Engine Design

The Design of the Engine

Overview

The engine is a component that processes problem-sized data. There are serial and parallel versions of the data. Like most components in VisIt, it is driven through RPCs, and those RPCs and are wrapped with a module, named EngineProxy.

The engine itself has a small code base. Instead, it offloads much of its work to the AVT library.

Modules

The primary modules in the Engine are:

  • Engine
  • NetworkManager
  • LoadBalancer
  • "Executors"
  • DataNetwork
  • Netnodes

Engine diagram.jpg

Engine

The Engine module does mostly management. It offloads all data processing to the NetworkManager. It's primary responsiblities are:

  • Initializing connections with the viewer
  • Communicating with the viewer (it does both the writing of big data and the writing of small progress reports)
  • Setting up Xfer objects and connecting them to RPCExecutors in Executors.h
  • The main event loop
  • Processing command line arguments and re-routing them to other modules.

NetworkManager

The NetworkManager is the link from the engine to the AVT library. Note that the phrase "Network" here refers to data flow networks, and is not meant to suggest things like socket communication. As RPCs come in from the viewer (OpenDatabase, AddOperator, AddPlot), the NetworkManager builds up a data flow network, which it stores in a DataNetwork module. The NetworkManager can manage multiple networks at a time, which is does through a vector of DataNetworks.

The NetworkManager also has code for rendering and querying, although the heavy lifting is ultimately offloaded to AVT modules.

Most changes to the engine module are done in the NetworkManager.

LoadBalancer

The LoadBalancer should probably be an AVT module. AVT modules inquire this module as to what portions of the larger data set they should be operating on. This is done through callbacks, since the AVT modules are not aware of the engine per se. More about load balancing can be found here.

Executors

Each RPC is connected to a specific method in Executors.h to respond to that RPC. The action of the method, of course, it tailored to the RPC and its arguments. It usually results in going to the NetworkManager.

DataNetwork

A DataNetwork contains a vector of Netnodes and provides convenience methods on that vector.

Netnodes

A netnode is an abstract type. Concrete, derived types exist for databases (NetnodeDB) and filters (NetnodeFilter). Each of these classes is connected to an AVT module and is involved with connecting the pipeline.

Additional topics

How the engine processes data

Pipelines

Databases

Parallel

Load balancing

Error handling

Global communication

Using MPI wrapper functions

Scalable rendering

Scalable rendering allows VisIt to render in software using multiple processors and then composite all of the images together and display the image instead of geometry in the viewer. Follow this link for more information on scalable rendering.