Silo Multi Mesh Adjacency Object

Using Silo's Mulitmeshadj Object

(Work in progress -Cyrus)

Silo's Multimeshadj object provides a downstream performance optimization that allows VisIt to generate ghost zone data for structured grid meshes. For a detailed overview on the Multimeshadj object, please see the DBPutMultimeshadj documentation in the Silo User's Guide. This wiki page provides a simple example to supplement the info in the Silo User's Guide.

Multimeshadj Object Example

To create a Multimeshadj object we need to fill several arrays with info to describe how domains abutt. We will construct multimeshmadj object for an 8x8 zone rectangular grid, split into 4 domains (2 in each logical dimension):

Mmadj example.png

Required Arrays

To create the object using DBPutMultimeshadj we need to construct the following arrays:

Array Description Size
mesh_types Specifies type of each domain.

(For our example mesh all domains are rectilinear, so creating this array is trivial)

(# of domains)
nneighbors Number of neighbors for each domain.

(The sum of all of the values in the nneighbors defines the number of nodelists we need to create.)

(# of domains)
neighbors The domain index of each neighbor. (packed for all domains) (# of nodelists)
back The local index of each domain in its neighbor's nodelists. (packed for all domains) (# of nodelists)
nnodes The size of each nodelist. For structured grids every nodelist is 15 integers long.

(packed for all domains)

(# of nodelists)
nodelist_ptrs Pointers to the beginning of each nodelist. (# of nodelists)
nodelists Node adjacency info for each domain seam. (packed for all domains) (# of nodelists) * 15

Our example mesh has a very simple domain neighbor list:

Domain Neighbors
0 1,2,2
1 0,2,3
2 0,1,3
3 0,1,2

Note that they share a single node

From this we can create the nneighbors, neighbors, back & nnodes arrays:

nneighbors = {3,3,3,3}
neighbors  = {1,2,3 0,2,3 0,1,3 0,1,2}
back       = {0,0,0 0,1,1 1,2,2 2,2,2}
nnodes     = {15,15,15 15,15,15 15,15,15 15,15,15}

Note the implicit packing order: entries for domain 0, followed by entries for domain 1, ... followed by entries for domain n. The back array construction is covered in more detail in the MI pointer section of the Domain Boundaries page.

Nodelist Construction

The node lists are the most difficult component to construct, we need to create a total of 12 - three for each domain. Lets take a close look at the info we need to pack into each list.

For the structured grid case each node list contains 15 integers:

Entry Details Size
Logical node extents of the current domain Logical block in interleaved order: i_min,i_max, j_min,j_max, k_min,k_max

(For our 2D example we will use k_min,k_max = -1,-1)

(6 values)
Logical node overlap between the current domain and the neighbor Logical block in interleaved order: i_min,i_max, j_min,j_max, k_min,k_max

(For our 2D example we will use k_min,k_max = -1,-1)

(6 values)
Logical orientation 1,2,3 specifies i->i, j->j, k->k.

If you have reduced or enhanced connectivity zones you may need to adjust these.

(3 values)

To calculate the overlapping nodes we need to understand the logical blocks of zones and nodes for each domain.

Mmadj domain logical blocks.png

Note that it is easy to obtain the logical block of nodes from corresponding logical block of zones by adding one the upper extent in each dimension. Lets create the first nodelist, which describes the node overlap between domain 0 & 1.

Fill the first six values with the interleaved logical node extents for domain 0:

0,4 0,4 -1,-1

For the next six we calculate the node overlap between these two domains by noting they overlap in logical j:

4,4 0,4 -1,-1

The orientation between these two domains is identical so we fill the next three values with:

1,2,3

Here is our first complete node list:

0,4 0,4 -1,-1 4,4 0,4 -1,-1 1,2,3


Lets complete the remaining nodelists:

Domain Domain's Logical Node Block Neighbor Domain Logical Node Overlap Node List Values
0 0,4 0,4 -1,-1 1 4,4 0,4 -1,-1
 0,4 0,4 -1,-1  4,4 0,4 -1,-1  1,2,3
2 0,4 4,4 -1,-1
 0,4 0,4 -1,-1  0,4 4,4 -1,-1  1,2,3
3 4,4 4,4 -1,-1
 0,4 0,4 -1,-1  4,4 4,4 -1,-1  1,2,3
1 4,8 0,4, -1,-1 0 4,4 0,4 -1,-1
 4,8 0,4 -1,-1  4,4 0,4 -1,-1  1,2,3
2 4,4 4,4 -1,-1
 4,8 0,4 -1,-1  4,4 4,4 -1,-1  1,2,3
3 4,8 4,4 -1,-1
 4,8 0,4 -1,-1  4,8 4,4 -1,-1  1,2,3
2 0,4 4,8 -1,-1 0 0,4 4,4 -1,-1
 0,4 4,8 -1,-1  0,4 4,4 -1,-1  1,2,3
1 4,4 4,4 -1,-1
 0,4 4,8 -1,-1  4,4 4,4 -1,-1  1,2,3
3 4,4 4,8 -1,-1
 0,4 4,8 -1,-1  4,4 4,8 -1,-1  1,2,3
3 4,8 4,8 -1,-1 0 4,4 4,4 -1,-1
 4,8 4,8 -1,-1  4,4 4,4 -1,-1  1,2,3
1 4,8 4,4 -1,-1
 4,8 4,8 -1,-1  4,8 4,4 -1,-1  1,2,3
2 4,4 4,8 -1,-1
 4,8 4,8 -1,-1  4,4 4,8 -1,-1  1,2,3

Re-Design of Nodelist Handoff from Silo plugin to VisIt Proper

The nodelists defined by a block-structured code as described above are interpreted in the Silo plugin and then converted to a node-centered variable that is painted onto the mesh. It is this mesh variable that is handed off (e.g. returned to) VisIt proper from the plugin. The value of the variable at each node is used to indicate each node's membership in various of the nodelists defined in the input database.

In its original design, it was assumed that this node-centered variable needed to behave more or less like a normal scalar mesh variable such as a integer-valued vtkDataArray. For example an integer value at each node could be used as a bit-field to indicate the node's membership in at most 32 nodelists. But, this is too constraining and leads to a very strict upper limit on the total number of nodelists defined in the input database or, at the very least, a strict (as well as difficult to predict) upper limit on the number of coincident nodelists any given node can be a member of. Furthermore, the assumption that the node-centered variable we create needs to be a normal scalar mesh variable is also not necessary.

Instead, in our re-design, we will create a vtkDataArray of either bit, int or long long type with a number of components defined to the number of nodelists in the associated enumerated scalar defined in the database metadata. We will use this vtkDataArray as a per-node bit field to indicate each node's membership in various of the nodelists. For a vtkBitArray, the number of components would be equal to the number of nodelists. But for a vtkIntArray or vtkLongLongArray, the number of components would be the number of nodelists divided by sizeof(int)*8 or sizeof(long long)*8 respectively. For example, for each node, we might have bit-fields such as...


Node Bit-Field Value
0 0011000010100111000101000000...
1 0011001000100111000101000000...
2 0011001000100111000101000000...
3 0011001110100111000101000000...
4 0011001000100000000101001100...

For each node, a 1 in bit-position i in the node-centered variable indicates the node is a member of nodelist i in the associated enumerated scalar variable. The particular set of nodelists that are on in the current SIL selection would be represented by another bit-field, a selection mask, indicating the on/off state of each corresponding nodelists in the current SIL selection. When the node-centered variable indicating nodelist membership is returned from the plugin to avtGenericDatabase, the selection mask is then applied by a bit-wise and operation to each node's value. The process of applying the selection mask is handled down in the bowels of vtkEnumThreshold with some enhancements to the way its ByBitMask enumeration mode works. Currently, vtkEnumThreshold is coded to assume a 1-component float, int, double kind of data array. We will have to adjust it to handle a many component vtkBitArray for the ByBitMask enumeration mode.

VisIt's Silo Plugin Support for Multimeshadj Objects

VisIt's Silo Plugin currently only supports a single Multimeshadj object, and it must exist in the root file @

Decomposition/Domain_Decomposition

It also requires an integer variable containing the number of domains @

Decomposition/NumDomains

Example Source

The full source for this example is located here: mmadj_rect_2d.C

The example creates the following files:

mmadj_rect_2d_root.silo (Open this file in VisIt)
mmadj_rect_2d_data/domain000.silo
mmadj_rect_2d_data/domain001.silo
mmadj_rect_2d_data/domain002.silo

Example Plots

To check if the Multimeshadj object valid, open mmadj_rect_2d_root.silo & create valid ghost zones create a Pseudocolor Plot of "value":

Mmadj value plot zonal.png

Change the centering of the Pseudocolor plot to "nodal" & you should see blending across the domains:

Mmadj value plot nodal.png

You can also use the inverse ghost zone operator to view the generated ghost zones:

Mmadj value plot nodal gz.png