Domain Boundaries

VisIt uses ghost data at the boundaries between different domains when it needs more data to get interpolation of values correct. When ghost data are available, VisIt can eliminate discontinuities that would otherwise cause seams in the dataset. Ghost data can be provided to VisIt in various ways. For example, the mesh can be expanded by 1 or more layers of zones along domain boundaries and those zones can be designated as ghost zones. To create ghost data in that manner, the code generating the data must create the extra mesh zones and write the appropriate values into those zones. For many simulations, this is a difficult challenge.

Another approach that can be used is a domain boundaries object, which informs VisIt as to how domains abut. VisIt can then use the domain boundary information to automatically create ghost zones, performing whichever data exchanges and communication are required to populate the ghost data arrays. This page will describe how one can provide domain boundary information to VisIt. The context used herein is that of a simulation that is using libsim to return data to VisIt though the patterns are broadly applicable to database readers that also return avtDomainBoundaries objects to VisIt.

Code examples:

This topic is also covered by the Silo Multi Mesh Adjacency Object page.

avtDomainBoundaries

The avtDomainBoundaries object is the base class that VisIt uses for representing domain boundaries. There are subclasses for structured, rectilinear, and unstructured types that let VisIt communicate boundary information for different mesh types.

VisIt_DomainBoundaries

In Libsim, the avtDomainBoundaries objects are exposed via the functions that begin with VisIt_DomainBoundaries. Libsim simply wraps the avtDomainBoundaries so the libsim functions corrspond 1:1 with methods that you would call on the AVT object itself.

Regular boundaries

When domains divide space uniformly in a structured mesh, you can call the VisIt_DomainBoundaries_set_rectIndices method to specify the extents for each domain in the mesh. Domains must overlap completely on edges and edges can only meet at domain corners. Look at the GetDomainBoundaries function in the domainbounds.c example simulation to see how regular domain boundaries are created.

Irregular boundaries

Domains are not required to divide space uniformly or touch only along nice, aligned edges. Domains can be arranged to form reduced or enhanced connectivity points or even T-junctions. In this case, you must manually specify the domain boundaries for each domain so VisIt will know how the domains are arranged.

Manually specifying domains means that you must understand how to tell VisIt how domains abut in more detail. This example shows how to create domain boundaries for a set of domains that use enhanced and reduced connectivity points. The node shared by domains 0,1,2 is a reduced connectivity point since only 3 domains instead of the typical 4 domains for a 2D structured grid contain the node. Likewise, the node shared by domains 1,2,3,4,5 is an enhanced connectivity point because more than the typical 4 domains contain the node.

DomainBoundaries domains.png
The entire mesh divided into 6 domains. The domain number labels the mesh.

Per domain boundaries

These images show each domain from the original mesh surrounded by a layer of zones from the adjacent domains. The layer of zones from the adjacent domains indicates where ghost zones will be created and they are colored using pastelle versions of each domain's color from the original picture.

Each edge between domains will produce 2 boundary cases in the domain boundaries object. For instance, the edge between domain 0 and domain 1 has 2 domains involved. There is the part of domain 0 that will be mapped across the boundary onto domain 1. Also, there is the part of domain 1 that will be mapped across the boundary onto domain 0. The size of the shared boundaries for the 2 domains must match.

DomainBoundaries domain0.png DomainBoundaries domain1.png DomainBoundaries domain2.png
Domain 0's boundaries Domain 1's boundaries Domain 2's boundaries
DomainBoundaries domain3.png DomainBoundaries domain4.png DomainBoundaries domain5.png
Domain 3's boundaries Domain 4's boundaries Domain 5's boundaries

Encoding boundaries

Boundaries are typically put into a stream of 11-integer tuples with the following layout:

struct
{
    int neighbordomain;
    int mi;
    int orientation[3];
    int extents[6];
};

The neighbordomain value is the index of the neighboring domain, in other words: its 0-based domain number.

The mi value is an index to where the boundary for the current domain falls in the other domain's boundary list. You can think of each domain as having a list of boundaries and the mi value is a "pointer". We'll write the boundary table like this:

  Domain 0 
    1, mi, ...
  ...
  Domain 1
    0, ...

You'd compute "mi" for the 1st boundary in domain 0's list by saying: "Domain 0 in domain 1's list appears at which index?". Of course, the value is 0 in this case because the boundary for domain 0 is the first one in domain 1's boundary list.

The orientation values can specify which axes are first. For X,Y,Z, use 1,2,3.

The extents consist of 6 values (loX, hiX, loY, hiY, loZ, hiZ) that indicate where the cells in the neighbor domain touch the current domain. All of the extent values for a boundary will be relative to the domain that owns the boundary in its boundary list. Each boundary must have an equally sized boundary in the neighbor domain's boundary list. This means that each time you add a boundary, you'll need to add it for 2 domains.

DomainBoundaries extents.png

For a complete encoding of the example mesh, see the multiblock.c example simulation.

MI pointer

The MI pointer is a pointer that relates the boundary cases in the boundary lists for each domain. This example shows how the MI pointer points to the corresponding boundary case in other domains. The MI pointer is the index into the other domain's boundary list where the corresponding case is found.

DomainBoundaries encoding.png

Neighbor table

Once you have created the boundary lists for each domain, you can simply combine them into a larger table. Collectively, this is the neighbor table that is used to pass domain boundary information to VisIt.

DomainBoundaries neighbor.png

Providing boundaries

Once you have a table of 11-tuples describing the domain boundaries, this can be translated easily into a domain boundaries object.

  1. Create the domain boundaries object
  2. Set the number of domains
  3. Foreach domain:
    1. Set the domain extents (its size in number of zones)
    2. Add the neighbors from the table
    3. Call finish on the domain

AVT

If you are creating a database reader plugin, you can pass the neighbor table to one of VisIt's avtDomainBoundaries subclasses using code like the following:

void SetDomainBounds(avtStructuredDomainBoundaries *dbi,
    int ndomains, int *ext, int *nneighbors, int *neighbors)
{
    dbi->SetNumDomains(ndomains);

    int *tuple = neighbors;
    for (int j = 0; j < ndomains; j++)
    {
        dbi->SetExtents(j, &ext[j*6]);
        for (int k = 0; k < nneighbors[j]; k++)
        {
            // neighbors 11-tuple
            dbi->AddNeighbor(j, tuple[0], tuple[1],
                                tuple + 2, tuple + 5);
            tuple += 11;
        }
        dbi->Finish(j);
    }
}

Libsim

If you are using libsim to instrument a simulation, you can use the following functions to pass the neighbor table to VisIt in the simulation's GetDomainBoundaries callback function:

  • VisIt_DomainBoundaries_alloc / VisIt_DomainBoundaries_set_type
  • VisIt_DomainBoundaries_set_numDomains
  • VisIt_DomainBoundaries_set_extents
  • VisIt_DomainBoundaries_add_neighbor
  • VisIt_DomainBoundaries_finish

The code to turn the 11-tuple table into domain boundaries that VisIt can use looks like:

visit_handle
SimGetDomainBoundaries(const char *name, void *cbdata)
{
    visit_handle h = VISIT_INVALID_HANDLE;
    simulation_data *sim = (simulation_data *)cbdata;

    /* Note: VisIt currently relies on domain boundaries objects that hold the
     *       information for all domains in the problem, even ones not held
     *       on this processor. We're working to correct that. In the meantime,
     *       we construct the global domain boundaries info for each domain.
     */
    if(VisIt_DomainBoundaries_alloc(&h) != VISIT_ERROR)
    {
        int dom, k, offset = 0;

        /* Set the type. 0=rectilinear, 1=curvilinear. */
        VisIt_DomainBoundaries_set_type(h, 1);

        /* Set the total number of domains in the problem. */
        VisIt_DomainBoundaries_set_numDomains(h, NDOMAINS);

        for(dom = 0; dom < NDOMAINS; ++dom)
        {
            /* Number of zones in this domain. */
            int extents[6];
            extents[0] = 0;
            extents[1] = mesh_dims[dom][0]-1;
            extents[2] = 0;
            extents[3] = mesh_dims[dom][1]-1;
            extents[4] = 0;
            extents[5] = mesh_dims[dom][2]-1;

            /* Set the domain's extents. */
            VisIt_DomainBoundaries_set_extents(h, dom, extents);

            /* Add the domain's neighbors. */
            for(k = 0; k < nneighbors[dom]; ++k)
            {
                const int *n = neighbors + offset;
                VisIt_DomainBoundaries_add_neighbor(h, dom, n[0], n[1],
                                                    &n[2], &n[5]);
                offset += 11;
            }

            /* We're done adding neighbors for this domain. */
            VisIt_DomainBoundaries_finish(h, dom);
        }
    }

    return h;
}

Interpolation results

This example shows what happens when the zonal domain number is interpolated to the nodes when domain boundaries are used to generate ghost zones.

DomainBoundaries zonal.png DomainBoundaries nodal.png
No interpolation Interpolation to the nodes