Additional in-situ work

VisIt currently interfaces to simulations via libsim and the SimV2 reader. Overall, SimV2 represents a significant improvement over SimV1 but there are additional features that could be added to make it work better. This page is meant to list some proposed new features and to eventually become a page for tracking progress on implementing new features.

Potential features

Make it easy to set up plots and save images of them while the simulation works -- without VisIt being connected. The current setup requires you to connect to the sim with VisIt and then you are free to make whatever plots you want as you drive it interactively. Some projects just want to dump images automatically while the sim churns. We don't really support this yet and what we do have once VisIt is connected is clumsy. For instance, I added a VisItSaveWindow (or something like that) that is supposed to save the current visualization to an image file. There's not a lot in terms of functions that can be called to set up the vis in the first place: VisItAddPlot("pressure") and so on. Putting an interface like that into place would be a good thing since it would let sims set up their own visualizations and save the pictures automatically. Our compute engine "API" does not make this easy so there's some work there. Our Python interface for VisIt is workable so should this sim API be similar but in C?

libsim uses runtime libraries for the current version of VisIt instead of for the version of VisIt that's trying to connect. This doesn't come up that much if folks use the current version of VisIt too. However, when linking a sim to 2.x.x's SimV1 or SimV2 libraries, often the user will think he can only use that version of VisIt to connect to the sim. Not so. What happens is that 2.x.x will load the most recent VisIt's sim runtime libraries. If they differ from the version of VisIt that the user ran then the sim can't connect and the user wonders why. It would be nice if the sim was able to load the runtime libraries for the version of VisIt that's connecting. This isn't that important though.

The SimV1/2 readers in VisIt end up copying the memory that gets passed to them. This might not always be necessary -- it's just safer. There are various other places in VisIt that could benefit from a smaller memory footprint. As an example, when I ran VisIt on dawn last summer there was just 1Gb of memory and just loading VisIt seemed to take about 300Mb of it. Another structure for domain connectivity pretty much sucked up the remaining memory. My comment is that sims are going to be more and more memory-constrained so we'll have to fit in a smaller space if we want to piggyback on them. One possibility might be to have a libsim or something like that can ship data to other nodes that do the vis.

Missing features

This table lists some features that should appear in the next iteration of libsim. These are features that have been requested by VisIt users.

Features Status
Ghost zones returned as a field on a mesh Done for VisIt 2.4.0
GetDomainList callback does not pass the name of the mesh, preventing simulations from having more than one domain per processor - or other funky domain decompositions Done in 2.9.0
libsim on Win32 Done in 2.2
C interface for calling individual VisIt queries from the simulation. Additionally, it would be nice if opening the GUI were not required for this. For example a simulation running in batch mode, time dependently querying VisIt. Not started
Extend CSG mesh support so individual zones can be given names. Not started
It would be nice if simulations could use VisIt's export facility to write files. Done in 2.9.0
Python bindings for libsim (Python calls C) Done in 2.3.0
Java bindings for libsim (Java calls C) Started
Support for variables defined on part of a mesh (history variables) Done
Productize custom UI's (finish it, document it) I fixed up the code and rewrote some of it so I can do sim ui's. I don't support all widgets quite yet. I created an example for the mandelbrot example.
Add ability to pass global zone and node ids for the mesh Done in 2.9.0
Add callback that lets simulations create VisIt's communicator. The reason is that sims often have custom communicators and not just COMM_WORLD that VisIt has always used. Thus, a sim must restrict its VisIt operations to the same set of processors as the simulation communicator. Ouch! This will be a can of worms! This is handled via VisItSetMPICommunicator
C simulations are quitting on exit because of missing exception handling code. Linking sims with libstdc++ gets rid of the problem but we should see whether we need to change how we link the simV2 runtime or maybe remove C++ from it. Not started

Debugging Simulations

On Windows

You can set a simulation as your startup project in Visual Studio and set breakpoints in it. Run an installed version of VisIt and connect to the simulation Perform whichever VisIt actions will get you to your breakpoints

If you're more interested in debugging the viewer side of things then you can run the simulation from the windows shell command line. Then you can run the installed version of VisIt, which you built with debugging information. Attach to the viewer process in Visual Studio and set the breakpoints you want. Next, open the .simv2 file in VisIt so VisIt connects to the simulation. Then perform whichever VisIt actions will get you to your breakpoints.

valgrind

Sometimes it is useful to run valgrind on a simulation to look for memory errors. Here is a useful command line for debugging simulations in the tools/DataManualExamples/Simulations directory.

valgrind --tool=memcheck --error-limit=no --num-callers=50 --leak-check=full \
  --log-file=foo.txt mandelbrot -dir ../../..

Development Efforts

This section describes some current in-situ development efforts

Porting to Windows

An initial port of SimV2 has been checked into VisIt's repo. Further changes on a Windows development machine have been made to VisIt, SimV2, and example simulations that allow VisIt to connect to the example simulations successfully and render data from them. The most significant change is that the engine/engineproxy are now creating a new socket for asynchronous state when a simulation gets connected. Since this is handled by the runtime library, it should be safe. This change was made because we can't use the same socket for RPC's and socket notifiers on Windows since it ends up causing a lost connection exception.

  • These changes fixed lost connection problems on Windows
  • These changes also fixed socket deadlocks on Mac

SWIG Bindings

I decided to try SWIG to generate bindings for Python and Java. I have a branch that can be checked out using:

co_branch SWIGBindings src -user whitlocb

I created simple .i files that tell swig what to do. I then generated Python and Java bindings but I have so far only concentrated on Python.

After having completed the Python wrapper, I became aware of a typemaps.i file that can be included into the simV2.i file that could aid in how SWIG generates bindings for pointer types like float * and so on. There is also some support for handling output arguments like getting a handle from the _alloc() routines. Since I've already created a filtered subset of the functions that I want to expose from the library, that representation could be further augmented with INPUT, OUTPUT, INOUT to help SWIG. That seems like it could reduce the amount of additional typemaps I need to worry about when creating a binding for additional languages.

Python

I generated bindings and started coding up a simple example. I was able to dump a .sim2 file and get VisIt to connect to the Python sim. After adding a lot of typemaps and some thunk code to handle user-defined callbacks written in Python, the data interface can return meshes, variables, and it can execute callbacks.

Todo:

Task Status
Hide writer callbacks Done
Hide certain get methods Done
Try updateplots with mpicomm to make a parallel version Done
Write materials example Done
See if I can pass class methods as callback functions. If so, make it work. Done
Make libsimV2 module get installed to the right place Done

Java

On Ubuntu, I needed to install sun-java6-sdk package so I could get a Java compiler and the required jni.h include file. This required a ton of other packages to update as well.

CMake has a FindJNI.cmake module that I put into VisIt's top level CMakeLists.txt. When JNI is found, I will enable compilation of the JNI bindings to libsim.

I have used SWIG to generate Java bindings to libsim but they don't look like they will be very useful because of the same problems experienced with Python. Namely, our use of callback functions requires SWIG typemaps in order to produce any sensible wrapping code. In Java, I'm not even sure what we can do as far as callbacks.

Idea

Maybe something like int VisItSetGetMesh(int (*cb)(int, const char *, void*), void *) becomes something like int VisItSetGetMesh(IntStringObjectCallback obj, Object).

from simV2 import *;

public interface IntStringObjectCallback
{
    // entry point for C to call into Java
    public visit_handle execute(int arg1, String a2, Object a3);
};

public class GetMesh implements IntStringObjectCallback
{
    public visit_handle execute(int domain, String name, Object cbdata)
    {
        visit_handle h = VISIT_INVALID_HANDLE;
        if(VisIt_RectilinearMesh_alloc(h) == VISIT_OKAY)
        {
            // Can we do this?
            Simulation sim = (Simulation)cbdata;

            // fill it in
        }

        return visit_handle;
    }
};

public class Simulation
{
    public int mainloop()
    {
        while(true)
        {
            int status = simV2.VisItDetectInput(1, -1);
            if(status == 0)
                SimulateOneTimestep();
            else if(status == 1)
            {
                if(simV2.VisItAttemptToCompleteConnection())
                {
                    // Create new GetMesh object, passing "this" as the callback data.
                    VisItSetGetMesh(new GetMesh(), this);
                }
            }
        }
    }
};

In the C wrapper side of things, we'd have a thunk function that we'd register with VisItSetGetMetaData and it would take the input GetMesh() object and get a handle to it. From there, we could probably use some JNI functions to get a pointer to its execute function that it defined via the IntStringObjectCallback interface. Then we'd create temp objects to pass to it, wrapping the domain and name passed into the thunk function.

// Libsim calls this function 
int
jnilibsim_int_pconstchar_pvoid(int domain, const char *name, void *cbdata)
{
    // Suppose cbdata is a void*[2] if we set it up that way in the wrapper function
    void *jclass = ((void **)cbdata)[0];
    void *jcbdata = ((void **)cbdata)[1];

    // Convert domain, name to Java int and String objects that we can pass to the Java callback

    // Dispatch the (domain, name, jcbdata) arguments to jclass' execute method

    // return the value that was returned
}

Simulation User Interfaces

VisIt once had some fragile support for simulation UI's but it never worked all that well. I've taken an initial stab at improving it and making it work again. This work has been merged into the trunk but there are still tasks to do to finish it.

Todo:

  • Add support for more signals/slots
  • Make the sim transmit the contents of the ui file to the local client ~/.visit/ui directory. We can probably leverage the mechanism where the sim sends a command to the viewer.

Setting Communicators

MPI simulations can do very exotic things with communicators. VisIt has been limited to using a duplicate of MPI_COMM_WORLD. We need the simulation to be able to install a communicator for VisIt to use.

int VisItSetMPICommunicator(void *mpicomm);

This functionality has been coded up on a branch called SimCommunicators but it is not known whether it works. There are likely other implementation concerns to deal with and there may be places in VisIt that are not using the VISIT_MPI_COMM communicator, which is a pointer to a communicator. The new VisItSetMPICommunicator function changes that pointer to an arbitrary communicator used by the simulation. Passing NULL for the communicator changes the internal pointer back to VisIt's duplicated copy of MPI_COMM_WORLD.

If the communicator that the simulation uses spans all of the ranks then it might be safe to just call VisItSetMPICommunicator once. VisItSetMPICommunicator will install a new communicator immediately if the VisIt runtimes have been loaded. Otherwise, the communicator pointer is cached in the control interface and the communicator gets installed when the VisIt runtimes are initialized.

If the communicator that the simulation uses varies over time then VisItSetMPICommunicator might need to be called many times when there is actual VisIt work to be done. The broadcast of commands from the viewer to other ranks should continue to use a communicator that involves all ranks -- especially COMM_WORLD's rank 0 since that is the only processor connected to the viewer. Before and after VisIt work is being done, it may be possible to install a temporary communicator and then revert back to the original once the work is done.