Getting the best performance when running in parallel

Database block count and engine processor count

The single biggest factor impacting VisIt performance is the number of blocks in the input database and how those blocks are assigned to processors for parallel computation. VisIt will assign the same number of blocks to process to each processor as best as it can. In an ideal situation the number of blocks is a multiple of the number of processors and each processor has the same number of blocks to work on. VisIt can handle any number of blocks with any number of processors with the only caveat that it may not be as efficient as possible or as fast as possible.

Going faster with more processors

More processors will be faster, right? Yes, but its important to understand the limitations here as well. The number of processors VisIt's engine can effectively use is primarily determined by the multi-block decomposition output by the data producer. If the input database is decomposed into K blocks, then VisIt can use no more than K processors effectively. Launching an engine with more than K processors will mean that the extra processors just idle. Also, certain processor communications will be slower due to additional unnecessary overhead due to the extra processors.

Overloading blocks to processors

VisIt can use fewer processors than blocks. Some processors will have more than one block. For example, if you have a mesh of 60 blocks, you can use VisIt on 60 processors where each processor gets one block, on 30 processors at 2 blocks per processor, on 20 processors at 3 blocks per processor or on 18 processors where 12 processors get 3 blocks and 6 processors get 4 blocks. If all blocks involve equal computational cost given the operations VisIt is trying to perform, then the last case would not be ideally load balanced. But, it still works fine.

If you are processing multiple databases within the same VisIt session and each has a different block count, this works fine. Just be aware that the databases with smaller block counts may wind up using less than all available compute resources.

Engine load balancing

By default, VisIt assigns blocks to processors in a round-robin fashion. In addition, as different operations in VisIt are able to cull blocks based on spatial or data extents, the assignment of blocks to processors can vary throughout a VisIt session. Different load balancing options are available as command-line options. From the VisIt usage, these options are described below. Depending on circumstances one of these options may lead to faster performance.

    Load balance options
       Note: Each time VisIt executes a pipeline the relevant domains for the
       execution are assigned to processors. This list of domains is sorted in
       increasing global domain number. The options below effect how domains
       in this list are assigned to processors. Assuming there are D domains
       and P processors...
       -lb-block            Assign the first D/P domains to processor 0, the
                            next D/P domains to processor 1, etc.
       -lb-stride           Assign every Pth domain starting from the first
                            to processor 0, every Pth domain starting from the
                            second to processor 1, etc.
       -lb-absolute         Assign domains by absolute domain number % P. This
                            guarentees a given domain is always processed
                            by the same processor but can also lead to poor
                            balance when only a subset of domains is selected.
       -lb-random           Randomly assign domains to processors.
       -allowdynamic        Dedicate one processor to spreading the work 
                            dynamically among the other processors.  This mode
                            has limitations in the types of queries it can 
                            perform.  Under development.
       -lb-stream           Similar to -lb-block, but have the domains travel
                            down the pipeline one at a time, instead of all
                            together.  Under development.

Dynamic decomposition

In some limited cases, VisIt does have the ability to perform on-the-fly parallel decomposition of an input dataset. So far, these cases are limited to a single, monolithic structured mesh object. For example, the Image and Basic NetCDF readers support this option. In this case, the database plugin does the work of decomposing the input mesh into pieces. For example, if a 3D structured grid of dimensions 2x8x60 is read from the netCDF reader onto 2 processors, the plugin will operate in such a way that each processors reads the entire mesh object and then selects out of that object the portion assigned to that processor and then throws the rest of what it reads away. So, I/O is by no means optimized in this situation but down-stream processing, and memory requirements of each rank thereafter is reduced.

Going faster with fewer blocks

An often very simple way to speed up VisIt performance is to turn off blocks. Maybe certain blocks are not relevant. Maybe you are planning on zooming in so far that many blocks are outside the field of view. Or, maybe you are setting up a complex session and need to see how changes in plot or operator attributes effect the final image before you are satisfied. You can turn blocks off to speed up your interactions and then turn them back on once you're happy with your settings. You can even move some blocks of a very large computation to your laptop and the use VisIt there to interact with the subset of blocks you have moved there. Here is an exampleSuppose the following dataset is generated on a large compute resource

-rw-------  1 miller86  3268  70425948 Mar 10 14:33 ucd3d0.silo
-rw-------  1 miller86  3268  70879820 Mar 10 14:33 ucd3d1.silo
-rw-------  1 miller86  3268  71163832 Mar 10 14:33 ucd3d2.silo
-rw-------  1 miller86  3268  71312792 Mar 10 14:33 ucd3d3.silo
-rw-------  1 miller86  3268  71310724 Mar 10 14:33 ucd3d4.silo
-rw-------  1 miller86  3268  71163644 Mar 10 14:34 ucd3d5.silo
-rw-------  1 miller86  3268  70872804 Mar 10 14:34 ucd3d6.silo
-rw-------  1 miller86  3268  70427956 Mar 10 14:34 ucd3d7.silo
-rw-------  1 miller86  3268     70522 Mar 10 14:34 ucd3d_root.silo

and the blocks of interest to you are contained in the file, ucd3d4.silo.

Here are the steps to move this dataset to your laptop and view it there.

  1. Copy the root file, ucd3d_root.silo, the file containing the first block, ucd3d0.silo and the file containing the block of interest, ucd3d4.silo to your laptop
    • Currently, you must also include the file with the first non-empty block because VisIt uses that block as part of the bootstrap to open the Silo database.
  2. Open the root file in VisIt
  3. Add the plot you want to plot but DO NOT HIT DRAW.
    • Also, don't have auto apply in effect
  4. Go to Controls->Subset and turn all blocks off by hitting the button for "All Sets" and selecting "Turn Off". Hit Apply
  5. Now hit Draw.
    • VisIt will read and display only those domains that are selected.
    • This works because VisIt will not attempt to open files or read domains that are not selected. So, even though those sub-files are not available on your laptop, VisIt doesn't care.
  6. For example, the image below shows blocks 1-10 and 148, 149 from the 288 block example database

Block selection.png

Using block selection in this way is often a great time-saving way to use VisIt to set up views, try various operator or expression settings, adjust lighting or rendering options, etc., and quickly/interactively observe the results. Then, once you've got all the settings the way you want, you can turn the missing domains back on and let VisIt work away on processing the whole dataset.

Other issues that impact Performance

There are many things that impact performance, ranging from the size of your visualization windows to a myriad of settings distributed across various portions of the interface. In this discussion, keep in mind that you could have, in the past, saved settings in such a way that they are now different from the defaults. If you did, VisIt will remember those settings and use them each time you start VisIt. Performance could be negatively effected.

Smaller windows are faster

Scalable Rendering and Volume Rendering are two operations in VisIt where some of the performance is dictated by the size of the visualization window. If you need to make a number of changes in settings that effect what is rendered or how it is rendered including viewing and lighting parameters, while in these modes it is best to make all these changes when the window is smaller. Once you are satisfied with all the settings you want, you can resize the visualizatoin window and it will re-render at the higher resolution.

Avoid starting VisIt in directories with large numbers of files

Whenever VisIt starts, it scans the startup directory (see below) for candidate files to open. If the startup directory contains a large number of files, this scan process can delay VisIt's timely initialization. If the filesystem containing the startup directory is itself bogged down due to other I/O activity or otherwise susceptible to delays, this too can cause VisIt startup to be slow. In particular, when the Lustre filesystem is unreliable, this can have the effect of causing VisIt startup to simply stall, waiting for the filesystem to respond to scan requests from VisIt. For this reason, it is best to avoid using a startup directory that contains a large number of files.

By default, the startup directory is the current working directory. However, there is a setting in VisIt that controls whether or not to use the current working directory. It is found in the File->Open… dialog, pictured below…

Open window.jpg

If you have ever in the past UNchecked this box and saved settings, VisIt will forevermore start using the directory to which this dialog was last pointing when settings were saved or, if that directory is not present, your home directory as the startup directory. This could mean that each time you start VisIt, it is constantly scanning and re-scanning a large directory.

Manage Directory History with the Remove Paths... Button

VisIt remembers various directories you have visited to open data files. This saves you having to type in or navigate through a long directory hierarchy. However, over time, the paths VisIt has saved in your directory history can become obsolete. You can remove obsolete or unneeded paths using the Remove Paths… button in the File->Open… dialog, pictured below.

Remove paths button.jpg

When you press this button, a directory history browser appears as shown on the right in the example below. . .

Remove paths example.jpg

Note that the browser shows all the paths available in the Path pull-down of the File->Open… dialog. To remove obsolete or unneeded paths, either click on them individually to select them or click and drag across a group of paths to select them. Them, hit the Remove button to remove the selected paths.

Turn off Auto apply

Auto apply is controlled by a small check box near the top of the main GUI panel as pictured below.

Auto apply button.jpg

For best performance, it should be disabled. When it is enabled, it has the effect of hitting the Apply button after each and every change to a setting, plot or operator. As you make a series of changes, you can wind up waiting for VisIt to execute each change and render an updated image. By default, Auto apply is not checked. That allows you to queue up a whole series of changes and then execute all of them in one Apply operation.

Most likely, the only time Auto apply is useful to you is when you are looking at small enough datasets that executions are fast and you need VisIt to behave more interactively than it does ordinarily.

Avoid setting scalable rendering to Never

Scalable rendering refers to VisIt's ability to render in parallel on the compute engine and send only the rendered image pixels back to the viewer. Scalable rendering controls are found under Options->Rendering... and then under the Advanced tab as shown in the screen shot below.

Sr mode pulldown.jpg

Scalable rendering has three settings; Always, Auto and Never. For best performance, user's should avoid the Never setting. In fact, for many scenarios, Never can easily lead to running the VisIt viewer out of memory. The Auto mode means that VisIt will automatically decide, based on dataset size, when to use scalable rendering.

In the very rare circumstances where a user is looking at small datasets (small enough to store the whole polygonal representation on the viewer), the Always mode can be slower than Auto or Never.

Faster opens by turning off database preferences

A portion of the preferences window is shown below for the preferences controlling database opens.

Visit db prefs.jpg

Checking any of these boxes can reduce performance of Opens. In addition, the three boxes that control generation of useful expressions can also slow VisIts GUI performance. This is particularly true for databases with large numbers of variables (say more than a couple hundred). Faster performance is possible by unchecking these boxes.

In particular, the check box that causes VisIt to try harder to get all cycles/times should be used only when absolutely necessary.

Turn off animation caching

Animation caching allows the viewer to cache all timesteps that have already been visited so that you can move through time quickly. However, it is a memory hog. When memory limits are approached or exceeded, it can significantly bog down performance. In addition, it cannot work when scalable rendering is in effect. Animation caching is off by default. It is accessed by going to Controls->Animation... as shown below.

Animation caching.jpg

Use transparancy for transparency

When users do a filled boundary (e.g. material) plot in 3D, they can wind up trying to use the plot's transparency options to hide some materials. Although this can sometimes work, it is not the most performant way to use VisIt. Removing some materials (or other subsets) from the view is what subset controls are for. Using transparency for this purpose can lead to significantly worse performance. The reason is that rendering with transparency requires a depth sort of the geometry to be rendered.

Forcing material interface reconstruction

There are various options that control how VisIt handles materials and material interface reconstruction (MIR). These are found under Controls->Material Options…

Mir options.jpg

One important option many users often have enabled by default is the force interface reconstruction. Forcing material interface reconstruction often improves the appearance of various plots. However, it can also degrade performance. So, use this option only when you know its essential to quality or accuracy of results.

Use Selected Plots to Quickly Toggle Views

A common situation is trying to see if some interesting feature in a plot of one variable lines up with (occurs on the same nodes/zones) as another variable. By carefully selecting plots in the plot list and setting their Hide/Slow status, you can easily toggle back and forth between the two variable's plots in a single button click. For example. . .

  1. Open noise2d.silo
  2. Put up a Mesh plot of Mesh
  3. Put up a Pseudocolor plot of hardyglobal
  4. Hit Draw
  5. Put up a Pseudocolor plot of airVf
  6. Hit Draw
    • Note that once the second Pseudocolor plot is drawn, it now totally obscures the first. See below for an example of changing the plot ordering.
    • We would like to be able to quickly toggle back and forth between the two Pseudocolor plots.
  7. Select the Pseudocolor plot for airVf in the plot list and hit the Hide/Show button.
    • The airVf plot is now hidden and the hardyglobal plot is visible.
  8. Now, select both the hardyglobal and airVf plots
    • With both plots selected a single click on the Hide/Slow button now quickly alternates between the two plots visibility.

Use Right Click Menu on Plot List

Did you know you can control the order of rendering by adjusting the order of plots in the plot list? Plots are rendered in top-to-bottom order. In 2D, this means plots near the bottom (e.g. last) in the list are rendered last. Pictured below, you can see the menu you get by right clicking on a given plot. . .

Right click plot.jpg

You can move the plot up or down in the plot list thereby allowing you to control the order in which the plots are rendered. This is most helpful in 2D settings. However, there may even be cases in 3D settings where adjusting the plot order may be useful. Note that there are several other things you can do to a plot including individually redrawing, clearing and cloning a given plot.

Using mxterms to grab and hold onto compute resources

What is an mxterm?

An mxterm is a general purpose Moab xterm script.

To our knowledge, the mxterm command is available only on LLNL LC systems. However, its conceivable the same processes can be applied elsewhere once the basic process of resource reservation is established.

An mxterm is the combination of an xterm window and a node reservation for a specified period of time.

Why use an mxterm?

Why are mxterms useful? They allow you to avoid having to re-join the queue and wait again for resources to become available if you need to re-run your jobs. In the context of VisIt, this means you can lanch and re-launch a parallel engine for VisIt without having to wait in the queue upon each engine launch. This is potentially a large time saver. In particular, if the VisIt engine crashes for any reason, this is an easy way to avoid having to wait in the queue again to launch a new engine.

How to use an mxterm


  1. Decide how many nodes you need and how long you need them
    • You may also need to consider which partition in which you will want to reserve those resources
  2. Launch the mxterm with appropriate options and wait for the xterm window to appear. For example
    • mxterm 1 8 30 -q pdebug
  3. Once the xterm window has launched, the resources have been reserved and you may now run and re-run VisIt using them.
  4. From the VisIt GUI, when you open a database, VisIt will present you with several options to launch the engine. If you are running on a system where VisIt installations are managed by the VisIt team, in all likelihood there is already a host profile defined for mxterm. Select the mxterm host profile and then be sure to set the number of processors as appropriate.
    • If you select more processors than your mxterm has, the engine will not launch due to resource violations in the Moab scheduler.
    • You may select fewer processors than you mxterm has. Note that if you launch multiple parallel engines within the mxterm (maybe you want multiple VisIt instances each with a parallel engine taking a portion of the mxterm reserved resources), there is no guarantee at present that each will be placed on different processors.
  5. From the VisIt command-line, you can launch the engine in the available mxterm processors using a command like the one below
    • visit -np <#procs> -l srun

Here are some examples of launching mxterm jobs

  mxterm <nnodes> <ntasks> <minutes> [<msub arg>]...
  Get 1 node with 8 tasks for 30 minutes in pdebug for workload myuse:
    mxterm 1 8 30 -q pdebug -l wckey=myuse

  Get 8 nodes with 128 tasks for 4 hours after 1pm today using mybank:
    mxterm 8 128 240 -a 1300 -A mybank

  On Blue Gene, <ntasks> is ignored, but some value (0) must be given for it:
    mxterm 2048 0 120 -q pdebug

  One command option, -g[eometry] widthxheight+-x+-y, may precede <nnodes>:
    mxterm -g 160x25+0-44 1 8 60 -l feature=16GB,gres=ignore,qos=standby

At NERSC (cori/edison)

  1. At NERSC, the command to reserve nodes is called salloc. You can read more about it here.
    • salloc does not start an xterm window for the nodes you've allocated. Instead it exec's a new shell in the current window.
  2. Here is an example salloc command for 2 nodes on cori in the debug partition for 30 minutes.
    • salloc -N 2 -p debug -t 00:30:00
  3. Once the salloc job has started, you can run VisIt with the command
    • visit -np <#procs> -l srun
    • Note that we have seen problems running the VisIt viewer in this mode that we think are related to certain X servers

Other aspects of mxterms

  • mxterms have time limits. You specify those limits when you start the mxterm. So, its important to be aware of those limits while running your instances of VisIt in them. If you start a session or activity too close to the time limit for the mxterm, you are likely to hit the time limit and then loose all the work you had done up to that point.
  • It is possible to use mxterms to abuse resources too. Try to avoid using mxterms to grab resources for an extended period if you don't intend to actually use those resources. Its best to give up (exit out of) the mxterm when you are done.
  • Currently, there is no way to run an engine in an mxterm using client/server mode.

Saving and restoring state in VisIt

There are several mechanisms to save state in VisIt to make it easier to save steps in getting VisIt to a particular state. These range from saving the default settings, to saving and restoring the entire state of VisIt, to saving and restoring the state of individual windows.

Saving the default settings

Saving the default settings saves all the default settings for all of the windows in VisIt. This includes the size and position of every window and if the window is visible, iconified or posted. It will also save the number of visualization windows and their location and size. If a window has a default setting it will save the default settings rather than what may currently be shown in the window. This primarily applies to Plot and Operator attribute windows. Saving your settings will not save any plots in your visualization windows and will not save any information about open databases.

To save settings go to Options->Save Settings. This will save the settings in your .visit directory. If you are on a Linux system this will be in your home directory. If you want to restore VisIt back to the default settings that come with a new installation, you will need to delete the config and guiconfig files in that directory. You could also rename the files to something else in case you didn't want to permanently remove the files. Sometimes you can get in a state where VisIt crashes at startup. To see if this is caused by a corrupt configuration file you can start visit with the -noconfig option if starting VisIt from a command line prompt.

Saving and restoring the entire state of VisIt by saving and restoring sessions

Saving your session will save the entire state of VisIt and will return it to the exact same state as it was when you saved the session when you restore the session. It primarily differs from saving settings in that it saves all the plots and knowledge of any open databases. When you save and restore sessions VisIt will bring up a dialog that will allow you to navigate the file system to find the session file and will be default look in the directory that you started VisIt in. When you restore your session you have the option of restoring the session as is or restoring the session with sources, where you can replace the databases that are open and that were used in the plots with different ones. If you restore a session with sources the new databases should have same variables, number of blocks, number of material, etc. or it may generate error messages when trying to recreate the same state.

To save your session go to File->Save session to save your session with the same session name as previously used. VisIt displays the current session name in the title of the main control panel. VisIt will overwrite the current session file without prompting you. If you want to save the current session with a new name use File->Save session as.... To restore a session go to File->Restore session.... To restore a session with sources go to File->Restore session with sources....

Saving and restoring the state of a window with the Save and Load Buttons

Plot and operator attribute windows also support Save and Load buttons as pictured below. . .

Save load buttons.jpg

These buttons allow you to save a plot's or operator's attributes to an xml file that you can be re-loaded at another time. These xml files can easily be shared with collaborators. Here is an example. . .

  1. Open the file noise.silo.
  2. Create a Pseudocolor plot of hardyglobal.
  3. Go to Operators->Slicing->Isosurface... to add an Isosurface operator.
  4. Bring up the Isosurface operator attributes window and change the number of levels to 5.
  5. Bring up the Pseudocolor attributes window and set the attributes as follows.
    • Set Minimum to 0.5.
    • Set Maximum to 4.0.
    • Set Color table to rainbow and check Invert.
    • Set Smoothing to High.
    • Click Apply.
  6. Click Save.
    • It should open the file browser in the directory in which you started VisIt.
    • Choose a file name to save the settings to.
  7. Exit and restart VisIt and open noise.silo.
  8. Create a Pseudocolor plot of hardyglobal.
  9. Add an Isosurface operator with 5 levels.
  10. Click Draw.
  11. Bring up the Pseudocolor attributes window.
  12. Click Load.
    • Select the file to load.
  13. Click Apply and the plot will update to the loaded settings.

Automating tasks with macros

Macros can be used to automate tasks that are performed on a frequent basis.

Let's create a macro to automatically create the following image.

Llnl tutorial time savers macros1.png

  1. Open the file noise.silo.
  2. Turn off Apply operators to all plots on the main control window.
  3. Go to Controls->Command... to bring up the Commands window. Make sure you are in an empty numbered tab.
  4. Click on the Record button to start recording the GUI actions. The CLI window will pop up and VisIt will now start recording all your GUI actions. You can either iconify the CLI window or move it out of the way.
  5. Llnl tutorial time savers macros2.png

  6. Create a Pseudocolor plot of hardyglobal.
  7. Go to Operators->Slicing->Isosurface... to add an Isosurface operator.
  8. Bring up the Isosurface operator attributes window and change the number of levels to 5.
  9. Create another Pseudocolor plot of hardyglobal.
  10. Bring up the Pseudocolor plot attributes window and turn off the legend.
  11. Go to Operators->Slicing->ThreeSlice... to add a ThreeSlice operator.
  12. Bring up the ThreeSlice operator attributes window and change the X, Y and Z slice coordinates to -10..
  13. Click Draw.
  14. Adjust the view until it approximates the image above.
  15. Click on the Stop button in the Commands window to stop recording the GUI actions. Python code to replicate the actions you performed will appear in the first tab of the Commands window.
  16. Llnl tutorial time savers macros3.png

  17. It is always a good idea to test your Python code while it is in one of the numbered tabs before creating a macro. Let's go ahead and do so now. Reset the view in the window and delete all the plots from the window. Click on the Execute button to execute the code in the tab. Your plots should get recreated.
  18. Llnl tutorial time savers macros4.png

  19. Now let's turn the code in the tab into a macro. Click on the Make macro button in the Commands window. You will now be prompted for the name of your macro. Enter Pseudocolor isosurface threeslice and click Ok. This will copy the code in the current numbered tab into the Macros tab wrapped in Python code to make it into a macro. The code should not need modification and will be properly indented.
  20. Llnl tutorial time savers macros4b.png

  21. At this point you will need to save the macro definitions by clicking the Update macros button in the Commands window.
  22. Llnl tutorial time savers macros5.png

  23. Now you can try out your new macro. Once again, reset the view in the window and delete all the plots in the window. Go to Controls->Macros... to bring up the Macros window. Click on the Pseudocolor isosurface threeslice button to execute the macro. The plots should get recreated.
  24. Llnl tutorial time savers macros6.png

Code in the numbered tabs and macros will be automatically saved by VisIt. There is no need to save settings.

When you bring up VisIt in a future session, the CLI window will automatically popup. You can either iconify the CLI window or move it out of the way.

Content Notes

These notes will be replaced with real tutorial content as we fill it in

  • Not starting VisIt in a directory with many files
    • Eric to find prefs for this that doesn't look at dirs
      • not show files and dirs in sep lists
  • Selected files window
    • It is a better interface
    • Save's space on the main panel
    • have to get used to active source
  • Dir history in open/selected files
    • remove paths button
    • does history get saved always, or is save settings required
      • is there a pref. that controls that?
  • apply selection and operators to all plots
  • Client/Server
    • best way to run VisIt
    • better on poor networks
    • rendering improvement
    • is there a way to reserver resources (like mxterm) that you can then launch the engine
      • Mark to file a ticket about this
  • Default settings for various plots/operatos
    • down side is its your default goin forward
    • How decide between "Make Default" and "Save/Load"
  • Multi-window usage
    • cloning
    • locking buttons views and times, operator attributs
    • copy plots? clone a window