Leveraging VisIt in Sim Code Regression Testing

Intro

Starting with VisIt 2.7.0, VisIt's regression testing infrastructure was exposed as an easy to use python module. Developers of simulation codes at computing centers with VisIt installed can now use this python module to write regression tests that incorporate the rendering and analysis capabilities of VisIt.

This page provides a quick start tutorial to help simulation code developers leverage this new VisIt capability.

To begin, download the example scripts and data files for this tutorial:

visit_sim_code_testing_tutorial.tar.gz

The Basics

A quick review of VisIt's Python Client Interface

VisIt's testing infrastructure is built on VisIt's Python Client Interface. This interface (often called the CLI) provides a powerful way to script visualization and analysis. To review Python Client Interface, please see:

http://visitusers.org/index.php?title=VisIt-tutorial-Python-scripting

The example scripts for this section are in the folder cli_examples in the tutorial tarball.

VisIt's Testing primitives

VisIt's testing infrastructure extends the standard Python Client Interface with a few test specific functions.

Baseline Based Tests

These functions execute tests that compare a rendered image or an text result to an existing baseline.

• Test("case_name")
  • Test is the workhorse for VisIt's own testing, it renders the current window and does an image comparison with the baseline image for the given case name.
• TestText("case_name","{test text}")
  • TestText compares the string passed in with the baseline text for the given case name.

Assert Based Tests

(Available in VisIt 2.7.1)

These functions execute tests that do simple value assertions, allowing you to test quantities without managing baselines.

• AssertTrue("case_name",val)
• AssertFalse("case_name",val)
• AssertEqual("case_name",val_a,val_b)
• AssertLT("case_name",val_a,val_b)
• AssertLTE("case_name",val_a,val_b)
• AssertGT("case_name",val_a,val_b)
• AssertGTE("case_name",val_a,val_b)

Termination

By default VisIt's client interface does not terminate after running a script. For test scripts we require an explicit Exit() call to ensure that all test cases were reached.

Using the visit_testing module

Test scripts are organized in the following hierarchy:

• Category: A directory containing a set of test scripts
• File: A python script file hat implements a set of test cases
• Case: Each call to Test(), TestImage(), AssertTrue(), etc

For this tutorial the example test scripts are located under tests/tutorial.

To get started, we review a python script that defines a few example test cases.

Example: A Basic Test Script

This script demonstrates how to:

• Open a dataset
• Create a Pseudocolor plot of a scalar field from the dataset.
• Add an operator to modify the mesh.
• Compare rendered views to images baselines.
• Execute queries to obtain the total volume of the mesh.

"""
  CLASSES: nightly
  Test Case:  basic_example.py
"""
OpenDatabase(data_path("example.silo"))
#
# Test simple read and display of a variable
#
AddPlot("Pseudocolor","temp")
DrawPlots()
Test("example_render_test_00")
#
# Change the view and re-render
#
v=GetView3D()
v.viewNormal=(-0.5, 0.296198, 0.813798)
SetView3D(v)
Test("example_render_test_01")
#
# Run a query to get the volume of the entire dataset
#
Query("Volume")
v = GetQueryOutputValue()
TestText("example_text_test_00",
         "Volume = %0.2f" % v)
#
# Add an isovolume operator to select a submesh.
#
AddOperator("Isovolume")
iatts = IsovolumeAttributes()
iatts.lbound = 4
iatts.ubound = 1e+37
iatts.variable = "temp"
SetOperatorOptions(iatts)
DrawPlots()
#
# re-render after isovolume
#
Test("example_render_test_02")
#
# Check new volume
#
Query("Volume",use_actual=True)
v = GetQueryOutputValue()
TestText("example_text_test_01",
              "Volume = %0.2f" % v)
Exit()

Example: Running the Basic Test Script

Next, we use the visit_testing module to execute our script. This will coordinate launching VisIt's test environment, executing the test cases, and will return the results in nested python dictionary.

"""
file: visit_basic_example_driver.py

usage: visit -v 2.7.0 -nowin -cli -s visit_basic_example_driver.py

description:
 Basic driver that runs a visit test and consumes the results of all the test cases run.
"""

from visit_testing import *
#
# Call a VisIt test script using VisIt's testing infrastructure:
#
res = run_visit_test("tests/tutorial/basic_example.py",
                     data_dir="data",
                     baseline_dir="baselines",
                     output_dir="_results",
                     visit_bin="/usr/gapps/visit/bin/visit -v 2.7.0",
                     verbose=False)
#
# Traverse test results:
#
print ""
print "[example results]"
for script_result in res["results"]:
    print "%s/%s" % (script_result["category"],script_result["file"]) ,
    print ": " , script_result["status"]
    for sect in script_result["details"]["sections"]:
        print " section:", sect["name"]
        print "  cases:"
        for case in sect["cases"]:
            print "   %s: %s" % (case["name"],case["status"])

sys.exit()

To quickly view the html results:

firefox _results/html/index.html 

After executing run_visit_test:

1. res will be a python dictionary that holds a tree of results
2. A bunch of goodies will live in the directory _results
   • To view the generated html pages open _results/html/index.html
   • The contents of res are also stored in _results/results.json (you can load them manually via json.load(open("_results/results.json"))
3. The current results (images and text) are in _results/current
   • If you need to rebaseline, you can simply replace the contexts of baseline with the appropriate files from _results/current

For this example driver, we simply traverse the result tree and print the details of each script tested

Example: Printing results using the json module

We can easily print to entire results data structure using python's json module.

#
# [full source: visit_basic_example_driver_with_pretty_print.py]
#
# usage: visit -v 2.7.0 -nowin -cli -s visit_basic_example_driver_with_pretty_print.py 
#

from visit_testing import *
#
# Call a VisIt test script using VisIt's testing infrastructure:
# ...

#
# Use python's json module to pretty print the entire result structure:
#
import json
print ""
print "<example pretty print via json.dumps>"
print json.dumps(res,indent=2)

Example: Running the Basic Test Script from your Python Interpreter

The visit_testing module is a pure python module that can easily be imported into your Python Interpreter. Here is an example demonstrating the proper way to setup your python sys.path to import the module.

#
# [full source: standalone_driver_basic_example.py]
#
# usage: python standalone_driver_basic_example.py
#
import sys

# Add VisIt's python site packages to your python search path:
# sys.path.append("/path/to/visit/<version>/<arch>/lib/site-packages/")
#
# example for LLNL's OCF:
#
sys.path.append("/usr/gapps/visit/2.7.0/linux-x86_64/lib/site-packages/")

#
# Invoke VisIt tests as in previous example:
#

from visit_testing import *
#
# Call a VisIt test script using VisIt's testing infrastructure:
# ...

Stubs for Running test scripts in VisIt without using the Testing Infrastructure

In certain cases it is helpful to run test scripts in VisIt's standard Python Client Interface or from the GUI to debug and explore test scripts.

Recall the standard VisIt Python Client interface does not provide the testing specific functions -- the quickest way to avoid a problem is to provide dummy functions that stub out these calls:

from os.path import join as pjoin

def data_path(file):
    return pjoin("data",file)

def Test(case):
   pass

def TestText(case,txt):
   pass

def Exit():
   pass

def AssertTrue(val):
   pas

#
# ...
#

In-depth Test Examples

These examples provide recipes for common tests that are useful to simulation codes. You can run all of these examples via:

python standalone_driver_all_examples.py

Integration

• tests/tutorial/integration.py

This example demonstrates how to integrate the mass of a specific material. Demonstrates this integration using material selection and using the volume, matvf, and value_for_material expressions to calculate per proper mixed material masses.

Using Weighted Variable Sum Query

The Weighted Variable Sum Query sums a scalar value at each cell scaled by its volume (3D), area (2D) or revolved volume (RZ).

When used a density plot, it allows you to find the total mass:

#
# Integrate den scaled by vol at each cell (to obtain total mass)
#
Query("Weighted Variable Sum")
v = GetQueryOutputValue()
TestText("full_mass_total",
         "Mass = %0.3f" % v)

Selecting a specific Material / Region

Subsetting is VisIt is handled via a Subset Inclusion Lattice (SIL). To select a specific material or region in python, we use a SILRestriction object.

#
# Select only material 1 using the SIL
#
silr = SILRestriction()
mats = silr.SetsInCategory("mat")
mats = {silr.SetName(m_id) : m_id for m_id in mats }
for k,v in mats.items():
    if k != "1":
        silr.TurnOffSet(v)
SetPlotSILRestriction(silr)
DrawPlots()
Test("den_mat1")

Using expressions to construct a mass sum

VisIt provides expressions that allow you access the volume fraction of a material at each cell, mixed slot values for scalars, and the volume of each cell. From these we can construct a expression with the desired per cell mass for a given material. To get the total mass in this case we simply use the Variable Sum query.

#
# Define an expression that allows us to look at the volume fraction of material 1
#
DefineScalarExpression("mat1_volfrac", "matvf(mat,1)")
#
# Define an expression that cals the proper mass using mixed slot values of den for material 1
#
DefineScalarExpression("mat1_mass", "volume(mesh) * matvf(mat,1) * value_for_material(den,1)")
#
# Render the volfrac and mass for exprs for mat1
#
ChangeActivePlotsVar("mat1_volfrac")
DrawPlots()
Test("mat1_volfrac")
ChangeActivePlotsVar("mat1_mass")
Test("mat1_mass_mixed_slot_expr")

#
# check total mass from our expr
#
Query("Variable Sum")
v = GetQueryOutputValue()
TestText("mat1_mass_mixed_slot_exp_total",
         "Mass = %0.3f" % v)

Lineouts

• tests/tutorial/lineout.py

This example executes a lineout on a Pseudocolor plot of 3D mesh and demonstrates how to access the resulting x,y pairs.

Executing a Lineout

The Lineout function executes a lineout from python.

AddPlot("Pseudocolor","d")
DrawPlots()

#...

#
# Execute the lineout
#
Lineout( [0,0],[.8,1.2])

Accessing resulting x,y pairs

You can access the resulting lineout data via the GetPlotInformation function.

Note: The lineout operation creates a new visualization window, so we first need to make sure that window is active.

#
# The lineout creates a new window, make that window active:
#
SetActiveWindow(2)

# ...

#
# We can also access to the x,y pairs
#
lineout_data = GetPlotInformation()
TestText("lineout_text",str(lineout_data["Curve"]))

Cross Mesh Field Evals

To map fields from one mesh to another, VisIt provides a set of Cross Mesh Field Evaluation (CMFE) expressions.

CMFEs allow you:

• Compare different time steps from a simulation
• Probe mesh symmetry
• Compare different simulations (even if their mesh topology differs)

You can find more details about VisIt's CMFEs here.

For this tutorial, we review test scripts that demonstrate a symmetry comparison and time differencing of scalar field on a mesh.

Symmetry Comparison

• tests/tutorial/cmfe_symm.py

This example demonstrates using a symm_plane CMFE to probe the symmetry of a field on a mesh. In addition to mapping the field across the plane, symm_plane includes the difference operation.

#
# Define our cmfe that does a diff across a plane
#
DefineScalarExpression("symm_diff", "symm_plane(d, [-1,0,0, 0,0,0])")

The field data on the mesh in the example dataset is symmetric. To demonstrate breaking symmetry we deform the example mesh using a vector field.

#
# Create expr to get access to the coordiantes at each node of the mesh
#
DefineVectorExpression("coords", "coord(curvmesh2d)")
#
# Create an expression we can use to distort the mesh to test symm_plane cmfe
#
DefineVectorExpression("deform", "if(lt(coords[0],0),{coords[0]*.5,0},{coords[0]*0,0})")

OpenDatabase(data_path("curv2d.silo"))

AddPlot("Pseudocolor", "symm_diff")
AddOperator("Displace")
dsp_atts = DisplaceAttributes()
# our first test will have not displacement
dsp_atts.factor = 0
dsp_atts.variable = "deform"
SetOperatorOptions(dsp_atts)
AddOperator("DeferExpression")
dfr_atts = DeferExpressionAttributes()
dfr_atts.exprs = ("symm_diff")
SetOperatorOptions(dfr_atts)
DrawPlots()

for disp_mult in [0,1,2]:
    dsp_atts = DisplaceAttributes()
    # set displacement factor that mults our deforming vector field
    dsp_atts.factor = disp_mult
    dsp_atts.variable = "deform"
    SetOperatorOptions(dsp_atts)
    ChangeActivePlotsVar("d")
    pc_atts = PseudocolorAttributes()
    pc_atts.minFlag = 0
    pc_atts.maxFlag = 0
    SetPlotOptions(pc_atts)
    Test("d_disp_mult_%d" % disp_mult)
    ChangeActivePlotsVar("symm_diff")
    pc_atts = PseudocolorAttributes()
    pc_atts.min = -2.0
    pc_atts.max =  2.0
    pc_atts.minFlag = 1
    pc_atts.maxFlag = 1
    SetPlotOptions(pc_atts)
    Test("symm_plane_disp_mult_%d" % disp_mult)

Renders Demonstrating symm_plane Test Cases

Density w/o Distortion	Symm Diff w/o Distortion
Density w/ Distortion Mult =1	Symm Diff w/ Distortion = 1
Density w/ Distortion Mult =2	Symm Diff w/ Distortion = 2

Time Differencing

• tests/tutorial/cmfe_time.py

The example dataset for this case is Eulerian (mesh topology does not change over time). This allow us to use a conn_cmfe (a Connectivity-based CMFE) to map a scalar field from one time step to another time step of simulation. For this example, we use this capability to create an expression that calculates the absolute difference between each time step and the initial conditions of the simulation (time step = 0). In cases where the mesh topology does change over time you can use a pos_cmfe (a Position-based CMFE) to do a similar differencing.

#
# Define our cmfe that maps "alpha1" from time zero to the current  timestep
#

DefineScalarExpression("alpha_t0_diff", "abs(alpha1-conn_cmfe(<[0]i:alpha1>, mesh))")

# ...

AddPlot("Pseudocolor","alpha_t0_diff")

# ...

DrawPlots()

#
# Render the diff over time
#
nts = TimeSliderGetNStates()
for ts in range(nts):
    TimeSliderSetState(ts)
    Test("alpha_t0_diff_render_%04d" % ts)

Renders Demonstrating conn_cmfe Test Cases

Diff at ts=0

Diff at ts=5

Diff at ts=10