VisIt and HDF5

Many who wish to use VisIt have data stored in HDF5 files. The important point about HDF5 is that it is a very flexible, free-form, file format. In using HDF5 to represent meshes and variables defined on those meshes, there are many choices to make. The names of datasets and their locations in the file(s), how information such as whether a variable is node- or zone-centered is stored, whether a vector variable is stored with all its components in a single dataset (and whether they are interleaved there or not) or multiple datasets are examples of just a few of those choices. Each set of choices leads to a unique way of using HDF5 to represent scientific data input to VisIt. We call each of these ways a flavor of HDF5. A flavor of HDF5 represents a set of conventions in how HDF5 is used to represent some data.

For each flavor of HDF5 VisIt supports, there is a database plugin that reads it. By their nature, these plugins make certain assumptions about the HDF5 files they read. Those assumptions are tailored to the set of conventions used in the representation of the data in the file. For example, the x coordinate field of a mesh will be stored in a dataset whose name is 'Xcoords'. Or, any dataset with an attribute name of 'nodal' is a node-centered field.

As of July 2008 the different flavors of HDF5 VisIt supports are: SAMRAI, Tetrad, GTC, Vista, FLASH, UNIC, Pixie, Chombo, Enzo, XDMF, PFLOTRAN, CosmosPP, H5Nimrod, M3D, and H5Part. These formats are described in somewhat more detail at Reading HDF5 into VisIt.

A common misconception users have is that because they know or have heard VisIt reads HDF5 files, it will be able to read their, possibly different, flavor of HDF5 files. Worse, when users try to use VisIt to open their HDF5 files, it is often the case that one of the existing plugins maybe partially succesful in opening the file. Because the Pixie plugin was written with the greatest generality, at least for structured data, it often succeeds in opening a file and presenting at least some of the data in the file for plotting. Do not be mislead or confused by this outcome. Unless you know for certain ahead of time that your HDF5 files are of a flavor VisIt already supports, be aware that VisIt may not be able to read your HDF5 files wholly or even partially.

What are your options if you have a new flavor of HDF5 files

If you believe you have a new flavor of HDF5 file, you have several options for getting VisIt to read it.

1. Try to open your HDF5 file with all the different flavors of HDF5 plugins VisIt supports. You can use the File->Open feature to specify which plugin to try or the -o <filename>,<plugin-id> option on the command line to start VisIt. Who knows, you might get lucky and an existing plugin might read your data. For example,

   % visit -o <my-filename>,Enzo_1.0

Note that you need to spell the plugin name exactly the way VisIt is expecting it, including proper capitalization followed by underscore followed by the plugin version number (in almost all cases its 1.0). Here are some of the other names: SAMRAI, Tetrad, GTC, Vista, FLASH, UNIC, Chombo, Enzo, XDMF, PFLOTRAN, CosmosPP, H5Nimrod, M3D, H5Part.

2. If you are willing and able to re-generate the data AND have any control over the application that produces the data, you might consider changing the application to write one of the supported flavors. Unfortunately, there is not always a lot of documentation on the structure of a given flavor of an HDF5 file. So, you might need to do a little reverse engineering from an existing HDF5 file in the flavor you are targeting to decide how to modify your application to create that flavor.

3. Write a new database plugin for VisIt that reads your specific flavor of HDF5. If you can identify an existing plugin that is close to what you need, then working from that plugin to develop the new one can be straightforward.

4. If you have some spare change, you might see if you can pay someone to develop a plugin for you ;). The VisIt project is always willing to consider new funding sources.

5. Create an XML schema for reading your file and use the XDMF reader.

Special Issues Compiling HDF5 and/or Linking HDF5 with VisIt

As of July, 2008, the most up to date release of the HDF5 library is version 1.8.1. However, all of the HDF5-based plugins in VisIt were written for HDF5 version 1.6. These two versions of HDF5 have slightly different application programming interfaces (APIs). Fortunately, the HDF5 folks provide some API-compatibility functionality so that 1.6 version client code can correctly work with a 1.8 version library.

If you have an HDF5-based database plugin that was written to the HDF5 1.6 API, then getting that plugin to correctly work with HDF5 version 1.8 API requires only that you define H5_USE_16_API macro pre-processor symbol prior to including hdf5.h. For example, from the SAMRAI plugin...

// Define this symbol BEFORE including hdf5.h to indicate the HDF5 code
// in this file uses version 1.6 of the HDF5 API. This is harmless for
// versions of HDF5 before 1.8 and ensures correct compilation with
// version 1.8 and thereafter. When, and if, the HDF5 code in this file
// is explicitly upgraded to the 1.8 API, this symbol should be removed.
#define H5_USE_16_API
#include <hdf5.h>

See HDF5 API compatiblity for more information on API compabitility.

Finally, if you use version 1.8.X of HDF5 with VisIt, you must take care to NOT use certain configuration flags when configuring HDF5. Do not use --with-default-api-version=v16 or --disable-deprecated-symbols when configuring HDF5. Due to a bug in HDF5, use of the former will result in the redefinition of the macro pre-processor symbol H5_USE_16_API. Use of the latter will remove from the installed HDF5 library all the version 1.6 API functions making it impossible for VisIt's HDF5 plugins to compile or link with the resulting library.