Static linking and SimV2

VisIt's libsim (a.k.a. SimV2) usually is separated into a small static library and a shared library runtime that gets loaded on demand. The static library contains C functions that can be linked to C and Fortran simulations. Some of those functions do nothing until a simulation is connected, when libsim dynamically loads the runtime library that contains the functions that do the actual work (and the rest of the compute engine functionality).

In order to link a simulation statically, the libsim front end and runtime libraries must be changed so that the front end can call functions in the runtime and resolve the functions at link time. This is made possible by the use of preprocessor macros internally. Instead of calling a function to dynamically look up a function based on a name, we can simply use the name of the function by changing a macro and then the linker will resolve the functions. This is handled by the VisIt build system.

Most VisIt installations that you'll come across are dynamically linked. Therefore, you will probably have to build your own static VisIt installation. You will need to get familiar with the build_visit script and pass it the --static flag.

Producing a static build of VisIt and all of the libraries that comprise it can be done by running the following command:

cmake -DVISIT_STATIC:BOOL=ON .

Linking a simulation

Linking a simulation to libsim in the usual dynamic build is as simple as linking the simulation with -lsimV2. Unfortunately, that will not suffice when you want to build the entire simulation statically and have it contain all of VisIt. The only choice is to link the simulation with libsimV2_static_ser or libsimV2_static_par and many of VisIt's dependent libraries such as VTK, Ice-T, Mesa, MPI, and OpenGL.

Combined libraries

The VisIt build system will create libsimV2_static_ser.a and libsimV2_static_par.a when a top-level build is performed. These libraries are essentially larger archives containing the various .o's from the rest of VisIt's .a libraries.

The build system still has a few quirks with libraries created like this:

  1. The libsimV2_static_ser.a and libsimV2_static_par.a libraries do not get installed. You can copy them into the installation's libsim/V2/lib directory for the time being.
  2. The build system dependencies are not 100% correct so changing the libraries that go into these libraries does not cause them to regenerate.

A serial simulation would link the _ser.a version of the library and a parallel simulation would link to the the _par.a version. You will also have to link with the VTK libraries other VisIt dependencies.

Hopper (Cray/Linux)

The following Makefile shows how a simple simulation using the combined static libraries for VisIt was linked on NERSC's hopper system. Note the large set of dependent libraries that must be linked. VisIt was built statically on a Linux system so it has a lot of X11 dependencies introduced via VTK, which for a typical Linux build will include X11 render windows. These features are not used in situ but the libraries must be provided to link the program.

###############################################################################

# Installed VisIt location for
VISIT_DIR=/global/homes/w/whitlocb/Development/install/2.9.0/linux-x86_64
VISIT_BUILD_DIR=/global/homes/w/whitlocb/Development/visit2.9.0/build_static

# VTK
VTK_DIR=/global/homes/w/whitlocb/Development/thirdparty_static/2.9.0/vtk/6.1.0/linux-x86_64_gcc-4.9

# ICET
ICET_DIR=/global/homes/w/whitlocb//Development/thirdparty_static/2.9.0/icet/1.0.0/linux-x86_64_gcc-4.9

# Mesa
MESA_DIR=/global/homes/w/whitlocb/Development/thirdparty_static/2.9.0/mesa/7.10.2/linux-x86_64_gcc-4.9

# hopper-specific MPI flags.
MPI_CFLAGS=-I/opt/cray/mpt/7.1.1/gni/mpich2-gnu/49/include
MPI_LDFLAGS=-L/opt/cray/mpt/7.1.1/gni/mpich2-gnu/49/lib \
-L/opt/cray/pmi/5.0.6-1.0000.10439.140.3.gem/lib64 \
-L/opt/cray/alps/5.2.1-2.0502.9041.11.6.gem/lib64
MPI_LIBS=-lmpich -lpmi -lalpslli -lalpsutil -lpthread -lrt

###############################################################################

# static simV2 flags.
SIMV2_CFLAGS=-I${VISIT_DIR}/libsim/V2/include -DPARALLEL
SIMV2_LDFLAGS=-L${VISIT_DIR}/libsim/V2/lib -L${VISIT_BUILD_DIR}/lib \
-Wl,-rpath,${VISIT_BUILD_DIR}/lib
SIMV2_LIBS=-lEFieldViewXDBDatabase_par \
-lsimV2_static_par -lEFieldViewXDBDatabase_par \
-lVXDB_par -lIFieldViewXDBDatabase -lsimV2_static_par  \
-lXDBLib -lvisit_vtk_osmesa

# Ice-T
ICET_LDFLAGS=-L${ICET_DIR}/lib
ICET_LIBS=-licet -licet_strategies -licet_mpi

# X11
X11_LIBS = -lGLU -lSM -lICE -lX11 -lXext -lSM -lICE -lX11 -lXext -lXt

GL_LDFLAGS=-L${MESA_DIR}/lib
GL_LIBS=-lOSMesa ${X11_LIBS}

# VTK
VTK_LDFLAGS=-L${VTK_DIR}/lib
VTK_LIBS = -lvtkCommonCore-6.1 \
-lvtkImagingCore-6.1 \
-lvtkRenderingCore-6.1 \
-lvtkIOLegacy-6.1 \
-lvtkIOPLY-6.1 \
-lvtkInteractionStyle-6.1 \
-lvtkRenderingOpenGL-6.1 \
-lvtkRenderingFreeTypeOpenGL-6.1 \
-lvtkFiltersHybrid-6.1 \
-lvtkFiltersModeling-6.1 \
-lvtkGeovisCore-6.1 \
-lvtkRenderingOpenGL-6.1 \
-lvtkIOXML-6.1 \
-lvtkIOGeometry-6.1 \
-lvtkjsoncpp-6.1 \
-lvtkIOXMLParser-6.1 \
-lvtkexpat-6.1 \
-lvtkInfovisLayout-6.1 \
-lvtkInfovisCore-6.1 \
-lvtkViewsCore-6.1 \
-lvtkInteractionWidgets-6.1 \
-lvtkInteractionStyle-6.1 \
-lvtkFiltersHybrid-6.1 \
-lvtkFiltersModeling-6.1 \
-lvtkImagingHybrid-6.1 \
-lvtkIOImage-6.1 \
-lvtkIOCore-6.1 \
-lvtkpng-6.1 \
-lvtktiff-6.1 \
-lvtkjpeg-6.1 \
-lvtkmetaio-6.1 \
-lvtkDICOMParser-6.1 \
-lvtkImagingGeneral-6.1 \
-lvtkImagingSources-6.1 \
-lvtkRenderingAnnotation-6.1 \
-lvtkRenderingFreeType-6.1 \
-lvtkftgl-6.1 \
-lvtkfreetype-6.1 \
-lvtkzlib-6.1 \
-lvtkImagingColor-6.1 \
-lvtkRenderingVolume-6.1 \
-lvtkRenderingCore-6.1 \
-lvtkFiltersSources-6.1 \
-lvtkFiltersGeometry-6.1 \
-lvtkFiltersExtraction-6.1 \
-lvtkFiltersGeneral-6.1 \
-lvtkCommonComputationalGeometry-6.1 \
-lvtkFiltersCore-6.1 \
-lvtkFiltersStatistics-6.1 \
-lvtkImagingFourier-6.1 \
-lvtkImagingCore-6.1 \
-lvtkCommonExecutionModel-6.1 \
-lvtkCommonDataModel-6.1 \
-lvtkCommonMisc-6.1 \
-lvtkCommonSystem-6.1 \
-lvtkCommonTransforms-6.1 \
-lvtkCommonMath-6.1 \
-lvtkCommonCore-6.1 \
-lvtksys-6.1 \
-lvtkalglib-6.1 \
-lvtkproj4-6.1

###############################################################################
CC=gcc
CXX=g++
CFLAGS=-m64 -g -I. ${MPI_CFLAGS} ${SIMV2_CFLAGS}
LDFLAGS=${SIMV2_LDFLAGS} ${VTK_LDFLAGS} ${MPI_LDFLAGS} ${ICET_LDFLAGS} ${GL_LDFLAGS}
LIBS=${SIMV2_LIBS} ${VTK_LIBS} ${ICET_LIBS} ${MPI_LIBS} ${GL_LIBS} -lz

SRC=batch.c extract.c
OBJ=$(SRC:.c=.o)

PROGRAM=batch_par

$(PROGRAM): $(OBJ)
        $(CXX) -o $(PROGRAM) $(CFLAGS) $(OBJ) $(LDFLAGS) $(LIBS)

.c.o:
        $(CC) -c -o $@ ${CFLAGS} $<

clean:
        rm -f $(OBJ) $(PROGRAM)

BG/Q

As with static linking on a typical Linux box, linking on BG/Q will require a lot of 3rd party library dependencies to be linked to produce the statically-linked simulation that includes libsim. The following example is a template for a Makefile that could be used on a BG/Q system to statically link a Fortran simulation to libsim.

# Set these up for your system.
VISIT_BASE=
VTK_BASE=
ICET_BASE=
MESA_BASE=

# VTK
VTK_LDFLAGS =-L${VTK_BASE}/lib \
-lvtkCommonCore-6.1 \
-lvtkImagingCore-6.1 \
-lvtkRenderingCore-6.1 \
-lvtkIOLegacy-6.1 \
-lvtkIOPLY-6.1 \
-lvtkInteractionStyle-6.1 \
-lvtkRenderingOpenGL-6.1 \
-lvtkRenderingFreeTypeOpenGL-6.1 \
-lvtkFiltersHybrid-6.1 \
-lvtkFiltersModeling-6.1 \
-lvtkGeovisCore-6.1 \
-lvtkRenderingOpenGL-6.1 \
-lvtkIOXML-6.1 \
-lvtkIOGeometry-6.1 \
-lvtkjsoncpp-6.1 \
-lvtkIOXMLParser-6.1 \
-lvtkexpat-6.1 \
-lvtkInfovisLayout-6.1 \
-lvtkInfovisCore-6.1 \
-lvtkViewsCore-6.1 \
-lvtkInteractionWidgets-6.1 \
-lvtkInteractionStyle-6.1 \
-lvtkFiltersHybrid-6.1 \
-lvtkFiltersModeling-6.1 \
-lvtkImagingHybrid-6.1 \
-lvtkIOImage-6.1 \
-lvtkIOCore-6.1 \
-lvtkpng-6.1 \
-lvtktiff-6.1 \
-lvtkjpeg-6.1 \
-lvtkmetaio-6.1 \
-lvtkDICOMParser-6.1 \
-lvtkImagingGeneral-6.1 \
-lvtkImagingSources-6.1 \
-lvtkRenderingAnnotation-6.1 \
-lvtkRenderingFreeType-6.1 \
-lvtkftgl-6.1 \
-lvtkfreetype-6.1 \
-lvtkImagingColor-6.1 \
-lvtkRenderingVolume-6.1 \
-lvtkRenderingCore-6.1 \
-lvtkFiltersSources-6.1 \
-lvtkFiltersGeometry-6.1 \
-lvtkFiltersExtraction-6.1 \
-lvtkFiltersGeneral-6.1 \
-lvtkCommonComputationalGeometry-6.1 \
-lvtkFiltersCore-6.1 \
-lvtkFiltersStatistics-6.1 \
-lvtkImagingFourier-6.1 \
-lvtkImagingCore-6.1 \
-lvtkCommonExecutionModel-6.1 \
-lvtkCommonDataModel-6.1 \
-lvtkCommonMisc-6.1 \
-lvtkCommonSystem-6.1 \
-lvtkCommonTransforms-6.1 \
-lvtkCommonMath-6.1 \
-lvtkCommonCore-6.1 \
-lvtksys-6.1 \
-lvtkalglib-6.1 \
-lvtkproj4-6.1

# Ice-T
ICET_LDFLAGS=-L${ICET_BASE}/lib \
-licet \
-licet_strategies \
-licet_mpi

# MESA
MESA_LDFLAGS=-L${MESA_BASE}/lib \
-lOSMesa

# SYSTEM (XL Fortran libraries)
XLF_LDFLAGS = -L${IBMCMPXLF_BASE}/lib64 \
-lxlopt \
-lxl \
-lxlf90_r \
-lxlfmath

VISIT_INCFLAGS = -I$(VISIT_BASE)/include
VISIT_LDFLAGS  = -L$(VISIT_BASE)/lib \
                 -lsimV2f \
                 -lsimV2 \
                 -lsimV2runtime_par \
                 -lengine_par \
                 -lsimV2_static_par \
                 -lsimV2runtime_par \
                 -lsimV2_static_par \
                 -lcognomen \
                 -lvisit_vtk \
                 ${VTK_LDFLAGS} \
                 ${ICET_LDFLAGS} \
                 ${MESA_LDFLAGS}

###############################################################################
CC=bgxlc_r
CXX=bgxlC_r
FC=bgxlf90_r
CFLAGS=-I. ${VISIT_INCFLAGS}
FFLAGS=-I. ${VISIT_INCFLAGS}
LDFLAGS=${VISIT_LDFLAGS}

# A Fortran program
FSRC=fscalarp.f
FOBJ=$(FSRC:.f=.o)
FPROGRAM=fscalarp

$(FPROGRAM): $(FOBJ)
        $(FC) -o $(FPROGRAM) $(FFLAGS) $(FOBJ) $(LDFLAGS) ${XLF_LDFLAGS} -lz -ldl -lstdc++

.f.o:
        $(FC) -c -o $@ ${FFLAGS} $<

clean:
        rm -f $(FOBJ) $(FPROGRAM)

Separate libraries

You can link your simulation to VisIt's libraries without combining them first. There are many libraries that make up AVT and VisIt's Core libraries, and don't forget the plugins that you want to include. The result is a large number of libraries as shown in the following Makefile for a statically linked version of the updateplots example simulation.

Note: The inclusion of frameworks shown in the Makefile is specific to MacOS X where this test was performed.

CXX=g++
CXXFLAGS=-O3 -m64 

VISITDIR=/path/to/visit/thirdparty_libraries
VISITARCH=i386-apple-darwin10_gcc-4.2

VTK_DIR=$(VISITDIR)/vtk/5.8.0/$(VISITARCH)
VTK_LDFLAGS=-L$(VTK_DIR)/lib
VTK_LIBS=-lvtkCommon -lvtkGraphics -lvtkHybrid -lvtkRendering -lvtkImaging -lvtkGenericFiltering -lvtkFiltering -lvtkIO -lvtksys -lvtkfreetype -lvtkftgl -lvtkjpeg -lvtktiff -lvtkpng -lvtkexpat -lvtklibxml2 -lvtkzlib 

MESA_DIR=$(VISITDIR)/mesa/7.8.2/$(VISITARCH)
MESA_LDFLAGS=-L$(MESA_DIR)/lib
MESA_LIBS=-lOSMesa -lMesaGLU

# Link against all libraries
SLIVR_LIBS=-lair -lalan -lbiff -lbow -ldye -lell -lgage -lhest -lmoss -lnrrd -lslivrG -lslivrV -lunrrdu
VISIT_PLOT_LIBS=-lIBoundaryPlot -lEBoundaryPlot_ser -lIContourPlot -lEContourPlot_ser -lICurvePlot -lECurvePlot_ser -lIFilledBoundaryPlot -lEFilledBoundaryPlot_ser -lIHistogramPlot -lEHistogramPlot_ser -lILabelPlot -lELabelPlot_ser -lIMeshPlot -lEMeshPlot_ser -lIMoleculePlot -lEMoleculePlot_ser -lIMultiCurvePlot -lEMultiCurvePlot_ser -lIParallelCoordinatesPlot -lEParallelCoordinatesPlot_ser -lIPoincarePlot -lEPoincarePlot_ser -lIPseudocolorPlot -lEPseudocolorPlot_ser -lIScatterPlot -lEScatterPlot_ser -lISpreadsheetPlot -lESpreadsheetPlot_ser -lIStreamlinePlot -lEStreamlinePlot_ser -lISubsetPlot -lESubsetPlot_ser -lISurfacePlot -lESurfacePlot_ser -lITensorPlot -lETensorPlot_ser -lITruecolorPlot -lETruecolorPlot_ser -lIVectorPlot -lEVectorPlot_ser -lIVolumePlot -lEVolumePlot_ser -lIWellBorePlot -lEWellBorePlot_ser
VISIT_OPERATOR_LIBS=-lIAMRStitchCellOperator -lEAMRStitchCellOperator_ser -lIBoundaryOpOperator -lEBoundaryOpOperator_ser -lIBoxOperator -lEBoxOperator_ser -lIClipOperator -lEClipOperator_ser -lIConeOperator -lEConeOperator_ser -lIConnectedComponentsOperator -lEConnectedComponentsOperator_ser -lICoordSwapOperator -lECoordSwapOperator_ser -lICracksClipperOperator -lECracksClipperOperator_ser -lICreateBondsOperator -lECreateBondsOperator_ser -lICylinderOperator -lECylinderOperator_ser -lIDataBinningOperator -lEDataBinningOperator_ser -lIDecimateOperator -lEDecimateOperator_ser -lIDelaunayOperator -lEDelaunayOperator_ser -lIDeferExpressionOperator -lEDeferExpressionOperator_ser -lIDisplaceOperator -lEDisplaceOperator_ser -lIDualMeshOperator -lEDualMeshOperator_ser -lIEdgeOperator -lEEdgeOperator_ser -lIElevateOperator -lEElevateOperator_ser -lIExternalSurfaceOperator -lEExternalSurfaceOperator_ser -lIExtrudeOperator -lEExtrudeOperator_ser -lIFFTOperator -lEFFTOperator_ser -lIFiveFoldTetSubdivisionOperator -lEFiveFoldTetSubdivisionOperator_ser -lIFluxOperator -lEFluxOperator_ser -lIIndexSelectOperator -lEIndexSelectOperator_ser -lIInverseGhostZoneOperator -lEInverseGhostZoneOperator_ser -lIIsosurfaceOperator -lEIsosurfaceOperator_ser -lIIsovolumeOperator -lEIsovolumeOperator_ser -lILineoutOperator -lELineoutOperator_ser -lILineSamplerOperator -lELineSamplerOperator_ser -lIMergeOperator -lEMergeOperator_ser -lIMultiresControlOperator -lEMultiresControlOperator_ser -lIOnionPeelOperator -lEOnionPeelOperator_ser -lIPDFOperator -lEPDFOperator_ser -lIPersistentParticlesOperator -lEPersistentParticlesOperator_ser -lIProjectOperator -lEProjectOperator_ser -lIReflectOperator -lEReflectOperator_ser -lIReplicateOperator -lEReplicateOperator_ser -lIResampleOperator -lEResampleOperator_ser -lIRevolveOperator -lERevolveOperator_ser -lISliceOperator -lESliceOperator_ser -lISmoothOperator -lESmoothOperator_ser -lISphereSliceOperator -lESphereSliceOperator_ser -lISurfaceNormalOperator -lESurfaceNormalOperator_ser -lIThreeSliceOperator -lEThreeSliceOperator_ser -lIThresholdOperator -lEThresholdOperator_ser -lITransformOperator -lETransformOperator_ser -lITriangulateRegularPointsOperator -lETriangulateRegularPointsOperator_ser -lITubeOperator -lETubeOperator_ser -lIZoneDumpOperator -lEZoneDumpOperator_ser
VISIT_DATABASE_LIBS=-lISimV2Database -lESimV2Database_ser
VISIT_LIBS=-lsimV2runtime_ser -lengine_ser -lavtdbatts -lavtivp -lavtmath -lavtqtviswindow -lavtshapelets -lavtview -lavtdatabase_ser -lavtdbin_ser -lavtexpressions_ser -lavtfilters_ser -lavtmir_ser -lavtpipeline_ser -lavtplotter_ser -lavtquery_ser -lavtviswindow_ser -lavtwriter_ser -lenginerpc -llightweight_visit_vtk -lmdserverproxy -lmdserverrpc -lvisit_verdict -lvisit_vtk -lvisitcommon -lsimV2 -lGLEW -lz
FRAMEWORKS=-framework OpenGL -framework AGL -framework SystemConfiguration -framework IOKit -framework Cocoa

CPPFLAGS=-I. -I/Users/whitlock2/Development/STATIC/trunk/src/sim/V2/lib
LDFLAGS=-L/Users/whitlock2/Development/STATIC/trunk/src/lib $(VTK_LDFLAGS) $(MESA_LDFLAGS)
LIBS=$(SLIVR_LIBS) $(VISIT_PLOT_LIBS) $(VISIT_OPERATOR_LIBS) $(VISIT_DATABASE_LIBS) $(VISIT_LIBS) $(VTK_LIBS) $(MESA_LIBS) $(FRAMEWORKS)

SRC=updateplots.C
OBJ=$(SRC:.C=.o)

all: updateplots

updateplots: $(OBJ)
	$(CXX) -o updateplots $(OBJ) $(CXXFLAGS) $(LDFLAGS) $(LIBS)

clean:
	$(RM) $(OBJ) updateplots

.C.o:
	$(CXX) $(CXXFLAGS) $(CPPFLAGS) -c $<

Flaws

More recent versions of VisIt build libsimV2_static_ser.a and libsimV2_static_par.a libraries that include the pieces of VisIt that we need to get simulations to link more easily. There are still a number of flaws in the creation and use of this library.

  1. The libsimV2_static_ser.a and libsimV2_static_par.a files get created during a top level "make" but they do not get installed in a top level "make install".
  2. When linking with the libsimV2_static_*.a libraries, we also need to link with visit_vtk_osmesa. This means that we did not put visit_vtk_osmesa into the libsimV2_static_*.a libraries like we should.
  3. Linking applications still requires a boatload of VTK libraries on the link line
  4. Linking with the VTK libraries seems to imply that we need to do the final application link with the c++ compiler, which may have some implications for linking Fortran programs.
  5. Example simulations did not link statically on hopper because they could not find IceT. The CMakeLists.txt only use IceT on BG/Q instead of using it for parallel apps whenever ICET_FOUND is true.
  6. The FieldViewXDB plugin fails during the build because it assumes a ".a" extension for libXDBLib. Also, it assumes the same for the VXDB libraries so we need static versions of that library in addition to the dynamic version. Without that, static binaries can't be built because the VXDB shared libraries rely on shared versions of VTK.
  7. Problems with Mesa rendering are causing my libsim runtime to die during initialization when we show all of the windows via viewercore. By going into VisWinRenderingWithoutWindow.C and commenting out the bodies of the RenderRenderWindow() and RealizeRenderWindow() methods then I can work around the problem for now. However, that would prevent scalable rendering from working. It is looking more like build_visit needs a libsim only mode that can build an X11-less VTK and use Mesa as GL like is done on BG/Q.
  8. When I finally got batch_par running statically-linked, it did not output XDBs. The ExportDatabase method is not giving an indication why. The batch_par program needs to have tracing and debug log ability added to it.
  9. The SimStaticSymbolLocator.C code is only adding SimV2 to the list of database plugins, which is preventing exporting from working.
  10. The process for creating the libsimV2_static_ libraries seems tied to the generation of the _ser library only.
  11. The FieldViewXDB plugin .a's do not seem to have been included in the libsimV2_static_ libraries.
  12. The simv2 runtime is reading a lot of color tables -- we need to broadcast them
  13. The VisItExportDatabase function does not return VISIT_ERROR when the dataset cannot be exported.
  14. This is a general problem with VisItExportDatabase but it cannot export to formats that have database options. I altered the VTK writer so it can handle exceptions from the DBOptionsAttributes when a key is not found in the object. It's better to use some default values than to not export. Meanwhile, SimEngine.C needs to be modified to pass the database options in the export database attributes if it can get them.

Hopper

There are additional flaws on hopper that prevent a static libsim from building and working well without heroic efforts.

  • Need to build a Mesa to use as OpenGL to so we can link a static GL.
  • Rendering has issues anyway, go into VisWinRenderingWithoutWindow.C and disable the RealizeRenderWindow and RenderRenderWindow methods. Also go into avtOpenGLExtensionManager.C and disable the calls to GLEW.

In the end, a successful dynamic link to libsim was performed. This was needed because the static binaries produced for hopper did not function well on the compute nodes. The most obvious problem was that libsim was only requesting the mesh from rank 0. By switching to dynamic linking, the runtime problems ceased. A couple important changes were needed to make dynamic linking work:

  1. DO NOT USE THE WRAPPER COMPILERS! Use gcc/g++/gfortran directly after loading PrgEnv-gnu environment.
  2. Link with -lsimV2dyn instead of -lsimV2
  3. In the job submission script, be sure to pass a valid LD_LIBRARY_PATH variable:
#PBS -q debug
#PBS -l mppwidth=24
#PBS -l walltime=00:30:00
#PBS -A m636
#PBS -v DISPLAY

cd $PBS_O_WORKDIR
aprun -n 24 -e LD_LIBRARY_PATH=/global/u1/w/whitlocb/Development/install_dyn2/2.9.0/linux-x86_64/lib:/opt/gcc/4.9.2/snos/lib64 ./flamev.x