Libsim on Titan

I'm basically following the procedure that I developed for creating a static Libsim for cori and edison. The procedure is to build a static Mesa/VTK/VisIt that we can link with Libsim into applications since shared libraries at large scale are too costly.

Environment

On Titan, I'm going to try the Intel compilers.

module unload PrgEnv-pgi
module load PrgEnv-intel
module load cmake

Workspace

I'll be building some of the VisIt packages manually and I'll use build_visit for some others. I'll adopt the build_visit directory structures.

cd
cd Development
mkdir thirdparty_static_server

If I use build_visit, I'll pass --thirdparty-path $HOME/Development/thirdparty_static_server --static in the options.

CMake

We'll need to build CMake. I ran into problems linking VisIt executables with the Intel compiler due to an -rdynamic flag. I built a CMake and edited some of its installation files to prevent adding flags we don't want during a static build.

wget https://cmake.org/files/v3.5/cmake-3.5.1.tar.gz
tar zxvf cmake-3.5.1.tar.gz
cd cmake-3.5.1
env CC=icc CXX=icpc ./bootstrap --prefix=$HOME/Development/thirdparty_static_server/cmake/3.5.1/linux-x86_64_icc
make -j 4 
make -j 4 install

CMake Modifications

We're doing totally static builds but somehow, some dynamic flags are making their way into our executable link lines via the MPI link line.

  • We need to make some changes to cmake's Linux support to work around the problems.
  • Another solution could be to change VisIt's parallel executable macros so they will add any target link libraries we add before MPI libraries.

NOTE - This was not necessary on cori.

rdynamic

Edit the Linux-Intel.cmake file in the installed cmake modules:

vi $HOME/Development/thirdparty_static_server/cmake/3.5.1/linux-x86_64_icc/share/cmake-3.5/Modules/Platform/Linux-Intel.cmake

Change this line:

 set(CMAKE_SHARED_LIBRARY_LINK_${lang}_FLAGS "-rdynamic")

to:

 set(CMAKE_SHARED_LIBRARY_LINK_${lang}_FLAGS "") #"-rdynamic")

Bdynamic

Edit the Linux.cmake file in the installed cmake modules:

vi $HOME/Development/thirdparty_static_server/cmake/3.5.1/linux-x86_64_icc/share/cmake-3.5/Modules/Platform/Linux.cmake

Change this line:

 set(CMAKE_${type}_LINK_DYNAMIC_C_FLAGS "-Wl,-Bdynamic")

to:

 #set(CMAKE_${type}_LINK_DYNAMIC_C_FLAGS "-Wl,-Bdynamic")

Mesa

I can follow the same basic procedure as for cori, using a new Mesa, and adjusting the configure line to use the Intel compilers. Run the following script:

#!/bin/sh
INSTALLDIR=$HOME/Development/thirdparty_static_server/mesa/11.1.2/linux-x86_64_icc

wget ftp://ftp.freedesktop.org/pub/mesa/11.1.2/mesa-11.1.2.tar.gz
tar zxvf mesa-11.1.2.tar.gz
cd mesa-11.1.2
 
env CC=icc CXX=icpc ./configure --prefix=$INSTALLDIR --disable-shared --enable-static --enable-osmesa \
--disable-dri --disable-egl --disable-glx --disable-driglx-direct --disable-gallium-llvm \
--with-gallium-drivers=swrast --disable-gles1 --disable-gles2 --disable-gbm \
--disable-vdpau --disable-xvmc --disable-driglx-direct --disable-va \
--disable-llvm-shared-libs --with-osmesa-lib-name=GL --with-sha1=libcrypto
 
make -j 8
make install

Mesa Run

I am following the same test procedure outlined in Static Libsim Mesa Test Application. There are some differences in getting this program to run, namely making sure that the PBS job is submitted from the right place.

submit_osmesa:

#!/bin/bash
#PBS -A csc194
#PBS -q debug
#PBS -l walltime=00:02:00
#PBS -l nodes=1
cd $PBS_O_WORKDIR
aprun -n 8 ./osmesatest 800 800

I submitted and ran the job from my $MEMBERWORK/csc194 directory, which is a Lustre work directory for the project I'm working on. When I submitted the qsub job elsewhere, my program seemed to die with an illegal instruction when it ran.

  • Update: I now think that was because I had set my job limit to be 2 seconds instead of 2 minutes.
cd $MEMBERWORK/csc194
qsub submit_osmesa

Each processor wrote out a ppm file of the rendering scene. Osmesatest.png

GLU

I made some small changes to the script I used to build GLU so it will use the Intel compilers:

#!/bin/sh
wget ftp://ftp.freedesktop.org/pub/mesa/glu/glu-9.0.0.tar.gz
tar zxvf glu-9.0.0.tar.gz
MESA_DIR=$HOME/Development/thirdparty_static_server/mesa/11.1.2/linux-x86_64_icc
cd glu-9.0.0
env PKG_CONFIG_LIBDIR=$MESA_DIR/lib CC=icc CXX=icpc ./configure --prefix=$MESA_DIR --disable-shared --enable-static
make -j 8 install

VTK

The procedure for building VTK follows what I did for cori except that I'm using the Intel compilers.

#/bin/sh
TPDIR=$HOME/Development/thirdparty_static_server
CMAKE=cmake
ARCH=linux-x86_64_icc
CXX=icpc
CC=icc

MESA_DIR=$TPDIR/mesa/11.1.2/$ARCH
VTK_DIR=$TPDIR/vtk/6.1.0/$ARCH

tar zxvf VTK-6.1.0.tar.gz
mkdir build_vtk
cd build_vtk

$CMAKE -DCMAKE_BUILD_TYPE:STRING=Release \
-DCMAKE_INSTALL_PREFIX:PATH=$VTK_DIR \
-DBUILD_SHARED_LIBS:BOOL=OFF \
-DCMAKE_C_COMPILER:STRING=$CC \
-DCMAKE_CXX_COMPILER:STRING=$CXX \
-DVTK_ALL_NEW_OBJECT_FACTORY:BOOL=true \
-DVTK_REPORT_OPENGL_ERRORS:BOOL=false \
-DVTK_USE_X:BOOL=OFF \
-DVTK_OPENGL_HAS_OSMESA:BOOL=ON \
-DOPENGL_INCLUDE_DIR:PATH=$MESA_DIR/include \
-DOPENGL_gl_LIBRARY:PATH="$MESA_DIR/lib/libGL.a;$MESA_DIR/lib/libglapi.a;crypto;dl" \
-DOPENGL_glu_LIBRARY:PATH=$MESA_DIR/lib/libGLU.a \
-DOSMESA_LIBRARY:FILEPATH="$MESA_DIR/lib/libGL.a;$MESA_DIR/lib/libglapi.a;crypto;dl" \
-DOSMESA_INCLUDE_DIR:FILEPATH=$MESA_DIR/include \
-DVTK_DEBUG_LEAKS:BOOL=false \
-DBUILD_TESTING:BOOL=false \
-DBUILD_DOCUMENTATION:BOOL=false \
-DVTK_Group_Imaging:BOOL=false \
-DVTK_Group_MPI:BOOL=false \
-DVTK_Group_Qt:BOOL=false \
-DVTK_Group_Rendering:BOOL=false \
-DVTK_Group_StandAlone:BOOL=false \
-DVTK_Group_Tk:BOOL=false \
-DVTK_Group_Views:BOOL=false \
-DModule_vtkCommonCore:BOOL=true \
-DModule_vtkFiltersFlowPaths:BOOL=true \
-DModule_vtkFiltersHybrid:BOOL=true \
-DModule_vtkFiltersModeling:BOOL=true \
-DModule_vtkGeovisCore:BOOL=true \
-DModule_vtkIOEnSight:BOOL=true \
-DModule_vtkIOGeometry:BOOL=true \
-DModule_vtkIOLegacy:BOOL=true \
-DModule_vtkIOPLY:BOOL=true \
-DModule_vtkIOXML:BOOL=true \
-DModule_vtkInteractionStyle:BOOL=true \
-DModule_vtkRenderingAnnotation:BOOL=true \
-DModule_vtkRenderingFreeType:BOOL=true \
-DModule_vtkRenderingFreeTypeOpenGL:BOOL=true \
-DModule_vtkRenderingOpenGL:BOOL=true \
-DModule_vtklibxml2:BOOL=true \
../VTK-6.1.0

make -j 8
make -j 8 install

VTK Modifications

When I went to link VisIt later on, I found that it wanted to link some X11-related libraries even though I configured without X11 support!

NOTE - This was not necessary on cori.

Find the file $HOME/Development/thirdparty_static_server/vtk/6.1.0/linux-x86_64_icc/lib/cmake/vtk-6.1/VTKTargets-release.cmake and remove the following stuff:

/usr/lib64/libSM.so;/usr/lib64/libICE.so;/usr/lib64/libX11.so;/usr/lib64/libXext.so

VTK Test Application

I followed the procedure outlined in VTK Test Application to create a program to use for testing rendering with VTK.

VTK Run

Here is the batch script I used on titan:

#!/bin/bash
#PBS -A csc194
#PBS -q debug
#PBS -l walltime=00:02:00
#PBS -l nodes=1
cd $PBS_O_WORKDIR
aprun -n 8 ./osmesavtktest

Here is how I submitted after copying the executable to $MEMBERWORK/csc194.

cp osmesavtktest $MEMBERWORK/csc194
cd $MEMBERWORK/csc194
qsub runvtk.sh

The following image was produced (same result as cori):

Osmesavtktest.png

Ice-T

After disappointing results on cori using Ice-T, I am skipping Ice-T for the Titan build.

Other VisIt Dependencies

This command will build some other VisIt dependencies. Note the use of the Intel compilers.

env PAR_INCLUDE=-I/opt/cray/mpt/7.2.5/gni/mpich2-intel/14.0/include ./build_visit --no-visit --no-thirdparty \
   --thirdparty-path $HOME/Development/thirdparty_static_server \
   --static --server-components-only --cc icc --cxx icpc --silo --hdf5 --szip

VisIt

Here is a config file based on cori. I had to edit the compiler, MPI, and the linker flags to ensure that the icpc compiler produced valid cmake test programs. The VisIt sources were from one of my branches with some scalability improvements.

titan-ext3.cmake

#/ccs/home/bjw/Development/thirdparty_static_server/cmake/3.0.2/linux-x86_64_icc/bin/cmake
##
## ./build_visit generated host.cmake
## created: Mon Apr  4 20:04:49 EDT 2016
## system: Linux titan-ext3 3.0.101-0.46-default #1 SMP Wed Dec 17 11:04:10 UTC 2014 (8356111) x86_64 x86_64 x86_64 GNU/Linux
## by: bjw

##
## Setup VISITHOME & VISITARCH variables.
##
SET(VISITHOME /ccs/home/bjw/Development/thirdparty_static_server)
SET(VISITARCH linux-x86_64_icc)

## Compiler flags.
##
VISIT_OPTION_DEFAULT(VISIT_C_COMPILER icc TYPE FILEPATH)
VISIT_OPTION_DEFAULT(VISIT_CXX_COMPILER icpc TYPE FILEPATH)
VISIT_OPTION_DEFAULT(VISIT_C_FLAGS "-m64 " TYPE STRING)
VISIT_OPTION_DEFAULT(VISIT_CXX_FLAGS "-m64 " TYPE STRING)
VISIT_OPTION_DEFAULT(VISIT_EXE_LINKER_FLAGS "-static" TYPE STRING)

##
## Static build
##
VISIT_OPTION_DEFAULT(VISIT_STATIC ON TYPE BOOL)

##
## VisIt Thread Option
##
VISIT_OPTION_DEFAULT(VISIT_THREAD OFF TYPE BOOL)

##
## Settings for this machine
##
VISIT_OPTION_DEFAULT(VISIT_MPI_C_FLAGS   "-I/opt/cray/mpt/7.2.5/gni/mpich2-intel/14.0/include" TYPE STRING)
VISIT_OPTION_DEFAULT(VISIT_MPI_CXX_FLAGS "-I/opt/cray/mpt/7.2.5/gni/mpich2-intel/14.0/include" TYPE STRING)
VISIT_OPTION_DEFAULT(VISIT_MPI_LD_FLAGS "-static-libgcc -static-libstdc++ -pthread -Wl,-Bstatic -L/opt/cray/xe-sysroot/default/lib64" TYPE STRING)
VISIT_OPTION_DEFAULT(VISIT_MPI_LIBS -Wl,--start-group /opt/cray/mpt/default/gni/mpich2-intel/14.0/lib/libmpich_intel.a /opt/cray/pmi/default/lib64/libpmi.a /opt/cray/alps/default/lib64/libalpsutil.a /opt/cray/alps/default/lib64/libalpslli.a /opt/cray/alps/default/lib64/libalps.a /opt/cray/wlm_detect/default/lib64/libwlm_detect.a /opt/cray/ugni/default/lib64/libugni.a /opt/cray/udreg/default/lib64/libudreg.a /opt/cray/xpmem/default/lib64/libxpmem.a pthread /usr/lib64/librt.a -Wl,--end-group)
VISIT_OPTION_DEFAULT(VISIT_PARALLEL ON TYPE BOOL)

##
## Special 
##
VISIT_OPTION_DEFAULT(VISIT_SERVER_COMPONENTS_ONLY ON TYPE BOOL)
SET(VISIT_SELECTED_DATABASE_PLUGINS "BOV;Cale;CaleHDF5;Curve2D;EnSight;Miranda;PDB;PlainText;SAMRAI;Silo;VTK")
VISIT_OPTION_DEFAULT(VISIT_USE_X            OFF TYPE BOOL)
VISIT_OPTION_DEFAULT(VISIT_USE_GLEW         OFF TYPE BOOL)
VISIT_OPTION_DEFAULT(VISIT_SLIVR            OFF TYPE BOOL)
VISIT_OPTION_DEFAULT(VISIT_PYTHON_FILTERS   OFF TYPE BOOL)
VISIT_OPTION_DEFAULT(VISIT_PYTHON_SCRIPTING OFF TYPE BOOL)
VISIT_OPTION_DEFAULT(VISIT_OPENGL_DIR       ${VISITHOME}/mesa/11.1.2/${VISITARCH})

# Database reader plugin support libraries
##
## The HDF4, HDF5 and NetCDF libraries must be first so that
## their libdeps are defined for any plugins that need them.
##
## For libraries with LIBDEP settings, order matters.
## Libraries with LIBDEP settings that depend on other
## Library's LIBDEP settings must come after them.
##############################################################
##

##
## VTK
##
SETUP_APP_VERSION(VTK 6.1.0)
VISIT_OPTION_DEFAULT(VISIT_VTK_DIR ${VISITHOME}/vtk/${VTK_VERSION}/${VISITARCH})
##

##
## SZIP
##
VISIT_OPTION_DEFAULT(VISIT_SZIP_DIR ${VISITHOME}/szip/2.1/${VISITARCH})

##
## HDF5
##
VISIT_OPTION_DEFAULT(VISIT_HDF5_DIR ${VISITHOME}/hdf5/1.8.14/${VISITARCH})
VISIT_OPTION_DEFAULT(VISIT_HDF5_LIBDEP ${VISITHOME}/szip/2.1/${VISITARCH}/lib sz /usr/lib z TYPE STRING)

##
## Silo
##
VISIT_OPTION_DEFAULT(VISIT_SILO_DIR ${VISITHOME}/silo/4.10.1/${VISITARCH})
VISIT_OPTION_DEFAULT(VISIT_SILO_LIBDEP HDF5_LIBRARY_DIR hdf5 ${VISIT_HDF5_LIBDEP} TYPE STRING)

VisIt Build

Here is the command I used to configure the VisIt build:

$HOME/Development/thirdparty_static_server/cmake/3.5.1/linux-x86_64_icc/bin/cmake \
-DCMAKE_BUILD_TYPE:STRING=Release \
-DCMAKE_INSTALL_PREFIX:PATH=$HOME/Development/install_static ../src

I like to build the diagnostic tools first to see that I get the linking behavior I expect for simple applications.

cd tools/diagnostics
make VERBOSE=1
cd ../..

Then I build the whole thing:

make install

I am running into some problems linking executables because cmake is adding a -Wl,-Bdynamic to some of my parallel links.

Problems

  • Link problems with diagnostic tools that required changes to the CMake install.
  • Link problems with visitconvert_par that required changes to the CMake install and to remove errant X11 dependencies from VTK install.
  • I had repeated problems linking the example simulations so I turned them off.
    • The libsimV2_static_par.a was not created right during one of my failed builds so the domainbounds sim failed to link.
    • Once I corrected the libsimV2_static_par.a library, domainbounds failed to link with MPI errors. This is because all of the MPI stuff is getting added before the target link libraries. I think we need to fix up our macros so MPI is added dead last.

Installation

After the build completed, I issued make install. This does not install the combined Libsim runtime libraries. That has to be done manually now.

cp lib/libsimV2_static_*  $HOME/Development/install_static/2.11.0/linux-x86_64/libsim/V2/lib/

Testing Libsim

I used my modified batch.c example and built it on Titan.

Makefile (nearly the same as the cori version)

CC=/opt/cray/craype/2.4.2/bin/cc
CXX=/opt/cray/craype/2.4.2/bin/CC

CFLAGS=-O3
THIRDPARTY_DIR=/ccs/home/bjw/Development/thirdparty_static_server
ARCH=linux-x86_64_icc

VTK_LIBDIR=$(THIRDPARTY_DIR)/vtk/6.1.0/$(ARCH)/lib
VTK_INCDIR=$(THIRDPARTY_DIR)/vtk/6.1.0/$(ARCH)/include/vtk-6.1
VTK_LDFLAGS=-L$(VTK_LIBDIR)
VTK_LIBS=-lvtkCommonCore-6.1\
 -lvtkImagingCore-6.1\
 -lvtkRenderingCore-6.1\
 -lvtkIOLegacy-6.1\
 -lvtkIOPLY-6.1\
 -lvtkIOXML-6.1\
 -lvtkInteractionStyle-6.1\
 -lvtkRenderingOpenGL-6.1\
 -lvtkRenderingFreeTypeOpenGL-6.1\
 -lvtkFiltersHybrid-6.1\
 -lvtkFiltersModeling-6.1\
 -lvtkGeovisCore-6.1\
 -lvtkRenderingOpenGL-6.1\
 -lvtkIOGeometry-6.1\
 -lvtkjsoncpp-6.1\
 -lvtkIOXMLParser-6.1\
 -lvtkexpat-6.1\
 -lvtkInfovisLayout-6.1\
 -lvtkInfovisCore-6.1\
 -lvtkViewsCore-6.1\
 -lvtkInteractionWidgets-6.1\
 -lvtkInteractionStyle-6.1\
 -lvtkFiltersHybrid-6.1\
 -lvtkFiltersModeling-6.1\
 -lvtkImagingHybrid-6.1\
 -lvtkIOImage-6.1\
 -lvtkIOCore-6.1\
 -lvtkpng-6.1\
 -lvtktiff-6.1\
 -lvtkjpeg-6.1\
 -lvtkmetaio-6.1\
 -lvtkDICOMParser-6.1\
 -lvtkImagingGeneral-6.1\
 -lvtkImagingSources-6.1\
 -lvtkRenderingAnnotation-6.1\
 -lvtkRenderingFreeType-6.1\
 -lvtkftgl-6.1\
 -lvtkzlib-6.1\
 -lvtkImagingColor-6.1\
 -lvtkRenderingVolume-6.1\
 -lvtkRenderingCore-6.1\
 -lvtkFiltersSources-6.1\
 -lvtkFiltersGeometry-6.1\
 -lvtkFiltersExtraction-6.1\
 -lvtkFiltersGeneral-6.1\
 -lvtkCommonComputationalGeometry-6.1\
 -lvtkFiltersCore-6.1\
 -lvtkFiltersStatistics-6.1\
 -lvtkImagingFourier-6.1\
 -lvtkImagingCore-6.1\
 -lvtkCommonExecutionModel-6.1\
 -lvtkCommonDataModel-6.1\
 -lvtkCommonMisc-6.1\
 -lvtkCommonSystem-6.1\
 -lvtkCommonTransforms-6.1\
 -lvtkCommonMath-6.1\
 -lvtkCommonCore-6.1\
 -lvtkfreetype-6.1\
 -lvtksys-6.1\
 -lvtkalglib-6.1\
 -lvtkproj4-6.1

#ICET_LIBDIR=$(THIRDPARTY_DIR)/icet/1.0.0/$(ARCH)/lib
#ICET_INCDIR=$(THIRDPARTY_DIR)/icet/1.0.0/$(ARCH)/include
#ICET_LDFLAGS=-L$(ICET_LIBDIR)
#ICET_LIBS=-licet -licet_mpi -licet_strategies

LIBSIM_INCDIR=/ccs/home/bjw/Development/install_static/2.11.0/linux-x86_64/libsim/V2/include
LIBSIM_LIBDIR=/ccs/home/bjw/Development/install_static/2.11.0/linux-x86_64/libsim/V2/lib
LIBSIM_LDFLAGS=-L$(LIBSIM_LIBDIR)
LIBSIM_LIBS=-lsimV2_static_par

GL_LIBDIR=$(THIRDPARTY_DIR)/mesa/11.1.2/$(ARCH)/lib
GL_LDFLAGS=
GL_LIBS=$(GL_LIBDIR)/libGL.a $(GL_LIBDIR)/libglapi.a -lcrypto -ldl

CPPFLAGS=-I. -I$(LIBSIM_INCDIR) -I$(VTK_INCDIR) -DPARALLEL
LDFLAGS=$(LIBSIM_LDFLAGS) $(VTK_LDFLAGS) $(ICET_LDFLAGS) $(MPI_LDFLAGS)
LIBS=$(LIBSIM_LIBS) $(VTK_LIBS) $(ICET_LIBS) $(MPI_LIBS) $(GL_LIBS) -ldl -lpthread -lz -lm

SRC=batch.c extract.c
OBJ=$(SRC:.c=.o)

all: batch_par

batch_par: $(OBJ)
	$(CXX) -o batch_par $(OBJ) $(CFLAGS) $(LDFLAGS) $(LIBS)

clean:
	$(RM) $(OBJ) batch_par

.c.o:
	$(CC) $(CFLAGS) $(CPPFLAGS) -c $<

Modifications

On Titan, the code seemed to hang up on setting up the environment. That code should not be necessary anyway and it represents a big chunk of the slowdown we see with SENSEI. Let's cut it out.

#if 0
    if(sim.par_rank == 0)
        env = VisItGetEnvironment();

    /* Pass the environment to all other processors collectively. */
    VisItSetupEnvironment2(env);
    if(env != NULL)
        free(env);
#else
    putenv("VISITARCHHOME=/ccs/home/bjw/Development/install_static/2.11.0/linux-x86_64");
    putenv("VISITHOME=/ccs/home/bjw/Development/install_static/2.11.0");
    putenv("VISITPLUGINDIR=/ccs/home/bjw/.visit/2.11.0/linux-x86_64/plugins:/ccs/home/bjw/Development/install_static/2.11.0/linux-x86_64/plugins");
#endif

Hey, that seemed to work. Of course, now the code is making images but they don't look right (bad compositing?). Sigh.

  • I replaced __restrict__ with RESTRICT in the ProgrammableCompositor so I could easily remove vectorization. That worked for now as a workaround.

16 cores

Here is my job script for running on 16 cores:

#!/bin/bash
#PBS -A csc194
#PBS -q debug
#PBS -l walltime=00:05:00
#PBS -l nodes=1
cd $PBS_O_WORKDIR
SIMDIR=/ccs/home/bjw/Development/libsimrender/batch_par
date
aprun -n 16 $SIMDIR/batch_par -domains 4,2,2 -maxcycles 10 -export 0 -render 1 -image-width 1024 -image-height 1024 -trace foo -timing 
date