VTKm

VTKm is a visualization library that enables data processing on manycore devices such as GPU's.

Getting started with VTK-m requires a certain set of tools:

  • C++11 Compiler
  • CMake 3.3 or later

VTKm benefits from additional optional parallel toolkits:

Windows

I first tried building VisIt with VTKm support on Windows. I started with CUDA 8.0.

A C++11 compiler is required to build VTKm. If you develop on Windows you have to use Visual Studio 2013 or later. Note that the Visual Studio 2017 release candidate is not compatible with CUDA 8.0 so don't try using it or the nvcc compiler will complain that the C++ compiler is not supported.

I built VTKm in Visual Studio 2013 and installed into the windowsbuild/MSVC2013-x64/VTKm/170301 directory so I could have an install that I could point VisIt to in my config-site file. I added this line to my config-site file:

VISIT_OPTION_DEFAULT(VISIT_VTKM_DIR ${VISITHOME}/VTKm/170301)

When I tried to configure VisIt in CMake, it flopped. The FindVTKm.cmake file didn't work on Windows with the latest version of VTKm.

I made some changes to FindVTKm.cmake so it locates the VTKmConfig.cmake file via find_package. I also changed the CMakeLists.txt for the VisIt vtkm directory so it makes a visit_vtkm shared library always that can be linked to the AVT Pipeline libraries.

I ended up also having to port VisIt's VTKm code to the newest version of the VTKm library. When I did that, the data for a Contour plot that VTKm produced was not right. I had to rewrite the VTKmToVTK function to produce the right connectivity information.

Mac

I wanted to switch to my Mac to continue development. This means that I had to get VTKm working on Mac. First, I had to get a new CMake since VTKm requires at least version 3.3.

I wanted to try out the TBB execution environment for VTKm so I downloaded TBB sources and Mac build and unzipped them into my VISITHOME/thirdparty_shared/2.11.0 directory. Then I typed make to build TBB in place.

Next, I built VTKm with support for TBB:

git clone  https://gitlab.kitware.com/vtk/vtk-m.git
mkdir vtk-m-build
cd vtk-m-build
cmake -DCMAKE_BUILD_TYPE:STRING=Release \
-DCMAKE_INSTALL_PREFIX:PATH=$HOME/Development/thirdparty_shared/2.11.0/VTKm/170306/i386-apple-darwin16_clang \
-DVTKm_ENABLE_TBB:BOOL=ON -DTBB_ROOT=$HOME/Development/thirdparty_shared/2.11.0/tbb2017_20170118oss \
../vtk-m
make
make test
make install

After installing VTKm and TBB, I added lines to my VisIt config-site file so it would know to look for VTKm and another line that selects C++11 compilation since that is needed for VTKm. Setting the TBB_ROOT variable is also necessary to make VTKm's TBB location cmake logic find TBB, even if the library was built with it.

VISIT_OPTION_DEFAULT(VISIT_CXX_FLAGS "-std=c++11 -Wno-deprecated-declarations -fno-common -fexceptions" TYPE STRING)
VISIT_OPTION_DEFAULT(VISIT_VTKM_DIR ${VISITHOME}/VTKm/170306/${VISITARCH})
VISIT_OPTION_DEFAULT(TBB_ROOT ${VISITHOME}/tbb2017_20170118oss)

With those changes, VTKm was detected by VisIt's build:

--   VTKM found - headers only - no libs
-- Successfully loaded VTK-m component Base
-- Successfully loaded VTK-m component Serial
-- Failed to configure VTK-m component CUDA: !VTKm_ENABLE_CUDA
-- Could not load optional VTK-m component CUDA
-- Successfully loaded VTK-m component TBB
--   VTKm_LIBRARIES = vtkm;vtkm_cont;optimized;$HOME/Development/thirdparty_shared/2.11.0/tbb2017_20170118oss/lib/libtbb.dylib;debug;$HOME/Development/thirdparty_shared/2.11.0/tbb2017_20170118oss/lib/libtbb_debug.dylib
--   VTKm_INCLUDE_DIRS = $HOME/Development/thirdparty_shared/2.11.0/VTKm/170306/i386-apple-darwin16_clang/include;$HOME/Development/thirdparty_shared/2.11.0/tbb2017_20170118oss/include
--   VTKm_LIBRARIES = vtkm;vtkm_cont;optimized;$HOME/Development/thirdparty_shared/2.11.0/tbb2017_20170118oss/lib/libtbb.dylib;debug;$HOME/Development/thirdparty_shared/2.11.0/tbb2017_20170118oss/lib/libtbb_debug.dylib
--   VTKM_FOUND = 1
--   VTKM_INCLUDE_DIR = $HOME/Development/thirdparty_shared/2.11.0/VTKm/170306/i386-apple-darwin16_clang/include;$HOME/Development/thirdparty_shared/2.11.0/tbb2017_20170118oss/include
--   VTKM_LIBRARY_DIR = $HOME/Development/thirdparty_shared/2.11.0/VTKm/170306/i386-apple-darwin16_clang/lib
--   VTKM_LIB = vtkm;vtkm_cont;optimized;$HOME/Development/thirdparty_shared/2.11.0/tbb2017_20170118oss/lib/libtbb.dylib;debug;$HOME/Development/thirdparty_shared/2.11.0/tbb2017_20170118oss/lib/libtbb_debug.dylib
--   VTKm_CUDA_FOUND = 
--   CUDA_INCLUDE_DIRS = 
--   CUDA_CUDART_LIBRARY = 

After making some other build changes to VisIt to permit a visit_vtkm library that includes TBB classes, I ran a Contour plot of noise.silo, reflected 64 times. I observed the engine taking ~1400% of the CPU in top on my 12 CPU system. So, TBB is running! Not faster yet! I'll build release now.

Initial Results

VTKm+TBB did beat the serial implementation of VisIt's VTK-based contour filter but not by a lot and it used threads on 12 cores to do it. By comparison, parallel VisIt on 8 cores was far faster than VTKm+TBB. This was on an 8 domain dataset where domains were about 3M nodes. I'll have to try some other tests where I've seen VTKm working on large domain sizes of 400^3 or so.

==================================================
Dataset: 8-domain noise.silo, reflected 2x2x2
==================================================

==================================================
Timings - serial engine with VTKm+TBB on 12 core Mac
==================================================
VTKToVTKm: 1.275116
ExecuteDataTree_VTKM: 16.208744
VTKmToVTK: 4.180051
total:  21.663911

avtContourFilter: 21.674423

==================================================
Timings - serial engine with VTK
==================================================
ExecuteDataTree_VTK: 38.885068
total:  38.885068

avtContourFilter: 38.889326

==================================================
Timings - parallel engine with VTK, 8 cores
==================================================
ExecuteDataTree_VTK: 4.873866
total:  4.873866

avtContourFilter: 4.893914