Using VisIt in an mxterm

Parallel computers at LLNL provide a script called mxterm that allows you allocate a parallel xterm for many nodes of a parallel computer. Inside the mxterm, you can repeatedly run your parallel jobs using mpirun or srun, depending on the system. The nice thing about invoking VisIt from within an mxterm is that you don't lose your nodes when the compute engine exits, as would be the case if you ran VisIt conventionally.

mxterm is also good for:

  • getting dedicated nodes for compiling VisIt
  • running many short-lived parallel programs as when debugging
  • preventing your nodes from being lost if your program dies

invoking mxterm

The mxterm command takes 3 basic arguments: #nodes, #procs, time in minutes.

# Get 100 processors for 30 minutes on a machine that has 2 processors per node
mxterm 50 100 30
# Get 1 node with 8 tasks for 30 minutes in pdebug:
mxterm 1 8 30 -q pdebug

Running VisIt at LC

This section will show how to run VisIt from within an mxterm at LC, displaying back to your desktop computer.

x86_64 machines

LC's more recent machines run 64-bit Linux. This includes machines like zeus, hera, yana, prism, aztec. These machines use MOAB for their batch scheduling and they use srun internally as the parallel launcher. If you are running VisIt at LC and you want to run within an mxterm, this is the method to use. Once you have a parallel mxterm opened, you can run a parallel instance of VisIt like this:

visit -np 100 -l srun


Running the engine directly

In VisIt 2.0 and later, it is possible to launch the compute engine directly so that it creates the viewer. This is in reverse order of typical operation but it can be useful if you want to start the engine inside of totalview. If you wanted to run the VisIt compute engine directly without the use of the visit script, you could do this:

setenv LD_LIBRARY_PATH /usr/gapps/visit/2.3.1/linux-x86_64/lib:/usr/local/lib
setenv VISITHOME /usr/gapps/visit/2.3.1
setenv VISITPLUGINDIR /usr/gapps/visit/2.3.1/linux-x86_64/plugins
srun -n 100 /usr/gapps/visit/2.3.1/linux-x86_64/bin/engine_par -reverse_launch
  • Be sure to substitute the right version number for the one you want to run.

Of course, this pattern of running your parallel application directly under srun in the mxterm will work for applications other than VisIt.

Running VisIt in Client/Server mode

Whenever possible, we recommend using VisIt in Client/Server mode. This means that you run the native version of VisIt for your desktop computer on your desktop and then you connect through VisIt to LC where your data will be read and processed. Running in Client/Server mode means that you will have faster, more interactive graphics because you are not incurring costs for encryption of your X11 session, you are less susceptible to networking latency.

VisIt provides host profiles that tell it how to reserve nodes and launch its compute engine at LC. When you use VisIt's Open window to select files that exist at LC, VisIt will prompt you to choose a profile that will govern how you would like to launch your compute engine. However, this means that VisIt will be taking care of the whole launch process for you. When you have an existing mxterm you don't want VisIt handling the entire launch process or you'll end up with additional nodes being reserved for VisIt's compute engine. When you want to run in Client/Server mode using an mxterm, you'll need to enter the VisIt launch commands yourself at the mxterm prompt. Fortunately, VisIt does provide a way for you to do that.

Create a new host profile

It's best to create a new host profile for the machine for which you have an mxterm.

Mxterm-HostProfile.png
Host Profile window

Follow these steps:

  1. Open the Options->Host profile window
  2. Click on the host that you want to use (e.g. LLNL Hera)
  3. Click on the Launch Profiles tab
  4. Select the host profile that matches best. This is usually a parallel host profile such as "parallel interactive debug".
  5. Click the Copy button to create a copy of the parallel launch profile
  6. Append (mxterm) to the end of the profile name so you can tell it apart from your other profiles
  7. Add -norun engine_par in the Additional arguments to tell VisIt that you will be launching the compute engine yourself.
  8. Click the window's Apply button and dismiss it
  9. You may want to save your settings so you don't need to repeat this procedure (Options->Save settings)

Launch the compute engine yourself

Since you've created a new host profile for your mxterm, you now need to go and select files on the remote computer as you normally would using the Open File window. Once you select a file and open it, VisIt will prompt you to select which host profile you want to use. Select your new mxterm profile.

Mxterm-chooser.png
Engine chooser window

Since you're choosing a profile that contains -norun engine_par, magic words that tell VisIt not to launch the engine, you'll have to launch the engine yourself. Fortunately, VisIt prints the instructions for doing so in the terminal where you launched VisIt. You'll need to set the environment as indicated and run the given visit command by pasting these values from your terminal into your mxterm window. The commands will look like the following. You should paste the commands from your terminal into the mxterm.

   setenv LD_LIBRARY_PATH /usr/gapps/visit/2.3.1/linux-x86_64/lib:/usr/local/tools/mvapich-gnu/lib/shared:/usr/local/lib
   setenv LD_LIBRARY32_PATH /usr/gapps/visit/2.3.1/linux-x86_64/lib
   setenv LD_LIBRARYN32_PATH /usr/gapps/visit/2.3.1/linux-x86_64/lib
   setenv LD_LIBRARY64_PATH /usr/gapps/visit/2.3.1/linux-x86_64/lib
   setenv LIBPATH /usr/gapps/visit/2.3.1/linux-x86_64/lib
   setenv VISITHOME /usr/gapps/visit/2.3.1
   setenv VISITHELPHOME /usr/gapps/visit/2.3.1/linux-x86_64/help
   setenv VISITULTRAHOME /usr/gapps/visit/2.3.1/linux-x86_64/ultrawrapper
   setenv VISITPLUGINDIR :/g/g19/whitlocb/.visit/linux-x86_64/plugins:/usr/gapps/visit/2.3.1/linux-x86_64/plugins
   setenv VISIT_MESA_LIB /usr/gapps/visit/2.3.1/linux-x86_64/lib/libOSMesa.so
   setenv PYTHONHOME 
   setenv TRAP_FPE 
   setenv MESA_GLX_FX disable
   srun -n 8 -p pdebug /usr/gapps/visit/2.3.1/linux-x86_64/bin/engine_par -host 128.255.255.255 \
   -port 5600 -dir /usr/gapps/visit -noloopback -forcestatic -timeout 480 -norun engine_par \
   -clobber_vlogs -key edd3215c3f35f0291927

Before you paste any setenv commands into your mxterm shell, you better be running csh or tcsh. If you run sh or bash as your shell then you'll need to use an alterate syntax for setting the environment variables.

Once you paste the commands into your mxterm, VisIt should launch and connect back to the VisIt running on your desktop computer. You can verify that VisIt has connected when the launch progress window goes away and your engine information appears in the File->Compute engines window.

Mxterm-cewindow.png
Compute Engine window

Things to note:

  • If your VisIt compute engine crashes and VisIt needs to relaunch it, you'll have to paste a new launch command into your mxterm
  • The new command will be similar but not identical to the previous launch command