Every VisIt component (gui, viewer, engine, mdserver, etc) can employ a "debug stream" to output debugging information into files.
Multiple levels of debug information
There are multiple levels of debug information, ranging from "1" to "5". To date, the conventions for assigning a message to a debug level are somewhat lax. Debug level 1 is typically reserved for:
- error conditions;
- violations of assumptions;
- very high level flow information for orienting where the rest of the debug info is coming from.
Debug level 5 information contains the most mundane information. Debug levels 2 through 4 are used less commonly and their level importance ranges between that of level 1 and level 5. Many database plugins write method names and metadata contained in a file format to debug4 to facilitate debugging in a released version of VisIt.
How to enable
You can turn on debug information with on the command line:
visit -debug N
On Windows, there is a shortcut available from VisIt's Start Menu entry: VisIt with debug logging. It uses debug level 5. 'N' is a number between 1 and 5. If the debug level is specified as '3', then each of the first three debug levels are outputted to files. Note that debug level 1 is mirrored to debug levels 2 through 5, debug level 2 is mirrored to debug levels 3 through 5, and so on. Then each debug level contains all of the information for its level and all lower-numbered levels.
Finding the output
The output of the debug stream depends on the component and the debug level. The output of the fifth debug stream for the mdserver component is "A.mdserver.5.vlog". The output of the first debug stream for the mdserver component is "A.mdserver.1.vlog".
You can encode the processor ID (pid) of a component into the filename (A.mdserver.26133.5.vlog) by also adding the "-pid" flag on the command line. This is especially useful when debugging a case where VisIt has a component core, then re-launches that component. For example, if the engine crashes, VisIt will launch a new engine. Without the -pid flag, the engine_ser.5.log from the first engine process will be overwritten by that of the second engine, meaning that potential information for debugging the crash would be lost.
- On Windows, look for the debug logs in My Documents/VisIt.
- On Windows, the files will always have pid information in their filenames.
Each of the five debug streams are implemented as C++ std::ostream's. They are accessed by including the header "DebugStream.h". You use them like any other ostream:
debug5 << "I am writing to debug level " << 5 << endl;
In actuality, each "debugN" is a macro which are simply used as ostreams. These macros reference five true ostreams (debug1_real, debug2_real, debug3_real, debug4_real, and debug5_real). The purpose of the macro is to determine if debug files have been enabled. If so, the debug messages will get sent to the right ostream. Otherwise, the debug messages are effectively "no-ops". This allows developers to write code that always writes to the debug stream. There will only be a performance degradation if debug info is on (in which case it is necessary).
Note that sometimes it is necessary to access the actual ostream and not a macro. An example of this would be the VTK method "PrintSelf", which takes an ostream as an argument. In this case, a developer can pass in the debugN_real argument, as in:
Exceptions and debug files
Every exception in VisIt logs an entry into debug1. The message contains the word "Exception". This allows a developer to look at a debug file and determine if any exceptions occurred by searching for this keyword. In addition, the filename and line number that the exception occurred at is logged. For example:
(InvalidFilesException) MDServerConnection.C, line 664: There was an error opening /usr/common/homes/h/hrchilds/marcday/src/bin/junk.jnk. It may be an invalid file. VisIt tried using the following file format readers to open the file: <No suitable plugins were identified>
- Some components are launched on both the local and remote machines. If your problem is on the remote machine, make sure you are looking at mdserver.5.log on the remote machine, not the local machine.
- If you are using a parallel engine, the resulting number of small writes to the engine_par debug files can slow the file system considerably.
Useful things in debug files
- Every exception (debug1)
- The beginning and of every avtFilter execution (debug1)
- Which plugins were loaded by a component (debug1)
- Which plugins failed to load (debug1)
- The amount of geometry (measured in number of triangles) for a plot (debug5)
- Every communication between processes, which is logged through Xfer (debug5)
- Out of memory errors, provided that the out of memory problem occurred with a
newand not a
malloc. Thus finding this error means you have ran out of memory, but the lack of such a message doesn't mean you aren't hitting an out of memory condition.
Be sure to also look here UserDebug