Ctest

Where

The cdash dashbaord is hosted at cdash.visit.lbl.gov

Quick start

Ctest support is integrated in to VisIt's build system and may be enabled by setting BUILD_TESTING=ON in the cmake cache (CMakeCache.txt). Note that you need to checkout data and test directories next to your src checkout. After building with testing enabled invoking the ctest command from the build directory will run the tests and generate a report. Such a run could even be uploaded to the dashboard by adding the track, for example ctest -D Experimental. Nightly and continuous submissions are best automated using a CMake script.

Organization

The dashboard is organized as a table with a row for each submitting system. The columns represent various stats and show deltas. Each of the table entries can be clicked to access more detailed information. The 3 tracks, Nightly, Continuous and Experimental, can also be grouped. We're using the trusted label to further organize submissions.

Nightly

Nightly submissions are run once per day using a dated checkout so that submitters grab the same revision no matter when the nightly submission is executed. Nightly runs are typically not incremental, a completely fresh build is submitted.

Continuous

Continuous submissions probe the repository periodically for updates, grab them, build incrementally and run the regression suite. Continuous submissions give developers quick feedback about their changes. As developers make commits the Continuous submissions can be used to verify that the commit hasn't broken key features or the build process on other platforms. Results should show up in the Continuous track any where withing a within a few minutes to an hour of the commit.

Features

Continuous submissions

The figure above shows a snapshot of the continuous section of the dashboard. We'll use the highlighted entry to point out some of the useful features. This also applies to the Nightly and Experimental tracks as well. For example we can see that there were 12 files changed which resulted in 70 new test failures. The new failures are indicated by +70 superscript in the Fail column and -70 subscript in the Pass column. Had instead there been 70 tests fixed, this would be reversed.

Drilling down

viewing the new failures.

The first thing we might want to do is look at the new failures. The test report will contain image diffs and stderr output either of which may be enough information for us to amend the commit. If we click on the 80 in the Test Fail column we will see all the failures not just the new one. To isolate the new failures, we click on the +70. This will bring up a list of just the test that failed as a result of our commit. From there we can click on the Failed status to see the test's output and an image difference.

New failures

viewing the source changes

We also may be interested in which files changed and their diffs. These can be conveniently accessed from the dashboard through it's websvn integration. For example clicking on the 12 in the Update Files column will bring us to a page listing the modified files. Clicking on the individual files will bring up a diff of the change.

Modified files
Diff of one of the changed files

Configuring a Nightly submission

Nightly runs are usually kicked off automatically using cron or the Windows task scheduler. Each site will need a CMake configuration that configures various build options, for example the VISIT_SITE_CONFIG must be set. Example scripts and instructions are located here.