Buildbot

VisIt has a buildbot to test VisIt after (almost) on every commit, and on more platforms, than the existing nightly regression system.


Update: VisIt's buildbot setup is now hosted at LLNL.

Access to the webserver is not generally available, however we may configure outgoing emails to provide feedback to developers outside of LLNL.

Current buildbots:

Dependencies:

  • deps-osx
  • deps-linux

Per commit builds:

  • main-osx
  • main-linux

Nightly full builds:

  • full-osx
  • full-linux

On the horizon:

  • Nightly static builds
  • Packing tests
  • A windows build

New Buildslave Setup Instructions

First, make sure python is installed & that you can install python packages to it. Most Linux and MacOS X computers will already have a suitable version of Python.

Some additional Windows instructions can be found at http://trac.buildbot.net/wiki/RunningBuildbotOnWindows.

Python for Windows

If you can install a prebuilt Python binary for Windows, good for you. If you are not allowed to install software then it is possible to use VisIt's prebuilt Python binaries from the windowsbuild directory. Copy all of the files and directories into C:\Python27\ so you will be able to easily run python.exe. It's best to put Python in an easy-to-type location since you're probably on a computer where IT policies prevent you from editing your PATH environment variables.

After you have copied the files, your C:\Python27\ directory should contain:

 Directory of c:\Python27

04/06/2012  02:14 PM    <DIR>          .
04/06/2012  02:14 PM    <DIR>          ..
04/06/2012  02:13 PM    <DIR>          DLLs
04/06/2012  02:13 PM    <DIR>          include
04/06/2012  02:22 PM    <DIR>          Lib
04/06/2012  02:14 PM    <DIR>          libs
11/23/2011  10:29 AM            26,624 python.exe
11/23/2011  10:29 AM         2,942,976 python27.dll
11/23/2011  10:30 AM            27,136 pythonw.exe
04/06/2012  02:22 PM    <DIR>          Scripts
               3 File(s)      2,996,736 bytes
               7 Dir(s)  858,754,478,080 bytes free

Open a command shell by typing cmd into the command line in the Windows menu. In that command shell, verify that you can run Python by typing: C:\Python27\python.exe. If Python runs then type import site at the >>> prompt. If you don't get any errors then you're in good shape and you can proceed.

Installing setuptools

Install the distribute package for setuptools in order to get the easy_install program, which will simplify installing your buildbot. Obtain the distribute package at http://pypi.python.org/pypi/distribute.

Linux and Mac

You should be able to unzip the distribute tar file & run:

>tar zxvf distribute-0.6.25.tar.gz
>cd distribute-0.6.25
>python setup.py install

Then install 'virtualenv'

>easy_install virtualenv

Windows

The procedure is slightly different on Windows, mostly due path issues. Extract the files from the distribute tar file.

> cd distribute-0.6.25
> C:\Python27\python.exe setup.py install

Then install 'virtualenv'

> C:\Python27\Scripts\easy_install.exe virtualenv

Setting up buildslave

Now, setup the buildslave: (all of the files will live under "buildbot")

>mkdir buildbot
>cd buildbot
>virtualenv --no-site-packages sandbox
>sandbox\activate.bat
>easy_install buildbot-slave
>cd sandbox
# setup the buildslave
>buildslave create-slave [buildbot-dir] beetown:9989 [buildbot-name] [buildbot-password]
# start the buildslave
>buildslave start $VIRTUAL_ENV\[buildbot-dir]

Windows

There are several Python packages that you'll need to install in order to get buildbot working on Windows. With nearly all of the packages being installed via executable installers, you'll need admin access to set up Python. The pywin32 package will want to install 3 DLLs into your Windows/system32 directory. Since you'll need admin access, skip setting up the sandbox and virtualenv.

> cd C:\
> mkdir buildbot
> cd buildbot

Unzip the buildbot slave zip file into C:\buildbot

> cd buildbot\buildbot-slave-0.8.6p1
> C:\Python27\python.exe setup.py install
> cd ..

Now you can create your build slave.

# setup the buildslave
> C:\Python27\Scripts\buildslave create-slave [buildbot-dir] beetown:9989 [buildbot-name] [buildbot-password]
# start the buildslave
> C:\Python27\Scripts\buildslave start [buildbot-dir]

buildbot-dir is the name of a directory such as visit-win7-x86. It should match the name of your build bot slave. buildbot-name is the name of the build bot slave and is typically something like visit-win7-x86. buildbot-password is the password that the buildbot admin would have given you to connect to the buildbot master.

Previous SCI Setup

Build Master

Administered by: User:tfogal.

Status interface: ci.sci.utah.edu, port 8010 (bookmark it!)

Master/slave communication port: 9989

Build Delays

The build master watches the anonymous subversion repository. To avoid hammering the svn server, they only check for changes every few minutes. Further, the anonymous repository is 10 to 20 minutes behind the main repository. Finally, there is a little bit of wiggle room associated with checking for commits, so that if you do two commits within 30 seconds of one another, that won't trigger the builds to run twice.

All of this adds up to mean: if you do a commit, be patient in waiting for the build results to appear. A good rule of thumb would be to check back thirty minutes after you've committed; if compilation broke, it's likely that one of the slaves would have caught it by then.

Builders

There are a variety of builders meant to test VisIt in a variety of configurations.

  • deps -- These builders will build all of VisIt's dependencies. These only get run when someone commits a change to src/svn_bin/build_visit.
  • main -- These perform incremental builds on every commit (or set of commits if they hit the repo close together). The intention is to catch build issues quickly.
  • full -- These run complete builds every night, by wiping the directory and doing a fresh svn export.
  • test-*-s -- These run some subset of the test suite, based on the name. The "s" here indicates serial mode test runs.

Slave Setup

Summary

  1. Install the relevant software stack
    • any of VisIt's build dependencies that aren't in build_visit. A compiler, for example.
    • python, in /usr.
      • Python Imaging Library (Linux only)
      • twisted
      • zope.interface
    • the latest buildbot release.
  2. Setup a new, unprivileged account (do not use your normal system account!).
  3. Setup your slave name and password with Tom
  4. run 'buildbot create-slave `pwd` ci.sci.utah.edu:9989 name password' in a new, empty directory
  5. run 'make start' in the aforementioned directory.

Details

Install the latest buildbot on your host. Setup a directory which is meant to hold your VisIt builds. If at all possible, this should be on a separate account which is used exclusively for the purpose of the build slave. Do not use an account with full system privileges for your build slave!

In the current configuration, you will need about 10 gigabytes of free space. This is likely to grow in the near future, but 20 gigabytes seems like a reasonable upper limit for the medium term.

You'll then need a slave name and password to be able to register with the buildbot master. Unfortunately, there is no automated method to generate these at present. Tom is currently managing the build master; you can email User:tfogal, or send a ping to the -developers list, where he will see it.

Once you've got buildbot installed and have received your slave name and password from Tom, run:

buildbot create-slave `pwd` ci.sci.utah.edu:9989 name password

in the directory you created to host your buildbot configuration and VisIt builds. Replace name with your slave's name and password with your slave's password.

Manually Installing Dependencies

You should install the packages you need from your distribution's package manager. If for some reason that is not possible...

  • easy_install:
curl -o setuptools-0.6c11.tar.gz http://pypi.python.org/packages/source/s/setuptools/setuptools-0.6c11.tar.gz#md5=7df2a529a074f613b509fb44feefe74e 
tar -xzvf setuptools-0.6c11.tar.gz
cd setuptools-0.6c11
python setup.py install
  • install necessary python packages
easy_install twisted
easy_install zope.interface
easy_install buildbot
  • Local PIL (easy_install is broken, it seems)
curl -o Imaging-1.1.6.tar.gz http://effbot.org/media/downloads/Imaging-1.1.6.tar.gz 
tar -xzvf Imaging-1.1.6.tar.gz
cd Imaging-1.1.6
python setup.py install
  • setup the slave:
 buildbot create-slave /path/to/sandbox/build-bot-slave ci.sci.utah.edu:9989 [slave-name] [password]
 cd /path/to/sandbox/
 edit build-bot-slave/info/admin: add your name & email address.
 edit build-bot-slave/info/host: add info about your machine.

Running

Once configured, cd to the slave directory and run make start to start the slave.

Do note that the build can be a significant drain on system resources. It is highly recommended that you nice the startup command to keep the system responsive for other tasks. Linux users may want to consider adding ionice -n7 in addition to nice.

We encourage you to run the buildbot slave under env -i. However, currently parts of the build require the HOME environment variable to be set to succeed.

Detailed Details

  • launch the slave using "env" and "nice":
export EXE_PATH="/path/to/python/2.6.2/bin/:/other/paths:/etc:/etc"
cd /path/to/sandbox/
env -i PATH="$EXE_PATH" HOME="$PWD" nice buildbot start build-bot-slave/

Make sure to define HOME in the limited environment b/c VisIt tries to write perl to $HOME at launch.

Depending on its configuration, your bot should start building on its own. For now, you can also point your browser at the status interface and force builds.

  • Force a "deps" build.

Every slave system has an associated "deps" builder. This builder is responsible for building all of VisIt's dependencies, and therefore it must be run before any of the other builders. Furthermore, it will only fire on its own if somebody modifies build_visit, so it won't run on its own very often. Therefore it is recommended that you force an associated dependency build manually the first time you've started up your new slave, to make sure the system bootstraps itself.

Running: Windows

Do note that you must run the slave under the Visual Studio shell of the VS install you'd like to compile with.

You can also setup Buildbot to run as a service, using the buildbot_service.py script. See the documentation on the buildbot website.

Running @ LLNL

Issues/Roadblocks:

  • The buildmaster lives on a non-standard port & requires a (daily) firewall exemption.
  • The buildbots export the http mirror repo & require (daily) outgoing authentication.