SVN Scripts for VisIt Development

Overview of using Subversion with the VisIt project

There are two classes of actions you will make with Subversion. One class centers around making and accessing directories (like branches) to do development in. The other class centers around operations that happen at the per file level. Subversion does not draw this distinction. They consider these two classes to be part of the same, larger class, all of which are done using native Subversion commands. The motivation, then, for this artificial distinction is that the VisIt project will provide scripts for the first class, so that development for the project will be done consistently. The second class will still be done directly using native Subversion commands.

As previously stated, Subversion is simply a repository for managing directories and files in those directories. Each project creates their own directory structure, although many utilize the trunk, tags, and branches pattern prescribed by the Subversion team. Keep in mind, however, that concepts such as "making a branch" really consist of copying a directory structure from one location (usually trunk) into another (in this case, the branches directory). In addition, this directory copy results solely in bookkeeping on a remote server. To actually access the files in the directory, you need to do a "checkout" (in Subversion parlance).

With that in mind, we can then explicitly distinguish between the activities done by VisIt scripts and those done with native SVN commands.

Done with scripts:

  1. Creating, moving, or deleting branches or tags (i.e. top-level directories)
  2. Merging between branches, tags, and/or the trunk
  3. Listing branches and tags (i.e. top-level directories in the directory structure)
  4. "Checking out" (read: accessing the VisIt tree or a portion of a VisIt tree from a single top-level directory for the first time)

Done with native SVN commands:

  1. Deleting a file in a directory
  2. Change the execution permission of a file
  3. Updating changes from the server to an existing "checked out" directory
  4. Committing changes from the current "checked out" directory to the server


VisIt scripts for Subversion development

These scripts live in the src/tools/dev/scripts directory. To retrieve them into a common location, like ~/scripts where you can add them to your path permanently, cd into your home directory and type:

svn co svn+ssh://USERNAME@portal-auth.nersc.gov/project/projectdirs/visit/svn/visit/trunk/src/tools/dev/scripts

You might want to periodically cd into ~/scripts and type svn update.

mk_branch_from_trunk

mk_branch_from_trunk will make a new directory structure from the trunk using Subversion cp commands. Note that this does not "mount" the directory structure so that it is available to you on your machine. That is done with the co_branch script described subsequently.

Examples:

  • mk_branch_from_trunk vr_work

There is no "mk_trunk" command because the trunk has already been made.

There is no "mk_tag" command, because the directories only are placed in the tags directory when they have finished their cycle of being a "release candidate". The "finalize_release" script is the one that places directories into the tags directory.

co_[branch|tag|trunk|rctrunk]

There are four scripts to do checkouts, co_branch, co_tag, co_trunk, and co_rctrunk that do "checkouts" from the different top-level directories (branches, tags, trunk, and rctrunk respectively) in the VisIt repository structure. So, mk_branch_from_trunk can be thought of as "mkdev" in ClearCase, where co_branch can be thought of as "cleartool setview -login". The checkout copies the appropriate subdirectory of the Subversion repository into your current present working directory (PWD). Note that co_tag allows you to access the release, but that you should never do development in that directory, since it is considered locked.

Examples:

  1. co_trunk /src
    • Checks out the /src directory of the trunk into the PWD. (So you get $PWD/src)
  2. co_tag 1.6
    • Checks out the entire 1.6 release into the PWD. (So you get $PWD/1.6/src, $PWD/1.6/data, etc, all from the 1.6 release)
  3. co_branch vr_work /third_party
    • Checks out the /branches/<username>/vr_work/third_party directory into the PWD. (So you get $PWD/third_party from that branch)

del_[branch|tag]

These scripts will delete a branch or a tag. del_tag should never be used, unless a mistake has been made.

mk_rc

This script will make a copy of the trunk into the branches directory to form the Release Candidate. It is run one time per release, presumably by the VisIt project leader. If a tagged release needs to be patched, the script mk_rc_from_tag can be used, which makes the copy from the tags directory instead of the trunk directory.

mk_branch_from_rc

This script will make a copy of the Release Candidate into the branches directory so that a developer can do development work for the release candidate that is not directly on the RC trunk. After creating the branch, changes to the RC can be picked up using the script merge_RC_to_branch. Changes done on the branch can be put on the RC using the script merge_branch_to_RC.

finalize_release

This script will change a release candidate to become a tag of the release. It is run one time per release, presumably by the VisIt project leader.

ls_[branches|tags]

These scripts will list the contents of the branches and tags directory, primarily to help developers locate existing top-level directories.

VisIt scripts for merging

There are five scripts that manage merging for you. They are:

  1. merge_trunk_to_branch
  2. merge_branch_to_trunk
  3. merge_RC_to_branch
  4. merge_branch_to_RC
  5. merge_rctrunk_to_trunk

Advanced topics

Merging is still being hashed out, so consider this section under development. If you are just familiarizing yourself with SVN, ignore this section.

There is a special directory, peer to src, test, etc, named svninfo. That directory contains two files: Rev_fromTrunk and Rev_toTrunk. Inside of the trunk, the values of these two files are always the same. But on branches, they can differ. These files simply contain a revision number. Each time a merge script is executed, it updates the revision number.

Consider the following example. The trunk is at revision 923. Branch1 and Branch2 are created from revision 923. Branch3 is later created at revision 940. (Since no one merged into the trunk, the original directory structure of Branch3 will be identical to that of Branch1 and Branch2.) Work happens on all three branches. Branch1 gets up to revision 960, Branch2 gets to 970 and Branch3 to 980.

The developer of Branch2 decides to merge. So the script merge_branch_to_trunk is called. The script knows that it needs to merge from revision 923->, since it spawned from a 923 version of the trunk. At the end, the Branch2 file of "Rev_toTrunk" is updated to contain 981, the current revision number. Rev_fromTrunk and Rev_toTrunk are also updated to be 981 on the trunk.

Now the tricky part, additional branch work. If the developer of Branch1 wants to merge in the latest version of the trunk, that developer would call "merge_trunk_to_branch". That script would merge versions 923-981 onto Branch1. At the end, the file "Rev_fromTrunk" would be updated to 981 (or maybe 982). Then, the developer would call "merge_branch_to_trunk". Because the value of "Rev_toTrunk" is still 923, it will merge all of its changes from 923 to 981. The new revision on the trunk would be 983 and the trunk would be update so that its "Rev_fromTrunk" and "Rev_toTrunk" would both contain 983. Branch1's "Rev_toTrunk" would also be updated with 983.

When Branch3 is ready to go, it follows the same pattern. Merging in from the trunk would bring in revisions 940-983, since that branch was created at revision 940.

RC development will be similar, except that it will also contain the files "Rev_fromRCTrunk" and "Rev_toRCTrunk".

Making Changes That Stay Only on the RC

In most instances, changes made to a release candidate also need to be propogated to the main trunk. So, developers get in the habit of committing their work on the release candidate and then using merge_rctrunk_to_trunk to merge their changes to the main trunk and commit them there. So, during the week or so while a release candidate is getting tested and patched, commits tend to occur in pairs; one one the release candidate and one on the trunk.

However, sometimes it is necessary to make a change that stays only on the release candidate and never makes it to the main trunk. How is that possible if everyone is tending to use the merge_rctrunk_to_trunk script? If one developer makes a change on the RC that is supposed to stay on the RC and another developer makes a change on the RC that s/he intends to merge to the trunk with merge_rc_trunk_to_trunk, how can we prevent the change that is supposed to stay in the RC from making it into the trunk?

The solution is to after committing a change to the RC, immediately do a merge_rctrunk_to_trunk but then revert that change from your checked out copy of the trunk. This will have the effect of fooling the scripts we use into thinking the RC change has already been merged to trunk and it will never again be considered for merge to trunk. Given that the operations of committing to the RC and then merging the RC to the trunk are not atonomous, it is possible that some other developer could beat you to the merge_rctrunk_to_trunk step. This could happen because of work they have done and want to merge to the trunk). So, when they run the merge_rctrunk_to_trunk script, they will pick up your changes as well and commit them to the trunk. The odds of this happening are very small but it is something to be aware of.

Useful Subversion commands

Making modifications

Unlike ClearCase, you make modifications by directly modifying the files. There is no notion of checkouts from a ClearCase point of view; SVN simply detects when files have changed. The basic model is referred to as ``copy-modify-merge" in the Subversion documentation. There is also a notion of file locking that will be necessary for binary files. This is done with svn lock filename.tif, although SVN recognizes binary files automatically.

Adding files

You do have to tell Subversion when to start tracking files, which is similar to ClearCase's Create VOB Element.

  • svn add newfile.C
  • svn mkdir newdir

Updating for changes from others

With ClearCase, as soon as someone makes a change to the mainline, those changes are immediately visible to all other developers. This is not the case with Subversion. With Subversion, you must manually force an update.

svn update

will update everything in the $PWD and below. So if this was issued from the src directory in your checkout, you would receive all of the files in src and below.

Checking for differences from the main repo

Your working copy may differ from the current version of the trunk either because you have made changes that have not been checked in, or because someone else has made changes that you have not picked up. You can get the status of these differences with:

svn status

This output can be somewhat verbose. It includes status for files that aren't even Subversion elements (for example, ".o files", although we will see how to eliminate those from showing up here).

You can get a status for only the Subversion elements with:

svn status -q

Note that if you need to do an svn add on a file, the "-q" option will cause you not to not see that there is a new file that is not a Subversion element.

Commits

You commit with:

svn commit -m "commit comment"

or

svn commit -F file_with_commit_comment

or

svn commit (which will then bring up an editor so you can generate a comment)

Note that this command does not update your local checked out tree to the HEAD of the repository, even if it was up to date prior to the commit. For example, if HEAD and BASE are at version 899, and you commit version 900, then: BASE is still at 899, so svn log will only report to version 899 by default. Only with a further svn update will BASE become 900 despite the fact that no files will change because of this update.

Note: it is probably a bad idea to give a brief checkin comment. We should all decide what the format for a checkin comment should be, since they will now be much more visible than they were previously.

Checking differences

You can check differences with:

svn diff filename

The resulting format is a bit clunky. Jeremy recommends "tkdiff", which is not available by default. However, you can download it from here. "tkdiff" is an Tcl/Tk interpreted script, so you can just download it and put it anywhere in your path like ~/bin without worrying about compilation or binary compatibility.

If you know a specific revision you want to difference with, use:

svn diff -r 151 filename

If you want to diff to the previous change of this file, use:

svn diff -r HEAD filename

(tkdiff also understands these revision arguments, so e.g. tkdiff -r HEAD filename also works.)

SVN understands several keywords for the revision specifier (BASE is the default for many commands like svn diff):

  • HEAD: latest (youngest) revision in the server
  • BASE: the revision your checkout was last updated against
  • COMMITTED: last revision a file or directory was changed
  • PREV: the last revision the file or directory was changed immediately before COMMITTED

Here is a good example taken directly from the KDE wiki:

1. You check out trunk at revision 200
2. You make a change to trunk/a_file and commit it: revision 201 is created
3. A day later, you update your working dir and find out it's now revision 208
4. You make another modification and commit: revision 209
5. One day later, you update again, this time to revision 212
6. The following day, before updating, the server has progressed to revision 218

Under those circumstances, here's what each one of those 4 mean:

* HEAD = 218
* BASE = 212
* COMMITTED (for a_file) = 209
* PREV (for a_file) = 201

Basic commands for controlling the versioning of files

  1. Making a new file
    • svn add filename
  2. Making a new directory
    • svn mkdir dirname
  3. Deleting a file
    • svn delete filename
  4. Deleting a directory
    • svn delete dirname
    • (Often leaves leftover files around)
    • rm -rf dirname
  5. Making a file have executable permissions
    • svn propset svn:executable 1 filename
  6. Symbolic links
    • (This example assumes you are in /include/visit.)
    • ln -s ../../components/Filters/avtMyNewFilter.h .
    • svn add avtMyNewFilter.h