Patching VisIt

Submitting Patches

I have no experience actually creating a patch file for submission. Can somone who has experience elaborate on how to do this?

Applying Patches

This section outlines a sequence of steps that has worked at least once in applying a patch that was submitted by a VisIt user/developer in the UK, Huw Jones (thanks Huw).

Here is an outline of the process

  a) Identify version of VisIt source code tree patch originates from
  b) Identify SVN revision number associated with a)
  c) Checkout the trunk using revision number in b)
  d) Apply the patch to the checked out trunk
  e) Be aware of code generation issues
  f) Update the checked out trunk to the head
  g) Resolve any conflicts
  h) Eye-ball all the changes
  i) Commit the changes to the trunk

Identify version of VisIt source code

First, identify the version of the VisIt release that the patch was created with. In my case, the release of VisIt that Huw submitted a patch for was 1.8.1. Now, we were just releasing 1.9 at the time, so this was the most recent release available to Huw to patch. If the patch had come from an earlier release, say 1.7, then I think the right strategy is to ask the submitter to at least bring the patch up to date with the most currently available public version of the code. So, if the user had submitted a patch from 1.7 release, I would have asked the user to submit an updated patch file that was up to date with the 1.8.1 release.

Identify SVN Revision number

Once you know the version of VisIt the patch originates from, you then need to determine the SVN revision number. This can be accomplished in several ways. Probably the easiest thing to do is to run an installed version of VisIt that is the version identified above. The splash screen displays the SVN revision number. Alternatively, you can use either 'svn info' or 'svn log' command on the tags dir like so...

  % svn info svn+ssh://miller86@portal-auth.nersc.gov/project/projectdirs/visit/svn/visit/tags/1.8.1
  Path: 1.8.1
  URL: svn+ssh://miller86@portal-auth.nersc.gov/project/projectdirs/visit/svn/visit/tags/1.8.1
  Repository Root: svn+ssh://miller86@portal-auth.nersc.gov/project/projectdirs/visit/svn/visit
  Repository UUID: 18c085ea-50e0-402c-830e-de6fd14e8384
  Revision: 4135
  Node Kind: directory
  Last Changed Author: hrchilds
  Last Changed Rev: 3880
  Last Changed Date: 2008-04-16 09:03:13 -0700 (Wed, 16 Apr 2008)

Now, in my case, the splash screen and 'svn info' numbers didn't agree. The splash screen indicated rev #3454 but the 'svn info' indicated rev #3880. This is because we sometimes forget to tag a release after it has been made and because when we finally did decide to tag it, we tagged the current head at the time and not the specific rev # of the release. Of these two numbers, I picked the smaller (earlier in time) one. I used 3454

Checkout trunk using revision #

The only issue here is deciding how much of the trunk you want to check out. Typically, for a patch comming from a user, it won't include changes to items in 'data' or 'test' and that is good because those directories are large and take time to checkout. In my case, the patch I had consisted only of changes to trunk/src. So, here is the svn checkout command I used...

   % svn co -r 3454 svn+ssh://miller86@portal-auth.nersc.gov/project/projectdirs/visit/svn/visit/trunk/src visit_181_3454_trunk_src

This tells SVN to check out only the 'src' part of the trunk for rev #3454 and put it in a directory named visit_181_3454_trunk_src.

Apply the patch to the checked out code

Note that a 'patch file' is a specifically formatted kind of file. It is ASCII, human readable, containing a bunch of diffs (like the 'diff' command might output) between the original files and the 'patched' files. A single patch file can contain pathes for many files in the source code tree. You use the special patch command the apply the patches to the files in source code tree. But, this can be a bit tricky depending on several factors. In my case, the repo had undergone a rather extensive directory and file renaming (e.g. 'components' dir was renamed 'avt') between the version of the code the user submitted the patch for and the version I had checked out. This is why getting the precise tag/rev # for a patch is so important. In any case, because the patch file lists the names of the files it is patching, I applied some sed commands to replace, for example, "/components/" with "/avt/" before I applied the patch to my checkout rev of the trunk. Finally, because the patch file originates on some other computer, the names of files in the patch file can contain leading stuff you don't want. You need to make sure you use the '-p' option to patch command to strip off this leading stuff.

   For example, supposing the file name in the patch file was
       /u/howard/src/blurfl/blurfl.c
   setting -p0 gives the entire file name unmodified, -p1 gives
       u/howard/src/blurfl/blurfl.c
   without the leading slash, -p4 gives
       blurfl/blurfl.c
   and not specifying -p at all just gives you blurfl.c.

Next, you want to use the '--dry-run' option to the patch command to see what it will do before you actually run it for real. I did this several times before I was sure it seemed to be working correctly. As patch runs, it will tell you which files it is operating on. If it encounters problems, it will put the 'rejects' in a file 'next to' the one it was trying to patch with a name ending in '.rej'. Here is the patch command I wound up using...

   % patch -d visit_181_3454_trunk_src -p2 < visit181_gcc43.patch.rename4

which tells patch to start in directory visit_181_3454_trunk_src, strip the 2 leading slashes from all filenames (-p2) and apply the patches from the file visit181_gcc43.patch.rename4 which was the original patch file I recieved but edited with sed to account for things like the directory name change from 'components' to 'avt'

When it completed, it touched 67 files and rejected 3 files. The command...

   % find . -name '*.rej'

will find all the rejects. Then, by examining the original patch file, visit181_gcc43.patch.rename4, the reject and the file it was trying to patch, I was able to resolve problems with the rejects. In my case, the rejects were a result of the fact that the changes the patch was trying to make were apparently already made in the version of the code I had checked out.

Be aware of code generation issues

It probably won't be all that clear to a typical user but a number of source files in VisIt's source code tree are really generated from other (typically xml) files. If a patch changes the generated files, those changes are likely to get lost the next time the file gets re-generated. For this reason, it is very important to pay attention to which files are getting patched and recognize a situation when a generated file is being patched. Take for example an attributes object in common/state. If some attributes object .C or .h file was patched, the right solution is to edit the .xml file from which the .C or .h are generated. This is a tricky issue but it is something to pay close attention to when patching.

Update your patched trunk to the head

Ok, you've applied the patch successfully now. But, you can't commit it because the version of the code you have checked out is an old revision. Now, you need to update to it to the current head. You do this simply by svn updating it. But, before you do, you might want to get a list of the files you've changed as a result of applying the patch. The command...

   % svn status -v | grep -v '^[ ?]' > files_changed_by_patch.txt

Should list all files you've changed by applying the patch. Now, update your checkout

   % svn update

And a new status command

   % svn status -v | grep -v '^[ ?]' > files_changed_by_update.txt

should list all files that get changed by the update. Find any files in conflict...

    % grep ^C files_changed_by_update.txt

and resolve any conflicts. Hopefully, there won't be any conflicts. But, if there are, it is because the patch you applied changed files in places near where those same files have been changed by all development on the trunk since the release the patch was created for.

Eye-ball all the changes (and add comments as necessary)

Ok, you're about ready to 'svn commit' your changes. But, you haven't really looked at them in detail. Also, unless the person sending you the patch bothered to add comments (hopefully they did NOT because comments in function pre-ambles tend to be a source of conflicts), you will probably want to comment on the changes you've made and maybe credit the person who submitted the patch in your comments. Here is what I do to eye-ball changes...

   % grep -v '^[ ?]' files_changed_by_update.txt | sort | tr -s ' ' | cut -d' ' -f5 | xargs -i -t xxdiff {} ~/visit/trunk/src/'{}'

This is a long command that greps for all lines in files_changed_by_update.txt which DO NOT have either a space or question mark as the first character (gre -v '^[ ?]'), sorts them (sort), squeezes out all spaces (tr -s ' '), takes the 5th field of each line (cut -d' ' -f5) and pipes each resulting filename to xargs which will in turn run xxdiff comparing the file in my patched trunk to another checkout of trunk I always keep in ~/visit/trunk/src (the {} refers to the name of the file passed to xargs from the pipe preceding it). This command will iterate through all files that are different and display the differences in xxdiff. In my case, 95% of the files contained a one line change to include some C-language related header. In each file, xxdiff points me to where the differences are. Then, in a separate editor, I can go into each of the files to the line numbers displayed by xxdiff and, as necessary, add comments regarding the changes.

Commit the changes

Ok, you're finally ready to commit the patch. Now, depending on the situation, you may want configure, compile and build everything and even run tests. In my example, all the changes I made were #include directives. So, I was pretty confident it would compile fine. Here is the command I used...

   % svn commit -m 'applying patches for gcc-4.3 compile from Huw Jones'