- 1 Developing with GitHub
- 2 Moving VisIt Source Code Control to Git + Github
Developing with GitHub
We have moved VisIt's main source code repository to github.
What do things look like now?
The good news is that less things changed than we originally expected.
We migrated these top level directories:
Our develop git branch is now our default branch and is most similar to our old svn trunk The git repo contains RC branches since 2.0 Releases are tagged off of the RC branches, just like before.
Moving forward we will use a simple branch-based extension to our current release workflow.
The main difference is that all proposed changes will be done on a branch and vetted before they are merged into develop or any RC branch.
Restated: No direct commits to develop or any RC branch.
We will use continuous integration and code reviews as extra checks to avoid introducing errors.
To introduce this process, lets walk through changing a text file and creating a pull request to merge these changes.
The VisIt repo is hosted under the visit-dav GitHub organization.
This walk through assumes are are a member of the visit-dav organization and have been added to the VisIt Developers team.
Create a branch with proposed changes
First, clone the Repo:
git clone https://github.com/visit-dav/visit.git
When you clone the default branch is the develop branch.
The first step is to create a branch off of develop, here is an example:
cd visit git checkout -b task/cyrush/2019_01_update_github_map
Edit src/resources/github_usernames.txt and add a line with your username, full name, and organization - here is an example:
cyrush, Cyrus Harrison, LLNL
After making changes, you stage specific files to commit with git add
git add src/resources/github_usernames.txt
Finally, commit using:
git commit -m "Added my name to the github usernames file"
Once you have a set of commits you want to review, push your branch to github.
Create a Pull Request for your branch
Open up https://github.com/visit-dav/visit in your web browser.
After pushing, you should see a yellow box that asks if you want to create a Pull Request for your branch
Follow the prompts to create the pull request. Add cyrush as a reviewer and wait for approval. In the future, we will also have continuous integration checks that will need to pass before merging.
Merge your Pull Request
You can merge your pull request after continuous integration checks pass and you obtain approvals.
GitHub provides a few options to merge. The two methods we use are: Squash and merge and Create a merge commit. For most changes Squash and merge is preferred. Squashing turns all of the changes from the commits on your branch into a single commit in the target branch. If you have many or very complex changes, the other preferred method is to use a "Create a merge commit" This will preserve your branch commits in the target and create a commit that records the merge.
For this walk through use the Squash and merge option. When you Squash and merge, you have a final change to cleanup any commit messages. After you merge, GitHub provides a button to delete the source branch. Use it! This will help us avoid stale branches.
Setting up Git LFS
Git LFS (Large File Storage) is a mechanism to help revison control large files efficiently with git. Instead of storing large files in the repo, LFS provides an extension that stores small text files with meta data in the repo and the actual files on another server. These meta data files are called `pointer` files. We use LFS for binary data including our test data tarballs, regression test baseline images, and other ascents like shared binary builds of shared libraries.
Git LFS is not part of the standard git client. See https://git-lfs.github.com/ for how to obtain Git LFS.
When installing, use the following option:
git lfs install --force --skip-smudge
The "skip smudge" command sets up LFS in way that skips automatically pulling our large files on clone. We do this to conserve bandwith.
To obtain these files you will need to do some extra incantations followed by an explicit:
git lfs pull
Bash Shell Customization
If you use bash, you can add the following to your .bash_rc file to setup your shell to disable the current git branch.
PS1='[\u@\h \W$(__git_ps1 " (%s)")]\$ '
Moving VisIt Source Code Control to Git + Github
We feel a transition to git and github can improve how we develop VisIt. That said, we also recognize this shift will incur transition costs and there will be bumps in the road.
This document includes discussion and suggestions for a path to move VisIt's development to github.
The VisIt team has used its own SVN model since our source code repo was moved to NERSC (~2007 -- ~2008) to bolster open source development. While this workflow has worked well for the team for many years, support for SVN is diminishing and development workflows based on distributed version control systems, primarily git or mercurial, are the first choice for new projects and many existing teams are migrating SVN repos to these systems. Several VisIt developers are already contributing to or managing other projects on github, so it is a natural choice to host VisIt's source:
Why git + github?
- git has substantial and growing mindshare
- git's robust branching and distributed version control features bring substantial benefits over svn
- github provides access to a large ecosystem of software engineering tools (continuous integration (CI) systems for linux, osx, and windows, code coverage, etc)
- github's tools for Windows are maturing and making it easy to use git + github on Windows.
- github's pull request support enables code reviews
Overview of Changes
These slides outline repo structure issues and proposed changes:
Branching is much more robust in git than in SVN. We will be able to use branching to improve our workflow and many of our current scripts to manage branching will not be necessary.
We will use the commonly used "git-flow" workflow https://www.atlassian.com/git/tutorials/comparing-workflows#gitflow-workflow, it's well documented and widely understood. Doing so will reduce barriers vs a custom process.
Our current process actually maps well to git-flow.
Here is how we currently use branches to manage releases:
To move to git-flow, our current trunk becomes the develop branch, and we merge our RC branches into master at release points:
With this the master branch, and tags on it contain the release history. We could still choose to retain the RC branches, but they will no longer be necessary to preserve the release history.
In addition to this strategy for releasing, following git-flow all work will be done via "topic" branches. "topic" branches is a fancy name for small branches used to bundle the changes for a specific feature or bugfix. All development will be done on branches off of develop or RCs and all merges to develop or RC branches will be done via github pull requests.
(Restated: no commits will be done directly to develop, RC, or master branches.)
Using pull requests on github to merge code is a ubiquitous practice. It creates a public record of any discussions related to changes and also allows us to leverage CI checks as part of the process.
Blessed developers will be part of the VisIt github project team and will not have to use forks of the source code repo to do development. Blessed developers *will* be able to merge their own pull requests, but having at least one additional reviewer will be preferred. PRs must pass a core set of CI tests (these will be integrated into the PR process -- facilitated by github and ci services) Upon approval, all pull requests will use squash merges (which are facilitated by github) Remote (github) branches for development work will be deleted after they are merged via a pull request.
Yes, this is a big change to our development workflow. We will work to make sure this process is outlined clearly and smooth for all VisIt developers. We will provide a cheat sheet with the subset of git commands necessary to support this workflow -- we don't want folks to have to be git experts to contribute to VisIt.
We will eventually migrate our issue tracking to Github's issue tracker. Having a github account should be the only account necessary to contribute to VisIt. Cyrus' Note: I think redmine actually works well, but a simplified integrated workflow tips the scales.
- github limits repo sizes & our current repo is large (>70GB). (This is mainly due to our practice of revision controlling all of our assets -- example data files, releases etc)
- We will split our source code and data supporting testing into separate repos
- We plan to use git lfs for binary data in all repos except for the main source repo.
- We can host release binaries using github's release feature. The limit is 1GB *per file*. There aren't any limits on the number of files per release.
- We will stop hosting tarballs of TPL source in git, and use another solution.
- We will still revision control the Windows TPL builds
- We will split our source code and data supporting testing into separate repos
- CI setups (travis, etc) will be stressed by building and testing VisIt
- We need to develop a solution for TPLs sets for CI systems, build_visit will likely timeout on free tiers of CI systems
- We will curate a reduced set of regression tests for quick turn around on CI systems
- We will explore moving our blessed regression baselines to a container-based system that anyone could wield to test changes
- We will explore using LLNL AWS to support longer running CI.
- [x] Establish VisIt github org (https://github.com/visit-dav/)
- [x] Resolve Major feedback, settle workflow and repo changes (Dec 18 2017)
- [x] Develop scripts that automate the port: construct develop and master, tags, etc (Dec 2017 / Jan 2018)
- [ ] For trunk, change to new top level repo structure in SVN (Before 4/3/2018)
- [ ] Teach Pull Request (PR) merge process (Before 4/3/2018)
- [ ] Migrate source repo to github, data and test will stay SVN (4/3/2018)
- [ ] Change Nightly testing to use source from github, data, and test from SVN (April 2018)
- [ ] Setup Basic CI (Linux + OSX + Windows) (April 2018)
- [ ] Port SVN hooks to scripts for use as CI checks (April 2018)
- [ ] Test minimal compile in CI(April 2018)
- [ ] Migrate non-source content
- [ ] Gain git-lfs experience, develop scripts to filter data and test repos to create git-lfs repos
- [ ] Setup contracts to purchase Github Data Packs
- [ ] Identify and copy assets we wont rev control with git (third_party tarballs, release tarballs) to visit project webserver on NERSC for hosting
- [ ] Write up processes on visitusers, replace SVN Stuff
- [ ] Telecons with PR demos
- Interested People:
- Cyrus, Kevin, Mark
- Hank: Testing, Help with Hosting Costs
- [ ] restructure top level layout in both to 2.13 RC and trunk github_migration_dir_restructure
- [ ] migrate source checkers to ci (check for tabs etc) (longer term)
- [ ] audit scripts that use svn (mainly in svn_bin, which includes build_visit -- but also things like masonry auto build )
- [ ] audit sphinx docs
- [X] audit visitusers.org txt (I've created a list: Pages_to_modify_with_move_to_github)
- [ ] have some check the new repo vs the old (Mark can help)
- [ ] pioneer how migrate our issues from redmine to github
- [ ] describe general github process
- [ ] document our branching process
- [ ] document our PR process and test
- [ ] sit in same room as a team and try the process out
- [ ] basic ci (in circle-ci since you can run longer than travis) (aim for one or two tests)
- [ ] in git, we will keep a simple text file to link data and test to svn revs to the current git source (we may need a script to checkout)
- [ ] finish git graft, and history rewrites
- we will have a cutoff date for using svn for source commits, after this date any svn commits will need to be moved by their authors to github.