Difference between revisions of "Github"

m (Official Plan)
(Replacing page with '= Developing with GitHub = This content has been moved to the [https://visit-sphinx-github-user-manual.readthedocs.io/en/develop/dev_manual/GitHub.html VisIt Developer Manual...')
 
(63 intermediate revisions by 7 users not shown)
Line 1: Line 1:
= Moving VisIt Source Code Control to Git + Github =
+
= Developing with GitHub =  
  
== Intro ==
+
This content has been moved to the [https://visit-sphinx-github-user-manual.readthedocs.io/en/develop/dev_manual/GitHub.html VisIt Developer Manual]
  
We feel a transition to git and github can improve how we develop VisIt. That said, we also recognize this shift will incur transition costs and there will be bumps in the road.
+
[[Category: Software Engineering]]
 
 
This document includes discussion and suggestions for a path to move VisIt's development to github.
 
 
 
== Motivation ==
 
The VisIt team has used its own SVN model since our source code repo was moved to NERSC (~2007 -- ~2008) to bolster open source development. While this workflow has worked well for the team for many years, support for SVN is diminishing and development workflows based on distributed version control systems, primarily git or mercurial, are the first choice for new projects and many existing teams are migrating SVN repos to these systems. Several VisIt developers are already contributing to or managing other projects on github, so it is a natural choice to host VisIt's source:
 
 
 
Why git + github?
 
* git has substantial and growing mindshare
 
* git's robust branching and distributed version control features bring substantial benefits over svn
 
* github provides access to a large ecosystem of software engineering tools (continuous integration (CI) systems for linux, osx, and windows, code coverage, etc)
 
* github's tools for Windows are maturing and making it easy to use git + github on Windows.
 
* github's pull request support enables code reviews
 
 
 
 
 
== Overview of Changes ==
 
 
 
'''These slides outline repo structure issues and proposed changes:'''
 
 
 
[http://portal.nersc.gov/project/visit/cyrush/2017_11_30_visit_git_notes.pdf 2017_11_30_visit_git_notes.pdf]
 
 
 
=== Git Workflow ===
 
 
 
Branching is much more robust in git than in SVN. We will be able to use branching to improve our workflow and many of our current scripts to manage branching will not be necessary.
 
 
 
We will use the commonly used "git-flow" workflow https://www.atlassian.com/git/tutorials/comparing-workflows#gitflow-workflow, it's well documented and widely understood.  Doing so will reduce barriers vs a custom process.
 
 
 
Our current process actually maps well to git-flow.
 
 
 
Here is how we currently use branches to manage releases:
 
 
 
[[Image:visit_git_trans_svn_branches.png|center]]
 
 
 
To move to git-flow, our current '''trunk''' becomes the '''develop''' branch, and we merge our '''RC''' branches into '''master''' at release points:
 
 
 
 
 
[[Image:visit_git_trans_gitflow_branches.png|center]]  
 
 
 
 
 
With this the '''master''' branch, and tags on it contain the release history.  We could still choose to retain the RC branches, but they will no longer be necessary to preserve the release history. 
 
 
 
 
 
In addition to this strategy for releasing, following git-flow all work will be done via "topic" branches. "topic" branches is a fancy name for small branches used to bundle the changes for a specific feature or bugfix.
 
All development will be done on branches off of develop or RCs and all merges to develop or RC branches will be done via github pull requests.
 
 
 
(Restated: no commits will be done directly to develop, RC, or master branches.)
 
 
 
Using pull requests on github to merge code is a ubiquitous practice. It creates a public record of any discussions related to changes and also allows us to leverage CI checks as part of the process.
 
 
 
 
 
Blessed developers will be part of the VisIt github project team and will not have to use forks of the source code repo to do development.
 
Blessed developers *will* be able to merge their own pull requests, but having at least one additional reviewer will be preferred.
 
PRs must pass a core set of CI tests (these will be integrated into the PR process -- facilitated by github and ci services)
 
Upon approval, all pull requests will use squash merges (which are facilitated by github)
 
Remote (github) branches for development work will be deleted after they are merged via a pull request.
 
 
 
 
 
Yes, this is a big change to our development workflow. We will work to make sure this process is outlined clearly and smooth for all VisIt developers.
 
We will provide a cheat sheet with the subset of git commands necessary to support this workflow -- we don't want folks to have to be git experts to contribute to VisIt.
 
 
 
------
 
 
 
===Issue Tracking ===
 
 
 
We will eventually migrate our issue tracking to Github's issue tracker. Having a github account should be the only account necessary to contribute to VisIt.
 
Cyrus' Note: I think redmine actually works well, but a simplified integrated workflow tips the scales.
 
 
 
 
 
 
 
=== Challenges ===
 
 
 
*  github limits repo sizes & our current repo is large (>70GB). (This is mainly due to our practice of revision controlling all of our assets  -- example data files, releases etc)
 
** We will split our source code and data supporting testing into separate repos
 
*** We plan to use git lfs for binary data in all repos except for the main source repo.
 
** We can host release binaries using github's release feature. The limit is 1GB *per file*. There aren't any limits on the number of files per release.
 
** We will stop hosting tarballs of TPL source in git, and use another solution.
 
** We will still revision control the Windows TPL builds
 
 
 
 
* CI setups (travis, etc) will be stressed by building and testing VisIt
 
** We need to develop a solution for TPLs sets for CI systems, build_visit will likely timeout on free tiers of CI systems
 
** We will curate a reduced set of regression tests for quick turn around on CI systems
 
** We will explore moving our blessed regression baselines to a container-based system that anyone could wield to test changes
 
** We will explore using LLNL AWS to support longer running CI.
 
 
 
=== Official Plan ===
 
*[x] Establish VisIt github org (https://github.com/visit-dav/)
 
*[x] Resolve Major feedback, settle workflow and repo changes ''(Dec 18 2017)''
 
*[x] Develop scripts that automate the port: construct develop and master, tags, etc ''(Dec 2017 / Jan 2018)''
 
*[  ] For trunk, change to new top level repo structure in SVN ''(Before 4/3/2018)''
 
*[ ] Teach Pull Request (PR) merge process ''(Before 4/3/2018)''
 
*[  ] Migrate source repo to github, data and test will stay SVN ''(4/3/2018)''
 
**[ ] Change Nightly testing to use source from github, data, and test from SVN ''(April 2018)''
 
**[ ] Setup Basic CI (Linux + OSX + Windows) ''(April 2018)''
 
**[ ]  Port SVN hooks to scripts for use as CI checks ''(April 2018)''
 
**[ ] Test minimal compile in CI''(April 2018)''
 
 
 
 
 
*[ ] Migrate non-source content
 
**[ ] Gain git-lfs experience, develop scripts to filter data and test repos to create git-lfs repos
 
**[ ] Setup contracts to purchase Github Data Packs
 
**[ ] Identify and copy assets we wont rev control with git (third_party tarballs, release tarballs) to visit project webserver on NERSC for hosting 
 
 
 
 
 
* [ ] Write up processes on visitusers, replace SVN Stuff
 
* [ ] Telecons with PR demos
 
 
 
 
 
*Interested People:
 
**Cyrus, Kevin, Mark
 
**Hank: Testing, Help with Hosting Costs
 

Latest revision as of 14:36, 13 May 2021

Developing with GitHub

This content has been moved to the VisIt Developer Manual