Difference between revisions of "Github"

(Create a branch with proposed changes)
(Replacing page with '= Developing with GitHub = This content has been moved to the [https://visit-sphinx-github-user-manual.readthedocs.io/en/develop/dev_manual/GitHub.html VisIt Developer Manual...')
(2 intermediate revisions by one other user not shown)
Line 1: Line 1:
= Developing with GitHub =  
= Developing with GitHub =  
We have moved VisIt's main source code repository to github.
This content has been moved to the [https://visit-sphinx-github-user-manual.readthedocs.io/en/develop/dev_manual/GitHub.html VisIt Developer Manual]
What do things look like now?
The good news is that less things changed than we originally expected.
We migrated these top level directories:
Our ''develop'' git branch is now our default branch and is most similar to our old svn trunk
The git repo contains RC branches since 2.0
Releases are tagged off of the RC branches, just like before.
Moving forward we will use a simple branch-based extension to our current release workflow.
The main difference is that all proposed changes will be done on a branch and vetted before they are merged into ''develop'' or any RC branch.
Restated: No direct commits to ''develop'' or any RC branch.
We will use continuous integration and code reviews as extra checks to avoid introducing errors.
To introduce this process, lets walk through changing a text file and creating a pull request to merge these changes.
The VisIt repo is hosted under the ''visit-dav'' GitHub organization.
This walk through assumes you are a member of the ''visit-dav'' organization and have been added to the VisIt Developers team.
== Setting up Git LFS ==
Git LFS (Large File Storage) is a mechanism to help revison control large files efficiently with git.  Instead of storing large files in the repo, LFS provides an extension that stores small text files with meta data in the repo and the actual files on another server.  These meta data files are called `pointer` files. We use LFS for binary data including our test data tarballs, regression test baseline images,  and other ascents like shared binary builds of shared libraries.
Git LFS is not part of the standard git client.  See https://git-lfs.github.com/ for how to obtain Git LFS.
When installing, use the following option:
git lfs install --force --skip-smudge
The "skip smudge" command sets up LFS in way that skips automatically pulling our large files on clone.
We do this to conserve bandwith.
To obtain these files you will need to do some extra incantations followed by an explicit:
git lfs pull
== Accessing GitHub ==
The following link points to a page for creating a personal access token to use for the password when accessing GitHub through the command line. Use the following scopes for the token:
The following link describes how to add your ssh key to your GitHub account.
== Create a branch with proposed changes ==
You can access GitHub either through https or ssh. If you use the https you will be prompted for your password whenever you push to GitHub. There are ways you can have your password cached for a period of time to reduce the frequency of entering your password. However, if you have two-factor authentication set up you will need to create a personal access token to use in place of the password. If you use the ssh protocol you can set things up so that you never have to enter a password by adding your ssh key to your GitHub account.
First, clone the Repo:
git clone https://github.com/visit-dav/visit.git
git clone ssh://git@github.com/visit-dav/visit.git
Setup our hooks:
cd visit
When you clone the default branch is the ''develop'' branch.
The first step is to create a branch off of develop. This is where your proposed changes will be done, then once pushed will be vetted, will be merged into develop (or RC) branch. Here is an example:
git checkout -b task/cyrush/2019_01_update_github_map
COMMENT - explain the naming. Does it matter? How does this work with the staging vetting??
Edit ''src/resources/github_usernames.txt'' and add a line with your username, full name, and organization - here is an example:
cyrush, Cyrus Harrison, LLNL
After making changes, you stage specific files to commit with ''git add''
git add src/resources/github_usernames.txt
COMMENT - I am not sure this step is needed as a file is not being added just edited.
Finally, commit using:
git commit -m "Added my name to the github usernames file"
Once you have a set of commits you want to review, push your branch to github.
git push
Git may complain about not knowing where to push.
There is no tracking information for the current branch.
Please specify which branch you want to merge with.
See git-pull(1) for details.
    git pull <remote> <branch>
If you wish to set tracking information for this branch you can do so with:
    git branch --set-upstream-to=origin/<branch> task/cyrush/2019_01_update_github_map
Here is an example:
git branch --set-upstream-to=origin/task/cyrush/2019_01_update_github_map task/cyrush/2019_01_update_github_map
git push
If you would like your local branches to automatically reference an upstream branch, configure your default:
git config --global push.default current
== Create a Pull Request for your branch  ==
Open up https://github.com/visit-dav/visit in your web browser.
After pushing, you should see a yellow box that asks if you want to create a Pull Request for your branch
Follow the prompts to create the pull request. Add ''cyrush'' as a reviewer and wait for approval.
In the future, we will also have continuous integration checks that will need to pass before merging.
== Merge your Pull Request ==
You can merge your pull request after continuous integration checks pass and you obtain approvals.
GitHub provides a few options to merge. The two methods we use are: ''Squash and merge'' and ''Create a merge commit''. For most changes '''Squash and merge''' is preferred. Squashing turns all of the changes from the commits on your branch into a single commit in the target branch. If you have many or very complex changes, the other preferred method is to use a "Create a merge commit" This will preserve your branch commits in the target and create a commit that records the merge.
For this walk through use the ''Squash and merge'' option. When you ''Squash and merge'', you have a final change to cleanup any commit messages. After you merge, GitHub provides a button to delete the source branch. Use it! This will help us avoid stale branches.
After the remote branch is merged and deleted, you'll want to do two things on your local side:
(1) Update your list of remote branches
git remote prune origin
git remote update
(2) Delete your local branch. First switch to another branch (e.g. develop)
git checkout develop
git pull
git branch -D <name of your branch that was merged>
COMMENT - not really necessary to delete the branch if one wants to continue to use it for development????
== Bash Shell Customization ==
If you use bash, you can add the following to your ''.bash_rc'' file to setup your shell to display the current git branch.
PS1='[\u@\h \W$(__git_ps1 " (%s)")]\$ '
= Moving VisIt Source Code Control to Git + Github =
== Intro ==
We feel a transition to git and github can improve how we develop VisIt. That said, we also recognize this shift will incur transition costs and there will be bumps in the road.
This document includes discussion and suggestions for a path to move VisIt's development to github.
== Motivation ==
The VisIt team has used its own SVN model since our source code repo was moved to NERSC (~2007 -- ~2008) to bolster open source development. While this workflow has worked well for the team for many years, support for SVN is diminishing and development workflows based on distributed version control systems, primarily git or mercurial, are the first choice for new projects and many existing teams are migrating SVN repos to these systems. Several VisIt developers are already contributing to or managing other projects on github, so it is a natural choice to host VisIt's source:
Why git + github?
* git has substantial and growing mindshare
* git's robust branching and distributed version control features bring substantial benefits over svn
* github provides access to a large ecosystem of software engineering tools (continuous integration (CI) systems for linux, osx, and windows, code coverage, etc)
* github's tools for Windows are maturing and making it easy to use git + github on Windows.
* github's pull request support enables code reviews
== Overview of Changes ==
'''These slides outline repo structure issues and proposed changes:'''
[http://portal.nersc.gov/project/visit/cyrush/2017_11_30_visit_git_notes.pdf 2017_11_30_visit_git_notes.pdf]
=== Git Workflow ===
Branching is much more robust in git than in SVN. We will be able to use branching to improve our workflow and many of our current scripts to manage branching will not be necessary.
We will use the commonly used "git-flow" workflow https://www.atlassian.com/git/tutorials/comparing-workflows#gitflow-workflow, it's well documented and widely understood.  Doing so will reduce barriers vs a custom process.
Our current process actually maps well to git-flow.
Here is how we currently use branches to manage releases:
To move to git-flow, our current '''trunk''' becomes the '''develop''' branch, and we merge our '''RC''' branches into '''master''' at release points:
With this the '''master''' branch, and tags on it contain the release history.  We could still choose to retain the RC branches, but they will no longer be necessary to preserve the release history. 
In addition to this strategy for releasing, following git-flow all work will be done via "topic" branches. "topic" branches is a fancy name for small branches used to bundle the changes for a specific feature or bugfix.
All development will be done on branches off of develop or RCs and all merges to develop or RC branches will be done via github pull requests.
(Restated: no commits will be done directly to develop, RC, or master branches.)
Using pull requests on github to merge code is a ubiquitous practice. It creates a public record of any discussions related to changes and also allows us to leverage CI checks as part of the process.
Blessed developers will be part of the VisIt github project team and will not have to use forks of the source code repo to do development.
Blessed developers *will* be able to merge their own pull requests, but having at least one additional reviewer will be preferred.
PRs must pass a core set of CI tests (these will be integrated into the PR process -- facilitated by github and ci services)
Upon approval, all pull requests will use squash merges (which are facilitated by github)
Remote (github) branches for development work will be deleted after they are merged via a pull request.
Yes, this is a big change to our development workflow. We will work to make sure this process is outlined clearly and smooth for all VisIt developers.
We will provide a cheat sheet with the subset of git commands necessary to support this workflow -- we don't want folks to have to be git experts to contribute to VisIt.
===Issue Tracking ===
We will eventually migrate our issue tracking to Github's issue tracker. Having a github account should be the only account necessary to contribute to VisIt.
Cyrus' Note: I think redmine actually works well, but a simplified integrated workflow tips the scales.
=== Challenges ===
*  github limits repo sizes & our current repo is large (>70GB). (This is mainly due to our practice of revision controlling all of our assets  -- example data files, releases etc)
** We will split our source code and data supporting testing into separate repos
*** We plan to use git lfs for binary data in all repos except for the main source repo.
** We can host release binaries using github's release feature. The limit is 1GB *per file*. There aren't any limits on the number of files per release.
** We will stop hosting tarballs of TPL source in git, and use another solution.
** We will still revision control the Windows TPL builds
* CI setups (travis, etc) will be stressed by building and testing VisIt
** We need to develop a solution for TPLs sets for CI systems, build_visit will likely timeout on free tiers of CI systems
** We will curate a reduced set of regression tests for quick turn around on CI systems
** We will explore moving our blessed regression baselines to a container-based system that anyone could wield to test changes
** We will explore using LLNL AWS to support longer running CI.
=== Official Plan ===
*[x] Establish VisIt github org (https://github.com/visit-dav/)
*[x] Resolve Major feedback, settle workflow and repo changes ''(Dec 18 2017)''
*[x] Develop scripts that automate the port: construct develop and master, tags, etc ''(Dec 2017 / Jan 2018)''
*[  ] For trunk, change to new top level repo structure in SVN ''(Before 4/3/2018)''
*[ ] Teach Pull Request (PR) merge process ''(Before 4/3/2018)''
*[  ] Migrate source repo to github, data and test will stay SVN ''(4/3/2018)''
**[ ] Change Nightly testing to use source from github, data, and test from SVN ''(April 2018)''
**[ ] Setup Basic CI (Linux + OSX + Windows) ''(April 2018)''
**[ ]  Port SVN hooks to scripts for use as CI checks ''(April 2018)''
**[ ] Test minimal compile in CI''(April 2018)''
*[ ] Migrate non-source content
**[ ] Gain git-lfs experience, develop scripts to filter data and test repos to create git-lfs repos
**[ ] Setup contracts to purchase Github Data Packs
**[ ] Identify and copy assets we wont rev control with git (third_party tarballs, release tarballs) to visit project webserver on NERSC for hosting 
* [ ] Write up processes on visitusers, replace SVN Stuff
* [ ] Telecons with PR demos
*Interested People:
**Cyrus, Kevin, Mark
**Hank: Testing, Help with Hosting Costs
== Checklist ==
* [ ] restructure top level layout in both to 2.13 RC and trunk [[github_migration_dir_restructure]]
* [ ] migrate source checkers to ci (check for tabs etc) (longer term)
* [ ] audit scripts that use svn (mainly in svn_bin, which includes build_visit -- but also things like masonry auto build )
* [ ] audit sphinx docs
* [X] audit visitusers.org txt  (I've created a list: [[Pages_to_modify_with_move_to_github]])
* [ ] have some check the new repo vs the old (Mark can help)
* [ ] pioneer how migrate our issues from redmine to github
* [ ] describe general github process
* [ ] document our branching process
* [ ] document our PR process and test
* [ ] sit in same room as a team and try the process out
* [ ] basic ci (in circle-ci since you can run longer than travis) (aim for one or two tests)
* [ ] in git, we will keep a simple text file to link data and test to svn revs to the current git source (we may need a script to checkout)
* [ ] finish git graft, and history rewrites
== notes ==
* we will have a cutoff date for using svn for source commits, after this date any svn commits will need to be moved by their authors to github.
== Things to Todo ==
* Add info + links on how to clone with SSH
* Add info about setting up hooks
* Basic CI style checks
== Things to Fix ==
* any path with `third_party` in the name, was skipped due to regex matching
* we need to find a way to prevent direct pushes to develop and rcs. We have github's seatbelts on, however these don't block admins (or owners) from pushing
* lfs pattern matching does example as far as we  (doc/*png does not match pngs in lower sub dirs)
* revisit email mappings
[[Category: Software Engineering]]
[[Category: Software Engineering]]

Latest revision as of 14:36, 13 May 2021

Developing with GitHub

This content has been moved to the VisIt Developer Manual