Synchronizing Repositories
You can synchronize not just to Daya Bay repositories but between personal and/or 3rd party ones. One way is to use git to glue CVS and SVN repositories together. This topic describes ways to exploit this strategy.
Why do this?
- Track changes in 3rd party code with local modifications not acceptable to upstream
- Allow for offline commits
- Allow for "out of band" peer-to-peer code sharing
GIT and CVS and SVN
Previous attempts (below) failed to put both upstream CVS and downstream SVN into one GIT repository. The trick to doing this is maintaining strict separation between the pristine CVS branch and the one used to upload to SVN. Also important is to make sure that the SVN branch has the CVS branch in its history so subsequent updates to the CVS branch can be more easily merged into the SVN branch.
The example below uses the Relax project which provides Reflex dictionaries for common HEP and STL classes.
Initial CVS importation
bviren@lycastus:nuwa-git> git cvsimport -r upstream-cvs -o upstream-cvs-head -d :pserver:anonymous@relax.cvs.cern.ch:/cvs/relax -C relax relax bviren@lycastus:(master):relax> git branch -a * master upstream-cvs/HEAD upstream-cvs/LCG53_b upstream-cvs/RELAX_1_1_2_branch upstream-cvs/avendor upstream-cvs/upstream-cvs-head bviren@lycastus:(master):relax> rm -r *
Initialize SVN project
svn mkdir http://dayabay.ihep.ac.cn/svn/dybsvn/relax svn mkdir http://dayabay.ihep.ac.cn/svn/dybsvn/relax/trunk svn mkdir http://dayabay.ihep.ac.cn/svn/dybsvn/relax/tags svn mkdir http://dayabay.ihep.ac.cn/svn/dybsvn/relax/branches svn co http://dayabay.ihep.ac.cn/svn/dybsvn/relax cd relax/trunk/ echo "This area tracks Relax CVS at :pserver:anonymous@relax.cvs.cern.ch:/cvs/relax." > README svn add README svn commit -m "Prime this project area." README
Initialize GIT's SVN branch
bviren@lycastus:(master):relax> git svn init -s --username=bv --prefix=downstream-svn/ http://dayabay.ihep.ac.cn/svn/dybsvn/relax bviren@lycastus:(master):relax> git svn fetch bviren@lycastus:(master):relax> git checkout --track -b downstream-svn-trunk downstream-svn/trunk bviren@lycastus:(downstream-svn-trunk):relax> git branch -a * downstream-svn-trunk master downstream-svn/trunk upstream-cvs/HEAD upstream-cvs/LCG53_b upstream-cvs/RELAX_1_1_2_branch upstream-cvs/avendor upstream-cvs/upstream-cvs-head
Merge CVS head branch into SVN trunk branch
bviren@lycastus:(downstream-svn-trunk):relax> git merge --no-commit upstream-cvs/HEAD Automatic merge went well; stopped before committing as requested bviren@lycastus:(downstream-svn-trunk):relax> git commit -m "Merge of Relax's CVS HEAD." Created commit 30e7dc9: Merge of Relax's CVS HEAD.
Upload changes to SVN
bviren@lycastus:(downstream-svn-trunk):relax> git svn dcommit bviren@lycastus:(downstream-svn-trunk):relax> rm $(find . -name .cvsignore -print) bviren@lycastus:(downstream-svn-trunk):relax> git commit -a -m "Remove .cvsignore cruft."
Update to latest in CVS
t.b.d.
GIT and CVS
This blog posting is the main inspiration for this section. It is reproduced here for safety.
This uses two git repositories:
- myrepo.git.cvs-gateway
- a pristine git mirror of the state of the CVS repository
- myrepo.git.devel
- a tracking clone of the above that supports editting
Initalize git to cvs gateway repository
mkdir myrepo.git.cvs-gateway git cvsimport -p x -v -d $CVSROOT [optional_module_name]
Initialize the git repository for development
git clone -l myrepo.git.cvs-gateway myrepo.git.devel
Local mods can be done in this repository. By default you will have a newly created "master" branch which tracks the CVS HEAD. Normal git commands can be done in this repository. Like usual, work on branches that track remote branches and not on remote branches directly.
Update the git-to-cvs gateway repository
cd myrepo.git.cvs-gateway git cvsimport [-a] -p x -v -d $CVSROOT
The same command is used, but the -a is useful if you want to get very recent CVS commits.
Updating development git
cd myrepo.git.devel git pull
Commit back to CVS
This step is a little wonky. You will run a git cvs command from the a checked out copy of the CVS module but will point git at the devel repository.
cd cvs-checked-out-project/[optional_module_name] GIT_DIR=/path/to/myrepo.git.devel git cvsexportcommit <HASH>
The <HASH> is the SHA1 hash reported by git commit.
If you missed when committing, use gitk to find it.
Finally follow the printed CVS commit command which will be something like:
cvs commit -F .msg file1 file2 ...
GIT and SVN
Using GIT and SVN repositories together is done through the git svn tool.
It is used to access SVN as a remote GIT repository.
Creating a git svn repository
Create a git svn repository with "git svn init" like:
git svn init [--prefix=prefix/] --username [SVNUSER] [SVNURL] [TARGET]
The target is a directory that git svn will create to hold the
repository. You can run this in a pre-existing GIT project (and leave off the "TARGET" argument).
If you don't specify a prefix then the remote git svn branch will be named "git-svn".
Note, that depending on the SVNURL you will download different parts of the repository. This may include branches and/or tags sub directories. Modify the SVNURL to limit the download to a sub-part of the repository as desired.
Getting a working copy of SVN files
The init will not pull down any SVN files. To do that for the first time and for any subsequent updates of SVN use:
git svn fetch
If your URL specifies something other than the top level of the SVN repository you may see a "Path Not Found" warning. It is apparently innocuous.
After the fetch you will be sitting in a working branch "master". To get a working branch with a more descriptive name, am explicit checkout is recommended.
git checkout --track -b git-svn-working git-svn/remote-branch
Updating GIT
To update your git repository with any potential changes made in SVN do a fetch:
git svn fetch
If any come in they will live in a branch in your GIT repository coming off you recent local commits. To "replay" your locally committed modifications use
git svn rebase
This may throw conflicts which you will have to resolve by hand.
Updating SVN
To update SVN with locally committed modifications (not working directory mods) first update your git repository as above then issue:
git svn dcommit
Note the "d" in dcommit.
There is always a potential race condition between your last fetch/rebase and the dcommit. If someone happened to commit in the mean time you will get a somewhat cryptic message:
shell-prompt> git svn dcommit Merge conflict during commit: Your file or directory 'git/path/to/some/committed/file' is probably out-of-date: The version resource does not correspond to the resource within the transaction. Either the requested version resource is out of date (needs to be updated), or the requested version resource is newer than the transaction root (restart the commit). at /path/to/git-svn line 405
Just redo this tripple:
git svn fetch git svn rebase git svn dcommit
Checking where you are at
If a git svn fetch causes remote commits to be downloaded you can check where your currently at in your local development.
- Find out changes your working directory has which haven't been locally committed
git status
- Find out what branch you are on and what others are available
git branch -a
- Browse the commits and branche structure.
gitk --all
Advanced: Linking another, remote GIT repository to your GIT and GIT-SVN
You can use an intermediate GIT repository to syncronize an SVN repository to another GIT repository, be it a native GIT or GIT-CVS or GIT-SVN one. To do this, first set up an intermediate git-svn repository like above. If it is a new repository, it should be "primed" with at least a single file. Then, add the remote GIT repository:
git remote add <LABLE_FOR_REMOTE> <REMOTE_GIT_URL> git fetch <LABLE_FOR_REMOTE>
This links the labeled remote GIT repository and fetches its single file.
To upload remote history to SVN first make sure you are working in
git-svn-working (git branch -a will tell).
Then decide between one of two methods:
Mostly lossless history with git svn set-tree
Warning: use this command with caution as it forces a commit that makes the SVN area appear just like the state at the given GIT commit.
To upload a single remote commit to SVN use
git svn set-tree <COMMIT>
or
git svn set-tree <COMMIT1>..<COMMIT2>
To upload all commits between #1 and #2 inclusive. Note: use set-tree only in this special case of syncing with a remote branch. If you are using git only to track SVN you should use git svn dcommit to upload changes.
The commit ID can be taken from running gitk on the remote branch like:
gitk $(cat .git/refs/remotes/LABEL_FOR_REMOTE/REMOTE_BRANCH_NAME)
While this does replay remote commits into SVN, the commits will have current dates and will be attributed to the git-svn user and not the original author.
Your working branch will still be at the pre set-tree state. To catch up do
git reset --hard git-svn
Where "git-svn" is the branch created during git svn init. If you choose something besides the default, substitute it here.
Lossy history with git merge + git svn dcommit
If you want to control how the remote history is uploaded you can periodically merge the remote branch with the git-svn one:
git merge [-m "some merge commit message"] <LABEL_FOR_REMOTE/REMOTE_BRANCH_NAME>
Followed by a "dcommit"
git svn dcommit
This will not preserve any history of the remote branch in SVN. That
is, it will look like a single commit regardless of how many commits
were made to the remote branch. It will be attributed to the git-svn
user and will have a commit message like "Merge commit 'LABEL_FOR_REMOTE/REMOTE_BRANCH_NAME' into
git-svn-working" unless the -m option was given.
GIT and GIT
I keep a GIT/GIT-SVN repository as well as a "local-working-copy" GIT repository. Each has the other as a remote.
git remote add -f local-working-copy /path/to/working/copy/.git
To sync I do the following in the git/git-svn repository:
git cherry HEAD remotes/local-working-copy/dybsvn-trunk
to see how they differ:
gitk $(cat .git/refs/heads/dybsvn-trunk .git/refs/remotes/local-working-copy/dybsvn-trunk )& git merge --no-commit [top commit of local-working-copy]
(I think I can also use just remotes/local-working-copy/dybsvn-trunk)
This may have conflicts, fix them and
git commit
This collapses all the commit in the local-working-copy into one so some fine grained details of the history are lost. I would like to keep them, but I don't see how.
From the working directory I just do a git pull.