Synchronizing Repositories

From Daya Bay

Jump to: navigation, search

You can synchronize not just to Daya Bay repositories but between personal and/or 3rd party ones. One way is to use git to glue CVS and SVN repositories together. This topic describes ways to exploit this strategy.


Contents

Why do this?

  • Track changes in 3rd party code with local modifications not acceptable to upstream
  • Allow for offline commits
  • Allow for "out of band" peer-to-peer code sharing

GIT and CVS and SVN

Previous attempts (below) failed to put both upstream CVS and downstream SVN into one GIT repository. The trick to doing this is maintaining strict separation between the pristine CVS branch and the one used to upload to SVN. Also important is to make sure that the SVN branch has the CVS branch in its history so subsequent updates to the CVS branch can be more easily merged into the SVN branch.

The example below uses the Relax project which provides Reflex dictionaries for common HEP and STL classes.

Initial CVS importation

bviren@lycastus:nuwa-git> git cvsimport -r upstream-cvs -o upstream-cvs-head -d :pserver:anonymous@relax.cvs.cern.ch:/cvs/relax -C relax relax
bviren@lycastus:(master):relax> git branch -a
* master
  upstream-cvs/HEAD
  upstream-cvs/LCG53_b
  upstream-cvs/RELAX_1_1_2_branch
  upstream-cvs/avendor
  upstream-cvs/upstream-cvs-head
bviren@lycastus:(master):relax> rm -r *

Initialize SVN project

svn mkdir http://dayabay.ihep.ac.cn/svn/dybsvn/relax
svn mkdir http://dayabay.ihep.ac.cn/svn/dybsvn/relax/trunk
svn mkdir http://dayabay.ihep.ac.cn/svn/dybsvn/relax/tags
svn mkdir http://dayabay.ihep.ac.cn/svn/dybsvn/relax/branches
svn co http://dayabay.ihep.ac.cn/svn/dybsvn/relax
cd relax/trunk/
echo "This area tracks Relax CVS at :pserver:anonymous@relax.cvs.cern.ch:/cvs/relax." > README
svn add README 
svn commit -m "Prime this project area." README 

Initialize GIT's SVN branch

bviren@lycastus:(master):relax> git svn init -s --username=bv --prefix=downstream-svn/ http://dayabay.ihep.ac.cn/svn/dybsvn/relax
bviren@lycastus:(master):relax> git svn fetch
bviren@lycastus:(master):relax> git checkout --track -b downstream-svn-trunk downstream-svn/trunk
bviren@lycastus:(downstream-svn-trunk):relax> git branch -a
* downstream-svn-trunk
  master
  downstream-svn/trunk
  upstream-cvs/HEAD
  upstream-cvs/LCG53_b
  upstream-cvs/RELAX_1_1_2_branch
  upstream-cvs/avendor
  upstream-cvs/upstream-cvs-head

Merge CVS head branch into SVN trunk branch

bviren@lycastus:(downstream-svn-trunk):relax> git merge --no-commit upstream-cvs/HEAD
Automatic merge went well; stopped before committing as requested
bviren@lycastus:(downstream-svn-trunk):relax> git commit -m "Merge of Relax's CVS HEAD."  
Created commit 30e7dc9: Merge of Relax's CVS HEAD.

Upload changes to SVN

bviren@lycastus:(downstream-svn-trunk):relax> git svn dcommit
bviren@lycastus:(downstream-svn-trunk):relax> rm $(find . -name .cvsignore -print)
bviren@lycastus:(downstream-svn-trunk):relax> git commit -a -m "Remove .cvsignore cruft."

Update to latest in CVS

t.b.d.

GIT and CVS

This blog posting is the main inspiration for this section. It is reproduced here for safety.

This uses two git repositories:

myrepo.git.cvs-gateway 
a pristine git mirror of the state of the CVS repository
myrepo.git.devel 
a tracking clone of the above that supports editting

Initalize git to cvs gateway repository

mkdir myrepo.git.cvs-gateway
git cvsimport -p x -v -d $CVSROOT [optional_module_name]

Initialize the git repository for development

git clone -l myrepo.git.cvs-gateway myrepo.git.devel

Local mods can be done in this repository. By default you will have a newly created "master" branch which tracks the CVS HEAD. Normal git commands can be done in this repository. Like usual, work on branches that track remote branches and not on remote branches directly.

Update the git-to-cvs gateway repository

cd myrepo.git.cvs-gateway
git cvsimport [-a] -p x -v -d $CVSROOT

The same command is used, but the -a is useful if you want to get very recent CVS commits.

Updating development git

cd myrepo.git.devel
git pull

Commit back to CVS

This step is a little wonky. You will run a git cvs command from the a checked out copy of the CVS module but will point git at the devel repository.

cd cvs-checked-out-project/[optional_module_name]
GIT_DIR=/path/to/myrepo.git.devel git cvsexportcommit <HASH>

The <HASH> is the SHA1 hash reported by git commit. If you missed when committing, use gitk to find it. Finally follow the printed CVS commit command which will be something like:

cvs commit -F .msg file1 file2 ...

GIT and SVN

Using GIT and SVN repositories together is done through the git svn tool. It is used to access SVN as a remote GIT repository.

Creating a git svn repository

Create a git svn repository with "git svn init" like:

git svn init [--prefix=prefix/] --username [SVNUSER] [SVNURL] [TARGET]

The target is a directory that git svn will create to hold the repository. You can run this in a pre-existing GIT project (and leave off the "TARGET" argument). If you don't specify a prefix then the remote git svn branch will be named "git-svn".

Note, that depending on the SVNURL you will download different parts of the repository. This may include branches and/or tags sub directories. Modify the SVNURL to limit the download to a sub-part of the repository as desired.

Getting a working copy of SVN files

The init will not pull down any SVN files. To do that for the first time and for any subsequent updates of SVN use:

git svn fetch

If your URL specifies something other than the top level of the SVN repository you may see a "Path Not Found" warning. It is apparently innocuous.

After the fetch you will be sitting in a working branch "master". To get a working branch with a more descriptive name, am explicit checkout is recommended.

git checkout --track -b git-svn-working git-svn/remote-branch

Updating GIT

To update your git repository with any potential changes made in SVN do a fetch:

git svn fetch

If any come in they will live in a branch in your GIT repository coming off you recent local commits. To "replay" your locally committed modifications use

git svn rebase

This may throw conflicts which you will have to resolve by hand.

Updating SVN

To update SVN with locally committed modifications (not working directory mods) first update your git repository as above then issue:

git svn dcommit

Note the "d" in dcommit.

There is always a potential race condition between your last fetch/rebase and the dcommit. If someone happened to commit in the mean time you will get a somewhat cryptic message:

shell-prompt> git svn dcommit
Merge conflict during commit: Your file or directory
'git/path/to/some/committed/file' is probably out-of-date: The
version resource does not correspond to the resource within the
transaction.  Either the requested version resource is out of date
(needs to be updated), or the requested version resource is newer
than the transaction root (restart the commit). at /path/to/git-svn
line 405

Just redo this tripple:

git svn fetch
git svn rebase
git svn dcommit

Checking where you are at

If a git svn fetch causes remote commits to be downloaded you can check where your currently at in your local development.

  • Find out changes your working directory has which haven't been locally committed
git status
  • Find out what branch you are on and what others are available
git branch -a
  • Browse the commits and branche structure.
gitk --all

Advanced: Linking another, remote GIT repository to your GIT and GIT-SVN

You can use an intermediate GIT repository to syncronize an SVN repository to another GIT repository, be it a native GIT or GIT-CVS or GIT-SVN one. To do this, first set up an intermediate git-svn repository like above. If it is a new repository, it should be "primed" with at least a single file. Then, add the remote GIT repository:

git remote add <LABLE_FOR_REMOTE> <REMOTE_GIT_URL>
git fetch <LABLE_FOR_REMOTE>

This links the labeled remote GIT repository and fetches its single file.

To upload remote history to SVN first make sure you are working in git-svn-working (git branch -a will tell). Then decide between one of two methods:

Mostly lossless history with git svn set-tree

Warning: use this command with caution as it forces a commit that makes the SVN area appear just like the state at the given GIT commit.

To upload a single remote commit to SVN use

git svn set-tree <COMMIT>

or

git svn set-tree <COMMIT1>..<COMMIT2>

To upload all commits between #1 and #2 inclusive. Note: use set-tree only in this special case of syncing with a remote branch. If you are using git only to track SVN you should use git svn dcommit to upload changes.


The commit ID can be taken from running gitk on the remote branch like:

gitk $(cat .git/refs/remotes/LABEL_FOR_REMOTE/REMOTE_BRANCH_NAME)

While this does replay remote commits into SVN, the commits will have current dates and will be attributed to the git-svn user and not the original author.

Your working branch will still be at the pre set-tree state. To catch up do

git reset --hard git-svn

Where "git-svn" is the branch created during git svn init. If you choose something besides the default, substitute it here.

Lossy history with git merge + git svn dcommit

If you want to control how the remote history is uploaded you can periodically merge the remote branch with the git-svn one:

git merge [-m "some merge commit message"] <LABEL_FOR_REMOTE/REMOTE_BRANCH_NAME>

Followed by a "dcommit"

git svn dcommit

This will not preserve any history of the remote branch in SVN. That is, it will look like a single commit regardless of how many commits were made to the remote branch. It will be attributed to the git-svn user and will have a commit message like "Merge commit 'LABEL_FOR_REMOTE/REMOTE_BRANCH_NAME' into git-svn-working" unless the -m option was given.

GIT and GIT

I keep a GIT/GIT-SVN repository as well as a "local-working-copy" GIT repository. Each has the other as a remote.

git remote add -f local-working-copy /path/to/working/copy/.git

To sync I do the following in the git/git-svn repository:

git cherry HEAD remotes/local-working-copy/dybsvn-trunk

to see how they differ:

gitk $(cat .git/refs/heads/dybsvn-trunk .git/refs/remotes/local-working-copy/dybsvn-trunk )&
git merge --no-commit [top commit of local-working-copy]

(I think I can also use just remotes/local-working-copy/dybsvn-trunk)

This may have conflicts, fix them and

git commit

This collapses all the commit in the local-working-copy into one so some fine grained details of the history are lost. I would like to keep them, but I don't see how.

From the working directory I just do a git pull.

Personal tools