Upgrading Projects

This describes the external projects are tracked and how upgrades are done..

Overview

Daya Bay relies on four projects from CERN:

• lcgcmt,
• relax,
• gaudi and
• lhcb.

These are available via their respective CVS repositories. CERN is transitioning to SVN starting with Gaudi v21. Gaudi is the test bed for this transition so expect the others to follow later. For now we stick with v20 and CVS.

Prepare Dyb's SVN

To avoid disturbing work going on in the trunk, we branch each project to an "upgrade" branch and make a "release" called "NuWa-upgrade". This generic term is not given any version as it is just a vehicle to make the upgrade.

Branch dancing goes like:

svn mkdir http://dayabay.ihep.ac.cn/svn/dybsvn/NuWa/branches/NuWa-upgrade
svn co http://dayabay.ihep.ac.cn/svn/dybsvn/NuWa/branches/NuWa-upgrade
svn propedit svn:externals NuWa-upgrade

In the editor, write:

lcgcmt http://dayabay.ihep.ac.cn/svn/dybsvn/lcgcmt/branches/upgrade
gaudi http://dayabay.ihep.ac.cn/svn/dybsvn/gaudi/branches/upgrade
lhcb http://dayabay.ihep.ac.cn/svn/dybsvn/lhcb/branches/upgrade
relax http://dayabay.ihep.ac.cn/svn/dybsvn/relax/branches/upgrade
dybgaudi http://dayabay.ihep.ac.cn/svn/dybsvn/dybgaudi/branches/upgrade
ldm http://dayabay.ihep.ac.cn/svn/dybsvn/ldm/branches/upgrade
tutorial http://dayabay.ihep.ac.cn/svn/dybsvn/tutorial/branches/upgrade

Confirm this took hold:

cd NuWa-trunk
svn update
...
cd ..
rm -rf NuWa-trunk

Setup Git

The general idea is for each project that is tracked:

1. use cvs to download tagged releases from CERN's CVS
2. clean up the result (remove CVS/ dir, etc)
3. commit to a branch in git that only holds pristine upstream releases
4. use git-svn to link Dyb's SVN to git
5. sync the local git branch associated with git-svn to the pristine upstream branch

Git provides git-cvsimport which, in principle, would allow fine-grained tracking of upstream development. However, in practice, it fails to ingest some of the repositories (failure is actually in the program cvsps).

If one doesn't already have a git repository tracking a project in our SVN, initiating one goes like:

mkdir nuwa-area/<project>
cd nuwa-area/<project>
git-svn init -s --prefix=downstream/http://dayabay.ihep.ac.cn/svn/dybsvn/<project>
git-svn fetch

This leaves us with remote branches under remotes/upstream/ and remotes/downstream.

To create the pristine upstream branch:

1. Create a branch from the very first git commit, this should result in an empty working directory.
2. Do a cvs checkout of the release that we first used. Checking past commits via gitk should make this obvious.
3. Remove any extraneous files such as CVS/ directories, .cvsignore files and any packages not wanted.
4. Do git add *
5. Do git commit -m "Upstream release vXrY, tag PROJECT_vXrY"

To add the next release to the upstream branch:

1. Check out the branch
2. Do rm -rf * to clear the working directory
3. Proceed from the cvs checkout above.

Merging Upstream code

The upgrade process consists of:

1. Checkout to a local branch from "downstream" which is at the point of the last upgrade merge (use gitk to determine this)
2. Merge upstream's commit which is at this same point of development
   1. Fix any conflicts and commit
3. Merge upstream's commit which is at the target point for the upgrade
   1. Fix any conflicts and commit
4. Checkout to a local branch, the head of the downstream "upgrade" branch
5. Merge in the branch used in the steps above
   1. Fix any conflicts and commit
6. Upload the results to Dyb's SVN

This looks like:

gitk --all &
git-checkout -b sync-for-upgrade <commit, branch or tag of last upgrade/merge>
git-merge <upstream tag corresponding to last upgrade/merge>
# fix conflicts
git-add -u
git-commit -m "Merge upstream <upstream past tag>"
git-merge <upstream tag corresponding to upgrade target>
git-add -u
git-commit -m "Merge upstream <upstream target tag>"
git-checkout -b downstream-upgrade remotes/downstream/upgrade
git-merge sync-for-upgrade
# fix conflicts
git-add -u
git-commit -m "Upgrade merge for <target tag>"

If a previous sync-for-upgrade branch from a previous upgrade exists it can be reused.

Final merge of upgrade branches back into trunk

The merge can be done with pure SVN but it is easier to do it in git and then upload the results. However, because SVN is limited in what it can represent, one has to follow a somewhat non-git philosophy in the merge. Namely, git-cherry-pick must be used to copy the commits from the git upgrade branch to the git trunk branch. In the case of cherry-picking a merge commit, the "-m 1" flag must be added.

This bit of scripting helps to cherry pick all commits in one go:

# Cherry picks commits starting with the on AFTER <commit1>
# and continues to and includes <commit2>.
for c in $(git-rev-list --reverse ${commit1}...${commit2})
do 
    echo "git cherry-pick $c"
    git cherry-pick $c
    if [ "$?" != "0" ] ; then
        echo "Failed to cherry pick $c"
        exit 1
    fi
    echo "...done with $c"
    echo
done

For example, lhcb was merged for the April 09 upgrade like:

git-checkout dayabay-svn-trunk
git-cherry-pick -m 1 77dac13
git-multi-cherry 77dac13 5591fd
git-svn dcommit

The commit IDs are easily found using "gitk".

The April 2009 upgrade

This was driven by the desire to support GCC 4.3. Some of the critical version changes are:

Externals:

• ROOT 5.18.00f -> 5.22.00a
• GCCXML 0.7 -> 0.9
• OpenScientist 16.2 -> 16.8

Projects:

• LCGCMT v53a -> v56
• Gaudi v19r7 -> v20r4
• LHCb v23r3 -> v26r2

For this upgrade, fresh git repositories were made to document how it is done.

LCGCMT

# Commit 1cf4... is "Merge in LCG_54", which was actually 54a
git-checkout -b sync-to-lcg-cvs 1cf4c357e37692b7019832306d0763e86a7d8568
# commit any thing if needed
git-commit -a -m "Start merging LCG_54a"

git-merge LCGCMT_54a
# fix any conflicts.
git-add -u
git commit -m "Merge upstream tag LCGCMT_54a"

git-checkout -b dybsvn-upgrade remotes/downstream/upgrade
git-merge sync-to-lcg-cvs
# fix conflicts

git-checkout sync-to-lcg-cvs
git-merge LCGCMT_55
# carry conflicts forward
git-add -u
git-commit -m "...."
git-merge LCGCMT_56

Notes:

• This did two syncs to dybsvn-upgrade

Relax

Like LCGCMT. Typically very little development so upgrades are trivial.

Gaudi

This script automates parts of this upgrade:

dybinst/scripts/upgrade-gaudi.sh

Initializing

For unknown reason, git-cvsimport does not work with Gaudi. It crashes cvsps. So, the strategy here is different:

1. Use usual "cvs checkout" to bring layer releases onto a "pristine" branch
2. Only edits on this branch are to clean out CVS cruft or remove unwanted packages
3. Treat this as an "upstream" remote branch

Do usual downstream init

mkdir -p nuwa-git/gaudi
cd nuwa-git/gaudi
git-svn init -s --prefix=downstream/ http://dayabay.ihep.ac.cn/svn/dybsvn/gaudi

Branch from the very initial commit, should leave you with an empty working dir

git-checkout -b upstream-gaudi-releases cbf3f6824853f323cd7cd334448628ecc830a269

Check out the starting release

cvs -d :pserver:anonymous@isscvs.cern.ch:/local/reps/Gaudi checkout -r GAUDI_v19r4 .

Clean up anything not wanted

rm -r GAUDI GaudiGridSvc

Also remove any CVS cruft

rm -r $(find . -name CVS -type d -print)
rm -r $(find . -name .cvsignore -print)

Sync up with git

git-add *
git-rm $(git-status|grep 'deleted:'|awk '{print $3}')
git-commit -m "Gaudi v19r7, upstream tag GAUDI_v19r7"</nowiki>

Notes:

• Note the CVS module is "."
• Must do checkout because can't do "export ."
• For subsequent sync'ing of upstream-gaudi-releases do rm -rf * before the cvs checkout

Upgrade Pristine Upstream Branch

The "upstream-gaudi-releases" pristine branch here is just "gaudi-releases". It was up to v19r9 in my git repo. To have some way points, bring in all v20rY with Y=0,1,2,3,4 releases.

Check out the branch and confirm it is clean

git-checkout gaudi-releases
git status
rm -rf *
cvs -d :pserver:anonymous@isscvs.cern.ch:/local/reps/Gaudi checkout -r GAUDI_v20r0 .

Remove the cruft and check what is new

git status
rm ...

Any new files will have to be added

git add *

And, commit.

git commit -m "Gaudi v20r0, upstream tag GAUDI_v20r0"

Repeat for r1-r4.

Note: right now there is no GAUDI_v20r4 tag in place so I use the more recent which is GAUDI_v20r4-pre. I've asked for this tag to be made.

Merge

Checkout upgrade branch, merge and commit:

git-checkout -t -b dayabay-svn-upgrade remotes/dayabay-svn/upgrade
git-merge gaudi-releases
# fix conflicts
git add *
git-commit -m "Merge in Gaudi v20r4-pre, upstream tag GAUDI_v20r4-pre."

LHCb

The script faciliates parts of the upgrade of LHCb project

dybinst/scripts/lhcb.sh

For this upgrade the these "Sys" version variables are used

• lhcb_sys v26r2
• pano_sys v16r4
• gauss_sys v36r2

Update pristine branch

git checkout lhcb-releases
git-status  # should be clean
rm -rf *
cvsconnect :pserver:anonymous@isscvs.cern.ch:/local/reps/lhcb ../../installation/upgrade/dybinst/scripts/lhcb.sh

Remove cvs cruft

rm -r $(find . -name CVS -type d)
rm $(find . -name .cvsignore)

Commit

git add -f *
git commit -m "LHCbSys v26r2, PanoramixSys v16r4, GaussSys v36r2"

Merge

git checkout -t -b dayabay-svn-upgrade remotes/dayabay-svn/upgrade
git-merge lhcb-releases
# fix conflicts (should be only a few)
# remove (git rm) unneeded files, these will be marked as CONFLICT (delete/modify):
# their removal has been added to lhcb.sh

Commit the merge and upload

git add -f *
git-commit -m "Merge in LHCbSys v26r2, PanoramixSys v16r4, GaussSys v36r2 from lhcb-releases."
git-svn dcommit

git add *

Past Upgrades

2008/02/20

LHCb:

• LHCbSys v23r3
• PanoramixSys v15r9
• GaussSys v30r5