Software Upgrades

From Daya Bay
Jump to navigation Jump to search

Periodically we must upgrade to newer versions of the externals and/or the CERN projects. Note: this needs merging with Offline Software Upgrades.

How upgrades are done

Upgrading involves re-synchronizing our SVN with repositories of CERN projects, merging in our modifications and making sure everything builds.

How to install upgrades

Normally one does not install upgrade "releases" unless to help out with its testing. Installing an "upgrade" is much like installing "trunk". First get the latest "dybinst" from the upgrade branch: 

svn export http://dayabay.ihep.ac.cn/svn/dybsvn/installation/branches/upgrade/dybinst/dybinst

April 2009 upgrade

This was motivated to support GCC 4.3. This version of GCC is being used in current Debian stable and Fedora 9 and is or will become standard for others. Most of the software we use has been fixed for GCC 4.3 so it is better to upgrade than to re-do the wheel.

The branches were made 2009/04/01. The upgrade will be merged with the trunk after the 1.2.0 release is made.

How to install upgrade branch

svn export http://dayabay.ihep.ac.cn/svn/dybsvn/installation/branches/upgrade/dybinst/dybinst
./dybinst upgrade all

How to test upgrade branch

cd NuWa-upgrade 
source setup.sh
cd dybgaudi/DybRelease/cmt
source setup.sh
cd $ROOTIOTESTROOT
nosetests

You should see: 

........
----------------------------------------------------------------------
Ran 8 tests in 282.535s

OK

Still to do

  • Mac PyROOT problem Trac ticket #222 fixed
  • Geant4 data files - this causes some RootIOTests to fail actually, this is due to a FPE / uninitialized variable bug in Geant4.9.1.p03 that is fixed in Geant4.9.2.p01.
  • RootIOTest tests passing. The hold up was due to G4FPE_DEBUG being on. This triggers all sorts of FPE to be asserted where they used to be quietly ignored. See Trac ticket #225.
  • LCG_Builder for libmore
  • Move to new LCG_Builder for AIDA

Intermediate merge with trunk development

It's been decided to wait to merge upgrade into trunk until after the June collaboration meeting. To do more extensive testing an intermediate merge is done by branching trunk on May 21st and merging upgrade into it.

Merge by SVN

Here is how this was done for the installation project. It follows the Red Bean book's chapter 4.

Branch trunk: 

svn cp http://dayabay.ihep.ac.cn/svn/dybsvn/installation/trunk http://dayabay.ihep.ac.cn/svn/dybsvn/installation/branches/trunk-upgrade-merge

Checkout upgrade and branched trunk branches 

mkdir -p /some/path/toinstallation 
cd /some/path/to/installation 
svn co http://dayabay.ihep.ac.cn/svn/dybsvn/installation/branches/trunk-upgrade-merge

Because SVN is stupid you have to tell it explicitly what range of commits on the upgrade branch to merge. It is also too stupid to know when your branch was started. Find this info thusly 

cd upgrade/
svn log --verbose --stop-on-copy

The last commit listed is the first of the branch and is r5826.

Merge: 

cd trunk-upgrade-merge/
svn update
At revision 6359.
svn merge -r r5826:HEAD http://dayabay.ihep.ac.cn/svn/dybsvn/installation/branches/upgrade
(deal with conflicts - there was one in nosebit.sh)
svn status
svn commit -m "Merged upgrade branch from r5826 into trunk-upgrade-merge"

Again, because SVN is so stupid, it is "critical" that the release number is included in the comment. Actually the Read Bean book says I should have included r6359 as well.

Merge by GIT

Just for fun, lets do it in git too. 

cd /path/to/installation/upgrade/
(git's working directories are not bound to branch, so the "upgrade/" name is just to make dybinst happy)

Get new branch based on the one made through SVN, merge and commit 

git-svn fetch && git-svn rebase
git-checkout -t -b dybsvn-trunk-upgrade-merge dybsvn/trunk
git-merge -m "Merge with dybsvn-upgrade" dybsvn-upgrade
(fix conflicts - there were 2, same nosebit.sh and complained about deleting geant.sh which has seen updates)
git commit -a -m "Merge with upgrade branch."


Much easier, faster and safer. This is then one "dcommit" away from going into svn.

Intermediate trunk->upgrade merge by GIT

git-checkout dayabay-svn-trunk
git-svn rebase
git-checkout -t -b git-upgrade-trunk-merge dayabay-svn-upgrade
git-merge dayabay-svn-trunk
(fix conflicts)
git commit -m "Merge trunk."

Two conflicts:

  • DybPythonAlg.py - trunk wins
  • GtGunGenTool.cc - trunk wins but need to replace the HepLorentzVector with
   HepMC::ThreeVector p(momentum*dir);
   HepMC::FourVector fourmom(p.x(),p.y(),p.z(),energy);

Overview of Changes

  • Move to Geant4.9.2.p01 which requires a major upgrade to CLHEP 2.0.4.2
  • Move to Gaudi 20.4 and associated LHCb versions
  • Move to ROOT 5.22.00a
  • Add sqlite
  • Improved platform auto-detection support
  • Support separate debug and optimized builds
  • Standardized platform labels
  • Simplify building of externals by using LCG_Builders
  • Pull external sources from a single web cache

Controlling platform with CMTCONFIG

A clean up of the tags/macros used in lcgcmt and a move to explicitly stating CMTCONFIG has been done. When you install the upgrade you may see something like: 

Checking your CMTCONFIG="i686-debian-gcc40-dbg"...

ERROR: CMTCONFIG is not compatible!
...

Followed by list of instructions. You can reproduce this error quickly by doing: 

source external/CMT/v1r20p20080222/mgr/setup.sh
unset CMTCONFIG     # <---- important, CMT's setup.sh gives initial, useless value
./installation/upgrade/dybinst/scripts/check-cmtconfig.sh upgrade

If you have any "Unknown" parts to the CMTCONFIG you will have to add host-* and target- tags to: 

NuWa-upgrade/lcgcmt/LCG_Settings/cmt/requirements

Use the output of cmt show tags that is printed in the error message as a starting point. For example, the coop has: 

bviren@coop:offline-upgrade-test> cmt show tags
CMTv1 (from CMTVERSION)
CMTr20 (from CMTVERSION)
CMTp20080222 (from CMTVERSION)
Linux (from uname) package CMT implies [Unix]
Linux-i686 (from CMTCONFIG) package CMT implies [Linux]
i686 (from package CMT)
debian (from package CMT)
gcc404 (from package CMT)
Unix (from package CMT) excludes [WIN32 Win32]

In this case the missing line was: 

tag debian              host-debian

Rechecking with check-cmtconfig.sh shows no more "Unknown" labels but still fails to be able to use i686-debian-gcc40. To fix this, this tag needs to be added in two places: 

### Daya Bay CMTCONFIG tags
...
tag i686-debian-gcc40-dbg target-i686 target-debian target-gcc40 target-linux target-dbg
tag i686-debian-gcc40-opt target-i686 target-debian target-gcc40 target-linux target-opt
### end Daya Bay CMTCONFIG tags

and 

macro LCG_basesystem                            "uncharacterized-platform" \
     target-i686&target-debian&target-gcc40     "i686-debian-gcc40" \
     ...

You should then be greeted with success: 

bviren@coop:offline-upgrade-test> ./installation/upgrade/dybinst/scripts/check-cmtconfig.sh upgrade
~/work/dayabay/offline/NuWa-upgrade/lcgcmt/LCG_Settings/cmt ~/work/dayabay/offline
Found CMTCONFIG="i686-debian-gcc43-dbg" from lcgcmt
~/work/dayabay/offline
Checking your CMTCONFIG="i686-debian-gcc43-dbg"...
...ok.
export CMTCONFIG=i686-debian-gcc43-dbg

If it still fails, make sure that you have done 

unset CMTCONFIG

Specific Platforms

The platforms that are initially supported by this upgrade. Additional support is usually added as above. This is being updated as new info comes in.

Currently we are stopped by porting the code to the new Geant4 version.

Ubuntu Intrepid, i686-debian-gcc43
Compiles.
Ubuntu Hardy, i686-debian-gcc42
Compiles, RootIOTests pass
Debian Lenny, i686-debian-gcc43
Compiles, RootIOTests pass
Debian Etch (coop), i686-debian-gcc40
Compiles, RootIOTests pass
Mac OS X 10.5 , i686-darwin-gcc40
Compiles, RootIOTests pass
SL 4.4 (PDSF), x86_64-sl44-gcc34
Compiles, RootIOTests pass
SLC 4.5 (IHEP), i686-sl45-gcc34
Compiles, RootIOTessts pass
Fedora 9, (Christine's laptop), i686-slc9-gcc43
Compiles.
SLC 5.2(UW cluster), x86_64-slc52-gcc346
Compiles.


LCG/AA will be standardizing on ATLAS styled tags which are like: 

cpu-os-compiler[-opt/dbg]

This upgrade follows this convention.

Externals

Most externals are now build with LCGCMT/LCG_Builders.

  • Results in a mostly similar layout of external/.
  • All building is done in external/build/LCG/
  • Source code tar balls are taken from external/tarFiles.
  • They are downloaded from a single web cache.

The base URL of the web cache is set by the LCG_tarurl macro in 

LCG_Builders/LCG_BuildPolicy/cmt/requirements

Default is a BNL URL. If you want to change it you can must break up the build into two parts so you can download lcgcmt, change the URL and continue. To do this, follow these steps: 

./dybinst upgrade cmt checkout
emacs NuWa-upgrade/lcgcmt/LCG_Builders/LCG_BuildPolicy/cmt/requirements
# change LCG_tarurl macro
./dybinst upgrade external projects

Alternatively you can manually pre-fill the area external/tarFiles/ area. For example: 

cd external/tarFiles/
wget -m --no-parent http://dayabay.bnl.gov/software/offline/tarFiles/

This may give you more files than you need.

Retrieved from "https://wiki.bnl.gov/dayabay/index.php?title=Software_Upgrades&oldid=1536"