Auxillary data

Auxillary data storage, etc. is in dybaux

Muon data files

Organize muon data files and resolve #709.

• make a versioned tar file of a given release of the data files, put on our web cache

Web cache is http://dayabay.bnl.gov/software/offline/tarFiles/

In /dayabay/u/djaffed/dayabay/offline/NuWa-trunk/people/djaffe/MuonDataFiles/data

$ tar  -czvf muondata-0.0.tgz Muon/*
Muon/GroundLevelmuonGenerator
Muon/mountain_DYB
Muon/mountain_Far
Muon/mountain_LA

Put tarball into cache:

 scp muondata-0.0.tgz dayabay@dayabay.bnl.gov:~/public_html/software/offline/tarFiles/.

• add the version to LCG_Configuration/cmt/requirements

 emacs -nw  NuWa-trunk/lcgcmt/LCG_Configuration/cmt/requirements 

macro cx_oracle_config_version       "4.3.1"
macro Cygwin_config_version          "1.5"
macro data_config_version            "0.0"
macro david_config_version           "1_36a"
macro dawn_config_version            "3_88a"
macro dcache_client_config_version   "1.8.0p1"

• make LCG_Builder packages to download and unpack the data under external/

Use AIDA as an example since it is only de-tars.

  mkdir NuWa-trunk/lcgcmt/LCG_Builders/data
  cd  NuWa-trunk/lcgcmt/LCG_Builders/data
  mkdir cmt
  mkdir scripts
  cp  ../aida/scripts/* scripts/.
  echo v1 > cmt/version.cmt
  cp  ../aida/cmt/requirements  cmt/.
 emacs -nw cmt/requirements 
  mv aida_make.sh data_make.sh
  mv aida_install.sh data_install.sh
 mv aida_config.sh data_config.sh
  emacs -nw data_config.sh

• make LCG_Interface package to set any env. var. and "use" it in our package.

Use AIDA and Geant4 as examples

 cd ../LCG_Interfaces/
  mkdir data
  cd data
  mkdir cmt
 cd cmt
  echo v1 > version.cmt
  cp  ../../AIDA/cmt/requirements  .
  emacs -nw requirements 

• add the external package to list of what dybinst should install

Add as 'optional' in

/Users/dayabay/offline-trunk/installation/trunk/dybinst/scripts/dybinst-common.sh

• Test.

 ./dybinst trunk optional
[deletia]

Starting dybinst commands: optional

Stage: "optional"... 

Found CMTCONFIG="i386-darwin-gcc42-dbg" from your environment.
Checking your CMTCONFIG="i386-darwin-gcc42-dbg"...
...ok.
not directed to build data

Brett points out that building can be tested by itself. in cmt/ of builder

 cd NuWa-trunk/lcgcmt/LCG_Builders/data/cmt/
dayabay@bnlku24:cmt> cmt config
Creating setup scripts.
Creating cleanup scripts.
dayabay@bnlku24:cmt> source setup.sh
dayabay@bnlku24:cmt> cmt pkg_get
Execute action pkg_get => python /Users/dayabay/offline-trunk/NuWa-trunk/lcgcmt/LCG_Builders/LCG_BuildPolicy/scripts/pkg_get.py
/Users/dayabay/offline-trunk/NuWa-trunk/lcgcmt/LCG_Builders/LCG_BuildPolicy/scripts/pkg_get.py : INFO: Downloading http://dayabay.bnl.gov/software/offline/tarFiles/muondata-0.0.tgz to /Users/dayabay/offline-trunk/NuWa-trunk/../external/tarFiles/muondata-0.0.tgz
/Users/dayabay/offline-trunk/NuWa-trunk/lcgcmt/LCG_Builders/LCG_BuildPolicy/scripts/pkg_get.py : INFO: File /Users/dayabay/offline-trunk/NuWa-trunk/../external/tarFiles/muondata-0.0.tgz copied to /Users/dayabay/offline-trunk/NuWa-trunk/../external/build/LCG
dayabay@bnlku24:cmt> cmt pkg_config
Execute action pkg_config => sh -x /Users/dayabay/offline-trunk/NuWa-trunk/lcgcmt/LCG_Builders/data/scripts/data_config.sh
+ mkdir -p /Users/dayabay/offline-trunk/NuWa-trunk/../external/data//Muon
+ tar -C /Users/dayabay/offline-trunk/NuWa-trunk/../external/data//Muon -xzf /Users/dayabay/offline-trunk/NuWa-trunk/../external/build/LCG/muondata-0.0.tgz

Hmm...not what I intended. Should be installed in external/data/0.0/Muon, not external/data/Muon/Muon.

1. Fixed typo in requirements
2. Now have to re-tar files since the directory Muon/ came along for the ride.
3. scp new tgz file to cache
4. new session cuz I screwed up setup by using the deadly cleanup
5. now ok

Decided to place data in external instead of optional because configuring optional requires intervention at each site. Only problem is that dybinst does not appear to be idempotent for this external/data directory installation.

Had to back out addition of data to external, builds were failing....

Add EH1, EH2, EH3 data files

Want to add to external/data/0.0/Daq

1. tar file must contain files with directory prefix Daq/
2. add name of tar file to LCG_sourcefiles in lcgcmt/LCG_Builders/data/cmt/requirements
3. add tar file to repository

$tar -czvf EH123-data-0.0.tgz Daq/*
Daq/daq.Neutrino.0021375.Physics.EH3-Merged.SFO-3._0011.data
Daq/daq.Neutrino.0021520.Physics.EH1-Merged.SFO-1._0317.data
Daq/daq.Neutrino.0022150.Physics.EH2-Merged.SFO-2._0001.data
$ scp EH123-data-0.0.tgz dayabay@dayabay.bnl.gov:~/public_html/software/offline/tarFiles/.

1. test
2. doesn't work. get snotty error

IOError: [Errno socket error] [Errno 60] Operation timed out
CMT> Error: execution_failed : pkg_get

1. unset http_proxy, HTTP_PROXY, etc. now ok.