Center for Functional Nanomaterials (CFN) Linux Cluster

From CFN Theory and Computation Group
Jump to: navigation, search

The page is about the old CFN cluster. For information about the new Institutional Cluster that CFN is using, please go to

The CFN Cluster is part of the user facilities provided by the Center for Functional Nanomaterials at Brookhaven National Laboratory for external users. You must have a valid Life or Guest Numbers before following the necessary steps in acquiring access to the CFN cluster. Guest numbers are assigned to external users upon proposal approvals. To submit a proposal, follow instructions here [1]. Once you have an approved proposal, you can start using the CFN cluster by reading this web site.

Accounts and access

  1. Access to our facility is granted to external users through a peer-reviewed proposal system. To become a user, please visit
  2. Users must have a valid Life or Guest Number before they can request access to a computer cluster. To obtain a guest number, users must complete the guest registration process at
  3. Complete Cyber Security training that is available online (
  4. Apply for a CFN cluster account at
    • A CFN cluster account (
    • A RSA SecurID Token (required for offsite users)
    • A BNL SSH Gateway Account (required for offsite users) or VPN Access (for BNL employees)
  5. Applicants will be notified by the BNL Account Office of their log in information (username and initial password). If there are any problems, contact the BNL Help Desk at (631) 344-5522 or by email.
  6. To access the CFN cluster:
    • Inside the BNL firewall (i.e. intranet), ssh to the head node, nano (or
    • Outside the BNL firewall, ssh to the gateway node, Follow the instructions received with your crypto card account to login. Once logged into the gateway, proceed to ssh to the head node, nano (or nano2) with your username and password.
    • Using BNL VPN also gets you inside the BNL firewall.
  • Note: In order to use the SSH Gateway you must have an SSH client installed first. You can find help here.

About the Cluster

The CFN cluster currently has two head nodes for interactive login (nano and nano2) plus ~212 computing nodes. The total number of cores is about 2144. The TORQUE Resource and Queue Manager, based on the original PBS resource manager developed by NASA and others, is currently installed to manage jobs.

Technical summary of the Cluster nodes

Numbers of Nodes Node Description /lscr size
gen01 26 2 Intel E5345 quad-core Xeon at 2.33GHz, 16GB RAM, DDR Infiniband 64 GB
gen02 10 2 Intel E5450 quad-core Xeon at 3.00 GHz, 32GB RAM, DDR Infiniband 903 GB
gen03 64 2 Intel E5530 quad-core Xeon at 2.4 GHz, 48GB RAM, QDR Infiniband 271 GB
gen04 112 2 Intel E5645 six-core Xeon at 2.4 GHz, 48GB RAM, QDR Infiniband 1.8 TB

Hard disk quotas

Please note the following changes in CFN Cluster operations for all users: (updated on Mar. 7, 2011)

  1. Please use the disk space allotted under /work/[userid] to initiate all jobs so that disk i/o to/from the computing nodes will go to the /work partition for input/output files.
  2. Continue to use /lscr for large temporary files that only need disk i/o from one compute node during the run.
  3. Start to use /gscr for large temporary files that need disk i/o from multiple compute nodes during the run.

These changes will take full advantage of the new, fast-access file system and should result in more efficient operation for all users.

Also, to assure stable operation and fairness for all users, a disk quota system has been implemented for the /work and /gscr fast-access file systems as well as the "/home" directory. These quotas and the backup policy for the files are as follows:

  • /home(1 or 2) --- 30GBytes (soft)/40 GBytes (hard); image backup of all files.
  • /work --- 30GBytes (soft)/40 GBytes (hard); image backup of all files.
  • /gscr --- 80GBytes (soft)/100 GBytes (hard); no backup; 45-day lifetime limit on all files.

The soft limit is your actual quota; while the hard limit allows you to temporarily write to the file system beyond your quota, so that your job will not be immediately killed. However, you must delete some files and go back below your quota within 14 days.

For users who do need larger file space than their quota, please make a request through your point of contact. We provide larger quotas on "/work" and "/gscr", but not "/home", for a limited number of users.

Useful commands to check your quota usage:

$ lfs quota -u <username> /work
$ lfs quota -u <username> /gscr

Available software

Various scientific software is available for exploring fundamental properties of nanomaterials. Please see a complete list in the software page.

Using the queuing system

Queues and attributes


A queuing system is enforced to ensure that resources will be properly allocated. The queuing applies limits, preventing arbitrarily long jobs, and limits the number of processors being used for arbitrarily long jobs. There are three queues currently in use. The default is 'cfn_regular' with a wall clock limit of 2 hours. Please note that starting from March 28, 2013, there is no more 'cfn_gen04' queue. You will need to remove the "-q cfn_gen04" option from your job submission scripts. The scheduler should automatically assign jobs to gen04 nodes if you have ppn=12. The 'cfn_short' queue has also been replaced with a 'cfn_debug' queue which has a max jobs of 2 per user at a time and max walltime of 1 hour per job.

  • cfn_regular
    • per job - max walltime - 72 hours
  • cfn_long
    • limited to gen01, gen02 and gen03 nodes only
    • per job - min walltime - 72 hours
    • per job - max walltime - 28 days
    • per user - max processor hours - 28 node days
    • per user - max 32 processors
    • for queue - max 128 processors
  • cfn_debug
    • per job - max walltime - 30 minutes
    • per user - max 2 jobs

There are two basic ways to specify a given queue:

  • To specify queue on the command line, add the option "-q [queue]" to the qsub command. For example, to run a one hour interactive job on 2 full nodes of the 'cfn_debug'
    qsub -l walltime=1:00:00 -I -l nodes=2:ppn=8 -q cfn_debug
  • To specify the queue with a submission script, you may either include it on the command line, or add the following line to the beginning of the script:
    #PBS -q cfn_debug 


  • In addition to these new queues, we will also be applying attributes to nodes. The purpose of these is to allow users to submit requests based on the specific resources required. In addition to standard resource requests such as total processors, processors per node, memory per node, you can specify whether you want DDR infiniband or QDR infiniband.
  • These attributes are specified in the "-l nodes=" option to qsub (or the #PBS line). Example, qsub -l nodes=10:gen01 would request 10 gen01 nodes. The new attributes are:
    • gen01 - Specifically gen01 nodes
    • gen02 - Specifically gen02 nodes
    • gen03 - Specifically gen03 nodes
    • gen04 - Specifically gen04 nodes (We encourage users to request gen04 nodes only when 12 ppn are needed.)
    • ddr (dual data rate infiniband, the IB fabric for the gen01/gen02 nodes)
    • qdr (quad data rate infiniband, the IB fabric for gen03/gen04 nodes)

You can find more information about PBS from TORQUE documentation. Note especially the Chapter 2.0 relating to submitting and managing jobs.

A sample PBS script file explained

  #PBS -S /bin/bash             -- the bash shell will be used to interpret the script
  # comment                     -- a line begins with # (not #PBS) is a comment 
  #PBS -N myjob                 -- the name of the batch job will be 'myjob'
  #PBS -q cfn_debug             -- specify the queue
  #PBS -m e                     -- mail is sent when the job terminates 
  #PBS -M my_email              -- specify your email address 
  #PBS -j oe                    -- the standard error stream will be merged into the
                                      standard error stream file in [jobname].o[jobID]. 
  #PBS -l nodes=1:ppn=4:gen03   -- request 4 processors on a gen03 node
  #PBS -l mem=1gb               -- request 1GB of physical memory

  cd $PBS_O_WORKDIR             -- go to the directory where the script is submitted.
  ./myprog                      -- execute the program 'myprog' 

Common PBS command

  • qsub [flags] [pbs_script]
    • Submits a PBS batch script called pbs_script
    • "qsub -I" submits an interactive job.
  • qstat [flags]
    • Check status of PBS queues.
    • "qstat -u [user] " checks status of the specified user's jobs only.
  • qdel [flags] jobid
    • Delete a job.
    • Use "qstat" to find out the job ID; only need to supply the number.
  • pbsnodes [hostname]
    • Displays that properties of the compute nodes in the cluster.

Environmental modules

  • The "environment-modules" package is now necessary in order to set your processing environment correctly. We will no longer be placing MPI implementations in your default environment. You will need to specifically select your MPI implementation.
  • "environment-modules", or "modules" as it is sometimes referred, is a tool to allow users to add or remove grouped sets of similar environment variables. It allows for multiple versions of the same software to be installed on the same system at a given time and provides an easy for you, the user, to select that software. The command line for this function is "module", and the manual page for this is available via "man module".
  • The "environment-modules" package will modify your environment variables such as PATH, MANPATH, LIBRARY_PATH, LD_LIBRARY_PATH and other variables in order to enable these software packages to work. The basic format for naming modules will be "[software]/[version]".
  • If you know you will be using certain software on a regular basis (mvapich2/1.4-gcc for example), then you may add the command to load that module into your shell startup scripts. The file .cshrc for tcsh/csh or .bashrc for bash/sh.

Common module commands

  • module avail
    • List all available package modules
  • module load module_name
    • Load settings for module_name into current environment
  • module unload module_name
    • Remove settings for module_name into current environment
  • module display module_name
    • Display settings for module_name
  • module purge
    • Remove all module settings from current environment

Data backup

It is recommended that users retrieve important data from CFN Cluster to their local storage. CFN Cluster's own backup storage can be accessed through "/backup" directory. We keep a daily backup for all "/home" and "/work" directories, and a weekly backup for "/home" directories only. Our update schedule is the following:

  • updates 10:00 PM every Sunday - Thursday
    • /backup/daily/home0
  • updates 12:00 AM every Monday - Friday
    • /backup/daily/home1
  • updates 8:00PM daily
    • /backup/daily/work
  • updates 10:00PM every Sunday
    • /backup/weekly/home0
  • updates 12:00AM every Monday
    • /backup/weekly/home1

Current status

Web pages that display the cluster's status are updated hourly and can be viewed with information on free nodes only, in detail, or in full.

Frequently asked questions

The FAQ page gives answers to questions about user proposal, file transfer, and others.