batch job 관련 참고자료.

http://www.ncsa.illinois.edu/UserInfo/Resources/Hardware/Intel64Cluster/Doc/Jobs.html

1. Interactive Use

Jobs should not be run on the interactive nodes. Their use is primarily for compiling and building your programs. Instead, please run jobs on the compute nodes. See the section on qsub -I for instructions on how to run an interactive job on the compute nodes.

2. Running Programs

MPI

All the implementations of MPI on the NCSA Intel 64 Linux Cluster have the mpirun script for running an MPI program. See the sample batch scripts for syntax details for the MPI implementations.

Notes:

• The environment variable $PBS_NODEFILE is automatically defined in a batch job to point to a temporary file that contains the list of nodes assigned to the job.
• The arguments to mpirun need to come before your executable. Any arguments after your executable are considered to be arguments to your executable.
• The VMI2 MPI implementation does not propagate environment variables well. The workaround is to create a wrapper script that sets all the environment variables that your code will need along with the executable. Then in your batch script use the wrapper script as the executable in your mpirun line.
• As noted in the MVAPICH2 sample batch script, in order to run MVAPICH2 jobs, a file named .mpd.conf needs to exist in your home directory with the line:

  MPD_SECRETWORD=XXXXXXX     
  
  

  where XXXXXXX is a string of random alphanumeric characters, with at least one alphabetic character.

  The file should also be readable and writeable only by the owner, so the permissions need to be set as follows:

  chmod 700 $HOME/.mpd.conf
  
  

OpenMP

Before you run an OpenMP program, set the environment variable OMP_NUM_THREADS to the number of thtreads you want. For example, to run program a.out interactively with two threads:

  setenv OMP_NUM_THREADS 2

  ./a.out

The following environment variables may also be useful in running your OpenMP programs:

OMP_SCHEDULE	Sets the schedule type and (optionally) the chunk size for DO and PARALLEL DO loops declared with a schedule of RUNTIME. The default is STATIC.
KMP_LIBRARY	sets the run-time execution mode. The default is throughput, but it can be set to turnaround so worker threads do not yield while waiting for work.
KMP_STACKSIZE	Sets the number of bytes to allocate for the stack of each parallel thread. You can use a suffix k, m, or g to specify kilobytes, megabytes or gigabytes. The default is 4m.

Hybrid MPI/OpenMP

To run a MPI/OpenMP hybrid program, you need to set the envionment variable OMP_NUM_THREADS to the number of threads you want, and change the number of cpus per node for MPI accordingly. For example, to run a program with 10 MPI ranks and 8 threads for each rank, do the following in your batch script:

  #PBS -l nodes=10:ppn=1

  setenv OMP_NUM_THREADS 8

See the exception with VMI2 in the MPI section above on using a wrapper.

(See the qsub section for information on PBS directives.)

3. Batch System (Torque)

The NCSA Intel 64 Linux Cluster uses the Torque Resource Manager with the Moab Workload Manager for running jobs. Torque is based upon OpenPBS, so the commands are the same as PBS commands.

3.1 Scheduling Policies

The scheduling policy on Abe is set to highly favor large node-count jobs.

Also, as with other HPC systems at NCSA, the scheduling policy includes fair-share. This is a policy whereby a job's priority may be increased or decreased because of other jobs that the user's project may be running or have recently run. Basically, in order to give everyone a fair opportunity to run jobs, a user's job will have a higher priority if users in their project haven't run jobs in the recent past. Fair-share also factors in the ratio of the service units the user's project is allocated and the time to the allocation expiration.

To maximize utilization, the scheduler will also back-fill jobs. When trying to schedule large blocks of nodes for large jobs, there are often "holes" where some nodes are idle waiting to be added to a pool to start a large waiting job. The scheduler back-fills smaller jobs into these holes.

When figuring out a job's priority relative to other jobs, there are several factors which are taken into account. Some of these factors include:

• job size (how many nodes)
• job expansion factor (the ratio of the time the job has spent eligible to be run versus how much time the job has requested)
• the raw amount of time the job has spent eligible to be run
• fair-share factors

A relative weighting of these factors contributes to a job's priority.

A debug queue is available to facilitate fast turnaround on debugging/testing jobs. Jobs in this queue have an intrinsically higher priority; additionally, they accrue priority at a much higher rate because the expansion factor (and its associated priority factor) increases very quickly.

In order to keep jobs from the long queue from dominating the system and causing shorter jobs to wait behind them, there is a limit on the nodes currently running jobs from the long queue. Given the fluid nature of our job load, this limit is adjusted from time to time, but in the general case we tend to keep it between 1/4 and 1/3 of the available nodes. When that limit is reached, subsequent jobs in the queue may go into a blocked state until running jobs finish and free up resources. Then the jobs will automatically be moved from the blocked state and get scheduled to run.

3.2 Queues

The following queues are currently available for users:

Queue	Walltime	Max # Nodes
debug	30 mins	16
normal(default)	48 hours	256 (as of July 1 2009)
wide	48 hours	600 (as of September 16 2009)
long	168 hours	256 (as of July 1 2009)

NOTES:

1. Jobs submitted to all but the wide queue will stay within the bounds of either the 16GB memory or the 8GB memory nodes i.e., jobs will not span across the two types of resources unless submitted to the wide queue.
2. The minimum node count for the wide queue is 64.
3. Access to resources over 600 nodes (up to a maximum of 1024 nodes) is available by special request. Please send email to consult@ncsa.uiuc.edu to request access. Include the number of nodes and the wall time required, and the number of jobs to be run.

3.3 Batch Commands

Below are brief descriptions of the useful batch commands. For more detailed information, refer to the individual man pages.

3.3.1 qsub

The qsub command is used to submit a batch job to a queue. All options to qsub can be specified either on the command line or as a line in a script (known as an embedded option). Command line options have precedence over embedded options. Scripts can be submitted using

qsub [list of qsub options] script_name

The main qsub commands are listed below. The sample batch scripts illustrates qsub usage and options. Also see the qsub man page for other options.

• -l resource-list: specifies resource limits. The resource_list argument is of the form:

  resource_name[=[value]][,resource_name[=[value]],...]:resource
  
  

  The resource_names are:
  
  walltime: maximum wall clock time (hh:mm:ss) [default: 10 mins]
  nodes: number of 8-core nodes [default: 1 node]
  ppn: how many cores per node to use (1 through 8) [default: ppn=1]
  resource: resource to be used. The available resource is himem to access the 16 GB memory nodes.
  Note: Specify the himem resource only if you absolutely need the higher memory nodes since it can impact turnaround time of the job.

  Examples:

  #PBS -l walltime=00:30:00,nodes=2:ppn=8
  
  #PBS -l walltime=00:30:00,nodes=2:ppn=8:himem
  
  

• -q queue_name: specify queue name.[default: normal]

• -N jobname: specifies the job name.

• -o out_file: store the standard output of the job to file out_file. After the job is done, this file will be found in the directory from which the qsub command was issued. [default :<jobname>.o<PBS_JOBID>]

• -e err_file: store the standard error of the job to file err_file. After the job is done, this file will be found in the directory from which the qsub command was issued. [default :<jobname>.e<PBS_JOBID>]

• -j oe: merge standard output and standard error into standard output file.

• -V: export all your environment variables to the batch job.

• -m be: send mail at the beginning and end of a job.

• -M myemail@myuniv.edu : send any email to given email address.

• -A project: charge your job to a specific project (TeraGrid project or NCSA PSN). (for users in more than one project)

• -X: enables X11 forwarding.      

Notes:

• Using the -N option will generate stdout and stderr files of the form: <jobname>.o<jobid> and <jobname>.o<jobid> respectively in the directory from where the batch job was submitted when used without the -o and -e options.
• Temporary stdout/stderr files while the job is running are located in the home directory [$HOME/.pbs_spool or $HOME], and named <jobid>.abem5.OU and <jobid>.abem5.ER.

3.3.2 qsub -I

The -I option tells qsub you want to run an interactive job. You can also use other qsub options such as those documented in the batch sample scripts. For example, the following command:

   qsub -I -V -l walltime=00:30:00,nodes=2:ppn=8

will run an interactive job with a wall clock limit of 30 minutes, using two nodes and eight cores per node.

After you enter the command, you will have to wait for Torque to start the job. As with any job, your interactive job will wait in the queue until the specified number of nodes is available. If you specify a small number of nodes for smaller amounts of time, the wait should be shorter because your job will backfill among larger jobs. Once the job starts, you will see something like this:

qsub: waiting for job 1244.abem5.ncsa.uiuc.edu to start

qsub: job 1244.abem5.ncsa.uiuc.edu ready

Now you are logged into the launch node. At this point, you can use the appropriate command to start your program.

When you are done with your runs, you can use the exit command to end the job.

3.3.3 qstat

The qstat command displays the status of batch jobs.

• qstat -a gives the status of all jobs on the system.
• qstat -n lists nodes allocated to a running job in addition to basic information.
• qstat -f PBS_JOBID gives detailed information on a particular job.
  Note: Currently PBS_JOBID needs to be the full extension: <jobid>.abem5.ncsa.uiuc.edu.
• qstat -q provides summary information on all the queues.

See the man page for other options available.

3.3.4 qhist

qhist, a locally written tool available on the NCSA Intel 64 Linux Cluster, summarizes the raw accounting record(s) for one or more jobs. See the output of "qhist --help" for details.
NOTE: As of May 6 2009, SU charges for a job are available the day after the job completes.

To display information about a specific job, the syntax is qhist PBS_JOBID.

3.3.5 qdel

The qdel command deletes a queued job or kills a running job. The syntax is qdel PBS_JOBID.

Note: You only need to use the numeric part of the Job ID.

3.4 Sample Batch Scripts

Sample batch scripts are available in the directory /usr/local/doc/batch_scripts for use as a template.

3.5 Disk Space for Batch Jobs

Scratch space for batch jobs is provided via a per-job scratch directory that is created at the beginning of the job. This directory is created under /scratch/batch, and is based on the JobID. If the batch script uses one of the sample scripts as a template, the name of this scratch directory is available to job scripts with the $SCR environment variable.

Your job scratch directory may be deleted soon [possibly immediately] after your job completes, so you should take care to transfer results to the mass storage system (see the section Automated Saving of Files from Batch Jobs).

The cdjob command can be used to change the working directory to the scratch directory of a running batch job. The syntax is

cdjob PBS_JOBID

3.6 Automated Saving of Files from Batch Jobs

The saveafterjob utility is available for automated, guaranteed saving of output files from batch jobs to the mass storage system. It needs to be added with the SoftEnv key: +saj
For details on its use, see the saveafterjob page and the sample PBS batch scripts.

4. Notes

• To avoid excessive paging, we recommend restricting job memory to 875MB/core or 7GB/node.
• While a job is running, you can ssh to the compute nodes on which your job is running. qstat -n provides the list of hosts assigned to your job. The first host on the list is the launch node.