Common commands in TORQUE and LoadLeveler

To help you migrate from the TORQUE job submission system (also called PBS, and used by Quarry at Indiana University) to IBM's LoadLeveler (LL) job submission system (used by Big Red), the following tables show commands, environment variables, and resource specifications for the two batch systems side by side.

If you are new to Big Red, see Getting started on Big Red.

On this page:

Common commands

  TORQUE command LL command
Job submission qsub [scriptfile] llsubmit [scriptfile]
Job deletion qdel [job_id] llcancel [job_id]
Job status (for user) qstat -u [username] llq -u [username]
Extended job status qstat -f [job_id] llq -l [job_id]
Hold a job temporarily qhold [job_id] llhold [job_id]
Resume job on hold qrls [job_id] llhold -r [job_id]
List of usable queues qstat -Q llclass
GUI for batch system xpbs xload

Environment variables

  TORQUE command LL command
Job ID $PBS_JOBID $LOADL_STEP_ID
Submission directory $PBS_O_WORKDIR $LOADL_STEP_INITDIR
Processor list $PBS_NODEFILE $LOADL_PROCESSOR_LIST (See note below.)

Note: For jobs using more than 128 tasks (processors/cores), the value of $LOADL_PROCESSOR_LIST will be set to NULL. If your job uses more than 128 tasks, run: /opt/ibmll/LoadL/full/bin/ll_get_machine_list > /tmp/machinelist.$LOADL_STEP_ID

Then use the temporary file (/tmp/machinelist.$LOADL_STEP_ID) created to get the list of machines assigned to your job.

Resource specifications

  TORQUE command LL command
Queue #PBS -q [queue] #@ class=[queue]
Nodes #PBS -l nodes=[#] #@ node=[#]
Processors #PBS -l ppn=[#] #@ tasks_per_node=[#]
Wall clock limit #PBS -l walltime=[hh:mm:ss] #@ wall_clock_limit=[hh:mm:ss]
Standard output file #PBS -o [file] #@ output=[file]
Standard error #PBS -e [file] #@ error=[file]
Copy environment #PBS -V #@ environment=COPY_ALL
Notification event #PBS -m abe #@ notification=start|error|complete|never|always
Email address #PBS -M [email] #@ notify_user=[email]
Job name #PBS -N [name] #@ job_name=[name]
Job restart #PBS -r [y|n] #@ restart=[yes|no]
Job type N/A #@ job_type=[type]
Initial directory N/A #@ initialdir=[directory]
Node usage N/A? #@ node_usage=not_shared
Memory requirement N/A #@ requirements=(Memory >= NumMegaBytes)

Common Moab scheduler commands

Big Red and Quarry use the Moab job scheduler. Frequently used Moab scheduler commands include:

Command LL and TORQUE
Show currently running/queued jobs showq | less
Check a job's status checkjob [jobid] or checkjob -v [jobid] (for details)
Show when your job might start showstart [job_id]
Show fairshare information (priority, etc.) diagnose -f | less
Check a nodes status checknode [nodename]
Show current reservations showres

For more on Moab, refer to the Moab Workload Manager User's Manual.

This document was developed with support from the National Science Foundation (NSF) under Grant No. 0503697 to the University of Chicago and subcontracted to Indiana University. Additional support was provided by IU through its participation in the TeraGrid, which is supported by the NSF under Grants No. 0833618, SCI451237, SCI535258, and SCI504075. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF.