Quarry usage policies

On this page:

Accounts

Quarry serves multiple communities as a research cluster and as a general purpose Linux environment. Accounts are open to all Indiana University students, faculty, and staff, as well as affiliated researchers.

Home directories

Quarry home directory disk space is allocated on the IBM N5500 NAS storage device. New accounts have a 10GB disk quota which is shared with Big Red and the Research Database Complex (RDC), if you have accounts on those systems; see At IU, how much disk space is available to me on the research systems? If you need additional permanent storage space, apply for an account on IU's Massive Data Storage Service (MDSS).

Computational resources (queues)

Each queue has properties defined in order to maximize job throughput and minimize time spent waiting in queue. Quarry uses a TORQUE routing queue to funnel jobs from the default BATCH queue into the SERIAL, NORMAL, and LONG queues. This sorting is done according to the size of the job's resource requests as documented below. By default, your jobs will run in the batch queue unless you specifically submit your jobs to one of the special queues such as DEBUG or OSG.

Note: Cluster-wide, the maximum number of tasks is 928 (116 compute nodes available [112 in queues, 4 in user-selectable debug] X 8 tasks per node).

SERIAL queue properties

  • Nodes: 5 serial (q029-q033) + 33 normal (q034-q066) + 46 long (q067-q112) + 28 himem (q113-q140) = 112 total
  • Maximum walltime: 12 hours
  • Maximum nodes per job: 1 node
  • Maximum cores per job: 8 cores
  • Maximum number of jobs per queue: 3,500
  • Maximum number of jobs per user: 1,500
  • Direct submission: No

NORMAL queue properties

  • Nodes: 33 normal (q034-q066) + 46 long (q067-q112) = 79 total
  • Maximum walltime: 7 days
  • Maximum nodes per job: 2 nodes
  • Maximum cores per job: 16 cores
  • Maximum number of jobs per queue: None
  • Maximum number of jobs per user: None
  • Direct submission: No

LONG queue properties

  • Nodes: 46 long (q067-q112) = 46 total
  • Maximum walltime: 14 days
  • Maximum nodes per job: 42 nodes
  • Maximum cores per job: 336 cores
  • Maximum number of jobs per queue: 4,000
  • Maximum number of jobs per user: 500
  • Direct submission: No

Note: The LONG queue was formerly known as DEFAULTQ. An alias for the old name is in place for job submissions, but when you use commands like llclass and showq -w class=XXXXX, you must use the new name.

DEBUG queue properties

  • Nodes: 4 blades dedicated
  • Maximum walltime: 30 min
  • Maximum nodes per job: 2 nodes
  • Maximum cores per job: 16 cores
  • Maximum number of jobs per queue: None
  • Maximum number of jobs per user: 2
  • Direct submission: Yes

Note: The DEBUG queue was formerly known as FASTQ. An alias for the old name is in place for job submissions, but when you use commands like llclass and showq -w class=XXXXX, you must use the new name.

HIMEM queue properties

  • Nodes: 28 himem (q113-q140) = 28 total
  • Maximum walltime: 14 days
  • Maximum nodes per job: 28 (= maximum in queue)
  • Maximum cores per job: 224 (= maximum in queue)
  • Maximum jobs per queue: None
  • Maximum jobs per user: None
  • Direct submission: Yes, for now

OSG queue properties (group restricted access)

  • Nodes: 16 blades dedicated
  • Maximum walltime: 14 days
  • Maximum nodes per job: None
  • Maximum cores per job: None
  • Maximum number of jobs per queue: None
  • Maximum number of jobs per user: None
  • Direct submission: Yes

Note: Access to the osg queue is restricted and jobs are routed to this queue via Globus. The debug queue is for debugging only; once code has been debugged, you may submit it to the long queue. If you have questions about the job queues, email High Performance Systems.

Note: The OSG queue was formerly known as OSGQ. An alias for the old name is in place for job submissions, but when you use commands like llclass and showq -w class=XXXXX, you must use the new name.

CPU limits and batch jobs

User processes on the login nodes are limited to 20 minutes of CPU time. Processes exceeding this limit are automatically terminated with no warning. You may run interactive jobs requiring more than 20 minutes of CPU time on nodes b005, b006, b007, and b008. There is a 24-hour CPU time limit on all user processes running on these nodes. If you require more than 24 hours of CPU time, submit a batch job to the TORQUE batch queuing system (also called PBS) with the qsub command.

Mail usage

UITS does not provide a production mail service on the Quarry cluster; however, TORQUE communicates via email. Quarry will send email about your jobs to the address specified in the ~/.forward file in your home directory. (Note the period [ . ] preceding the filename.) By default, this is the email address you provided when you requested your account.

If you'd like to change this email address, enter a command similar to the following, replacing username@host.com with your email address:

jdoe@Quarry:~> echo "username@host.com" > ~/.forward

Be sure to use a valid email address; if you do not, you will not be notified about the status of your jobs. For help, see How do I forward my mail from a Unix account?

Scheduled down time

The Quarry maintenance window is scheduled for the first Tuesday of each month. A window will be used only if necessary; an announcement regarding the status of maintenance will be posted the preceding Friday morning on both the Quarry Message of the Day and the Scheduled Downtime page.

Questions and comments

The policies described in this document were established with the goal of providing a stable, powerful, and efficient research computing environment for IU faculty, graduate students, and staff. If you have any questions or comments about these policies, UITS welcomes your input; email High Performance Systems.