Indiana University
 

Mason

Mason at Indiana University

On this page:

System overview

Mason (mason.indiana.edu) at Indiana University is a large memory computer cluster configured to support data-intensive high-performance computing tasks for IU researchers. Mason consists of 16 Hewlett Packard DL580 servers, each containing four Intel Xeon L7555 8-core processors and 512 GB of RAM. The total RAM in the system is 8 TB. Each server chassis has a 10-gigabit Ethernet connection that connects to the other research systems at IU. For hardware configuration details, see System information.

The Mason nodes run Red Hat Enterprise Linux 6.0. Job management is provided by the TORQUE resource manager (also called PBS) and the Moab job scheduler. The Modules system is used to simplify application and environment configuration. Users may log into the cluster via SSH, using their IU Network IDs.

Mason is intended for use by IU researchers (including faculty, postdoctoral fellows, and research staff), and is also available to IU students using genome assembly software (particularly software suitable for assembly of data from next-generation sequencers), large-scale phylogenetic software, or other genome analysis applications requiring large amounts of computer memory. IU educators providing instruction on genome analysis software, and developers of such software, are also welcome to use Mason.

Back to top

System information


System configuration Aggregate information Per-node information (when applicable)
Machine type Intel Xeon (L7555), large memory (8 TB) MPP cluster 4 x Intel Xeon L7555, 512 GB of memory
Operating system Red Hat Enterprise Linux 6
Memory model Distributed
Computational systems details Total Per node
Processing capability Rmax = 4,302 GFLOPS Rmax = 239 GFLOPS
Benchmark data HPL GFLOPS 3129.75 HPL GFLOPS 222.22
Power usage 0.000153 TFLOPS/W 0.000173 TFLOPS/W
Disk space 8 TB (local) 500 GB (local)
Login nodes Rmax = 478 GFLOPS Rmax = 239 GFLOPS
Homogeneous compute nodes Rmax = 3,824 GFLOPS Rmax = 239 GFLOPS
Heterogeneous compute nodes (GPU or other accelerator) Rmax = 3,824 GFLOPS
Storage information
Data Capacitor Available to IU students, faculty, and staff
Scholarly Data Archive (SDA) Available to IU students, faculty, and staff

Back to top

System access

Methods of access

Interactive access to the Mason cluster is provided via SSH only. You can use SSH2 clients to connect to mason.indiana.edu, which resolves to either h1.mason.indiana.edu or h2.mason.indiana.edu, Mason's two login nodes.

File transfer is supported via SCP or SFTP (see SSH/SFTP clients supported at IU).

Use of public key authentication is permitted on Mason (see In SSH and SSH2 for Unix, how do I set up public key authentication?).

Logging into Mason

Access policy: A limited portion of the Mason cluster is available for IU students, faculty, and staff. Access to the remainder of the cluster is provided at the discretion of the National Center for Genome Analysis Support (NCGAS) allocations committee. To request an allocation, email NCGAS.

Passphrases: Mason login nodes use the IU Active Directory Service for user authentication. As a result, local passwords/passphrases are not supported. For information about changing your ADS passphrase, see At IU, how do I change my Network ID passphrase? For helpful information regarding secure passphrases, see Passwords and passphrases

User responsibilities: The Mason cluster is an IU resource; access via the Extreme Science and Engineering Discovery Environment (XSEDE) will be available soon. For information about your responsibilities as a user of this resource, see:

Back to top

Computing environment

Unix shell

The shell is the primary method of interacting with the Mason cluster. The command line interface provided by the shell lets users run built-in commands, utilities installed on the system, and even short ad hoc programs.

Mason supports the Bourne-again (bash) and TC (tcsh) shells. New user accounts are assigned the bash shell by default. For more on bash, see the Bash Reference Manual and the Bash (Unix shell) Wikipedia page.

To change your shell on Mason, use the changeshell command.

Note: Running chsh (instead of changeshell) changes your shell only on the node on which you run it, and leaves the other nodes of the cluster unchanged; changeshell prompts you with the shells available on the system, and changes your login shell system-wide within 15 minutes.

Environment variables: The shell uses environment variables primarily to modify shell behavior and the operation of certain commands. A good example is the PATH variable.

When the shell parses a command you have entered (i.e., after you hit Enter or Return), it interprets certain words you've typed as program files that should be executed. The shell then searches various directories on the system to locate these files. The PATH variable determines which directories are searched, and the order in which they are searched. In the bash shell, the PATH variable is a string of directories separated by colons (e.g., /bin:/usr/bin:/usr/local/bin). The shell searches for an executable file in the /bin directory, then the /usr/bin directory, and finally the /usr/local/bin directory. If files of the same name (e.g., foo) exist in all three directories, /bin/foo will be run, because the shell will find it first.

To display and change the values of environment variables:

Shell: bash tcsh
Display value: echo $VARNAME echo $VARNAME
Change value: export VARNAME=VALUE setenv VARNAME VALUE

Startup scripts: Shells offer much flexibility in terms of startup configuration. On login, bash by default reads and executes commands from the following directories (and in this order): /etc/profile ~/.bash_profile ~/.bashrc

Note: The ~ (tilde) represents your home directory (e.g., ~/.bash_profile is the .bash_profile file in your home directory).

On logout, the shell reads and executes ~/.bash_logout. For more on bash startup files, see the "Bash Startup Files" section of the Bash Reference Manual.

On login, the tcsh shell reads and executes commands from the following directories (also in this order):

/etc/csh.cshrc /etc/csh.login ~/.tcshrc (if it exists, otherwise ~/.cshrc) ~/.history ~/.login ~/.cshdirs

In practice, on Mason, only the first two files exist. You may create the others, and add commands and variables to them as you see fit.

Modules: On Mason, Modules provide a convenient method for dynamically modifying your environment. A few simple commands provide easy access to various applications on the cluster:

Command Action
$ module avail List available modules on the system
$ module load Load (modifies your environment)
$ module list Display your currently loaded modules
$ module rm Unload (modifies your environment)

For more about the Environment Modules package, see the module man page and the modulefile man page.

Back to top

Transferring your files to Mason

Presently, Mason supports only SCP and SFTP for transferring files. SCP is a command line utility included with OpenSSH. Basic use is:

scp [[user@]host1:]file1 [[user@]host2:]file2

For example, to copy foo.txt from the current directory on your computer to your home directory on Mason, use (replacing username with your Network ID username): scp foo.txt username@mason.indiana.edu:foo.txt

You may specify absolute paths or paths relative to your home directory:

scp foo.txt username@mason.indiana.edu:some/path/for/data/foo.txt

You also may leave the destination filename unspecified, in which case it will become the same as the source filename. For more, see In Unix, how do I use the scp command to securely transfer files between two computers?

The SSH File Transfer Protocol (SFTP) provides file access, transfer, and management, and offers client functionality much like FTP. For example, from a computer with a command line SFTP client (e.g., a Linux or Mac OS X workstation), you could transfer files as follows:

$ sftp username@mason.indiana.edu username@mason.indiana.edu's password: Connected to mason.indiana.edu. sftp> ls -l -rw------- 1 username group 113 May 19 2011 loadit.pbs.e897 -rw------- 1 username group 695 May 19 2011 loadit.pbs.o897 -rw-r--r-- 1 username group 693 May 19 2011 local_limits sftp> put foo.txt Uploading foo.txt to /N/hd00/username/Mason/foo.txt foo.txt 100% 43MB 76.9KB/s 09:39 sftp> exit $

Graphical SFTP clients are also available for many systems. For more, see Transferring files with SFTP

Back to top

Application development

Programming models

Mason is designed to support codes that have extremely large memory requirements. As these codes typically do not implement a distributed memory model, Mason is geared toward a serial or shared memory parallel programming paradigm. However, Mason can support distributed memory parallelism.

Compiling

  • Compilers available on Mason:

    • Intel, Portland Group, GNU (default)
    • Fortran, C, C++
    • OpenMPI

  • Compiler options:

    • Recommended options are -O3 -xHost

  • Optimization:

    • Serial codes
    • Parallel codes

  • Example sessions include:

    • MPI codes
    • OpenMP codes
    • Hybrid jobs

  • Libraries:

    • Libraries available on Mason are MKL, ACML

  • Debugging:

    • Debuggers available on Mason are gdb, idb

Back to top

Running your applications

CPU limits and batch jobs

User processes on the login nodes are limited to 20 minutes of CPU time. Processes exceeding this limit are automatically terminated without warning. If you require more than 24 hours of CPU time, submit a batch job to the TORQUE batch queuing system (also called PBS) using the qsub command.

Short definitions

A job is an instance of an application you wish to run.

A queue is a pool of compute resources that accepts job to run, and executes them according to a fairshare policy.

Job schedulers are the applications responsible for scheduling jobs.

Queues

  • Mason queues:

    • BATCH (general purpose)

  • Queue policies:

    • Fairshare scheduling
    • Walltime limit of two weeks

Job control

To delete queued or running jobs, use qdel. Occasionally, a node will become unresponsive and unable to respond to the TORQUE server's requests to kill a job. In such cases, try using qdel -W.

How to submit a job

Use qsub to submit jobs to run on Mason. If the command exits successfully, it will return a job ID, for example:

[jdoe@Quarry]$ qsub job.script 123456.qm2 [jdoe@Quarry]$

If you need attribute values different from the defaults, but less than the maximum allowed, specify these either in the job script using TORQUE directives, or on the command line with the -l switch. For example, to submit a job that needs more than the default 30 minutes of walltime, use:

qsub -l walltime=10:00:00 job.script

Note: Command-line arguments override directives in the job script, and you may specify many attributes on the command line, either as comma-separated options following the -l switch, or each with its own -l switch. The following two commands are equivalent:

qsub -l ncpus=16,mem=1024mb job.script qsub -l ncpus=16 -l mem=1024mb job.script

Useful qsub switches include:

-q <queue name> Specify user-selectable queues
-r Job is rerunnable
-a <date_time> Execute the job only after specified date and time
-V Export environment variables in your current environment to the job
-I Run interactively

For more, see the qsub man page.

How to monitor a job

Use qstat for monitoring the status of a queued or running job. Switches include:

-u <user_list> Display jobs for users in user list
-a Display all jobs
-r Display running jobs
-f Display full listing of jobs (excessive detail)
-n Display nodes allocated to jobs

How to delete a job

Use qdel to delete queued or running jobs. Occasionally, a node will become unresponsive and unable to respond to the TORQUE server's requests to kill a job. In such cases, try qdel -W.

Back to top

Tools

The performance tools currently available on Mason are VampirTrace and VAMPIR. VampirTrace is a profiling tool for Message Passing Interface (MPI) applications. VampirTrace produces tracefiles that can be analyzed with the VAMPIR performance analysis tool. For more, see the VAMPIR web site.

The corresponding modules for VampirTrace and VAMPIR are:

vampirtrace/5.11.2-gnu vampir/server-2.4.0

Back to top

Reference

Back to top

This is document bbhh in domain all.
Last modified on April 19, 2012.

Search the Knowledge Base