Getting Started with Unity

Logging in, managing files, and running jobs

Unity Cluster
Research Computing

In this Talk

  1. What is Unity, Anyway?

  2. Accessing Unity

  3. Storage on Unity

  4. Transferring Data

  5. Introduction to Slurm

  6. Using Unity OnDemand

  7. The Unity Software Stack

  8. Get Help

What is Unity?

The Unity Research Computing Platform…​

  • supports research computing tasks of all scales.

  • makes collaborating simple.

  • provides access to specialized equipment, including GPUs and large-memory servers.

  • connects researchers with interdisciplinary support staff.

Unity’s Facets

team icon

Research Groups

storage icon

Storage

compute icon

Compute

support icon

Support

Unity by the Numbers

ResourceCount

Nodes

~450

CPUs

~25000

GPUs

~1400

Users

Over 2000

PIs

Over 450

Research Group Management

team icon

  • Faculty or senior research staff (PIs) own PI Groups.

  • PIs control user access to Unity and their group.

  • The Unity Portal is the management interface.

Getting an Account

  1. Sign in to the Unity Portal.

  2. Request a PI group (PIs) or access to an existing PI group (students, research staff, collaborators).

Students must join an existing PI group to get Unity access

See the "Requesting an Account" documentation for detailed instructions.

portal signup

The Unity Portal

unity.rc.umass.edu

Accessing Unity

Accessing Unity: Unity OnDemand Portal

ood.unity.rc.umass.edu
Unity Unity OnDemand Portal

The Unity OnDemand Portal

ood.unity.rc.umass.edu

Features:

  • Log in with your school credentials rather than SSH keys.

  • Access a shell from your browser.

  • Use interactive GUI applications like JupyterLab, RStudio, and a graphical desktop.

  • Get Slurm templates for common job setups.

Accessing Unity: SSH

  • SSH stands for secure shell

  • Unity only allows key-based authentication

  • Your Unity username is based off your email

Hostname/Address: unity.rc.umass.edu
Username: NETID_school_edu
Example email: alovelace@umass.edu
Example username: alovelace_umass_edu
ssh -i <path to private key> NETID_school_edu@unity.rc.umass.edu

Generating SSH Keys

SSH keys can be generated on the Unity portal

  1. Under Account Settings, click the plus button under the SSH Keys heading

  2. Select Generate Key

  3. Place the downloaded private key in the appropriate spot for your access method on your local computer (generally ~/.ssh)

Never share your private key with anyone

Adding an Existing SSH Key

To add an existing ssh key:

  1. Copy the public ssh key (often located in ~/.ssh)

  2. Log in to the Unity Portal

  3. Click the red "plus" button under SSH Keys

  4. Paste your public SSH key into the box and click Add Key

See the Unity documentation for more detailed information on connecting via SSH, including with PuTTY.

Storage on Unity

UNITY IS NOT APPROPRIATE FOR PROTECTED DATA!

  • No human subjects data with identifying info, including HIPAA.

  • Deidentified data is usually OK, but defer to your IRB.

Types of Storage on Unity

Four storage options on Unity:

  • Home directory (50 GB quota): /home/<username>

  • PI group work directory (1 TB quota): /work/pi_<PI username>

  • Project storage (Free and paid options. Free tier project quota set by home institution): /project

  • Scratch space: via hpc-workspace: /scratch<2,3>/workspace

See storage documentation here.

Data Persistence on Unity

  • 3 day snapshots for data in $HOME and the PI Work directory

  • There are no backups on Unity

  • Do not rely on Unity to store data long term

To discuss options for long term storage, contact the facilitation team at hpc@umass.edu.

THERE ARE CURRENTLY NO BACKUPS ON UNITY.
However, data backup is in the product roadmap.

Transferring Data

  1. Command line tools like rsync and scp

  2. Unity OnDemand file browser

  3. Globus

For detailed information, visit our file management documentation.

Which to choose?

  1. CLI tools if you’re comfortable with the command line

  2. Unity OnDemand for an intuitive interface and file size less than 5G

  3. Globus for large files to and from existing endpoints or Globus Personal Endpoint

Welcome to Slurm

Slurm

What is Slurm?

Slurm is a resource manager that

  • controls access to resources (CPUs, GPUs, memory, etc) for jobs

  • organizes a queue of jobs and arbitrates which jobs get resources next

See the Quickstart Guide for a crash course in Slurm

Running Interactive Jobs

An interactive session lets you access the command line on a compute node

# This requests an interactive job with
# four CPU cores for two hours on the cpu partition
# -c number of cores
# -p partition
# -t time in the format Days-Hours:Minutes:Seconds
# --mem memory per node (defaults to MB unless you include "G")
salloc -c 4 -p cpu -t 2:00:00 --mem=8G
Requests for high demand resources (like A100s) could wait a while!

Running Batch Jobs

A batch job submits a pre-determined script to the queue for non-interactive execution

#!/bin/bash

#SBATCH -N 1            # Number of nodes requested
#SBATCH -n 1            # Number of tasks requested
#SBATCH -p cpu  # Partition
#SBATCH -t 2:00:00      # Time (Days-Hours:Minutes:Seconds)
#SBATCH --mem=8G        # memory per node (defaults to MB without "G")
#SBATCH -o o-%j.out     # Output file (%j is job numbers)

echo "Hello World!"

To run: sbatch <filename>

Slurm Commands

CommandDescriptionNotes

salloc \ srun

Runs an executable in a parallel job

sbatch

Submits a batch job

Slurm options are prefixed with #SBATCH

squeue

View information about running and queued jobs

Use --me to restrict to only your jobs

scancel <job id>

Cancels a running or queued job

If you’re using job arrays, this will cancel the whole array unless you specify a sub job

sinfo

View partition, node, and cluster information

Use -p <partition> to restrict to a certain partition

sacct

Slurm accounting information

Use -X to hide "job steps"

See our Slurm cheat sheet here!

Tasks vs CPUs per Task

Slurm draws the distinction between tasks and CPUs.

For most applications using threading, use

#SBATCH -N 1
#SBATCH -n 1
#SBATCH --cpus-per-task=<number of CPUS desired>

For MPI applications, use

#SBATCH -N <number of nodes desired>
#SBATCH -n <number of tasks desired>
There are situations where code needs both multiple tasks and multiple CPUs per task, but they are rare! When in doubt, contact us for help.

MPI Jobs

  1. Load mpi with module load openmpi/<VERSION>

  2. Compile your code with the mpicc compiler

  3. Run with srun or mpirun

#!/bin/bash

#SBATCH -N 1
#SBATCH -n 8
#SBATCH --mem=1G

module load openmpi/5.0.3 # Or other MPI version

srun ./my-mpi-program
# OR
mpirun ./my-mpi-program

Multi-node MPI Jobs

Only some nodes have infiniband. If you need to use MPI across nodes, you can request only those nodes with --constraint=ib

#!/bin/bash

#SBATCH -N 4
#SBATCH -n 200
#SBATCH --mem=1G
#SBATCH --constraint=ib
#SBATCH -p cpu

module load openmpi/5.0.3 # Or other MPI version

mpirun ./my-mpi-program

Using GPUs

Unity has a variety of GPUs available! To see available GPUs by node and partition, see our documentation.

#!/bin/bash

#SBATCH -N 1
#SBATCH -n 1
#SBATCH --gpus=1
#SBATCH -p gpu,gpu-preempt
#SBATCH --mem=8G
Using -p gpu,gpu-preempt will get you access to the most GPUs

Using Specific GPUs - Method 1

If you need a specific GPU, you can specify that in the --gpus option in the format gpu-name:number:

#!/bin/bash

#SBATCH -N 1
#SBATCH -n 1
#SBATCH --gpus=2080ti:1
#SBATCH -p gpu,gpu-preempt
#SBATCH --mem=8G
The GPU name will always be lowercase

Using Specific GPUs - Method 2

If you need one of several types of GPU, but you don’t care about which, you can use constraint with --gpus=1:

#!/bin/bash

#SBATCH -N 1
#SBATCH -n 1
#SBATCH --constraint=[a100|m40|rtx8000]
#SBATCH --gpus=1
#SBATCH -p gpu,gpu-preempt
#SBATCH --mem=8G
Constraints can be combined with | for "or" or & for "and": 'vram40&sm_80'

Unity Partitions

Unity has three types of partitions:

  • General use partitions (open to all): cpu, gpu

  • Condo partitions (open to specific groups)

  • Preempt partitions (open to all, but your job may be killed and requeued after 2 hours): cpu-preempt, gpu-preempt

Preempt partitions allow anyone to run on private hardware.

Unity Constraints

Constraints are a way to specify specific hardware features. Unity is a heterogeneous cluster. Unity nodes have constraints based on…​

  • GPU type

  • Infiniband

  • Architecture / Microarchitecture

  • Vectorization instructions

For a full list of available constraints, run unity-slurm-list-constraints or see our documentation.

The Unity User Software Stack

Unity hosts a variety of centrally-installed packages for cluster-wide use, including…​

  • Interactive Unity OnDemand apps

  • MPI stacks

  • Compilers (gnu, intel)

  • Interpreters (Python, R, Julia, etc)

  • Scientific Libraries (hdf5, netcdf, openblas, etc)

  • GPU utilities (cuda, etc)

We use lmod to manage most software

Seeing Available Software

Or, via the shell…​

  • To see currently loaded software, use module list

  • To see available software that is currently loadable, use module avail

  • To see all available software, use module spider or module spider <package name>

Loading Modules with lmod

To load a module, use

module load <package name>/<version>
Specifying a version is required. Module versions can be tab completed!

If a module says it can’t be loaded, use spider to find out requirements:

module spider <package name/version>

Software in User Space

You’re welcome to install software in your work directory! There are some situations where it’s preferable to install software in user-space:

  • Conda environments and packages

  • R packages

  • Very specific software

If you need assistance with software, please submit a ticket to hpc@umass.edu

Follow Up & Contact

Keep in touch! To contact us:

Get this talk at unity.unityhpc.page/education/workshops/unity-onboarding