High Performance Computing

• High performance computing (HPC) refers to aggregated computing power. This grouping of computers can be referred to as a cluster.
• The individual machines in a cluster are nodes.
• Each node is a computer, with processors, memory, storage, etc.
  • Nodes are often configured identically.
  • There might be specially configured subsets of nodes.
    • Larger amounts of memory
    • GPUs (graphics processing unit)
  • Nodes typically access a shared fileystem, where job data is stored.
  • Nodes are interconnected.
• On each node, the processor likely has multiple cores. Sometimes, these cores can perform multiple operations simultaneously, and the operating system might report this ability as additional cores.
  • A system with four physical cores might be treated as though it has eight cores.

Figure 1 - CPU diagram [1Understanding physical and logical cpus. URL: https://www.linkedin.com/pulse/understanding-physical-logical-cpus-akshay-deshpande.]

• Users access a cluster through a login node.
  • Once logged in, users can submit jobs to the queue, where the jobs wait until the resources required for the job are free.

Figure 2 - Accessing a Cluster [2Introduction to high performance computing for supercomputing wales: supercomputing wales introduction. URL: https://supercomputingwales.github.io/SCW-tutorial/01-HPC-intro/.]

Benefits of High Performance Computing

• Jobs can be submitted to a queue, where the scheduler runs them as resources become available.
  • Individual nodes may have more memory and processing power than an average desktop.
• Jobs can be executed in parallel rather than sequentially.
  • If the data can be divided into discrete units, then each subset of the data can be submitted to the queue as a separate job.
  • Jobs start running as resources become available without user intervention.
• Jobs that require large amounts of resources can access the combined resources of many nodes.

Introduction to Atlas

Atlas is a Cray CS500 Linux cluster.

• 11,520 Intel Xeon Platinum 8260 (Cascade Lake) processor cores
• 101 terabytes of RAM
• 8 NVIDIA V100 GPUs
• Mellanox HDR100 Infiniband (100 Gb/s) interconnect

Atlas is composed of 244 nodes, all of which contain two 2.40GHz Xeon Platinum 8260 2nd Generation Scalable Processors with 24 cores each, for a total of 48 cores per node.

Using Atlas

• Getting an Account
• Logging In
• Transferring Data
• Getting Help

Getting an Account

ARS Employees

• fill out the SCINet Account Request Form
• the request will be forwarded to your supervisor for approval.
• once your supervisor approves, the Virtual Research Support Core (VRSC) will create an account for you
• you will receive an email from the VRSC with logon information

Non-ARS collaborators, students, or postdocs

• all students, postdocs, visiting scientists, and collaborators must have an ARS sponsor
• all requests must also be approved by the Research Leader.
• the sponsor must fill out the Non-ARS SCINet Account Request Form
• the request will be forwarded to the sponsor’s supervisor for approval
• once their supervisor approves, the Virtual Research Support Core (VRSC) will create an account for you
• the student, postdoc, or visiting scientist will receive an email from the VRSC with logon information

Logging In

Secure shell (SSH)

• SSH is available on Linux and macOS in the terminal apps and in the Powershell app on Windows 10.
  • For older versions of Windows, use PuTTY.
• $ ssh <SCINet UserID>@Atlas-login.hpc.msstate.edu
  • access login node
  • don't run jobs on this node
• $ ssh <SCINet UserID>@Atlas-dtn.hpc.msstate.edu
  • access data transfer nodes
  • don't run jobs on this node
• use credentials provided by VRSC

Multi-factor Authentication (MFA)

• In addition to your password, an additional code is required to log on to Atlas.
• MFA requires the Google Authenticator (GA) app on your phone or the Authy app on your desktop computer.
• The Welcome email from VRSC contains a QR code and a key, either of which can be used by the GA app to set up your device.
• Once the app is setup, the GA app displays 6-digit codes that must be entered as part of logging in to Atlas.
• More information

Password Expiration

• Passwords on Atlas expire after 60 days.
• Users can still login after their password expires, but they will be prompted to change their password immediately.
• If you have forgotten your login password, please email SCINet Virtual Research Support Core.

Transferring Data

Methods

• Globus is a simple web interface for transferring data.
  • Log in via ORCiD, a Globus account, or an institutional account (USDA-ARS currently does not provide an organizational login)
  • Both data transfer nodes of Atlas can be accessed via the Globus endpoint msuhpc2#Atlas-dtn.

Figure 3 - Globus Interface Example [3How to log in and transfer files with globus. URL: https://docs.globus.org/how-to/get-started/.]

• Secure Copy (SCP)
  • for small amounts of data that need to be transferred to user home directories
  • copies files between hosts on a network
  • uses SSH for data transfer
  • will ask for passwords as well as two-factor authentication codes.

Quotas

• 5 GB on user home folder
• set by VRSC on project folders

Loading Modules

Atlas uses LMOD as an environment module system.

• allows a single system to accommodate multiple versions of the same software application
• allows the user to select the version they want to use
• module commands set, change, or delete environment variables
• module load blast or ml blast
  • load default version of BLAST
• LMOD user guide

Some applications are available as containers, self-contained application execution environments that contain the software and its dependencies.

• Software available as a container is shown in the results of module avail once the user has loaded the singularity module.

Submitting Jobs

Atlas uses the Slurm Workload Manager as a scheduler and resource manager.

Slurm has three primary job allocation commands which accept almost identical options.

• SBATCH Submits a job runscript for later execution (batch mode)
• SALLOC Creates a job allocation and starts a shell to use it (interactive mode)
• SRUN Creates a job allocation and launches the job step (typically an MPI job)

• More information

Example - Using Salmon to quantify RNA-seq data

• adapted from [4Getting started. URL: https://combine-lab.github.io/salmon/getting_started/.]
• download data
• use srun to run one step of an analysis
• use sbatch to submit a job array to run the second step of the analysis several times on different data

Set Up

cd /home/adam.thrash/training/

curl ftp://ftp.ensemblgenomes.org/pub/plants/release-28/fasta/arabidopsis_thaliana/cdna/Arabidopsis_thaliana.TAIR10.28.cdna.all.fa.gz -o athal.fa.gz

for i in `seq 25 40`; do 
  mkdir -p data/DRR0161${i}; 
  wget -P data/DRR0161${i} ftp://ftp.sra.ebi.ac.uk/vol1/fastq/DRR016/DRR0161${i}/DRR0161${i}_1.fastq.gz; 
  wget -P data/DRR0161${i} ftp://ftp.sra.ebi.ac.uk/vol1/fastq/DRR016/DRR0161${i}/DRR0161${i}_2.fastq.gz; 
done

• change directory to /home/adam.thrash/training/
• download the transcriptome as athal.fa.gz
• for all numbers from 25 to 40
  • mkdir a directory data/DRR0161 + the number
  • download two FASTQ files to that directory

Indexing Transcriptome

salloc -A scinet

salloc: Granted job allocation 292472
salloc: Waiting for resource configuration
salloc: Nodes Atlas-0042 are ready for job

• request default allocation using scinet account
  • walltime: 15 minutes
  • nodes: 1 node
  • number of tasks: 1 core

#!/usr/bin/env bash

# CHANGE TO THE CORRECT DIRECTORY
cd /home/adam.thrash/training/

# PRINT THE HOSTNAME FOR THE EXAMPLE
hostname

# LOAD SINGULARITY AND SALMON
ml singularity salmon/1.3.0--hf69c8f4_0

# INDEX THE TRANSCRIPTOME
salmon index -t athal.fa.gz -i athal_index

• change to correct directory
• print the hostname to show that the command is being run on compute node
• load singularity module and salmon module
  • salmon module is accessible as a container, so singularity has to be loaded first
• run salmon indexing command

srun /home/adam.thrash/training/index.sh

Atlas-0042.HPC.MsState.Edu
WARNING: Skipping mount /apps/singularity-3/singularity-3.7.1/var/singularity/mnt/session/etc/resolv.conf [files]: /etc/resolv.conf doesn't exist in container
index ["athal_index"] did not previously exist  . . . creating it
...

Running Salmon as a Job Array

#!/usr/bin/env bash
#SBATCH --account scinet            # set correct account
#SBATCH --nodes=1                   # request one node
#SBATCH --cpus-per-task=8           # ask for 8 CPUs
#SBATCH --time=0-00:30:00           # set job time to  30 minutes.
#SBATCH --array=0-15                # run 16 jobs of this script
#SBATCH --output=%x.%A_%a.log       # store output as jobname.jobid_arrayid.log
#SBATCH --error=%x.%A_%a.err        # store output as jobname.jobid_arrayid.err
#SBATCH --job-name="salmon_run"     # job name that will be shown in the queue

declare -A rna_files
rna_files['DRR016125']='data/DRR016125/DRR016125_1.fastq.gz data/DRR016125/DRR016125_2.fastq.gz'
rna_files['DRR016126']='data/DRR016126/DRR016126_1.fastq.gz data/DRR016126/DRR016126_2.fastq.gz'
rna_files['DRR016127']='data/DRR016127/DRR016127_1.fastq.gz data/DRR016127/DRR016127_2.fastq.gz'
rna_files['DRR016128']='data/DRR016128/DRR016128_1.fastq.gz data/DRR016128/DRR016128_2.fastq.gz'
rna_files['DRR016129']='data/DRR016129/DRR016129_1.fastq.gz data/DRR016129/DRR016129_2.fastq.gz'
rna_files['DRR016130']='data/DRR016130/DRR016130_1.fastq.gz data/DRR016130/DRR016130_2.fastq.gz'
rna_files['DRR016131']='data/DRR016131/DRR016131_1.fastq.gz data/DRR016131/DRR016131_2.fastq.gz'
rna_files['DRR016132']='data/DRR016132/DRR016132_1.fastq.gz data/DRR016132/DRR016132_2.fastq.gz'
rna_files['DRR016133']='data/DRR016133/DRR016133_1.fastq.gz data/DRR016133/DRR016133_2.fastq.gz'
rna_files['DRR016134']='data/DRR016134/DRR016134_1.fastq.gz data/DRR016134/DRR016134_2.fastq.gz'
rna_files['DRR016135']='data/DRR016135/DRR016135_1.fastq.gz data/DRR016135/DRR016135_2.fastq.gz'
rna_files['DRR016136']='data/DRR016136/DRR016136_1.fastq.gz data/DRR016136/DRR016136_2.fastq.gz'
rna_files['DRR016137']='data/DRR016137/DRR016137_1.fastq.gz data/DRR016137/DRR016137_2.fastq.gz'
rna_files['DRR016138']='data/DRR016138/DRR016138_1.fastq.gz data/DRR016138/DRR016138_2.fastq.gz'
rna_files['DRR016139']='data/DRR016139/DRR016139_1.fastq.gz data/DRR016139/DRR016139_2.fastq.gz'
rna_files['DRR016140']='data/DRR016140/DRR016140_1.fastq.gz data/DRR016140/DRR016140_2.fastq.gz'

declare -a samples=( 'DRR016125' 'DRR016126' 'DRR016127' 'DRR016128' 'DRR016129' 'DRR016130' 'DRR016131' 'DRR016132' 'DRR016133' 'DRR016134' 'DRR016135' 'DRR016136' 'DRR016137' 'DRR016138' 'DRR016139' 'DRR016140' )

cd /home/adam.thrash/training/

ml singularity salmon/1.3.0--hf69c8f4_0

SAMPLE=${samples[$SLURM_ARRAY_TASK_ID]}
read -a FILES <<< ${rna_files[$SAMPLE]}

salmon quant -i athal_index -l A \
       -1 ${FILES[0]} \
       -2 ${FILES[1]} \
       -p $SLURM_CPUS_PER_TASK --validateMappings \
       -o quants/$SAMPLE

• set up RNA files
• set up sample names
• change to the correct directory
• load singularity and salmon
• get sample from slurm_array_task_id
• get reads from rna_files
• run salmon

sbatch salmon-run.sh

Submitted batch job 292484

squeue --me

     JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
  292484_0     atlas salmon_r adam.thr  R       0:03      1 Atlas-0047
  292484_1     atlas salmon_r adam.thr  R       0:03      1 Atlas-0051
  292484_2     atlas salmon_r adam.thr  R       0:03      1 Atlas-0060
  292484_3     atlas salmon_r adam.thr  R       0:03      1 Atlas-0067
  292484_4     atlas salmon_r adam.thr  R       0:03      1 Atlas-0074
  292484_5     atlas salmon_r adam.thr  R       0:03      1 Atlas-0076
  292484_6     atlas salmon_r adam.thr  R       0:03      1 Atlas-0079
  292484_7     atlas salmon_r adam.thr  R       0:03      1 Atlas-0083
  292484_8     atlas salmon_r adam.thr  R       0:03      1 Atlas-0131
  292484_9     atlas salmon_r adam.thr  R       0:03      1 Atlas-0136
 292484_10     atlas salmon_r adam.thr  R       0:03      1 Atlas-0144
 292484_11     atlas salmon_r adam.thr  R       0:03      1 Atlas-0146
 292484_12     atlas salmon_r adam.thr  R       0:03      1 Atlas-0149
 292484_13     atlas salmon_r adam.thr  R       0:03      1 Atlas-0152
 292484_14     atlas salmon_r adam.thr  R       0:03      1 Atlas-0156
 292484_15     atlas salmon_r adam.thr  R       0:03      1 Atlas-0158

Getting Help

• HPC² / Mississippi State University: help-usda@hpc.msstate.edu
• SCINet Virtual Research Support Core - scinet_vrsc@usda.gov
• SCINet Quick Start - https://scinet.usda.gov/guide/quickstart
• Atlas Documentation - https://www.hpc.msstate.edu/computing/atlas/

Questions

References

1. Understanding physical and logical cpus. URL: https://www.linkedin.com/pulse/understanding-physical-logical-cpus-akshay-deshpande.
2. Introduction to high performance computing for supercomputing wales: supercomputing wales introduction. URL: https://supercomputingwales.github.io/SCW-tutorial/01-HPC-intro/.
3. How to log in and transfer files with globus. URL: https://docs.globus.org/how-to/get-started/.
4. Getting started. URL: https://combine-lab.github.io/salmon/getting_started/.