Using Local /scratch (TMPDIR) on Compute Nodes

All nodes (compute and development) have their own locally storage mounted as /scratch. The /scratch storage is fast - faster than system-wide storage such as home folders but also /wynton/scratch - which make it ideal for holding intermediate data files. This will also lower the load on the system-wide storage and the local network. Using local /scratch is a win-win for everyone.

Instructions

Here is how to use /scratch:

Example

Here is a script called ex-scratch.sh that illustrates how to copy input files over from the $HOME folder to the local scratch folder (TMPDIR) of whatever node the job ends up running on. After processing of the input files is complete, the output files are moved from the local scratch (TMPDIR) to HOME.

#!/bin/env bash
#$ -cwd             ## use current working directory
#$ -l scratch=200G  ## needs 200 GB of /scratch space

## 0. In case TMPDIR is not set, e.g. on development nodes, set
##    it to local /scratch, if it exists, otherwise to /tmp
if [[ -z "$TMPDIR" ]]; then
  if [[ -d /scratch ]]; then TMPDIR=/scratch/$USER; else TMPDIR=/tmp/$USER; fi
  mkdir -p "$TMPDIR"
  export TMPDIR
fi

## 1. Use a temporary working directory
cd "$TMPDIR"

## 2. Copy input files from global disk to local scratch
cp ~/sample.fq .
cp ~/reference.fa .

## 3. Process input files
/path/to/my_pipeline --cores="$NSLOTS" reference.fa sample.fq > output.bam

## 4. Move output files back to global disk
mv output.bam ~

## 5. End-of-job summary
[[ -n "$JOB_ID" ]] && qstat -j "$JOB_ID"

Assume that the total amount of local scratch you need for your input files and your output files and whatever intermediate files my_pipeline needs is 100 GiB, and assume that the process requires up to 8 GiB of RAM (=4 GiB per core) to complete. Moreover, let’s say you wish to run in parallel using two cores. Then you should submit this job script as:

$ qsub -l scratch=100G -l mem_free=4G -pe smp 2 ex-scratch.sh

To understand the purpose of qstat -j at the end, see the Job Summary page.