Cluster Specifications

Overview

Compute nodes
429
Physical cores
10428
GPUs
132 GPUs on 38 GPU nodes (24/7 GPUs/nodes are communal and 108/31 GPUs/nodes are prioritized for GPU contributors)
RAM
48-1512 GiB/node
Local scratch
0.1-1.8 TiB/node
Global scratch
492 TiB
User home storage
383 TiB (maximum 500 GiB/user)
Group storage
3.8 PB
Number of accounts
987
Number of projects
0

Software environment

All nodes on the cluster runs CentOS 7 which is updated on a regular basis. The job scheduler is SGE 8.1.9 (Son of Grid Engine) which provides queues for both communal and lab-priority tasks.

Hardware

Compute Nodes

The majority of the compute nodes have Intel processors, while a few have AMD processes. Each compute node has a local /scratch drive (see above for size), which is either a hard disk drive (HDD), a solid state drive (SSD), or even a Non-Volatile Memory Express (NVMe) drive. Each node has a tiny /tmp drive (4-8 GiB).

The compute nodes can only be utilized by submitting jobs via the scheduler - it is not possible to explicitly log in to compute nodes.

Login Nodes

The cluster can be accessed via SSH to one of two login nodes:

  1. log1.wynton.ucsf.edu
  2. log2.wynton.ucsf.edu

Data Transfer Nodes

For transferring large data files, it is recommended to use one of the dedicate data transfer nodes:

  1. dt1.wynton.ucsf.edu
  2. dt2.wynton.ucsf.edu

which both has a 10 Gbps connection - providing a file transfer speed of up to (theoretical) 1.25 GB/s = 4.5 TB/h. As for the login nodes, the transfer nodes can be accessed via SSH.

Comment: You can also transfer data via the login nodes, but since those only have 1 Gbps connections, you will see much lower transfer rates.

Development Nodes

The cluster has development nodes for the purpose of validating scripts, prototyping pipelines, compiling software, and more. Development nodes can be accessed from the login nodes.

Node Physical Cores RAM Local /scratch CPU GPU
dev1 8 16 GiB 0.11 TiB Intel Xeon E5430 2.66GHz  
dev2 32 512 GiB 1.1 TiB Intel Xeon E5-2640 v3 2.60GHz  
dev3 32 512 GiB 1.1 TiB Intel Xeon E5-2640 v3 2.60GHz  
gpudev1 12 48 GiB 0.37 TiB Intel Xeon X5650 2.67GHz GeForce GTX 980 Ti

Comment: Please use the GPU development node only if you need to build or prototype GPU software.

Scratch Storage

The Wynton cluster provides two types of scratch storage:

There are no per-user quotas in these scratch spaces. Files not added or modified during the last two weeks will be automatically deleted on a nightly basis. Note, files with old timestamps that were “added” to the scratch place during this period will not be deleted, which covers the use case where files with old timestamps are extracted from tar.gz file. (Details: tmpwatch --ctime --dirmtime --all --force is used for the cleanup.)

User and Lab Storage

Each user may use up to 500 GiB disk space in the home directory. Research groups can add additional storage space under /wynton/group by either mounting their existing storage or purchase new.

Network

The majority of the compute nodes are connected to the local network with 1 Gbps and 10 Gbps network cards while a few got 40 Gbps cards.

The cluster itself connects to NSF’s Pacific Research Platform at a speed of 100 Gbps - providing a file transfer speed of up to (theoretical) 12.5 GB/s = 45 TB/h.