slideshow 3


March 26, 2019, Tuesday, 9am-5pm: Minerva full maintenance
This PM is scheduled to perform network modification and preparation to make Chimera open to all users. This will be a full maintenance, i.e., no jobs will be running, no login access to Minerva, and no access to the file system.

Chimera will be open to all on April 1, 2019!!
The Chimera compute partition is scheduled to open to all users on April 1, 2019. We are still configuring our cluster but friendly users will be able to access Chimera after the PMs. Please stay tuned for the announcements.

Spring Town Hall, April 1st, Monday, 1PM-3PM, HESS Seminar Room A
We will discuss the new partition, Chimera, and introduce the new environment. We also hope to have an update on the S10 BODE2 proposal.

TSM files expire in 2019
The TSM files that were archived in 2013 will expire in 2019. A reminder email will be sent to users 3 months ahead. There is no simple command to extend the file expiration date. If you would like to keep these files, you will need to retrieve them to disk and archive again. If you would like to query other users’ archived files, please send in a ticket.

Amazon AWS transfers from Minerva
If you are copying large amounts of data to or from Amazon AWS, then please keep an upper limit of 100MB/s via your AWS CLI configuration. Minerva is a shared environment and aggressive uploads from some processes is bound to throttle performance for everyone else.

V100 GPU nodes available in Minerva
5 new GPU nodes with a total of 20 V100 GPU cards are available in Minera. Currently, We have Centos7.6/Centos7.5 installed on all the GPU nodes, with nvidia driver version 410.79 which supports CUDA 10.0.

To access GPU node through LSF, please use following the command :
$ bsub -n 1 -P acc_xxx -q gpu -R "v100" -R "rusage[ngpus_excl_p=1]" -W 0:05 echo "Hello World"
$ bsub -n 1 -P acc_xxx -q gpu -R "p100" -R "rusage[ngpus_excl_p=1]" -W 0:05 echo "Hello World"

Chimera status update
Lenovo completed reliability test and HPL performance test. Chimera is able to achieve a peak performance of 833.3TF with 69.8% efficiency. HPC team is building OS image and setting up environment. Currently, 5 GPU nodes are made available in the Minerva partition.

Minerva user survey results
The results for user survey is available at
The response to the comments is available at

Slides for Minerva Town Hall meeting
The slides for Minerva Town Hall meeting are available on our website: (MUG Meeting Nov 29, 2018).

Highlights from the meeting:

  • Minerva usage status and our accomplishments in 2018
  • Main upgrade plans for 2019:
    • New compute partition Chimera: specs and installation plan
    • Storage upgrade: GPFS upgrade to 5.x and new filesystem /sc/hydra
    • OS upgrade, package rebuild and new module system
    • Queue structure changes
    • Container support

New compute partition: Chimera
We are happy to announce that we purchased a new compute partition Chimera. It will be available to users in early 2019. Specs of the new compute partition:

4x login nodes - Intel 8168 24C, 2.7GHz - 384 GB memory
280+ compute nodes - Intel 8168 24C, 2.7GHz - 192 GB memory
4x high memory nodes - Intel 8168 24C, 2.7GHz - 1.5 TB memory
48 V100 GPUs in 12 nodes - Intel 6142 16C, 2.6GHz - 384 GB memory - 4x V100-16 GB GPU
10x gateway nodes

New NFS storage (for users home directories) - 192 TB raw / 160 TB usable RAID6

Mellanox EDR Infiniband fat tree fabric (100Gb/s)

Total system memory (computes + GPU + high mem) = 65.7 TB
Total number of cores (computes + GPU + high mem) = 14,304 cores
Peak performance (computes + GPU + high mem, CPU only) = 1.2 PFlops/s

We will send out necessary announcement during the integration and make this transition seamless for all users.

We do not backup user data. It is the users' responsibility to ensure data is backed up in accordance with their funding agency or private organization requirements. Refer to "Preserving Data: Who’s Got Your Back(up)?" for helpful hints in developing your individual backup strategy.

All Announcements/Reminders.

Theme by Danetsoft and Danang Probo Sayekti inspired by Maksimer