Storage Overview & Usage Guidelines

Managing the data that we generate on DeepThought is one of the biggest challenges that we face, particularly as computational pipelines and software becomes more and more advanced. Genomics pipelines in particular frequently generate terabytes of data while running and we both need some way of hosting that and some way of getting rid of the unwanted data once they’re finished.

The HPC is a shared resource available to all Researchers at Flinders but we cant store everybody’s data for all time as there just isn’t enough space (particularly as the servers were procured in late 2025 after storage availability and pricing started becoming an issue).

This page details the configuration of the storage on DeepThought, how we expect you to use it and commentary on the intentions behind some of the design choices that we made given the resourcing constraints we had.

Danger

The data drives on DeepThought are all considered ‘volatile’ and are not backed up so if you accidentally delete data or we experience a hardware failure then we won’t be able to recover the data.

Note

The storage configuration of the legacy DeepThought cluster has been preserved in the second half of this page and will remain available while both clusters are run side-by-side.

Storage Network

Each of the nodes has a dedicated pair of bonded 25Gbps Network cards that provide bandwidth solely for storage traffic. The bonded pair is configured with both RDMA support and Jumbo frames allowing for significantly faster file transfer speeds than a regular IP network.

The following traffic on the cluster uses the storage network:

  • Network File System mounts (/home/hpc and /software)

  • Clustered File System traffic (/data and /cluster)

Note

Copy data to/from the R/T Drives or using SCP/SFTP to copy data to or from the HPC will be performed via the regular IP network that the HPC nodes also have and will be slower than copying data internally on the cluster

Network File System Storage

User home directories and the tooling folders are both located on the storage disks in the Login Node and are shared to the rest of the cluster via the Network File System (NFS).

NFS Storage - HPC Tooling

3TB of the storage in the Login Node is allocated to the various Tooling used on the HPC, primarily mounted under /software.

This volume contains the following:

  • /software/automated - Software automatically built using EasyBuild

  • /software/manual - Software compiled manually by IDS Research Success

  • /software/modules - Module definition files for the Lmod module load system

  • /software/system - System software compiled outside of EasyBuild such as the EasyBuild bootstrap and Lmod binaries

Tip

The EasyBuild EasyConfigs are installed at /software/system/easybuild/easybuild/easyconfigs if you’re looking to build software for your own use

NFS Storage - Home Directories

The remaining 22TB of the disk in the Login Node is allocated to home directory storage (/home/hpc) which is also mounted on every compute node at the same location.

Each user on the HPC has a home directory that corresponds to their FAN (e.g. /home/hpc/demo1234) with a 50GB quota allocated to it.

Tip

You can view your current quota usage using the quota command

Our intention with the home directories is to for them to be used to hold any job submission scripts, job error/output logs and small software packages that you’ve compiled yourself.

Design Intention

We had ongoing issues with the unrestricted space usage in home directories on the previous cluster so we’ve decided to start with a restrictive quota to begin with since it’s more viable to add more later than try and remove it once it’s already there.

We are aware that packages for conda/pip and IDEs such as VS Code tend to use lots of space in the home directory without any user input. We’re looking at ways to potentially automate the cleanup of some of the caches that apps we know create but in the meantime we’d recommend keeping an eye on your usage.

Note

Home directories do not have any access time-based automatic cleanup present on them

Important

Due to security concerns, the permissions on your home directory folder are regularly reset so that only you can access them

Clustered Storage Overview

The majority of the storage in the new cluster is provided by physical hard disks (spinning spindle-based rather than flash storage) that are located in each of the compute nodes that make up the General partition. Each node has approximately ~25TB of usable storage in it which means we have approximately 400TB of space split between Dataset storage, User-based Scratch space and temporary working space for SLURM jobs.

We are using two different clustered file systems:

  • BeeGFS: Used to provide the scratch and temporary working space mounted under /cluster

  • Ceph: Used to provide the persistent data storage mounted at /data

BeeGFS takes advantage of the RDMA support on the storage network to provide extremely fast read/write speeds to allow jobs and pipelines to run at their maximum efficiency.

Ceph does not have quite the same level of performance as BeeGFS but is an object store with metadata support under the hood that has a Linux-like filesystem over the top of it. Ceph volumes have higher resilience compared to BeeGFS and we’re hoping that we can integrate the object store side of things into the updated Research data storage system.

10 nodes are allocated to BeeGFS (approximately 250TB) and 6 nodes (approximately 150TB) are allocated to Ceph.

Design Intention

Similar to the home drives, we had ongoing issues with large amounts of data being left in scratch directories on the previous cluster. After discussing with our partners at NCI and Pawsey we’ve decided that rather than quotas we’re going to implement stale data collection on files that haven’t been accessed for more than 28 days instead. This will allow us to continue to share the BeeGFS clustered system to provide maximum performance for both user-based /cluster/scratch and the SLURM job /cluster/jobs/<jobid> storage while also accommodating the need for large quantities of temporary storage.

While we acknowledge that experienced users of the HPC can work out ways to update the access time on all of the files in their scratch space and effectively store things forever, we’re going explicitly ask you not to do that. We are monitoring the individual usage and if it becomes obvious that your space usage is unreasonable due to working around our cleanup method then we will be discussing it with you directly. Repeated occurrences are likely to result in access to the cluster being revoked.

BeeGFS Clustered Temporary Storage - User Scratch Space

In addition to your home directory, a directory specific to your account is also created under /cluster/scratch/<your_fan>. This location is where we intend for you to temporarily stage larger files that you need to work on as part of your jobs (e.g. copy them from the R Drive or via SCP before you submit your jobs via SLURM).

A symbolic link of scratch (e.g. ~/scratch) will be automatically created in your home directory to your scratch directory. While in a SLURM job, you can also use the $SCRATCH variable to refer to your directory.

Important

The scratch space will automatically remove any files that haven’t been accessed after 28 days.

One thing to note is that if you’re rsyncing data into your scratch folder from off the cluster with the time preservation flags set you may find them purged if their last access time is greater than 28 days. We’re specifically looking at the access time (can be viewed via stat <filename>) rather than the modified time for the cleanup so as long as you’re actually using the files they shouldn’t be purged.

Important

Due to security concerns, the permissions on your user-based scratch folder are regularly reset so that only you can access them

Design Intention

We’re aware that people were frequently modifying the permissions on their scratch (and to a lesser extent home) folders on the old HPC to share files with each other. We have a process for providing shared data storage for labs and projects so if you need to collaborate on data please see the Persistent Storage section for more details.

Warning

The rstudio-server package that we’re using in Open OnDemand makes some interesting assumptions about how it will be running and will set your working folder (which defaults to /cluster/scratch/<fan>) to be world readable and writable while it is running.

We’ve set up SLURM to automatically fix up the permissions on your scratch folder when it finishes but if you’re using the RStudio Web app through Open OnDemand please keep this in mind.

BeeGFS Clustered Temporary Storage - Temporary Job Working Space

Whenever you submit a job through SLURM, the prolog scripts that run before the job starts create a temporary working space for that job under /cluster/jobs/<job_number> and set the permissions so that only you have access to it.

Tip

You can use the $BGFS variable in your SLURM submission scripts to refer to the directory on a per-job basis. We suggest getting into the habit of changing into it at the start of your scripts so you don’t have to worry about cleaning up the temporary files generated during the jobs as SLURM does that for you after the job finishes

Like the user scratch folders, the working folders don’t have quota limits but instead are automatically cleaned up after each run to ensure that there’s enough space for everyone else to use.

Important

Please make sure you’re copy your outputs somewhere else as part of your scripts or you’ll lose them when the job finishes and the folder is removed

Note

The temporary job working space will automatically remove any files that haven’t been accessed after 28 days, although if everything is working well there shouldn’t be anything left to clean up after the job finishes

Ceph Semi-Persistent Storage - Datasets, Groups, Labs and Projects

In addition to the temporary storage on the clustered file system mentioned above, we are also providing persistent storage with enforced quotas for shared resources that will be used by multiple people. Our aim with this is to help reduce duplication of data and to improve the security by providing centrally managed access control.

Persistent storage will be available upon request for the following:

  • Datasets - Shared data that will be used by multiple areas that do not have a formal association

  • Groups - Working groups that are not necessarily part of the same Lab or Project but need some form of shared storage

  • Labs - Lab-specific storage

  • Projects - Storage for shared data focused on a specific Project

Tip

A report of the storage usage for each persistent storage subvolume is generated daily at 6am and placed inside each folder as ‘daily_storage_report.txt’ so you can get a relatively up to date idea of what the quota usage is

We have to individually maintain the group memberships used behind the scenes so access to this form of persistent shared storage will require a Data Management Plan (see the Library Research Data page for more info on DMPs), similar to getting a new R Drive created.

Design Intention

What we’re essentially trying to do here is shift the shared datasets that were scattered around scratch into centrally managed folders that can be shared by a lab or working group instead. At the same time, we’re also trying to reduce the amount of data that is sitting untouched on DeepThought as it is centrally funded and can only provide a fraction of the space that the R Drive does.

The new cluster has significantly better network connectivity than the old cluster so we’re intending for all research data other than temporary working files to be copied back out to R Drive after you’re done with them. The Research Storage refresh in 2026 has added the ability to purchase ad-hoc storage expansions for the R Drive which should hopefully help relieve the need to have as much internal storage in DeepThought.

If you need significantly more storage we encourage you and your College to work with the IDS Storage team to procuring additional space on the R drive rather than trying to store all data on DeepThought.

Warning

While the persistent storage file system isn’t subject to file cleanup like the rest of the clustered storage, it is still considered volatile. Please do not store the only copies of the data you have in this space as if something happens to it we will not be able to recover it

Note

Ceph is object store (which also has an S3 interface) rather than a traditional filesystem interface so in the future we’ll be looking to see what we can do with metadata and potential connectivity to other University systems.

Legacy Cluster Storage Tier Overview

The HPC is a little different that your desktop at home when it comes to storage, not just computing power. It’s a shared resource, so we cant store everybody’s data for all time - there just isn’t enough space.

So, before we start putting files onto the HPC, its best you know where to put them in the first place.

On DeepThought, are three main storage tiers. Firstly our bulk storage is the ‘Scratch’ area - and is slower, spinning Hard-Disk Drives (HDD’s). The next storage tier is the ‘Cluster’ parallel filesystem.

For distributed jobs, or jobs that required lots of staged files, this filesystem is the standard high-speed and low-latency HPC filesystem. Be aware that it is smaller that /scratch, and as such, is a transient filesystem. Finally, there are hyper-fast NVMe Solid-State Drives (SSD’s) located at /local. For the exact specifications and capacities, see the System Specifications.

We also integrate into R-Drive to allow you access to your Research Data Storage allocations. This is an automatic process for any shares you have access to.

There is a critical differences between these three locations, so please read this page carefully.

Legacy Cluster Storage Accessibility Overview

As general guide, the following table presents the overall storage for the HPC.

Filesystem Location Accessible From Capacity
/scratch All Nodes ~250TB
/cluster All Nodes ~41TB
/home All Nodes ~12TB
/local Individual Compute Nodes ~1TB (400GB on Node019)
/RDrive/\ Head Nodes Share Dependant

Warning

The HPC is classed as volatile storage. Your research data and datasets that you wanted backed up MUST be moved to /RDrive, or off the HPC.

Legacy Cluster Usage Guidelines

The following sections will go over the individual storage location/mounts along with some general guidelines of what should be stored where.

Legacy Cluster /scratch

Scratch is your working space and ‘Large Dataset Storage’ As you can see from the table above it have far more storage capacity than /home. If you need to store a large data-set, then it should live in here.

Legacy Cluster What to store in /scratch

Here is a rough guide as to what should live in your /scratch/$FAN directory. In general, anything large, bulky and only needed for a little while should go here.

  • Job Working Data-sets

  • Large Intermediate Files

Legacy Cluster /cluster

Cluster is the new, high speed parallel filesystem for DeepThought, deployed using BeeGFS. It is highly recommended that you take advantage of high speeds available to reduce the I/O times associated with /scratch - so please read this section carefully.

When performing parallel jobs, to prevent congestion, its best to have a random back-off to your job-scripts. This will take you stage-in times down from up to 30 minutes to less than 3 minutes. An example of this random backoff is below:

sleep $(echo $RANDOM%30 | bc)

The directories you can write to in /cluster are controlled by SLURM. When your job starts, SLURM sets multiple environment variables and creates directories for you to use on this filesystem. See the environment variables sections of the SLURM Guide for more information.

Once you job completes, is cancelled, or errors out, SLURM removes then entire directory of your job. That means, if you do not move your data from the /cluster filesystem, you will lose all of it. This is by design, and the HPC Team cannot recover any data lost this way.

When this quota is exceeded, files can still be written, but the HPC Team is notified of the user and their associated usage.

Legacy Cluster What to store in /cluster?

  • Your working data sets

  • Temporary job files

  • Results, before you copy them back to /scratch

Legacy Cluster /home

Your ‘home’ directories. This is a small amount of storage to store your small bits and pieces. This is the analogous to the Windows ‘Documents’ folder. At a command prompt, your home directory will be shortened to ~/.

Legacy Cluster What to store in /home

Here is a rough guide as to what should live in your /home/$FAN directory. In general, you want small, little things is here.

  • SLURM Scripts

  • ‘Small’ Results from Jobs

  • ‘Small’ Data-Sets (<5GB)

Legacy Cluster /local

Local is the per-node, high speed flash storage that is specific to each node.

Legacy Cluster What to Store in /local

Only transient files should live on /local. Anything that your job is currently working on can be on /local. Once your job has finished with these files, they should be copied (or moved) to /scratch. The directory you were working in on /local should then cleaned, removing all files from your job - is you use the automatic SLURM created directories, then this is done for you.