# Moving from Legacy DeepThought This document provides some guidance around potential changes and differences for people who had used the previous HPC cluster. ## Accessing the new HPC Direct access to the new HPC is still only available via on campus networks at this stage so you either need to be on site or connected via VPN. The web portal is also currently only accessible on campus but we're intending for that to be internet accessible eventually. You can access the new web interface via the `DeepThought HPC` button that should now be present in your Okta dashboard or directly via https://hpc.flinders.edu.au. There's a web-based Shell app in the new portal that you can use if you don't need to copy files. The web interface is also available via [Citrix VirtualApps](https://flinders.cloud.com) as 'DeepThought HPC' and is automatically published to anyone with access. The connection via VirtualApps is provided through Google Chrome hosted on our internal servers and can be accessed without requiring the VPN. Once the `deepthought.flinders.edu.au` name is moved to the new cluster you'll need to use the direct IP address to connect to the old cluster at `10.228.5.28`. ## New Web Portal One major new feature is we're using the Open OnDemand web portal system to allow for easier access to resources and to provide a convenient method to run interactive apps like Jupyter Notebooks. We've currently got Jupyter Notebooks, RStudio and VS Code working and integrated with the cluster along with the File browser and shell access. A few points to note about them: - Jupyter notebooks will cleanly shut down your SLURM job automatically if you shut down the notebook - RStudio's `rstudio-server` package is setting your working directory (defaults to scratch) world readable/writable while it runs. We're automatically fixing the scratch permissions via the SLURM job scripts but if you used a different working directory you might need to fix them yourself - VS Code isn't cleaning up the SLURM job at the moment and you need to stop it from the `My Interactive Sessions` page or let it time out - The `/RDrive` and `/TDrive` mappings in the Files app are only available on the login node itself - the actual compute nodes don't have them mapped - The shell app will stay connected as long as you're actively using it but it'll disconnect you if you're idle for 30 minutes and has a maximum runtime of 8 hours (even if you're actively using it). `tmux` is the easiest way to keep sessions active :::{tip} The full documentation is still WIP but you can configure and run your own personal web apps through the portal. The official documentation for it is available on the [OOD support pages](https://osc.github.io/ood-documentation/latest/how-tos/app-development.html) and in our case all you need to do is make a `~/ondemand/dev/` folder to get started. The apps we already have deployed are in `/var/www/ood/apps/sys/` if you want to copy them as a starting point. ::: ## Compute Hardware We have 16 new compute nodes available in the general partition and each node has 64C/128T CPU and 768GB of RAM. The new web portal has a status page where you can get a rough idea of what the cluster occupancy is like: https://hpc.flinders.edu.au/pun/sys/dashboard/system-status The new server hardware is all located back in the main part of the network so the bandwidth bottlenecks should be a thing of the past. From our performance testing we were seeing 3-5Gbps pulling data off the R Drive and internally all of the clustered storage is now using RDMA so it's really snappy. ## Storage The storage and paths are a bit different compared to the previous HPC: ### Home Folders User home folders are now stored in `/home/hpc//` and have a 50GB quota applied to them. These are mounted via NFS on all compute nodes so your home directory is the same everywhere. ### Clustered Filesystem - User based Scratch storage is available on the new clustered filesystem mount point at `/cluster/scratch/`, is automatically symlinked via `~/scratch` and is also accessible as $SCRATCH in SLURM jobs - The SLURM working folders are created at `/cluster/jobs/` and can still be accessed through $BGFS inside your jobs - Group/Lab/Project Datasets have their own separate storage with hard quotas under `/data` which has been broken down by College as `/data/college/lab|group` (e.g. `/data/cmph/mog`) One thing we're doing that's different (but will be familiar to anyone who's used Pawsey) is automatically cleaning up files under /cluster that haven't been accessed in 28 days. The datasets directory and home directories do **not** have any cleanup on them. ## SLURM Jobs Everything's fairly similar to the current version of DeepThought with the exception that you'll need to specify an account when running jobs using `--account=""` as part of your job submission scripts. For Researchers, your account will either be `-users` (e.g `cmph-users` for Staff/Full Status Holders), `-hdr` (e.g. `cmph-hdr` for HDR Students) or `-group` (e.g. `cmph-mog` for the Multi-omics Group). Non-Researchers will be placed into either `nonresearch-users` (for Flinders Professional/Teaching Staff), `nonresearch-sponsored` (for contractor staff or sponsored external affiliates) or individual team areas similar to College Labs (e.g. `nonresearch-esp` for Education Success Platforms). The main partition that most people will use is the 'general' partition which you will need to specify as there isn't a default partition. The list of all the partitions currently available is below: | Description | Partition Name | Nodes | Default Run Time | Max Run Time | | ------------------------------ | -------------- | ----- | ----------------------- | ----------------------- | | General Compute (CPU) | general | 6 | 60 minutes (0-01:00:00) | 7 days (7-00:00:00) | | IDS System Support | hpc-support | all | Unlimited | Unlimited | | Open OnDemand Interactive Apps | ood | 3\* | 60 minutes (0-01:00:00) | 12 hours (0-12:00:00) | | Short Runtime Jobs | short | 3\* | 60 minutes (0-01:00:00) | 1 day (1-0:00:00) | | Teaching | teaching | 3\* | 60 minutes (0-01:00:00) | 12 hours (0-12:00:00) | SLURM Fairshare weights are what determines the priority of your jobs compared to everyone else currently submitting jobs and are primarily applied at the user-level based on the type. The SLURM fairshare has been assigned with priority for HDR Students (100%) > College Users (50%) / Non-Research Users (50%) > Sponsored Users (5%). Groups/Labs/Projects also have their own separate fairshare allocations which are counted towards when using the `-group` account rather than against your own personal allocation. This means that even if you've submitted a significant number of jobs for your own research and are having to wait you still have the option of running jobs for work in your group/lab through their specific account. One thing we did get working was the method Pawsey use to allow interactive sessions through the use of `salloc` so you can happily run an interactive terminal now without having to resort to `srun bash` like you previously did. You can submit an interactive job with a command similar to the following (e.g. 1 task, 12 CPUs, 20GB of RAM, general partition): `salloc -n 1 -c 12 --mem=20G --partition="general" --account="cmph-users"` Pawsey also has documentation on using interactive jobs [available on their wiki](https://pawsey.atlassian.net/wiki/spaces/US/pages/51925964/Job+Scheduling#Interactive-jobs) although in our case the -A/account flag is the College affiliation rather than a project code. ## Software We're still using the module load system from before and have the `foss/2025a` toolchain compiled as the baseline we're building everything from. There's quite a bit there already but all of the main module dependencies are hidden to help prevent the module avail list from being unusable. You can view all the modules with `module --show-hidden avail` or use module spider to search for them. All of the software and modules are stored under `/software` (similar to Pawsey except without the Project part) and if you've got your own in-house stuff you've written we can work with you to turn it into a loadable module.