# Running Jobs The main function of DeepThought is to allow you to run your software with allocations of CPU, memory and graphics cards that are considerably more than what you can get in a desktop or laptop. DeepThought is comprised of a number of servers working together and we use [SLURM](https://slurm.schedmd.com) (Simple Linux Usage Resource Manager) to manage requests to access those resources. ## SLURM and the 'Fairshare' allocation system SLURM on DeepThought uses the 'Fairshare' work allocation algorithm which works by tracking how many resource your job takes and adjusting your position in queue depending upon your usage. We've allocated Fairshare units to everyone by what you do at the University and in general the weighting is HDR Students (100%) > Staff/Status Holders (50%) > Sponsored Users (5%). The intention is to give HDR students the highest priority as they don't have other options while still allowing other Researchers to use resources as they are available. ## Accounting in SLURM In order to better distribute access to resources evenly, we've split up the Fairshare allocations by College (along with Non-Research as a separate pool) and then created accounts underneath each College by role. :::{important} Accounts are specified during SLURM submissions using the `--account=` switch. Jobs must have an account specified or they won't start ::: You can see what accounts you're a member of by using the `sacctmgr show association format=account%20 where user=` command:\ ![](../_static/RunningJobs/01-AccountMembership.png) :::{tip} Slurm only shows 8 characters per field by default so the `%20` on account increases the length to 20 instead. You can add this to the end of any field name if you see a `+` instead of the full name ::: The College-based accounts are organised in the following structure: * College e.g. `cmph` > * HDR Students e.g. `cmph-hdr` > * Users e.g. `cmph-users` > * Groups which is listed as `cmph-groups` but is effectively a container rather than an actual account > > * Group/Lab/Project - e.g. `cmph-mog` :::{note} Labs, Projects and Research Groups have separate accounts with their own Fairshare pool equal to approximately 20 staff members worth of compute. This account can be used if you have work to do for your lab but you've already run enough jobs that you're having to wait a lot due to everyone else having higher priority ::: In addition to Colleges, there are also a few other account types used on DeepThought: * Staff and sponsored users that fall outside of Research will be placed into `nonresearch` (either under `nonresearch-users` or `nonresearch-sponsored`) * In some cases Non-Research can have will have the equivalent of a Group/Lab/Project allocation created under `nonresearch-groups` * Students that have access to the HPC as part of certain topics will be placed under `teaching` and sorted by College e.g. `cse-teaching` The full fairshare structure can be viewed with the `sshare` command:\ ![](../_static/RunningJobs/02-AccountHeirarchy.png) ## Partitions in SLURM SLURM uses partitions to differentiate between groups of similar resources. In our case, we split the partitions up by function e.g. `general` for General computing tasks that only use the CPU and `high-capacity` for jobs that require significantly more memory :::{important} Partitions are specified during SLURM submissions using the `--partition=` switch. Jobs must have a partition specified or they won't start ::: The partitions that are currently available are listed below: | Description | Partition Name | Nodes | Default Run Time | Max Run Time | | ------------------------------ | -------------- | ----- | ----------------------- | ----------------------- | | General Compute (CPU) | general | 5 | 60 minutes (0-01:00:00) | 7 days (7-00:00:00) | | IDS System Support | hpc-support | all | Unlimited | Unlimited | | Open OnDemand Interactive Apps | ood | 3\* | 60 minutes (0-01:00:00) | 12 hours (0-12:00:00) | | Short Runtime Jobs | short | 3\* | 60 minutes (0-01:00:00) | 1 day (1-0:00:00) | | Teaching | teaching | 3\* | 60 minutes (0-01:00:00) | 12 hours (0-12:00:00) | \* The `ood`, `short` and `teaching` queues all share the same nodes :::{note} Research groups that have contributed hardware to the DeepThought Cluster will have their own partitions that are exclusively available for their use. These partitions will have a higher priority than jobs running on the same hardware using the regular queues e.g. jobs submitted to `mcoote` will have priority over `high-capacity` jobs ::: ## Requesting System Resources There are several different resources you can request as part of your SLURM jobs, either as command line switches or as #SBATCH directives in your scripts: * Parallel Tasks (`-n ` or `--ntasks=`: The number of parallel process tasks to split the job into. Unless you're using software that supports OpenMPI or inter-node communication you can leave this at 1 * CPUs Per Task (`-c ` or `--cpus-per-task=`): The number of CPUs to assign to each task (equivalent to CPU Threads). If you're using 4 tasks and you ask for 8 CPUs per task you'll end up using 32 CPUs overall. * Memory Per CPU (`--mem-per-cpu=`): The amount of RAM to allocate per CPU core (e.g. `8G` for 8 GB of RAM). Mutually exclusive with `--mem` and allocates the memory alongside the CPU so can be spread across multiple nodes * Memory Per Job (`--mem=`): The amount of RAM to allocate to the overall job in a contiguous block (e.g. `8G` for 8 GB of RAM). Mutually exclusive with `--mem-per-cpu` and forces the memory to all be allocated on a single node * Nodes Per Job (`-N ` or `--nodes=`): The number of worker nodes that the job can run on and will automatically be set to 1 if omitted. Unless you're using software that supports OpenMPI or inter-node communication you can leave this at 1 * Maximum Run Time (`-t D-HH:MM:SS` or `--time=D-HH:MM:SS`): The amount of time the job can run for. Each queue has its own limit and entering a time past the max time per queue will return a `Requested time limit is invalid (missing or exceeds some limit)` warning and prevent the job from being submitted * GPU/Generic Resources (`--gres="gpu:0"`): The number of a type of generic resource (such as GPUs) to assign. If you're not requesting GPU resources you can leave this out See the [Putting it all together](#putting-it-all-together) section for a practical example of how to use these switches to request resources. In general, we recommend starting with the approximate amount you think you'll actually need rather than asking for lots and working back down. It is significantly easier for SLURM to schedule in smaller chunks of resources than it larger ones and you may be waiting longer if you request more than you actually need. :::{note}::: If you request GPU resources on a queue that doesn't have `-gpu` on the end of it the job submission will be rejected ::: ## Interactive SLURM Sessions If you're still working through what you actually need to do for your final SLURM script you may want to conduct a dry run and debug the process in real time. SLURM allows you to use the `salloc` command to request resources attached to an interactive shell session with the same environment that you'd be working in through a batch script. You can start a very basic interactive session using the following command: `salloc -n 1 -c 4 --mem=8G --partition=short --account=` You'll get some feedback from SLURM that you didn't include settings such as the Nodes per job or a maximum run time but it'll fill those in for you (at one node and 60 minutes, respectively) and launch the session anyway. Once the session starts you'll see the server name in the terminal change from `hpc-login01p` to the respective compute node that you've connected to e.g. `hpc-comp001p`. :::{tip} Interactive shell sessions work particularly well with a terminal multiplexer like `tmux`. Just remember to start the multiplexer *before* you run the `salloc` command ::: The command below should include everything to get a 2 hour session started without SLURM having to automatically adjust your input command in the background: `salloc -N 1 -n 1 -c 4 --mem=8G -t 0-02:00:00 --partition=short --account=` :::{note} We intend for people to use the `short` partition to run interactive jobs but if needed you can also run them on any partition other than `general`. If you submit an interactive job to `general` it will be moved to `short` automatically ::: ## Putting it all together Once you've worked out which account you're using, which partition you're going to run the job on, what resources you need and what you're actually going to run you can combine it together into a script and then submit it to SLURM to actually run your job. :::{tip} Some example scripts are below, you can save them as a `.sh` file (e.g. `example.sh` in your home folder on DeepThought and then use the `sbatch` command to submit them. ::: If you saved your script as `example.sh` in a `slurm` folder in your home directory, you could use the following command to submit your job: `sbatch ~/slurm/example.sh` ``` #!/bin/bash # An example script for a basic job submission. See https://deepthoughtdocs.flinders.edu.au/SystemInformation/SLURM.html for more detail on what can go in a SLURM job. # Lines with #SBATCH are SLURM Batch commands and ##SBATCH is a commented out SLURM Batch commands to use as an example # The additional comments starting with # to the right of the commands are not necessary and are for descriptive purposes only # General SLURM job parameters #SBATCH --job-name=_example # A descriptive name for the job #SBATCH --mail-user=@flinders.edu.au # The email address to send notifications to #SBATCH --mail-type=ALL # Which notifications emails should be sent for (ALL,END,FAIL,NONE) #SBATCH --output=/home/hpc//%x-%j.out.txt # Where to save the console output from the job #SBATCH --error=/home/hpc//%x-%j.err.txt # Where to save the console error output from the job # SLURM resource parameters #SBATCH --partition= # Request that the job is running on a specific partition e.g. general #SBATCH --account= # Replace with an account you have access to e.g. cmph-users #SBATCH --time=0-01:00:00 # The maximum time for the job in Days-Hours:Minutes:Seconds format #SBATCH --nodes=1 # How many physical computers can this be run on (defaults to 1 if not specified) #SBATCH --ntasks=1 # How many parallel compute tasks to run for OpenMPI (defalts to 1) #SBATCH --cpus-per-task=1 # How many CPUs to use (the equivalent of CPU Threads rather than Cores) #SBATCH --mem-per-cpu=1G # How much memory should be allocated to each CPU (exclusive with --mem) ##SBATCH --mem=1G # How much memory should be allocated in total (forces all to be allocated on one node) #SBATCH --gres="gpu:0" # How many GPU resources to use # Load modules needed for the job module purge # Clean out any currently loaded modules module load R # Substitute for the packages you want to load # Change to temporary working directory in /cluster/jobs/ cd $BGFS # Copy files from your home or scratch to work on cp $SCRATCH/workingfolder $BGFS # Run whatever commands you need the script to do echo 'Hello, world!' > example.txt # Copy results (excluding the helper symbolic links we automatically create) back out to scratch in a new folder under the Job ID as the temporary folder gets cleaned up after the job runs mkdir -p $SCRATCH/$SLURM_JOB_ID rsync -avh --no-links $BGFS/ $SCRATCH/$SLURM_JOB_ID/ ``` The script above with the commentary removed and example data entered: ``` #!/bin/bash # General SLURM job parameters #SBATCH --job-name=test1234_example #SBATCH --mail-user=test1234@flinders.edu.au #SBATCH --mail-type=ALL #SBATCH --output=/home/hpc/test1234/%x-%j.out.txt #SBATCH --error=/home/hpc/test1234/%x-%j.err.txt # SLURM resource parameters #SBATCH --partition=general #SBATCH --account=cmph-users #SBATCH --time=0-01:00:00 #SBATCH --nodes=1 #SBATCH --ntasks=1 #SBATCH --cpus-per-task=1 #SBATCH --mem-per-cpu=1G #SBATCH --gres="gpu:0" # Load modules needed for the job module purge module load R # Change to temporary working directory cd $BGFS # Copy files from scratch cp $SCRATCH/workingfolder $BGFS # Run commands echo 'Hello, world!' > example.txt # Copy results back to scratch mkdir -p $SCRATCH/$SLURM_JOB_ID rsync -avh --no-links $BGFS/ $SCRATCH/$SLURM_JOB_ID/ ``` ## Monitoring Job Progress If you've used the templates above you'll receive an email notification when the job starts and when it completes either through successfully ending or failing. :::{tip} The email notifications on completion include a summary of how many resources were actually used so you can adjust your resourcing requests for similar tasks in the future ::: You view the progress of your active jobs using the `squeue -u ` command. You can also filter for specific jobs based on their status (e.g. `RUNNING` or `PENDING`) using the `-t` flag e.g. `squeue -u test1234 -t RUNNING`.