The Module System¶
In mordern time, almost all HPC systems are multiuser systems and consequently, this brings many challenges. One of such challenges is effectively managing a complex and large volume of software application cohort.
DeepThought HPC uses the LMod (Load MODules) system to load/unload applications in the command line. Any modules you use frequently can be loaded on login using your .bash_profile file; modules required for a job should be automated in your SLURM script.
Best way to think of Module is a singular program version + all of its associated dependencies to run correctly.
How Do I Install Software?¶
There are multiple ways to install software on the HPC. Below is an expansion on some of the common ones. The short and sweet version is that, if you compile/install it yourself to your /home or a Virtual Environment of some kind, you are free to do whatever you want!
The HPC support team will need to action your request if you need something big and complicated like ANSYS, GNU Octave, a new version of R or other similar large and complicated programs.
Python / Conda¶
The HPC Team allows you to install your own packages by using the inbuilt package manager tools, like Pythons ‘pip’, or Conda.
As an example, you can create a Conda Virtual Environment - this is under your complete control and you may install, remove or alter it as you wish. This is also the same for Pythons ‘venv’, which functions in much the same way.
The Conda Guide is located at: Conda Guide
The Python Guide is located at: Python Guide
Compile Your Own¶
The HPC uses the FOSS Toolchain, as detailed in the Fair Usage Policy. Should you wish to compile and use your own software, simply load the associated module (eg, foss-2020a) which will load up the associated GCC Toolchains and libraries.
My Toolchain isn’t Listed¶
Should you require a different Toolchain, like LLVM or Go and it is not listed under the
module avail list, you can either:
1.) Bootstrap the compiler + libraries yourself in your /home directory, managing it and any programs yourself
2.) Contact the HPC Support Team, either via Email or ServiceOne
As Software requirements for research can be very specific, the modules follow suit as well. Newer modules will have the following syntax:
To break that down into its individual parts for BioPerl/1.7.2-GCCcore-8.2.0-Perl-5.28.1 in the following sections.
Manually installed software will always be a shorter format of:
This is program that was installed — in this case it’s BioPerl, version 1.7.2
The Compiler Toolchain that was used to compile the program — in this case it’s GCCcore, version 8.2.0.
Some programs have a dependence on another interpreter, like Perl or Python. In this case it’s Perl, version 5.28.1. This means, it will also load the module for Perl, Version 5.28.1.
Below are some common module commands to load, unload and reset your module system.
Will get you a list of something similar to this - a list of every single available module on the HPC. You can scroll by your ‘Up’ and ‘Down’ arrows and ‘q’ will exit the scroll if you don’t want to scroll all the way to then end. The screenshot below is not an exhaustive list of all modules, just an example!
Will get you a list of your current modules.
There are three main ways to load a module. For most of the time, they are functionally equivalent. For more information, head on over to LMod Loading Types and read up on how they differ.
- A good tip! Typing out a partial name and double-tapping ‘tab’ will attempt to complete the name of what you are typing. This means you don’t have to worry about typing the very-long name of a module.
module load BioPerl/1.7.2-GCCcore-8.2.0-Perl-5.28.1.
There is also a nice shortcut, that you can use:
‘ml’ is just short for ‘module load’. Handy!
Additional Software & Modules¶
Generally speaking, we can install almost all Linux/CentOS bounded software/applications on HPC, but we don’t always need to go thorough the effort to install things ‘globally’ for everybody.
- Are people other than just me going to use this software?
- If yes, create an ServiceOne Ticket, and Digital Research Services will get it installed on a Best-Effort basis
Otherwise, there is nothing stopping you installing the program locally for yourself! If you run into issues installing software then open an ServiceOne ticket and again, Digital Research Services will help on a best-effort basis.
An Important Note¶
The software must in all cases be appropriately licensed.
Currently Installed Modules¶
Be warned, that this is a long list. It’s updated on best effort basis to help you see what is already present, but the latest list is always available by running the
module avail command on the HPC. Its broken into several segments, as there are several tools used to facilitate the installation and management of software used on the HPC.
Manually Installed Software / Default Available¶
This is the list of software that has been ‘hand rolled’ as it contains either things that at that time, did not have an automated way of installation, or rather esoteric software that required extensive modification to work correctly on the HPC. It is available here
There are additional software collections will be made available in a near future.