# Large Language Models 

## Large Language Models / Machine Learning Overview

DeepThought supports the use of [Large Language Models](https://en.wikipedia.org/wiki/Large_language_model) (LLMs) and general Machine Learning tools through SLURM on the compute nodes.

:::{warning}
Allowing agents in programs like VS Code on your local computer to connect to DeepThought via an SSH to use resources is not a supported configuration and will be resource constrained due to our use of control groups. 

Please note that the legacy cluster is not in a position to support connections from agents in coding IDEs at all and repeated use after warnings may result in your access to the legacy cluster being restricted.
:::

:::{important}
If you wish to use coding assistance models in VS Code we recommend using something like llama.cpp to spin up your own model on a compute node via SLURM and then connecting VS Code running through Open OnDemand to it.
:::

## LLMs via llama.cpp

One of the easiest ways to interact with an LLM on the HPC is to use an inference tool such as [llama.cpp](https://github.com/ggml-org/llama.cpp) to load a copy of the model into memory and provide an active interface to chat with the model.

We have several different optimised versions of `llama.cpp` available through the module load system to help facilitate either building your own models or chatting with existing open models (primarily sourced from [HuggingFace](https://huggingface.co)):
* `llama.cpp` - Optimised for CPU-based inference on AMD hardware
* `llama.cpp-cuda` - Optimised for GPU-based inference on NVIDIA CUDA-based hardware

:::{tip}
We have some [quantised](https://localllm.in/blog/quantization-explained) versions of popular models such as [Gemma](https://deepmind.google/models/gemma/) and [Qwen Coder](https://coder.qwen.ai) available to load centrally from `/software/models`
:::

:::{note}
LLMs function best on modern GPUs with high amounts of VRAM which we only have a limited number of in the `high-capacity` nodes that are also needed for other CUDA-enabled workflows. We suggest starting with smaller models in the CPU-optimised version of llama.cpp unless you specifically have a GPU requirement for your model.

If you have a long term requirement for models that have a large GPU VRAM footprint we suggest getting in touch with the [IDS Research Success](mailto:researchsuccess@flinders.edu.au) team as there are more appropriate options available through our partners outside the HPC such as the Dell's [Pro Max range](https://www.dell.com/en-au/shop/computer-pcs/lp/dell-pro-max-nvidia-ai-dev) or Apple's [Mac Studio](https://www.apple.com/au/mac-studio/)
:::

## Loading interactive models through SLURM and llama.cpp

The simplest method to getting started with a basic LLM is to load an [interactive SLURM session](../GettingStarted/RunningJobs.md#interactive-slurm-sessions) on the short queue, load the `llama.cpp` module and then load one of the models we have stored centrally.

:::{note}
We've done quite a bit of performance testing and the best ratio seems to be 8 CPU cores and 16GB of RAM for an interactive session. If you want to run up models and serve them out with the http server to use with something like plugins in VS Code from Open OnDemand the ratio shifts to 64 CPUs and 32-64GB of RAM
:::

1. Start an interactive SLURM session on the short queue:\
`salloc -N 1 -n 1 -c 8 --mem=16G --partition="short" --time=0-01:00:00 --account=<your_account_goes_here`

2. Load the CPU-optimised version of `llama.cpp`:\
`module load llama.cpp`

3. Load the Gemma 4 quantised model in interactive mode:\
`llama-cli -m /software/models/google_gemma-4-E2B-it-Q5_K_M.gguf`

:::{tip}
You can also directly ask your question using the `-p "Question"` switch in step 3 e.g. `llama-cli -m /software/models/google_gemma-4-E2B-it-Q5_K_M.gguf -p "Why is the sky blue?"`
:::

## Starting models with the API service

llama.cpp also includes an API service (which includes a web server) to allow external connections to interact with the running model, most commonly used for coding assistance through an IDE such as VS Code. This can be combined with the VS Code web app available in Open OnDemand to allow for o

The process for starting the model with the associated API service is very similar to the interactive session although it can be run through a regular SLURM batch submission job on the other compute nodes instead:\
`llama-server -m /software/models/qwen2.5-coder-7b-instruct-q5_k_m.gguf --port 0` 

:::{note}
`--port 0` in the above command will automatically bind to an available port on the system to listen for the API connections which we recommend instead of specifying one as the server will fail to start if the specific port is in use
:::

To determine the port that is being used you will need to ensure you have the `#SBATCH --output=/home/hpc/<FAN>/%x-%j.out.txt` line in your submission script and then check the output file to see where it launched after your job starts. Once you know the server name and port you can then point your coding assistance extension of choice at `https://hpc-comp<00N>p.it.ad.flinders.edu.au:<port>`.

:::{tip}
You can also combine the API server with Open OnDemand's [Reverse Proxy](OpenOnDemand.md#reverse-proxy-via-websockets) functionality to connect directly to the integrated web service although we've noticed that in testing there are a lot of CSS and JavaScript errors that often prevent the page from rendering correctly
:::

:::{note}
If you're familiar with using Reverse Tunnelling in SSH you can also create a double reverse tunnel using the Login node as a bridge to connect a coding IDE on your local computer to a model running in a SLURM job on a compute node in DeepThought. This isn't a configuration our team can provide any support for but we aren't disallowing this method and would prefer that you do this rather than trying to use coding agents via IDEs on the Login node directly
:::