AirLLM

1. What is AirLLM?

AirLLM is a Python library that lets very large AI language models (70 billion parameters and up) run on a computer whose graphics card has only 4 GB of memory. Models of that size normally need expensive server hardware. AirLLM gets around this by never loading the whole model at once: it splits the model into layers, stores them on disk, and moves one layer at a time onto the graphics card to do the maths.

To be clear up front: this is a tool for people who can write a little Python. AirLLM has no chat window, no installer and no buttons. You use it by writing a few lines of code. The library is free and open source (Apache-2.0 licence), and the models are downloaded free of charge from Hugging Face (a public library of AI models).

The price you pay is speed. Every layer has to be read from disk, so answers arrive far more slowly than in a normal chat app.

A good fit if:

• You know some Python and want to try a very large model on your own machine.
• Your work is not urgent: summarising documents, experiments, jobs you leave running overnight.
• You just want a look without installing anything: the repo ships example notebooks that run on Google Colab.

Not a good fit if:

• You want an app where you type a question and get an instant reply.
• You only have a phone.
• Your disk is nearly full: large models take tens to hundreds of gigabytes.

2. What you need

Item	Details
Computer	A machine with an NVIDIA graphics card (the sample code targets NVIDIA), or a Mac with Apple Silicon. The project states that on macOS only Apple Silicon is supported.
Graphics memory	Depends on the model. The project's table lists roughly 1–2 GB for 8B-class models, about 4 GB for Llama 3.x 70B, about 8 GB for Llama 3.1 405B and about 12 GB for DeepSeek-V3.
Disk space	As much as you can spare. On the first run AirLLM downloads the original model, then splits it into layers and saves a second copy. Running out of space is the number one cause of errors.
Internet	A stable connection for the first download of a model.
Hugging Face account	Optional. Only needed for gated (restricted) models, which require a free access token from https://huggingface.co

Software to install first:

• Python: the programming language AirLLM runs on. Download it from https://www.python.org/downloads/ . On Windows, tick "Add python.exe to PATH" in the installer.
• PyTorch: the AI maths library AirLLM depends on (version 4.0.0 requires PyTorch 2.4 or newer). To use an NVIDIA card, pick the CUDA build as described at https://pytorch.org/get-started/locally/
• Mac only: Apple's mlx library, see https://github.com/ml-explore/mlx

Note: The official documentation only describes Linux and macOS. The install command is the same on Windows, but the project has no Windows-specific instructions.

3. Step-by-step installation

Option A: Try it on Google Colab with nothing installed

Google Colab is a Google service that lends you a computer with a graphics card and runs code inside your browser. It is the gentlest way to see AirLLM working.

Step 1: Open the project's example notebook

Open this address and sign in with a Google account:

https://colab.research.google.com/github/lyogavin/airllm/blob/main/air_llm/examples/run_all_types_of_models.ipynb

You should see a page made of code cells, each with a triangular run button on its left.

Step 2: Run the cells in order

Click the run button on the first cell, wait for it to finish, then move to the next. The install cell prints many lines of text; the cells below it download a model and print its answer.

Note: The free tier of Colab limits disk space and session time, so it only suits smaller models. The exact limits are set by Google; check the Colab site.

Option B: Install on your own computer

Step 1: Check Python

python --version

You should see a line such as Python 3.x.x. On macOS and Linux, if that fails, try python3 --version.

Step 2: Install AirLLM

pip install airllm

Your machine downloads AirLLM and the libraries it needs. It worked if the last line contains Successfully installed and airllm appears in the list.

Step 3 (Mac only): Install mlx

pip install mlx

The project also notes that you should be using a native Apple Silicon build of Python rather than an Intel-emulated one.

Step 4: Confirm the install

pip show airllm

You should see Name: airllm and a Version: line (the latest release at the time of writing is 4.0.0, published on 5 September 2026).

4. First use

You will run a small Python file that asks a model one question and prints the answer.

Step 1: Create the script

Create a file called try_airllm.py in Notepad or any text editor and paste in the following. It is a trimmed version of the example in the project's documentation:

from airllm import AutoModel

MAX_LENGTH = 128
model = AutoModel.from_pretrained("Qwen/Qwen3-32B")

input_text = ['What is the capital of United States?']

input_tokens = model.tokenizer(input_text,
    return_tensors="pt",
    return_attention_mask=False,
    truncation=True,
    max_length=MAX_LENGTH,
    padding=False)

generation_output = model.generate(
    input_tokens['input_ids'].cuda(),
    max_new_tokens=20,
    use_cache=True,
    return_dict_in_generate=True)

output = model.tokenizer.decode(generation_output.sequences[0])
print(output)

The AutoModel.from_pretrained(...) line is where you choose the model: replace the name in quotes with another model ID from Hugging Face.

Step 2: Run it

Open a command line in the folder that holds the file and type:

python try_airllm.py

Step 3: Wait, then read the result

The first run takes a long time because the model has to be downloaded and split into layers. You will see progress bars. When it finishes, the screen shows your question followed by the text the model wrote. Later runs are quicker because nothing is downloaded again.

Tip: go faster and save disk space

The project reports that turning on model compression can speed things up by as much as three times with almost negligible loss of accuracy. Install the compression library:

pip install -U bitsandbytes

Then change the model line to:

model = AutoModel.from_pretrained("Qwen/Qwen3-32B", compression='4bit')

Other options go in the same place:

Option	What it does
compression	'4bit' or '8bit'; off by default.
delete_original=True	Deletes the original download after splitting, saving about half the disk space.
layer_shards_saving_path	Saves the split layers in a folder you choose, for example on a drive with more room.
hf_token	Your Hugging Face token, for gated models.
profiling_mode=True	Prints how long each stage takes.

5. Common errors and fixes

Symptom	Cause	Fix
SafetensorError: Error while deserializing header: MetadataIncompleteBuffer	Almost always the disk filled up while the model was being split.	Free up space, clear the Hugging Face cache folder and run again. Consider delete_original=True or moving the output with layer_shards_saving_path.
ValueError: max() arg is an empty sequence	A Qwen or ChatGLM model was loaded with the old AirLLMLlama2 class.	Use from airllm import AutoModel and AutoModel.from_pretrained(...), which detects the model type for you.
401 Client Error ... Repo model ... is gated	The model is restricted and needs permission.	Request access on the model's Hugging Face page, create a token, then add hf_token='your-token' when creating the model.
ValueError: Asking to pad but the tokenizer does not have a padding token	The model has no padding token.	Set padding=False in the model.tokenizer(...) call, as in the example in section 4.
pip or python "is not recognized" / "command not found"	Python is missing or not on PATH.	Reinstall Python with "Add python.exe to PATH" ticked; on macOS/Linux try pip3 and python3.
An error containing Torch not compiled with CUDA enabled	Your PyTorch build has no NVIDIA support, or the machine has no NVIDIA card.	Reinstall the CUDA build of PyTorch from https://pytorch.org/get-started/locally/ . The project has supported CPU inference since version 2.10.1; see the repo for details.
A new model complains about a missing class or is not recognised	Some models need a specific transformers version (for example Qwen3.8-27B needs 5.8 or newer, Kimi K3 needs 4.56.x).	Read the "Updates" section of the repo's README for each model's own requirements.

6. Uninstalling and updating

Update to the newest version:

pip install -U airllm

Remove AirLLM:

pip uninstall airllm

Removing the library does not delete the models you downloaded, and those are what fill the disk. They live in the Hugging Face cache, which by default is the .cache/huggingface/hub folder inside your user folder. Delete the subfolders whose names start with models-- to get the space back. If you ever set layer_shards_saving_path, delete that folder too.

7. Frequently asked questions

Does AirLLM cost anything? No. The library is free and open source. You pay only in electricity, disk space and time.

Do I need an internet connection? Yes for installing and for downloading a model the first time. The computation itself happens on your machine, and you can point AirLLM at a model folder already on disk instead of a model ID.

Is the quality of the answers reduced? In its default mode AirLLM does not compress or trim the model, so the model is unchanged. Compression only happens if you switch on compression, and the project says the accuracy loss is close to negligible.

Will it run on a modest computer? A small graphics card is fine; that is the whole point. You do need plenty of disk space and patience, because the bottleneck is reading each layer from disk.

I cannot code. What should I do? Try Option A (Google Colab) to see what it is like. If what you want is to chat with an AI on your computer, AirLLM is not the right choice because it has no interface.