AirLLM: Run Huge LLMs on Small GPUs
AirLLM is an open-source Python library (lyogavin/airllm, Apache-2.0, also on PyPI) that dramatically cuts inference VRAM by keeping roughly one layer on the GPU at a time. The official README states you can run Llama-class 70B models on about 4GB, and lists even larger MoE models at hobbyist VRAM (e.g. DeepSeek-V3 ~12GB). Install with pip install airllm, then AutoModel.from_pretrained(...). Caveat: layer streaming is disk-heavy and often too slow for interactive chat — better for offline/batch jobs.
AirLLM at a glance
| AirLLM | |
|---|---|
| Repo | github.com/lyogavin/airllm |
| Install | pip install airllm (PyPI) |
| Licence | Apache-2.0 (README badge) |
| Trick | Stream one layer to the GPU at a time (VRAM ≈ layer size, not full model) |
| Not for | Real-time chat on a 4GB card (disk I/O dominates) |
Quick start (from the official README)
pip install airllm
from airllm import AutoModel
MAX_LENGTH = 128
model = AutoModel.from_pretrained("Qwen/Qwen3-32B") # any popular HF id
input_tokens = model.tokenizer(
["What is the capital of United States?"],
return_tensors="pt",
return_attention_mask=False,
truncation=True,
max_length=MAX_LENGTH,
padding=False,
)
generation_output = model.generate(
input_tokens["input_ids"].cuda(),
max_new_tokens=20,
use_cache=True,
return_dict_in_generate=True,
)
print(model.tokenizer.decode(generation_output.sequences[0]))
First run splits the checkpoint layer-wise under the Hugging Face cache — ensure lots of free disk. Set delete_original=True if you need to reclaim space after the split.
Vendor-reported VRAM table (README)
Figures below are from the AirLLM README (measured end-to-end claims by the project). Treat as project-reported, not Toolsiftly benchmarks.
| Model (examples in README) | Approx size | Reported GPU VRAM |
|---|---|---|
| ~8B dense | 8B | ~1–2 GB |
| Qwen3.8-27B dense VL | 27B | 3.33 GB |
| Llama 3.x 70B | 70B | ~4 GB |
| Qwen3.8-Flash-Next | ~125B MoE class | ~5.95 GB |
| DeepSeek-V3 | 671B | ~12 GB |
Speed-up: 4-bit / 8-bit compression
Per README v2+: install bitsandbytes, then:
model = AutoModel.from_pretrained("garage-bAInd/Platypus2-70B-instruct", compression="4bit")
The project describes this as reducing disk load size (weight-only block-wise quantization), aiming for up to ~3× faster inference versus uncompressed AirLLM — still not llama.cpp-class interactive speed on tiny GPUs.
Honest expectations
- Disk: a 70B fp16 checkpoint is on the order of ~140GB before split overhead.
- Speed: community write-ups often see many seconds to minutes per token when every layer streams from disk each step.
- Mac: README supports Apple silicon via MLX; Intel Macs are not covered the same way.
What this page does not do
- No hosted 70B demo (we cannot run that on the edge worker).
- No pirated model weights or gated-model key sharing — use your own Hugging Face token for gated repos (
hf_token=).
Frequently asked questions
Can AirLLM really run 70B on 4GB VRAM?
The official README says yes for Llama-class 70B by streaming one layer at a time. You still need large disk space, and generation is usually far too slow for chat.
How do I install AirLLM?
pip install airllm, then from airllm import AutoModel and AutoModel.from_pretrained("HF_ID").
Is AirLLM free?
Yes — Apache-2.0 open source on GitHub/PyPI. Model weights have their own licences.
AirLLM vs llama.cpp?
Different goals. AirLLM targets fitting huge HF models on tiny VRAM via layer streaming; llama.cpp/GGUF stacks usually win on interactive tokens/sec. Pick based on your workflow.