Colibri is proof that you don’t need a datacenter to talk to a frontier-scale model — you need a laptop, a fast disk, and very disciplined memory usage.
The project runs GLM-5.2, a Mixture-of-Experts model with 744 billion parameters, on a consumer machine with just ~25GB of RAM. No cluster, no rented H100s, no Mac with 256GB of unified memory. Just a C file doing something smart with how MoE models actually use their weights.
The Idea
A 744B MoE model doesn’t touch all 744B parameters on each token. GLM-5.2 activates only ~40B parameters per token — and of those, only ~11GB change from one token to the next (the routed experts). The rest is structurally constant.
Colibri splits the model exactly along that line:
- The dense part — attention, shared experts, embeddings, ~17B params — stays resident in RAM as int4 (~9.9GB).
- The 21,504 routed experts (75 MoE layers × 256 experts, plus the MTP head, ~19MB each as int4) live on disk (~370GB) and are streamed on demand, backed by an LRU cache per layer, an optional pinned hot-store, and the OS page cache as a free second layer.
The entire engine is a single C file, c/glm.c, about 1,300 lines, plus a couple of small headers. No BLAS, no Python at runtime, no GPU.
What’s Implemented
This isn’t a toy reimplementation. The forward pass in the repo — glm_moe_dsa — is validated token-exact against a transformers oracle (32/32 on teacher-forcing, 20/20 on greedy decoding against a tiny-random model built with the real architecture). Beyond the core forward pass:
- MLA attention with compressed KV-cache: 576 floats per token instead of 32,768 — a 57x reduction, because GLM-5.2 uses 64 heads without GQA.
- DeepSeek-V3–style router sigmoid (noaux_tc, routed_scaling_factor), shared expert, first-3-dense-layers pattern.
- Native MTP speculative decoding — GLM-5.2 comes with its own multi-token-prediction head, which drafts tokens that the main model verifies in a single batched forward pass. The author reports 2.0 tokens/forward with 100% acceptance on structured text, and it’s lossless: the output is identical to plain greedy decoding.
- Quantization kernels for int8/int4/int2 with per-row scales and AVX2, dequant-on-use.
- A byte-level BPE tokenizer written in C (GPT-2 style, Unicode-aware, 320k merges) — with no Python dependency even for tokenization.
- RAM safety: the expert cache auto-sizes itself from
MemAvailableon startup, so the OOM killer has no vote.
Numbers (author’s dev box: WSL2, 12 cores, 25GB RAM, NVMe via VHDX)
| Metric | Value |
|---|---|
| Model on disk (int4 container) | ~370GB |
| RAM resident (dense, int4) | 9.9GB |
| Load time | ~30s |
| Peak RSS during chat | ~20GB |
| Decode cost on cold | ~11GB of disk reads/token |
| Disk ceiling on this machine | ~1GB/s → ~0.05–0.1 tok/s cold |
| MTP speculation | 2.0 tok/forward |
Slow, by design and by hardware. The point isn’t throughput — it’s that a frontier-class 744B model responds correctly on a machine that costs less than a single H100 fan. The repo includes its own back-of-envelope projections for better hardware — setups with PCIe4/5 NVMe land somewhere between 0.5 and 4 tok/s, and a workstation with 128–256GB of RAM with a mostly-cached hot expert set lands in the 5-15 tok/s range, interactive territory. These are the project’s own estimates, not independent measurements, and it invites people to send real numbers.
Quickstart
cd c
./setup.sh # checks gcc/OpenMP, compiles, runs self-tests
# convert the model (resumable, needs ~400GB free on ext4/NVMe):
./coli convert # downloads from zai-org/GLM-5.2-FP8
# chat (RAM budget and expert cache auto-size):
COLI_MODEL=/path/to/glm52_i4 ./coli chat
There’s also a pre-converted int4 build on Hugging Face if you’d rather skip the FP8→int4 conversion (which otherwise streams and deletes 5GB shards one at a time, so the full 756GB checkpoint never has to exist all at once on disk).
Useful knobs: --topp 0.7 for adaptive expert top-p (30–40% less disk I/O), THINK=1 for GLM-5.2’s reasoning block, DRAFT=n for MTP draft depth, and STATS=f / PIN=f PIN_GB=g to log which experts you actually use and pin the hottest ones in free RAM — the engine gets faster the more you use it.
Where This Fits
Most local paths to GLM-5.2 assume serious hardware: Unsloth’s dynamic GGUF wants a Mac with 256GB+ of unified memory or a multi-GPU workstation to hit single-digit tokens per second. Colibri bets the other way — treat RAM as the scarce resource and let a fast NVMe carry the weight. It’s not a replacement for a well-resourced inference stack, but as a way to run a real frontier open-weights model on hardware you already have, it’s a genuinely different angle on self-hosted local LLMs — and worth trying on your own machine to see where it lands.
The project is Apache 2.0 (GLM-5.2 weights are MIT from Z.ai), built by a single developer, and is gaining traction on GitHub since launch.
Has anyone tried it on their own laptop yet? Tell us what tok/s you got and what hardware you used.