BitNet is Microsoft’s family of 1-bit language models, and bitnet.cpp is the open source engine that runs them on a regular CPU, without a GPU and with a fraction of the usual memory. Its main model, BitNet b1.58 2B4T, takes up about 0.4 GB in its weights without embeddings, compared to 1.4 to 4.8 GB that comparable small models need.
The repository isn’t new: bitnet.cpp 1.0 came out in October 2024. But it hasn’t stopped moving. In January 2026, a CPU optimization added another speedup of between 1.15x and 2.1x. In July 2026, Microsoft released two 1-bit embedding models and VibeASR.cpp, a speech recognition engine built on the same quantization. Here’s what it is, how to install it, and where it really fits in your toolbox.
What does it mean for an LLM to be 1-bit?
It means each weight of the model can only take one of three values: -1, 0, or +1. Technically they’re 1.58 bits, and that’s where the “b1.58” in the name comes from. A conventional model stores each weight as a 16-bit floating point number. With ternary weights, expensive matrix multiplications reduce to additions, and that’s why the gain shows up at once in memory, latency, and energy. In English you’ll find it as 1-bit LLM or ternary LLM.
The important detail is in the model card: BitNet b1.58 2B4T was trained from scratch in this format. It’s not a full-precision model that was compressed afterward. That’s the difference from Q4 or Q8 quantizations you download for Ollama or LM Studio: those take a finished model and compress it, while BitNet is born ternary. Activations are quantized to 8 bits, so the full scheme is W1.58A8.
The model has about 2 billion parameters, was trained on 4 trillion tokens, uses the Llama 3 tokenizer, and has a maximum context of 4,096 tokens.
How fast is BitNet and how good is it?
According to Microsoft’s data, which are not independent benchmarks, bitnet.cpp is between 2.37x and 6.17x faster than full-precision inference on x86 CPUs. On those same processors it consumes between 71.9 and 82.2 percent less energy. On ARM, the speedup goes from 1.37x to 5.07x and energy savings, from 55.4 to 70.0 percent. The larger models are the ones that gain the most.
On quality, the model card compares BitNet b1.58 2B with similarly-sized instruct models:
| LLaMA 3.2 1B | Gemma-3 1B | Qwen2.5 1.5B | BitNet b1.58 2B | |
|---|---|---|---|---|
| Memory (without embeddings) | 2 GB | 1.4 GB | 2.6 GB | 0.4 GB |
| CPU decoding latency | 48 ms | 41 ms | 65 ms | 29 ms |
| Benchmark average | 44.90 | 43.74 | 55.23 | 54.19 |
| HumanEval+ | 31.10 | 37.20 | 50.60 | 38.40 |
| GSM8K | 38.21 | 31.16 | 56.79 | 58.38 |
The table shows quality close to Qwen2.5 1.5B with approximately one-sixth the memory. The exception is code: on HumanEval+ it falls quite short of Qwen.
The README also claims that bitnet.cpp can run a BitNet model with 100 billion parameters on a single CPU, at between 5 and 7 tokens per second, roughly human reading speed. Take that as a claim about the engine. There’s no public 100B BitNet model. The largest supported models today are from the Falcon3 family, up to 10B. The repository includes a dummy model generator precisely to measure architectures that don’t yet have a public model.
Why isn’t BitNet faster with transformers?
Because specialized kernels only exist in bitnet.cpp. The model card says it plainly: running BitNet with Hugging Face’s standard transformers library doesn’t give you any advantage in speed, latency, or energy, and it can even be slower than a full-precision model. You’ll see less memory usage, but compute benefits require bitnet.cpp.
It’s the most common mistake with BitNet. People load it in Python, don’t notice a difference, and conclude it was all smoke. The engine is the product.
How to install bitnet.cpp?
You need Python 3.10 or higher, CMake 3.22 or higher, and clang 18 or higher. Using conda is highly recommended. On Debian or Ubuntu, the README points to the automatic LLVM installation script to get clang:
bash -c "$(wget -O - https://apt.llvm.org/llvm.sh)"
1. Clone the repository with its submodules:
git clone --recursive https://github.com/microsoft/BitNet.git
cd BitNet
2. Create the environment and install dependencies:
conda create -n bitnet-cpp python=3.10
conda activate bitnet-cpp
pip install -r requirements.txt
3. Download the model and compile:
huggingface-cli download microsoft/BitNet-b1.58-2B-4T-gguf --local-dir models/BitNet-b1.58-2B-4T
python setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_s
setup_env.py accepts -q i2_s or -q tl1 to choose the quantization kernel. It also has these options:
--use-pretuned(-p): uses already-tuned kernel parameters.--hf-repo(-hr): downloads directly from one of the supported community models.
4. Chat with the model:
python run_inference.py -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf -p "You are a helpful assistant" -cnv
-cnv activates chat mode, and in that mode -p becomes the system prompt. Other useful flags:
-t: number of threads.-c: context size.-n: how many tokens to generate.-temp: temperature.
To measure performance on your own machine:
python utils/e2e_benchmark.py -m /path/to/model -n 200 -p 256 -t 4
That command generates 200 tokens from a 256-token prompt with 4 threads. The default is 2 threads, so adjust -t to your cores before comparing numbers.
Does BitNet work on Windows and Mac?
Yes, on both, with caveats.
Windows needs Visual Studio 2022 with these components enabled:
- Desktop development with C++
- C++ CMake Tools for Windows
- Git for Windows
- C++ Clang Compiler for Windows
- MS-Build Support for LLVM-Toolset (clang)
All commands must be run from a VS2022 Developer Command Prompt or Developer PowerShell. If clang -v responds that the command is not recognized, your terminal isn’t initialized with Visual Studio tools. The README FAQ has the solution with VsDevCmd.bat.
Mac with Apple Silicon and other ARM chips run the chat model with the I2_S and TL1 kernels. Microsoft’s own demo video shows a 3B BitNet model running on an M2.
The new embedding models are x86 only. The supported models table doesn’t list any ARM kernel for BitNet-embedding-0.6B or the 270M one, so on Apple Silicon you can’t run them with bitnet.cpp.
There’s also an official GPU kernel, released in May 2025. At the time of publishing this note, the README mentions NPU support as “coming next”, without a date.
What are the new BitNet 1-bit embedding models?
They’re the first 1-bit embedding models, released on July 20, 2026 in two sizes:
- BitNet-embedding-0.6B: prefill between 1.42x and 2.28x faster than F16, with 8 threads.
- BitNet-embedding-270M: between 1.32x and 1.74x faster, aimed at edge deployments.
Microsoft reports embedding quality competitive with full-precision models using 2 bits per weight. Embeddings are the engine of any RAG pipeline, semantic search, or code search, so this might be the most practical part of the release for a developer. The repository includes a specific conversion and inference guide (docs/bitnet-embeddings-i2s-guide.md).
What is BitNet good for and what isn’t it?
It fits well in these cases:
- Local embeddings for RAG.
- Classification and extraction on servers without a GPU.
- Offline deployments and on the edge.
- CI jobs or background workers, where a small model on free CPU is a better choice than an API call.
- Any case where energy or memory is the main constraint.
It doesn’t fit in these:
- Coding assistance. Its HumanEval+ score is modest; it doesn’t replace your code agent.
- Long documents. The context window is 4,096 tokens.
- Workloads mainly in languages other than English. The model card warns that support for non-English languages is limited.
Microsoft also doesn’t recommend using BitNet b1.58 in commercial or real-world applications without further testing and development. Treat it as a research model with a production-level engine.
BitNet, llama.cpp, or Ollama: which one to use?
It depends on how the model was trained. bitnet.cpp is built on top of llama.cpp; the difference is in the kernels. The bitnet.cpp ones are designed for ternary weights, using lookup table methods from Microsoft’s T-MAC project. For a general quantized model (Q4, Q8, or similar), llama.cpp, Ollama, or LM Studio are still the right tools. Microsoft itself recommends T-MAC for low-bit models that aren’t ternary. Use bitnet.cpp when the model was trained as a BitNet model.
If you’re still choosing between general runtimes, we compare the three here: LM Studio, Ollama or llama.cpp: what actually runs on your machine
What about you? What task in your workflow would you move to a local CPU model?
- Embeddings for RAG or semantic search
- Classification or extraction on servers without a GPU
- Experimenting on my laptop
- I prefer another option (tell us what)
Comment below or, if this article reached you by email, reply directly to the email: your reply will be published here.
Related: Ternlight: How to Install a 7 MB Embedding Engine That Runs Entirely in Your Browser