A Qwen 80B in 4.3 GB of RAM: How Swiftlet Streams Experts from the SSD (and What It Costs You in Speed)
By Devy · Category: AI Dev Tools — General
The number in the headline is real, and it’s worth stating it precisely before doing anything with it: Swiftlet runs Qwen3-Next-80B-A3B with 4.3 GB of peak RAM, with 42 GB backed by disk, at 4.5–5 tokens per second — measured on an M5 Mac. The 35B sibling, Qwen3.6-35B-A3B, comes in at 2.6 GB of RAM, 18 GB of disk, and 7–11 tok/s on the same machine. And yes, the 35B runs on an iPhone 17, with about 2.5 GB and around 1 tok/s.
Those are decode figures on Apple Silicon, and the M5 matters: it’s not a number you can mentally transpose to an M1 Air and expect to hold up. But the interesting part isn’t the benchmark. It’s that the number tells you where the model fits, not how fast it runs, and Swiftlet’s author himself is refreshingly clear about that difference. Let’s install it, run it, and then talk honestly about the part the headline leaves out.
The Mechanism, in a Paragraph
Swiftlet is a runtime in Swift + Metal, Apache-2.0, for the Qwen3-Next / Qwen3.6 MoE family. The trick is a separation that MoE architectures invite but most runtimes don’t take advantage of: it keeps only the dense core resident in memory—attention, DeltaNet projections, routers, shared experts, embeddings—and leaves the routed experts on the SSD. When a token needs an expert, Swiftlet issues a single pread to fetch it into a bounded cache pool, with eviction by LFU plus recency.
The detail that enables it is the file format. Swiftlet repackages the model into a .qpack container that stores experts as fixed-stride blobs, so any expert is a single positioned read at a calculable offset — no memory mapping, no page fault roulette, no digging around inside a HuggingFace checkpoint. That’s why the RAM figure is a bound and not an average: the resident set is the dense core plus a cache you size yourself.
Install and Run
You need Apple Silicon, macOS 14+ (or iOS 17+), and enough free SSD space to hold the model.
1. Clone and build.
git clone https://github.com/leonickson1/Swiftlet.git && cd Swiftlet
swift build -c release
2. Repackage the model to .qpack. This is the step that converts the checkpoint to the fixed-stride layout that the streaming path needs. You can fetch it straight from HuggingFace:
.build/release/swiftlet-repack \
--from-hf Leonickson/Qwen3.6-35B-A3B-qpack \
--output ~/models/qwen3.6-35b.qpack
There’s a --source flag if you already have a local checkpoint.
3. Chat.
.build/release/swiftlet chat ~/models/qwen3.6-35b.qpack "Your prompt"
4. Or spin it up as a server. This is the part that lets Swiftlet fit into an existing setup instead of replacing it:
.build/release/swiftlet-server --model ~/models/qwen3.6-35b.qpack --port 8080
swiftlet-server speaks the OpenAI chat-completions API loopback only, not over the network. Any chat UI that talks to an OpenAI-compatible endpoint points there and gets local streaming generation. Start with the 35B: it’s 18 GB instead of 42 and roughly double the decode speed, and it’ll tell you whether the whole approach fits your workflow before you commit the disk.
Integrating into an app. There’s also a path as a Swift package: add SwiftletCore to a macOS or iOS target and manage it through SwiftletSession, which already handles streaming, conversation caching, and sampling.
On the iPhone
The iPhone claim is true and deserves a clarification, because “a 35B on an iPhone” invites the wrong mental image. You don’t compile the CLI on your phone. The path is the Priv AI app on the App Store, or building from the leonickson1/localLLM repo with Swiftlet cloned alongside. And the speed is about 1 token per second: it’s a result of “the model physically fits and generates”, not “replaces your chat app”. Which, for a model with 35 billion parameters running on a phone, is still the impressive version of the statement.
Now the Honest Part: RAM Was Never the Bottleneck
This is the finding around which I’d frame your expectations, and it comes from the author themselves in the Hacker News thread.
RAM is tunable: you can give the cache more memory. So someone asked the obvious question: does more help? The author tested it. Going from a 1 GB cache to a 6 GB one moved the expert cache hit rate from 43% to 70% — and left throughput essentially unchanged. The bottleneck isn’t SSD reads or cache misses. It’s GPU dispatch. Which means the efficiency number in the headline is, in a sense, free: you’re not trading speed for that small resident set, because the resident set was never what was slowing you down. It also means what’s going to make Swiftlet significantly faster is kernel work, not more memory.
The second caveat is bigger and the README doesn’t cover it at all: there are no published prefill numbers. All the figures above are decode — tokens coming out. Prompt processing is a separate cost, and in the HN thread a commenter did the math and landed on roughly half an hour to process 10k tokens on an M5. Take that as an unverified reader estimate and not a measured benchmark, but the structural point holds and matters: if your use case is “I paste a big file and ask questions about it”, prefill is the wall you’re going to hit, and it’s the number nobody’s citing. Short prompts and long generations is where this architecture feels comfortable.
And a third one, which the README states outright and I’m repeating because it’s one of those things you discover the hard way: only about 3B parameters are active per token. As the docs put it, these models “chat and write like large models but recall facts like small ones” — they converse and write like large models, but remember facts like small ones. An 80B MoE with 3B active isn’t a dense 80B model in disguise. Reasoning and fluency hold up; encyclopedic memory doesn’t.
The Part I Really Want to Talk About: The Credits Section
There’s a section in the Swiftlet README called “Relationship to TurboFieldfare”, and I think it’s the most quietly interesting thing in the repo.
On July 31st we covered TurboFieldfare, the Swift runtime that fit Gemma 4 26B into ~2 GB of RAM by streaming experts from the SSD. Swiftlet is the same idea pointed at another model family, and instead of reimplementing it in silence or proclaiming novelty loudly, the README does something almost never seen: it lists which design lessons it adopted and which code it wrote from scratch.
What it says it borrowed from TurboFieldfare: stream experts with pread into a bounded slot pool; evict with LFU plus recency; pack experts with fixed stride; compile shaders at runtime. What it claims as new: an entirely different architecture to support Qwen’s Gated DeltaNet linear attention, the quantized compute path, the validation infrastructure, and iOS integration.
That’s a real distinction, and it’s worth naming precisely because our industry is bad at it today. Ideas travel; code doesn’t have to. TurboFieldfare proved that expert streaming was viable — that the SSD could stand in for RAM in a MoE without everything collapsing. Once that’s established, the next person doesn’t need the first person’s code, they need their conclusions, and Gated DeltaNet is different enough from Gemma’s attention that a fresh implementation was the only honest option anyway. Writing out which is which costs the author nothing and gives every reader a precise map of where the work happened.
If you’re publishing a project that stands on someone else’s shoulders, this is the template. Not a line of Thanks to X for the inspiration at the end. A section that says: here are the four design decisions I made for you, and here’s everything I wrote.
To take with you
Swiftlet is worth an afternoon if you’re on Apple Silicon and want a genuinely large MoE running local without a 64 GB machine. Start with the 35B, run it behind swiftlet-server and point your existing chat UI to loopback. Come in expecting a model that fits beautifully and generates at a rate that works for latency-tolerant tasks: background summaries, batch processing, anything where you’re not watching the cursor. Don’t come in expecting to feed it a 10k token file.
And read the credits section before the benchmarks. It’s the best piece of engineering writing of the two.
Have you already tried running a large MoE streaming from the SSD? In what use case did the decode speed you get actually serve you — and in which one did prefill slow you down?
Sources: