Klingon as a Test Bed: The $8 Chip That Trains Its Own Model (And What’s Missing for Quechua)
By Devy · Category: Open Source / AI & LLMs
Almost every “language model on a microcontroller” project you’ve seen does inference. The weights were trained elsewhere — a GPU, a rented instance, someone’s workstation — and the chip’s job is to run a forward pass on numbers that arrived already finished. qapla' does the other thing. The ESP32-S3 starts with random weights and runs the complete training loop on the device: tokenization, forward pass, cross-entropy loss, backpropagation, weight updates, checkpoints to flash. No PyTorch, no TensorFlow, no autodiff library. Every derivative in the network was deduced by hand and written in C.
That’s an easy sentence to say and hard to believe, which is why the most important number in this repo isn’t the parameter count.
The Number That Gives the Project Credibility
When you write backprop by hand, failure mode is silent: a wrong gradient doesn’t blow up, it just trains a slightly worse model and you never find out. The standard defense is a finite difference check — you nudge a weight by epsilon, measure how much the loss actually moved, and compare it against what your analytical gradient says should have happened.
qapla' publishes that check. Worst relative error: 1.07e-08, against a threshold of 1e-4. Four orders of magnitude of margin. Before any training result means anything, that’s the line saying the chain rule was implemented correctly and not approximately.
What’s Running, Exactly
- ~319,000 parameters, a single transformer block, single-head causal attention
- Character-level vocabulary of ~31 symbols, context window of 32 characters
- SGD with momentum 0.9 plus a cosine learning rate schedule
- 5,000 steps, roughly two days, powered from a phone charger
- Final training loss ≈ 1.87 (moving average)
- RNG seed: 1701, which tells you the author was having fun
The chip saves the checkpoint with the lowest training loss moving average it’s seen. On the next boot it loads that checkpoint and generates. If you connect the optional SH1106 OLED via I2C, you see the loss drop live — which, as the README rightly says, is half the point.
Why Klingon, Specifically
This is where the project stops being a trick and becomes an argument. Three constraints had to be satisfied at once:
Real grammar. There has to be structure worth learning. Random character soup would train toward nothing and prove nothing.
A vocabulary that fits. Klingon works with about 30 symbols. Every character you add widens the embedding table and output projection — on a chip where the whole model weighs ~1.3 MB, the character set is a hard line in the budget, not a detail.
A corpus you’re allowed to use. The training data comes from the boQwI’ dictionary, Apache 2.0. Clean license, no scraping.
The README says it straight: “Klingon is an ideal test bed. A real language, with real grammar, but a size that fits in the palm of your hand.”
And then it names what the test bed is for: the perfect lab to wring out the engine before taking it — maybe in another phase of the project — to real minority languages, with real speakers. “Sami? Quechua? Nahuatl? That… is another story.”
Notice the nuance. It’s an open door, not a roadmap. We come back to that below.
The Hardware Requirement Most People Will Overlook
The headline says “$8 ESP32-S3”. It’s correct, but the board has to be the N16R8 — the variant with PSRAM. Not just any S3.
The memory math explains why. The model weights alone are ~1.3 MB. But training isn’t inference: you also need the gradients (another full copy of the parameter shape), momentum buffers (another one), the cached activations from the forward pass for the backward to consume, and the corpus, compiled in. That adds up to several MB of working set. A bare S3 without PSRAM has a few hundred KB of SRAM. It can’t sustain a training step of this model, no matter how the code is written.
If you buy the wrong board, nothing in the software will save you. Look for the R8 in the part number.
Running It on Your Own Text
The tooling is generic, and that’s the part I’d emphasize. Nothing in the pipeline is Klingon-specific beyond the file you point it at.
1. Convert a corpus into a C header.
python tools/gen_header.py corpus/klingon.txt src/corpus_klingon.h
Change corpus/klingon.txt to any plain text file. The script writes the training data as a header so the chip has it at compile time — no filesystem reads, no network, no SD card.
2. Compile and flash.
pio run -t upload
3. Watch it learn.
pio device monitor
That’s the complete loop. Three commands, one substitution, and the microcontroller on your desk starts inferring statistics from your text from scratch.
Budget the time honestly. The README does: “This takes hours. Many. Even days.” Leave it plugged in.
The Caveat: What the Numbers Don’t Tell You
This is what the repo publishes about model quality: a training loss. This is what it doesn’t publish: no validation loss on held-out data, no perplexity on text the model never saw, no evaluation split.
That absence matters more at this scale than it would elsewhere. With ~319K parameters and a corpus derived from a dictionary, the honest question is whether the model learned Klingon morphology or memorized dictionary chunks — and a training loss can’t distinguish those two outcomes. It goes down in both cases.
The evidence offered instead is qualitative, and it’s decent evidence: the model produces forms like SuvwI' (warrior — the root Suv plus the agentive suffix -wI’) and DujDaq (in the ship), which is well-assembled morphology and not strings you’d expect to come out of pure memorization. The README is also clear that the output isn’t semantically perfect Klingon. Solid in structure, weak in meaning.
So: treat the qualitative examples as the real result and the loss curve as a training diagnostic. And if you retrain on your own corpus, set aside a portion before generating the header — the tooling won’t do it for you, and you’ll want the number the repo doesn’t have.
What’s Missing, Specifically, for Quechua or Nahuatl
It’s worth crossing the door the README leaves open, because the honest answer is that the engine transfers and the configuration doesn’t.
What goes clean: backprop deduced by hand doesn’t care what language you’re tuning, and gen_header.py accepts any text file. You could point this at a Quechua corpus this afternoon.
What would need rethinking, and here I’m reading between the lines and not the repo:
The 32-character context window. Both Quechua and Nahuatl are strongly agglutinative — meaning is built by stacking affixes onto a root, and a single word can carry what Spanish spreads across a clause. A 32-character window that comfortably spans several Klingon words may not span a single long inflected form plus enough of its neighbors to learn from there. That’s the first parameter I’d raise, and it costs attention compute quadratically.
The license of the corpus, which is the real wall. boQwI’ is Apache 2.0 and that’s why the project was able to be published. Free and machine-readable corpora exist for indigenous languages, but they’re scattered, inconsistent in spelling, and often governed by community protocols about how the material can be used — which is a question to answer with speakers, not a scraping problem to get around. That’s the real bottleneck, and no amount of PSRAM will fix it.
Orthographic variation. Klingon has a single canonical writing system. Nahuatl has several competing orthographies depending on the region, and a character-level model trained on mixed conventions is learning the disagreement along with the language. Normalizing first, or deliberately choosing a variant, becomes a modeling decision rather than a preprocessing task.
None of that is reason to think the project won’t get there. It’s actually the list of what that “other phase” contains.
What I’m taking away
The interesting claim of qapla' isn’t that a transformer fits on cheap silicon. It’s that the complete training loop — the part we collectively agreed requires a datacenter — fits on cheap silicon, as long as the model is small enough and someone is willing to do the calculation by hand and demonstrate that they did it right.
That combination puts on-device training within reach of anyone with an $8 board and patience, which is a different phrase from what we’ve been saying about machine learning for the past decade. And that the author wrote a Spanish version of the README himself, and named Quechua and Nahuatl out loud instead of leaving the test field as his own final joke, suggests he already knows what the interesting audience is for that phrase.
The tooling is open, it’s Apache 2.0, and it accepts any text file. The next corpus is a choice, not a technical barrier.
(Klingon was created by Marc Okrand; trademarks belong to CBS Studios / Paramount. The project is educational and not affiliated.)
What about you? If you had this chip training for two days on whatever corpus you wanted, what text would you load into it?
Sources: