Serving a decision model like Jev with vLLM recalculates the same context once for each question. Flide calculates it only once and, according to the project, responds up to 9.5 times faster than vLLM and SGLang without changing the answers.
Status as of November 9, 2026: the current version on PyPI is 0.1.10, released that same day. The project has released a new version every few days since 0.1.0 (October 24), and the list of supported models grows with each one.
Flide is an open source inference server (Apache-2.0) for decision models: models that read a state and answer typed questions about it with a probability for each option, instead of generating text. It installs from PyPI and is signed by a team from the National University of Singapore and the University of Oxford, who describe it in the paper Flide: Accelerating Decision Inference with Exact State Forking (arXiv 2610.42148). It was presented on Hacker News on November 9.
This article is based on the README, the documentation, the paper, and the team’s responses on Hacker News. We haven’t run it: Flide requires an NVIDIA GPU, and all performance figures you’ll see are from the project itself.
What is a decision model like Jev?
A decision model receives a state (a ticket, a conversation, an agent’s trajectory) and several questions about it, and returns a probability distribution over the options for each question in a single pass. It never writes text, so its output connects directly to a threshold: route the ticket, block the command, escalate to a human.
The closed model Jev, by TypeSafe, created the category; we explain how it works in Jev by TypeSafe: how AI that makes decisions without generating text works. Flide serves the same Jev API, /v1/systemone, but with open models on your own GPU: it’s, in practice, a way to have a self-hosted open source Jev.
Why are vLLM and SGLang slow with decision models?
Because they’re designed to generate text, where the unit of work is a linear sequence, and a decision request is a tree: a shared state from which many independent questions hang.
According to Flide’s paper:
- vLLM and SGLang treat each question as its own prompt, so the shared context is rendered, tokenized, and scheduled again for each question. The latest version of vLLM calculates the shared context only once, but each question still runs as a separate sequence that resumes from the prefix cache.
- In hybrid models like Qwen3.5, whose Gated DeltaNet layers compress all previous tokens into a single recurrent state, that cache can only restore state at block boundaries. Each question recalculates the tokens remaining after the last boundary.
- SGLang’s multi-item scoring puts all questions in the same sequence, with a delimiter token in front of each one. It’s fast, but each question reads that extra token, which shifts its probabilities relative to doing it alone.
In the Hacker News thread, a developer commented that they achieved similar speedups by modifying vLLM for their own experiments with decision models. The maintainer’s response: that works with full attention, but not with Qwen3.5’s Gated DeltaNet layers, and that’s why they built a new engine. If you’ve been comparing vLLM vs SGLang, this is the missing piece in that comparison: neither of them is designed for this type of workload.
How does Flide’s state forking work?
Flide calculates the shared state only once and executes all questions together in a single forward pass, while each question calculates exactly what it would if done alone.
In practice:
- In attention layers, a mask ensures each question only sees the state and itself, and each question receives the position indices it would have just after the state.
- In Gated DeltaNet layers, the calculation is done in two calls: one on the states and another on all questions, each starting from the exact recurrent state of its own request.
- The entire pass is executed as a pre-recorded CUDA graph. A short decision request takes only milliseconds of GPU, so launching kernels one by one would dominate the total time.
It’s the same idea as prefix caching or KV cache reuse, taken to the extreme: the common context isn’t resumed from a cache, but computed within the same pass as the questions.
The paper claims that, in exact arithmetic, each answer is identical to the question asked separately. In floating point, differences stay at the rounding level, and the paper places Flide as the engine closest to a float32 reference of those it compared. If you need exactly the same bits for the same request regardless of server load, there’s an optional --batch-invariant mode, at the cost of some speed.
How much faster is Flide than vLLM?
It depends on how many questions share the same state. The 9.5× figure corresponds to Qwen3.5-0.8B on QuALITY, where a long article (about 5,800 tokens) has an average of 18 questions, according to the maintainer’s explanation on Hacker News.
With a single question per request there’s nothing to share, and the gain comes solely from lower per-request cost. The maintainer places it between a technical tie (27B model with a long input) and over 3 times (small model with a short input).
The README table gives the speedup by model family as median latency on an H200, compared to the faster of vLLM and SGLang in the case of chat models. Some rows:
| Model | DI | MultiRC | QuALITY |
|---|---|---|---|
| Qwen3.5 (average across sizes) | 2.1× | 4.8× | 1.3× |
| Gemma 4 (average across sizes) | 1.3× | 1.7× | 1.0× |
| OneJev (vs qev) | 6.8× | 7.3× | 4.7× |
| Senova | 3.1× | 28.1× | 1.2× |
| Clef | 1.3× | 1.0× | 1.0× |
DI is the Decision Index; MultiRC has an average of about 48 questions per paragraph. The practical takeaway: put everything shared in the state and ask all your questions in a single request. That’s the case Flide is designed for.
How do you install Flide and send your first request?
Create an environment with Python 3.12 and install from PyPI:
uv venv -p 3.12 && source .venv/bin/activate
uv pip install flide torchvision
Start a server with any compatible model. Any chat model based on Qwen3.5 or Gemma 4 works as-is:
flide serve --model Qwen/Qwen3.5-4B
Or the project’s own decision model:
flide serve --model OmniJev/OneJev1.6-0.8B
By default, flide serve listens on all network interfaces on port 8000. Add --host 127.0.0.1 if you want it to be accessible only from your machine.
Then, send a request to /v1/systemone:
curl http://localhost:8000/v1/systemone -H "Content-Type: application/json" -d '{
"state": "Order #1182 arrived with a cracked screen. The customer asks for a refund and attached two photos.",
"questions": {
"action": {"type": "choice", "instructions": "What should support do next?",
"criteria": {"refund": "refund the order", "replace": "send a replacement", "escalate": "ask a human"}},
"urgent": {"type": "noul", "instructions": "Does this need a reply within the hour?"}
}
}'
Each response includes the probability of each option, ready to apply a threshold to.
What Flide options matter for agents?
The README documents four:
| Option | What it does |
|---|---|
--session-cache-gb 40 |
saves states from previous requests, so the next step of an agent only calculates what’s new |
--batch-invariant |
same bits for a request regardless of which others it’s batched with, at the cost of some speed |
--quantize fp8 |
Qwen3.5 in FP8 on Ada, Hopper, and Blackwell GPUs |
--frontends 2 |
HTTP, rendering, and tokenization in separate processes, for many clients |
--session-cache-gb is the one that matters in agent loops: each step’s state is almost the previous step’s state plus a few new tokens, and Flide resumes from a checkpoint instead of recalculating the entire history.
What decision models can you serve with Flide?
As of November 9, 2026, the README lists over 30 families. Among them are Qwen3.5 and Gemma 4 chat models, the project’s own OneJev, and decision models like Open-Jev, Kev, Perplexity’s pplx-decider, Cloudflare’s Clef, Intern-Decision, and KnowLine. LoRA adapters are served as published, and the flide verify command checks exact state forking with your own model and your own requests.
Does Flide work on Windows? Do you need a GPU?
Yes, you need a GPU: Flide is a CUDA engine for NVIDIA GPUs, and its documentation describes no mode for CPU. As for Windows, the maintainer indicated on Hacker News that the team has only tested it on Linux and that on Windows you might need WSL2. They also said an RTX 3050 is enough for the 0.8B model.
Flide, Ollaya, or vLLM: which to use?
Flide and Ollaya run open decision models locally behind the Jev API, but they solve different problems, and vLLM still has its place.
- Ollaya follows the Ollama flow: a binary,
pullandrun, ONNX graphs, and CPU on any platform, including Windows and Mac. It’s the simple way to put a decision step in front of an agent on a laptop. - Flide is an inference engine for GPU, designed for decoder models (Qwen3.5, Gemma 4, and their fine-tunes) and optimized for the case where many questions share the same state.
- vLLM is still the tool for generating text. If you already use it to serve generative models, as in our guide to serving open source models on your own hardware with vLLM, don’t change it: Flide is only a vLLM alternative for decision workloads.
If you’re prototyping on your machine, Ollaya. If you’re serving decision models on an NVIDIA GPU and paying for each recalculated token, Flide.
Who benefits from Flide today?
Three profiles get the most out of it:
- Teams already serving decision models with vLLM or SGLang, especially with Qwen3.5.
- Anyone building agents and asking several questions about the same trajectory at each step.
- Anyone who needs the same answer for the same request, regardless of server load.
It has three weeks of life and changes every few days, so pin the version you install and run flide verify with your own requests before trusting it with production traffic.
What about you? What decision in your agent or backend does a decision model already solve instead of an LLM that generates text?
- On a hosted API (Jev or other)
- Locally, with Ollaya or another tool
- On my own GPU, with vLLM, SGLang, or Flide
- I prefer another option (tell us which)
Comment below or, if this article reached you by email, reply directly to the email: your answer will be published here.
Related: Jev by TypeSafe: how AI that makes decisions without generating text works