Separating Attention from FFN: the plugin that squeezes 11% more throughput out of vLLM without touching a single line
By Devy · Category: Open Source / AI & LLMs
Inside a MoE model, two things happen at every token that have almost nothing in common. Attention is memory-bound: it drags the KV cache back and forth and its appetite grows with context length. FFN — in a Mixture-of-Experts, the set of experts — is compute-bound: it wants pure FLOPs and cares very little how long your conversation is. We serve both with the same worker topology, in the same process, on the same GPUs, because that’s how the engine was built. And then we scale the entire cluster according to whichever of the two is drowning first, and overprovision the other.
The AFD plugin for vLLM, published on July 23, 2026 under the vLLM project organization, breaks that assumption. It separates Attention and FFN into two independently deployed services, each scaling on its own axis. The headline number is +11.3% throughput per die and, before you keep reading, it’s worth knowing that number was measured on Ascend 910C NPUs running DeepSeek-V3.2 W8A8, not on NVIDIA hardware. The plugin does support CUDA. Published benchmarks don’t cover it. That distinction matters and we’ll come back to it.
What I want to show you here is why the idea is solid, how it hooks into vLLM without needing a fork, and how to run it on a single machine with a model that actually fits.
The asymmetry nobody was exploiting
Think about what your serving cluster looks like when you increase context length. Attention scales with sequence length: more tokens, more KV cache, more memory bandwidth burned per forward pass. FFN doesn’t care: a request with 16K tokens and one with 2K light up the same experts with the same compute cost per token. The two components diverge under load, and diverge more the longer your contexts are.
When both live in the same worker, you can’t respond to that divergence. You add GPUs and get more of both, whether you need more of both or not. Expert parallelism (EP) already spreads experts across devices, but attention comes along for the ride: in an EP64 deployment you have 64 ranks doing both jobs.
AFD lets you say something different: give me 64 ranks of attention and 16 of FFN. That’s the 64A16F configuration from the benchmarks, and that’s where the 11.3% comes from. Same silicon, different proportion, more output tokens. You’re not buying performance: you’re stopping wasting attention capacity that you provisioned as FFN capacity you didn’t need.
How it hooks in without forking vLLM
This is the part that feels most elegant to me, and it’s the reason this story matters even if you never deploy it.
The plugin registers via vllm.general_plugins, a standard entry point that vLLM exposes exactly for this. No patched source tree, no vendored fork that drifts from upstream, no rebase pain with each release. You install a package, vLLM discovers it on startup, and the AFD runtime is active.
That runtime has three pieces:
- an Attention service that handles scheduling and KV cache,
- an FFN service that runs as a daemon in the background,
- a connector layer that moves activations between them.
Today there are three connectors: P2pNcclAFDConnector for CUDA decode, CAMP2pAFDConnector for decode on Ascend NPU over HCCL, and CAMAsyncAFDConnector for the async prefill path on NPU.
The strategic reading: in the vLLM tracker there have been open RFCs on attention-FFN disaggregation for almost a year—#21644, #22799 and, more recently, #27584 on elastic AFD—. A change so invasive to the execution model is hard to land in a core engine that thousands of deployments depend on. Publishing it as an external plugin under the project’s own organization sidesteps the merge entirely: the feature can be experimental and iterate fast, and the core stays boring. If you maintain infrastructure, that pattern is worth stealing whether you care about MoE serving or not.
The numbers, broken down
There are two figures being cited together. They come from different experiments and deserve to be read separately.
Synchronous decode — DeepSeek-V3.2 W8A8 on Ascend 910C:
| Input length | Baseline EP64 | 64A16F | Delta |
|---|---|---|---|
| 16K | 232.6 tok/s/die | 258.9 tok/s/die | +11.3% |
| 32K | 168.2 tok/s/die | 183.3 tok/s/die | +9.0% |
Asynchronous prefill — same model, 2× Ascend 910C: median time-to-first-token drops from 15.1s to 8.0s at 12 requests per second, approximately −47%.
Three honest reads. First: these are measurements from the plugin authors themselves, with no independent reproduction yet. Second: the throughput gain shrinks as context grows, from 11.3% at 16K to 9.0% at 32K, something worth chewing on considering long context is exactly where you’d most expect disaggregation to pay off. Third: the TTFT number comes from a completely different configuration—async prefill over two devices, not the synchronous decode path on one device that produced the throughput figures—. That’s not a combo you get by flipping a single flag.
And again: Ascend. If you’re on H100, the mechanism applies to you and the CUDA connector exists, but those specific percentages aren’t yours until someone measures them.
Running it on a single machine
You don’t need a 671B model or a rack of NPUs to see it working. The example in the repo itself points to DeepSeek-V2-Lite—a MoE of around 16B with roughly 2.4B active parameters—with DP=1, TP=1. That’s two processes on a single host.
Installation. Requires Python 3.10–3.13 and uv.
git clone https://github.com/vllm-project/afd-plugin.git
cd afd-plugin
uv sync --group dev
On Linux with CUDA, add the vLLM extra, which pins vllm==0.19.1:
uv sync --group dev --extra vllm
Start the FFN service on port 18001:
vllm serve /path/to/DeepSeek-V2-Lite \
--served-model-name deepseek-v2-lite-afd-ffn \
--data-parallel-size 1 \
--tensor-parallel-size 1 \
--enable-expert-parallel \
--enforce-eager \
--host 127.0.0.1 \
--port 18001 \
--additional-config '{"afd":{"role":"ffn","connector":"P2pNcclAFDConnector","host":"127.0.0.1","port":6239,"num_attention_ranks":1,"num_ffn_ranks":1}}'
Start the Attention service on port 18000, pointing to the same rendezvous port as the connector, port 6239:
vllm serve /path/to/DeepSeek-V2-Lite \
--served-model-name deepseek-v2-lite-afd-attention \
--data-parallel-size 1 \
--tensor-parallel-size 1 \
--enable-expert-parallel \
--enforce-eager \
--host 127.0.0.1 \
--port 18000 \
--additional-config '{"afd":{"role":"attention","connector":"P2pNcclAFDConnector","host":"127.0.0.1","port":6239,"num_attention_ranks":1,"num_ffn_ranks":1}}'
Everything AFD-specific lives inside --additional-config. The rest are flags you already know. The two roles differ in exactly one field.
Send traffic only to the Attention server, port 18000. This is the gotcha that’s going to bite you first:
curl http://127.0.0.1:18000/v1/completions \
-H "Content-Type: application/json" \
-d '{"model":"deepseek-v2-lite-afd-attention","prompt":"Explain MoE routing in a paragraph.","max_tokens":128}'
FFN workers are connector-driven. They’re not a second endpoint to balance load across. Calls to execute_model() directed by the FFN-side scheduler fail immediately by design, and that’s fine: you find out instantly instead of debugging silent weirdness.
What you’re going to trip over
The order of ranks is a hard constraint. FFN ranks come before attention ranks, and num_attention_ranks must be greater than or equal to num_ffn_ranks and divisible by it. So 64A16F is legal, 16A64F is not, and 64A24F is not either. Plan the ratio before you plan the hosts.
The vLLM version is pinned to 0.19.1. It’s not “0.19.1 or later”. If your platform is on another version, this is a separate environment, not a drop-in.
Model coverage is narrow. The DeepSeek V2/V3 families, including V3.2, plus GLM MoE DSA. Your Qwen or Mixtral deployment is not covered today.
It is explicitly experimental. The repository says it needs more large-scale testing across different hardware backends, with known limitations in graph support (eager and FULL_DECODE_ONLY only) and in the number of ubatches. This is something to prototype, not to put in front of your customers this quarter.
Where I land
The percentages will shift: they’re self-reported, from a single provider, on a single silicon, and will look different on your hardware. What won’t shift is the observation underneath: attention and FFN are different workloads dressed in the same uniform, and serving them as a single unit means overpaying for one of the two permanently. That idea survives what this particular plugin’s benchmark table says six months from now.
If you run MoE models at any scale, spin up the example with V2-Lite and watch two processes serving a single model. It takes twenty minutes and changes the way you think about inference cluster architecture. And if you’re on NVIDIA and you measure it, publish what you get: that number doesn’t exist publicly today, and the community could really use it.
And you—are you serving MoE models in production? Did you end up over-provisioning GPUs on the attention side knowing the experts were running at half capacity? Tell me in the comments how you’re solving it today.
Sources: