Gemma 4 with MTP in vLLM 0.29: zero prefix-cache hits and a one-flag workaround

Setup

All runs used the same model, drafter and runtime on one NVIDIA DGX Spark (GB10), shared with another vLLM process: Gemma 4 12B instruction-tuned (RedHatAI NVFP4 checkpoint), MTP with Google’s gemma-4-12B-it-assistant drafter, vllm/vllm-openai:v0.29.0, prefix caching and fp8 KV cache. Three configurations differ in the rest:

Each cache measurement was taken once with greedy decoding.

Symptom

We send each prompt twice and read cached_tokens from the second response. Every MTP variant we tried cached nothing; the LoRA adapter made no difference.

For a router that sends repeat conversations to the server that already holds their prefix, a zero hit rate removes the point of doing so, and every turn pays the full prefill again. Turning MTP off was not attractive either: in the speed benchmark, generation fell from 85.8 to 23.8 tokens per second.

Cause

We patched vLLM’s KV cache coordinator in-process to print each lookup, then read the v0.29.0 source for the store side. The trace prompt had 6,392 tokens and was sent twice.

Six cache groups, one hit

In this configuration Gemma 4 has six KV cache groups: five sliding-window groups (16-token blocks, 1,024-token window) and one global-attention group (64-token blocks). The scheduler works in 64-token blocks. A prefix hit is the shortest hit across all groups, so every group has to find its blocks.

Without MTP

The global group hit 6,336 tokens, and the sliding-window groups held the 1,024-token window ending there. Result: 6,336 cached tokens.

With MTP

With speculative decoding, vLLM drops the last cached block from the drafter’s cache group on lookup, so that block is recomputed. Gemma’s drafter has its own decoder layers, but they read the target model’s KV cache and own none. vLLM 0.29.0 found no group belonging to the drafter and fell back to applying the drop to all six groups:

• the global group dropped one 64-token block, so the hit fell from 6,336 to 6,272;
• to back a hit at 6,272, each sliding-window group needs the window plus one block past it: 65 sixteen-token blocks, numbers 328–392;
• under the default retention (0), the sliding-window groups had stored only the 65 blocks around the replay point computed from the prompt length, which aligns to 6,336: blocks 332–396. The store side did not account for the global group’s drop.

Blocks 328–331 were missing, so the sliding-window hit was 0, and so was the whole hit. Our trace printed only the blocks inside the lookup range, up to 392, so it showed 332–392; the stored range 332–396 comes from the v0.29.0 source.

Workaround on 0.29.0

--prefix-cache-retention-interval controls which sliding-window (and Mamba) checkpoints vLLM keeps. The default 0 keeps only semantic checkpoints such as the replay point. A positive value also keeps a checkpoint every N tokens; N must be a multiple of the scheduler block size, 64 here. The extra checkpoints include the blocks the shifted lookup needs.

We chose 1024, between the two, and run it in production. On the test server:

• Hits: a 9,887-token prompt sent twice reused 9,792 tokens on the second request.
• Output: ten long documents of about 6K tokens, each asked under both model names with greedy decoding: 19 of 20 cold and warm answers were identical; the twentieth differed only in where bold markup fell. Follow-up turns reused 5,888–5,952 of about 6,020 prompt tokens.

In production, our deployment check reused 5,056 of 5,202 prompt tokens on a second turn (12 October 2026).

The cost is cache space: extra sliding-window checkpoints stay in the prefix cache until evicted. They do not take memory from running requests. When we resent an identical prompt, the hit stopped 95 to 120 tokens short of its end (under two 64-token scheduler blocks), as long as the checkpoints had not been evicted.

--enable-prefix-caching --enable-chunked-prefill \
--prefix-cache-retention-interval 1024 \
--speculative-config '{"method":"mtp","model":"/drafter","num_speculative_tokens":7}'

Upstream status (checked 26 October 2026)

• vllm#53945, merged 8 September 2026 and included from v0.30.0, moves the retained replay checkpoint to where a lookup lands after the speculative-decoding drop. That is the mismatch above. vllm#54713 (also from v0.30.0) covers prompts whose length is an exact multiple of the scheduler block.
• vllm#61455, merged 17 October 2026 and not yet in a release, replaces how draft cache groups are identified; its description says Gemma 4 MTP still falls back to applying the drop to all groups, because its drafter shares the target’s KV.
• vllm#40624 (Gemma 4 with DFlash at 0% hits, closed May 2026) and vllm#54360 (hybrid models losing hits under MTP or DFlash, open) track related failures.
• The engine arguments document --prefix-cache-retention-interval and its default of 0.

Our node still runs 0.29.0 with the flag. If you can upgrade, try 0.30.0 or later without the flag and run the check below.

Check your server

The test is worth running on any model that mixes sliding-window (or Mamba) and global attention with speculative decoding. The server needs --enable-prefix-caching and --enable-prompt-tokens-details; set URL and MODEL to yours.

URL=http://localhost:8000/v1/chat/completions MODEL=your-model
body=$(MODEL=$MODEL python3 -c 'import json, os; print(json.dumps({"model": os.environ["MODEL"], "max_tokens": 1,
  "messages": [{"role": "user", "content": " ".join(f"w{i}" for i in range(4000))}]}))')
for i in 1 2; do
  out=$(curl -sS "$URL" -H 'content-type: application/json' -d "$body") || break
  printf '%s' "$out" | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d.get("usage", {}).get("prompt_tokens_details") or d)'
done     # if the prefix stays cached, the second line shows most of the prompt as cached_tokens

To see the per-group lookups, mtp_cache_trace.py patches vLLM 0.29.0’s coordinator in-process and prints, for the sliding-window groups, the cached block ranges, and for every lookup the hit length. It relies on internal names that may change between versions.

FAQ

Why is the vLLM prefix cache hit rate zero for Gemma 4 with MTP?

In vLLM 0.29.0, Gemma 4's MTP drafter owns no KV cache, so vLLM applies the speculative-decoding block drop to all six KV cache groups. The global-attention group's drop moves the hit one 64-token scheduler block earlier than the checkpoint the sliding-window groups kept under the default retention, so the sliding-window groups find nothing and the whole hit is zero.

How do I get prefix-cache hits back without turning off MTP?

On vLLM 0.29.0, start the server with --prefix-cache-retention-interval set to a multiple of the scheduler block size (64 tokens in our setup); we use 1024. vLLM 0.30.0 and later move the retained checkpoint to where the lookup lands; we have not tested those releases on our node.

What does --prefix-cache-retention-interval cost?

It keeps extra sliding-window blocks as periodic checkpoints. They are evictable cache, not memory reserved by running requests. When we resent an identical prompt, the hit stopped 95 to 120 tokens short of its end (under two 64-token scheduler blocks), as long as the checkpoints had not been evicted.

More from tokenflow

tokenflow routes OpenAI- and Anthropic-compatible traffic across our own GPUs and sends each conversation back to the GPU that holds its prefix, which is why this mattered to us. The Qwen3.8-27B deployment on the same DGX Spark is written up in Serving Qwen3.8-27B on One DGX Spark. Read the other notes or request access. Found an error in this note? Write to support@tokenflow.dev.