tokenflowNotes
Notes from running open models in production
What we measured while serving open models across our own GPUs: benchmarks with their full protocol, the settings that mattered, and the mistakes we made on the way. Each note says when it was verified.
-
Gemma 4 with MTP in vLLM 0.29: zero prefix-cache hits and a one-flag workaround
Why vLLM 0.29.0 never reused the prefix cache for Gemma 4 12B with MTP speculative decoding on, traced block by block, the flag that brings the hits back, and the release that fixes it.
-
Qwen3-Embedding vs bge-m3 vs EmbeddingGemma 2 for Chinese retrieval
Four open embedding models and two rerankers on T2Retrieval and DuRetrieval with the full C-MTEB corpora, with SciFact and NFCorpus for contrast. Pinned revisions, scripts and aggregate results included.