Qwen3-Embedding vs bge-m3 vs EmbeddingGemma 2 for Chinese retrieval

Protocol

Every model saw the same four evaluation sets, byte for byte: we built them once with a fixed seed and checked their SHA-256 on every machine.

Queries are sampled but corpora are complete, so these scores compare with each other, not with the full-query C-MTEB leaderboard.

Embedding results

• Small models, Chinese: Qwen3-Embedding-0.6B leads bge-m3 by 3.9 points on T2 and ties it on Du (+0.1). EmbeddingGemma 2 trails Qwen3-Embedding-0.6B by 7.3 / 8.0 points and bge-m3 by 3.4 / 7.9.
• Going to 4B: +3.8 on T2 and +5.7 on Du, with 6.7× the parameters and 2.5× the vector size of 0.6B.
• EmbeddingGemma 2 leads on SciFact by 6.7 points over the 4B model while trailing it on the other three sets. We did not audit training-data overlap for any model; validate all candidates on held-out data from your own domain.
• bge-m3 was measured in dense mode only. Its sparse and multi-vector outputs are not part of vLLM’s embeddings endpoint.

Reranking results

The rerankers trade places by language. Qwen’s model card (checked 26 October 2026) also puts bge-reranker-v2-m3 slightly ahead on Chinese, 72.16 against 71.31 for Qwen3-Reranker-0.6B on its CMTEB-R suite, reranking the top 100 from the same retriever; on our two sets the gap is larger. Possible reasons we have not tested: T2Retrieval passages contain a lot of HTML markup, and we rerank only the top 50 with documents clipped to 512 tokens. If your traffic is mostly Chinese, test bge-reranker-v2-m3 first.

Pitfalls we hit

The two numbers in the first two pitfalls come from exploratory runs that we discarded; they are not in the published results.

Truncating the pair breaks Qwen3-Reranker

Qwen3-Reranker is a causal language model that scores the probability of “yes” at the position after its prompt suffix (<|im_start|>assistant …). Our first run truncated the whole query-plus-document string to 1,024 tokens from the end, which cut the suffix off long passages. Reranking then pushed T2 down from 84.29 to 68.28, with no error anywhere. Clip the document text first and wrap it in the template afterwards.

Random distractors make Chinese retrieval look easier

Our first Chinese sets used the relevant passages plus 20,000 random ones. Qwen3-Embedding-0.6B scored 91–93 nDCG@10, compressing the differences between models; with the full corpora the same model scores 84–85. To cut evaluation cost without making the task easier, sample queries and keep the full corpus.

EmbeddingGemma 2 needs a newer stack

Its config declares EmbeddingGemma2Model and was saved with a transformers 5.18 development build. Transformers 5.16 has no such architecture, and vLLM 0.29.0’s native model registry does not list it; we did not try vLLM’s Transformers fallback. Transformers 5.20.2 with sentence-transformers 6.2.0 loads it once Pillow is installed for its processor. Use bfloat16 or float32: the model card says float16 returns NaN or degraded embeddings.

Limits

• One run per configuration and 300 queries per Chinese set. We did not compute confidence intervals; we would not decide on a difference of about one point.
• Two Chinese and two English sets, all public. Your domain may rank models differently.
• Quality only. Latency, throughput and memory on the hardware you deploy to are a separate measurement.
• Training-data overlap with these benchmarks was not audited for any model.

Reproduce

The scripts, pinned model and dataset revisions, set hashes and aggregate results are below. We rebuilt the four sets from the pinned dataset revisions on a second machine and their hashes matched sets.sha256.

• download.py: models and datasets at pinned revisions
• eval.py: set building, embedding and reranking evaluation
• sets.sha256: hashes of the evaluation sets
• Aggregate results (renamed for readability): Qwen3-Embedding-0.6B, 4B (T2), 4B (other sets), bge-m3, EmbeddingGemma 2, bge-reranker-v2-m3, Qwen3-Reranker-0.6B

Two Python 3.12 environments: one for vLLM, one for sentence-transformers (EmbeddingGemma 2 only).

export EVAL_WORKDIR=/workspace      # default
python3.12 -m venv vllm-env && vllm-env/bin/pip install vllm==0.29.0 numpy pyarrow huggingface_hub
python3.12 -m venv st-env && st-env/bin/pip install transformers==5.20.2 sentence-transformers==6.2.0 torch pillow numpy pyarrow huggingface_hub

vllm-env/bin/python download.py      # models + datasets, pinned
vllm-env/bin/python eval.py prep
(cd $EVAL_WORKDIR && sha256sum -c /path/to/sets.sha256)

vllm-env/bin/python eval.py embed Qwen3-Embedding-0.6B vllm
vllm-env/bin/python eval.py embed Qwen3-Embedding-4B vllm
vllm-env/bin/python eval.py embed bge-m3 vllm
st-env/bin/python eval.py embed embeddinggemma-2 st
vllm-env/bin/python eval.py rerank bge-reranker-v2-m3 Qwen3-Embedding-0.6B
vllm-env/bin/python eval.py rerank Qwen3-Reranker-0.6B Qwen3-Embedding-0.6B

Check each dataset’s license before using it beyond evaluation.

FAQ

Which small open embedding model is best for Chinese retrieval?

In our October 2026 run, Qwen3-Embedding-0.6B scored nDCG@10 84.29 on T2Retrieval and 84.71 on DuRetrieval with the full C-MTEB corpora, ahead of bge-m3 (80.39, 84.61) and EmbeddingGemma 2 (77.03, 76.68). Qwen3-Embedding-4B scored 3.8 and 5.7 points higher (88.07, 90.39) with about 6.7 times the parameters.

Is bge-reranker-v2-m3 or Qwen3-Reranker-0.6B better for Chinese?

On our two Chinese sets bge-reranker-v2-m3 was better: it raised nDCG@10 by 2.34 on T2Retrieval and 4.99 on DuRetrieval, while Qwen3-Reranker-0.6B changed it by −3.97 and +0.79. On our two English sets the order reversed.

Can vLLM serve EmbeddingGemma 2?

Its architecture, EmbeddingGemma2Model, is not in vLLM 0.29.0's native model registry. We did not try vLLM's Transformers fallback; we ran the model with sentence-transformers 6.2.0 and transformers 5.20.2 in bfloat16. The model card warns against float16.

Why does Qwen3-Reranker make my ranking worse?

Check how you truncate. Qwen3-Reranker scores the token that follows its prompt suffix. If you truncate the whole query-plus-document string from the end, long documents lose the suffix and the score becomes noise; in an exploratory run that dropped T2Retrieval from 84.29 to 68.28. Clip the document first, then wrap it in the template.

Sources

• Model cards: Qwen3-Embedding, Qwen3-Reranker, bge-m3, bge-reranker-v2-m3, EmbeddingGemma 2
• Datasets: C-MTEB/T2Retrieval, C-MTEB/DuRetrieval, mteb/scifact, mteb/nfcorpus
• vLLM 0.29.0 model registry

More from tokenflow

tokenflow routes OpenAI- and Anthropic-compatible traffic across our own GPUs. Read the other notes, see how to use it from coding agents, or request access. Found an error in this note? Write to support@tokenflow.dev.