Introducing mentee-embed-v3: How Far Can Multilingual Embeddings Be Trained from Scratch?

MenteE AI publishes mentee-embed-v3 — a 41M trilingual embedding model for Arabic, English and Urdu trained from random init with 2.1M triplets, MS-MARCO retrieval data, hard negatives and batch size 512. Protocol A avg MRR@10 0.655. Published at doi:10.5281/zenodo.22117673.

MenteE AI publishes mentee-embed-v3, the latest model in our open trilingual embedding series for Arabic, English and Urdu. The preprint is citable at doi:10.5281/zenodo.22117673 and the model weights are live at huggingface.co/MenteEAI/mentee-embed-v3.

What changed from v1 to v3?

Same 41M architecture. Completely different data and training regime. v1 used ~810K triplets (NLI + XNLI + OPUS), batch 192, one distillation round. v3 uses 2.1M triplets, batch 512, two distillation rounds with hard negative mining between rounds, and adds 700K MS-MARCO BM25 + 200K MS-MARCO hard negative passages that v1 never saw. The table below tells the story:

Version	Params	Batch	Prot-A MRR@10	Prot-C MRR@10
v1	41M	192	0.585	~0.20
v2	125M	128	0.429 ↓	0.215
v3	41M	512	0.655	0.645

The key scientific finding: batch size beats model size

v2 was our biggest bet — 125M parameters with a bigger hidden dimension, hoping scale would win. It didn't: Protocol A dropped from 0.585 to 0.429 and Protocol C stayed at 0.215. The culprit? The extra parameters consumed so much VRAM that batch size was forced down from 192 to 128. Fewer negatives per batch means less contrastive signal, and the model suffered for it.

v3 returned to 41M, raised batch to 512, and the result is unambiguous: Protocol C jumped from 0.215 to 0.645 — a 3× improvement while model size decreased. Under a fixed VRAM budget, choose a smaller model that enables a larger batch. That is the headline finding.

Training pipeline in full

Stage A — MLM (8,000 steps, batch 32, lr 5×10⁻⁴, bf16). Custom 50K BPE tokenizer trained from scratch on 1.1M trilingual sentences. Produces a language-aware backbone with no pretrained weights anywhere. ~45 minutes on RTX 5090.

Stage B — Distillation Round 1 (4K steps, batch 512, temp 0.05). Teacher is intfloat/multilingual-e5-base (768-dim), frozen. Student learns to reproduce the teacher's full batch cosine similarity matrix via MSE + InfoNCE. No hard negatives yet.

Stage B — Distillation Round 2 (10K steps, batch 512, temp 0.05). Hard negatives mined GPU-side via chunk-wise dot products after Round 1. Top-5 hardest negatives per anchor injected into training. This is where the MS-MARCO gap closes.

Benchmark results — honest numbers

Protocol A (in-batch, ~97 candidates): avg MRR@10 0.655. The only sub-50M model functional across all three languages — all-MiniLM-L6-v2 scores 0.144 AR and 0.140 UR (pretrained, 23M). mentee-embed-v3 scores 0.475 AR and 0.443 UR from random initialization.

Protocol B (MIRACL Wikipedia corpus, up to 15,201 passages): avg MRR@10 0.260. This is out-of-domain by design — zero Wikipedia passages in training. Pretrained mpnet-base-v2 reaches 0.670 here. We document the gap openly.

Protocol C (MS-MARCO, 10,296 passages): MRR@10 0.645, R@100 0.957. In-domain, strong. The 3× jump from v2 isolates MS-MARCO retrieval data and batch size as the causal factors.

STS-B (Spearman ρ): 0.683. Training was optimized for retrieval, not graded similarity — yet v3 generalizes to STS with no similarity supervision.

How to use v3 — three lines

from transformers import AutoModel, AutoTokenizer

tok   = AutoTokenizer.from_pretrained("MenteEAI/mentee-embed-v3", trust_remote_code=True)
model = AutoModel.from_pretrained("MenteEAI/mentee-embed-v3",    trust_remote_code=True)

embeddings = model.encode(texts, tokenizer=tok)
# torch.Tensor of shape (N, 384), L2-normalised

trust_remote_code=True is standard for custom-architecture HuggingFace models — the code runs locally on your machine.

What's honest and what's a gap

We beat all-MiniLM-L6-v2 on Protocol A and C despite training from scratch. We lag pretrained models on Protocol B (Wikipedia, out-of-domain) and STS-B (not our training objective). Single-seed results — variance unquantified. Full breakdown on /research.

Cite v3

@misc{mentee-embed-v3-2026,
  title   = {mentee-embed-v3: Trilingual Text Embeddings Trained from Scratch},
  author  = {Syed Syab Ahmad Shah and Team MenteE AI},
  year    = {2026},
  url     = {https://huggingface.co/MenteEAI/mentee-embed-v3},
  note    = {Apache-2.0 License. DOI: 10.5281/zenodo.22117673}
}

Model card: /embed-models · Full technical report: /research · GitHub: github.com/MenteE-s/mentee-embeddings