mentee-embed-v4: 146% Custom Bench Improvement, 18K sents/sec, mMARCO Arabic

MenteE AI publishes mentee-embed-v4, the latest model in our trilingual embedding series for Arabic, English, and Urdu. v4 retains the 41M architecture from v3 but expands training data to 2.6M triplets with mMARCO Arabic retrieval data (500K) and adds a third distillation round with re-mined hard negatives.

What changed from v3 to v4

• Training data: 2.1M → 2.6M triplets (+500K mMARCO Arabic)
• Distillation rounds: 2 → 3 rounds with progressive hard negative mining
• Custom bench MRR@10: 0.103 → 0.252 (+146%)
• MIRACL AR: 0.475 → 0.825 (acc@1) / 0.874 (MRR@10)
• Throughput: 18,115 sents/sec on RTX 5090 — 3.5× faster than mpnet-base
• Cost: $7.15 per 1B sentences

Training pipeline

Stage A: 50K-step MLM pretraining (6× more than v3). Stage B: three distillation rounds with GPU-accelerated hard negative mining between each. Teacher: multilingual-e5-base. The third round polished performance on dialect Arabic and Roman Urdu — the hardest slices.

Key finding: Arabic is strong

v4 scores 0.825/0.874 on MIRACL Arabic — close to pretrained mpnet-base (0.860/0.898) at 6.8× fewer parameters. This is the first sub-50M model to approach pretrained performance on Arabic retrieval from random initialization.

How to use v4

from transformers import AutoModel, AutoTokenizer

tok   = AutoTokenizer.from_pretrained("MenteEAI/mentee-embed-v4", trust_remote_code=True)
model = AutoModel.from_pretrained("MenteEAI/mentee-embed-v4",    trust_remote_code=True)

embeddings = model.encode(texts, tokenizer=tok)
# torch.Tensor of shape (N, 384), L2-normalised

Full report

All benchmarks, charts, and reproduction steps at /research. Model: MenteEAI/mentee-embed-v4. GitHub: github.com/MenteE-s/mentee-embeddings.