MenteE AI · Technical Report
mentee-embed research
Trilingual embedding models (Arabic, English, Urdu) trained entirely from scratch. Three versions: v1 (baseline, ~810K triplets), v3 (2.1M triplets, MS-MARCO, hard negatives), and v4 (2.6M triplets, mMARCO Arabic, 3-round distillation). Full benchmarks, training details, and honest evaluations below.
How Far Can Multilingual Text Embeddings Be Trained From Scratch? A Compute-Efficient Study of Arabic, English, and Urdu
Shah, Syed Syab Ahmad · Team MenteE AI — MenteE AI
41M-parameter trilingual embedding model trained from scratch with ~2.6M triplets, 50K MLM steps, and 3-round distillation. Bench MRR@10 0.252 (+146% vs v3), MIRACL AR 0.874 (+96%), 18,115 sents/sec on RTX 5090 — the fastest model tested.
Show BibTeX
@misc{mentee-embed-v4-2026,
title = {How Far Can Multilingual Text Embeddings Be Trained From Scratch?
A Compute-Efficient Study of Arabic, English, and Urdu},
author = {Shah, Syed Syab Ahmad and Team MenteE AI},
year = {2026},
url = {https://huggingface.co/MenteEAI/mentee-embed-v4},
note = {Apache-2.0 License}
}mentee-embed-v4
The Approach
v4 retains the same 41M architecture and 384-dim output from v3 but expands the training data to 2.6M triplets with mMARCO Arabic retrieval data (500K) and adds a third distillation round with re-mined hard negatives, using multilingual-e5-base as teacher.
How Far Can Multilingual Text Embeddings Be Trained From Scratch? A Compute-Efficient Study of Arabic, English, and Urdu
Shah, Syed Syab Ahmad · Team MenteE AI — MenteE AI
41M-parameter trilingual embedding model trained from scratch with ~2.6M triplets, 50K MLM steps, and 3-round distillation. Bench MRR@10 0.252 (+146% vs v3), MIRACL AR 0.874 (+96%), 18,115 sents/sec on RTX 5090 — the fastest model tested.
Show BibTeX
@misc{mentee-embed-v4-2026,
title = {How Far Can Multilingual Text Embeddings Be Trained From Scratch?
A Compute-Efficient Study of Arabic, English, and Urdu},
author = {Shah, Syed Syab Ahmad and Team MenteE AI},
year = {2026},
url = {https://huggingface.co/MenteEAI/mentee-embed-v4},
note = {Apache-2.0 License}
}Training pipeline
- 1
MLM pretraining — 50K steps
Random init → masked language modeling on trilingual corpus, batch 64, vocab 50K BPE. 6× more pretraining than v3.
- 2
Teacher encoding
multilingual-e5-base (768-dim) encodes all 2.6M triplets to soft labels.
- 3
Distillation round 1
InfoNCE + relational distillation, 4,000 steps, batch 512, temp 0.05.
- 4
Hard negative mining round 1
GPU-accelerated top-5 mining across the full 2.6M corpus.
- 5
Distillation round 2
Same objective + mined hard negatives, 10,000 steps.
- 6
Hard negative mining round 2
Re-mine with the improved round-2 model.
- 7
Distillation round 3 — final polish
Train on round-2 mined negatives, 15,000 cumulative steps.
Training data — ~2.6M triplets
v3 used 2.1M · v4 adds mMARCO Arabic (500K)
| Source | Lang | Type | Triplets |
|---|---|---|---|
| all-NLI (sentence-transformers) | EN | NLI triplets | 558K |
| XNLI | AR | NLI triplets | 128K |
| XNLI | UR | NLI triplets | 125K |
| OPUS-100 EN-UR | EN/UR | Parallel translation | 300K |
| OPUS-100 AR-EN | AR/EN | Parallel translation | 300K |
| MS-MARCO BM25 triplets | EN | Passage retrieval | 500K |
| MS-MARCO hard negatives | EN | Hard retrieval | 200K |
| mMARCO Arabic | AR | Passage retrieval | 500K |
| MIRACL | EN/AR/UR | Wikipedia retrieval | ~9K |
| Total | ~2.6M | ||
Training Data Composition — v3 vs v4
Benchmarks
All results on a single NVIDIA RTX 5090. Baselines use pretrained encoders — none trained from scratch.
MenteE Embed Bench — Custom EN/AR/UR
121 queries · 18 domains · dialect Arabic · Roman Urdu · hard negatives · MRR@10
| Model | Init | Params | EN | AR | UR | Dialect AR | Roman UR | All | NDCG@10 |
|---|---|---|---|---|---|---|---|---|---|
| mpnet-base-v2 | Pre | 278M | 0.764 | 0.602 | 0.611 | 0.445 | 0.661 | 0.661 | 0.714 |
| MiniLM-L12-v2 | Pre | 118M | 0.682 | 0.568 | 0.519 | 0.385 | 0.480 | 0.592 | 0.634 |
| e5-base | Pre | 278M | 0.667 | 0.352 | 0.413 | 0.310 | 0.430 | 0.481 | 0.470 |
| MiniLM-L6-v2 | Pre | 23M | 0.873 | 0.087 | 0.289 | 0.084 | 0.486 | 0.425 | 0.299 |
| mentee-v4 ★ | Rand | 41M | 0.369 | 0.120 | 0.261 | 0.095 | 0.399 | 0.252 | 0.238 |
| mentee-v3 | Rand | 41M | 0.225 | 0.021 | 0.055 | 0.000 | 0.079 | 0.103 | 0.147 |
Key: mentee-v4 (0.252 All) closes the gap vs v3 (0.103) by +146%. MiniLM-L6-v2 (0.425) scores high on EN (0.873) but collapses on dialect Arabic (0.084).
Custom Bench — MRR@10 by Language (121 queries, 18 domains)
Protocol A — In-batch Retrieval (MIRACL)
acc@1 / MRR@10 per language
| Model | Init | MIRACL-EN | MIRACL-AR | MIRACL-UR |
|---|---|---|---|---|
| e5-base | Pre | 0.980 / 0.990 | 0.940 / 0.958 | 0.960 / 0.970 |
| MiniLM-L6-v2 | Pre | 0.980 / 0.990 | 0.005 / 0.027 | 0.015 / 0.031 |
| mpnet-base-v2 | Pre | 0.965 / 0.982 | 0.860 / 0.898 | 0.755 / 0.824 |
| MiniLM-L12-v2 | Pre | 0.965 / 0.979 | 0.815 / 0.868 | 0.720 / 0.785 |
| mentee-v4 ★ | Rand | 0.870 / 0.916 | 0.825 / 0.874 | 0.475 / 0.572 |
AR is strong. mentee-v4 scores 0.825/0.874 on AR — close to pretrained mpnet-base (0.860/0.898) at 6.8× fewer parameters.
Protocol C — MS-MARCO Corpus Retrieval
10K passages · MRR@10 · R@5 · R@100
| Model | MRR | R@5 | R@100 |
|---|---|---|---|
| MiniLM-L6-v2 | 0.877 | 0.992 | 0.992 |
| e5-base | 0.856 | 0.992 | 0.996 |
| mpnet-base-v2 | 0.803 | 0.964 | 0.988 |
| mentee-v4 ★ | 0.706 | 0.913 | 0.980 |
0.706 MRR@10 — up from v3's 0.592-era result, +19%.
STS-B — Semantic Similarity
Spearman ρ · zero-shot
| Model | Init | ρ |
|---|---|---|
| mpnet-base-v2 | Pre | 0.868 |
| bge-small-v1.5 | Pre | 0.859 |
| MiniLM-L12-v2 | Pre | 0.844 |
| e5-base | Pre | 0.842 |
| mentee-v4 ★ | Rand | 0.725 |
0.725 ρ — improved over v3 (0.683) with retrieval-focused training.
Speed & Efficiency
Throughput and cost on RTX 5090 at batch size 128.
Throughput & Cost
sents/sec · latency (ms) · VRAM (MB) · cost per 1B sentences
| Model | Params | sents/sec | lat (ms) | VRAM | Cost |
|---|---|---|---|---|---|
| mentee-v4 ★ | 41M | 18,115 | 5.15 | 175 | $7.15 |
| MiniLM-L6-v2 | 23M | 12,445 | 4.71 | 103 | $10.40 |
| e5-small | 118M | 9,749 | 7.88 | 482 | $13.28 |
| MiniLM-L12-v2 | 118M | 9,456 | 8.17 | 573 | $13.69 |
| e5-base | 278M | 5,379 | 8.07 | 1126 | $24.07 |
| mpnet-base-v2 | 278M | 5,158 | 8.04 | 1126 | $25.10 |
Fastest and cheapest. mentee-v4 at 18,115 sents/sec — 3.5× faster than mpnet-base (5,158) and $7.15 per 1B sentences.
Throughput — sents/sec (RTX 5090, batch 128)
Throughput vs Cost — RTX 5090, batch 128
Efficiency Ratio
Bench MRR@10 per 1M params · sents/sec per 1M params
| Model | Params | Bench MRR/1M | Sents/sec/1M |
|---|---|---|---|
| mentee-v4 ★ | 41M | 0.525 | 441.82 |
| MiniLM-L12-v2 | 118M | 0.452 | 80.34 |
| mpnet-base-v2 | 278M | 0.224 | 18.55 |
| e5-base | 278M | 0.166 | 19.35 |
Best efficiency. 0.525 bench MRR per 1M params — highest of any model. 441.82 sents/sec throughput per 1M params.
Strengths
Massive v3 improvement
+146% bench MRR@10 (0.103 → 0.252), +96% MIRACL AR, +62% MIRACL UR — all from the same 41M architecture.
Fastest model tested
18,115 sents/sec at batch 128 — 3.5x faster than mpnet-base, 45% faster than MiniLM-L12.
MIRACL AR/EN competitive
0.874 MIRACL AR MRR@10 — close to pretrained mpnet-base (0.898) at 6.8x fewer parameters.
Lowest cost
$7.15 per 1B sentences — cheapest of all models tested on RTX 5090.
Strong efficiency
0.525 bench MRR per 1M params — highest efficiency ratio of any model in the comparison.
Limitations
We document what does not work alongside what does.
Custom bench gap
0.252 vs 0.661 for mpnet-base. The gap reflects 2.6M training pairs vs web-scale data.
English still dominates
EN MRR@10 = 0.369 vs AR 0.120 on custom bench. MIRACL AR is strong (0.874) but the custom bench uses dialect Arabic.
Not on MTEB full suite
Only targeted retrieval + STS-B evaluated. Full MTEB evaluation pending.
Single seed
All results from one training run; variance unquantified.
Roadmap
- 1
v1.5 completion
Stage-B contrastive run on top of the successful MLM foundation.
- 2
Public leaderboard
Head-to-head benchmarks on identical Arabic/English/Urdu retrieval sets.
- 3
Open release
Model weights, custom tokenizer, and full model card under Apache 2.0.
- 4
Standardized evaluation
MTEB-style benchmarks across multilingual retrieval and bitext-mining tasks.
- 5
Urdu-focused expansion
Adding parallel-corpus alignment objectives for the scarcest retrieval ground truth.
Reproduction
Everything is open: code on GitHub, models on Hugging Face.
Full benchmark report (v4)
All four v4 tables with every metric, per-domain results, and raw JSON — generated 2026-08-28 on RTX 5090.
Quick start (v4)
from transformers import AutoModel, AutoTokenizer
tok = AutoTokenizer.from_pretrained("MenteEAI/mentee-embed-v4", trust_remote_code=True)
model = AutoModel.from_pretrained("MenteEAI/mentee-embed-v4", trust_remote_code=True)
embeddings = model.encode(["Hello world", "مرحبا بالعالم"], tokenizer=tok)
# torch.Size([2, 384])The full training pipeline reproduces on a single GPU with 24GB+ VRAM. Benchmark scripts are in the repo under benchmarks/.
MenteE AI · Trilingual embeddings trained from scratch · Arabic · English · Urdu · v1: MLM → distillation · v3: + MS-MARCO + hard negatives + batch 512 · v4: + mMARCO Arabic + 3-round distillation.