MenteE AI · Embed Models

mentee-embed

Sentence EmbeddingsEnglishArabicUrduApache 2.041M params

Multilingual text embedding models trained entirely from scratch for Arabic, English, and Urdu — no pretrained backbone. v4 is the latest version: 2.6M triplets, 3-round distillation, 18K sents/sec. Three public versions: v1 (baseline), v3 (MS-MARCO + hard negatives), and v4 (mMARCO Arabic + full pipeline). Full benchmarks on the research page.

Use v4 on Hugging Face →Source on GitHubFull technical report

Overview

Most multilingual embedding models rely on a pretrained backbone like mBERT or XLM-R. mentee-embed starts from random initialization — a custom 50K BPE tokenizer, MLM pretraining on 1.1M trilingual sentences, then relational knowledge distillation from a teacher model.

41M
Parameters
Same across all versions
384
Embedding dim
Cosine similarity ready
3
Languages
EN · AR · UR
2.6M
Training triplets
v4 — with mMARCO Arabic

Versions

v4 adds mMARCO Arabic retrieval data (500K triplets) and a third distillation round with hard negatives — producing a +146% bench MRR improvement over v3 while remaining the same 41M architecture.

VersionParamsBatchDataProt-AProt-CWeights
v1

Baseline — NLI + XNLI + OPUS

41M192~810K triplets0.585~0.20MenteEAI/mentee-embed-v1
v3

+ MS-MARCO + hard negatives

41M5122.1M triplets0.6550.645MenteEAI/mentee-embed-v3
v4Recommended

+ mMARCO Arabic + 3-round distillation · 18K sents/sec

41M5122.6M triplets0.9160.706MenteEAI/mentee-embed-v4

Model Evolution — Prot-A & Prot-C MRR@10

1.000.750.500.250.000.5850.200v10.6550.645v30.9160.706v4Prot-AProt-C

Throughput — sents/sec (RTX 5090, batch 128)

0k5k10k15k20kmentee-v4 ★18,115MiniLM-L612,445e5-small9,749MiniLM-L129,456e5-base5,379mpnet-base5,158

MIRACL — MRR@10 by Language

1.000.750.500.250.00mentee-v4 ★0.9160.8740.572mpnet-base0.9820.8980.824MiniLM-L120.9790.8680.785e5-base0.9900.9580.970MiniLM-L60.990EnglishArabicUrdu

Quick Start

v4 and v3 load via standard transformers — no custom loader or extra library needed. v1 requires cloning the mentee-embeddings repo for the build_embedder helper.

# Install
pip install torch transformers tokenizers huggingface_hub

from transformers import AutoModel, AutoTokenizer

tok   = AutoTokenizer.from_pretrained("MenteEAI/mentee-embed-v4", trust_remote_code=True)
model = AutoModel.from_pretrained("MenteEAI/mentee-embed-v4",    trust_remote_code=True)

texts = [
    "Machine learning is a branch of artificial intelligence",
    "تعلم الآلة هو فرع من الذكاء الاصطناعي",   # AR
    "مشین لرننگ مصنوعی ذہانت کی ایک شاخ ہے",    # UR
]

embeddings = model.encode(texts, tokenizer=tok)
# embeddings: torch.Tensor of shape (3, 384), L2-normalised

Requires trust_remote_code=True — this is standard for custom-architecture models on HuggingFace. The code runs locally on your machine; nothing is sent to any server.

Cross-lingual retrieval works out of the box. Query in English, retrieve in Arabic or Urdu — no language-pair-specific fine-tuning required.

License

Apache 2.0

Cite this model

Published preprint — please cite if you use mentee-embed in your research.

Preprint · v4Zenodo · 28 Aug 2026 · DOI · CC BY 4.0

How Far Can Multilingual Text Embeddings Be Trained From Scratch? A Compute-Efficient Study of Arabic, English, and Urdu

Shah, Syed Syab Ahmad · Team MenteE AI — MenteE AI

41M-parameter trilingual embedding model trained from scratch with ~2.6M triplets, 50K MLM steps, and 3-round distillation. Bench MRR@10 0.252 (+146% vs v3), MIRACL AR 0.874 (+96%), 18,115 sents/sec on RTX 5090 — the fastest model tested.

View on ZenodoDownload PDFCode
Show BibTeX
@misc{mentee-embed-v4-2026,
  title   = {How Far Can Multilingual Text Embeddings Be Trained From Scratch?
             A Compute-Efficient Study of Arabic, English, and Urdu},
  author  = {Shah, Syed Syab Ahmad and Team MenteE AI},
  year    = {2026},
  url     = {https://huggingface.co/MenteEAI/mentee-embed-v4},
  note    = {Apache-2.0 License}
}