Why Urdu and Arabic Matter: Building AI for 300M+ Speakers the Industry Ignores

At MenteE AI (menteeai.org) we start with a simple observation: Urdu has ~230 million speakers yet almost no open embedding model treats it as a first-class language. Arabic has coverage but almost nothing competitive under 100M parameters. That is not a data problem — it is a priority problem.

The gap in open embeddings

OpenAI, Cohere and Google hide their embeddings behind APIs. Open models like MiniLM, E5 and BGE are excellent for English but Urdu evaluation is an afterthought — often zero Urdu in the benchmark table. MenteE published mentee-embed-v1 precisely to close this gap: 41M params, 50K BPE trained on Arabic+English+Urdu, beating all-MiniLM-L6-v2 on every in-batch slice and reaching EN↔UR cross-lingual acc@1 0.757.

Why we chose 50K BPE from scratch

Off-the-shelf tokenizers over-fragment Urdu (Nastaliq-influenced script) and Arabic. We trained a ByteLevel BPE on our trilingual corpus so Urdu and Arabic get fair vocabulary share. That plus 1.1M sentences of MLM gives the encoder real morphology, not transliteration hacks.

What this means for products at menteeai.org

MenteE does not sell consulting — we ship AI platforms. Urdu/Arabic-first embeddings unlock search, RAG and classification for fintech, legal and government workflows in Pakistan, MENA and diaspora. See Products and Embed Models.

Open and honest

We also publish where we lag: corpus-pool MRR@10 ~0.19. That honesty is rare and intentional — read the full report at /research and cite the preprint doi:10.5281/zenodo.22087139.