Hard Negative Mining: The Highest-Leverage Trick in Multilingual Embedding Training

Ask what drove the biggest quality jump between mentee-embed-v3 and v4 and the answer isn't more data or a bigger model — it's hard negative mining. This post explains what it is, why it matters disproportionately for low-resource languages, and how we do it without blowing up training time.

The problem with easy negatives

Standard contrastive training (InfoNCE) treats every other example in the batch as a negative. With random batches, those negatives are almost always trivially easy — an English query paired against a random Arabic document teaches the model nothing after the first few thousand steps. The loss drops, eval numbers plateau, and the model never learns fine-grained distinctions.

A hard negative is a document that is semantically close to the query but not the correct answer: a near-duplicate about a different entity, a same-language passage on a related topic, a dialect variant that says something subtly different. Training against these forces the model to learn actual relevance, not just language matching.

Why this matters more for Arabic and Urdu

Low-resource and morphologically rich languages suffer the most from easy-negative training. Arabic dialects and Roman Urdu share vocabulary with their formal counterparts, so a model trained on random negatives learns "Arabic query → any Arabic doc" and collapses dialect, script, and register distinctions. Our v3 error analysis showed exactly this: most retrieval failures were same-language, same-topic, wrong-answer cases.

Our pipeline: mine between rounds, on GPU

We run three distillation rounds. Between each round, the current model encodes the full corpus, retrieves the top-k nearest neighbors for every training query, and we sample negatives from ranks 5–50 (skipping the very top hits, which are often unlabeled positives). Everything — encoding, FAISS search, sampling — stays on GPU, so a mining pass over 2.6M triplets adds roughly 20 minutes instead of hours.

# conceptual sketch
emb = model.encode(corpus)                  # current-round encoder
D, I = faiss_index.search(emb, k=50)        # top-50 per query
hard = sample_from_ranks(I, lo=5, hi=50)    # skip likely positives
train_round(triplets_with(hard))            # next distillation round

Three lessons

• Re-mine every round. Negatives mined by a stale model go soft — the current model already ranks them correctly. Fresh negatives from the latest checkpoint are what keep the gradient informative.
• Skip the top ranks. The #1–4 retrieved "negatives" contain a surprising number of true positives the dataset never labeled. Training against them teaches the model to penalize correct answers.
• Mix in easy negatives too. 100% hard negatives destabilizes training. We keep roughly 70% mined / 30% in-batch random, which gave the best MIRACL and custom-bench numbers in our ablations.

Measured impact

Adding the third distillation round with re-mined hard negatives took custom bench MRR@10 from 0.103 to 0.252 and MIRACL Arabic acc@1 from 0.475 to 0.825, with no architecture change and no new labeled data. Full ablation tables are in the v4 report.