Skip to content

How well does multilingual retrieval find Sindhi Sufi verse?

Rehan Sathio

Status
Completed
Date
2026
Tags
RAG · Evaluation · Low-resource NLP

  • 9.80%

    Recall@1

  • 23.53%

    Recall@5

  • 27.45%

    Recall@10

  • 51

    queries

    hand-built evaluation set

Abstract

Off-the-shelf multilingual sentence embeddings are often assumed to work for any language. This benchmark tests that assumption on Shah Jo Risalo, classical Sindhi Sufi poetry that is metaphor-heavy, spelled inconsistently in Roman script and almost absent from training data. With a 384-dimensional multilingual encoder and exact FAISS search, the system reaches 27.45% Recall@10 on a hand-built 51-query set, giving a measured baseline for every future change.

01Setup

  • Corpus: 4,767 raw verses from AMBILE, cleaned into 4,730 normalized entries; 2,519 verse documents indexed.
  • Retriever: paraphrase-multilingual-MiniLM-L12-v2 (384-d) with FAISS IndexFlatL2 (exact L2 search).
  • Query-side preprocessing: Romanized-Sindhi spelling normalization (dictionary + regex) and thematic expansion for characters and places.

02Results

Recall@1 is 9.80%, Recall@5 is 23.53% and Recall@10 is 27.45% on the 51-query set. The right verse is usually not the first hit, but it is often in the neighbourhood.

03What it says

Off-the-shelf multilingual embeddings struggle with classical, metaphor-heavy, low-resource poetry. The benchmark exists so the next change (better embeddings, hybrid search, reranking) can be measured instead of guessed.