{"repo": "anonymous-submitter0104/neurips-submission", "files": {"indic-mmlu/README.md": {"6e9995688769bb4aea3224af97961761ed2e5679": {"t": 1785747245.7293606, "n": 19996, "h": "116c711498ccf534", "c": "# Indic MMLU — Benchmarking Open-Source LLMs on Indian Languages\n\nA rigorously prepared, multi-language adaptation of the CAIS/MMLU test set, translated and enhanced for **22 Indic languages**, designed to evaluate open-source LLMs’ real-world ability to understand Indian languages.\nWe additionally conducted large-scale **evaluation on 16 of these Indic languages (plus English)**, producing detailed accuracy reports and per-language visualizations.\n\n---\n# Table of Contents\n\n1. [Overview](#overview)\n2. [Why the Need for Indic MMLU](#why-the-need-for-indic-mmlu)\n   - [Data Distillation for Indian Languages](#data-distillation-for-indian-languages)\n   - [The Role of Indic MMLU](#the-role-of-indic-mmlu)\n3. [What This Dataset Has on Offer](#what-this-dataset-has-on-offer)\n4. [Coverage](#coverage)\n5. [High-Level Pipeline](#high-level-pipeline-what-we-did)\n   - [Indic MMLU Construction Algorithm](#indic-mmlu-construction-algorithm)\n6. [Translation Algorithm](#translation-algorithm)\n7. [Reproducible Commands](#reproducible-commands-examples)\n   - [Translate](#translate)\n   - [Enhance Translations (LLM-based)](#enhance-translations-llm-based)\n   - [Generate Embeddings](#generate-embeddings)\n   - [Cosine Similarity](#cosine-similarity-english-vs-translated-enhancements)\n   - [Linguist / Teacher Rating](#linguist--teacher-rating-llm-assisted-orchestration)\n   - [Model Evaluation with `lm-eval`](#model-evaluation-with-lm-eval)\n8. [Download Indic MMLU Dataset](#download-Indic-mmlu-dataset)\n9. [Summary of LLM-as-Judge Ratings](#summary-of-llm-as-judge-ratings)\n10. [Cosine Similarity](#cosine-similarity)\n11. [Model Evaluation Methodology](#model-evaluation-methodology)\n12. [Important Note](#important-note)\n13. [Indic MMLU Results](#indic-mmlu-results)\n14. [How to Reproduce (Quick Checklist)](#how-to-reproduce-quick-checklist)\n15. [Best Practices & Caveats](#best-practices--caveats)\n16. [Visual Assets & Tables](#visual-assets--tables)\n\n---\n\n## Why the Need for Indic MMLU\n\nThe rise of large language models (LLMs) has enabled advanced **synthetic data generation**, including:\n\n* Building LLM-based frameworks for **data curation and sandboxing**\n* Generating reasoning and coding examples in non-English languages\n* Creating SFT datasets, as well as off-policy datasets for **RLHF, DPO, GRPO**-style training\n\nTo fully leverage these capabilities for Indian languages, we need LLMs with at least **limited native proficiency** in these languages. Such models allow us to generate high-quality, culturally-aligned datasets, create virtual personas to introduce nuanced constructs, and maintain **full control and flexibility** over the synthetic data pipeline.\n\n### Data Distillation for Indian Languages\n\nData distillation provides a practical approach to achieve this:\n\n* **Easier and cleaner access to world knowledge** from state-of-the-art LLMs\n* **Domain control and flexibility** to shape datasets according to cultural and linguistic requirements\n* Embedding **strong Indian values** into datasets more effectively than via custom pipelines\n* Creating safety datasets that mitigate **anti-Indian stances** and promote positive responses\n\n**Distillation** is the process of generating data from large models to produce **task-specific, high-quality, and culturally-aligned datasets**. Using distillation, we can apply various synthetic data augmentation strategies, such as:\n\n* Chain-of-Thought (CoT) reasoning\n* Multi-turn conversational setups for richer data\n* Summaries and diverse QA pairs\n* Incorporation of strong Indian values through **virtual Indian personas**\n* Other augmentations, including translations and language-specific refinements\n\n### The Role of Indic MMLU\n\nBefore deploying LLMs for such Indian-language data generation, we first need to **evaluate their cross-lingual knowledge transfer capabilities**. Indic MMLU serves this purpose by providing a benchmark to test how well models understand Indian languages, rather than merely translating English content.\n\nWhile previous efforts have translated MMLU into Indian languages, none have **released their methodology**. Indic MMLU is the first benchmark to provide:\n\n* **Open recipe and open code** for translations\n* Coverage of **22 Indian languages**\n* A fully transparent and **reproducible evaluation setup**\n\nThis ensures that models can be assessed rigorously for genuine Indian-language understanding, forming a **foundation for culturally-aware synthetic data pipelines** in future research.\n\n---\n\n## What this dataset has on offer\n\nLarge language models are often benchmarked only on English datasets, which leaves open the question of whether they can *actually* understand Indian languages beyond surface-level translation.\n\nThe **Indic MMLU dataset** bridges this gap by:\n\n* Translating the widely used **[CAIS/MMLU](https://huggingface.co/datasets/cais/mmlu) benchmark** into **22 Indian languages**.\n* Enhancing those translations with high-quality open-source LLM refinement.\n* Providing a reproducible evaluation setup for **measuring open-source models across 16 Indic languages (evaluated subset)**.\n\nIf a model’s score on an Indic version is close to its score on the original English MMLU, it indicates **genuine language understanding** and strong cross-lingual generalization.\n\n---\n\n## Coverage\n\n* **Dataset scope:** 22 Indic languages (full benchmark availability).\n* **Evaluation scope:** 16 Indic languages + English (practically evaluated with ~24 open-source LLMs).\n\nThis separation ensures the dataset is **broad and inclusive**, while evaluation focuses on the languages with the most immediate support and resources.\n\n---\n\n## Overview\n\n* **Source benchmark:** CAIS/MMLU (general knowledge QA benchmark).\n* **Target:** 22 Indic languages + English (translated test set).\n* **Workflow:** Machine translation → LLM-based translation enhancement → embedding comparison → Persona based LLM-as-judge Scores → Correlation between Embeddings and LLM as Judge → human/linguist rating and automated evaluation.\n* **Deliverables:** Enhanced translated `.jsonl` datasets, per-language embeddings, cosine similarity reports, teacher/linguist ratings, model evaluation scores and visualizations (radar plots).\n\n---\n\n## Key principles applied\n\n1. **Preserve semantic parity:** Questions remain semantically the same as English; only language changes.\n2. **Quality control:** Model-Based Automated checks (embeddings, math correctness) + human linguist ratings.\n3. **Reproducibility:** Dockerized inference environment, scriptable pipeline, and clear CLI commands.\n4. **Transparency:** We provide similarity metrics to demonstrate translation fidelity and teacher ratings to highlight linguistic quality.\n\n---\n\n## High-level pipeline (what we did)\n\n1. **Translate** the CAIS/MMLU test set to each Indic language using *IndicTrans2* (baseline MT).\n2. **Enhance** those translations using an open-source LLM with task-specific instructions (to correct phrasing, clarify ambiguous translations, preserve math expressions, and maintain answer choices alignment).\n3. **Embed** both the original English items and the (enhanced) translated items with the same embedding model.\n4. **Compare** embeddings using cosine similarity to measure semantic closeness.\n5. **LLM as Judge and Human review**: linguists/teachers rated samples for language quality, math correctness, and coherence.\n6. **Evaluate models**: run ~24 open-source LLMs across all languages using the lm-eval framework and visualize results (radar plots, top-3 per language).\n\n---\n\n## Indic MMLU Construction Algorithm\n\nTo complement the pseudo-code, the following flowchart provides a **visual overview of the Indic MMLU pipeline**. It clearly illustrates the step-by-step workflow, starting from the English MMLU dataset and moving through translation, enhancement, evaluation, and validation:\n\n* **Translation Stage:** The English MMLU instance is translated into Indic languages using specialist MT.\n* **Enhancement Stage:** Translations are refined with large-scale LLMs for improved fluency and alignment.\n* **LLM Based Judging Phase:** Three types of evaluation are conducted — *fluency by native language experts, subject correctness by domain experts, and alignment by coherence judges*.\n* **Validation Stage:** Embedding similarity checks, n-gram metrics (BLEU, ROUGE, ChrF++), and discrepancy detection ensure quality.\n* **Final Output:** A validated Indic MMLU dataset instance ready for benchmarking.\n\n![Indic MMLU Pipeline](assets/indic-mmlu.png)\n\n# Translation Algorithm \n\n```\nInput:\n  original_mmlu\n  target_languages\n  instructions\n\nFor each language L in target_languages:\n    translated = translate(original_mmlu, tgt=L)\n    enhanced = enhance_using_llm(translated, instructions)\n\n    eng_emb = embed(original_mmlu)\n    trans_emb = embed(enhanced)\n    sim_scores = cosine_similarity(eng_emb, trans_emb)\n\n    llm_scores = LLM_as_Judge(original_mmlu, translated)\n\n    consistency = consistency_check(sim_scores, llm_scores)\n\n    If consistency is poor:\n        human_ratings = rate_sample(enhanced)\n    Else:\n        accept enhanced\n\nAggregate all scores:\n    combine sim_scores, llm_scores, human_ratings\n    compute per-language model scores\n\nProduce visualizations:\n    generate charts and plots for cross-language comparison\n\nOutput:\n    per-language evaluation scores\n    visualizations\n\n```\n\nThis visual representation makes it easier to understand how the **algorithmic steps, human evaluation, and automated checks** come together to ensure translation quality and dataset reliability.\n\n---\n\n## Reproducible commands (examples)\n\n> These are the canonical commands used in our pipeline. Replace file paths, ports, and concurrency values as appropriate for your environment.\n\n**Translate**\n\n```bash\npython3 trans_mmlu.py \\\n  --input_file_path test-00000-of-00001_with_ids.jsonl \\\n  --output_file_path mmlu_kn_in.jsonl \\\n  --tgt_lang \"kn\" \\\n  --src_lang \"en\"\n```\n\n**Enhance translations (LLM-based)**\n\n```bash\npython3 async_infr.py \\\n  --input-path mmlu_as_in.jsonl \\\n  --output-file ds_enhance_eval_hi.jsonl \\\n  --instruction-path instruction_prompts.yml \\\n  --task deepseek_enhance_instruct \\\n  --template-fields 'og_question' 'og_choices' 'question' 'answer' 'Hindi' \\\n  --max-concurrency 4096 \\\n  --backend sglang-oai-chat \\\n  --extra-request-body '{\"temperature\":0.7,\"top_p\":0.9,\"top_k\":50,\"max_tokens\":1024}' \\\n  --port 30000\n```\n\n**Generate embeddings**\n\n```bash\npython3 get_embeddings.py \\\n  --input-file test-00000-of-00001_with_ids.jsonl \\\n  --output-file mmlu_qwen3_embd.jsonl \\\n  --server-url http://127.0.0.1:30000 \\\n  --concurrency 4096\n```\n\n**Cosine similarity (English vs translated enhancements)**\n\n```bash\npython3 calculate_cosine_similarity.py \\\n  --input_dir deepseek_enhance \\\n  --output_dir mmmlu/similarity/cosine/enhance/deepseek_enhance \\\n  --english_file mmlu_eng.jsonl\n```\n\n**Linguist / teacher rating (LLM assisted orchestration)**\n\n```bash\npython3 async_infr.py \\\n  --input-path mmlu_as_in.jsonl \\\n  --output-file ds_enhance_eval_as_qwen3_instruct.jsonl \\\n  --instruction-path instruction_prompts.yml \\\n  --task rate_translated_text_linguist \\\n  --template-fields 'og_question' 'og_choices' 'enhanced_text' 'Assamese' \\\n  --max-concurrency 4096 \\\n  --backend sglang-oai-chat \\\n  --extra-request-body '{\"temperature\":0.7,\"top_p\":0.9,\"top_k\":50,\"max_tokens\":1024}' \\\n  --port 30000\n```\n\n**Model evaluation with `lm-eval`**\n\n```bash\n# Example wrapper which runs evaluation for a model snapshot against the Indic MMLU dataset\nbash lm-eval-llm.sh /path/to/model_snapshot /benchmark-result-path\n```\n\n---\n\n## Download Indic MMLU Dataset\n\nThe complete benchmark covering **22 Indic languages + English** is available here:  \n\n- [Download Full Benchmark (PARQUET)](https://github.com/anonymous-submitter0104/neurips-submission/tree/main/opensource-release/Indic%20MMLU)\n\n> ⚠️ Please cite this repository if you use the dataset in your research.\n\n\n---\n\n### Summary of LLM-as-Judge Ratings\n\nTo evaluate translation quality systematically, we employed **LLMs as judges**, operating under three carefully designed expert personas:\n\n* **Math Expert Persona** → Rated mathematical correctness, terminology, and conceptual fidelity.\n* **Linguistic Expert Persona** → Assessed fluency, grammar, idiomatic usage, and naturalness of the translation.\n* **Coherence Expert Persona** → Evaluated semantic alignment, logical flow, and clarity between English and the target language.\n\n**What we measured:**\n\n* Language quality, fluency, and linguistic suitability.\n* Mathematical correctness and fidelity of technical content.\n* Overall coherence and semantic alignment with the English source.\n\n| Language  | Avg Maths Rate | Avg Coherence Rate | Avg Linguist Rate | Total Records |\n|-----------|----------------|--------------------|-------------------|---------------|\n| Hindi\t    | 9.658029\t     | 9.722054\t          | 9.667891\t        | 14042         |\n| Bengali\t  | 9.182561\t     | 9.131505\t          | 8.202293\t        | 14042         |\n| Nepali    | 9.473793       | 9.331505           | 8.402293          | 14042         |\n| Telugu    | 9.220837       | 8.975929           | 7.986256          | 14042         |\n| Oriya     | 9.264421       | 8.944096           | 8.220268          | 14042         |\n| Punjabi   | 9.348027       | 9.049138           | 8.253454          | 14042         |\n| Assamese  | 9.147201       | 9.039097           | 8.133243          | 14042         |\n| Sanskrit  | 8.338485       | 8.112520           | 6.390115          | 14042         |\n| Kannada   | 9.322746       | 9.111736           | 8.131819          | 14042         |\n| Sindhi    | 2.070645       | 3.109956           | 2.478137          | 14042         |\n| Gujrati   | 9.260647       | 9.206523           | 8.368965          | 14042         |\n| Marathi   | 9.301951       | 9.342045           | 8.381926          | 14042         |\n| Tamil     | 9.047856       | 9.042943           | 8.071144          | 14042         |\n| Malayalam | 9.134952       | 8.998148           | 7.956488          | 14042         |\n| Maithili  | 9.263282       | 9.118146           | 8.218772          | 14042         |\n\n**How to read the table:**\n\n* Ratings are on a **1–10 scale**, where higher scores reflect better translation and enhancement quality.\n* Most languages achieved **consistently high averages (>8.9)**, showing strong quality after enhancement.\n* A few outliers suggest areas where further targeted refinement may be beneficial.\n\n---\n\n## Cosine similarity \n\n* Cosine similarity gives a quick automated signal of semantic closeness between the translated/enhanced items and their English originals.\n* Typical mean similarities range from ~0.76 to 0.85 across languages; higher means closer embeddings (better semantic preservation).\n* Use the similarity scores as a triage: low similarity → manual review; high similarity → likely semantically faithful.\n\n| Language               | Mean Similarity | Std Similarity | Min Similarity | Max Similarity |\n|-------------------------|-----------------|----------------|----------------|----------------|\n| mmlu_as_in_qwen3_embed  | 0.8045          | 0.0597         | 0.4644         | 1              |\n| mmlu_bn_in_qwen3_embed  | 0.8133          | 0.0504         | 0.5276         | 1              |\n| mmlu_gu_in_qwen3_embed  | 0.8211          | 0.0535         | 0.4016         | 1              |\n| mmlu_hi_in_qwen3_embed  | 0.8472          | 0.0489         | 0.3688         | 1              |\n| mmlu_kn_in_qwen3_embed  | 0.8106          | 0.0565         | 0.4660         | 1              |\n| mmlu_mai_in_qwen3_embed | 0.8226          | 0.0504         | 0.4936         | 1              |\n| mmlu_ml_in_qwen3_embed  | 0.8158          | 0.0531         | 0.5109         | 1              |\n| mmlu_mr_in_qwen3_embed  | 0.8129          | 0.0513         | 0.5281         | 1              |\n| mmlu_ne_in_qwen3_embed  | 0.8242          | 0.0502         | 0.4802         | 1              |\n| mmlu_or_in_qwen3_embed  | 0.8159          | 0.0555         | 0.4684         | 1              |\n| mmlu_pa_in_qwen3_embed  | 0.8246          | 0.0516         | 0.5450         | 1              |\n| mmlu_sa_in_qwen3_embed  | 0.7912          | 0.0574         | 0.4981         | 1              |\n| mmlu_sdd_in_qwen3_embed | 0.7646          | 0.0735         | 0.3633         | 0.9676         |\n| mmlu_ta_in_qwen3_embed  | 0.7964          | 0.0559         | 0.5242         | 1              |\n| mmlu_te_in_qwen3_embed  | 0.8006          | 0.0524         | 0.5379         | 1              |\n\n---\n\n## Model Evaluation Methodology\n\nWe evaluated **~24 open-source LLMs** across **16 Indic languages** (subset of the 22 total dataset languages) plus English, using the Indic MMLU benchmark.\n\nWorkflow:\n\n1. **Environment Setup:** Dockerized `vllm/vllm-openai` for reproducibility.\n2. **Framework:** `lm-eval` for standardized evaluation.\n3. **Metrics:** per-language accuracy, top-3 rankings per language, aggregate averages.\n4. **Visualization:** radar plots for each language and comparative top-3 plots.\n\n---\n\n## Important Note\n\n* **Indic MMLU Dataset = 22 Indian languages**\n* **Current Model Evaluation = 16 Indian languages + English**\n* Work is ongoing to expand the evaluation to cover all 22 languages.\n\n---\n### Indic MMLU Results \n\n| <div align=\"center\">![Assamese](assets/assamese.png)<br/>*Assamese*</div> | <div align=\"center\">![Bengali](assets/bengali.png)<br/>*Bengali*</div> |\n|---------------------------------------------------------------------------|------------------------------------------------------------------------|\n| <div align=\"center\">![English](assets/English.png)<br/>*English*</div>    | <div align=\"center\">![Gujarati](assets/Gujrati.png)<br/>*Gujarati*</div> |\n| <div align=\"center\">![Hindi](assets/Hindi.png)<br/>*Hindi*</div>          | <div align=\"center\">![Kannada](assets/kannada.png)<br/>*Kannada*</div> |\n| <div align=\"center\">![Maithili](assets/maithili.png)<br/>*Maithili*</div> | <div align=\"center\">![Malayalam](assets/Malayalam.png)<br/>*Malayalam*</div> |\n| <div align=\"center\">![Marathi](assets/marathi.png)<br/>*Marathi*</div>    | <div align=\"center\">![Nepali](assets/Nepali.png)<br/>*Nepali*</div>   |\n| <div align=\"center\">![Oriya](assets/oriya.png)<br/>*Oriya*</div>          | <div align=\"center\">![Punjabi](assets/Punjabi.png)<br/>*Punjabi*</div> |\n| <div align=\"center\">![Sanskrit](assets/sanskrit.png)<br/>*Sanskrit*</div> | <div align=\"center\">![Sindhi](assets/sindhi.png)<br/>*Sindhi*</div>   |\n| <div align=\"center\">![Tamil](assets/Tamil.png)<br/>*Tamil*</div>          | <div align=\"center\">![Telugu](assets/Telugu.png)<br/>*Telugu*</div>   |\n\n\n---\n\n## How to reproduce (quick checklist)\n\n1. Prepare the English CAIS/MMLU test set (`.jsonl` with IDs).\n2. Run `trans_mmlu.py` to produce per-language MT outputs.\n3. Run `async_infr.py` with the enhancement task to obtain human-readable improved translations.\n4. Generate embeddings for both English and enhanced translations.\n5. Run `calculate_cosine_similarity.py` to produce similarity reports.\n6. Optionally run human/linguist rating orchestration for additional quality assessment.\n7. Evaluate models with `lm-eval` and generate visualizations.\n\n---\n\n## Best practices & caveats\n\n* **Math and formatting:** preserve mathematical expressions and numerals during translation—these are critical for correctness.\n* **Choice alignment:** ensure answer choices remain in the same order and map to the same ground-truth labels.\n* **Human review:** automated metrics are helpful but not sufficient; include periodic linguist sampling.\n* **Constrained inference:** use deterministic LLM settings for enhancement (lower temperature, consistent decoding) to reduce hallucination.\n\n---\n\n## Visual assets & tables\n\nWe include per-language radar plots and a consolidated results panel in `indic-mmlu/assets/`. The README in the repo contains thumbnails; click through for full-resolution visuals. \n\n---\n"}, "a6e2a4ae2c3599addd74a5ffb49f85e39193208b": {"t": 1785771709.843152, "n": 19996, "h": "116c711498ccf534", "c": "# Indic MMLU — Benchmarking Open-Source LLMs on Indian Languages\n\nA rigorously prepared, multi-language adaptation of the CAIS/MMLU test set, translated and enhanced for **22 Indic languages**, designed to evaluate open-source LLMs’ real-world ability to understand Indian languages.\nWe additionally conducted large-scale **evaluation on 16 of these Indic languages (plus English)**, producing detailed accuracy reports and per-language visualizations.\n\n---\n# Table of Contents\n\n1. [Overview](#overview)\n2. [Why the Need for Indic MMLU](#why-the-need-for-indic-mmlu)\n   - [Data Distillation for Indian Languages](#data-distillation-for-indian-languages)\n   - [The Role of Indic MMLU](#the-role-of-indic-mmlu)\n3. [What This Dataset Has on Offer](#what-this-dataset-has-on-offer)\n4. [Coverage](#coverage)\n5. [High-Level Pipeline](#high-level-pipeline-what-we-did)\n   - [Indic MMLU Construction Algorithm](#indic-mmlu-construction-algorithm)\n6. [Translation Algorithm](#translation-algorithm)\n7. [Reproducible Commands](#reproducible-commands-examples)\n   - [Translate](#translate)\n   - [Enhance Translations (LLM-based)](#enhance-translations-llm-based)\n   - [Generate Embeddings](#generate-embeddings)\n   - [Cosine Similarity](#cosine-similarity-english-vs-translated-enhancements)\n   - [Linguist / Teacher Rating](#linguist--teacher-rating-llm-assisted-orchestration)\n   - [Model Evaluation with `lm-eval`](#model-evaluation-with-lm-eval)\n8. [Download Indic MMLU Dataset](#download-Indic-mmlu-dataset)\n9. [Summary of LLM-as-Judge Ratings](#summary-of-llm-as-judge-ratings)\n10. [Cosine Similarity](#cosine-similarity)\n11. [Model Evaluation Methodology](#model-evaluation-methodology)\n12. [Important Note](#important-note)\n13. [Indic MMLU Results](#indic-mmlu-results)\n14. [How to Reproduce (Quick Checklist)](#how-to-reproduce-quick-checklist)\n15. [Best Practices & Caveats](#best-practices--caveats)\n16. [Visual Assets & Tables](#visual-assets--tables)\n\n---\n\n## Why the Need for Indic MMLU\n\nThe rise of large language models (LLMs) has enabled advanced **synthetic data generation**, including:\n\n* Building LLM-based frameworks for **data curation and sandboxing**\n* Generating reasoning and coding examples in non-English languages\n* Creating SFT datasets, as well as off-policy datasets for **RLHF, DPO, GRPO**-style training\n\nTo fully leverage these capabilities for Indian languages, we need LLMs with at least **limited native proficiency** in these languages. Such models allow us to generate high-quality, culturally-aligned datasets, create virtual personas to introduce nuanced constructs, and maintain **full control and flexibility** over the synthetic data pipeline.\n\n### Data Distillation for Indian Languages\n\nData distillation provides a practical approach to achieve this:\n\n* **Easier and cleaner access to world knowledge** from state-of-the-art LLMs\n* **Domain control and flexibility** to shape datasets according to cultural and linguistic requirements\n* Embedding **strong Indian values** into datasets more effectively than via custom pipelines\n* Creating safety datasets that mitigate **anti-Indian stances** and promote positive responses\n\n**Distillation** is the process of generating data from large models to produce **task-specific, high-quality, and culturally-aligned datasets**. Using distillation, we can apply various synthetic data augmentation strategies, such as:\n\n* Chain-of-Thought (CoT) reasoning\n* Multi-turn conversational setups for richer data\n* Summaries and diverse QA pairs\n* Incorporation of strong Indian values through **virtual Indian personas**\n* Other augmentations, including translations and language-specific refinements\n\n### The Role of Indic MMLU\n\nBefore deploying LLMs for such Indian-language data generation, we first need to **evaluate their cross-lingual knowledge transfer capabilities**. Indic MMLU serves this purpose by providing a benchmark to test how well models understand Indian languages, rather than merely translating English content.\n\nWhile previous efforts have translated MMLU into Indian languages, none have **released their methodology**. Indic MMLU is the first benchmark to provide:\n\n* **Open recipe and open code** for translations\n* Coverage of **22 Indian languages**\n* A fully transparent and **reproducible evaluation setup**\n\nThis ensures that models can be assessed rigorously for genuine Indian-language understanding, forming a **foundation for culturally-aware synthetic data pipelines** in future research.\n\n---\n\n## What this dataset has on offer\n\nLarge language models are often benchmarked only on English datasets, which leaves open the question of whether they can *actually* understand Indian languages beyond surface-level translation.\n\nThe **Indic MMLU dataset** bridges this gap by:\n\n* Translating the widely used **[CAIS/MMLU](https://huggingface.co/datasets/cais/mmlu) benchmark** into **22 Indian languages**.\n* Enhancing those translations with high-quality open-source LLM refinement.\n* Providing a reproducible evaluation setup for **measuring open-source models across 16 Indic languages (evaluated subset)**.\n\nIf a model’s score on an Indic version is close to its score on the original English MMLU, it indicates **genuine language understanding** and strong cross-lingual generalization.\n\n---\n\n## Coverage\n\n* **Dataset scope:** 22 Indic languages (full benchmark availability).\n* **Evaluation scope:** 16 Indic languages + English (practically evaluated with ~24 open-source LLMs).\n\nThis separation ensures the dataset is **broad and inclusive**, while evaluation focuses on the languages with the most immediate support and resources.\n\n---\n\n## Overview\n\n* **Source benchmark:** CAIS/MMLU (general knowledge QA benchmark).\n* **Target:** 22 Indic languages + English (translated test set).\n* **Workflow:** Machine translation → LLM-based translation enhancement → embedding comparison → Persona based LLM-as-judge Scores → Correlation between Embeddings and LLM as Judge → human/linguist rating and automated evaluation.\n* **Deliverables:** Enhanced translated `.jsonl` datasets, per-language embeddings, cosine similarity reports, teacher/linguist ratings, model evaluation scores and visualizations (radar plots).\n\n---\n\n## Key principles applied\n\n1. **Preserve semantic parity:** Questions remain semantically the same as English; only language changes.\n2. **Quality control:** Model-Based Automated checks (embeddings, math correctness) + human linguist ratings.\n3. **Reproducibility:** Dockerized inference environment, scriptable pipeline, and clear CLI commands.\n4. **Transparency:** We provide similarity metrics to demonstrate translation fidelity and teacher ratings to highlight linguistic quality.\n\n---\n\n## High-level pipeline (what we did)\n\n1. **Translate** the CAIS/MMLU test set to each Indic language using *IndicTrans2* (baseline MT).\n2. **Enhance** those translations using an open-source LLM with task-specific instructions (to correct phrasing, clarify ambiguous translations, preserve math expressions, and maintain answer choices alignment).\n3. **Embed** both the original English items and the (enhanced) translated items with the same embedding model.\n4. **Compare** embeddings using cosine similarity to measure semantic closeness.\n5. **LLM as Judge and Human review**: linguists/teachers rated samples for language quality, math correctness, and coherence.\n6. **Evaluate models**: run ~24 open-source LLMs across all languages using the lm-eval framework and visualize results (radar plots, top-3 per language).\n\n---\n\n## Indic MMLU Construction Algorithm\n\nTo complement the pseudo-code, the following flowchart provides a **visual overview of the Indic MMLU pipeline**. It clearly illustrates the step-by-step workflow, starting from the English MMLU dataset and moving through translation, enhancement, evaluation, and validation:\n\n* **Translation Stage:** The English MMLU instance is translated into Indic languages using specialist MT.\n* **Enhancement Stage:** Translations are refined with large-scale LLMs for improved fluency and alignment.\n* **LLM Based Judging Phase:** Three types of evaluation are conducted — *fluency by native language experts, subject correctness by domain experts, and alignment by coherence judges*.\n* **Validation Stage:** Embedding similarity checks, n-gram metrics (BLEU, ROUGE, ChrF++), and discrepancy detection ensure quality.\n* **Final Output:** A validated Indic MMLU dataset instance ready for benchmarking.\n\n![Indic MMLU Pipeline](assets/indic-mmlu.png)\n\n# Translation Algorithm \n\n```\nInput:\n  original_mmlu\n  target_languages\n  instructions\n\nFor each language L in target_languages:\n    translated = translate(original_mmlu, tgt=L)\n    enhanced = enhance_using_llm(translated, instructions)\n\n    eng_emb = embed(original_mmlu)\n    trans_emb = embed(enhanced)\n    sim_scores = cosine_similarity(eng_emb, trans_emb)\n\n    llm_scores = LLM_as_Judge(original_mmlu, translated)\n\n    consistency = consistency_check(sim_scores, llm_scores)\n\n    If consistency is poor:\n        human_ratings = rate_sample(enhanced)\n    Else:\n        accept enhanced\n\nAggregate all scores:\n    combine sim_scores, llm_scores, human_ratings\n    compute per-language model scores\n\nProduce visualizations:\n    generate charts and plots for cross-language comparison\n\nOutput:\n    per-language evaluation scores\n    visualizations\n\n```\n\nThis visual representation makes it easier to understand how the **algorithmic steps, human evaluation, and automated checks** come together to ensure translation quality and dataset reliability.\n\n---\n\n## Reproducible commands (examples)\n\n> These are the canonical commands used in our pipeline. Replace file paths, ports, and concurrency values as appropriate for your environment.\n\n**Translate**\n\n```bash\npython3 trans_mmlu.py \\\n  --input_file_path test-00000-of-00001_with_ids.jsonl \\\n  --output_file_path mmlu_kn_in.jsonl \\\n  --tgt_lang \"kn\" \\\n  --src_lang \"en\"\n```\n\n**Enhance translations (LLM-based)**\n\n```bash\npython3 async_infr.py \\\n  --input-path mmlu_as_in.jsonl \\\n  --output-file ds_enhance_eval_hi.jsonl \\\n  --instruction-path instruction_prompts.yml \\\n  --task deepseek_enhance_instruct \\\n  --template-fields 'og_question' 'og_choices' 'question' 'answer' 'Hindi' \\\n  --max-concurrency 4096 \\\n  --backend sglang-oai-chat \\\n  --extra-request-body '{\"temperature\":0.7,\"top_p\":0.9,\"top_k\":50,\"max_tokens\":1024}' \\\n  --port 30000\n```\n\n**Generate embeddings**\n\n```bash\npython3 get_embeddings.py \\\n  --input-file test-00000-of-00001_with_ids.jsonl \\\n  --output-file mmlu_qwen3_embd.jsonl \\\n  --server-url http://127.0.0.1:30000 \\\n  --concurrency 4096\n```\n\n**Cosine similarity (English vs translated enhancements)**\n\n```bash\npython3 calculate_cosine_similarity.py \\\n  --input_dir deepseek_enhance \\\n  --output_dir mmmlu/similarity/cosine/enhance/deepseek_enhance \\\n  --english_file mmlu_eng.jsonl\n```\n\n**Linguist / teacher rating (LLM assisted orchestration)**\n\n```bash\npython3 async_infr.py \\\n  --input-path mmlu_as_in.jsonl \\\n  --output-file ds_enhance_eval_as_qwen3_instruct.jsonl \\\n  --instruction-path instruction_prompts.yml \\\n  --task rate_translated_text_linguist \\\n  --template-fields 'og_question' 'og_choices' 'enhanced_text' 'Assamese' \\\n  --max-concurrency 4096 \\\n  --backend sglang-oai-chat \\\n  --extra-request-body '{\"temperature\":0.7,\"top_p\":0.9,\"top_k\":50,\"max_tokens\":1024}' \\\n  --port 30000\n```\n\n**Model evaluation with `lm-eval`**\n\n```bash\n# Example wrapper which runs evaluation for a model snapshot against the Indic MMLU dataset\nbash lm-eval-llm.sh /path/to/model_snapshot /benchmark-result-path\n```\n\n---\n\n## Download Indic MMLU Dataset\n\nThe complete benchmark covering **22 Indic languages + English** is available here:  \n\n- [Download Full Benchmark (PARQUET)](https://github.com/anonymous-submitter0104/neurips-submission/tree/main/opensource-release/Indic%20MMLU)\n\n> ⚠️ Please cite this repository if you use the dataset in your research.\n\n\n---\n\n### Summary of LLM-as-Judge Ratings\n\nTo evaluate translation quality systematically, we employed **LLMs as judges**, operating under three carefully designed expert personas:\n\n* **Math Expert Persona** → Rated mathematical correctness, terminology, and conceptual fidelity.\n* **Linguistic Expert Persona** → Assessed fluency, grammar, idiomatic usage, and naturalness of the translation.\n* **Coherence Expert Persona** → Evaluated semantic alignment, logical flow, and clarity between English and the target language.\n\n**What we measured:**\n\n* Language quality, fluency, and linguistic suitability.\n* Mathematical correctness and fidelity of technical content.\n* Overall coherence and semantic alignment with the English source.\n\n| Language  | Avg Maths Rate | Avg Coherence Rate | Avg Linguist Rate | Total Records |\n|-----------|----------------|--------------------|-------------------|---------------|\n| Hindi\t    | 9.658029\t     | 9.722054\t          | 9.667891\t        | 14042         |\n| Bengali\t  | 9.182561\t     | 9.131505\t          | 8.202293\t        | 14042         |\n| Nepali    | 9.473793       | 9.331505           | 8.402293          | 14042         |\n| Telugu    | 9.220837       | 8.975929           | 7.986256          | 14042         |\n| Oriya     | 9.264421       | 8.944096           | 8.220268          | 14042         |\n| Punjabi   | 9.348027       | 9.049138           | 8.253454          | 14042         |\n| Assamese  | 9.147201       | 9.039097           | 8.133243          | 14042         |\n| Sanskrit  | 8.338485       | 8.112520           | 6.390115          | 14042         |\n| Kannada   | 9.322746       | 9.111736           | 8.131819          | 14042         |\n| Sindhi    | 2.070645       | 3.109956           | 2.478137          | 14042         |\n| Gujrati   | 9.260647       | 9.206523           | 8.368965          | 14042         |\n| Marathi   | 9.301951       | 9.342045           | 8.381926          | 14042         |\n| Tamil     | 9.047856       | 9.042943           | 8.071144          | 14042         |\n| Malayalam | 9.134952       | 8.998148           | 7.956488          | 14042         |\n| Maithili  | 9.263282       | 9.118146           | 8.218772          | 14042         |\n\n**How to read the table:**\n\n* Ratings are on a **1–10 scale**, where higher scores reflect better translation and enhancement quality.\n* Most languages achieved **consistently high averages (>8.9)**, showing strong quality after enhancement.\n* A few outliers suggest areas where further targeted refinement may be beneficial.\n\n---\n\n## Cosine similarity \n\n* Cosine similarity gives a quick automated signal of semantic closeness between the translated/enhanced items and their English originals.\n* Typical mean similarities range from ~0.76 to 0.85 across languages; higher means closer embeddings (better semantic preservation).\n* Use the similarity scores as a triage: low similarity → manual review; high similarity → likely semantically faithful.\n\n| Language               | Mean Similarity | Std Similarity | Min Similarity | Max Similarity |\n|-------------------------|-----------------|----------------|----------------|----------------|\n| mmlu_as_in_qwen3_embed  | 0.8045          | 0.0597         | 0.4644         | 1              |\n| mmlu_bn_in_qwen3_embed  | 0.8133          | 0.0504         | 0.5276         | 1              |\n| mmlu_gu_in_qwen3_embed  | 0.8211          | 0.0535         | 0.4016         | 1              |\n| mmlu_hi_in_qwen3_embed  | 0.8472          | 0.0489         | 0.3688         | 1              |\n| mmlu_kn_in_qwen3_embed  | 0.8106          | 0.0565         | 0.4660         | 1              |\n| mmlu_mai_in_qwen3_embed | 0.8226          | 0.0504         | 0.4936         | 1              |\n| mmlu_ml_in_qwen3_embed  | 0.8158          | 0.0531         | 0.5109         | 1              |\n| mmlu_mr_in_qwen3_embed  | 0.8129          | 0.0513         | 0.5281         | 1              |\n| mmlu_ne_in_qwen3_embed  | 0.8242          | 0.0502         | 0.4802         | 1              |\n| mmlu_or_in_qwen3_embed  | 0.8159          | 0.0555         | 0.4684         | 1              |\n| mmlu_pa_in_qwen3_embed  | 0.8246          | 0.0516         | 0.5450         | 1              |\n| mmlu_sa_in_qwen3_embed  | 0.7912          | 0.0574         | 0.4981         | 1              |\n| mmlu_sdd_in_qwen3_embed | 0.7646          | 0.0735         | 0.3633         | 0.9676         |\n| mmlu_ta_in_qwen3_embed  | 0.7964          | 0.0559         | 0.5242         | 1              |\n| mmlu_te_in_qwen3_embed  | 0.8006          | 0.0524         | 0.5379         | 1              |\n\n---\n\n## Model Evaluation Methodology\n\nWe evaluated **~24 open-source LLMs** across **16 Indic languages** (subset of the 22 total dataset languages) plus English, using the Indic MMLU benchmark.\n\nWorkflow:\n\n1. **Environment Setup:** Dockerized `vllm/vllm-openai` for reproducibility.\n2. **Framework:** `lm-eval` for standardized evaluation.\n3. **Metrics:** per-language accuracy, top-3 rankings per language, aggregate averages.\n4. **Visualization:** radar plots for each language and comparative top-3 plots.\n\n---\n\n## Important Note\n\n* **Indic MMLU Dataset = 22 Indian languages**\n* **Current Model Evaluation = 16 Indian languages + English**\n* Work is ongoing to expand the evaluation to cover all 22 languages.\n\n---\n### Indic MMLU Results \n\n| <div align=\"center\">![Assamese](assets/assamese.png)<br/>*Assamese*</div> | <div align=\"center\">![Bengali](assets/bengali.png)<br/>*Bengali*</div> |\n|---------------------------------------------------------------------------|------------------------------------------------------------------------|\n| <div align=\"center\">![English](assets/English.png)<br/>*English*</div>    | <div align=\"center\">![Gujarati](assets/Gujrati.png)<br/>*Gujarati*</div> |\n| <div align=\"center\">![Hindi](assets/Hindi.png)<br/>*Hindi*</div>          | <div align=\"center\">![Kannada](assets/kannada.png)<br/>*Kannada*</div> |\n| <div align=\"center\">![Maithili](assets/maithili.png)<br/>*Maithili*</div> | <div align=\"center\">![Malayalam](assets/Malayalam.png)<br/>*Malayalam*</div> |\n| <div align=\"center\">![Marathi](assets/marathi.png)<br/>*Marathi*</div>    | <div align=\"center\">![Nepali](assets/Nepali.png)<br/>*Nepali*</div>   |\n| <div align=\"center\">![Oriya](assets/oriya.png)<br/>*Oriya*</div>          | <div align=\"center\">![Punjabi](assets/Punjabi.png)<br/>*Punjabi*</div> |\n| <div align=\"center\">![Sanskrit](assets/sanskrit.png)<br/>*Sanskrit*</div> | <div align=\"center\">![Sindhi](assets/sindhi.png)<br/>*Sindhi*</div>   |\n| <div align=\"center\">![Tamil](assets/Tamil.png)<br/>*Tamil*</div>          | <div align=\"center\">![Telugu](assets/Telugu.png)<br/>*Telugu*</div>   |\n\n\n---\n\n## How to reproduce (quick checklist)\n\n1. Prepare the English CAIS/MMLU test set (`.jsonl` with IDs).\n2. Run `trans_mmlu.py` to produce per-language MT outputs.\n3. Run `async_infr.py` with the enhancement task to obtain human-readable improved translations.\n4. Generate embeddings for both English and enhanced translations.\n5. Run `calculate_cosine_similarity.py` to produce similarity reports.\n6. Optionally run human/linguist rating orchestration for additional quality assessment.\n7. Evaluate models with `lm-eval` and generate visualizations.\n\n---\n\n## Best practices & caveats\n\n* **Math and formatting:** preserve mathematical expressions and numerals during translation—these are critical for correctness.\n* **Choice alignment:** ensure answer choices remain in the same order and map to the same ground-truth labels.\n* **Human review:** automated metrics are helpful but not sufficient; include periodic linguist sampling.\n* **Constrained inference:** use deterministic LLM settings for enhancement (lower temperature, consistent decoding) to reduce hallucination.\n\n---\n\n## Visual assets & tables\n\nWe include per-language radar plots and a consolidated results panel in `indic-mmlu/assets/`. The README in the repo contains thumbnails; click through for full-resolution visuals. \n\n---\n"}, "bad7cfae4a545e3f48015b3f4f74aa7b5ef84bb5": {"t": 1785771711.23567, "n": 19993, "h": "1bd87ff6b92898cb", "c": "# Indic MMLU — Benchmarking Open-Source LLMs on Indian Languages\n\nA rigorously prepared, multi-language adaptation of the CAIS/MMLU test set, translated and enhanced for **22 Indic languages**, designed to evaluate open-source LLMs’ real-world ability to understand Indian languages.\nWe additionally conducted large-scale **evaluation on 16 of these Indic languages (plus English)**, producing detailed accuracy reports and per-language visualizations.\n\n---\n# Table of Contents\n\n1. [Overview](#overview)\n2. [Why the Need for Indic MMLU](#why-the-need-for-indic-mmlu)\n   - [Data Distillation for Indian Languages](#data-distillation-for-indian-languages)\n   - [The Role of Indic MMLU](#the-role-of-indic-mmlu)\n3. [What This Dataset Has on Offer](#what-this-dataset-has-on-offer)\n4. [Coverage](#coverage)\n5. [High-Level Pipeline](#high-level-pipeline-what-we-did)\n   - [Indic MMLU Construction Algorithm](#indic-mmlu-construction-algorithm)\n6. [Translation Algorithm](#translation-algorithm)\n7. [Reproducible Commands](#reproducible-commands-examples)\n   - [Translate](#translate)\n   - [Enhance Translations (LLM-based)](#enhance-translations-llm-based)\n   - [Generate Embeddings](#generate-embeddings)\n   - [Cosine Similarity](#cosine-similarity-english-vs-translated-enhancements)\n   - [Linguist / Teacher Rating](#linguist--teacher-rating-llm-assisted-orchestration)\n   - [Model Evaluation with `lm-eval`](#model-evaluation-with-lm-eval)\n8. [Download Indic MMLU Dataset](#download-Indic-mmlu-dataset)\n9. [Summary of LLM-as-Judge Ratings](#summary-of-llm-as-judge-ratings)\n10. [Cosine Similarity](#cosine-similarity)\n11. [Model Evaluation Methodology](#model-evaluation-methodology)\n12. [Important Note](#important-note)\n13. [Indic MMLU Results](#indic-mmlu-results)\n14. [How to Reproduce (Quick Checklist)](#how-to-reproduce-quick-checklist)\n15. [Best Practices & Caveats](#best-practices--caveats)\n16. [Visual Assets & Tables](#visual-assets--tables)\n\n---\n\n## Why the Need for Indic MMLU\n\nThe rise of large language models (LLMs) has enabled advanced **synthetic data generation**, including:\n\n* Building LLM-based frameworks for **data curation and sandboxing**\n* Generating reasoning and coding examples in non-English languages\n* Creating SFT datasets, as well as off-policy datasets for **RLHF, DPO, GRPO**-style training\n\nTo fully leverage these capabilities for Indian languages, we need LLMs with at least **limited native proficiency** in these languages. Such models allow us to generate high-quality, culturally-aligned datasets, create virtual personas to introduce nuanced constructs, and maintain **full control and flexibility** over the synthetic data pipeline.\n\n### Data Distillation for Indian Languages\n\nData distillation provides a practical approach to achieve this:\n\n* **Easier and cleaner access to world knowledge** from state-of-the-art LLMs\n* **Domain control and flexibility** to shape datasets according to cultural and linguistic requirements\n* Embedding **strong Indian values** into datasets more effectively than via custom pipelines\n* Creating safety datasets that mitigate **anti-Indian stances** and promote positive responses\n\n**Distillation** is the process of generating data from large models to produce **task-specific, high-quality, and culturally-aligned datasets**. Using distillation, we can apply various synthetic data augmentation strategies, such as:\n\n* Chain-of-Thought (CoT) reasoning\n* Multi-turn conversational setups for richer data\n* Summaries and diverse QA pairs\n* Incorporation of strong Indian values through **virtual Indian personas**\n* Other augmentations, including translations and language-specific refinements\n\n### The Role of Indic MMLU\n\nBefore deploying LLMs for such Indian-language data generation, we first need to **evaluate their cross-lingual knowledge transfer capabilities**. Indic MMLU serves this purpose by providing a benchmark to test how well models understand Indian languages, rather than merely translating English content.\n\nWhile previous efforts have translated MMLU into Indian languages, none have **released their methodology**. Indic MMLU is the first benchmark to provide:\n\n* **Open recipe and open code** for translations\n* Coverage of **22 Indian languages**\n* A fully transparent and **reproducible evaluation setup**\n\nThis ensures that models can be assessed rigorously for genuine Indian-language understanding, forming a **foundation for culturally-aware synthetic data pipelines** in future research.\n\n---\n\n## What this dataset has on offer\n\nLarge language models are often benchmarked only on English datasets, which leaves open the question of whether they can *actually* understand Indian languages beyond surface-level translation.\n\nThe **Indic MMLU dataset** bridges this gap by:\n\n* Translating the widely used **[CAIS/MMLU](https://huggingface.co/datasets/cais/mmlu) benchmark** into **22 Indian languages**.\n* Enhancing those translations with high-quality open-source LLM refinement.\n* Providing a reproducible evaluation setup for **measuring open-source models across 16 Indic languages (evaluated subset)**.\n\nIf a model’s score on an Indic version is close to its score on the original English MMLU, it indicates **genuine language understanding** and strong cross-lingual generalization.\n\n---\n\n## Coverage\n\n* **Dataset scope:** 22 Indic languages (full benchmark availability).\n* **Evaluation scope:** 16 Indic languages + English (practically evaluated with ~24 open-source LLMs).\n\nThis separation ensures the dataset is **broad and inclusive**, while evaluation focuses on the languages with the most immediate support and resources.\n\n---\n\n## Overview\n\n* **Source benchmark:** CAIS/MMLU (general knowledge QA benchmark).\n* **Target:** 22 Indic languages + English (translated test set).\n* **Workflow:** Machine translation → LLM-based translation enhancement → embedding comparison → Persona based LLM-as-judge Scores → Correlation between Embeddings and LLM as Judge → human/linguist rating and automated evaluation.\n* **Deliverables:** Enhanced translated `.jsonl` datasets, per-language embeddings, cosine similarity reports, teacher/linguist ratings, model evaluation scores and visualizations (radar plots).\n\n---\n\n## Key principles applied\n\n1. **Preserve semantic parity:** Questions remain semantically the same as English; only language changes.\n2. **Quality control:** Model-Based Automated checks (embeddings, math correctness) + human linguist ratings.\n3. **Reproducibility:** Dockerized inference environment, scriptable pipeline, and clear CLI commands.\n4. **Transparency:** We provide similarity metrics to demonstrate translation fidelity and teacher ratings to highlight linguistic quality.\n\n---\n\n## High-level pipeline (what we did)\n\n1. **Translate** the CAIS/MMLU test set to each Indic language using *IndicTrans2* (baseline MT).\n2. **Enhance** those translations using an open-source LLM with task-specific instructions (to correct phrasing, clarify ambiguous translations, preserve math expressions, and maintain answer choices alignment).\n3. **Embed** both the original English items and the (enhanced) translated items with the same embedding model.\n4. **Compare** embeddings using cosine similarity to measure semantic closeness.\n5. **LLM as Judge and Human review**: linguists/teachers rated samples for language quality, math correctness, and coherence.\n6. **Evaluate models**: run ~24 open-source LLMs across all languages using the lm-eval framework and visualize results (radar plots, top-3 per language).\n\n---\n\n## Indic MMLU Construction Algorithm\n\nTo complement the pseudo-code, the following flowchart provides a **visual overview of the Indic MMLU pipeline**. It clearly illustrates the step-by-step workflow, starting from the English MMLU dataset and moving through translation, enhancement, evaluation, and validation:\n\n* **Translation Stage:** The English MMLU instance is translated into Indic languages using specialist MT.\n* **Enhancement Stage:** Translations are refined with large-scale LLMs for improved fluency and alignment.\n* **LLM Based Judging Phase:** Three types of evaluation are conducted — *fluency by native language experts, subject correctness by domain experts, and alignment by coherence judges*.\n* **Validation Stage:** Embedding similarity checks, n-gram metrics (BLEU, ROUGE, ChrF++), and discrepancy detection ensure quality.\n* **Final Output:** A validated Indic MMLU dataset instance ready for benchmarking.\n\n![Indic MMLU Pipeline](assets/indic-mmlu.png)\n\n# Translation Algorithm \n\n```\nInput:\n  original_mmlu\n  target_languages\n  instructions\n\nFor each language L in target_languages:\n    translated = translate(original_mmlu, tgt=L)\n    enhanced = enhance_using_llm(translated, instructions)\n\n    eng_emb = embed(original_mmlu)\n    trans_emb = embed(enhanced)\n    sim_scores = cosine_similarity(eng_emb, trans_emb)\n\n    llm_scores = LLM_as_Judge(original_mmlu, translated)\n\n    consistency = consistency_check(sim_scores, llm_scores)\n\n    If consistency is poor:\n        human_ratings = rate_sample(enhanced)\n    Else:\n        accept enhanced\n\nAggregate all scores:\n    combine sim_scores, llm_scores, human_ratings\n    compute per-language model scores\n\nProduce visualizations:\n    generate charts and plots for cross-language comparison\n\nOutput:\n    per-language evaluation scores\n    visualizations\n\n```\n\nThis visual representation makes it easier to understand how the **algorithmic steps, human evaluation, and automated checks** come together to ensure translation quality and dataset reliability.\n\n---\n\n## Reproducible commands (examples)\n\n> These are the canonical commands used in our pipeline. Replace file paths, ports, and concurrency values as appropriate for your environment.\n\n**Translate**\n\n```bash\npython3 trans_mmlu.py \\\n  --input_file_path test-00000-of-00001_with_ids.jsonl \\\n  --output_file_path mmlu_kn_in.jsonl \\\n  --tgt_lang \"kn\" \\\n  --src_lang \"en\"\n```\n\n**Enhance translations (LLM-based)**\n\n```bash\npython3 async_infr.py \\\n  --input-path mmlu_as_in.jsonl \\\n  --output-file ds_enhance_eval_hi.jsonl \\\n  --instruction-path instruction_prompts.yml \\\n  --task deepseek_enhance_instruct \\\n  --template-fields 'og_question' 'og_choices' 'question' 'answer' 'Hindi' \\\n  --max-concurrency 4096 \\\n  --backend sglang-oai-chat \\\n  --extra-request-body '{\"temperature\":0.7,\"top_p\":0.9,\"top_k\":50,\"max_tokens\":1024}' \\\n  --port 30000\n```\n\n**Generate embeddings**\n\n```bash\npython3 get_embeddings.py \\\n  --input-file test-00000-of-00001_with_ids.jsonl \\\n  --output-file mmlu_qwen3_embd.jsonl \\\n  --server-url http://127.0.0.1:30000 \\\n  --concurrency 4096\n```\n\n**Cosine similarity (English vs translated enhancements)**\n\n```bash\npython3 calculate_cosine_similarity.py \\\n  --input_dir deepseek_enhance \\\n  --output_dir mmmlu/similarity/cosine/enhance/deepseek_enhance \\\n  --english_file mmlu_eng.jsonl\n```\n\n**Linguist / teacher rating (LLM assisted orchestration)**\n\n```bash\npython3 async_infr.py \\\n  --input-path mmlu_as_in.jsonl \\\n  --output-file ds_enhance_eval_as_qwen3_instruct.jsonl \\\n  --instruction-path instruction_prompts.yml \\\n  --task rate_translated_text_linguist \\\n  --template-fields 'og_question' 'og_choices' 'enhanced_text' 'Assamese' \\\n  --max-concurrency 4096 \\\n  --backend sglang-oai-chat \\\n  --extra-request-body '{\"temperature\":0.7,\"top_p\":0.9,\"top_k\":50,\"max_tokens\":1024}' \\\n  --port 30000\n```\n\n**Model evaluation with `lm-eval`**\n\n```bash\n# Example wrapper which runs evaluation for a model snapshot against the Indic MMLU dataset\nbash lm-eval-llm.sh /path/to/model_snapshot /benchmark-result-path\n```\n\n---\n\n## Download Indic MMLU Dataset\n\nThe complete benchmark covering **22 Indic languages + English** is available here:  \n\n- [Download Full Benchmark (PARQUET)](https://github.com/anonymous-submitter0104/iclr-submission/tree/main/opensource-release/Indic%20MMLU)\n\n> ⚠️ Please cite this repository if you use the dataset in your research.\n\n\n---\n\n### Summary of LLM-as-Judge Ratings\n\nTo evaluate translation quality systematically, we employed **LLMs as judges**, operating under three carefully designed expert personas:\n\n* **Math Expert Persona** → Rated mathematical correctness, terminology, and conceptual fidelity.\n* **Linguistic Expert Persona** → Assessed fluency, grammar, idiomatic usage, and naturalness of the translation.\n* **Coherence Expert Persona** → Evaluated semantic alignment, logical flow, and clarity between English and the target language.\n\n**What we measured:**\n\n* Language quality, fluency, and linguistic suitability.\n* Mathematical correctness and fidelity of technical content.\n* Overall coherence and semantic alignment with the English source.\n\n| Language  | Avg Maths Rate | Avg Coherence Rate | Avg Linguist Rate | Total Records |\n|-----------|----------------|--------------------|-------------------|---------------|\n| Hindi\t    | 9.658029\t     | 9.722054\t          | 9.667891\t        | 14042         |\n| Bengali\t  | 9.182561\t     | 9.131505\t          | 8.202293\t        | 14042         |\n| Nepali    | 9.473793       | 9.331505           | 8.402293          | 14042         |\n| Telugu    | 9.220837       | 8.975929           | 7.986256          | 14042         |\n| Oriya     | 9.264421       | 8.944096           | 8.220268          | 14042         |\n| Punjabi   | 9.348027       | 9.049138           | 8.253454          | 14042         |\n| Assamese  | 9.147201       | 9.039097           | 8.133243          | 14042         |\n| Sanskrit  | 8.338485       | 8.112520           | 6.390115          | 14042         |\n| Kannada   | 9.322746       | 9.111736           | 8.131819          | 14042         |\n| Sindhi    | 2.070645       | 3.109956           | 2.478137          | 14042         |\n| Gujrati   | 9.260647       | 9.206523           | 8.368965          | 14042         |\n| Marathi   | 9.301951       | 9.342045           | 8.381926          | 14042         |\n| Tamil     | 9.047856       | 9.042943           | 8.071144          | 14042         |\n| Malayalam | 9.134952       | 8.998148           | 7.956488          | 14042         |\n| Maithili  | 9.263282       | 9.118146           | 8.218772          | 14042         |\n\n**How to read the table:**\n\n* Ratings are on a **1–10 scale**, where higher scores reflect better translation and enhancement quality.\n* Most languages achieved **consistently high averages (>8.9)**, showing strong quality after enhancement.\n* A few outliers suggest areas where further targeted refinement may be beneficial.\n\n---\n\n## Cosine similarity \n\n* Cosine similarity gives a quick automated signal of semantic closeness between the translated/enhanced items and their English originals.\n* Typical mean similarities range from ~0.76 to 0.85 across languages; higher means closer embeddings (better semantic preservation).\n* Use the similarity scores as a triage: low similarity → manual review; high similarity → likely semantically faithful.\n\n| Language               | Mean Similarity | Std Similarity | Min Similarity | Max Similarity |\n|-------------------------|-----------------|----------------|----------------|----------------|\n| mmlu_as_in_qwen3_embed  | 0.8045          | 0.0597         | 0.4644         | 1              |\n| mmlu_bn_in_qwen3_embed  | 0.8133          | 0.0504         | 0.5276         | 1              |\n| mmlu_gu_in_qwen3_embed  | 0.8211          | 0.0535         | 0.4016         | 1              |\n| mmlu_hi_in_qwen3_embed  | 0.8472          | 0.0489         | 0.3688         | 1              |\n| mmlu_kn_in_qwen3_embed  | 0.8106          | 0.0565         | 0.4660         | 1              |\n| mmlu_mai_in_qwen3_embed | 0.8226          | 0.0504         | 0.4936         | 1              |\n| mmlu_ml_in_qwen3_embed  | 0.8158          | 0.0531         | 0.5109         | 1              |\n| mmlu_mr_in_qwen3_embed  | 0.8129          | 0.0513         | 0.5281         | 1              |\n| mmlu_ne_in_qwen3_embed  | 0.8242          | 0.0502         | 0.4802         | 1              |\n| mmlu_or_in_qwen3_embed  | 0.8159          | 0.0555         | 0.4684         | 1              |\n| mmlu_pa_in_qwen3_embed  | 0.8246          | 0.0516         | 0.5450         | 1              |\n| mmlu_sa_in_qwen3_embed  | 0.7912          | 0.0574         | 0.4981         | 1              |\n| mmlu_sdd_in_qwen3_embed | 0.7646          | 0.0735         | 0.3633         | 0.9676         |\n| mmlu_ta_in_qwen3_embed  | 0.7964          | 0.0559         | 0.5242         | 1              |\n| mmlu_te_in_qwen3_embed  | 0.8006          | 0.0524         | 0.5379         | 1              |\n\n---\n\n## Model Evaluation Methodology\n\nWe evaluated **~24 open-source LLMs** across **16 Indic languages** (subset of the 22 total dataset languages) plus English, using the Indic MMLU benchmark.\n\nWorkflow:\n\n1. **Environment Setup:** Dockerized `vllm/vllm-openai` for reproducibility.\n2. **Framework:** `lm-eval` for standardized evaluation.\n3. **Metrics:** per-language accuracy, top-3 rankings per language, aggregate averages.\n4. **Visualization:** radar plots for each language and comparative top-3 plots.\n\n---\n\n## Important Note\n\n* **Indic MMLU Dataset = 22 Indian languages**\n* **Current Model Evaluation = 16 Indian languages + English**\n* Work is ongoing to expand the evaluation to cover all 22 languages.\n\n---\n### Indic MMLU Results \n\n| <div align=\"center\">![Assamese](assets/assamese.png)<br/>*Assamese*</div> | <div align=\"center\">![Bengali](assets/bengali.png)<br/>*Bengali*</div> |\n|---------------------------------------------------------------------------|------------------------------------------------------------------------|\n| <div align=\"center\">![English](assets/English.png)<br/>*English*</div>    | <div align=\"center\">![Gujarati](assets/Gujrati.png)<br/>*Gujarati*</div> |\n| <div align=\"center\">![Hindi](assets/Hindi.png)<br/>*Hindi*</div>          | <div align=\"center\">![Kannada](assets/kannada.png)<br/>*Kannada*</div> |\n| <div align=\"center\">![Maithili](assets/maithili.png)<br/>*Maithili*</div> | <div align=\"center\">![Malayalam](assets/Malayalam.png)<br/>*Malayalam*</div> |\n| <div align=\"center\">![Marathi](assets/marathi.png)<br/>*Marathi*</div>    | <div align=\"center\">![Nepali](assets/Nepali.png)<br/>*Nepali*</div>   |\n| <div align=\"center\">![Oriya](assets/oriya.png)<br/>*Oriya*</div>          | <div align=\"center\">![Punjabi](assets/Punjabi.png)<br/>*Punjabi*</div> |\n| <div align=\"center\">![Sanskrit](assets/sanskrit.png)<br/>*Sanskrit*</div> | <div align=\"center\">![Sindhi](assets/sindhi.png)<br/>*Sindhi*</div>   |\n| <div align=\"center\">![Tamil](assets/Tamil.png)<br/>*Tamil*</div>          | <div align=\"center\">![Telugu](assets/Telugu.png)<br/>*Telugu*</div>   |\n\n\n---\n\n## How to reproduce (quick checklist)\n\n1. Prepare the English CAIS/MMLU test set (`.jsonl` with IDs).\n2. Run `trans_mmlu.py` to produce per-language MT outputs.\n3. Run `async_infr.py` with the enhancement task to obtain human-readable improved translations.\n4. Generate embeddings for both English and enhanced translations.\n5. Run `calculate_cosine_similarity.py` to produce similarity reports.\n6. Optionally run human/linguist rating orchestration for additional quality assessment.\n7. Evaluate models with `lm-eval` and generate visualizations.\n\n---\n\n## Best practices & caveats\n\n* **Math and formatting:** preserve mathematical expressions and numerals during translation—these are critical for correctness.\n* **Choice alignment:** ensure answer choices remain in the same order and map to the same ground-truth labels.\n* **Human review:** automated metrics are helpful but not sufficient; include periodic linguist sampling.\n* **Constrained inference:** use deterministic LLM settings for enhancement (lower temperature, consistent decoding) to reduce hallucination.\n\n---\n\n## Visual assets & tables\n\nWe include per-language radar plots and a consolidated results panel in `indic-mmlu/assets/`. The README in the repo contains thumbnails; click through for full-resolution visuals. \n\n---\n"}, "ac61730cf2f4134df3473dc8543b76d8d72d99f4": {"t": 1785771713.0376034, "n": 19981, "h": "81d82c3c7f58c327", "c": "# Indic MMLU — Benchmarking Open-Source LLMs on Indian Languages\n\nA rigorously prepared, multi-language adaptation of the CAIS/MMLU test set, translated and enhanced for **22 Indic languages**, designed to evaluate open-source LLMs’ real-world ability to understand Indian languages.\nWe additionally conducted large-scale **evaluation on 16 of these Indic languages (plus English)**, producing detailed accuracy reports and per-language visualizations.\n\n---\n# Table of Contents\n\n1. [Overview](#overview)\n2. [Why the Need for Indic MMLU](#why-the-need-for-indic-mmlu)\n   - [Data Distillation for Indian Languages](#data-distillation-for-indian-languages)\n   - [The Role of Indic MMLU](#the-role-of-indic-mmlu)\n3. [What This Dataset Has on Offer](#what-this-dataset-has-on-offer)\n4. [Coverage](#coverage)\n5. [High-Level Pipeline](#high-level-pipeline-what-we-did)\n   - [Indic MMLU Construction Algorithm](#indic-mmlu-construction-algorithm)\n6. [Translation Algorithm](#translation-algorithm)\n7. [Reproducible Commands](#reproducible-commands-examples)\n   - [Translate](#translate)\n   - [Enhance Translations (LLM-based)](#enhance-translations-llm-based)\n   - [Generate Embeddings](#generate-embeddings)\n   - [Cosine Similarity](#cosine-similarity-english-vs-translated-enhancements)\n   - [Linguist / Teacher Rating](#linguist--teacher-rating-llm-assisted-orchestration)\n   - [Model Evaluation with `lm-eval`](#model-evaluation-with-lm-eval)\n8. [Download Indic MMLU Dataset](#download-Indic-mmlu-dataset)\n9. [Summary of LLM-as-Judge Ratings](#summary-of-llm-as-judge-ratings)\n10. [Cosine Similarity](#cosine-similarity)\n11. [Model Evaluation Methodology](#model-evaluation-methodology)\n12. [Important Note](#important-note)\n13. [Indic MMLU Results](#indic-mmlu-results)\n14. [How to Reproduce (Quick Checklist)](#how-to-reproduce-quick-checklist)\n15. [Best Practices & Caveats](#best-practices--caveats)\n16. [Visual Assets & Tables](#visual-assets--tables)\n\n---\n\n## Why the Need for Indic MMLU\n\nThe rise of large language models (LLMs) has enabled advanced **synthetic data generation**, including:\n\n* Building LLM-based frameworks for **data curation and sandboxing**\n* Generating reasoning and coding examples in non-English languages\n* Creating SFT datasets, as well as off-policy datasets for **RLHF, DPO, GRPO**-style training\n\nTo fully leverage these capabilities for Indian languages, we need LLMs with at least **limited native proficiency** in these languages. Such models allow us to generate high-quality, culturally-aligned datasets, create virtual personas to introduce nuanced constructs, and maintain **full control and flexibility** over the synthetic data pipeline.\n\n### Data Distillation for Indian Languages\n\nData distillation provides a practical approach to achieve this:\n\n* **Easier and cleaner access to world knowledge** from state-of-the-art LLMs\n* **Domain control and flexibility** to shape datasets according to cultural and linguistic requirements\n* Embedding **strong Indian values** into datasets more effectively than via custom pipelines\n* Creating safety datasets that mitigate **anti-Indian stances** and promote positive responses\n\n**Distillation** is the process of generating data from large models to produce **task-specific, high-quality, and culturally-aligned datasets**. Using distillation, we can apply various synthetic data augmentation strategies, such as:\n\n* Chain-of-Thought (CoT) reasoning\n* Multi-turn conversational setups for richer data\n* Summaries and diverse QA pairs\n* Incorporation of strong Indian values through **virtual Indian personas**\n* Other augmentations, including translations and language-specific refinements\n\n### The Role of Indic MMLU\n\nBefore deploying LLMs for such Indian-language data generation, we first need to **evaluate their cross-lingual knowledge transfer capabilities**. Indic MMLU serves this purpose by providing a benchmark to test how well models understand Indian languages, rather than merely translating English content.\n\nWhile previous efforts have translated MMLU into Indian languages, none have **released their methodology**. Indic MMLU is the first benchmark to provide:\n\n* **Open recipe and open code** for translations\n* Coverage of **22 Indian languages**\n* A fully transparent and **reproducible evaluation setup**\n\nThis ensures that models can be assessed rigorously for genuine Indian-language understanding, forming a **foundation for culturally-aware synthetic data pipelines** in future research.\n\n---\n\n## What this dataset has on offer\n\nLarge language models are often benchmarked only on English datasets, which leaves open the question of whether they can *actually* understand Indian languages beyond surface-level translation.\n\nThe **Indic MMLU dataset** bridges this gap by:\n\n* Translating the widely used **[CAIS/MMLU](https://huggingface.co/datasets/cais/mmlu) benchmark** into **22 Indian languages**.\n* Enhancing those translations with high-quality open-source LLM refinement.\n* Providing a reproducible evaluation setup for **measuring open-source models across 16 Indic languages (evaluated subset)**.\n\nIf a model’s score on an Indic version is close to its score on the original English MMLU, it indicates **genuine language understanding** and strong cross-lingual generalization.\n\n---\n\n## Coverage\n\n* **Dataset scope:** 22 Indic languages (full benchmark availability).\n* **Evaluation scope:** 16 Indic languages + English (practically evaluated with ~24 open-source LLMs).\n\nThis separation ensures the dataset is **broad and inclusive**, while evaluation focuses on the languages with the most immediate support and resources.\n\n---\n\n## Overview\n\n* **Source benchmark:** CAIS/MMLU (general knowledge QA benchmark).\n* **Target:** 22 Indic languages + English (translated test set).\n* **Workflow:** Machine translation → LLM-based translation enhancement → embedding comparison → Persona based LLM-as-judge Scores → Correlation between Embeddings and LLM as Judge → human/linguist rating and automated evaluation.\n* **Deliverables:** Enhanced translated `.jsonl` datasets, per-language embeddings, cosine similarity reports, teacher/linguist ratings, model evaluation scores and visualizations (radar plots).\n\n---\n\n## Key principles applied\n\n1. **Preserve semantic parity:** Questions remain semantically the same as English; only language changes.\n2. **Quality control:** Model-Based Automated checks (embeddings, math correctness) + human linguist ratings.\n3. **Reproducibility:** Dockerized inference environment, scriptable pipeline, and clear CLI commands.\n4. **Transparency:** We provide similarity metrics to demonstrate translation fidelity and teacher ratings to highlight linguistic quality.\n\n---\n\n## High-level pipeline (what we did)\n\n1. **Translate** the CAIS/MMLU test set to each Indic language using *IndicTrans2* (baseline MT).\n2. **Enhance** those translations using an open-source LLM with task-specific instructions (to correct phrasing, clarify ambiguous translations, preserve math expressions, and maintain answer choices alignment).\n3. **Embed** both the original English items and the (enhanced) translated items with the same embedding model.\n4. **Compare** embeddings using cosine similarity to measure semantic closeness.\n5. **LLM as Judge and Human review**: linguists/teachers rated samples for language quality, math correctness, and coherence.\n6. **Evaluate models**: run ~24 open-source LLMs across all languages using the lm-eval framework and visualize results (radar plots, top-3 per language).\n\n---\n\n## Indic MMLU Construction Algorithm\n\nTo complement the pseudo-code, the following flowchart provides a **visual overview of the Indic MMLU pipeline**. It clearly illustrates the step-by-step workflow, starting from the English MMLU dataset and moving through translation, enhancement, evaluation, and validation:\n\n* **Translation Stage:** The English MMLU instance is translated into Indic languages using specialist MT.\n* **Enhancement Stage:** Translations are refined with large-scale LLMs for improved fluency and alignment.\n* **LLM Based Judging Phase:** Three types of evaluation are conducted — *fluency by native language experts, subject correctness by domain experts, and alignment by coherence judges*.\n* **Validation Stage:** Embedding similarity checks, n-gram metrics (BLEU, ROUGE, ChrF++), and discrepancy detection ensure quality.\n* **Final Output:** A validated Indic MMLU dataset instance ready for benchmarking.\n\n![Indic MMLU Pipeline](assets/indic-mmlu.png)\n\n# Translation Algorithm \n\n```\nInput:\n  original_mmlu\n  target_languages\n  instructions\n\nFor each language L in target_languages:\n    translated = translate(original_mmlu, tgt=L)\n    enhanced = enhance_using_llm(translated, instructions)\n\n    eng_emb = embed(original_mmlu)\n    trans_emb = embed(enhanced)\n    sim_scores = cosine_similarity(eng_emb, trans_emb)\n\n    llm_scores = LLM_as_Judge(original_mmlu, translated)\n\n    consistency = consistency_check(sim_scores, llm_scores)\n\n    If consistency is poor:\n        human_ratings = rate_sample(enhanced)\n    Else:\n        accept enhanced\n\nAggregate all scores:\n    combine sim_scores, llm_scores, human_ratings\n    compute per-language model scores\n\nProduce visualizations:\n    generate charts and plots for cross-language comparison\n\nOutput:\n    per-language evaluation scores\n    visualizations\n\n```\n\nThis visual representation makes it easier to understand how the **algorithmic steps, human evaluation, and automated checks** come together to ensure translation quality and dataset reliability.\n\n---\n\n## Reproducible commands (examples)\n\n> These are the canonical commands used in our pipeline. Replace file paths, ports, and concurrency values as appropriate for your environment.\n\n**Translate**\n\n```bash\npython3 trans_mmlu.py \\\n  --input_file_path test-00000-of-00001_with_ids.jsonl \\\n  --output_file_path mmlu_kn_in.jsonl \\\n  --tgt_lang \"kn\" \\\n  --src_lang \"en\"\n```\n\n**Enhance translations (LLM-based)**\n\n```bash\npython3 async_infr.py \\\n  --input-path mmlu_as_in.jsonl \\\n  --output-file ds_enhance_eval_hi.jsonl \\\n  --instruction-path instruction_prompts.yml \\\n  --task deepseek_enhance_instruct \\\n  --template-fields 'og_question' 'og_choices' 'question' 'answer' 'Hindi' \\\n  --max-concurrency 4096 \\\n  --backend sglang-oai-chat \\\n  --extra-request-body '{\"temperature\":0.7,\"top_p\":0.9,\"top_k\":50,\"max_tokens\":1024}' \\\n  --port 30000\n```\n\n**Generate embeddings**\n\n```bash\npython3 get_embeddings.py \\\n  --input-file test-00000-of-00001_with_ids.jsonl \\\n  --output-file mmlu_qwen3_embd.jsonl \\\n  --server-url http://127.0.0.1:30000 \\\n  --concurrency 4096\n```\n\n**Cosine similarity (English vs translated enhancements)**\n\n```bash\npython3 calculate_cosine_similarity.py \\\n  --input_dir deepseek_enhance \\\n  --output_dir mmmlu/similarity/cosine/enhance/deepseek_enhance \\\n  --english_file mmlu_eng.jsonl\n```\n\n**Linguist / teacher rating (LLM assisted orchestration)**\n\n```bash\npython3 async_infr.py \\\n  --input-path mmlu_as_in.jsonl \\\n  --output-file ds_enhance_eval_as_qwen3_instruct.jsonl \\\n  --instruction-path instruction_prompts.yml \\\n  --task rate_translated_text_linguist \\\n  --template-fields 'og_question' 'og_choices' 'enhanced_text' 'Assamese' \\\n  --max-concurrency 4096 \\\n  --backend sglang-oai-chat \\\n  --extra-request-body '{\"temperature\":0.7,\"top_p\":0.9,\"top_k\":50,\"max_tokens\":1024}' \\\n  --port 30000\n```\n\n**Model evaluation with `lm-eval`**\n\n```bash\n# Example wrapper which runs evaluation for a model snapshot against the Indic MMLU dataset\nbash lm-eval-llm.sh /path/to/model_snapshot /benchmark-result-path\n```\n\n---\n\n## Download Indic MMLU Dataset\n\nThe complete benchmark covering **22 Indic languages + English** is available here:  \n\n- [Download Full Benchmark (PARQUET)](https://github.com/anonymous-submitter0104/iclr-submission/tree/main/opensource-release/Indic%20MMLU)\n\n> ⚠️ Please cite this repository if you use the dataset in your research.\n\n\n---\n\n### Summary of LLM-as-Judge Ratings\n\nTo evaluate translation quality systematically, we employed **LLMs as judges**, operating under three carefully designed expert personas:\n\n* **Math Expert Persona** → Rated mathematical correctness, terminology, and conceptual fidelity.\n* **Linguistic Expert Persona** → Assessed fluency, grammar, idiomatic usage, and naturalness of the translation.\n* **Coherence Expert Persona** → Evaluated semantic alignment, logical flow, and clarity between English and the target language.\n\n**What we measured:**\n\n* Language quality, fluency, and linguistic suitability.\n* Mathematical correctness and fidelity of technical content.\n* Overall coherence and semantic alignment with the English source.\n\n| Language  | Avg Maths Rate | Avg Coherence Rate | Avg Linguist Rate | Total Records |\n|-----------|----------------|--------------------|-------------------|---------------|\n| Hindi\t    | 9.658029\t     | 9.722054\t          | 9.667891\t        | 14042         |\n| Bengali\t  | 9.182561\t     | 9.131505\t          | 8.202293\t        | 14042         |\n| Nepali    | 9.473793       | 9.331505           | 8.402293          | 14042         |\n| Telugu    | 9.220837       | 8.975929           | 7.986256          | 14042         |\n| Oriya     | 9.264421       | 8.944096           | 8.220268          | 14042         |\n| Punjabi   | 9.348027       | 9.049138           | 8.253454          | 14042         |\n| Assamese  | 9.147201       | 9.039097           | 8.133243          | 14042         |\n| Sanskrit  | 8.338485       | 8.112520           | 6.390115          | 14042         |\n| Kannada   | 9.322746       | 9.111736           | 8.131819          | 14042         |\n| Sindhi    | 2.070645       | 3.109956           | 2.478137          | 14042         |\n| Gujrati   | 9.260647       | 9.206523           | 8.368965          | 14042         |\n| Marathi   | 9.301951       | 9.342045           | 8.381926          | 14042         |\n| Tamil     | 9.047856       | 9.042943           | 8.071144          | 14042         |\n| Malayalam | 9.134952       | 8.998148           | 7.956488          | 14042         |\n| Maithili  | 9.263282       | 9.118146           | 8.218772          | 14042         |\n\n**How to read the table:**\n\n* Ratings are on a **1–10 scale**, where higher scores reflect better translation and enhancement quality.\n* Most languages achieved **consistently high averages (>8.9)**, showing strong quality after enhancement.\n* A few outliers suggest areas where further targeted refinement may be beneficial.\n\n---\n\n## Cosine similarity \n\n* Cosine similarity gives a quick automated signal of semantic closeness between the translated/enhanced items and their English originals.\n* Typical mean similarities range from ~0.76 to 0.85 across languages; higher means closer embeddings (better semantic preservation).\n* Use the similarity scores as a triage: low similarity → manual review; high similarity → likely semantically faithful.\n\n| Language               | Mean Similarity | Std Similarity | Min Similarity | Max Similarity |\n|-------------------------|-----------------|----------------|----------------|----------------|\n| mmlu_as_in_qwen3_embed  | 0.8045          | 0.0597         | 0.4644         | 1              |\n| mmlu_bn_in_qwen3_embed  | 0.8133          | 0.0504         | 0.5276         | 1              |\n| mmlu_gu_in_qwen3_embed  | 0.8211          | 0.0535         | 0.4016         | 1              |\n| mmlu_hi_in_qwen3_embed  | 0.8472          | 0.0489         | 0.3688         | 1              |\n| mmlu_kn_in_qwen3_embed  | 0.8106          | 0.0565         | 0.4660         | 1              |\n| mmlu_mai_in_qwen3_embed | 0.8226          | 0.0504         | 0.4936         | 1              |\n| mmlu_ml_in_qwen3_embed  | 0.8158          | 0.0531         | 0.5109         | 1              |\n| mmlu_mr_in_qwen3_embed  | 0.8129          | 0.0513         | 0.5281         | 1              |\n| mmlu_ne_in_qwen3_embed  | 0.8242          | 0.0502         | 0.4802         | 1              |\n| mmlu_or_in_qwen3_embed  | 0.8159          | 0.0555         | 0.4684         | 1              |\n| mmlu_pa_in_qwen3_embed  | 0.8246          | 0.0516         | 0.5450         | 1              |\n| mmlu_sa_in_qwen3_embed  | 0.7912          | 0.0574         | 0.4981         | 1              |\n| mmlu_sdd_in_qwen3_embed | 0.7646          | 0.0735         | 0.3633         | 0.9676         |\n| mmlu_ta_in_qwen3_embed  | 0.7964          | 0.0559         | 0.5242         | 1              |\n| mmlu_te_in_qwen3_embed  | 0.8006          | 0.0524         | 0.5379         | 1              |\n\n---\n\n## Model Evaluation Methodology\n\nWe evaluated **~24 open-source LLMs** across **16 Indic languages** (subset of the 22 total dataset languages) plus English, using the Indic MMLU benchmark.\n\nWorkflow:\n\n1. **Environment Setup:** Dockerized `vllm/vllm-openai` for reproducibility.\n2. **Framework:** `lm-eval` for standardized evaluation.\n3. **Metrics:** per-language accuracy, top-3 rankings per language, aggregate averages.\n4. **Visualization:** radar plots for each language and comparative top-3 plots.\n\n---\n\n## Important Note\n\n* **Indic MMLU Dataset = 22 Indian languages**\n* **Current Model Evaluation = 16 Indian languages + English**\n* Work is ongoing to expand the evaluation to cover all 22 languages.\n\n---\n### Indic MMLU Results \n\n| <div align=\"center\">![Assamese](assets/assamese.png)<br/>*Assamese*</div> | <div align=\"center\">![Bengali](assets/bengali.png)<br/>*Bengali*</div> |\n|---------------------------------------------------------------------------|------------------------------------------------------------------------|\n| <div align=\"center\">![English](assets/English.png)<br/>*English*</div>    | <div align=\"center\">![Gujarati](assets/Gujrati.png)<br/>*Gujarati*</div> |\n| <div align=\"center\">![Hindi](assets/Hindi.png)<br/>*Hindi*</div>          | <div align=\"center\">![Kannada](assets/kannada.png)<br/>*Kannada*</div> |\n| <div align=\"center\">![Maithili](assets/maithili.png)<br/>*Maithili*</div> | <div align=\"center\">![Malayalam](assets/Malayalam.png)<br/>*Malayalam*</div> |\n| <div align=\"center\">![Marathi](assets/marathi.png)<br/>*Marathi*</div>    | <div align=\"center\">![Nepali](assets/Nepali.png)<br/>*Nepali*</div>   |\n| <div align=\"center\">![Oriya](assets/oriya.png)<br/>*Oriya*</div>          | <div align=\"center\">![Punjabi](assets/Punjabi.png)<br/>*Punjabi*</div> |\n| <div align=\"center\">![Sanskrit](assets/sanskrit.png)<br/>*Sanskrit*</div> | <div align=\"center\">![Sindhi](assets/sindhi.png)<br/>*Sindhi*</div>   |\n| <div align=\"center\">![Tamil](assets/Tamil.png)<br/>*Tamil*</div>          | <div align=\"center\">![Telugu](assets/Telugu.png)<br/>*Telugu*</div>   |\n\n\n---\n\n## How to reproduce (quick checklist)\n\n1. Prepare the English CAIS/MMLU test set (`.jsonl` with IDs).\n2. Run `trans_mmlu.py` to produce per-language MT outputs.\n3. Run `async_infr.py` with the enhancement task to obtain human-readable improved translations.\n4. Generate embeddings for both English and enhanced translations.\n5. Run `calculate_cosine_similarity.py` to produce similarity reports.\n6. Optionally run human/linguist rating orchestration for additional quality assessment.\n7. Evaluate models with `lm-eval` and generate visualizations.\n\n---\n\n## Best practices & caveats\n\n* **Math and formatting:** preserve mathematical expressions and numerals during translation—these are critical for correctness.\n* **Choice alignment:** ensure answer choices remain in the same order and map to the same ground-truth labels.\n* **Human review:** automated metrics are helpful but not sufficient; include periodic linguist sampling.\n* **Constrained inference:** use deterministic LLM settings for enhancement (lower temperature, consistent decoding) to reduce hallucination.\n\n---\n\n## Visual assets & tables\n\nWe include per-language radar plots and a consolidated results panel in `plots/`. The README in the repo contains thumbnails; click through for full-resolution visuals. \n\n---\n"}, "62032558a800543c04426b0626e4c953e61b35be": {"t": 1785771714.3951683, "n": 19915, "h": "61a3105234fcbfde", "c": "# Indic MMLU — Benchmarking Open-Source LLMs on Indian Languages\n\nA rigorously prepared, multi-language adaptation of the CAIS/MMLU test set, translated and enhanced for **22 Indic languages**, designed to evaluate open-source LLMs’ real-world ability to understand Indian languages.\nWe additionally conducted large-scale **evaluation on 16 of these Indic languages (plus English)**, producing detailed accuracy reports and per-language visualizations.\n\n---\n# Table of Contents\n\n1. [Overview](#overview)\n2. [Why the Need for Indic MMLU](#why-the-need-for-indic-mmlu)\n   - [Data Distillation for Indian Languages](#data-distillation-for-indian-languages)\n   - [The Role of Indic MMLU](#the-role-of-indic-mmlu)\n3. [What This Dataset Has on Offer](#what-this-dataset-has-on-offer)\n4. [Coverage](#coverage)\n5. [High-Level Pipeline](#high-level-pipeline-what-we-did)\n   - [Indic MMLU Construction Algorithm](#indic-mmlu-construction-algorithm)\n6. [Translation Algorithm](#translation-algorithm)\n7. [Reproducible Commands](#reproducible-commands-examples)\n   - [Translate](#translate)\n   - [Enhance Translations (LLM-based)](#enhance-translations-llm-based)\n   - [Generate Embeddings](#generate-embeddings)\n   - [Cosine Similarity](#cosine-similarity-english-vs-translated-enhancements)\n   - [Linguist / Teacher Rating](#linguist--teacher-rating-llm-assisted-orchestration)\n   - [Model Evaluation with `lm-eval`](#model-evaluation-with-lm-eval)\n8. [Summary of LLM-as-Judge Ratings](#summary-of-llm-as-judge-ratings)\n9. [Cosine Similarity](#cosine-similarity)\n10. [Model Evaluation Methodology](#model-evaluation-methodology)\n11. [Important Note](#important-note)\n12. [Indic MMLU Results](#indic-mmlu-results)\n13. [How to Reproduce (Quick Checklist)](#how-to-reproduce-quick-checklist)\n14. [Best Practices & Caveats](#best-practices--caveats)\n15. [Visual Assets & Tables](#visual-assets--tables)\n\n---\n\n## Why the Need for Indic MMLU\n\nThe rise of large language models (LLMs) has enabled advanced **synthetic data generation**, including:\n\n* Building LLM-based frameworks for **data curation and sandboxing**\n* Generating reasoning and coding examples in non-English languages\n* Creating SFT datasets, as well as off-policy datasets for **RLHF, DPO, GRPO**-style training\n\nTo fully leverage these capabilities for Indian languages, we need LLMs with at least **limited native proficiency** in these languages. Such models allow us to generate high-quality, culturally-aligned datasets, create virtual personas to introduce nuanced constructs, and maintain **full control and flexibility** over the synthetic data pipeline.\n\n### Data Distillation for Indian Languages\n\nData distillation provides a practical approach to achieve this:\n\n* **Easier and cleaner access to world knowledge** from state-of-the-art LLMs\n* **Domain control and flexibility** to shape datasets according to cultural and linguistic requirements\n* Embedding **strong Indian values** into datasets more effectively than via custom pipelines\n* Creating safety datasets that mitigate **anti-Indian stances** and promote positive responses\n\n**Distillation** is the process of generating data from large models to produce **task-specific, high-quality, and culturally-aligned datasets**. Using distillation, we can apply various synthetic data augmentation strategies, such as:\n\n* Chain-of-Thought (CoT) reasoning\n* Multi-turn conversational setups for richer data\n* Summaries and diverse QA pairs\n* Incorporation of strong Indian values through **virtual Indian personas**\n* Other augmentations, including translations and language-specific refinements\n\n### The Role of Indic MMLU\n\nBefore deploying LLMs for such Indian-language data generation, we first need to **evaluate their cross-lingual knowledge transfer capabilities**. Indic MMLU serves this purpose by providing a benchmark to test how well models understand Indian languages, rather than merely translating English content.\n\nWhile previous efforts have translated MMLU into Indian languages, none have **released their methodology**. Indic MMLU is the first benchmark to provide:\n\n* **Open recipe and open code** for translations\n* Coverage of **22 Indian languages**\n* A fully transparent and **reproducible evaluation setup**\n\nThis ensures that models can be assessed rigorously for genuine Indian-language understanding, forming a **foundation for culturally-aware synthetic data pipelines** in future research.\n\n---\n\n## What this dataset has on offer\n\nLarge language models are often benchmarked only on English datasets, which leaves open the question of whether they can *actually* understand Indian languages beyond surface-level translation.\n\nThe **Indic MMLU dataset** bridges this gap by:\n\n* Translating the widely used **[CAIS/MMLU](https://huggingface.co/datasets/cais/mmlu) benchmark** into **22 Indian languages**.\n* Enhancing those translations with high-quality open-source LLM refinement.\n* Providing a reproducible evaluation setup for **measuring open-source models across 16 Indic languages (evaluated subset)**.\n\nIf a model’s score on an Indic version is close to its score on the original English MMLU, it indicates **genuine language understanding** and strong cross-lingual generalization.\n\n---\n\n## Coverage\n\n* **Dataset scope:** 22 Indic languages (full benchmark availability).\n* **Evaluation scope:** 16 Indic languages + English (practically evaluated with ~24 open-source LLMs).\n\nThis separation ensures the dataset is **broad and inclusive**, while evaluation focuses on the languages with the most immediate support and resources.\n\n---\n\n## Overview\n\n* **Source benchmark:** CAIS/MMLU (general knowledge QA benchmark).\n* **Target:** 22 Indic languages + English (translated test set).\n* **Workflow:** Machine translation → LLM-based translation enhancement → embedding comparison → Persona based LLM-as-judge Scores → Correlation between Embeddings and LLM as Judge → human/linguist rating and automated evaluation.\n* **Deliverables:** Enhanced translated `.jsonl` datasets, per-language embeddings, cosine similarity reports, teacher/linguist ratings, model evaluation scores and visualizations (radar plots).\n\n---\n\n## Key principles applied\n\n1. **Preserve semantic parity:** Questions remain semantically the same as English; only language changes.\n2. **Quality control:** Model-Based Automated checks (embeddings, math correctness) + human linguist ratings.\n3. **Reproducibility:** Dockerized inference environment, scriptable pipeline, and clear CLI commands.\n4. **Transparency:** We provide similarity metrics to demonstrate translation fidelity and teacher ratings to highlight linguistic quality.\n\n---\n\n## High-level pipeline (what we did)\n\n1. **Translate** the CAIS/MMLU test set to each Indic language using *IndicTrans2* (baseline MT).\n2. **Enhance** those translations using an open-source LLM with task-specific instructions (to correct phrasing, clarify ambiguous translations, preserve math expressions, and maintain answer choices alignment).\n3. **Embed** both the original English items and the (enhanced) translated items with the same embedding model.\n4. **Compare** embeddings using cosine similarity to measure semantic closeness.\n5. **LLM as Judge and Human review**: linguists/teachers rated samples for language quality, math correctness, and coherence.\n6. **Evaluate models**: run ~24 open-source LLMs across all languages using the lm-eval framework and visualize results (radar plots, top-3 per language).\n\n---\n\n## Indic MMLU Construction Algorithm\n\nTo complement the pseudo-code, the following flowchart provides a **visual overview of the Indic MMLU pipeline**. It clearly illustrates the step-by-step workflow, starting from the English MMLU dataset and moving through translation, enhancement, evaluation, and validation:\n\n* **Translation Stage:** The English MMLU instance is translated into Indic languages using specialist MT.\n* **Enhancement Stage:** Translations are refined with large-scale LLMs for improved fluency and alignment.\n* **LLM Based Judging Phase:** Three types of evaluation are conducted — *fluency by native language experts, subject correctness by domain experts, and alignment by coherence judges*.\n* **Validation Stage:** Embedding similarity checks, n-gram metrics (BLEU, ROUGE, ChrF++), and discrepancy detection ensure quality.\n* **Final Output:** A validated Indic MMLU dataset instance ready for benchmarking.\n\n![Indic MMLU Pipeline](assets/indic-mmlu.png)\n\n# Translation Algorithm \n\n```\nInput:\n  original_mmlu\n  target_languages\n  instructions\n\nFor each language L in target_languages:\n    translated = translate(original_mmlu, tgt=L)\n    enhanced = enhance_using_llm(translated, instructions)\n\n    eng_emb = embed(original_mmlu)\n    trans_emb = embed(enhanced)\n    sim_scores = cosine_similarity(eng_emb, trans_emb)\n\n    llm_scores = LLM_as_Judge(original_mmlu, translated)\n\n    consistency = consistency_check(sim_scores, llm_scores)\n\n    If consistency is poor:\n        human_ratings = rate_sample(enhanced)\n    Else:\n        accept enhanced\n\nAggregate all scores:\n    combine sim_scores, llm_scores, human_ratings\n    compute per-language model scores\n\nProduce visualizations:\n    generate charts and plots for cross-language comparison\n\nOutput:\n    per-language evaluation scores\n    visualizations\n\n```\n\nThis visual representation makes it easier to understand how the **algorithmic steps, human evaluation, and automated checks** come together to ensure translation quality and dataset reliability.\n\n---\n\n## Download Indic MMLU Dataset\n\nThe complete benchmark covering **22 Indic languages + English** is available here:  \n\n- [Download Full Benchmark (PARQUET)](https://github.com/anonymous-submitter0104/iclr-submission/tree/main/opensource-release/Indic%20MMLU)\n\n> ⚠️ Please cite this repository if you use the dataset in your research.\n---\n\n## Reproducible commands (examples)\n\n> These are the canonical commands used in our pipeline. Replace file paths, ports, and concurrency values as appropriate for your environment.\n\n**Translate**\n\n```bash\npython3 trans_mmlu.py \\\n  --input_file_path test-00000-of-00001_with_ids.jsonl \\\n  --output_file_path mmlu_kn_in.jsonl \\\n  --tgt_lang \"kn\" \\\n  --src_lang \"en\"\n```\n\n**Enhance translations (LLM-based)**\n\n```bash\npython3 async_infr.py \\\n  --input-path mmlu_as_in.jsonl \\\n  --output-file ds_enhance_eval_hi.jsonl \\\n  --instruction-path instruction_prompts.yml \\\n  --task deepseek_enhance_instruct \\\n  --template-fields 'og_question' 'og_choices' 'question' 'answer' 'Hindi' \\\n  --max-concurrency 4096 \\\n  --backend sglang-oai-chat \\\n  --extra-request-body '{\"temperature\":0.7,\"top_p\":0.9,\"top_k\":50,\"max_tokens\":1024}' \\\n  --port 30000\n```\n\n**Generate embeddings**\n\n```bash\npython3 get_embeddings.py \\\n  --input-file test-00000-of-00001_with_ids.jsonl \\\n  --output-file mmlu_qwen3_embd.jsonl \\\n  --server-url http://127.0.0.1:30000 \\\n  --concurrency 4096\n```\n\n**Cosine similarity (English vs translated enhancements)**\n\n```bash\npython3 calculate_cosine_similarity.py \\\n  --input_dir deepseek_enhance \\\n  --output_dir mmmlu/similarity/cosine/enhance/deepseek_enhance \\\n  --english_file mmlu_eng.jsonl\n```\n\n**Linguist / teacher rating (LLM assisted orchestration)**\n\n```bash\npython3 async_infr.py \\\n  --input-path mmlu_as_in.jsonl \\\n  --output-file ds_enhance_eval_as_qwen3_instruct.jsonl \\\n  --instruction-path instruction_prompts.yml \\\n  --task rate_translated_text_linguist \\\n  --template-fields 'og_question' 'og_choices' 'enhanced_text' 'Assamese' \\\n  --max-concurrency 4096 \\\n  --backend sglang-oai-chat \\\n  --extra-request-body '{\"temperature\":0.7,\"top_p\":0.9,\"top_k\":50,\"max_tokens\":1024}' \\\n  --port 30000\n```\n\n**Model evaluation with `lm-eval`**\n\n```bash\n# Example wrapper which runs evaluation for a model snapshot against the Indic MMLU dataset\nbash lm-eval-llm.sh /path/to/model_snapshot /benchmark-result-path\n```\n\n---\n\n### Summary of LLM-as-Judge Ratings\n\nTo evaluate translation quality systematically, we employed **LLMs as judges**, operating under three carefully designed expert personas:\n\n* **Math Expert Persona** → Rated mathematical correctness, terminology, and conceptual fidelity.\n* **Linguistic Expert Persona** → Assessed fluency, grammar, idiomatic usage, and naturalness of the translation.\n* **Coherence Expert Persona** → Evaluated semantic alignment, logical flow, and clarity between English and the target language.\n\n**What we measured:**\n\n* Language quality, fluency, and linguistic suitability.\n* Mathematical correctness and fidelity of technical content.\n* Overall coherence and semantic alignment with the English source.\n\n| Language  | Avg Maths Rate | Avg Coherence Rate | Avg Linguist Rate | Total Records |\n|-----------|----------------|--------------------|-------------------|---------------|\n| Hindi\t    | 9.658029\t     | 9.722054\t          | 9.667891\t        | 14042         |\n| Bengali\t  | 9.182561\t     | 9.131505\t          | 8.202293\t        | 14042         |\n| Nepali    | 9.473793       | 9.331505           | 8.402293          | 14042         |\n| Telugu    | 9.220837       | 8.975929           | 7.986256          | 14042         |\n| Oriya     | 9.264421       | 8.944096           | 8.220268          | 14042         |\n| Punjabi   | 9.348027       | 9.049138           | 8.253454          | 14042         |\n| Assamese  | 9.147201       | 9.039097           | 8.133243          | 14042         |\n| Sanskrit  | 8.338485       | 8.112520           | 6.390115          | 14042         |\n| Kannada   | 9.322746       | 9.111736           | 8.131819          | 14042         |\n| Sindhi    | 2.070645       | 3.109956           | 2.478137          | 14042         |\n| Gujrati   | 9.260647       | 9.206523           | 8.368965          | 14042         |\n| Marathi   | 9.301951       | 9.342045           | 8.381926          | 14042         |\n| Tamil     | 9.047856       | 9.042943           | 8.071144          | 14042         |\n| Malayalam | 9.134952       | 8.998148           | 7.956488          | 14042         |\n| Maithili  | 9.263282       | 9.118146           | 8.218772          | 14042         |\n\n**How to read the table:**\n\n* Ratings are on a **1–10 scale**, where higher scores reflect better translation and enhancement quality.\n* Most languages achieved **consistently high averages (>8.9)**, showing strong quality after enhancement.\n* A few outliers suggest areas where further targeted refinement may be beneficial.\n\n---\n\n## Cosine similarity \n\n* Cosine similarity gives a quick automated signal of semantic closeness between the translated/enhanced items and their English originals.\n* Typical mean similarities range from ~0.76 to 0.85 across languages; higher means closer embeddings (better semantic preservation).\n* Use the similarity scores as a triage: low similarity → manual review; high similarity → likely semantically faithful.\n\n| Language               | Mean Similarity | Std Similarity | Min Similarity | Max Similarity |\n|-------------------------|-----------------|----------------|----------------|----------------|\n| mmlu_as_in_qwen3_embed  | 0.8045          | 0.0597         | 0.4644         | 1              |\n| mmlu_bn_in_qwen3_embed  | 0.8133          | 0.0504         | 0.5276         | 1              |\n| mmlu_gu_in_qwen3_embed  | 0.8211          | 0.0535         | 0.4016         | 1              |\n| mmlu_hi_in_qwen3_embed  | 0.8472          | 0.0489         | 0.3688         | 1              |\n| mmlu_kn_in_qwen3_embed  | 0.8106          | 0.0565         | 0.4660         | 1              |\n| mmlu_mai_in_qwen3_embed | 0.8226          | 0.0504         | 0.4936         | 1              |\n| mmlu_ml_in_qwen3_embed  | 0.8158          | 0.0531         | 0.5109         | 1              |\n| mmlu_mr_in_qwen3_embed  | 0.8129          | 0.0513         | 0.5281         | 1              |\n| mmlu_ne_in_qwen3_embed  | 0.8242          | 0.0502         | 0.4802         | 1              |\n| mmlu_or_in_qwen3_embed  | 0.8159          | 0.0555         | 0.4684         | 1              |\n| mmlu_pa_in_qwen3_embed  | 0.8246          | 0.0516         | 0.5450         | 1              |\n| mmlu_sa_in_qwen3_embed  | 0.7912          | 0.0574         | 0.4981         | 1              |\n| mmlu_sdd_in_qwen3_embed | 0.7646          | 0.0735         | 0.3633         | 0.9676         |\n| mmlu_ta_in_qwen3_embed  | 0.7964          | 0.0559         | 0.5242         | 1              |\n| mmlu_te_in_qwen3_embed  | 0.8006          | 0.0524         | 0.5379         | 1              |\n\n---\n\n## Model Evaluation Methodology\n\nWe evaluated **~24 open-source LLMs** across **16 Indic languages** (subset of the 22 total dataset languages) plus English, using the Indic MMLU benchmark.\n\nWorkflow:\n\n1. **Environment Setup:** Dockerized `vllm/vllm-openai` for reproducibility.\n2. **Framework:** `lm-eval` for standardized evaluation.\n3. **Metrics:** per-language accuracy, top-3 rankings per language, aggregate averages.\n4. **Visualization:** radar plots for each language and comparative top-3 plots.\n\n---\n\n## Important Note\n\n* **Indic MMLU Dataset = 22 Indian languages**\n* **Current Model Evaluation = 16 Indian languages + English**\n* Work is ongoing to expand the evaluation to cover all 22 languages.\n\n---\n### Indic MMLU Results \n\n| <div align=\"center\">![Assamese](assets/assamese.png)<br/>*Assamese*</div> | <div align=\"center\">![Bengali](assets/bengali.png)<br/>*Bengali*</div> |\n|---------------------------------------------------------------------------|------------------------------------------------------------------------|\n| <div align=\"center\">![English](assets/English.png)<br/>*English*</div>    | <div align=\"center\">![Gujarati](assets/Gujrati.png)<br/>*Gujarati*</div> |\n| <div align=\"center\">![Hindi](assets/Hindi.png)<br/>*Hindi*</div>          | <div align=\"center\">![Kannada](assets/kannada.png)<br/>*Kannada*</div> |\n| <div align=\"center\">![Maithili](assets/maithili.png)<br/>*Maithili*</div> | <div align=\"center\">![Malayalam](assets/Malayalam.png)<br/>*Malayalam*</div> |\n| <div align=\"center\">![Marathi](assets/marathi.png)<br/>*Marathi*</div>    | <div align=\"center\">![Nepali](assets/Nepali.png)<br/>*Nepali*</div>   |\n| <div align=\"center\">![Oriya](assets/oriya.png)<br/>*Oriya*</div>          | <div align=\"center\">![Punjabi](assets/Punjabi.png)<br/>*Punjabi*</div> |\n| <div align=\"center\">![Sanskrit](assets/sanskrit.png)<br/>*Sanskrit*</div> | <div align=\"center\">![Sindhi](assets/sindhi.png)<br/>*Sindhi*</div>   |\n| <div align=\"center\">![Tamil](assets/Tamil.png)<br/>*Tamil*</div>          | <div align=\"center\">![Telugu](assets/Telugu.png)<br/>*Telugu*</div>   |\n\n\n---\n\n## How to reproduce (quick checklist)\n\n1. Prepare the English CAIS/MMLU test set (`.jsonl` with IDs).\n2. Run `trans_mmlu.py` to produce per-language MT outputs.\n3. Run `async_infr.py` with the enhancement task to obtain human-readable improved translations.\n4. Generate embeddings for both English and enhanced translations.\n5. Run `calculate_cosine_similarity.py` to produce similarity reports.\n6. Optionally run human/linguist rating orchestration for additional quality assessment.\n7. Evaluate models with `lm-eval` and generate visualizations.\n\n---\n\n## Best practices & caveats\n\n* **Math and formatting:** preserve mathematical expressions and numerals during translation—these are critical for correctness.\n* **Choice alignment:** ensure answer choices remain in the same order and map to the same ground-truth labels.\n* **Human review:** automated metrics are helpful but not sufficient; include periodic linguist sampling.\n* **Constrained inference:** use deterministic LLM settings for enhancement (lower temperature, consistent decoding) to reduce hallucination.\n\n---\n\n## Visual assets & tables\n\nWe include per-language radar plots and a consolidated results panel in `plots/`. The README in the repo contains thumbnails; click through for full-resolution visuals. \n\n---\n"}, "ff114c9057111f6c10ae31fe58b8a55cd8180bc3": {"t": 1785771715.6367164, "n": 19576, "h": "33bc81e3ba20e406", "c": "# Indic MMLU — Benchmarking Open-Source LLMs on Indian Languages\n\nA rigorously prepared, multi-language adaptation of the CAIS/MMLU test set, translated and enhanced for **22 Indic languages**, designed to evaluate open-source LLMs’ real-world ability to understand Indian languages.\nWe additionally conducted large-scale **evaluation on 16 of these Indic languages (plus English)**, producing detailed accuracy reports and per-language visualizations.\n\n---\n# Table of Contents\n\n1. [Overview](#overview)\n2. [Why the Need for Indic MMLU](#why-the-need-for-indic-mmlu)\n   - [Data Distillation for Indian Languages](#data-distillation-for-indian-languages)\n   - [The Role of Indic MMLU](#the-role-of-indic-mmlu)\n3. [What This Dataset Has on Offer](#what-this-dataset-has-on-offer)\n4. [Coverage](#coverage)\n5. [High-Level Pipeline](#high-level-pipeline-what-we-did)\n   - [Indic MMLU Construction Algorithm](#indic-mmlu-construction-algorithm)\n6. [Translation Algorithm](#translation-algorithm)\n7. [Reproducible Commands](#reproducible-commands-examples)\n   - [Translate](#translate)\n   - [Enhance Translations (LLM-based)](#enhance-translations-llm-based)\n   - [Generate Embeddings](#generate-embeddings)\n   - [Cosine Similarity](#cosine-similarity-english-vs-translated-enhancements)\n   - [Linguist / Teacher Rating](#linguist--teacher-rating-llm-assisted-orchestration)\n   - [Model Evaluation with `lm-eval`](#model-evaluation-with-lm-eval)\n8. [Summary of LLM-as-Judge Ratings](#summary-of-llm-as-judge-ratings)\n9. [Cosine Similarity](#cosine-similarity)\n10. [Model Evaluation Methodology](#model-evaluation-methodology)\n11. [Important Note](#important-note)\n12. [Indic MMLU Results](#indic-mmlu-results)\n13. [How to Reproduce (Quick Checklist)](#how-to-reproduce-quick-checklist)\n14. [Best Practices & Caveats](#best-practices--caveats)\n15. [Visual Assets & Tables](#visual-assets--tables)\n\n---\n\n## Why the Need for Indic MMLU\n\nThe rise of large language models (LLMs) has enabled advanced **synthetic data generation**, including:\n\n* Building LLM-based frameworks for **data curation and sandboxing**\n* Generating reasoning and coding examples in non-English languages\n* Creating SFT datasets, as well as off-policy datasets for **RLHF, DPO, GRPO**-style training\n\nTo fully leverage these capabilities for Indian languages, we need LLMs with at least **limited native proficiency** in these languages. Such models allow us to generate high-quality, culturally-aligned datasets, create virtual personas to introduce nuanced constructs, and maintain **full control and flexibility** over the synthetic data pipeline.\n\n### Data Distillation for Indian Languages\n\nData distillation provides a practical approach to achieve this:\n\n* **Easier and cleaner access to world knowledge** from state-of-the-art LLMs\n* **Domain control and flexibility** to shape datasets according to cultural and linguistic requirements\n* Embedding **strong Indian values** into datasets more effectively than via custom pipelines\n* Creating safety datasets that mitigate **anti-Indian stances** and promote positive responses\n\n**Distillation** is the process of generating data from large models to produce **task-specific, high-quality, and culturally-aligned datasets**. Using distillation, we can apply various synthetic data augmentation strategies, such as:\n\n* Chain-of-Thought (CoT) reasoning\n* Multi-turn conversational setups for richer data\n* Summaries and diverse QA pairs\n* Incorporation of strong Indian values through **virtual Indian personas**\n* Other augmentations, including translations and language-specific refinements\n\n### The Role of Indic MMLU\n\nBefore deploying LLMs for such Indian-language data generation, we first need to **evaluate their cross-lingual knowledge transfer capabilities**. Indic MMLU serves this purpose by providing a benchmark to test how well models understand Indian languages, rather than merely translating English content.\n\nWhile previous efforts have translated MMLU into Indian languages, none have **released their methodology**. Indic MMLU is the first benchmark to provide:\n\n* **Open recipe and open code** for translations\n* Coverage of **22 Indian languages**\n* A fully transparent and **reproducible evaluation setup**\n\nThis ensures that models can be assessed rigorously for genuine Indian-language understanding, forming a **foundation for culturally-aware synthetic data pipelines** in future research.\n\n---\n\n## What this dataset has on offer\n\nLarge language models are often benchmarked only on English datasets, which leaves open the question of whether they can *actually* understand Indian languages beyond surface-level translation.\n\nThe **Indic MMLU dataset** bridges this gap by:\n\n* Translating the widely used **[CAIS/MMLU](https://huggingface.co/datasets/cais/mmlu) benchmark** into **22 Indian languages**.\n* Enhancing those translations with high-quality open-source LLM refinement.\n* Providing a reproducible evaluation setup for **measuring open-source models across 16 Indic languages (evaluated subset)**.\n\nIf a model’s score on an Indic version is close to its score on the original English MMLU, it indicates **genuine language understanding** and strong cross-lingual generalization.\n\n---\n\n## Coverage\n\n* **Dataset scope:** 22 Indic languages (full benchmark availability).\n* **Evaluation scope:** 16 Indic languages + English (practically evaluated with ~24 open-source LLMs).\n\nThis separation ensures the dataset is **broad and inclusive**, while evaluation focuses on the languages with the most immediate support and resources.\n\n---\n\n## Overview\n\n* **Source benchmark:** CAIS/MMLU (general knowledge QA benchmark).\n* **Target:** 22 Indic languages + English (translated test set).\n* **Workflow:** Machine translation → LLM-based translation enhancement → embedding comparison → Persona based LLM-as-judge Scores → Correlation between Embeddings and LLM as Judge → human/linguist rating and automated evaluation.\n* **Deliverables:** Enhanced translated `.jsonl` datasets, per-language embeddings, cosine similarity reports, teacher/linguist ratings, model evaluation scores and visualizations (radar plots).\n\n---\n\n## Key principles applied\n\n1. **Preserve semantic parity:** Questions remain semantically the same as English; only language changes.\n2. **Quality control:** Model-Based Automated checks (embeddings, math correctness) + human linguist ratings.\n3. **Reproducibility:** Dockerized inference environment, scriptable pipeline, and clear CLI commands.\n4. **Transparency:** We provide similarity metrics to demonstrate translation fidelity and teacher ratings to highlight linguistic quality.\n\n---\n\n## High-level pipeline (what we did)\n\n1. **Translate** the CAIS/MMLU test set to each Indic language using *IndicTrans2* (baseline MT).\n2. **Enhance** those translations using an open-source LLM with task-specific instructions (to correct phrasing, clarify ambiguous translations, preserve math expressions, and maintain answer choices alignment).\n3. **Embed** both the original English items and the (enhanced) translated items with the same embedding model.\n4. **Compare** embeddings using cosine similarity to measure semantic closeness.\n5. **LLM as Judge and Human review**: linguists/teachers rated samples for language quality, math correctness, and coherence.\n6. **Evaluate models**: run ~24 open-source LLMs across all languages using the lm-eval framework and visualize results (radar plots, top-3 per language).\n\n---\n\n## Indic MMLU Construction Algorithm\n\nTo complement the pseudo-code, the following flowchart provides a **visual overview of the Indic MMLU pipeline**. It clearly illustrates the step-by-step workflow, starting from the English MMLU dataset and moving through translation, enhancement, evaluation, and validation:\n\n* **Translation Stage:** The English MMLU instance is translated into Indic languages using specialist MT.\n* **Enhancement Stage:** Translations are refined with large-scale LLMs for improved fluency and alignment.\n* **LLM Based Judging Phase:** Three types of evaluation are conducted — *fluency by native language experts, subject correctness by domain experts, and alignment by coherence judges*.\n* **Validation Stage:** Embedding similarity checks, n-gram metrics (BLEU, ROUGE, ChrF++), and discrepancy detection ensure quality.\n* **Final Output:** A validated Indic MMLU dataset instance ready for benchmarking.\n\n![Indic MMLU Pipeline](assets/indic-mmlu.png)\n\n# Translation Algorithm \n\n```\nInput:\n  original_mmlu\n  target_languages\n  instructions\n\nFor each language L in target_languages:\n    translated = translate(original_mmlu, tgt=L)\n    enhanced = enhance_using_llm(translated, instructions)\n\n    eng_emb = embed(original_mmlu)\n    trans_emb = embed(enhanced)\n    sim_scores = cosine_similarity(eng_emb, trans_emb)\n\n    llm_scores = LLM_as_Judge(original_mmlu, translated)\n\n    consistency = consistency_check(sim_scores, llm_scores)\n\n    If consistency is poor:\n        human_ratings = rate_sample(enhanced)\n    Else:\n        accept enhanced\n\nAggregate all scores:\n    combine sim_scores, llm_scores, human_ratings\n    compute per-language model scores\n\nProduce visualizations:\n    generate charts and plots for cross-language comparison\n\nOutput:\n    per-language evaluation scores\n    visualizations\n\n```\n\nThis visual representation makes it easier to understand how the **algorithmic steps, human evaluation, and automated checks** come together to ensure translation quality and dataset reliability.\n\n---\n\n## Reproducible commands (examples)\n\n> These are the canonical commands used in our pipeline. Replace file paths, ports, and concurrency values as appropriate for your environment.\n\n**Translate**\n\n```bash\npython3 trans_mmlu.py \\\n  --input_file_path test-00000-of-00001_with_ids.jsonl \\\n  --output_file_path mmlu_kn_in.jsonl \\\n  --tgt_lang \"kn\" \\\n  --src_lang \"en\"\n```\n\n**Enhance translations (LLM-based)**\n\n```bash\npython3 async_infr.py \\\n  --input-path mmlu_as_in.jsonl \\\n  --output-file ds_enhance_eval_hi.jsonl \\\n  --instruction-path instruction_prompts.yml \\\n  --task deepseek_enhance_instruct \\\n  --template-fields 'og_question' 'og_choices' 'question' 'answer' 'Hindi' \\\n  --max-concurrency 4096 \\\n  --backend sglang-oai-chat \\\n  --extra-request-body '{\"temperature\":0.7,\"top_p\":0.9,\"top_k\":50,\"max_tokens\":1024}' \\\n  --port 30000\n```\n\n**Generate embeddings**\n\n```bash\npython3 get_embeddings.py \\\n  --input-file test-00000-of-00001_with_ids.jsonl \\\n  --output-file mmlu_qwen3_embd.jsonl \\\n  --server-url http://127.0.0.1:30000 \\\n  --concurrency 4096\n```\n\n**Cosine similarity (English vs translated enhancements)**\n\n```bash\npython3 calculate_cosine_similarity.py \\\n  --input_dir deepseek_enhance \\\n  --output_dir mmmlu/similarity/cosine/enhance/deepseek_enhance \\\n  --english_file mmlu_eng.jsonl\n```\n\n**Linguist / teacher rating (LLM assisted orchestration)**\n\n```bash\npython3 async_infr.py \\\n  --input-path mmlu_as_in.jsonl \\\n  --output-file ds_enhance_eval_as_qwen3_instruct.jsonl \\\n  --instruction-path instruction_prompts.yml \\\n  --task rate_translated_text_linguist \\\n  --template-fields 'og_question' 'og_choices' 'enhanced_text' 'Assamese' \\\n  --max-concurrency 4096 \\\n  --backend sglang-oai-chat \\\n  --extra-request-body '{\"temperature\":0.7,\"top_p\":0.9,\"top_k\":50,\"max_tokens\":1024}' \\\n  --port 30000\n```\n\n**Model evaluation with `lm-eval`**\n\n```bash\n# Example wrapper which runs evaluation for a model snapshot against the Indic MMLU dataset\nbash lm-eval-llm.sh /path/to/model_snapshot /benchmark-result-path\n```\n\n---\n\n### Summary of LLM-as-Judge Ratings\n\nTo evaluate translation quality systematically, we employed **LLMs as judges**, operating under three carefully designed expert personas:\n\n* **Math Expert Persona** → Rated mathematical correctness, terminology, and conceptual fidelity.\n* **Linguistic Expert Persona** → Assessed fluency, grammar, idiomatic usage, and naturalness of the translation.\n* **Coherence Expert Persona** → Evaluated semantic alignment, logical flow, and clarity between English and the target language.\n\n**What we measured:**\n\n* Language quality, fluency, and linguistic suitability.\n* Mathematical correctness and fidelity of technical content.\n* Overall coherence and semantic alignment with the English source.\n\n| Language  | Avg Maths Rate | Avg Coherence Rate | Avg Linguist Rate | Total Records |\n|-----------|----------------|--------------------|-------------------|---------------|\n| Hindi\t    | 9.658029\t     | 9.722054\t          | 9.667891\t        | 14042         |\n| Bengali\t  | 9.182561\t     | 9.131505\t          | 8.202293\t        | 14042         |\n| Nepali    | 9.473793       | 9.331505           | 8.402293          | 14042         |\n| Telugu    | 9.220837       | 8.975929           | 7.986256          | 14042         |\n| Oriya     | 9.264421       | 8.944096           | 8.220268          | 14042         |\n| Punjabi   | 9.348027       | 9.049138           | 8.253454          | 14042         |\n| Assamese  | 9.147201       | 9.039097           | 8.133243          | 14042         |\n| Sanskrit  | 8.338485       | 8.112520           | 6.390115          | 14042         |\n| Kannada   | 9.322746       | 9.111736           | 8.131819          | 14042         |\n| Sindhi    | 2.070645       | 3.109956           | 2.478137          | 14042         |\n| Gujrati   | 9.260647       | 9.206523           | 8.368965          | 14042         |\n| Marathi   | 9.301951       | 9.342045           | 8.381926          | 14042         |\n| Tamil     | 9.047856       | 9.042943           | 8.071144          | 14042         |\n| Malayalam | 9.134952       | 8.998148           | 7.956488          | 14042         |\n| Maithili  | 9.263282       | 9.118146           | 8.218772          | 14042         |\n\n**How to read the table:**\n\n* Ratings are on a **1–10 scale**, where higher scores reflect better translation and enhancement quality.\n* Most languages achieved **consistently high averages (>8.9)**, showing strong quality after enhancement.\n* A few outliers suggest areas where further targeted refinement may be beneficial.\n\n---\n\n## Cosine similarity \n\n* Cosine similarity gives a quick automated signal of semantic closeness between the translated/enhanced items and their English originals.\n* Typical mean similarities range from ~0.76 to 0.85 across languages; higher means closer embeddings (better semantic preservation).\n* Use the similarity scores as a triage: low similarity → manual review; high similarity → likely semantically faithful.\n\n| Language               | Mean Similarity | Std Similarity | Min Similarity | Max Similarity |\n|-------------------------|-----------------|----------------|----------------|----------------|\n| mmlu_as_in_qwen3_embed  | 0.8045          | 0.0597         | 0.4644         | 1              |\n| mmlu_bn_in_qwen3_embed  | 0.8133          | 0.0504         | 0.5276         | 1              |\n| mmlu_gu_in_qwen3_embed  | 0.8211          | 0.0535         | 0.4016         | 1              |\n| mmlu_hi_in_qwen3_embed  | 0.8472          | 0.0489         | 0.3688         | 1              |\n| mmlu_kn_in_qwen3_embed  | 0.8106          | 0.0565         | 0.4660         | 1              |\n| mmlu_mai_in_qwen3_embed | 0.8226          | 0.0504         | 0.4936         | 1              |\n| mmlu_ml_in_qwen3_embed  | 0.8158          | 0.0531         | 0.5109         | 1              |\n| mmlu_mr_in_qwen3_embed  | 0.8129          | 0.0513         | 0.5281         | 1              |\n| mmlu_ne_in_qwen3_embed  | 0.8242          | 0.0502         | 0.4802         | 1              |\n| mmlu_or_in_qwen3_embed  | 0.8159          | 0.0555         | 0.4684         | 1              |\n| mmlu_pa_in_qwen3_embed  | 0.8246          | 0.0516         | 0.5450         | 1              |\n| mmlu_sa_in_qwen3_embed  | 0.7912          | 0.0574         | 0.4981         | 1              |\n| mmlu_sdd_in_qwen3_embed | 0.7646          | 0.0735         | 0.3633         | 0.9676         |\n| mmlu_ta_in_qwen3_embed  | 0.7964          | 0.0559         | 0.5242         | 1              |\n| mmlu_te_in_qwen3_embed  | 0.8006          | 0.0524         | 0.5379         | 1              |\n\n---\n\n## Model Evaluation Methodology\n\nWe evaluated **~24 open-source LLMs** across **16 Indic languages** (subset of the 22 total dataset languages) plus English, using the Indic MMLU benchmark.\n\nWorkflow:\n\n1. **Environment Setup:** Dockerized `vllm/vllm-openai` for reproducibility.\n2. **Framework:** `lm-eval` for standardized evaluation.\n3. **Metrics:** per-language accuracy, top-3 rankings per language, aggregate averages.\n4. **Visualization:** radar plots for each language and comparative top-3 plots.\n\n---\n\n## Important Note\n\n* **Indic MMLU Dataset = 22 Indian languages**\n* **Current Model Evaluation = 16 Indian languages + English**\n* Work is ongoing to expand the evaluation to cover all 22 languages.\n\n---\n### Indic MMLU Results \n\n| <div align=\"center\">![Assamese](assets/assamese.png)<br/>*Assamese*</div> | <div align=\"center\">![Bengali](assets/bengali.png)<br/>*Bengali*</div> |\n|---------------------------------------------------------------------------|------------------------------------------------------------------------|\n| <div align=\"center\">![English](assets/English.png)<br/>*English*</div>    | <div align=\"center\">![Gujarati](assets/Gujrati.png)<br/>*Gujarati*</div> |\n| <div align=\"center\">![Hindi](assets/Hindi.png)<br/>*Hindi*</div>          | <div align=\"center\">![Kannada](assets/kannada.png)<br/>*Kannada*</div> |\n| <div align=\"center\">![Maithili](assets/maithili.png)<br/>*Maithili*</div> | <div align=\"center\">![Malayalam](assets/Malayalam.png)<br/>*Malayalam*</div> |\n| <div align=\"center\">![Marathi](assets/marathi.png)<br/>*Marathi*</div>    | <div align=\"center\">![Nepali](assets/Nepali.png)<br/>*Nepali*</div>   |\n| <div align=\"center\">![Oriya](assets/oriya.png)<br/>*Oriya*</div>          | <div align=\"center\">![Punjabi](assets/Punjabi.png)<br/>*Punjabi*</div> |\n| <div align=\"center\">![Sanskrit](assets/sanskrit.png)<br/>*Sanskrit*</div> | <div align=\"center\">![Sindhi](assets/sindhi.png)<br/>*Sindhi*</div>   |\n| <div align=\"center\">![Tamil](assets/Tamil.png)<br/>*Tamil*</div>          | <div align=\"center\">![Telugu](assets/Telugu.png)<br/>*Telugu*</div>   |\n\n\n---\n\n## How to reproduce (quick checklist)\n\n1. Prepare the English CAIS/MMLU test set (`.jsonl` with IDs).\n2. Run `trans_mmlu.py` to produce per-language MT outputs.\n3. Run `async_infr.py` with the enhancement task to obtain human-readable improved translations.\n4. Generate embeddings for both English and enhanced translations.\n5. Run `calculate_cosine_similarity.py` to produce similarity reports.\n6. Optionally run human/linguist rating orchestration for additional quality assessment.\n7. Evaluate models with `lm-eval` and generate visualizations.\n\n---\n\n## Best practices & caveats\n\n* **Math and formatting:** preserve mathematical expressions and numerals during translation—these are critical for correctness.\n* **Choice alignment:** ensure answer choices remain in the same order and map to the same ground-truth labels.\n* **Human review:** automated metrics are helpful but not sufficient; include periodic linguist sampling.\n* **Constrained inference:** use deterministic LLM settings for enhancement (lower temperature, consistent decoding) to reduce hallucination.\n\n---\n\n## Visual assets & tables\n\nWe include per-language radar plots and a consolidated results panel in `plots/`. The README in the repo contains thumbnails; click through for full-resolution visuals. \n\n---\n"}, "e605f3ce24f64a26ea34d58a8801b909d5ed812e": {"t": 1785771717.0582702, "n": 19772, "h": "40f3a541f95fd8b6", "c": "# Indic MMLU — Benchmarking Open-Source LLMs on Indian Languages\n\nA rigorously prepared, multi-language adaptation of the CAIS/MMLU test set, translated and enhanced for **22 Indic languages**, designed to evaluate open-source LLMs’ real-world ability to understand Indian languages.\nWe additionally conducted large-scale **evaluation on 16 of these Indic languages (plus English)**, producing detailed accuracy reports and per-language visualizations.\n\n---\n\n# Open Source Release \n[Click here to view Release](https://github.com/anonymous-submitter0104/iclr-submission/tree/66bfdab3aa2f7b44bc3ccd67bf9b2ea7c3450d5f/opensource-release/Indic%20MMLU)\n\n---\n# Table of Contents\n\n1. [Overview](#overview)\n2. [Why the Need for Indic MMLU](#why-the-need-for-indic-mmlu)\n   - [Data Distillation for Indian Languages](#data-distillation-for-indian-languages)\n   - [The Role of Indic MMLU](#the-role-of-indic-mmlu)\n3. [What This Dataset Has on Offer](#what-this-dataset-has-on-offer)\n4. [Coverage](#coverage)\n5. [High-Level Pipeline](#high-level-pipeline-what-we-did)\n   - [Indic MMLU Construction Algorithm](#indic-mmlu-construction-algorithm)\n6. [Translation Algorithm](#translation-algorithm)\n7. [Reproducible Commands](#reproducible-commands-examples)\n   - [Translate](#translate)\n   - [Enhance Translations (LLM-based)](#enhance-translations-llm-based)\n   - [Generate Embeddings](#generate-embeddings)\n   - [Cosine Similarity](#cosine-similarity-english-vs-translated-enhancements)\n   - [Linguist / Teacher Rating](#linguist--teacher-rating-llm-assisted-orchestration)\n   - [Model Evaluation with `lm-eval`](#model-evaluation-with-lm-eval)\n8. [Summary of LLM-as-Judge Ratings](#summary-of-llm-as-judge-ratings)\n9. [Cosine Similarity](#cosine-similarity)\n10. [Model Evaluation Methodology](#model-evaluation-methodology)\n11. [Important Note](#important-note)\n12. [Indic MMLU Results](#indic-mmlu-results)\n13. [How to Reproduce (Quick Checklist)](#how-to-reproduce-quick-checklist)\n14. [Best Practices & Caveats](#best-practices--caveats)\n15. [Visual Assets & Tables](#visual-assets--tables)\n\n---\n\n## Why the Need for Indic MMLU\n\nThe rise of large language models (LLMs) has enabled advanced **synthetic data generation**, including:\n\n* Building LLM-based frameworks for **data curation and sandboxing**\n* Generating reasoning and coding examples in non-English languages\n* Creating SFT datasets, as well as off-policy datasets for **RLHF, DPO, GRPO**-style training\n\nTo fully leverage these capabilities for Indian languages, we need LLMs with at least **limited native proficiency** in these languages. Such models allow us to generate high-quality, culturally-aligned datasets, create virtual personas to introduce nuanced constructs, and maintain **full control and flexibility** over the synthetic data pipeline.\n\n### Data Distillation for Indian Languages\n\nData distillation provides a practical approach to achieve this:\n\n* **Easier and cleaner access to world knowledge** from state-of-the-art LLMs\n* **Domain control and flexibility** to shape datasets according to cultural and linguistic requirements\n* Embedding **strong Indian values** into datasets more effectively than via custom pipelines\n* Creating safety datasets that mitigate **anti-Indian stances** and promote positive responses\n\n**Distillation** is the process of generating data from large models to produce **task-specific, high-quality, and culturally-aligned datasets**. Using distillation, we can apply various synthetic data augmentation strategies, such as:\n\n* Chain-of-Thought (CoT) reasoning\n* Multi-turn conversational setups for richer data\n* Summaries and diverse QA pairs\n* Incorporation of strong Indian values through **virtual Indian personas**\n* Other augmentations, including translations and language-specific refinements\n\n### The Role of Indic MMLU\n\nBefore deploying LLMs for such Indian-language data generation, we first need to **evaluate their cross-lingual knowledge transfer capabilities**. Indic MMLU serves this purpose by providing a benchmark to test how well models understand Indian languages, rather than merely translating English content.\n\nWhile previous efforts have translated MMLU into Indian languages, none have **released their methodology**. Indic MMLU is the first benchmark to provide:\n\n* **Open recipe and open code** for translations\n* Coverage of **22 Indian languages**\n* A fully transparent and **reproducible evaluation setup**\n\nThis ensures that models can be assessed rigorously for genuine Indian-language understanding, forming a **foundation for culturally-aware synthetic data pipelines** in future research.\n\n---\n\n## What this dataset has on offer\n\nLarge language models are often benchmarked only on English datasets, which leaves open the question of whether they can *actually* understand Indian languages beyond surface-level translation.\n\nThe **Indic MMLU dataset** bridges this gap by:\n\n* Translating the widely used **[CAIS/MMLU](https://huggingface.co/datasets/cais/mmlu) benchmark** into **22 Indian languages**.\n* Enhancing those translations with high-quality open-source LLM refinement.\n* Providing a reproducible evaluation setup for **measuring open-source models across 16 Indic languages (evaluated subset)**.\n\nIf a model’s score on an Indic version is close to its score on the original English MMLU, it indicates **genuine language understanding** and strong cross-lingual generalization.\n\n---\n\n## Coverage\n\n* **Dataset scope:** 22 Indic languages (full benchmark availability).\n* **Evaluation scope:** 16 Indic languages + English (practically evaluated with ~24 open-source LLMs).\n\nThis separation ensures the dataset is **broad and inclusive**, while evaluation focuses on the languages with the most immediate support and resources.\n\n---\n\n## Overview\n\n* **Source benchmark:** CAIS/MMLU (general knowledge QA benchmark).\n* **Target:** 22 Indic languages + English (translated test set).\n* **Workflow:** Machine translation → LLM-based translation enhancement → embedding comparison → Persona based LLM-as-judge Scores → Correlation between Embeddings and LLM as Judge → human/linguist rating and automated evaluation.\n* **Deliverables:** Enhanced translated `.jsonl` datasets, per-language embeddings, cosine similarity reports, teacher/linguist ratings, model evaluation scores and visualizations (radar plots).\n\n---\n\n## Key principles applied\n\n1. **Preserve semantic parity:** Questions remain semantically the same as English; only language changes.\n2. **Quality control:** Model-Based Automated checks (embeddings, math correctness) + human linguist ratings.\n3. **Reproducibility:** Dockerized inference environment, scriptable pipeline, and clear CLI commands.\n4. **Transparency:** We provide similarity metrics to demonstrate translation fidelity and teacher ratings to highlight linguistic quality.\n\n---\n\n## High-level pipeline (what we did)\n\n1. **Translate** the CAIS/MMLU test set to each Indic language using *IndicTrans2* (baseline MT).\n2. **Enhance** those translations using an open-source LLM with task-specific instructions (to correct phrasing, clarify ambiguous translations, preserve math expressions, and maintain answer choices alignment).\n3. **Embed** both the original English items and the (enhanced) translated items with the same embedding model.\n4. **Compare** embeddings using cosine similarity to measure semantic closeness.\n5. **LLM as Judge and Human review**: linguists/teachers rated samples for language quality, math correctness, and coherence.\n6. **Evaluate models**: run ~24 open-source LLMs across all languages using the lm-eval framework and visualize results (radar plots, top-3 per language).\n\n---\n\n## Indic MMLU Construction Algorithm\n\nTo complement the pseudo-code, the following flowchart provides a **visual overview of the Indic MMLU pipeline**. It clearly illustrates the step-by-step workflow, starting from the English MMLU dataset and moving through translation, enhancement, evaluation, and validation:\n\n* **Translation Stage:** The English MMLU instance is translated into Indic languages using specialist MT.\n* **Enhancement Stage:** Translations are refined with large-scale LLMs for improved fluency and alignment.\n* **LLM Based Judging Phase:** Three types of evaluation are conducted — *fluency by native language experts, subject correctness by domain experts, and alignment by coherence judges*.\n* **Validation Stage:** Embedding similarity checks, n-gram metrics (BLEU, ROUGE, ChrF++), and discrepancy detection ensure quality.\n* **Final Output:** A validated Indic MMLU dataset instance ready for benchmarking.\n\n![Indic MMLU Pipeline](assets/indic-mmlu.png)\n\n# Translation Algorithm \n\n```\nInput:\n  original_mmlu\n  target_languages\n  instructions\n\nFor each language L in target_languages:\n    translated = translate(original_mmlu, tgt=L)\n    enhanced = enhance_using_llm(translated, instructions)\n\n    eng_emb = embed(original_mmlu)\n    trans_emb = embed(enhanced)\n    sim_scores = cosine_similarity(eng_emb, trans_emb)\n\n    llm_scores = LLM_as_Judge(original_mmlu, translated)\n\n    consistency = consistency_check(sim_scores, llm_scores)\n\n    If consistency is poor:\n        human_ratings = rate_sample(enhanced)\n    Else:\n        accept enhanced\n\nAggregate all scores:\n    combine sim_scores, llm_scores, human_ratings\n    compute per-language model scores\n\nProduce visualizations:\n    generate charts and plots for cross-language comparison\n\nOutput:\n    per-language evaluation scores\n    visualizations\n\n```\n\nThis visual representation makes it easier to understand how the **algorithmic steps, human evaluation, and automated checks** come together to ensure translation quality and dataset reliability.\n\n---\n\n## Reproducible commands (examples)\n\n> These are the canonical commands used in our pipeline. Replace file paths, ports, and concurrency values as appropriate for your environment.\n\n**Translate**\n\n```bash\npython3 trans_mmlu.py \\\n  --input_file_path test-00000-of-00001_with_ids.jsonl \\\n  --output_file_path mmlu_kn_in.jsonl \\\n  --tgt_lang \"kn\" \\\n  --src_lang \"en\"\n```\n\n**Enhance translations (LLM-based)**\n\n```bash\npython3 async_infr.py \\\n  --input-path mmlu_as_in.jsonl \\\n  --output-file ds_enhance_eval_hi.jsonl \\\n  --instruction-path instruction_prompts.yml \\\n  --task deepseek_enhance_instruct \\\n  --template-fields 'og_question' 'og_choices' 'question' 'answer' 'Hindi' \\\n  --max-concurrency 4096 \\\n  --backend sglang-oai-chat \\\n  --extra-request-body '{\"temperature\":0.7,\"top_p\":0.9,\"top_k\":50,\"max_tokens\":1024}' \\\n  --port 30000\n```\n\n**Generate embeddings**\n\n```bash\npython3 get_embeddings.py \\\n  --input-file test-00000-of-00001_with_ids.jsonl \\\n  --output-file mmlu_qwen3_embd.jsonl \\\n  --server-url http://127.0.0.1:30000 \\\n  --concurrency 4096\n```\n\n**Cosine similarity (English vs translated enhancements)**\n\n```bash\npython3 calculate_cosine_similarity.py \\\n  --input_dir deepseek_enhance \\\n  --output_dir mmmlu/similarity/cosine/enhance/deepseek_enhance \\\n  --english_file mmlu_eng.jsonl\n```\n\n**Linguist / teacher rating (LLM assisted orchestration)**\n\n```bash\npython3 async_infr.py \\\n  --input-path mmlu_as_in.jsonl \\\n  --output-file ds_enhance_eval_as_qwen3_instruct.jsonl \\\n  --instruction-path instruction_prompts.yml \\\n  --task rate_translated_text_linguist \\\n  --template-fields 'og_question' 'og_choices' 'enhanced_text' 'Assamese' \\\n  --max-concurrency 4096 \\\n  --backend sglang-oai-chat \\\n  --extra-request-body '{\"temperature\":0.7,\"top_p\":0.9,\"top_k\":50,\"max_tokens\":1024}' \\\n  --port 30000\n```\n\n**Model evaluation with `lm-eval`**\n\n```bash\n# Example wrapper which runs evaluation for a model snapshot against the Indic MMLU dataset\nbash lm-eval-llm.sh /path/to/model_snapshot /benchmark-result-path\n```\n\n---\n\n### Summary of LLM-as-Judge Ratings\n\nTo evaluate translation quality systematically, we employed **LLMs as judges**, operating under three carefully designed expert personas:\n\n* **Math Expert Persona** → Rated mathematical correctness, terminology, and conceptual fidelity.\n* **Linguistic Expert Persona** → Assessed fluency, grammar, idiomatic usage, and naturalness of the translation.\n* **Coherence Expert Persona** → Evaluated semantic alignment, logical flow, and clarity between English and the target language.\n\n**What we measured:**\n\n* Language quality, fluency, and linguistic suitability.\n* Mathematical correctness and fidelity of technical content.\n* Overall coherence and semantic alignment with the English source.\n\n| Language  | Avg Maths Rate | Avg Coherence Rate | Avg Linguist Rate | Total Records |\n|-----------|----------------|--------------------|-------------------|---------------|\n| Hindi\t    | 9.658029\t     | 9.722054\t          | 9.667891\t        | 14042         |\n| Bengali\t  | 9.182561\t     | 9.131505\t          | 8.202293\t        | 14042         |\n| Nepali    | 9.473793       | 9.331505           | 8.402293          | 14042         |\n| Telugu    | 9.220837       | 8.975929           | 7.986256          | 14042         |\n| Oriya     | 9.264421       | 8.944096           | 8.220268          | 14042         |\n| Punjabi   | 9.348027       | 9.049138           | 8.253454          | 14042         |\n| Assamese  | 9.147201       | 9.039097           | 8.133243          | 14042         |\n| Sanskrit  | 8.338485       | 8.112520           | 6.390115          | 14042         |\n| Kannada   | 9.322746       | 9.111736           | 8.131819          | 14042         |\n| Sindhi    | 2.070645       | 3.109956           | 2.478137          | 14042         |\n| Gujrati   | 9.260647       | 9.206523           | 8.368965          | 14042         |\n| Marathi   | 9.301951       | 9.342045           | 8.381926          | 14042         |\n| Tamil     | 9.047856       | 9.042943           | 8.071144          | 14042         |\n| Malayalam | 9.134952       | 8.998148           | 7.956488          | 14042         |\n| Maithili  | 9.263282       | 9.118146           | 8.218772          | 14042         |\n\n**How to read the table:**\n\n* Ratings are on a **1–10 scale**, where higher scores reflect better translation and enhancement quality.\n* Most languages achieved **consistently high averages (>8.9)**, showing strong quality after enhancement.\n* A few outliers suggest areas where further targeted refinement may be beneficial.\n\n---\n\n## Cosine similarity \n\n* Cosine similarity gives a quick automated signal of semantic closeness between the translated/enhanced items and their English originals.\n* Typical mean similarities range from ~0.76 to 0.85 across languages; higher means closer embeddings (better semantic preservation).\n* Use the similarity scores as a triage: low similarity → manual review; high similarity → likely semantically faithful.\n\n| Language               | Mean Similarity | Std Similarity | Min Similarity | Max Similarity |\n|-------------------------|-----------------|----------------|----------------|----------------|\n| mmlu_as_in_qwen3_embed  | 0.8045          | 0.0597         | 0.4644         | 1              |\n| mmlu_bn_in_qwen3_embed  | 0.8133          | 0.0504         | 0.5276         | 1              |\n| mmlu_gu_in_qwen3_embed  | 0.8211          | 0.0535         | 0.4016         | 1              |\n| mmlu_hi_in_qwen3_embed  | 0.8472          | 0.0489         | 0.3688         | 1              |\n| mmlu_kn_in_qwen3_embed  | 0.8106          | 0.0565         | 0.4660         | 1              |\n| mmlu_mai_in_qwen3_embed | 0.8226          | 0.0504         | 0.4936         | 1              |\n| mmlu_ml_in_qwen3_embed  | 0.8158          | 0.0531         | 0.5109         | 1              |\n| mmlu_mr_in_qwen3_embed  | 0.8129          | 0.0513         | 0.5281         | 1              |\n| mmlu_ne_in_qwen3_embed  | 0.8242          | 0.0502         | 0.4802         | 1              |\n| mmlu_or_in_qwen3_embed  | 0.8159          | 0.0555         | 0.4684         | 1              |\n| mmlu_pa_in_qwen3_embed  | 0.8246          | 0.0516         | 0.5450         | 1              |\n| mmlu_sa_in_qwen3_embed  | 0.7912          | 0.0574         | 0.4981         | 1              |\n| mmlu_sdd_in_qwen3_embed | 0.7646          | 0.0735         | 0.3633         | 0.9676         |\n| mmlu_ta_in_qwen3_embed  | 0.7964          | 0.0559         | 0.5242         | 1              |\n| mmlu_te_in_qwen3_embed  | 0.8006          | 0.0524         | 0.5379         | 1              |\n\n---\n\n## Model Evaluation Methodology\n\nWe evaluated **~24 open-source LLMs** across **16 Indic languages** (subset of the 22 total dataset languages) plus English, using the Indic MMLU benchmark.\n\nWorkflow:\n\n1. **Environment Setup:** Dockerized `vllm/vllm-openai` for reproducibility.\n2. **Framework:** `lm-eval` for standardized evaluation.\n3. **Metrics:** per-language accuracy, top-3 rankings per language, aggregate averages.\n4. **Visualization:** radar plots for each language and comparative top-3 plots.\n\n---\n\n## Important Note\n\n* **Indic MMLU Dataset = 22 Indian languages**\n* **Current Model Evaluation = 16 Indian languages + English**\n* Work is ongoing to expand the evaluation to cover all 22 languages.\n\n---\n### Indic MMLU Results \n\n| <div align=\"center\">![Assamese](assets/assamese.png)<br/>*Assamese*</div> | <div align=\"center\">![Bengali](assets/bengali.png)<br/>*Bengali*</div> |\n|---------------------------------------------------------------------------|------------------------------------------------------------------------|\n| <div align=\"center\">![English](assets/English.png)<br/>*English*</div>    | <div align=\"center\">![Gujarati](assets/Gujrati.png)<br/>*Gujarati*</div> |\n| <div align=\"center\">![Hindi](assets/Hindi.png)<br/>*Hindi*</div>          | <div align=\"center\">![Kannada](assets/kannada.png)<br/>*Kannada*</div> |\n| <div align=\"center\">![Maithili](assets/maithili.png)<br/>*Maithili*</div> | <div align=\"center\">![Malayalam](assets/Malayalam.png)<br/>*Malayalam*</div> |\n| <div align=\"center\">![Marathi](assets/marathi.png)<br/>*Marathi*</div>    | <div align=\"center\">![Nepali](assets/Nepali.png)<br/>*Nepali*</div>   |\n| <div align=\"center\">![Oriya](assets/oriya.png)<br/>*Oriya*</div>          | <div align=\"center\">![Punjabi](assets/Punjabi.png)<br/>*Punjabi*</div> |\n| <div align=\"center\">![Sanskrit](assets/sanskrit.png)<br/>*Sanskrit*</div> | <div align=\"center\">![Sindhi](assets/sindhi.png)<br/>*Sindhi*</div>   |\n| <div align=\"center\">![Tamil](assets/Tamil.png)<br/>*Tamil*</div>          | <div align=\"center\">![Telugu](assets/Telugu.png)<br/>*Telugu*</div>   |\n\n\n---\n\n## How to reproduce (quick checklist)\n\n1. Prepare the English CAIS/MMLU test set (`.jsonl` with IDs).\n2. Run `trans_mmlu.py` to produce per-language MT outputs.\n3. Run `async_infr.py` with the enhancement task to obtain human-readable improved translations.\n4. Generate embeddings for both English and enhanced translations.\n5. Run `calculate_cosine_similarity.py` to produce similarity reports.\n6. Optionally run human/linguist rating orchestration for additional quality assessment.\n7. Evaluate models with `lm-eval` and generate visualizations.\n\n---\n\n## Best practices & caveats\n\n* **Math and formatting:** preserve mathematical expressions and numerals during translation—these are critical for correctness.\n* **Choice alignment:** ensure answer choices remain in the same order and map to the same ground-truth labels.\n* **Human review:** automated metrics are helpful but not sufficient; include periodic linguist sampling.\n* **Constrained inference:** use deterministic LLM settings for enhancement (lower temperature, consistent decoding) to reduce hallucination.\n\n---\n\n## Visual assets & tables\n\nWe include per-language radar plots and a consolidated results panel in `plots/`. The README in the repo contains thumbnails; click through for full-resolution visuals. \n\n---\n"}, "8cee70e3d071261d5ca43ecf24e3177ccf939d28": {"t": 1785771719.0358877, "n": 19772, "h": "60c8d9fd72d39e08", "c": "# Indic MMLU — Benchmarking Open-Source LLMs on Indian Languages\n\nA rigorously prepared, multi-language adaptation of the CAIS/MMLU test set, translated and enhanced for **22 Indic languages**, designed to evaluate open-source LLMs’ real-world ability to understand Indian languages.\nWe additionally conducted large-scale **evaluation on 16 of these Indic languages (plus English)**, producing detailed accuracy reports and per-language visualizations.\n\n---\n\n# Open Source Release \n[Click here to view Release](https://github.com/anonymous-submitter0104/iclr-submission/tree/ebd3accac49c6f0c4441ef5fd2691322e26da4d8/opensource-release/Indic%20MMLU)\n\n---\n# Table of Contents\n\n1. [Overview](#overview)\n2. [Why the Need for Indic MMLU](#why-the-need-for-indic-mmlu)\n   - [Data Distillation for Indian Languages](#data-distillation-for-indian-languages)\n   - [The Role of Indic MMLU](#the-role-of-indic-mmlu)\n3. [What This Dataset Has on Offer](#what-this-dataset-has-on-offer)\n4. [Coverage](#coverage)\n5. [High-Level Pipeline](#high-level-pipeline-what-we-did)\n   - [Indic MMLU Construction Algorithm](#indic-mmlu-construction-algorithm)\n6. [Translation Algorithm](#translation-algorithm)\n7. [Reproducible Commands](#reproducible-commands-examples)\n   - [Translate](#translate)\n   - [Enhance Translations (LLM-based)](#enhance-translations-llm-based)\n   - [Generate Embeddings](#generate-embeddings)\n   - [Cosine Similarity](#cosine-similarity-english-vs-translated-enhancements)\n   - [Linguist / Teacher Rating](#linguist--teacher-rating-llm-assisted-orchestration)\n   - [Model Evaluation with `lm-eval`](#model-evaluation-with-lm-eval)\n8. [Summary of LLM-as-Judge Ratings](#summary-of-llm-as-judge-ratings)\n9. [Cosine Similarity](#cosine-similarity)\n10. [Model Evaluation Methodology](#model-evaluation-methodology)\n11. [Important Note](#important-note)\n12. [Indic MMLU Results](#indic-mmlu-results)\n13. [How to Reproduce (Quick Checklist)](#how-to-reproduce-quick-checklist)\n14. [Best Practices & Caveats](#best-practices--caveats)\n15. [Visual Assets & Tables](#visual-assets--tables)\n\n---\n\n## Why the Need for Indic MMLU\n\nThe rise of large language models (LLMs) has enabled advanced **synthetic data generation**, including:\n\n* Building LLM-based frameworks for **data curation and sandboxing**\n* Generating reasoning and coding examples in non-English languages\n* Creating SFT datasets, as well as off-policy datasets for **RLHF, DPO, GRPO**-style training\n\nTo fully leverage these capabilities for Indian languages, we need LLMs with at least **limited native proficiency** in these languages. Such models allow us to generate high-quality, culturally-aligned datasets, create virtual personas to introduce nuanced constructs, and maintain **full control and flexibility** over the synthetic data pipeline.\n\n### Data Distillation for Indian Languages\n\nData distillation provides a practical approach to achieve this:\n\n* **Easier and cleaner access to world knowledge** from state-of-the-art LLMs\n* **Domain control and flexibility** to shape datasets according to cultural and linguistic requirements\n* Embedding **strong Indian values** into datasets more effectively than via custom pipelines\n* Creating safety datasets that mitigate **anti-Indian stances** and promote positive responses\n\n**Distillation** is the process of generating data from large models to produce **task-specific, high-quality, and culturally-aligned datasets**. Using distillation, we can apply various synthetic data augmentation strategies, such as:\n\n* Chain-of-Thought (CoT) reasoning\n* Multi-turn conversational setups for richer data\n* Summaries and diverse QA pairs\n* Incorporation of strong Indian values through **virtual Indian personas**\n* Other augmentations, including translations and language-specific refinements\n\n### The Role of Indic MMLU\n\nBefore deploying LLMs for such Indian-language data generation, we first need to **evaluate their cross-lingual knowledge transfer capabilities**. Indic MMLU serves this purpose by providing a benchmark to test how well models understand Indian languages, rather than merely translating English content.\n\nWhile previous efforts have translated MMLU into Indian languages, none have **released their methodology**. Indic MMLU is the first benchmark to provide:\n\n* **Open recipe and open code** for translations\n* Coverage of **22 Indian languages**\n* A fully transparent and **reproducible evaluation setup**\n\nThis ensures that models can be assessed rigorously for genuine Indian-language understanding, forming a **foundation for culturally-aware synthetic data pipelines** in future research.\n\n---\n\n## What this dataset has on offer\n\nLarge language models are often benchmarked only on English datasets, which leaves open the question of whether they can *actually* understand Indian languages beyond surface-level translation.\n\nThe **Indic MMLU dataset** bridges this gap by:\n\n* Translating the widely used **[CAIS/MMLU](https://huggingface.co/datasets/cais/mmlu) benchmark** into **22 Indian languages**.\n* Enhancing those translations with high-quality open-source LLM refinement.\n* Providing a reproducible evaluation setup for **measuring open-source models across 16 Indic languages (evaluated subset)**.\n\nIf a model’s score on an Indic version is close to its score on the original English MMLU, it indicates **genuine language understanding** and strong cross-lingual generalization.\n\n---\n\n## Coverage\n\n* **Dataset scope:** 22 Indic languages (full benchmark availability).\n* **Evaluation scope:** 16 Indic languages + English (practically evaluated with ~24 open-source LLMs).\n\nThis separation ensures the dataset is **broad and inclusive**, while evaluation focuses on the languages with the most immediate support and resources.\n\n---\n\n## Overview\n\n* **Source benchmark:** CAIS/MMLU (general knowledge QA benchmark).\n* **Target:** 22 Indic languages + English (translated test set).\n* **Workflow:** Machine translation → LLM-based translation enhancement → embedding comparison → Persona based LLM-as-judge Scores → Correlation between Embeddings and LLM as Judge → human/linguist rating and automated evaluation.\n* **Deliverables:** Enhanced translated `.jsonl` datasets, per-language embeddings, cosine similarity reports, teacher/linguist ratings, model evaluation scores and visualizations (radar plots).\n\n---\n\n## Key principles applied\n\n1. **Preserve semantic parity:** Questions remain semantically the same as English; only language changes.\n2. **Quality control:** Model-Based Automated checks (embeddings, math correctness) + human linguist ratings.\n3. **Reproducibility:** Dockerized inference environment, scriptable pipeline, and clear CLI commands.\n4. **Transparency:** We provide similarity metrics to demonstrate translation fidelity and teacher ratings to highlight linguistic quality.\n\n---\n\n## High-level pipeline (what we did)\n\n1. **Translate** the CAIS/MMLU test set to each Indic language using *IndicTrans2* (baseline MT).\n2. **Enhance** those translations using an open-source LLM with task-specific instructions (to correct phrasing, clarify ambiguous translations, preserve math expressions, and maintain answer choices alignment).\n3. **Embed** both the original English items and the (enhanced) translated items with the same embedding model.\n4. **Compare** embeddings using cosine similarity to measure semantic closeness.\n5. **LLM as Judge and Human review**: linguists/teachers rated samples for language quality, math correctness, and coherence.\n6. **Evaluate models**: run ~24 open-source LLMs across all languages using the lm-eval framework and visualize results (radar plots, top-3 per language).\n\n---\n\n## Indic MMLU Construction Algorithm\n\nTo complement the pseudo-code, the following flowchart provides a **visual overview of the Indic MMLU pipeline**. It clearly illustrates the step-by-step workflow, starting from the English MMLU dataset and moving through translation, enhancement, evaluation, and validation:\n\n* **Translation Stage:** The English MMLU instance is translated into Indic languages using specialist MT.\n* **Enhancement Stage:** Translations are refined with large-scale LLMs for improved fluency and alignment.\n* **LLM Based Judging Phase:** Three types of evaluation are conducted — *fluency by native language experts, subject correctness by domain experts, and alignment by coherence judges*.\n* **Validation Stage:** Embedding similarity checks, n-gram metrics (BLEU, ROUGE, ChrF++), and discrepancy detection ensure quality.\n* **Final Output:** A validated Indic MMLU dataset instance ready for benchmarking.\n\n![Indic MMLU Pipeline](assets/indic-mmlu.png)\n\n# Translation Algorithm \n\n```\nInput:\n  original_mmlu\n  target_languages\n  instructions\n\nFor each language L in target_languages:\n    translated = translate(original_mmlu, tgt=L)\n    enhanced = enhance_using_llm(translated, instructions)\n\n    eng_emb = embed(original_mmlu)\n    trans_emb = embed(enhanced)\n    sim_scores = cosine_similarity(eng_emb, trans_emb)\n\n    llm_scores = LLM_as_Judge(original_mmlu, translated)\n\n    consistency = consistency_check(sim_scores, llm_scores)\n\n    If consistency is poor:\n        human_ratings = rate_sample(enhanced)\n    Else:\n        accept enhanced\n\nAggregate all scores:\n    combine sim_scores, llm_scores, human_ratings\n    compute per-language model scores\n\nProduce visualizations:\n    generate charts and plots for cross-language comparison\n\nOutput:\n    per-language evaluation scores\n    visualizations\n\n```\n\nThis visual representation makes it easier to understand how the **algorithmic steps, human evaluation, and automated checks** come together to ensure translation quality and dataset reliability.\n\n---\n\n## Reproducible commands (examples)\n\n> These are the canonical commands used in our pipeline. Replace file paths, ports, and concurrency values as appropriate for your environment.\n\n**Translate**\n\n```bash\npython3 trans_mmlu.py \\\n  --input_file_path test-00000-of-00001_with_ids.jsonl \\\n  --output_file_path mmlu_kn_in.jsonl \\\n  --tgt_lang \"kn\" \\\n  --src_lang \"en\"\n```\n\n**Enhance translations (LLM-based)**\n\n```bash\npython3 async_infr.py \\\n  --input-path mmlu_as_in.jsonl \\\n  --output-file ds_enhance_eval_hi.jsonl \\\n  --instruction-path instruction_prompts.yml \\\n  --task deepseek_enhance_instruct \\\n  --template-fields 'og_question' 'og_choices' 'question' 'answer' 'Hindi' \\\n  --max-concurrency 4096 \\\n  --backend sglang-oai-chat \\\n  --extra-request-body '{\"temperature\":0.7,\"top_p\":0.9,\"top_k\":50,\"max_tokens\":1024}' \\\n  --port 30000\n```\n\n**Generate embeddings**\n\n```bash\npython3 get_embeddings.py \\\n  --input-file test-00000-of-00001_with_ids.jsonl \\\n  --output-file mmlu_qwen3_embd.jsonl \\\n  --server-url http://127.0.0.1:30000 \\\n  --concurrency 4096\n```\n\n**Cosine similarity (English vs translated enhancements)**\n\n```bash\npython3 calculate_cosine_similarity.py \\\n  --input_dir deepseek_enhance \\\n  --output_dir mmmlu/similarity/cosine/enhance/deepseek_enhance \\\n  --english_file mmlu_eng.jsonl\n```\n\n**Linguist / teacher rating (LLM assisted orchestration)**\n\n```bash\npython3 async_infr.py \\\n  --input-path mmlu_as_in.jsonl \\\n  --output-file ds_enhance_eval_as_qwen3_instruct.jsonl \\\n  --instruction-path instruction_prompts.yml \\\n  --task rate_translated_text_linguist \\\n  --template-fields 'og_question' 'og_choices' 'enhanced_text' 'Assamese' \\\n  --max-concurrency 4096 \\\n  --backend sglang-oai-chat \\\n  --extra-request-body '{\"temperature\":0.7,\"top_p\":0.9,\"top_k\":50,\"max_tokens\":1024}' \\\n  --port 30000\n```\n\n**Model evaluation with `lm-eval`**\n\n```bash\n# Example wrapper which runs evaluation for a model snapshot against the Indic MMLU dataset\nbash lm-eval-llm.sh /path/to/model_snapshot /benchmark-result-path\n```\n\n---\n\n### Summary of LLM-as-Judge Ratings\n\nTo evaluate translation quality systematically, we employed **LLMs as judges**, operating under three carefully designed expert personas:\n\n* **Math Expert Persona** → Rated mathematical correctness, terminology, and conceptual fidelity.\n* **Linguistic Expert Persona** → Assessed fluency, grammar, idiomatic usage, and naturalness of the translation.\n* **Coherence Expert Persona** → Evaluated semantic alignment, logical flow, and clarity between English and the target language.\n\n**What we measured:**\n\n* Language quality, fluency, and linguistic suitability.\n* Mathematical correctness and fidelity of technical content.\n* Overall coherence and semantic alignment with the English source.\n\n| Language  | Avg Maths Rate | Avg Coherence Rate | Avg Linguist Rate | Total Records |\n|-----------|----------------|--------------------|-------------------|---------------|\n| Hindi\t    | 9.658029\t     | 9.722054\t          | 9.667891\t        | 14042         |\n| Bengali\t  | 9.182561\t     | 9.131505\t          | 8.202293\t        | 14042         |\n| Nepali    | 9.473793       | 9.331505           | 8.402293          | 14042         |\n| Telugu    | 9.220837       | 8.975929           | 7.986256          | 14042         |\n| Oriya     | 9.264421       | 8.944096           | 8.220268          | 14042         |\n| Punjabi   | 9.348027       | 9.049138           | 8.253454          | 14042         |\n| Assamese  | 9.147201       | 9.039097           | 8.133243          | 14042         |\n| Sanskrit  | 8.338485       | 8.112520           | 6.390115          | 14042         |\n| Kannada   | 9.322746       | 9.111736           | 8.131819          | 14042         |\n| Sindhi    | 2.070645       | 3.109956           | 2.478137          | 14042         |\n| Gujrati   | 9.260647       | 9.206523           | 8.368965          | 14042         |\n| Marathi   | 9.301951       | 9.342045           | 8.381926          | 14042         |\n| Tamil     | 9.047856       | 9.042943           | 8.071144          | 14042         |\n| Malayalam | 9.134952       | 8.998148           | 7.956488          | 14042         |\n| Maithili  | 9.263282       | 9.118146           | 8.218772          | 14042         |\n\n**How to read the table:**\n\n* Ratings are on a **1–10 scale**, where higher scores reflect better translation and enhancement quality.\n* Most languages achieved **consistently high averages (>8.9)**, showing strong quality after enhancement.\n* A few outliers suggest areas where further targeted refinement may be beneficial.\n\n---\n\n## Cosine similarity \n\n* Cosine similarity gives a quick automated signal of semantic closeness between the translated/enhanced items and their English originals.\n* Typical mean similarities range from ~0.76 to 0.85 across languages; higher means closer embeddings (better semantic preservation).\n* Use the similarity scores as a triage: low similarity → manual review; high similarity → likely semantically faithful.\n\n| Language               | Mean Similarity | Std Similarity | Min Similarity | Max Similarity |\n|-------------------------|-----------------|----------------|----------------|----------------|\n| mmlu_as_in_qwen3_embed  | 0.8045          | 0.0597         | 0.4644         | 1              |\n| mmlu_bn_in_qwen3_embed  | 0.8133          | 0.0504         | 0.5276         | 1              |\n| mmlu_gu_in_qwen3_embed  | 0.8211          | 0.0535         | 0.4016         | 1              |\n| mmlu_hi_in_qwen3_embed  | 0.8472          | 0.0489         | 0.3688         | 1              |\n| mmlu_kn_in_qwen3_embed  | 0.8106          | 0.0565         | 0.4660         | 1              |\n| mmlu_mai_in_qwen3_embed | 0.8226          | 0.0504         | 0.4936         | 1              |\n| mmlu_ml_in_qwen3_embed  | 0.8158          | 0.0531         | 0.5109         | 1              |\n| mmlu_mr_in_qwen3_embed  | 0.8129          | 0.0513         | 0.5281         | 1              |\n| mmlu_ne_in_qwen3_embed  | 0.8242          | 0.0502         | 0.4802         | 1              |\n| mmlu_or_in_qwen3_embed  | 0.8159          | 0.0555         | 0.4684         | 1              |\n| mmlu_pa_in_qwen3_embed  | 0.8246          | 0.0516         | 0.5450         | 1              |\n| mmlu_sa_in_qwen3_embed  | 0.7912          | 0.0574         | 0.4981         | 1              |\n| mmlu_sdd_in_qwen3_embed | 0.7646          | 0.0735         | 0.3633         | 0.9676         |\n| mmlu_ta_in_qwen3_embed  | 0.7964          | 0.0559         | 0.5242         | 1              |\n| mmlu_te_in_qwen3_embed  | 0.8006          | 0.0524         | 0.5379         | 1              |\n\n---\n\n## Model Evaluation Methodology\n\nWe evaluated **~24 open-source LLMs** across **16 Indic languages** (subset of the 22 total dataset languages) plus English, using the Indic MMLU benchmark.\n\nWorkflow:\n\n1. **Environment Setup:** Dockerized `vllm/vllm-openai` for reproducibility.\n2. **Framework:** `lm-eval` for standardized evaluation.\n3. **Metrics:** per-language accuracy, top-3 rankings per language, aggregate averages.\n4. **Visualization:** radar plots for each language and comparative top-3 plots.\n\n---\n\n## Important Note\n\n* **Indic MMLU Dataset = 22 Indian languages**\n* **Current Model Evaluation = 16 Indian languages + English**\n* Work is ongoing to expand the evaluation to cover all 22 languages.\n\n---\n### Indic MMLU Results \n\n| <div align=\"center\">![Assamese](assets/assamese.png)<br/>*Assamese*</div> | <div align=\"center\">![Bengali](assets/bengali.png)<br/>*Bengali*</div> |\n|---------------------------------------------------------------------------|------------------------------------------------------------------------|\n| <div align=\"center\">![English](assets/English.png)<br/>*English*</div>    | <div align=\"center\">![Gujarati](assets/Gujrati.png)<br/>*Gujarati*</div> |\n| <div align=\"center\">![Hindi](assets/Hindi.png)<br/>*Hindi*</div>          | <div align=\"center\">![Kannada](assets/kannada.png)<br/>*Kannada*</div> |\n| <div align=\"center\">![Maithili](assets/maithili.png)<br/>*Maithili*</div> | <div align=\"center\">![Malayalam](assets/Malayalam.png)<br/>*Malayalam*</div> |\n| <div align=\"center\">![Marathi](assets/marathi.png)<br/>*Marathi*</div>    | <div align=\"center\">![Nepali](assets/Nepali.png)<br/>*Nepali*</div>   |\n| <div align=\"center\">![Oriya](assets/oriya.png)<br/>*Oriya*</div>          | <div align=\"center\">![Punjabi](assets/Punjabi.png)<br/>*Punjabi*</div> |\n| <div align=\"center\">![Sanskrit](assets/sanskrit.png)<br/>*Sanskrit*</div> | <div align=\"center\">![Sindhi](assets/sindhi.png)<br/>*Sindhi*</div>   |\n| <div align=\"center\">![Tamil](assets/Tamil.png)<br/>*Tamil*</div>          | <div align=\"center\">![Telugu](assets/Telugu.png)<br/>*Telugu*</div>   |\n\n\n---\n\n## How to reproduce (quick checklist)\n\n1. Prepare the English CAIS/MMLU test set (`.jsonl` with IDs).\n2. Run `trans_mmlu.py` to produce per-language MT outputs.\n3. Run `async_infr.py` with the enhancement task to obtain human-readable improved translations.\n4. Generate embeddings for both English and enhanced translations.\n5. Run `calculate_cosine_similarity.py` to produce similarity reports.\n6. Optionally run human/linguist rating orchestration for additional quality assessment.\n7. Evaluate models with `lm-eval` and generate visualizations.\n\n---\n\n## Best practices & caveats\n\n* **Math and formatting:** preserve mathematical expressions and numerals during translation—these are critical for correctness.\n* **Choice alignment:** ensure answer choices remain in the same order and map to the same ground-truth labels.\n* **Human review:** automated metrics are helpful but not sufficient; include periodic linguist sampling.\n* **Constrained inference:** use deterministic LLM settings for enhancement (lower temperature, consistent decoding) to reduce hallucination.\n\n---\n\n## Visual assets & tables\n\nWe include per-language radar plots and a consolidated results panel in `plots/`. The README in the repo contains thumbnails; click through for full-resolution visuals. \n\n---\n"}, "fe23fea4272aea6fe0f390e1dc0cc82cd3af234b": {"t": 1785771720.24547, "n": 19770, "h": "3372d9f3595670c9", "c": "# Indic MMLU — Benchmarking Open-Source LLMs on Indian Languages\n\nA rigorously prepared, multi-language adaptation of the CAIS/MMLU test set, translated and enhanced for **22 Indic languages**, designed to evaluate open-source LLMs’ real-world ability to understand Indian languages.\nWe additionally conducted large-scale **evaluation on 16 of these Indic languages (plus English)**, producing detailed accuracy reports and per-language visualizations.\n\n---\n\n# Open Source Release \n[Click here to view files](https://github.com/anonymous-submitter0104/iclr-submission/tree/ebd3accac49c6f0c4441ef5fd2691322e26da4d8/opensource-release/Indic%20MMLU)\n\n---\n# Table of Contents\n\n1. [Overview](#overview)\n2. [Why the Need for Indic MMLU](#why-the-need-for-indic-mmlu)\n   - [Data Distillation for Indian Languages](#data-distillation-for-indian-languages)\n   - [The Role of Indic MMLU](#the-role-of-indic-mmlu)\n3. [What This Dataset Has on Offer](#what-this-dataset-has-on-offer)\n4. [Coverage](#coverage)\n5. [High-Level Pipeline](#high-level-pipeline-what-we-did)\n   - [Indic MMLU Construction Algorithm](#indic-mmlu-construction-algorithm)\n6. [Translation Algorithm](#translation-algorithm)\n7. [Reproducible Commands](#reproducible-commands-examples)\n   - [Translate](#translate)\n   - [Enhance Translations (LLM-based)](#enhance-translations-llm-based)\n   - [Generate Embeddings](#generate-embeddings)\n   - [Cosine Similarity](#cosine-similarity-english-vs-translated-enhancements)\n   - [Linguist / Teacher Rating](#linguist--teacher-rating-llm-assisted-orchestration)\n   - [Model Evaluation with `lm-eval`](#model-evaluation-with-lm-eval)\n8. [Summary of LLM-as-Judge Ratings](#summary-of-llm-as-judge-ratings)\n9. [Cosine Similarity](#cosine-similarity)\n10. [Model Evaluation Methodology](#model-evaluation-methodology)\n11. [Important Note](#important-note)\n12. [Indic MMLU Results](#indic-mmlu-results)\n13. [How to Reproduce (Quick Checklist)](#how-to-reproduce-quick-checklist)\n14. [Best Practices & Caveats](#best-practices--caveats)\n15. [Visual Assets & Tables](#visual-assets--tables)\n\n---\n\n## Why the Need for Indic MMLU\n\nThe rise of large language models (LLMs) has enabled advanced **synthetic data generation**, including:\n\n* Building LLM-based frameworks for **data curation and sandboxing**\n* Generating reasoning and coding examples in non-English languages\n* Creating SFT datasets, as well as off-policy datasets for **RLHF, DPO, GRPO**-style training\n\nTo fully leverage these capabilities for Indian languages, we need LLMs with at least **limited native proficiency** in these languages. Such models allow us to generate high-quality, culturally-aligned datasets, create virtual personas to introduce nuanced constructs, and maintain **full control and flexibility** over the synthetic data pipeline.\n\n### Data Distillation for Indian Languages\n\nData distillation provides a practical approach to achieve this:\n\n* **Easier and cleaner access to world knowledge** from state-of-the-art LLMs\n* **Domain control and flexibility** to shape datasets according to cultural and linguistic requirements\n* Embedding **strong Indian values** into datasets more effectively than via custom pipelines\n* Creating safety datasets that mitigate **anti-Indian stances** and promote positive responses\n\n**Distillation** is the process of generating data from large models to produce **task-specific, high-quality, and culturally-aligned datasets**. Using distillation, we can apply various synthetic data augmentation strategies, such as:\n\n* Chain-of-Thought (CoT) reasoning\n* Multi-turn conversational setups for richer data\n* Summaries and diverse QA pairs\n* Incorporation of strong Indian values through **virtual Indian personas**\n* Other augmentations, including translations and language-specific refinements\n\n### The Role of Indic MMLU\n\nBefore deploying LLMs for such Indian-language data generation, we first need to **evaluate their cross-lingual knowledge transfer capabilities**. Indic MMLU serves this purpose by providing a benchmark to test how well models understand Indian languages, rather than merely translating English content.\n\nWhile previous efforts have translated MMLU into Indian languages, none have **released their methodology**. Indic MMLU is the first benchmark to provide:\n\n* **Open recipe and open code** for translations\n* Coverage of **22 Indian languages**\n* A fully transparent and **reproducible evaluation setup**\n\nThis ensures that models can be assessed rigorously for genuine Indian-language understanding, forming a **foundation for culturally-aware synthetic data pipelines** in future research.\n\n---\n\n## What this dataset has on offer\n\nLarge language models are often benchmarked only on English datasets, which leaves open the question of whether they can *actually* understand Indian languages beyond surface-level translation.\n\nThe **Indic MMLU dataset** bridges this gap by:\n\n* Translating the widely used **[CAIS/MMLU](https://huggingface.co/datasets/cais/mmlu) benchmark** into **22 Indian languages**.\n* Enhancing those translations with high-quality open-source LLM refinement.\n* Providing a reproducible evaluation setup for **measuring open-source models across 16 Indic languages (evaluated subset)**.\n\nIf a model’s score on an Indic version is close to its score on the original English MMLU, it indicates **genuine language understanding** and strong cross-lingual generalization.\n\n---\n\n## Coverage\n\n* **Dataset scope:** 22 Indic languages (full benchmark availability).\n* **Evaluation scope:** 16 Indic languages + English (practically evaluated with ~24 open-source LLMs).\n\nThis separation ensures the dataset is **broad and inclusive**, while evaluation focuses on the languages with the most immediate support and resources.\n\n---\n\n## Overview\n\n* **Source benchmark:** CAIS/MMLU (general knowledge QA benchmark).\n* **Target:** 22 Indic languages + English (translated test set).\n* **Workflow:** Machine translation → LLM-based translation enhancement → embedding comparison → Persona based LLM-as-judge Scores → Correlation between Embeddings and LLM as Judge → human/linguist rating and automated evaluation.\n* **Deliverables:** Enhanced translated `.jsonl` datasets, per-language embeddings, cosine similarity reports, teacher/linguist ratings, model evaluation scores and visualizations (radar plots).\n\n---\n\n## Key principles applied\n\n1. **Preserve semantic parity:** Questions remain semantically the same as English; only language changes.\n2. **Quality control:** Model-Based Automated checks (embeddings, math correctness) + human linguist ratings.\n3. **Reproducibility:** Dockerized inference environment, scriptable pipeline, and clear CLI commands.\n4. **Transparency:** We provide similarity metrics to demonstrate translation fidelity and teacher ratings to highlight linguistic quality.\n\n---\n\n## High-level pipeline (what we did)\n\n1. **Translate** the CAIS/MMLU test set to each Indic language using *IndicTrans2* (baseline MT).\n2. **Enhance** those translations using an open-source LLM with task-specific instructions (to correct phrasing, clarify ambiguous translations, preserve math expressions, and maintain answer choices alignment).\n3. **Embed** both the original English items and the (enhanced) translated items with the same embedding model.\n4. **Compare** embeddings using cosine similarity to measure semantic closeness.\n5. **LLM as Judge and Human review**: linguists/teachers rated samples for language quality, math correctness, and coherence.\n6. **Evaluate models**: run ~24 open-source LLMs across all languages using the lm-eval framework and visualize results (radar plots, top-3 per language).\n\n---\n\n## Indic MMLU Construction Algorithm\n\nTo complement the pseudo-code, the following flowchart provides a **visual overview of the Indic MMLU pipeline**. It clearly illustrates the step-by-step workflow, starting from the English MMLU dataset and moving through translation, enhancement, evaluation, and validation:\n\n* **Translation Stage:** The English MMLU instance is translated into Indic languages using specialist MT.\n* **Enhancement Stage:** Translations are refined with large-scale LLMs for improved fluency and alignment.\n* **LLM Based Judging Phase:** Three types of evaluation are conducted — *fluency by native language experts, subject correctness by domain experts, and alignment by coherence judges*.\n* **Validation Stage:** Embedding similarity checks, n-gram metrics (BLEU, ROUGE, ChrF++), and discrepancy detection ensure quality.\n* **Final Output:** A validated Indic MMLU dataset instance ready for benchmarking.\n\n![Indic MMLU Pipeline](assets/indic-mmlu.png)\n\n# Translation Algorithm \n\n```\nInput:\n  original_mmlu\n  target_languages\n  instructions\n\nFor each language L in target_languages:\n    translated = translate(original_mmlu, tgt=L)\n    enhanced = enhance_using_llm(translated, instructions)\n\n    eng_emb = embed(original_mmlu)\n    trans_emb = embed(enhanced)\n    sim_scores = cosine_similarity(eng_emb, trans_emb)\n\n    llm_scores = LLM_as_Judge(original_mmlu, translated)\n\n    consistency = consistency_check(sim_scores, llm_scores)\n\n    If consistency is poor:\n        human_ratings = rate_sample(enhanced)\n    Else:\n        accept enhanced\n\nAggregate all scores:\n    combine sim_scores, llm_scores, human_ratings\n    compute per-language model scores\n\nProduce visualizations:\n    generate charts and plots for cross-language comparison\n\nOutput:\n    per-language evaluation scores\n    visualizations\n\n```\n\nThis visual representation makes it easier to understand how the **algorithmic steps, human evaluation, and automated checks** come together to ensure translation quality and dataset reliability.\n\n---\n\n## Reproducible commands (examples)\n\n> These are the canonical commands used in our pipeline. Replace file paths, ports, and concurrency values as appropriate for your environment.\n\n**Translate**\n\n```bash\npython3 trans_mmlu.py \\\n  --input_file_path test-00000-of-00001_with_ids.jsonl \\\n  --output_file_path mmlu_kn_in.jsonl \\\n  --tgt_lang \"kn\" \\\n  --src_lang \"en\"\n```\n\n**Enhance translations (LLM-based)**\n\n```bash\npython3 async_infr.py \\\n  --input-path mmlu_as_in.jsonl \\\n  --output-file ds_enhance_eval_hi.jsonl \\\n  --instruction-path instruction_prompts.yml \\\n  --task deepseek_enhance_instruct \\\n  --template-fields 'og_question' 'og_choices' 'question' 'answer' 'Hindi' \\\n  --max-concurrency 4096 \\\n  --backend sglang-oai-chat \\\n  --extra-request-body '{\"temperature\":0.7,\"top_p\":0.9,\"top_k\":50,\"max_tokens\":1024}' \\\n  --port 30000\n```\n\n**Generate embeddings**\n\n```bash\npython3 get_embeddings.py \\\n  --input-file test-00000-of-00001_with_ids.jsonl \\\n  --output-file mmlu_qwen3_embd.jsonl \\\n  --server-url http://127.0.0.1:30000 \\\n  --concurrency 4096\n```\n\n**Cosine similarity (English vs translated enhancements)**\n\n```bash\npython3 calculate_cosine_similarity.py \\\n  --input_dir deepseek_enhance \\\n  --output_dir mmmlu/similarity/cosine/enhance/deepseek_enhance \\\n  --english_file mmlu_eng.jsonl\n```\n\n**Linguist / teacher rating (LLM assisted orchestration)**\n\n```bash\npython3 async_infr.py \\\n  --input-path mmlu_as_in.jsonl \\\n  --output-file ds_enhance_eval_as_qwen3_instruct.jsonl \\\n  --instruction-path instruction_prompts.yml \\\n  --task rate_translated_text_linguist \\\n  --template-fields 'og_question' 'og_choices' 'enhanced_text' 'Assamese' \\\n  --max-concurrency 4096 \\\n  --backend sglang-oai-chat \\\n  --extra-request-body '{\"temperature\":0.7,\"top_p\":0.9,\"top_k\":50,\"max_tokens\":1024}' \\\n  --port 30000\n```\n\n**Model evaluation with `lm-eval`**\n\n```bash\n# Example wrapper which runs evaluation for a model snapshot against the Indic MMLU dataset\nbash lm-eval-llm.sh /path/to/model_snapshot /benchmark-result-path\n```\n\n---\n\n### Summary of LLM-as-Judge Ratings\n\nTo evaluate translation quality systematically, we employed **LLMs as judges**, operating under three carefully designed expert personas:\n\n* **Math Expert Persona** → Rated mathematical correctness, terminology, and conceptual fidelity.\n* **Linguistic Expert Persona** → Assessed fluency, grammar, idiomatic usage, and naturalness of the translation.\n* **Coherence Expert Persona** → Evaluated semantic alignment, logical flow, and clarity between English and the target language.\n\n**What we measured:**\n\n* Language quality, fluency, and linguistic suitability.\n* Mathematical correctness and fidelity of technical content.\n* Overall coherence and semantic alignment with the English source.\n\n| Language  | Avg Maths Rate | Avg Coherence Rate | Avg Linguist Rate | Total Records |\n|-----------|----------------|--------------------|-------------------|---------------|\n| Hindi\t    | 9.658029\t     | 9.722054\t          | 9.667891\t        | 14042         |\n| Bengali\t  | 9.182561\t     | 9.131505\t          | 8.202293\t        | 14042         |\n| Nepali    | 9.473793       | 9.331505           | 8.402293          | 14042         |\n| Telugu    | 9.220837       | 8.975929           | 7.986256          | 14042         |\n| Oriya     | 9.264421       | 8.944096           | 8.220268          | 14042         |\n| Punjabi   | 9.348027       | 9.049138           | 8.253454          | 14042         |\n| Assamese  | 9.147201       | 9.039097           | 8.133243          | 14042         |\n| Sanskrit  | 8.338485       | 8.112520           | 6.390115          | 14042         |\n| Kannada   | 9.322746       | 9.111736           | 8.131819          | 14042         |\n| Sindhi    | 2.070645       | 3.109956           | 2.478137          | 14042         |\n| Gujrati   | 9.260647       | 9.206523           | 8.368965          | 14042         |\n| Marathi   | 9.301951       | 9.342045           | 8.381926          | 14042         |\n| Tamil     | 9.047856       | 9.042943           | 8.071144          | 14042         |\n| Malayalam | 9.134952       | 8.998148           | 7.956488          | 14042         |\n| Maithili  | 9.263282       | 9.118146           | 8.218772          | 14042         |\n\n**How to read the table:**\n\n* Ratings are on a **1–10 scale**, where higher scores reflect better translation and enhancement quality.\n* Most languages achieved **consistently high averages (>8.9)**, showing strong quality after enhancement.\n* A few outliers suggest areas where further targeted refinement may be beneficial.\n\n---\n\n## Cosine similarity \n\n* Cosine similarity gives a quick automated signal of semantic closeness between the translated/enhanced items and their English originals.\n* Typical mean similarities range from ~0.76 to 0.85 across languages; higher means closer embeddings (better semantic preservation).\n* Use the similarity scores as a triage: low similarity → manual review; high similarity → likely semantically faithful.\n\n| Language               | Mean Similarity | Std Similarity | Min Similarity | Max Similarity |\n|-------------------------|-----------------|----------------|----------------|----------------|\n| mmlu_as_in_qwen3_embed  | 0.8045          | 0.0597         | 0.4644         | 1              |\n| mmlu_bn_in_qwen3_embed  | 0.8133          | 0.0504         | 0.5276         | 1              |\n| mmlu_gu_in_qwen3_embed  | 0.8211          | 0.0535         | 0.4016         | 1              |\n| mmlu_hi_in_qwen3_embed  | 0.8472          | 0.0489         | 0.3688         | 1              |\n| mmlu_kn_in_qwen3_embed  | 0.8106          | 0.0565         | 0.4660         | 1              |\n| mmlu_mai_in_qwen3_embed | 0.8226          | 0.0504         | 0.4936         | 1              |\n| mmlu_ml_in_qwen3_embed  | 0.8158          | 0.0531         | 0.5109         | 1              |\n| mmlu_mr_in_qwen3_embed  | 0.8129          | 0.0513         | 0.5281         | 1              |\n| mmlu_ne_in_qwen3_embed  | 0.8242          | 0.0502         | 0.4802         | 1              |\n| mmlu_or_in_qwen3_embed  | 0.8159          | 0.0555         | 0.4684         | 1              |\n| mmlu_pa_in_qwen3_embed  | 0.8246          | 0.0516         | 0.5450         | 1              |\n| mmlu_sa_in_qwen3_embed  | 0.7912          | 0.0574         | 0.4981         | 1              |\n| mmlu_sdd_in_qwen3_embed | 0.7646          | 0.0735         | 0.3633         | 0.9676         |\n| mmlu_ta_in_qwen3_embed  | 0.7964          | 0.0559         | 0.5242         | 1              |\n| mmlu_te_in_qwen3_embed  | 0.8006          | 0.0524         | 0.5379         | 1              |\n\n---\n\n## Model Evaluation Methodology\n\nWe evaluated **~24 open-source LLMs** across **16 Indic languages** (subset of the 22 total dataset languages) plus English, using the Indic MMLU benchmark.\n\nWorkflow:\n\n1. **Environment Setup:** Dockerized `vllm/vllm-openai` for reproducibility.\n2. **Framework:** `lm-eval` for standardized evaluation.\n3. **Metrics:** per-language accuracy, top-3 rankings per language, aggregate averages.\n4. **Visualization:** radar plots for each language and comparative top-3 plots.\n\n---\n\n## Important Note\n\n* **Indic MMLU Dataset = 22 Indian languages**\n* **Current Model Evaluation = 16 Indian languages + English**\n* Work is ongoing to expand the evaluation to cover all 22 languages.\n\n---\n### Indic MMLU Results \n\n| <div align=\"center\">![Assamese](assets/assamese.png)<br/>*Assamese*</div> | <div align=\"center\">![Bengali](assets/bengali.png)<br/>*Bengali*</div> |\n|---------------------------------------------------------------------------|------------------------------------------------------------------------|\n| <div align=\"center\">![English](assets/English.png)<br/>*English*</div>    | <div align=\"center\">![Gujarati](assets/Gujrati.png)<br/>*Gujarati*</div> |\n| <div align=\"center\">![Hindi](assets/Hindi.png)<br/>*Hindi*</div>          | <div align=\"center\">![Kannada](assets/kannada.png)<br/>*Kannada*</div> |\n| <div align=\"center\">![Maithili](assets/maithili.png)<br/>*Maithili*</div> | <div align=\"center\">![Malayalam](assets/Malayalam.png)<br/>*Malayalam*</div> |\n| <div align=\"center\">![Marathi](assets/marathi.png)<br/>*Marathi*</div>    | <div align=\"center\">![Nepali](assets/Nepali.png)<br/>*Nepali*</div>   |\n| <div align=\"center\">![Oriya](assets/oriya.png)<br/>*Oriya*</div>          | <div align=\"center\">![Punjabi](assets/Punjabi.png)<br/>*Punjabi*</div> |\n| <div align=\"center\">![Sanskrit](assets/sanskrit.png)<br/>*Sanskrit*</div> | <div align=\"center\">![Sindhi](assets/sindhi.png)<br/>*Sindhi*</div>   |\n| <div align=\"center\">![Tamil](assets/Tamil.png)<br/>*Tamil*</div>          | <div align=\"center\">![Telugu](assets/Telugu.png)<br/>*Telugu*</div>   |\n\n\n---\n\n## How to reproduce (quick checklist)\n\n1. Prepare the English CAIS/MMLU test set (`.jsonl` with IDs).\n2. Run `trans_mmlu.py` to produce per-language MT outputs.\n3. Run `async_infr.py` with the enhancement task to obtain human-readable improved translations.\n4. Generate embeddings for both English and enhanced translations.\n5. Run `calculate_cosine_similarity.py` to produce similarity reports.\n6. Optionally run human/linguist rating orchestration for additional quality assessment.\n7. Evaluate models with `lm-eval` and generate visualizations.\n\n---\n\n## Best practices & caveats\n\n* **Math and formatting:** preserve mathematical expressions and numerals during translation—these are critical for correctness.\n* **Choice alignment:** ensure answer choices remain in the same order and map to the same ground-truth labels.\n* **Human review:** automated metrics are helpful but not sufficient; include periodic linguist sampling.\n* **Constrained inference:** use deterministic LLM settings for enhancement (lower temperature, consistent decoding) to reduce hallucination.\n\n---\n\n## Visual assets & tables\n\nWe include per-language radar plots and a consolidated results panel in `plots/`. The README in the repo contains thumbnails; click through for full-resolution visuals. \n\n---\n"}, "5ad814573c50d56a63460294ded8a7a016e25683": {"t": 1785771722.3660088, "n": 19771, "h": "ca863dfcff46a8be", "c": "# Indic MMLU — Benchmarking Open-Source LLMs on Indian Languages\n\nA rigorously prepared, multi-language adaptation of the CAIS/MMLU test set, translated and enhanced for **22 Indic languages**, designed to evaluate open-source LLMs’ real-world ability to understand Indian languages.\nWe additionally conducted large-scale **evaluation on 16 of these Indic languages (plus English)**, producing detailed accuracy reports and per-language visualizations.\n\n---\n\n# Open Source Release: \n[Click here to view files](https://github.com/anonymous-submitter0104/iclr-submission/tree/ebd3accac49c6f0c4441ef5fd2691322e26da4d8/opensource-release/Indic%20MMLU)\n\n---\n# Table of Contents\n\n1. [Overview](#overview)\n2. [Why the Need for Indic MMLU](#why-the-need-for-indic-mmlu)\n   - [Data Distillation for Indian Languages](#data-distillation-for-indian-languages)\n   - [The Role of Indic MMLU](#the-role-of-indic-mmlu)\n3. [What This Dataset Has on Offer](#what-this-dataset-has-on-offer)\n4. [Coverage](#coverage)\n5. [High-Level Pipeline](#high-level-pipeline-what-we-did)\n   - [Indic MMLU Construction Algorithm](#indic-mmlu-construction-algorithm)\n6. [Translation Algorithm](#translation-algorithm)\n7. [Reproducible Commands](#reproducible-commands-examples)\n   - [Translate](#translate)\n   - [Enhance Translations (LLM-based)](#enhance-translations-llm-based)\n   - [Generate Embeddings](#generate-embeddings)\n   - [Cosine Similarity](#cosine-similarity-english-vs-translated-enhancements)\n   - [Linguist / Teacher Rating](#linguist--teacher-rating-llm-assisted-orchestration)\n   - [Model Evaluation with `lm-eval`](#model-evaluation-with-lm-eval)\n8. [Summary of LLM-as-Judge Ratings](#summary-of-llm-as-judge-ratings)\n9. [Cosine Similarity](#cosine-similarity)\n10. [Model Evaluation Methodology](#model-evaluation-methodology)\n11. [Important Note](#important-note)\n12. [Indic MMLU Results](#indic-mmlu-results)\n13. [How to Reproduce (Quick Checklist)](#how-to-reproduce-quick-checklist)\n14. [Best Practices & Caveats](#best-practices--caveats)\n15. [Visual Assets & Tables](#visual-assets--tables)\n\n---\n\n## Why the Need for Indic MMLU\n\nThe rise of large language models (LLMs) has enabled advanced **synthetic data generation**, including:\n\n* Building LLM-based frameworks for **data curation and sandboxing**\n* Generating reasoning and coding examples in non-English languages\n* Creating SFT datasets, as well as off-policy datasets for **RLHF, DPO, GRPO**-style training\n\nTo fully leverage these capabilities for Indian languages, we need LLMs with at least **limited native proficiency** in these languages. Such models allow us to generate high-quality, culturally-aligned datasets, create virtual personas to introduce nuanced constructs, and maintain **full control and flexibility** over the synthetic data pipeline.\n\n### Data Distillation for Indian Languages\n\nData distillation provides a practical approach to achieve this:\n\n* **Easier and cleaner access to world knowledge** from state-of-the-art LLMs\n* **Domain control and flexibility** to shape datasets according to cultural and linguistic requirements\n* Embedding **strong Indian values** into datasets more effectively than via custom pipelines\n* Creating safety datasets that mitigate **anti-Indian stances** and promote positive responses\n\n**Distillation** is the process of generating data from large models to produce **task-specific, high-quality, and culturally-aligned datasets**. Using distillation, we can apply various synthetic data augmentation strategies, such as:\n\n* Chain-of-Thought (CoT) reasoning\n* Multi-turn conversational setups for richer data\n* Summaries and diverse QA pairs\n* Incorporation of strong Indian values through **virtual Indian personas**\n* Other augmentations, including translations and language-specific refinements\n\n### The Role of Indic MMLU\n\nBefore deploying LLMs for such Indian-language data generation, we first need to **evaluate their cross-lingual knowledge transfer capabilities**. Indic MMLU serves this purpose by providing a benchmark to test how well models understand Indian languages, rather than merely translating English content.\n\nWhile previous efforts have translated MMLU into Indian languages, none have **released their methodology**. Indic MMLU is the first benchmark to provide:\n\n* **Open recipe and open code** for translations\n* Coverage of **22 Indian languages**\n* A fully transparent and **reproducible evaluation setup**\n\nThis ensures that models can be assessed rigorously for genuine Indian-language understanding, forming a **foundation for culturally-aware synthetic data pipelines** in future research.\n\n---\n\n## What this dataset has on offer\n\nLarge language models are often benchmarked only on English datasets, which leaves open the question of whether they can *actually* understand Indian languages beyond surface-level translation.\n\nThe **Indic MMLU dataset** bridges this gap by:\n\n* Translating the widely used **[CAIS/MMLU](https://huggingface.co/datasets/cais/mmlu) benchmark** into **22 Indian languages**.\n* Enhancing those translations with high-quality open-source LLM refinement.\n* Providing a reproducible evaluation setup for **measuring open-source models across 16 Indic languages (evaluated subset)**.\n\nIf a model’s score on an Indic version is close to its score on the original English MMLU, it indicates **genuine language understanding** and strong cross-lingual generalization.\n\n---\n\n## Coverage\n\n* **Dataset scope:** 22 Indic languages (full benchmark availability).\n* **Evaluation scope:** 16 Indic languages + English (practically evaluated with ~24 open-source LLMs).\n\nThis separation ensures the dataset is **broad and inclusive**, while evaluation focuses on the languages with the most immediate support and resources.\n\n---\n\n## Overview\n\n* **Source benchmark:** CAIS/MMLU (general knowledge QA benchmark).\n* **Target:** 22 Indic languages + English (translated test set).\n* **Workflow:** Machine translation → LLM-based translation enhancement → embedding comparison → Persona based LLM-as-judge Scores → Correlation between Embeddings and LLM as Judge → human/linguist rating and automated evaluation.\n* **Deliverables:** Enhanced translated `.jsonl` datasets, per-language embeddings, cosine similarity reports, teacher/linguist ratings, model evaluation scores and visualizations (radar plots).\n\n---\n\n## Key principles applied\n\n1. **Preserve semantic parity:** Questions remain semantically the same as English; only language changes.\n2. **Quality control:** Model-Based Automated checks (embeddings, math correctness) + human linguist ratings.\n3. **Reproducibility:** Dockerized inference environment, scriptable pipeline, and clear CLI commands.\n4. **Transparency:** We provide similarity metrics to demonstrate translation fidelity and teacher ratings to highlight linguistic quality.\n\n---\n\n## High-level pipeline (what we did)\n\n1. **Translate** the CAIS/MMLU test set to each Indic language using *IndicTrans2* (baseline MT).\n2. **Enhance** those translations using an open-source LLM with task-specific instructions (to correct phrasing, clarify ambiguous translations, preserve math expressions, and maintain answer choices alignment).\n3. **Embed** both the original English items and the (enhanced) translated items with the same embedding model.\n4. **Compare** embeddings using cosine similarity to measure semantic closeness.\n5. **LLM as Judge and Human review**: linguists/teachers rated samples for language quality, math correctness, and coherence.\n6. **Evaluate models**: run ~24 open-source LLMs across all languages using the lm-eval framework and visualize results (radar plots, top-3 per language).\n\n---\n\n## Indic MMLU Construction Algorithm\n\nTo complement the pseudo-code, the following flowchart provides a **visual overview of the Indic MMLU pipeline**. It clearly illustrates the step-by-step workflow, starting from the English MMLU dataset and moving through translation, enhancement, evaluation, and validation:\n\n* **Translation Stage:** The English MMLU instance is translated into Indic languages using specialist MT.\n* **Enhancement Stage:** Translations are refined with large-scale LLMs for improved fluency and alignment.\n* **LLM Based Judging Phase:** Three types of evaluation are conducted — *fluency by native language experts, subject correctness by domain experts, and alignment by coherence judges*.\n* **Validation Stage:** Embedding similarity checks, n-gram metrics (BLEU, ROUGE, ChrF++), and discrepancy detection ensure quality.\n* **Final Output:** A validated Indic MMLU dataset instance ready for benchmarking.\n\n![Indic MMLU Pipeline](assets/indic-mmlu.png)\n\n# Translation Algorithm \n\n```\nInput:\n  original_mmlu\n  target_languages\n  instructions\n\nFor each language L in target_languages:\n    translated = translate(original_mmlu, tgt=L)\n    enhanced = enhance_using_llm(translated, instructions)\n\n    eng_emb = embed(original_mmlu)\n    trans_emb = embed(enhanced)\n    sim_scores = cosine_similarity(eng_emb, trans_emb)\n\n    llm_scores = LLM_as_Judge(original_mmlu, translated)\n\n    consistency = consistency_check(sim_scores, llm_scores)\n\n    If consistency is poor:\n        human_ratings = rate_sample(enhanced)\n    Else:\n        accept enhanced\n\nAggregate all scores:\n    combine sim_scores, llm_scores, human_ratings\n    compute per-language model scores\n\nProduce visualizations:\n    generate charts and plots for cross-language comparison\n\nOutput:\n    per-language evaluation scores\n    visualizations\n\n```\n\nThis visual representation makes it easier to understand how the **algorithmic steps, human evaluation, and automated checks** come together to ensure translation quality and dataset reliability.\n\n---\n\n## Reproducible commands (examples)\n\n> These are the canonical commands used in our pipeline. Replace file paths, ports, and concurrency values as appropriate for your environment.\n\n**Translate**\n\n```bash\npython3 trans_mmlu.py \\\n  --input_file_path test-00000-of-00001_with_ids.jsonl \\\n  --output_file_path mmlu_kn_in.jsonl \\\n  --tgt_lang \"kn\" \\\n  --src_lang \"en\"\n```\n\n**Enhance translations (LLM-based)**\n\n```bash\npython3 async_infr.py \\\n  --input-path mmlu_as_in.jsonl \\\n  --output-file ds_enhance_eval_hi.jsonl \\\n  --instruction-path instruction_prompts.yml \\\n  --task deepseek_enhance_instruct \\\n  --template-fields 'og_question' 'og_choices' 'question' 'answer' 'Hindi' \\\n  --max-concurrency 4096 \\\n  --backend sglang-oai-chat \\\n  --extra-request-body '{\"temperature\":0.7,\"top_p\":0.9,\"top_k\":50,\"max_tokens\":1024}' \\\n  --port 30000\n```\n\n**Generate embeddings**\n\n```bash\npython3 get_embeddings.py \\\n  --input-file test-00000-of-00001_with_ids.jsonl \\\n  --output-file mmlu_qwen3_embd.jsonl \\\n  --server-url http://127.0.0.1:30000 \\\n  --concurrency 4096\n```\n\n**Cosine similarity (English vs translated enhancements)**\n\n```bash\npython3 calculate_cosine_similarity.py \\\n  --input_dir deepseek_enhance \\\n  --output_dir mmmlu/similarity/cosine/enhance/deepseek_enhance \\\n  --english_file mmlu_eng.jsonl\n```\n\n**Linguist / teacher rating (LLM assisted orchestration)**\n\n```bash\npython3 async_infr.py \\\n  --input-path mmlu_as_in.jsonl \\\n  --output-file ds_enhance_eval_as_qwen3_instruct.jsonl \\\n  --instruction-path instruction_prompts.yml \\\n  --task rate_translated_text_linguist \\\n  --template-fields 'og_question' 'og_choices' 'enhanced_text' 'Assamese' \\\n  --max-concurrency 4096 \\\n  --backend sglang-oai-chat \\\n  --extra-request-body '{\"temperature\":0.7,\"top_p\":0.9,\"top_k\":50,\"max_tokens\":1024}' \\\n  --port 30000\n```\n\n**Model evaluation with `lm-eval`**\n\n```bash\n# Example wrapper which runs evaluation for a model snapshot against the Indic MMLU dataset\nbash lm-eval-llm.sh /path/to/model_snapshot /benchmark-result-path\n```\n\n---\n\n### Summary of LLM-as-Judge Ratings\n\nTo evaluate translation quality systematically, we employed **LLMs as judges**, operating under three carefully designed expert personas:\n\n* **Math Expert Persona** → Rated mathematical correctness, terminology, and conceptual fidelity.\n* **Linguistic Expert Persona** → Assessed fluency, grammar, idiomatic usage, and naturalness of the translation.\n* **Coherence Expert Persona** → Evaluated semantic alignment, logical flow, and clarity between English and the target language.\n\n**What we measured:**\n\n* Language quality, fluency, and linguistic suitability.\n* Mathematical correctness and fidelity of technical content.\n* Overall coherence and semantic alignment with the English source.\n\n| Language  | Avg Maths Rate | Avg Coherence Rate | Avg Linguist Rate | Total Records |\n|-----------|----------------|--------------------|-------------------|---------------|\n| Hindi\t    | 9.658029\t     | 9.722054\t          | 9.667891\t        | 14042         |\n| Bengali\t  | 9.182561\t     | 9.131505\t          | 8.202293\t        | 14042         |\n| Nepali    | 9.473793       | 9.331505           | 8.402293          | 14042         |\n| Telugu    | 9.220837       | 8.975929           | 7.986256          | 14042         |\n| Oriya     | 9.264421       | 8.944096           | 8.220268          | 14042         |\n| Punjabi   | 9.348027       | 9.049138           | 8.253454          | 14042         |\n| Assamese  | 9.147201       | 9.039097           | 8.133243          | 14042         |\n| Sanskrit  | 8.338485       | 8.112520           | 6.390115          | 14042         |\n| Kannada   | 9.322746       | 9.111736           | 8.131819          | 14042         |\n| Sindhi    | 2.070645       | 3.109956           | 2.478137          | 14042         |\n| Gujrati   | 9.260647       | 9.206523           | 8.368965          | 14042         |\n| Marathi   | 9.301951       | 9.342045           | 8.381926          | 14042         |\n| Tamil     | 9.047856       | 9.042943           | 8.071144          | 14042         |\n| Malayalam | 9.134952       | 8.998148           | 7.956488          | 14042         |\n| Maithili  | 9.263282       | 9.118146           | 8.218772          | 14042         |\n\n**How to read the table:**\n\n* Ratings are on a **1–10 scale**, where higher scores reflect better translation and enhancement quality.\n* Most languages achieved **consistently high averages (>8.9)**, showing strong quality after enhancement.\n* A few outliers suggest areas where further targeted refinement may be beneficial.\n\n---\n\n## Cosine similarity \n\n* Cosine similarity gives a quick automated signal of semantic closeness between the translated/enhanced items and their English originals.\n* Typical mean similarities range from ~0.76 to 0.85 across languages; higher means closer embeddings (better semantic preservation).\n* Use the similarity scores as a triage: low similarity → manual review; high similarity → likely semantically faithful.\n\n| Language               | Mean Similarity | Std Similarity | Min Similarity | Max Similarity |\n|-------------------------|-----------------|----------------|----------------|----------------|\n| mmlu_as_in_qwen3_embed  | 0.8045          | 0.0597         | 0.4644         | 1              |\n| mmlu_bn_in_qwen3_embed  | 0.8133          | 0.0504         | 0.5276         | 1              |\n| mmlu_gu_in_qwen3_embed  | 0.8211          | 0.0535         | 0.4016         | 1              |\n| mmlu_hi_in_qwen3_embed  | 0.8472          | 0.0489         | 0.3688         | 1              |\n| mmlu_kn_in_qwen3_embed  | 0.8106          | 0.0565         | 0.4660         | 1              |\n| mmlu_mai_in_qwen3_embed | 0.8226          | 0.0504         | 0.4936         | 1              |\n| mmlu_ml_in_qwen3_embed  | 0.8158          | 0.0531         | 0.5109         | 1              |\n| mmlu_mr_in_qwen3_embed  | 0.8129          | 0.0513         | 0.5281         | 1              |\n| mmlu_ne_in_qwen3_embed  | 0.8242          | 0.0502         | 0.4802         | 1              |\n| mmlu_or_in_qwen3_embed  | 0.8159          | 0.0555         | 0.4684         | 1              |\n| mmlu_pa_in_qwen3_embed  | 0.8246          | 0.0516         | 0.5450         | 1              |\n| mmlu_sa_in_qwen3_embed  | 0.7912          | 0.0574         | 0.4981         | 1              |\n| mmlu_sdd_in_qwen3_embed | 0.7646          | 0.0735         | 0.3633         | 0.9676         |\n| mmlu_ta_in_qwen3_embed  | 0.7964          | 0.0559         | 0.5242         | 1              |\n| mmlu_te_in_qwen3_embed  | 0.8006          | 0.0524         | 0.5379         | 1              |\n\n---\n\n## Model Evaluation Methodology\n\nWe evaluated **~24 open-source LLMs** across **16 Indic languages** (subset of the 22 total dataset languages) plus English, using the Indic MMLU benchmark.\n\nWorkflow:\n\n1. **Environment Setup:** Dockerized `vllm/vllm-openai` for reproducibility.\n2. **Framework:** `lm-eval` for standardized evaluation.\n3. **Metrics:** per-language accuracy, top-3 rankings per language, aggregate averages.\n4. **Visualization:** radar plots for each language and comparative top-3 plots.\n\n---\n\n## Important Note\n\n* **Indic MMLU Dataset = 22 Indian languages**\n* **Current Model Evaluation = 16 Indian languages + English**\n* Work is ongoing to expand the evaluation to cover all 22 languages.\n\n---\n### Indic MMLU Results \n\n| <div align=\"center\">![Assamese](assets/assamese.png)<br/>*Assamese*</div> | <div align=\"center\">![Bengali](assets/bengali.png)<br/>*Bengali*</div> |\n|---------------------------------------------------------------------------|------------------------------------------------------------------------|\n| <div align=\"center\">![English](assets/English.png)<br/>*English*</div>    | <div align=\"center\">![Gujarati](assets/Gujrati.png)<br/>*Gujarati*</div> |\n| <div align=\"center\">![Hindi](assets/Hindi.png)<br/>*Hindi*</div>          | <div align=\"center\">![Kannada](assets/kannada.png)<br/>*Kannada*</div> |\n| <div align=\"center\">![Maithili](assets/maithili.png)<br/>*Maithili*</div> | <div align=\"center\">![Malayalam](assets/Malayalam.png)<br/>*Malayalam*</div> |\n| <div align=\"center\">![Marathi](assets/marathi.png)<br/>*Marathi*</div>    | <div align=\"center\">![Nepali](assets/Nepali.png)<br/>*Nepali*</div>   |\n| <div align=\"center\">![Oriya](assets/oriya.png)<br/>*Oriya*</div>          | <div align=\"center\">![Punjabi](assets/Punjabi.png)<br/>*Punjabi*</div> |\n| <div align=\"center\">![Sanskrit](assets/sanskrit.png)<br/>*Sanskrit*</div> | <div align=\"center\">![Sindhi](assets/sindhi.png)<br/>*Sindhi*</div>   |\n| <div align=\"center\">![Tamil](assets/Tamil.png)<br/>*Tamil*</div>          | <div align=\"center\">![Telugu](assets/Telugu.png)<br/>*Telugu*</div>   |\n\n\n---\n\n## How to reproduce (quick checklist)\n\n1. Prepare the English CAIS/MMLU test set (`.jsonl` with IDs).\n2. Run `trans_mmlu.py` to produce per-language MT outputs.\n3. Run `async_infr.py` with the enhancement task to obtain human-readable improved translations.\n4. Generate embeddings for both English and enhanced translations.\n5. Run `calculate_cosine_similarity.py` to produce similarity reports.\n6. Optionally run human/linguist rating orchestration for additional quality assessment.\n7. Evaluate models with `lm-eval` and generate visualizations.\n\n---\n\n## Best practices & caveats\n\n* **Math and formatting:** preserve mathematical expressions and numerals during translation—these are critical for correctness.\n* **Choice alignment:** ensure answer choices remain in the same order and map to the same ground-truth labels.\n* **Human review:** automated metrics are helpful but not sufficient; include periodic linguist sampling.\n* **Constrained inference:** use deterministic LLM settings for enhancement (lower temperature, consistent decoding) to reduce hallucination.\n\n---\n\n## Visual assets & tables\n\nWe include per-language radar plots and a consolidated results panel in `plots/`. The README in the repo contains thumbnails; click through for full-resolution visuals. \n\n---\n"}, "d04c74dab7baaf43018aad69d3b19f3158b6ee45": {"t": 1785771724.0358133, "n": 19577, "h": "a5ea7970b13bba7b", "c": "# Indic MMLU — Benchmarking Open-Source LLMs on Indian Languages\n\nA rigorously prepared, multi-language adaptation of the CAIS/MMLU test set, translated and enhanced for **22 Indic languages**, designed to evaluate open-source LLMs’ real-world ability to understand Indian languages.\nWe additionally conducted large-scale **evaluation on 16 of these Indic languages (plus English)**, producing detailed accuracy reports and per-language visualizations.\n\n---\n\n# Table of Contents\n\n1. [Overview](#overview)\n2. [Why the Need for Indic MMLU](#why-the-need-for-indic-mmlu)\n   - [Data Distillation for Indian Languages](#data-distillation-for-indian-languages)\n   - [The Role of Indic MMLU](#the-role-of-indic-mmlu)\n3. [What This Dataset Has on Offer](#what-this-dataset-has-on-offer)\n4. [Coverage](#coverage)\n5. [High-Level Pipeline](#high-level-pipeline-what-we-did)\n   - [Indic MMLU Construction Algorithm](#indic-mmlu-construction-algorithm)\n6. [Translation Algorithm](#translation-algorithm)\n7. [Reproducible Commands](#reproducible-commands-examples)\n   - [Translate](#translate)\n   - [Enhance Translations (LLM-based)](#enhance-translations-llm-based)\n   - [Generate Embeddings](#generate-embeddings)\n   - [Cosine Similarity](#cosine-similarity-english-vs-translated-enhancements)\n   - [Linguist / Teacher Rating](#linguist--teacher-rating-llm-assisted-orchestration)\n   - [Model Evaluation with `lm-eval`](#model-evaluation-with-lm-eval)\n8. [Summary of LLM-as-Judge Ratings](#summary-of-llm-as-judge-ratings)\n9. [Cosine Similarity](#cosine-similarity)\n10. [Model Evaluation Methodology](#model-evaluation-methodology)\n11. [Important Note](#important-note)\n12. [Indic MMLU Results](#indic-mmlu-results)\n13. [How to Reproduce (Quick Checklist)](#how-to-reproduce-quick-checklist)\n14. [Best Practices & Caveats](#best-practices--caveats)\n15. [Visual Assets & Tables](#visual-assets--tables)\n\n---\n\n## Why the Need for Indic MMLU\n\nThe rise of large language models (LLMs) has enabled advanced **synthetic data generation**, including:\n\n* Building LLM-based frameworks for **data curation and sandboxing**\n* Generating reasoning and coding examples in non-English languages\n* Creating SFT datasets, as well as off-policy datasets for **RLHF, DPO, GRPO**-style training\n\nTo fully leverage these capabilities for Indian languages, we need LLMs with at least **limited native proficiency** in these languages. Such models allow us to generate high-quality, culturally-aligned datasets, create virtual personas to introduce nuanced constructs, and maintain **full control and flexibility** over the synthetic data pipeline.\n\n### Data Distillation for Indian Languages\n\nData distillation provides a practical approach to achieve this:\n\n* **Easier and cleaner access to world knowledge** from state-of-the-art LLMs\n* **Domain control and flexibility** to shape datasets according to cultural and linguistic requirements\n* Embedding **strong Indian values** into datasets more effectively than via custom pipelines\n* Creating safety datasets that mitigate **anti-Indian stances** and promote positive responses\n\n**Distillation** is the process of generating data from large models to produce **task-specific, high-quality, and culturally-aligned datasets**. Using distillation, we can apply various synthetic data augmentation strategies, such as:\n\n* Chain-of-Thought (CoT) reasoning\n* Multi-turn conversational setups for richer data\n* Summaries and diverse QA pairs\n* Incorporation of strong Indian values through **virtual Indian personas**\n* Other augmentations, including translations and language-specific refinements\n\n### The Role of Indic MMLU\n\nBefore deploying LLMs for such Indian-language data generation, we first need to **evaluate their cross-lingual knowledge transfer capabilities**. Indic MMLU serves this purpose by providing a benchmark to test how well models understand Indian languages, rather than merely translating English content.\n\nWhile previous efforts have translated MMLU into Indian languages, none have **released their methodology**. Indic MMLU is the first benchmark to provide:\n\n* **Open recipe and open code** for translations\n* Coverage of **22 Indian languages**\n* A fully transparent and **reproducible evaluation setup**\n\nThis ensures that models can be assessed rigorously for genuine Indian-language understanding, forming a **foundation for culturally-aware synthetic data pipelines** in future research.\n\n---\n\n## What this dataset has on offer\n\nLarge language models are often benchmarked only on English datasets, which leaves open the question of whether they can *actually* understand Indian languages beyond surface-level translation.\n\nThe **Indic MMLU dataset** bridges this gap by:\n\n* Translating the widely used **[CAIS/MMLU](https://huggingface.co/datasets/cais/mmlu) benchmark** into **22 Indian languages**.\n* Enhancing those translations with high-quality open-source LLM refinement.\n* Providing a reproducible evaluation setup for **measuring open-source models across 16 Indic languages (evaluated subset)**.\n\nIf a model’s score on an Indic version is close to its score on the original English MMLU, it indicates **genuine language understanding** and strong cross-lingual generalization.\n\n---\n\n## Coverage\n\n* **Dataset scope:** 22 Indic languages (full benchmark availability).\n* **Evaluation scope:** 16 Indic languages + English (practically evaluated with ~24 open-source LLMs).\n\nThis separation ensures the dataset is **broad and inclusive**, while evaluation focuses on the languages with the most immediate support and resources.\n\n---\n\n## Overview\n\n* **Source benchmark:** CAIS/MMLU (general knowledge QA benchmark).\n* **Target:** 22 Indic languages + English (translated test set).\n* **Workflow:** Machine translation → LLM-based translation enhancement → embedding comparison → Persona based LLM-as-judge Scores → Correlation between Embeddings and LLM as Judge → human/linguist rating and automated evaluation.\n* **Deliverables:** Enhanced translated `.jsonl` datasets, per-language embeddings, cosine similarity reports, teacher/linguist ratings, model evaluation scores and visualizations (radar plots).\n\n---\n\n## Key principles applied\n\n1. **Preserve semantic parity:** Questions remain semantically the same as English; only language changes.\n2. **Quality control:** Model-Based Automated checks (embeddings, math correctness) + human linguist ratings.\n3. **Reproducibility:** Dockerized inference environment, scriptable pipeline, and clear CLI commands.\n4. **Transparency:** We provide similarity metrics to demonstrate translation fidelity and teacher ratings to highlight linguistic quality.\n\n---\n\n## High-level pipeline (what we did)\n\n1. **Translate** the CAIS/MMLU test set to each Indic language using *IndicTrans2* (baseline MT).\n2. **Enhance** those translations using an open-source LLM with task-specific instructions (to correct phrasing, clarify ambiguous translations, preserve math expressions, and maintain answer choices alignment).\n3. **Embed** both the original English items and the (enhanced) translated items with the same embedding model.\n4. **Compare** embeddings using cosine similarity to measure semantic closeness.\n5. **LLM as Judge and Human review**: linguists/teachers rated samples for language quality, math correctness, and coherence.\n6. **Evaluate models**: run ~24 open-source LLMs across all languages using the lm-eval framework and visualize results (radar plots, top-3 per language).\n\n---\n\n## Indic MMLU Construction Algorithm\n\nTo complement the pseudo-code, the following flowchart provides a **visual overview of the Indic MMLU pipeline**. It clearly illustrates the step-by-step workflow, starting from the English MMLU dataset and moving through translation, enhancement, evaluation, and validation:\n\n* **Translation Stage:** The English MMLU instance is translated into Indic languages using specialist MT.\n* **Enhancement Stage:** Translations are refined with large-scale LLMs for improved fluency and alignment.\n* **LLM Based Judging Phase:** Three types of evaluation are conducted — *fluency by native language experts, subject correctness by domain experts, and alignment by coherence judges*.\n* **Validation Stage:** Embedding similarity checks, n-gram metrics (BLEU, ROUGE, ChrF++), and discrepancy detection ensure quality.\n* **Final Output:** A validated Indic MMLU dataset instance ready for benchmarking.\n\n![Indic MMLU Pipeline](assets/indic-mmlu.png)\n\n# Translation Algorithm \n\n```\nInput:\n  original_mmlu\n  target_languages\n  instructions\n\nFor each language L in target_languages:\n    translated = translate(original_mmlu, tgt=L)\n    enhanced = enhance_using_llm(translated, instructions)\n\n    eng_emb = embed(original_mmlu)\n    trans_emb = embed(enhanced)\n    sim_scores = cosine_similarity(eng_emb, trans_emb)\n\n    llm_scores = LLM_as_Judge(original_mmlu, translated)\n\n    consistency = consistency_check(sim_scores, llm_scores)\n\n    If consistency is poor:\n        human_ratings = rate_sample(enhanced)\n    Else:\n        accept enhanced\n\nAggregate all scores:\n    combine sim_scores, llm_scores, human_ratings\n    compute per-language model scores\n\nProduce visualizations:\n    generate charts and plots for cross-language comparison\n\nOutput:\n    per-language evaluation scores\n    visualizations\n\n```\n\nThis visual representation makes it easier to understand how the **algorithmic steps, human evaluation, and automated checks** come together to ensure translation quality and dataset reliability.\n\n---\n\n## Reproducible commands (examples)\n\n> These are the canonical commands used in our pipeline. Replace file paths, ports, and concurrency values as appropriate for your environment.\n\n**Translate**\n\n```bash\npython3 trans_mmlu.py \\\n  --input_file_path test-00000-of-00001_with_ids.jsonl \\\n  --output_file_path mmlu_kn_in.jsonl \\\n  --tgt_lang \"kn\" \\\n  --src_lang \"en\"\n```\n\n**Enhance translations (LLM-based)**\n\n```bash\npython3 async_infr.py \\\n  --input-path mmlu_as_in.jsonl \\\n  --output-file ds_enhance_eval_hi.jsonl \\\n  --instruction-path instruction_prompts.yml \\\n  --task deepseek_enhance_instruct \\\n  --template-fields 'og_question' 'og_choices' 'question' 'answer' 'Hindi' \\\n  --max-concurrency 4096 \\\n  --backend sglang-oai-chat \\\n  --extra-request-body '{\"temperature\":0.7,\"top_p\":0.9,\"top_k\":50,\"max_tokens\":1024}' \\\n  --port 30000\n```\n\n**Generate embeddings**\n\n```bash\npython3 get_embeddings.py \\\n  --input-file test-00000-of-00001_with_ids.jsonl \\\n  --output-file mmlu_qwen3_embd.jsonl \\\n  --server-url http://127.0.0.1:30000 \\\n  --concurrency 4096\n```\n\n**Cosine similarity (English vs translated enhancements)**\n\n```bash\npython3 calculate_cosine_similarity.py \\\n  --input_dir deepseek_enhance \\\n  --output_dir mmmlu/similarity/cosine/enhance/deepseek_enhance \\\n  --english_file mmlu_eng.jsonl\n```\n\n**Linguist / teacher rating (LLM assisted orchestration)**\n\n```bash\npython3 async_infr.py \\\n  --input-path mmlu_as_in.jsonl \\\n  --output-file ds_enhance_eval_as_qwen3_instruct.jsonl \\\n  --instruction-path instruction_prompts.yml \\\n  --task rate_translated_text_linguist \\\n  --template-fields 'og_question' 'og_choices' 'enhanced_text' 'Assamese' \\\n  --max-concurrency 4096 \\\n  --backend sglang-oai-chat \\\n  --extra-request-body '{\"temperature\":0.7,\"top_p\":0.9,\"top_k\":50,\"max_tokens\":1024}' \\\n  --port 30000\n```\n\n**Model evaluation with `lm-eval`**\n\n```bash\n# Example wrapper which runs evaluation for a model snapshot against the Indic MMLU dataset\nbash lm-eval-llm.sh /path/to/model_snapshot /benchmark-result-path\n```\n\n---\n\n### Summary of LLM-as-Judge Ratings\n\nTo evaluate translation quality systematically, we employed **LLMs as judges**, operating under three carefully designed expert personas:\n\n* **Math Expert Persona** → Rated mathematical correctness, terminology, and conceptual fidelity.\n* **Linguistic Expert Persona** → Assessed fluency, grammar, idiomatic usage, and naturalness of the translation.\n* **Coherence Expert Persona** → Evaluated semantic alignment, logical flow, and clarity between English and the target language.\n\n**What we measured:**\n\n* Language quality, fluency, and linguistic suitability.\n* Mathematical correctness and fidelity of technical content.\n* Overall coherence and semantic alignment with the English source.\n\n| Language  | Avg Maths Rate | Avg Coherence Rate | Avg Linguist Rate | Total Records |\n|-----------|----------------|--------------------|-------------------|---------------|\n| Hindi\t    | 9.658029\t     | 9.722054\t          | 9.667891\t        | 14042         |\n| Bengali\t  | 9.182561\t     | 9.131505\t          | 8.202293\t        | 14042         |\n| Nepali    | 9.473793       | 9.331505           | 8.402293          | 14042         |\n| Telugu    | 9.220837       | 8.975929           | 7.986256          | 14042         |\n| Oriya     | 9.264421       | 8.944096           | 8.220268          | 14042         |\n| Punjabi   | 9.348027       | 9.049138           | 8.253454          | 14042         |\n| Assamese  | 9.147201       | 9.039097           | 8.133243          | 14042         |\n| Sanskrit  | 8.338485       | 8.112520           | 6.390115          | 14042         |\n| Kannada   | 9.322746       | 9.111736           | 8.131819          | 14042         |\n| Sindhi    | 2.070645       | 3.109956           | 2.478137          | 14042         |\n| Gujrati   | 9.260647       | 9.206523           | 8.368965          | 14042         |\n| Marathi   | 9.301951       | 9.342045           | 8.381926          | 14042         |\n| Tamil     | 9.047856       | 9.042943           | 8.071144          | 14042         |\n| Malayalam | 9.134952       | 8.998148           | 7.956488          | 14042         |\n| Maithili  | 9.263282       | 9.118146           | 8.218772          | 14042         |\n\n**How to read the table:**\n\n* Ratings are on a **1–10 scale**, where higher scores reflect better translation and enhancement quality.\n* Most languages achieved **consistently high averages (>8.9)**, showing strong quality after enhancement.\n* A few outliers suggest areas where further targeted refinement may be beneficial.\n\n---\n\n## Cosine similarity \n\n* Cosine similarity gives a quick automated signal of semantic closeness between the translated/enhanced items and their English originals.\n* Typical mean similarities range from ~0.76 to 0.85 across languages; higher means closer embeddings (better semantic preservation).\n* Use the similarity scores as a triage: low similarity → manual review; high similarity → likely semantically faithful.\n\n| Language               | Mean Similarity | Std Similarity | Min Similarity | Max Similarity |\n|-------------------------|-----------------|----------------|----------------|----------------|\n| mmlu_as_in_qwen3_embed  | 0.8045          | 0.0597         | 0.4644         | 1              |\n| mmlu_bn_in_qwen3_embed  | 0.8133          | 0.0504         | 0.5276         | 1              |\n| mmlu_gu_in_qwen3_embed  | 0.8211          | 0.0535         | 0.4016         | 1              |\n| mmlu_hi_in_qwen3_embed  | 0.8472          | 0.0489         | 0.3688         | 1              |\n| mmlu_kn_in_qwen3_embed  | 0.8106          | 0.0565         | 0.4660         | 1              |\n| mmlu_mai_in_qwen3_embed | 0.8226          | 0.0504         | 0.4936         | 1              |\n| mmlu_ml_in_qwen3_embed  | 0.8158          | 0.0531         | 0.5109         | 1              |\n| mmlu_mr_in_qwen3_embed  | 0.8129          | 0.0513         | 0.5281         | 1              |\n| mmlu_ne_in_qwen3_embed  | 0.8242          | 0.0502         | 0.4802         | 1              |\n| mmlu_or_in_qwen3_embed  | 0.8159          | 0.0555         | 0.4684         | 1              |\n| mmlu_pa_in_qwen3_embed  | 0.8246          | 0.0516         | 0.5450         | 1              |\n| mmlu_sa_in_qwen3_embed  | 0.7912          | 0.0574         | 0.4981         | 1              |\n| mmlu_sdd_in_qwen3_embed | 0.7646          | 0.0735         | 0.3633         | 0.9676         |\n| mmlu_ta_in_qwen3_embed  | 0.7964          | 0.0559         | 0.5242         | 1              |\n| mmlu_te_in_qwen3_embed  | 0.8006          | 0.0524         | 0.5379         | 1              |\n\n---\n\n## Model Evaluation Methodology\n\nWe evaluated **~24 open-source LLMs** across **16 Indic languages** (subset of the 22 total dataset languages) plus English, using the Indic MMLU benchmark.\n\nWorkflow:\n\n1. **Environment Setup:** Dockerized `vllm/vllm-openai` for reproducibility.\n2. **Framework:** `lm-eval` for standardized evaluation.\n3. **Metrics:** per-language accuracy, top-3 rankings per language, aggregate averages.\n4. **Visualization:** radar plots for each language and comparative top-3 plots.\n\n---\n\n## Important Note\n\n* **Indic MMLU Dataset = 22 Indian languages**\n* **Current Model Evaluation = 16 Indian languages + English**\n* Work is ongoing to expand the evaluation to cover all 22 languages.\n\n---\n### Indic MMLU Results \n\n| <div align=\"center\">![Assamese](assets/assamese.png)<br/>*Assamese*</div> | <div align=\"center\">![Bengali](assets/bengali.png)<br/>*Bengali*</div> |\n|---------------------------------------------------------------------------|------------------------------------------------------------------------|\n| <div align=\"center\">![English](assets/English.png)<br/>*English*</div>    | <div align=\"center\">![Gujarati](assets/Gujrati.png)<br/>*Gujarati*</div> |\n| <div align=\"center\">![Hindi](assets/Hindi.png)<br/>*Hindi*</div>          | <div align=\"center\">![Kannada](assets/kannada.png)<br/>*Kannada*</div> |\n| <div align=\"center\">![Maithili](assets/maithili.png)<br/>*Maithili*</div> | <div align=\"center\">![Malayalam](assets/Malayalam.png)<br/>*Malayalam*</div> |\n| <div align=\"center\">![Marathi](assets/marathi.png)<br/>*Marathi*</div>    | <div align=\"center\">![Nepali](assets/Nepali.png)<br/>*Nepali*</div>   |\n| <div align=\"center\">![Oriya](assets/oriya.png)<br/>*Oriya*</div>          | <div align=\"center\">![Punjabi](assets/Punjabi.png)<br/>*Punjabi*</div> |\n| <div align=\"center\">![Sanskrit](assets/sanskrit.png)<br/>*Sanskrit*</div> | <div align=\"center\">![Sindhi](assets/sindhi.png)<br/>*Sindhi*</div>   |\n| <div align=\"center\">![Tamil](assets/Tamil.png)<br/>*Tamil*</div>          | <div align=\"center\">![Telugu](assets/Telugu.png)<br/>*Telugu*</div>   |\n\n\n---\n\n## How to reproduce (quick checklist)\n\n1. Prepare the English CAIS/MMLU test set (`.jsonl` with IDs).\n2. Run `trans_mmlu.py` to produce per-language MT outputs.\n3. Run `async_infr.py` with the enhancement task to obtain human-readable improved translations.\n4. Generate embeddings for both English and enhanced translations.\n5. Run `calculate_cosine_similarity.py` to produce similarity reports.\n6. Optionally run human/linguist rating orchestration for additional quality assessment.\n7. Evaluate models with `lm-eval` and generate visualizations.\n\n---\n\n## Best practices & caveats\n\n* **Math and formatting:** preserve mathematical expressions and numerals during translation—these are critical for correctness.\n* **Choice alignment:** ensure answer choices remain in the same order and map to the same ground-truth labels.\n* **Human review:** automated metrics are helpful but not sufficient; include periodic linguist sampling.\n* **Constrained inference:** use deterministic LLM settings for enhancement (lower temperature, consistent decoding) to reduce hallucination.\n\n---\n\n## Visual assets & tables\n\nWe include per-language radar plots and a consolidated results panel in `plots/`. The README in the repo contains thumbnails; click through for full-resolution visuals. \n\n---\n"}, "9352e0bad04c8b12aa70007be04d3bfa0472c980": {"t": 1785771727.0438097, "n": 19604, "h": "8ef580668db1f851", "c": "# Indic MMLU — Benchmarking Open-Source LLMs on Indian Languages\n\nA rigorously prepared, multi-language adaptation of the CAIS/MMLU test set, translated and enhanced for **22 Indic languages**, designed to evaluate open-source LLMs’ real-world ability to understand Indian languages.\nWe additionally conducted large-scale **evaluation on 16 of these Indic languages (plus English)**, producing detailed accuracy reports and per-language visualizations.\n\n---\n\n# Table of Contents\n\n1. [Overview](#overview)\n2. [Why the Need for Indic MMLU](#why-the-need-for-indic-mmlu)\n   - [Data Distillation for Indian Languages](#data-distillation-for-indian-languages)\n   - [The Role of Indic MMLU](#the-role-of-indic-mmlu)\n3. [What This Dataset Has on Offer](#what-this-dataset-has-on-offer)\n4. [Coverage](#coverage)\n5. [High-Level Pipeline](#high-level-pipeline-what-we-did)\n   - [Indic MMLU Pipeline Construction Algorithm](#indic-mmlu-pipeline-construction-algorithm)\n6. [Translation Algorithm](#translation-algorithm)\n7. [Reproducible Commands](#reproducible-commands-examples)\n   - [Translate](#translate)\n   - [Enhance Translations (LLM-based)](#enhance-translations-llm-based)\n   - [Generate Embeddings](#generate-embeddings)\n   - [Cosine Similarity](#cosine-similarity-english-vs-translated-enhancements)\n   - [Linguist / Teacher Rating](#linguist--teacher-rating-llm-assisted-orchestration)\n   - [Model Evaluation with `lm-eval`](#model-evaluation-with-lm-eval)\n8. [Summary of LLM-as-Judge Ratings](#summary-of-llm-as-judge-ratings)\n9. [Cosine Similarity](#cosine-similarity)\n10. [Model Evaluation Methodology](#model-evaluation-methodology)\n11. [Important Note](#important-note)\n12. [Indic MMLU Results](#indic-mmlu-results)\n13. [How to Reproduce (Quick Checklist)](#how-to-reproduce-quick-checklist)\n14. [Best Practices & Caveats](#best-practices--caveats)\n15. [Visual Assets & Tables](#visual-assets--tables)\n\n---\n\n## Why the Need for Indic MMLU\n\nThe rise of large language models (LLMs) has enabled advanced **synthetic data generation**, including:\n\n* Building LLM-based frameworks for **data curation and sandboxing**\n* Generating reasoning and coding examples in non-English languages\n* Creating SFT datasets, as well as off-policy datasets for **RLHF, DPO, GRPO**-style training\n\nTo fully leverage these capabilities for Indian languages, we need LLMs with at least **limited native proficiency** in these languages. Such models allow us to generate high-quality, culturally-aligned datasets, create virtual personas to introduce nuanced constructs, and maintain **full control and flexibility** over the synthetic data pipeline.\n\n### Data Distillation for Indian Languages\n\nData distillation provides a practical approach to achieve this:\n\n* **Easier and cleaner access to world knowledge** from state-of-the-art LLMs\n* **Domain control and flexibility** to shape datasets according to cultural and linguistic requirements\n* Embedding **strong Indian values** into datasets more effectively than via custom pipelines\n* Creating safety datasets that mitigate **anti-Indian stances** and promote positive responses\n\n**Distillation** is the process of generating data from large models to produce **task-specific, high-quality, and culturally-aligned datasets**. Using distillation, we can apply various synthetic data augmentation strategies, such as:\n\n* Chain-of-Thought (CoT) reasoning\n* Multi-turn conversational setups for richer data\n* Summaries and diverse QA pairs\n* Incorporation of strong Indian values through **virtual Indian personas**\n* Other augmentations, including translations and language-specific refinements\n\n### The Role of Indic MMLU\n\nBefore deploying LLMs for such Indian-language data generation, we first need to **evaluate their cross-lingual knowledge transfer capabilities**. Indic MMLU serves this purpose by providing a benchmark to test how well models understand Indian languages, rather than merely translating English content.\n\nWhile previous efforts have translated MMLU into Indian languages, none have **released their methodology**. Indic MMLU is the first benchmark to provide:\n\n* **Open recipe and open code** for translations\n* Coverage of **22 Indian languages**\n* A fully transparent and **reproducible evaluation setup**\n\nThis ensures that models can be assessed rigorously for genuine Indian-language understanding, forming a **foundation for culturally-aware synthetic data pipelines** in future research.\n\n---\n\n## What this dataset has on offer\n\nLarge language models are often benchmarked only on English datasets, which leaves open the question of whether they can *actually* understand Indian languages beyond surface-level translation.\n\nThe **Indic MMLU dataset** bridges this gap by:\n\n* Translating the widely used **[CAIS/MMLU](https://huggingface.co/datasets/cais/mmlu) benchmark** into **22 Indian languages**.\n* Enhancing those translations with high-quality open-source LLM refinement.\n* Providing a reproducible evaluation setup for **measuring open-source models across 16 Indic languages (evaluated subset)**.\n\nIf a model’s score on an Indic version is close to its score on the original English MMLU, it indicates **genuine language understanding** and strong cross-lingual generalization.\n\n---\n\n## Coverage\n\n* **Dataset scope:** 22 Indic languages (full benchmark availability).\n* **Evaluation scope:** 16 Indic languages + English (practically evaluated with ~24 open-source LLMs).\n\nThis separation ensures the dataset is **broad and inclusive**, while evaluation focuses on the languages with the most immediate support and resources.\n\n---\n\n## Overview\n\n* **Source benchmark:** CAIS/MMLU (general knowledge QA benchmark).\n* **Target:** 22 Indic languages + English (translated test set).\n* **Workflow:** Machine translation → LLM-based translation enhancement → embedding comparison → Persona based LLM-as-judge Scores → Correlation between Embeddings and LLM as Judge → human/linguist rating and automated evaluation.\n* **Deliverables:** Enhanced translated `.jsonl` datasets, per-language embeddings, cosine similarity reports, teacher/linguist ratings, model evaluation scores and visualizations (radar plots).\n\n---\n\n## Key principles applied\n\n1. **Preserve semantic parity:** Questions remain semantically the same as English; only language changes.\n2. **Quality control:** Model-Based Automated checks (embeddings, math correctness) + human linguist ratings.\n3. **Reproducibility:** Dockerized inference environment, scriptable pipeline, and clear CLI commands.\n4. **Transparency:** We provide similarity metrics to demonstrate translation fidelity and teacher ratings to highlight linguistic quality.\n\n---\n\n## High-level pipeline (what we did)\n\n1. **Translate** the CAIS/MMLU test set to each Indic language using *IndicTrans2* (baseline MT).\n2. **Enhance** those translations using an open-source LLM with task-specific instructions (to correct phrasing, clarify ambiguous translations, preserve math expressions, and maintain answer choices alignment).\n3. **Embed** both the original English items and the (enhanced) translated items with the same embedding model.\n4. **Compare** embeddings using cosine similarity to measure semantic closeness.\n5. **LLM as Judge and Human review**: linguists/teachers rated samples for language quality, math correctness, and coherence.\n6. **Evaluate models**: run ~24 open-source LLMs across all languages using the lm-eval framework and visualize results (radar plots, top-3 per language).\n\n---\n\n## Indic MMLU Pipeline Construction Algorithm\n\nTo complement the pseudo-code, the following flowchart provides a **visual overview of the Indic MMLU pipeline**. It clearly illustrates the step-by-step workflow, starting from the English MMLU dataset and moving through translation, enhancement, evaluation, and validation:\n\n* **Translation Stage:** The English MMLU instance is translated into Indic languages using specialist MT.\n* **Enhancement Stage:** Translations are refined with large-scale LLMs for improved fluency and alignment.\n* **LLM Based Judging Phase:** Three types of evaluation are conducted — *fluency by native language experts, subject correctness by domain experts, and alignment by coherence judges*.\n* **Validation Stage:** Embedding similarity checks, n-gram metrics (BLEU, ROUGE, ChrF++), and discrepancy detection ensure quality.\n* **Final Output:** A validated Indic MMLU dataset instance ready for benchmarking.\n\n![Indic MMLU Pipeline](assets/indic-mmlu.png)\n\n# Translation Algorithm \n\n```\nInput:\n  original_mmlu\n  target_languages\n  instructions\n\nFor each language L in target_languages:\n    translated = translate(original_mmlu, tgt=L)\n    enhanced = enhance_using_llm(translated, instructions)\n\n    eng_emb = embed(original_mmlu)\n    trans_emb = embed(enhanced)\n    sim_scores = cosine_similarity(eng_emb, trans_emb)\n\n    llm_scores = LLM_as_Judge(original_mmlu, translated)\n\n    consistency = consistency_check(sim_scores, llm_scores)\n\n    If consistency is poor:\n        human_ratings = rate_sample(enhanced)\n    Else:\n        accept enhanced\n\nAggregate all scores:\n    combine sim_scores, llm_scores, human_ratings\n    compute per-language model scores\n\nProduce visualizations:\n    generate charts and plots for cross-language comparison\n\nOutput:\n    per-language evaluation scores\n    visualizations\n\n```\n\nThis visual representation makes it easier to understand how the **algorithmic steps, human evaluation, and automated checks** come together to ensure translation quality and dataset reliability.\n\n---\n\n## Reproducible commands (examples)\n\n> These are the canonical commands used in our pipeline. Replace file paths, ports, and concurrency values as appropriate for your environment.\n\n**Translate**\n\n```bash\npython3 trans_mmlu.py \\\n  --input_file_path test-00000-of-00001_with_ids.jsonl \\\n  --output_file_path mmlu_kn_in.jsonl \\\n  --tgt_lang \"kn\" \\\n  --src_lang \"en\"\n```\n\n**Enhance translations (LLM-based)**\n\n```bash\npython3 async_infr.py \\\n  --input-path mmlu_as_in.jsonl \\\n  --output-file ds_enhance_eval_hi.jsonl \\\n  --instruction-path instruction_prompts.yml \\\n  --task deepseek_enhance_instruct \\\n  --template-fields 'og_question' 'og_choices' 'question' 'answer' 'Hindi' \\\n  --max-concurrency 4096 \\\n  --backend sglang-oai-chat \\\n  --extra-request-body '{\"temperature\":0.7,\"top_p\":0.9,\"top_k\":50,\"max_tokens\":1024}' \\\n  --port 30000\n```\n\n**Generate embeddings**\n\n```bash\npython3 get_embeddings.py \\\n  --input-file test-00000-of-00001_with_ids.jsonl \\\n  --output-file mmlu_qwen3_embd.jsonl \\\n  --server-url http://127.0.0.1:30000 \\\n  --concurrency 4096\n```\n\n**Cosine similarity (English vs translated enhancements)**\n\n```bash\npython3 calculate_cosine_similarity.py \\\n  --input_dir deepseek_enhance \\\n  --output_dir mmmlu/similarity/cosine/enhance/deepseek_enhance \\\n  --english_file mmlu_eng.jsonl\n```\n\n**Linguist / teacher rating (LLM assisted orchestration)**\n\n```bash\npython3 async_infr.py \\\n  --input-path mmlu_as_in.jsonl \\\n  --output-file ds_enhance_eval_as_qwen3_instruct.jsonl \\\n  --instruction-path instruction_prompts.yml \\\n  --task rate_translated_text_linguist \\\n  --template-fields 'og_question' 'og_choices' 'enhanced_text' 'Assamese' \\\n  --max-concurrency 4096 \\\n  --backend sglang-oai-chat \\\n  --extra-request-body '{\"temperature\":0.7,\"top_p\":0.9,\"top_k\":50,\"max_tokens\":1024}' \\\n  --port 30000\n```\n\n**Model evaluation with `lm-eval`**\n\n```bash\n# Example wrapper which runs evaluation for a model snapshot against the Indic MMLU dataset\nbash lm-eval-llm.sh /path/to/model_snapshot /benchmark-result-path\n```\n\n---\n\n### Summary of LLM-as-Judge Ratings\n\nTo evaluate translation quality systematically, we employed **LLMs as judges**, operating under three carefully designed expert personas:\n\n* **Math Expert Persona** → Rated mathematical correctness, terminology, and conceptual fidelity.\n* **Linguistic Expert Persona** → Assessed fluency, grammar, idiomatic usage, and naturalness of the translation.\n* **Coherence Expert Persona** → Evaluated semantic alignment, logical flow, and clarity between English and the target language.\n\n**What we measured:**\n\n* Language quality, fluency, and linguistic suitability.\n* Mathematical correctness and fidelity of technical content.\n* Overall coherence and semantic alignment with the English source.\n\n| Language  | Avg Maths Rate | Avg Coherence Rate | Avg Linguist Rate | Total Records |\n|-----------|----------------|--------------------|-------------------|---------------|\n| Hindi\t    | 9.658029\t     | 9.722054\t          | 9.667891\t        | 14042         |\n| Bengali\t  | 9.182561\t     | 9.131505\t          | 8.202293\t        | 14042         |\n| Nepali    | 9.473793       | 9.331505           | 8.402293          | 14042         |\n| Telugu    | 9.220837       | 8.975929           | 7.986256          | 14042         |\n| Oriya     | 9.264421       | 8.944096           | 8.220268          | 14042         |\n| Punjabi   | 9.348027       | 9.049138           | 8.253454          | 14042         |\n| Assamese  | 9.147201       | 9.039097           | 8.133243          | 14042         |\n| Sanskrit  | 8.338485       | 8.112520           | 6.390115          | 14042         |\n| Kannada   | 9.322746       | 9.111736           | 8.131819          | 14042         |\n| Sindhi    | 2.070645       | 3.109956           | 2.478137          | 14042         |\n| Gujrati   | 9.260647       | 9.206523           | 8.368965          | 14042         |\n| Marathi   | 9.301951       | 9.342045           | 8.381926          | 14042         |\n| Tamil     | 9.047856       | 9.042943           | 8.071144          | 14042         |\n| Malayalam | 9.134952       | 8.998148           | 7.956488          | 14042         |\n| Maithili  | 9.263282       | 9.118146           | 8.218772          | 14042         |\n\n**How to read the table:**\n\n* Ratings are on a **1–10 scale**, where higher scores reflect better translation and enhancement quality.\n* Most languages achieved **consistently high averages (>8.9)**, showing strong quality after enhancement.\n* A few outliers suggest areas where further targeted refinement may be beneficial.\n\n---\n\n## Cosine similarity \n\n* Cosine similarity gives a quick automated signal of semantic closeness between the translated/enhanced items and their English originals.\n* Typical mean similarities range from ~0.76 to 0.85 across languages; higher means closer embeddings (better semantic preservation).\n* Use the similarity scores as a triage: low similarity → manual review; high similarity → likely semantically faithful.\n\n| Language               | Mean Similarity | Std Similarity | Min Similarity | Max Similarity |\n|-------------------------|-----------------|----------------|----------------|----------------|\n| mmlu_as_in_qwen3_embed  | 0.8045          | 0.0597         | 0.4644         | 1              |\n| mmlu_bn_in_qwen3_embed  | 0.8133          | 0.0504         | 0.5276         | 1              |\n| mmlu_gu_in_qwen3_embed  | 0.8211          | 0.0535         | 0.4016         | 1              |\n| mmlu_hi_in_qwen3_embed  | 0.8472          | 0.0489         | 0.3688         | 1              |\n| mmlu_kn_in_qwen3_embed  | 0.8106          | 0.0565         | 0.4660         | 1              |\n| mmlu_mai_in_qwen3_embed | 0.8226          | 0.0504         | 0.4936         | 1              |\n| mmlu_ml_in_qwen3_embed  | 0.8158          | 0.0531         | 0.5109         | 1              |\n| mmlu_mr_in_qwen3_embed  | 0.8129          | 0.0513         | 0.5281         | 1              |\n| mmlu_ne_in_qwen3_embed  | 0.8242          | 0.0502         | 0.4802         | 1              |\n| mmlu_or_in_qwen3_embed  | 0.8159          | 0.0555         | 0.4684         | 1              |\n| mmlu_pa_in_qwen3_embed  | 0.8246          | 0.0516         | 0.5450         | 1              |\n| mmlu_sa_in_qwen3_embed  | 0.7912          | 0.0574         | 0.4981         | 1              |\n| mmlu_sdd_in_qwen3_embed | 0.7646          | 0.0735         | 0.3633         | 0.9676         |\n| mmlu_ta_in_qwen3_embed  | 0.7964          | 0.0559         | 0.5242         | 1              |\n| mmlu_te_in_qwen3_embed  | 0.8006          | 0.0524         | 0.5379         | 1              |\n\n---\n\n## Model Evaluation Methodology\n\nWe evaluated **~24 open-source LLMs** across **16 Indic languages** (subset of the 22 total dataset languages) plus English, using the Indic MMLU benchmark.\n\nWorkflow:\n\n1. **Environment Setup:** Dockerized `vllm/vllm-openai` for reproducibility.\n2. **Framework:** `lm-eval` for standardized evaluation.\n3. **Metrics:** per-language accuracy, top-3 rankings per language, aggregate averages.\n4. **Visualization:** radar plots for each language and comparative top-3 plots.\n\n---\n\n## Important Note\n\n* **Indic MMLU Dataset = 22 Indian languages**\n* **Current Model Evaluation = 16 Indian languages + English**\n* Work is ongoing to expand the evaluation to cover all 22 languages.\n\n---\n### Indic MMLU Results \n\n| <div align=\"center\">![Assamese](assets/assamese.png)<br/>*Assamese*</div> | <div align=\"center\">![Bengali](assets/bengali.png)<br/>*Bengali*</div> |\n|---------------------------------------------------------------------------|------------------------------------------------------------------------|\n| <div align=\"center\">![English](assets/English.png)<br/>*English*</div>    | <div align=\"center\">![Gujarati](assets/Gujrati.png)<br/>*Gujarati*</div> |\n| <div align=\"center\">![Hindi](assets/Hindi.png)<br/>*Hindi*</div>          | <div align=\"center\">![Kannada](assets/kannada.png)<br/>*Kannada*</div> |\n| <div align=\"center\">![Maithili](assets/maithili.png)<br/>*Maithili*</div> | <div align=\"center\">![Malayalam](assets/Malayalam.png)<br/>*Malayalam*</div> |\n| <div align=\"center\">![Marathi](assets/marathi.png)<br/>*Marathi*</div>    | <div align=\"center\">![Nepali](assets/Nepali.png)<br/>*Nepali*</div>   |\n| <div align=\"center\">![Oriya](assets/oriya.png)<br/>*Oriya*</div>          | <div align=\"center\">![Punjabi](assets/Punjabi.png)<br/>*Punjabi*</div> |\n| <div align=\"center\">![Sanskrit](assets/sanskrit.png)<br/>*Sanskrit*</div> | <div align=\"center\">![Sindhi](assets/sindhi.png)<br/>*Sindhi*</div>   |\n| <div align=\"center\">![Tamil](assets/Tamil.png)<br/>*Tamil*</div>          | <div align=\"center\">![Telugu](assets/Telugu.png)<br/>*Telugu*</div>   |\n\n\n---\n\n## How to reproduce (quick checklist)\n\n1. Prepare the English CAIS/MMLU test set (`.jsonl` with IDs).\n2. Run `trans_mmlu.py` to produce per-language MT outputs.\n3. Run `async_infr.py` with the enhancement task to obtain human-readable improved translations.\n4. Generate embeddings for both English and enhanced translations.\n5. Run `calculate_cosine_similarity.py` to produce similarity reports.\n6. Optionally run human/linguist rating orchestration for additional quality assessment.\n7. Evaluate models with `lm-eval` and generate visualizations.\n\n---\n\n## Best practices & caveats\n\n* **Math and formatting:** preserve mathematical expressions and numerals during translation—these are critical for correctness.\n* **Choice alignment:** ensure answer choices remain in the same order and map to the same ground-truth labels.\n* **Human review:** automated metrics are helpful but not sufficient; include periodic linguist sampling.\n* **Constrained inference:** use deterministic LLM settings for enhancement (lower temperature, consistent decoding) to reduce hallucination.\n\n---\n\n## Visual assets & tables\n\nWe include per-language radar plots and a consolidated results panel in `plots/`. The README in the repo contains thumbnails; click through for full-resolution visuals. \n\n---\n"}, "5429ee82c29be1f4eec36bd44e6947d88278c4cc": {"t": 1785771728.6240714, "n": 19621, "h": "f5cc1e425cccd0f0", "c": "# Indic MMLU — Benchmarking Open-Source LLMs on Indian Languages\n\nA rigorously prepared, multi-language adaptation of the CAIS/MMLU test set, translated and enhanced for **22 Indic languages**, designed to evaluate open-source LLMs’ real-world ability to understand Indian languages.\nWe additionally conducted large-scale **evaluation on 16 of these Indic languages (plus English)**, producing detailed accuracy reports and per-language visualizations.\n\n---\n\n# Table of Contents\n\n1. [Overview](#overview)\n2. [Why the Need for Indic MMLU](#why-the-need-for-indic-mmlu)\n   - [Data Distillation for Indian Languages](#data-distillation-for-indian-languages)\n   - [The Role of Indic MMLU](#the-role-of-indic-mmlu)\n3. [What This Dataset Has on Offer](#what-this-dataset-has-on-offer)\n4. [Coverage](#coverage)\n5. [High-Level Pipeline](#high-level-pipeline-what-we-did)\n   - [Flowchart Representation of the Indic MMLU Pipeline](#flowchart-representation-of-the-indic-mmlu-pipeline)\n6. [Translation Algorithm](#translation-algorithm)\n7. [Reproducible Commands](#reproducible-commands-examples)\n   - [Translate](#translate)\n   - [Enhance Translations (LLM-based)](#enhance-translations-llm-based)\n   - [Generate Embeddings](#generate-embeddings)\n   - [Cosine Similarity](#cosine-similarity-english-vs-translated-enhancements)\n   - [Linguist / Teacher Rating](#linguist--teacher-rating-llm-assisted-orchestration)\n   - [Model Evaluation with `lm-eval`](#model-evaluation-with-lm-eval)\n8. [Summary of LLM-as-Judge Ratings](#summary-of-llm-as-judge-ratings)\n9. [Cosine Similarity](#cosine-similarity)\n10. [Model Evaluation Methodology](#model-evaluation-methodology)\n11. [Important Note](#important-note)\n12. [Indic MMLU Results](#indic-mmlu-results)\n13. [How to Reproduce (Quick Checklist)](#how-to-reproduce-quick-checklist)\n14. [Best Practices & Caveats](#best-practices--caveats)\n15. [Visual Assets & Tables](#visual-assets--tables)\n\n---\n\n## Why the Need for Indic MMLU\n\nThe rise of large language models (LLMs) has enabled advanced **synthetic data generation**, including:\n\n* Building LLM-based frameworks for **data curation and sandboxing**\n* Generating reasoning and coding examples in non-English languages\n* Creating SFT datasets, as well as off-policy datasets for **RLHF, DPO, GRPO**-style training\n\nTo fully leverage these capabilities for Indian languages, we need LLMs with at least **limited native proficiency** in these languages. Such models allow us to generate high-quality, culturally-aligned datasets, create virtual personas to introduce nuanced constructs, and maintain **full control and flexibility** over the synthetic data pipeline.\n\n### Data Distillation for Indian Languages\n\nData distillation provides a practical approach to achieve this:\n\n* **Easier and cleaner access to world knowledge** from state-of-the-art LLMs\n* **Domain control and flexibility** to shape datasets according to cultural and linguistic requirements\n* Embedding **strong Indian values** into datasets more effectively than via custom pipelines\n* Creating safety datasets that mitigate **anti-Indian stances** and promote positive responses\n\n**Distillation** is the process of generating data from large models to produce **task-specific, high-quality, and culturally-aligned datasets**. Using distillation, we can apply various synthetic data augmentation strategies, such as:\n\n* Chain-of-Thought (CoT) reasoning\n* Multi-turn conversational setups for richer data\n* Summaries and diverse QA pairs\n* Incorporation of strong Indian values through **virtual Indian personas**\n* Other augmentations, including translations and language-specific refinements\n\n### The Role of Indic MMLU\n\nBefore deploying LLMs for such Indian-language data generation, we first need to **evaluate their cross-lingual knowledge transfer capabilities**. Indic MMLU serves this purpose by providing a benchmark to test how well models understand Indian languages, rather than merely translating English content.\n\nWhile previous efforts have translated MMLU into Indian languages, none have **released their methodology**. Indic MMLU is the first benchmark to provide:\n\n* **Open recipe and open code** for translations\n* Coverage of **22 Indian languages**\n* A fully transparent and **reproducible evaluation setup**\n\nThis ensures that models can be assessed rigorously for genuine Indian-language understanding, forming a **foundation for culturally-aware synthetic data pipelines** in future research.\n\n---\n\n## What this dataset has on offer\n\nLarge language models are often benchmarked only on English datasets, which leaves open the question of whether they can *actually* understand Indian languages beyond surface-level translation.\n\nThe **Indic MMLU dataset** bridges this gap by:\n\n* Translating the widely used **[CAIS/MMLU](https://huggingface.co/datasets/cais/mmlu) benchmark** into **22 Indian languages**.\n* Enhancing those translations with high-quality open-source LLM refinement.\n* Providing a reproducible evaluation setup for **measuring open-source models across 16 Indic languages (evaluated subset)**.\n\nIf a model’s score on an Indic version is close to its score on the original English MMLU, it indicates **genuine language understanding** and strong cross-lingual generalization.\n\n---\n\n## Coverage\n\n* **Dataset scope:** 22 Indic languages (full benchmark availability).\n* **Evaluation scope:** 16 Indic languages + English (practically evaluated with ~24 open-source LLMs).\n\nThis separation ensures the dataset is **broad and inclusive**, while evaluation focuses on the languages with the most immediate support and resources.\n\n---\n\n## Overview\n\n* **Source benchmark:** CAIS/MMLU (general knowledge QA benchmark).\n* **Target:** 22 Indic languages + English (translated test set).\n* **Workflow:** Machine translation → LLM-based translation enhancement → embedding comparison → Persona based LLM-as-judge Scores → Correlation between Embeddings and LLM as Judge → human/linguist rating and automated evaluation.\n* **Deliverables:** Enhanced translated `.jsonl` datasets, per-language embeddings, cosine similarity reports, teacher/linguist ratings, model evaluation scores and visualizations (radar plots).\n\n---\n\n## Key principles applied\n\n1. **Preserve semantic parity:** Questions remain semantically the same as English; only language changes.\n2. **Quality control:** Model-Based Automated checks (embeddings, math correctness) + human linguist ratings.\n3. **Reproducibility:** Dockerized inference environment, scriptable pipeline, and clear CLI commands.\n4. **Transparency:** We provide similarity metrics to demonstrate translation fidelity and teacher ratings to highlight linguistic quality.\n\n---\n\n## High-level pipeline (what we did)\n\n1. **Translate** the CAIS/MMLU test set to each Indic language using *IndicTrans2* (baseline MT).\n2. **Enhance** those translations using an open-source LLM with task-specific instructions (to correct phrasing, clarify ambiguous translations, preserve math expressions, and maintain answer choices alignment).\n3. **Embed** both the original English items and the (enhanced) translated items with the same embedding model.\n4. **Compare** embeddings using cosine similarity to measure semantic closeness.\n5. **LLM as Judge and Human review**: linguists/teachers rated samples for language quality, math correctness, and coherence.\n6. **Evaluate models**: run ~24 open-source LLMs across all languages using the lm-eval framework and visualize results (radar plots, top-3 per language).\n\n---\n\n## Indic MMLU Pipeline Construction Pipeline\n\nTo complement the pseudo-code, the following flowchart provides a **visual overview of the Indic MMLU pipeline**. It clearly illustrates the step-by-step workflow, starting from the English MMLU dataset and moving through translation, enhancement, evaluation, and validation:\n\n* **Translation Stage:** The English MMLU instance is translated into Indic languages using specialist MT.\n* **Enhancement Stage:** Translations are refined with large-scale LLMs for improved fluency and alignment.\n* **LLM Based Judging Phase:** Three types of evaluation are conducted — *fluency by native language experts, subject correctness by domain experts, and alignment by coherence judges*.\n* **Validation Stage:** Embedding similarity checks, n-gram metrics (BLEU, ROUGE, ChrF++), and discrepancy detection ensure quality.\n* **Final Output:** A validated Indic MMLU dataset instance ready for benchmarking.\n\n![Indic MMLU Pipeline](assets/indic-mmlu.png)\n\n# Translation Algorithm \n\n```\nInput:\n  original_mmlu\n  target_languages\n  instructions\n\nFor each language L in target_languages:\n    translated = translate(original_mmlu, tgt=L)\n    enhanced = enhance_using_llm(translated, instructions)\n\n    eng_emb = embed(original_mmlu)\n    trans_emb = embed(enhanced)\n    sim_scores = cosine_similarity(eng_emb, trans_emb)\n\n    llm_scores = LLM_as_Judge(original_mmlu, translated)\n\n    consistency = consistency_check(sim_scores, llm_scores)\n\n    If consistency is poor:\n        human_ratings = rate_sample(enhanced)\n    Else:\n        accept enhanced\n\nAggregate all scores:\n    combine sim_scores, llm_scores, human_ratings\n    compute per-language model scores\n\nProduce visualizations:\n    generate charts and plots for cross-language comparison\n\nOutput:\n    per-language evaluation scores\n    visualizations\n\n```\n\nThis visual representation makes it easier to understand how the **algorithmic steps, human evaluation, and automated checks** come together to ensure translation quality and dataset reliability.\n\n---\n\n## Reproducible commands (examples)\n\n> These are the canonical commands used in our pipeline. Replace file paths, ports, and concurrency values as appropriate for your environment.\n\n**Translate**\n\n```bash\npython3 trans_mmlu.py \\\n  --input_file_path test-00000-of-00001_with_ids.jsonl \\\n  --output_file_path mmlu_kn_in.jsonl \\\n  --tgt_lang \"kn\" \\\n  --src_lang \"en\"\n```\n\n**Enhance translations (LLM-based)**\n\n```bash\npython3 async_infr.py \\\n  --input-path mmlu_as_in.jsonl \\\n  --output-file ds_enhance_eval_hi.jsonl \\\n  --instruction-path instruction_prompts.yml \\\n  --task deepseek_enhance_instruct \\\n  --template-fields 'og_question' 'og_choices' 'question' 'answer' 'Hindi' \\\n  --max-concurrency 4096 \\\n  --backend sglang-oai-chat \\\n  --extra-request-body '{\"temperature\":0.7,\"top_p\":0.9,\"top_k\":50,\"max_tokens\":1024}' \\\n  --port 30000\n```\n\n**Generate embeddings**\n\n```bash\npython3 get_embeddings.py \\\n  --input-file test-00000-of-00001_with_ids.jsonl \\\n  --output-file mmlu_qwen3_embd.jsonl \\\n  --server-url http://127.0.0.1:30000 \\\n  --concurrency 4096\n```\n\n**Cosine similarity (English vs translated enhancements)**\n\n```bash\npython3 calculate_cosine_similarity.py \\\n  --input_dir deepseek_enhance \\\n  --output_dir mmmlu/similarity/cosine/enhance/deepseek_enhance \\\n  --english_file mmlu_eng.jsonl\n```\n\n**Linguist / teacher rating (LLM assisted orchestration)**\n\n```bash\npython3 async_infr.py \\\n  --input-path mmlu_as_in.jsonl \\\n  --output-file ds_enhance_eval_as_qwen3_instruct.jsonl \\\n  --instruction-path instruction_prompts.yml \\\n  --task rate_translated_text_linguist \\\n  --template-fields 'og_question' 'og_choices' 'enhanced_text' 'Assamese' \\\n  --max-concurrency 4096 \\\n  --backend sglang-oai-chat \\\n  --extra-request-body '{\"temperature\":0.7,\"top_p\":0.9,\"top_k\":50,\"max_tokens\":1024}' \\\n  --port 30000\n```\n\n**Model evaluation with `lm-eval`**\n\n```bash\n# Example wrapper which runs evaluation for a model snapshot against the Indic MMLU dataset\nbash lm-eval-llm.sh /path/to/model_snapshot /benchmark-result-path\n```\n\n---\n\n### Summary of LLM-as-Judge Ratings\n\nTo evaluate translation quality systematically, we employed **LLMs as judges**, operating under three carefully designed expert personas:\n\n* **Math Expert Persona** → Rated mathematical correctness, terminology, and conceptual fidelity.\n* **Linguistic Expert Persona** → Assessed fluency, grammar, idiomatic usage, and naturalness of the translation.\n* **Coherence Expert Persona** → Evaluated semantic alignment, logical flow, and clarity between English and the target language.\n\n**What we measured:**\n\n* Language quality, fluency, and linguistic suitability.\n* Mathematical correctness and fidelity of technical content.\n* Overall coherence and semantic alignment with the English source.\n\n| Language  | Avg Maths Rate | Avg Coherence Rate | Avg Linguist Rate | Total Records |\n|-----------|----------------|--------------------|-------------------|---------------|\n| Hindi\t    | 9.658029\t     | 9.722054\t          | 9.667891\t        | 14042         |\n| Bengali\t  | 9.182561\t     | 9.131505\t          | 8.202293\t        | 14042         |\n| Nepali    | 9.473793       | 9.331505           | 8.402293          | 14042         |\n| Telugu    | 9.220837       | 8.975929           | 7.986256          | 14042         |\n| Oriya     | 9.264421       | 8.944096           | 8.220268          | 14042         |\n| Punjabi   | 9.348027       | 9.049138           | 8.253454          | 14042         |\n| Assamese  | 9.147201       | 9.039097           | 8.133243          | 14042         |\n| Sanskrit  | 8.338485       | 8.112520           | 6.390115          | 14042         |\n| Kannada   | 9.322746       | 9.111736           | 8.131819          | 14042         |\n| Sindhi    | 2.070645       | 3.109956           | 2.478137          | 14042         |\n| Gujrati   | 9.260647       | 9.206523           | 8.368965          | 14042         |\n| Marathi   | 9.301951       | 9.342045           | 8.381926          | 14042         |\n| Tamil     | 9.047856       | 9.042943           | 8.071144          | 14042         |\n| Malayalam | 9.134952       | 8.998148           | 7.956488          | 14042         |\n| Maithili  | 9.263282       | 9.118146           | 8.218772          | 14042         |\n\n**How to read the table:**\n\n* Ratings are on a **1–10 scale**, where higher scores reflect better translation and enhancement quality.\n* Most languages achieved **consistently high averages (>8.9)**, showing strong quality after enhancement.\n* A few outliers suggest areas where further targeted refinement may be beneficial.\n\n---\n\n## Cosine similarity \n\n* Cosine similarity gives a quick automated signal of semantic closeness between the translated/enhanced items and their English originals.\n* Typical mean similarities range from ~0.76 to 0.85 across languages; higher means closer embeddings (better semantic preservation).\n* Use the similarity scores as a triage: low similarity → manual review; high similarity → likely semantically faithful.\n\n| Language               | Mean Similarity | Std Similarity | Min Similarity | Max Similarity |\n|-------------------------|-----------------|----------------|----------------|----------------|\n| mmlu_as_in_qwen3_embed  | 0.8045          | 0.0597         | 0.4644         | 1              |\n| mmlu_bn_in_qwen3_embed  | 0.8133          | 0.0504         | 0.5276         | 1              |\n| mmlu_gu_in_qwen3_embed  | 0.8211          | 0.0535         | 0.4016         | 1              |\n| mmlu_hi_in_qwen3_embed  | 0.8472          | 0.0489         | 0.3688         | 1              |\n| mmlu_kn_in_qwen3_embed  | 0.8106          | 0.0565         | 0.4660         | 1              |\n| mmlu_mai_in_qwen3_embed | 0.8226          | 0.0504         | 0.4936         | 1              |\n| mmlu_ml_in_qwen3_embed  | 0.8158          | 0.0531         | 0.5109         | 1              |\n| mmlu_mr_in_qwen3_embed  | 0.8129          | 0.0513         | 0.5281         | 1              |\n| mmlu_ne_in_qwen3_embed  | 0.8242          | 0.0502         | 0.4802         | 1              |\n| mmlu_or_in_qwen3_embed  | 0.8159          | 0.0555         | 0.4684         | 1              |\n| mmlu_pa_in_qwen3_embed  | 0.8246          | 0.0516         | 0.5450         | 1              |\n| mmlu_sa_in_qwen3_embed  | 0.7912          | 0.0574         | 0.4981         | 1              |\n| mmlu_sdd_in_qwen3_embed | 0.7646          | 0.0735         | 0.3633         | 0.9676         |\n| mmlu_ta_in_qwen3_embed  | 0.7964          | 0.0559         | 0.5242         | 1              |\n| mmlu_te_in_qwen3_embed  | 0.8006          | 0.0524         | 0.5379         | 1              |\n\n---\n\n## Model Evaluation Methodology\n\nWe evaluated **~24 open-source LLMs** across **16 Indic languages** (subset of the 22 total dataset languages) plus English, using the Indic MMLU benchmark.\n\nWorkflow:\n\n1. **Environment Setup:** Dockerized `vllm/vllm-openai` for reproducibility.\n2. **Framework:** `lm-eval` for standardized evaluation.\n3. **Metrics:** per-language accuracy, top-3 rankings per language, aggregate averages.\n4. **Visualization:** radar plots for each language and comparative top-3 plots.\n\n---\n\n## Important Note\n\n* **Indic MMLU Dataset = 22 Indian languages**\n* **Current Model Evaluation = 16 Indian languages + English**\n* Work is ongoing to expand the evaluation to cover all 22 languages.\n\n---\n### Indic MMLU Results \n\n| <div align=\"center\">![Assamese](assets/assamese.png)<br/>*Assamese*</div> | <div align=\"center\">![Bengali](assets/bengali.png)<br/>*Bengali*</div> |\n|---------------------------------------------------------------------------|------------------------------------------------------------------------|\n| <div align=\"center\">![English](assets/English.png)<br/>*English*</div>    | <div align=\"center\">![Gujarati](assets/Gujrati.png)<br/>*Gujarati*</div> |\n| <div align=\"center\">![Hindi](assets/Hindi.png)<br/>*Hindi*</div>          | <div align=\"center\">![Kannada](assets/kannada.png)<br/>*Kannada*</div> |\n| <div align=\"center\">![Maithili](assets/maithili.png)<br/>*Maithili*</div> | <div align=\"center\">![Malayalam](assets/Malayalam.png)<br/>*Malayalam*</div> |\n| <div align=\"center\">![Marathi](assets/marathi.png)<br/>*Marathi*</div>    | <div align=\"center\">![Nepali](assets/Nepali.png)<br/>*Nepali*</div>   |\n| <div align=\"center\">![Oriya](assets/oriya.png)<br/>*Oriya*</div>          | <div align=\"center\">![Punjabi](assets/Punjabi.png)<br/>*Punjabi*</div> |\n| <div align=\"center\">![Sanskrit](assets/sanskrit.png)<br/>*Sanskrit*</div> | <div align=\"center\">![Sindhi](assets/sindhi.png)<br/>*Sindhi*</div>   |\n| <div align=\"center\">![Tamil](assets/Tamil.png)<br/>*Tamil*</div>          | <div align=\"center\">![Telugu](assets/Telugu.png)<br/>*Telugu*</div>   |\n\n\n---\n\n## How to reproduce (quick checklist)\n\n1. Prepare the English CAIS/MMLU test set (`.jsonl` with IDs).\n2. Run `trans_mmlu.py` to produce per-language MT outputs.\n3. Run `async_infr.py` with the enhancement task to obtain human-readable improved translations.\n4. Generate embeddings for both English and enhanced translations.\n5. Run `calculate_cosine_similarity.py` to produce similarity reports.\n6. Optionally run human/linguist rating orchestration for additional quality assessment.\n7. Evaluate models with `lm-eval` and generate visualizations.\n\n---\n\n## Best practices & caveats\n\n* **Math and formatting:** preserve mathematical expressions and numerals during translation—these are critical for correctness.\n* **Choice alignment:** ensure answer choices remain in the same order and map to the same ground-truth labels.\n* **Human review:** automated metrics are helpful but not sufficient; include periodic linguist sampling.\n* **Constrained inference:** use deterministic LLM settings for enhancement (lower temperature, consistent decoding) to reduce hallucination.\n\n---\n\n## Visual assets & tables\n\nWe include per-language radar plots and a consolidated results panel in `plots/`. The README in the repo contains thumbnails; click through for full-resolution visuals. \n\n---\n"}, "ff0d81489659a3581ab86417edc338051bd8b612": {"t": 1785771730.2620528, "n": 19631, "h": "4a595905dc7cb287", "c": "# Indic MMLU — Benchmarking Open-Source LLMs on Indian Languages\n\nA rigorously prepared, multi-language adaptation of the CAIS/MMLU test set, translated and enhanced for **22 Indic languages**, designed to evaluate open-source LLMs’ real-world ability to understand Indian languages.\nWe additionally conducted large-scale **evaluation on 16 of these Indic languages (plus English)**, producing detailed accuracy reports and per-language visualizations.\n\n---\n\n# Table of Contents\n\n1. [Overview](#overview)\n2. [Why the Need for Indic MMLU](#why-the-need-for-indic-mmlu)\n   - [Data Distillation for Indian Languages](#data-distillation-for-indian-languages)\n   - [The Role of Indic MMLU](#the-role-of-indic-mmlu)\n3. [What This Dataset Has on Offer](#what-this-dataset-has-on-offer)\n4. [Coverage](#coverage)\n5. [High-Level Pipeline](#high-level-pipeline-what-we-did)\n   - [Flowchart Representation of the Indic MMLU Pipeline](#flowchart-representation-of-the-indic-mmlu-pipeline)\n6. [Translation Algorithm](#translation-algorithm)\n7. [Reproducible Commands](#reproducible-commands-examples)\n   - [Translate](#translate)\n   - [Enhance Translations (LLM-based)](#enhance-translations-llm-based)\n   - [Generate Embeddings](#generate-embeddings)\n   - [Cosine Similarity](#cosine-similarity-english-vs-translated-enhancements)\n   - [Linguist / Teacher Rating](#linguist--teacher-rating-llm-assisted-orchestration)\n   - [Model Evaluation with `lm-eval`](#model-evaluation-with-lm-eval)\n8. [Summary of LLM-as-Judge Ratings](#summary-of-llm-as-judge-ratings)\n9. [Cosine Similarity](#cosine-similarity)\n10. [Model Evaluation Methodology](#model-evaluation-methodology)\n11. [Important Note](#important-note)\n12. [Indic MMLU Results](#indic-mmlu-results)\n13. [How to Reproduce (Quick Checklist)](#how-to-reproduce-quick-checklist)\n14. [Best Practices & Caveats](#best-practices--caveats)\n15. [Visual Assets & Tables](#visual-assets--tables)\n\n---\n\n## Why the Need for Indic MMLU\n\nThe rise of large language models (LLMs) has enabled advanced **synthetic data generation**, including:\n\n* Building LLM-based frameworks for **data curation and sandboxing**\n* Generating reasoning and coding examples in non-English languages\n* Creating SFT datasets, as well as off-policy datasets for **RLHF, DPO, GRPO**-style training\n\nTo fully leverage these capabilities for Indian languages, we need LLMs with at least **limited native proficiency** in these languages. Such models allow us to generate high-quality, culturally-aligned datasets, create virtual personas to introduce nuanced constructs, and maintain **full control and flexibility** over the synthetic data pipeline.\n\n### Data Distillation for Indian Languages\n\nData distillation provides a practical approach to achieve this:\n\n* **Easier and cleaner access to world knowledge** from state-of-the-art LLMs\n* **Domain control and flexibility** to shape datasets according to cultural and linguistic requirements\n* Embedding **strong Indian values** into datasets more effectively than via custom pipelines\n* Creating safety datasets that mitigate **anti-Indian stances** and promote positive responses\n\n**Distillation** is the process of generating data from large models to produce **task-specific, high-quality, and culturally-aligned datasets**. Using distillation, we can apply various synthetic data augmentation strategies, such as:\n\n* Chain-of-Thought (CoT) reasoning\n* Multi-turn conversational setups for richer data\n* Summaries and diverse QA pairs\n* Incorporation of strong Indian values through **virtual Indian personas**\n* Other augmentations, including translations and language-specific refinements\n\n### The Role of Indic MMLU\n\nBefore deploying LLMs for such Indian-language data generation, we first need to **evaluate their cross-lingual knowledge transfer capabilities**. Indic MMLU serves this purpose by providing a benchmark to test how well models understand Indian languages, rather than merely translating English content.\n\nWhile previous efforts have translated MMLU into Indian languages, none have **released their methodology**. Indic MMLU is the first benchmark to provide:\n\n* **Open recipe and open code** for translations\n* Coverage of **22 Indian languages**\n* A fully transparent and **reproducible evaluation setup**\n\nThis ensures that models can be assessed rigorously for genuine Indian-language understanding, forming a **foundation for culturally-aware synthetic data pipelines** in future research.\n\n---\n\n## What this dataset has on offer\n\nLarge language models are often benchmarked only on English datasets, which leaves open the question of whether they can *actually* understand Indian languages beyond surface-level translation.\n\nThe **Indic MMLU dataset** bridges this gap by:\n\n* Translating the widely used **[CAIS/MMLU](https://huggingface.co/datasets/cais/mmlu) benchmark** into **22 Indian languages**.\n* Enhancing those translations with high-quality open-source LLM refinement.\n* Providing a reproducible evaluation setup for **measuring open-source models across 16 Indic languages (evaluated subset)**.\n\nIf a model’s score on an Indic version is close to its score on the original English MMLU, it indicates **genuine language understanding** and strong cross-lingual generalization.\n\n---\n\n## Coverage\n\n* **Dataset scope:** 22 Indic languages (full benchmark availability).\n* **Evaluation scope:** 16 Indic languages + English (practically evaluated with ~24 open-source LLMs).\n\nThis separation ensures the dataset is **broad and inclusive**, while evaluation focuses on the languages with the most immediate support and resources.\n\n---\n\n## Overview\n\n* **Source benchmark:** CAIS/MMLU (general knowledge QA benchmark).\n* **Target:** 22 Indic languages + English (translated test set).\n* **Workflow:** Machine translation → LLM-based translation enhancement → embedding comparison → Persona based LLM-as-judge Scores → Correlation between Embeddings and LLM as Judge → human/linguist rating and automated evaluation.\n* **Deliverables:** Enhanced translated `.jsonl` datasets, per-language embeddings, cosine similarity reports, teacher/linguist ratings, model evaluation scores and visualizations (radar plots).\n\n---\n\n## Key principles applied\n\n1. **Preserve semantic parity:** Questions remain semantically the same as English; only language changes.\n2. **Quality control:** Model-Based Automated checks (embeddings, math correctness) + human linguist ratings.\n3. **Reproducibility:** Dockerized inference environment, scriptable pipeline, and clear CLI commands.\n4. **Transparency:** We provide similarity metrics to demonstrate translation fidelity and teacher ratings to highlight linguistic quality.\n\n---\n\n## High-level pipeline (what we did)\n\n1. **Translate** the CAIS/MMLU test set to each Indic language using *IndicTrans2* (baseline MT).\n2. **Enhance** those translations using an open-source LLM with task-specific instructions (to correct phrasing, clarify ambiguous translations, preserve math expressions, and maintain answer choices alignment).\n3. **Embed** both the original English items and the (enhanced) translated items with the same embedding model.\n4. **Compare** embeddings using cosine similarity to measure semantic closeness.\n5. **LLM as Judge and Human review**: linguists/teachers rated samples for language quality, math correctness, and coherence.\n6. **Evaluate models**: run ~24 open-source LLMs across all languages using the lm-eval framework and visualize results (radar plots, top-3 per language).\n\n---\n\n## Flowchart Representation of the Indic MMLU Pipeline\n\nTo complement the pseudo-code, the following flowchart provides a **visual overview of the Indic MMLU pipeline**. It clearly illustrates the step-by-step workflow, starting from the English MMLU dataset and moving through translation, enhancement, evaluation, and validation:\n\n* **Translation Stage:** The English MMLU instance is translated into Indic languages using specialist MT.\n* **Enhancement Stage:** Translations are refined with large-scale LLMs for improved fluency and alignment.\n* **LLM Based Judging Phase:** Three types of evaluation are conducted — *fluency by native language experts, subject correctness by domain experts, and alignment by coherence judges*.\n* **Validation Stage:** Embedding similarity checks, n-gram metrics (BLEU, ROUGE, ChrF++), and discrepancy detection ensure quality.\n* **Final Output:** A validated Indic MMLU dataset instance ready for benchmarking.\n\n![Indic MMLU Pipeline](assets/indic-mmlu.png)\n\n# Translation Algorithm \n\n```\nInput:\n  original_mmlu\n  target_languages\n  instructions\n\nFor each language L in target_languages:\n    translated = translate(original_mmlu, tgt=L)\n    enhanced = enhance_using_llm(translated, instructions)\n\n    eng_emb = embed(original_mmlu)\n    trans_emb = embed(enhanced)\n    sim_scores = cosine_similarity(eng_emb, trans_emb)\n\n    llm_scores = LLM_as_Judge(original_mmlu, translated)\n\n    consistency = consistency_check(sim_scores, llm_scores)\n\n    If consistency is poor:\n        human_ratings = rate_sample(enhanced)\n    Else:\n        accept enhanced\n\nAggregate all scores:\n    combine sim_scores, llm_scores, human_ratings\n    compute per-language model scores\n\nProduce visualizations:\n    generate charts and plots for cross-language comparison\n\nOutput:\n    per-language evaluation scores\n    visualizations\n\n```\n\nThis visual representation makes it easier to understand how the **algorithmic steps, human evaluation, and automated checks** come together to ensure translation quality and dataset reliability.\n\n---\n\n## Reproducible commands (examples)\n\n> These are the canonical commands used in our pipeline. Replace file paths, ports, and concurrency values as appropriate for your environment.\n\n**Translate**\n\n```bash\npython3 trans_mmlu.py \\\n  --input_file_path test-00000-of-00001_with_ids.jsonl \\\n  --output_file_path mmlu_kn_in.jsonl \\\n  --tgt_lang \"kn\" \\\n  --src_lang \"en\"\n```\n\n**Enhance translations (LLM-based)**\n\n```bash\npython3 async_infr.py \\\n  --input-path mmlu_as_in.jsonl \\\n  --output-file ds_enhance_eval_hi.jsonl \\\n  --instruction-path instruction_prompts.yml \\\n  --task deepseek_enhance_instruct \\\n  --template-fields 'og_question' 'og_choices' 'question' 'answer' 'Hindi' \\\n  --max-concurrency 4096 \\\n  --backend sglang-oai-chat \\\n  --extra-request-body '{\"temperature\":0.7,\"top_p\":0.9,\"top_k\":50,\"max_tokens\":1024}' \\\n  --port 30000\n```\n\n**Generate embeddings**\n\n```bash\npython3 get_embeddings.py \\\n  --input-file test-00000-of-00001_with_ids.jsonl \\\n  --output-file mmlu_qwen3_embd.jsonl \\\n  --server-url http://127.0.0.1:30000 \\\n  --concurrency 4096\n```\n\n**Cosine similarity (English vs translated enhancements)**\n\n```bash\npython3 calculate_cosine_similarity.py \\\n  --input_dir deepseek_enhance \\\n  --output_dir mmmlu/similarity/cosine/enhance/deepseek_enhance \\\n  --english_file mmlu_eng.jsonl\n```\n\n**Linguist / teacher rating (LLM assisted orchestration)**\n\n```bash\npython3 async_infr.py \\\n  --input-path mmlu_as_in.jsonl \\\n  --output-file ds_enhance_eval_as_qwen3_instruct.jsonl \\\n  --instruction-path instruction_prompts.yml \\\n  --task rate_translated_text_linguist \\\n  --template-fields 'og_question' 'og_choices' 'enhanced_text' 'Assamese' \\\n  --max-concurrency 4096 \\\n  --backend sglang-oai-chat \\\n  --extra-request-body '{\"temperature\":0.7,\"top_p\":0.9,\"top_k\":50,\"max_tokens\":1024}' \\\n  --port 30000\n```\n\n**Model evaluation with `lm-eval`**\n\n```bash\n# Example wrapper which runs evaluation for a model snapshot against the Indic MMLU dataset\nbash lm-eval-llm.sh /path/to/model_snapshot /benchmark-result-path\n```\n\n---\n\n### Summary of LLM-as-Judge Ratings\n\nTo evaluate translation quality systematically, we employed **LLMs as judges**, operating under three carefully designed expert personas:\n\n* **Math Expert Persona** → Rated mathematical correctness, terminology, and conceptual fidelity.\n* **Linguistic Expert Persona** → Assessed fluency, grammar, idiomatic usage, and naturalness of the translation.\n* **Coherence Expert Persona** → Evaluated semantic alignment, logical flow, and clarity between English and the target language.\n\n**What we measured:**\n\n* Language quality, fluency, and linguistic suitability.\n* Mathematical correctness and fidelity of technical content.\n* Overall coherence and semantic alignment with the English source.\n\n| Language  | Avg Maths Rate | Avg Coherence Rate | Avg Linguist Rate | Total Records |\n|-----------|----------------|--------------------|-------------------|---------------|\n| Hindi\t    | 9.658029\t     | 9.722054\t          | 9.667891\t        | 14042         |\n| Bengali\t  | 9.182561\t     | 9.131505\t          | 8.202293\t        | 14042         |\n| Nepali    | 9.473793       | 9.331505           | 8.402293          | 14042         |\n| Telugu    | 9.220837       | 8.975929           | 7.986256          | 14042         |\n| Oriya     | 9.264421       | 8.944096           | 8.220268          | 14042         |\n| Punjabi   | 9.348027       | 9.049138           | 8.253454          | 14042         |\n| Assamese  | 9.147201       | 9.039097           | 8.133243          | 14042         |\n| Sanskrit  | 8.338485       | 8.112520           | 6.390115          | 14042         |\n| Kannada   | 9.322746       | 9.111736           | 8.131819          | 14042         |\n| Sindhi    | 2.070645       | 3.109956           | 2.478137          | 14042         |\n| Gujrati   | 9.260647       | 9.206523           | 8.368965          | 14042         |\n| Marathi   | 9.301951       | 9.342045           | 8.381926          | 14042         |\n| Tamil     | 9.047856       | 9.042943           | 8.071144          | 14042         |\n| Malayalam | 9.134952       | 8.998148           | 7.956488          | 14042         |\n| Maithili  | 9.263282       | 9.118146           | 8.218772          | 14042         |\n\n**How to read the table:**\n\n* Ratings are on a **1–10 scale**, where higher scores reflect better translation and enhancement quality.\n* Most languages achieved **consistently high averages (>8.9)**, showing strong quality after enhancement.\n* A few outliers suggest areas where further targeted refinement may be beneficial.\n\n---\n\n## Cosine similarity \n\n* Cosine similarity gives a quick automated signal of semantic closeness between the translated/enhanced items and their English originals.\n* Typical mean similarities range from ~0.76 to 0.85 across languages; higher means closer embeddings (better semantic preservation).\n* Use the similarity scores as a triage: low similarity → manual review; high similarity → likely semantically faithful.\n\n| Language               | Mean Similarity | Std Similarity | Min Similarity | Max Similarity |\n|-------------------------|-----------------|----------------|----------------|----------------|\n| mmlu_as_in_qwen3_embed  | 0.8045          | 0.0597         | 0.4644         | 1              |\n| mmlu_bn_in_qwen3_embed  | 0.8133          | 0.0504         | 0.5276         | 1              |\n| mmlu_gu_in_qwen3_embed  | 0.8211          | 0.0535         | 0.4016         | 1              |\n| mmlu_hi_in_qwen3_embed  | 0.8472          | 0.0489         | 0.3688         | 1              |\n| mmlu_kn_in_qwen3_embed  | 0.8106          | 0.0565         | 0.4660         | 1              |\n| mmlu_mai_in_qwen3_embed | 0.8226          | 0.0504         | 0.4936         | 1              |\n| mmlu_ml_in_qwen3_embed  | 0.8158          | 0.0531         | 0.5109         | 1              |\n| mmlu_mr_in_qwen3_embed  | 0.8129          | 0.0513         | 0.5281         | 1              |\n| mmlu_ne_in_qwen3_embed  | 0.8242          | 0.0502         | 0.4802         | 1              |\n| mmlu_or_in_qwen3_embed  | 0.8159          | 0.0555         | 0.4684         | 1              |\n| mmlu_pa_in_qwen3_embed  | 0.8246          | 0.0516         | 0.5450         | 1              |\n| mmlu_sa_in_qwen3_embed  | 0.7912          | 0.0574         | 0.4981         | 1              |\n| mmlu_sdd_in_qwen3_embed | 0.7646          | 0.0735         | 0.3633         | 0.9676         |\n| mmlu_ta_in_qwen3_embed  | 0.7964          | 0.0559         | 0.5242         | 1              |\n| mmlu_te_in_qwen3_embed  | 0.8006          | 0.0524         | 0.5379         | 1              |\n\n---\n\n## Model Evaluation Methodology\n\nWe evaluated **~24 open-source LLMs** across **16 Indic languages** (subset of the 22 total dataset languages) plus English, using the Indic MMLU benchmark.\n\nWorkflow:\n\n1. **Environment Setup:** Dockerized `vllm/vllm-openai` for reproducibility.\n2. **Framework:** `lm-eval` for standardized evaluation.\n3. **Metrics:** per-language accuracy, top-3 rankings per language, aggregate averages.\n4. **Visualization:** radar plots for each language and comparative top-3 plots.\n\n---\n\n## Important Note\n\n* **Indic MMLU Dataset = 22 Indian languages**\n* **Current Model Evaluation = 16 Indian languages + English**\n* Work is ongoing to expand the evaluation to cover all 22 languages.\n\n---\n### Indic MMLU Results \n\n| <div align=\"center\">![Assamese](assets/assamese.png)<br/>*Assamese*</div> | <div align=\"center\">![Bengali](assets/bengali.png)<br/>*Bengali*</div> |\n|---------------------------------------------------------------------------|------------------------------------------------------------------------|\n| <div align=\"center\">![English](assets/English.png)<br/>*English*</div>    | <div align=\"center\">![Gujarati](assets/Gujrati.png)<br/>*Gujarati*</div> |\n| <div align=\"center\">![Hindi](assets/Hindi.png)<br/>*Hindi*</div>          | <div align=\"center\">![Kannada](assets/kannada.png)<br/>*Kannada*</div> |\n| <div align=\"center\">![Maithili](assets/maithili.png)<br/>*Maithili*</div> | <div align=\"center\">![Malayalam](assets/Malayalam.png)<br/>*Malayalam*</div> |\n| <div align=\"center\">![Marathi](assets/marathi.png)<br/>*Marathi*</div>    | <div align=\"center\">![Nepali](assets/Nepali.png)<br/>*Nepali*</div>   |\n| <div align=\"center\">![Oriya](assets/oriya.png)<br/>*Oriya*</div>          | <div align=\"center\">![Punjabi](assets/Punjabi.png)<br/>*Punjabi*</div> |\n| <div align=\"center\">![Sanskrit](assets/sanskrit.png)<br/>*Sanskrit*</div> | <div align=\"center\">![Sindhi](assets/sindhi.png)<br/>*Sindhi*</div>   |\n| <div align=\"center\">![Tamil](assets/Tamil.png)<br/>*Tamil*</div>          | <div align=\"center\">![Telugu](assets/Telugu.png)<br/>*Telugu*</div>   |\n\n\n---\n\n## How to reproduce (quick checklist)\n\n1. Prepare the English CAIS/MMLU test set (`.jsonl` with IDs).\n2. Run `trans_mmlu.py` to produce per-language MT outputs.\n3. Run `async_infr.py` with the enhancement task to obtain human-readable improved translations.\n4. Generate embeddings for both English and enhanced translations.\n5. Run `calculate_cosine_similarity.py` to produce similarity reports.\n6. Optionally run human/linguist rating orchestration for additional quality assessment.\n7. Evaluate models with `lm-eval` and generate visualizations.\n\n---\n\n## Best practices & caveats\n\n* **Math and formatting:** preserve mathematical expressions and numerals during translation—these are critical for correctness.\n* **Choice alignment:** ensure answer choices remain in the same order and map to the same ground-truth labels.\n* **Human review:** automated metrics are helpful but not sufficient; include periodic linguist sampling.\n* **Constrained inference:** use deterministic LLM settings for enhancement (lower temperature, consistent decoding) to reduce hallucination.\n\n---\n\n## Visual assets & tables\n\nWe include per-language radar plots and a consolidated results panel in `plots/`. The README in the repo contains thumbnails; click through for full-resolution visuals. \n\n---\n"}, "f68791505286c760f5d85343138c0db9b16a427b": {"t": 1785771732.0458388, "n": 19654, "h": "8c4716bc8fa6cf68", "c": "# Indic MMLU — Benchmarking Open-Source LLMs on Indian Languages\n\n**Short description:**\nA rigorously prepared, multi-language adaptation of the CAIS/MMLU test set, translated and enhanced for **22 Indic languages**, designed to evaluate open-source LLMs’ real-world ability to understand Indian languages.\nWe additionally conducted large-scale **evaluation on 16 of these Indic languages (plus English)**, producing detailed accuracy reports and per-language visualizations.\n\n---\n\n# Table of Contents\n\n1. [Overview](#overview)\n2. [Why the Need for Indic MMLU](#why-the-need-for-indic-mmlu)\n   - [Data Distillation for Indian Languages](#data-distillation-for-indian-languages)\n   - [The Role of Indic MMLU](#the-role-of-indic-mmlu)\n3. [What This Dataset Has on Offer](#what-this-dataset-has-on-offer)\n4. [Coverage](#coverage)\n5. [High-Level Pipeline](#high-level-pipeline-what-we-did)\n   - [Flowchart Representation of the Indic MMLU Pipeline](#flowchart-representation-of-the-indic-mmlu-pipeline)\n6. [Translation Algorithm](#translation-algorithm)\n7. [Reproducible Commands](#reproducible-commands-examples)\n   - [Translate](#translate)\n   - [Enhance Translations (LLM-based)](#enhance-translations-llm-based)\n   - [Generate Embeddings](#generate-embeddings)\n   - [Cosine Similarity](#cosine-similarity-english-vs-translated-enhancements)\n   - [Linguist / Teacher Rating](#linguist--teacher-rating-llm-assisted-orchestration)\n   - [Model Evaluation with `lm-eval`](#model-evaluation-with-lm-eval)\n8. [Summary of LLM-as-Judge Ratings](#summary-of-llm-as-judge-ratings)\n9. [Cosine Similarity](#cosine-similarity)\n10. [Model Evaluation Methodology](#model-evaluation-methodology)\n11. [Important Note](#important-note)\n12. [Indic MMLU Results](#indic-mmlu-results)\n13. [How to Reproduce (Quick Checklist)](#how-to-reproduce-quick-checklist)\n14. [Best Practices & Caveats](#best-practices--caveats)\n15. [Visual Assets & Tables](#visual-assets--tables)\n\n---\n\n## Why the Need for Indic MMLU\n\nThe rise of large language models (LLMs) has enabled advanced **synthetic data generation**, including:\n\n* Building LLM-based frameworks for **data curation and sandboxing**\n* Generating reasoning and coding examples in non-English languages\n* Creating SFT datasets, as well as off-policy datasets for **RLHF, DPO, GRPO**-style training\n\nTo fully leverage these capabilities for Indian languages, we need LLMs with at least **limited native proficiency** in these languages. Such models allow us to generate high-quality, culturally-aligned datasets, create virtual personas to introduce nuanced constructs, and maintain **full control and flexibility** over the synthetic data pipeline.\n\n### Data Distillation for Indian Languages\n\nData distillation provides a practical approach to achieve this:\n\n* **Easier and cleaner access to world knowledge** from state-of-the-art LLMs\n* **Domain control and flexibility** to shape datasets according to cultural and linguistic requirements\n* Embedding **strong Indian values** into datasets more effectively than via custom pipelines\n* Creating safety datasets that mitigate **anti-Indian stances** and promote positive responses\n\n**Distillation** is the process of generating data from large models to produce **task-specific, high-quality, and culturally-aligned datasets**. Using distillation, we can apply various synthetic data augmentation strategies, such as:\n\n* Chain-of-Thought (CoT) reasoning\n* Multi-turn conversational setups for richer data\n* Summaries and diverse QA pairs\n* Incorporation of strong Indian values through **virtual Indian personas**\n* Other augmentations, including translations and language-specific refinements\n\n### The Role of Indic MMLU\n\nBefore deploying LLMs for such Indian-language data generation, we first need to **evaluate their cross-lingual knowledge transfer capabilities**. Indic MMLU serves this purpose by providing a benchmark to test how well models understand Indian languages, rather than merely translating English content.\n\nWhile previous efforts have translated MMLU into Indian languages, none have **released their methodology**. Indic MMLU is the first benchmark to provide:\n\n* **Open recipe and open code** for translations\n* Coverage of **22 Indian languages**\n* A fully transparent and **reproducible evaluation setup**\n\nThis ensures that models can be assessed rigorously for genuine Indian-language understanding, forming a **foundation for culturally-aware synthetic data pipelines** in future research.\n\n---\n\n## What this dataset has on offer\n\nLarge language models are often benchmarked only on English datasets, which leaves open the question of whether they can *actually* understand Indian languages beyond surface-level translation.\n\nThe **Indic MMLU dataset** bridges this gap by:\n\n* Translating the widely used **[CAIS/MMLU](https://huggingface.co/datasets/cais/mmlu) benchmark** into **22 Indian languages**.\n* Enhancing those translations with high-quality open-source LLM refinement.\n* Providing a reproducible evaluation setup for **measuring open-source models across 16 Indic languages (evaluated subset)**.\n\nIf a model’s score on an Indic version is close to its score on the original English MMLU, it indicates **genuine language understanding** and strong cross-lingual generalization.\n\n---\n\n## Coverage\n\n* **Dataset scope:** 22 Indic languages (full benchmark availability).\n* **Evaluation scope:** 16 Indic languages + English (practically evaluated with ~24 open-source LLMs).\n\nThis separation ensures the dataset is **broad and inclusive**, while evaluation focuses on the languages with the most immediate support and resources.\n\n---\n\n## Overview\n\n* **Source benchmark:** CAIS/MMLU (general knowledge QA benchmark).\n* **Target:** 22 Indic languages + English (translated test set).\n* **Workflow:** Machine translation → LLM-based translation enhancement → embedding comparison → Persona based LLM-as-judge Scores → Correlation between Embeddings and LLM as Judge → human/linguist rating and automated evaluation.\n* **Deliverables:** Enhanced translated `.jsonl` datasets, per-language embeddings, cosine similarity reports, teacher/linguist ratings, model evaluation scores and visualizations (radar plots).\n\n---\n\n## Key principles applied\n\n1. **Preserve semantic parity:** Questions remain semantically the same as English; only language changes.\n2. **Quality control:** Model-Based Automated checks (embeddings, math correctness) + human linguist ratings.\n3. **Reproducibility:** Dockerized inference environment, scriptable pipeline, and clear CLI commands.\n4. **Transparency:** We provide similarity metrics to demonstrate translation fidelity and teacher ratings to highlight linguistic quality.\n\n---\n\n## High-level pipeline (what we did)\n\n1. **Translate** the CAIS/MMLU test set to each Indic language using *IndicTrans2* (baseline MT).\n2. **Enhance** those translations using an open-source LLM with task-specific instructions (to correct phrasing, clarify ambiguous translations, preserve math expressions, and maintain answer choices alignment).\n3. **Embed** both the original English items and the (enhanced) translated items with the same embedding model.\n4. **Compare** embeddings using cosine similarity to measure semantic closeness.\n5. **LLM as Judge and Human review**: linguists/teachers rated samples for language quality, math correctness, and coherence.\n6. **Evaluate models**: run ~24 open-source LLMs across all languages using the lm-eval framework and visualize results (radar plots, top-3 per language).\n\n---\n\n## Flowchart Representation of the Indic MMLU Pipeline\n\nTo complement the pseudo-code, the following flowchart provides a **visual overview of the Indic MMLU pipeline**. It clearly illustrates the step-by-step workflow, starting from the English MMLU dataset and moving through translation, enhancement, evaluation, and validation:\n\n* **Translation Stage:** The English MMLU instance is translated into Indic languages using specialist MT.\n* **Enhancement Stage:** Translations are refined with large-scale LLMs for improved fluency and alignment.\n* **LLM Based Judging Phase:** Three types of evaluation are conducted — *fluency by native language experts, subject correctness by domain experts, and alignment by coherence judges*.\n* **Validation Stage:** Embedding similarity checks, n-gram metrics (BLEU, ROUGE, ChrF++), and discrepancy detection ensure quality.\n* **Final Output:** A validated Indic MMLU dataset instance ready for benchmarking.\n\n![Indic MMLU Pipeline](assets/indic-mmlu.png)\n\n# Translation Algorithm \n\n```\nInput:\n  original_mmlu\n  target_languages\n  instructions\n\nFor each language L in target_languages:\n    translated = translate(original_mmlu, tgt=L)\n    enhanced = enhance_using_llm(translated, instructions)\n\n    eng_emb = embed(original_mmlu)\n    trans_emb = embed(enhanced)\n    sim_scores = cosine_similarity(eng_emb, trans_emb)\n\n    llm_scores = LLM_as_Judge(original_mmlu, translated)\n\n    consistency = consistency_check(sim_scores, llm_scores)\n\n    If consistency is poor:\n        human_ratings = rate_sample(enhanced)\n    Else:\n        accept enhanced\n\nAggregate all scores:\n    combine sim_scores, llm_scores, human_ratings\n    compute per-language model scores\n\nProduce visualizations:\n    generate charts and plots for cross-language comparison\n\nOutput:\n    per-language evaluation scores\n    visualizations\n\n```\n\nThis visual representation makes it easier to understand how the **algorithmic steps, human evaluation, and automated checks** come together to ensure translation quality and dataset reliability.\n\n---\n\n## Reproducible commands (examples)\n\n> These are the canonical commands used in our pipeline. Replace file paths, ports, and concurrency values as appropriate for your environment.\n\n**Translate**\n\n```bash\npython3 trans_mmlu.py \\\n  --input_file_path test-00000-of-00001_with_ids.jsonl \\\n  --output_file_path mmlu_kn_in.jsonl \\\n  --tgt_lang \"kn\" \\\n  --src_lang \"en\"\n```\n\n**Enhance translations (LLM-based)**\n\n```bash\npython3 async_infr.py \\\n  --input-path mmlu_as_in.jsonl \\\n  --output-file ds_enhance_eval_hi.jsonl \\\n  --instruction-path instruction_prompts.yml \\\n  --task deepseek_enhance_instruct \\\n  --template-fields 'og_question' 'og_choices' 'question' 'answer' 'Hindi' \\\n  --max-concurrency 4096 \\\n  --backend sglang-oai-chat \\\n  --extra-request-body '{\"temperature\":0.7,\"top_p\":0.9,\"top_k\":50,\"max_tokens\":1024}' \\\n  --port 30000\n```\n\n**Generate embeddings**\n\n```bash\npython3 get_embeddings.py \\\n  --input-file test-00000-of-00001_with_ids.jsonl \\\n  --output-file mmlu_qwen3_embd.jsonl \\\n  --server-url http://127.0.0.1:30000 \\\n  --concurrency 4096\n```\n\n**Cosine similarity (English vs translated enhancements)**\n\n```bash\npython3 calculate_cosine_similarity.py \\\n  --input_dir deepseek_enhance \\\n  --output_dir mmmlu/similarity/cosine/enhance/deepseek_enhance \\\n  --english_file mmlu_eng.jsonl\n```\n\n**Linguist / teacher rating (LLM assisted orchestration)**\n\n```bash\npython3 async_infr.py \\\n  --input-path mmlu_as_in.jsonl \\\n  --output-file ds_enhance_eval_as_qwen3_instruct.jsonl \\\n  --instruction-path instruction_prompts.yml \\\n  --task rate_translated_text_linguist \\\n  --template-fields 'og_question' 'og_choices' 'enhanced_text' 'Assamese' \\\n  --max-concurrency 4096 \\\n  --backend sglang-oai-chat \\\n  --extra-request-body '{\"temperature\":0.7,\"top_p\":0.9,\"top_k\":50,\"max_tokens\":1024}' \\\n  --port 30000\n```\n\n**Model evaluation with `lm-eval`**\n\n```bash\n# Example wrapper which runs evaluation for a model snapshot against the Indic MMLU dataset\nbash lm-eval-llm.sh /path/to/model_snapshot /benchmark-result-path\n```\n\n---\n\n### Summary of LLM-as-Judge Ratings\n\nTo evaluate translation quality systematically, we employed **LLMs as judges**, operating under three carefully designed expert personas:\n\n* **Math Expert Persona** → Rated mathematical correctness, terminology, and conceptual fidelity.\n* **Linguistic Expert Persona** → Assessed fluency, grammar, idiomatic usage, and naturalness of the translation.\n* **Coherence Expert Persona** → Evaluated semantic alignment, logical flow, and clarity between English and the target language.\n\n**What we measured:**\n\n* Language quality, fluency, and linguistic suitability.\n* Mathematical correctness and fidelity of technical content.\n* Overall coherence and semantic alignment with the English source.\n\n| Language  | Avg Maths Rate | Avg Coherence Rate | Avg Linguist Rate | Total Records |\n|-----------|----------------|--------------------|-------------------|---------------|\n| Hindi\t    | 9.658029\t     | 9.722054\t          | 9.667891\t        | 14042         |\n| Bengali\t  | 9.182561\t     | 9.131505\t          | 8.202293\t        | 14042         |\n| Nepali    | 9.473793       | 9.331505           | 8.402293          | 14042         |\n| Telugu    | 9.220837       | 8.975929           | 7.986256          | 14042         |\n| Oriya     | 9.264421       | 8.944096           | 8.220268          | 14042         |\n| Punjabi   | 9.348027       | 9.049138           | 8.253454          | 14042         |\n| Assamese  | 9.147201       | 9.039097           | 8.133243          | 14042         |\n| Sanskrit  | 8.338485       | 8.112520           | 6.390115          | 14042         |\n| Kannada   | 9.322746       | 9.111736           | 8.131819          | 14042         |\n| Sindhi    | 2.070645       | 3.109956           | 2.478137          | 14042         |\n| Gujrati   | 9.260647       | 9.206523           | 8.368965          | 14042         |\n| Marathi   | 9.301951       | 9.342045           | 8.381926          | 14042         |\n| Tamil     | 9.047856       | 9.042943           | 8.071144          | 14042         |\n| Malayalam | 9.134952       | 8.998148           | 7.956488          | 14042         |\n| Maithili  | 9.263282       | 9.118146           | 8.218772          | 14042         |\n\n**How to read the table:**\n\n* Ratings are on a **1–10 scale**, where higher scores reflect better translation and enhancement quality.\n* Most languages achieved **consistently high averages (>8.9)**, showing strong quality after enhancement.\n* A few outliers suggest areas where further targeted refinement may be beneficial.\n\n---\n\n## Cosine similarity \n\n* Cosine similarity gives a quick automated signal of semantic closeness between the translated/enhanced items and their English originals.\n* Typical mean similarities range from ~0.76 to 0.85 across languages; higher means closer embeddings (better semantic preservation).\n* Use the similarity scores as a triage: low similarity → manual review; high similarity → likely semantically faithful.\n\n| Language               | Mean Similarity | Std Similarity | Min Similarity | Max Similarity |\n|-------------------------|-----------------|----------------|----------------|----------------|\n| mmlu_as_in_qwen3_embed  | 0.8045          | 0.0597         | 0.4644         | 1              |\n| mmlu_bn_in_qwen3_embed  | 0.8133          | 0.0504         | 0.5276         | 1              |\n| mmlu_gu_in_qwen3_embed  | 0.8211          | 0.0535         | 0.4016         | 1              |\n| mmlu_hi_in_qwen3_embed  | 0.8472          | 0.0489         | 0.3688         | 1              |\n| mmlu_kn_in_qwen3_embed  | 0.8106          | 0.0565         | 0.4660         | 1              |\n| mmlu_mai_in_qwen3_embed | 0.8226          | 0.0504         | 0.4936         | 1              |\n| mmlu_ml_in_qwen3_embed  | 0.8158          | 0.0531         | 0.5109         | 1              |\n| mmlu_mr_in_qwen3_embed  | 0.8129          | 0.0513         | 0.5281         | 1              |\n| mmlu_ne_in_qwen3_embed  | 0.8242          | 0.0502         | 0.4802         | 1              |\n| mmlu_or_in_qwen3_embed  | 0.8159          | 0.0555         | 0.4684         | 1              |\n| mmlu_pa_in_qwen3_embed  | 0.8246          | 0.0516         | 0.5450         | 1              |\n| mmlu_sa_in_qwen3_embed  | 0.7912          | 0.0574         | 0.4981         | 1              |\n| mmlu_sdd_in_qwen3_embed | 0.7646          | 0.0735         | 0.3633         | 0.9676         |\n| mmlu_ta_in_qwen3_embed  | 0.7964          | 0.0559         | 0.5242         | 1              |\n| mmlu_te_in_qwen3_embed  | 0.8006          | 0.0524         | 0.5379         | 1              |\n\n---\n\n## Model Evaluation Methodology\n\nWe evaluated **~24 open-source LLMs** across **16 Indic languages** (subset of the 22 total dataset languages) plus English, using the Indic MMLU benchmark.\n\nWorkflow:\n\n1. **Environment Setup:** Dockerized `vllm/vllm-openai` for reproducibility.\n2. **Framework:** `lm-eval` for standardized evaluation.\n3. **Metrics:** per-language accuracy, top-3 rankings per language, aggregate averages.\n4. **Visualization:** radar plots for each language and comparative top-3 plots.\n\n---\n\n## Important Note\n\n* **Indic MMLU Dataset = 22 Indian languages**\n* **Current Model Evaluation = 16 Indian languages + English**\n* Work is ongoing to expand the evaluation to cover all 22 languages.\n\n---\n### Indic MMLU Results \n\n| <div align=\"center\">![Assamese](assets/assamese.png)<br/>*Assamese*</div> | <div align=\"center\">![Bengali](assets/bengali.png)<br/>*Bengali*</div> |\n|---------------------------------------------------------------------------|------------------------------------------------------------------------|\n| <div align=\"center\">![English](assets/English.png)<br/>*English*</div>    | <div align=\"center\">![Gujarati](assets/Gujrati.png)<br/>*Gujarati*</div> |\n| <div align=\"center\">![Hindi](assets/Hindi.png)<br/>*Hindi*</div>          | <div align=\"center\">![Kannada](assets/kannada.png)<br/>*Kannada*</div> |\n| <div align=\"center\">![Maithili](assets/maithili.png)<br/>*Maithili*</div> | <div align=\"center\">![Malayalam](assets/Malayalam.png)<br/>*Malayalam*</div> |\n| <div align=\"center\">![Marathi](assets/marathi.png)<br/>*Marathi*</div>    | <div align=\"center\">![Nepali](assets/Nepali.png)<br/>*Nepali*</div>   |\n| <div align=\"center\">![Oriya](assets/oriya.png)<br/>*Oriya*</div>          | <div align=\"center\">![Punjabi](assets/Punjabi.png)<br/>*Punjabi*</div> |\n| <div align=\"center\">![Sanskrit](assets/sanskrit.png)<br/>*Sanskrit*</div> | <div align=\"center\">![Sindhi](assets/sindhi.png)<br/>*Sindhi*</div>   |\n| <div align=\"center\">![Tamil](assets/Tamil.png)<br/>*Tamil*</div>          | <div align=\"center\">![Telugu](assets/Telugu.png)<br/>*Telugu*</div>   |\n\n\n---\n\n## How to reproduce (quick checklist)\n\n1. Prepare the English CAIS/MMLU test set (`.jsonl` with IDs).\n2. Run `trans_mmlu.py` to produce per-language MT outputs.\n3. Run `async_infr.py` with the enhancement task to obtain human-readable improved translations.\n4. Generate embeddings for both English and enhanced translations.\n5. Run `calculate_cosine_similarity.py` to produce similarity reports.\n6. Optionally run human/linguist rating orchestration for additional quality assessment.\n7. Evaluate models with `lm-eval` and generate visualizations.\n\n---\n\n## Best practices & caveats\n\n* **Math and formatting:** preserve mathematical expressions and numerals during translation—these are critical for correctness.\n* **Choice alignment:** ensure answer choices remain in the same order and map to the same ground-truth labels.\n* **Human review:** automated metrics are helpful but not sufficient; include periodic linguist sampling.\n* **Constrained inference:** use deterministic LLM settings for enhancement (lower temperature, consistent decoding) to reduce hallucination.\n\n---\n\n## Visual assets & tables\n\nWe include per-language radar plots and a consolidated results panel in `plots/`. The README in the repo contains thumbnails; click through for full-resolution visuals. \n\n---\n"}, "effe2f6918995cc7305809063c08b52e1b78ddd0": {"t": 1785771733.8306417, "n": 17504, "h": "0d1805ed2704904c", "c": "# Indic MMLU — Benchmarking Open-Source LLMs on Indian Languages\n\n**Short description:**\nA rigorously prepared, multi-language adaptation of the CAIS/MMLU test set, translated and enhanced for **22 Indic languages**, designed to evaluate open-source LLMs’ real-world ability to understand Indian languages.\nWe additionally conducted large-scale **evaluation on 16 of these Indic languages (plus English)**, producing detailed accuracy reports and per-language visualizations.\n\n# What makes this Indic MMLU unique?\nWhile there have been previous efforts to translate MMLU into Indian languages, none have released their underlying methodology. Indic MMLU is the first benchmark to provide an **open recipe, open code, and translations across 22 Indian languages**, making it fully transparent and reproducible.\n\n---\n\n# Table of Contents\n\n* [Indic MMLU — Benchmarking Open-Source LLMs on Indian Languages](#indic-mmlu--benchmarking-open-source-llms-on-indian-languages)\n\n  * [Why this dataset exists](#why-this-dataset-exists)\n  * [Coverage](#coverage)\n  * [Overview](#overview)\n  * [Key principles applied](#key-principles-applied)\n  * [High-level pipeline (what we did)](#high-level-pipeline-what-we-did)\n  * [Flowchart Representation of the Indic MMLU Pipeline](#flowchart-representation-of-the-indic-mmlu-pipeline)\n  * [Translation Algorithm](#translation-algorithm)\n  * [Reproducible commands (examples)](#reproducible-commands-examples)\n\n    * [Translate](#translate)\n    * [Enhance translations (LLM-based)](#enhance-translations-llm-based)\n    * [Generate embeddings](#generate-embeddings)\n    * [Cosine similarity (English vs translated enhancements)](#cosine-similarity-english-vs-translated-enhancements)\n    * [Linguist / teacher rating (LLM assisted orchestration)](#linguist--teacher-rating-llm-assisted-orchestration)\n    * [Model evaluation with lm-eval](#model-evaluation-with-lm-eval)\n  * [Summary of LLM-as-Judge Ratings](#summary-of-llm-as-judge-ratings)\n  * [Cosine similarity](#cosine-similarity)\n  * [Model Evaluation Methodology](#model-evaluation-methodology)\n  * [Important Note](#important-note)\n  * [Indic MMLU Results](#indic-mmlu-results)\n  * [How to reproduce (quick checklist)](#how-to-reproduce-quick-checklist)\n  * [Best practices & caveats](#best-practices--caveats)\n  * [Visual assets & tables](#visual-assets--tables)\n\n---\n\n## Why this dataset exists\n\nLarge language models are often benchmarked only on English datasets, which leaves open the question of whether they can *actually* understand Indian languages beyond surface-level translation.\n\nThe **Indic MMLU dataset** bridges this gap by:\n\n* Translating the widely used **[CAIS/MMLU](https://huggingface.co/datasets/cais/mmlu) benchmark** into **22 Indian languages**.\n* Enhancing those translations with high-quality open-source LLM refinement.\n* Providing a reproducible evaluation setup for **measuring open-source models across 16 Indic languages (evaluated subset)**.\n\nIf a model’s score on an Indic version is close to its score on the original English MMLU, it indicates **genuine language understanding** and strong cross-lingual generalization.\n\n---\n\n## Coverage\n\n* **Dataset scope:** 22 Indic languages (full benchmark availability).\n* **Evaluation scope:** 16 Indic languages + English (practically evaluated with ~24 open-source LLMs).\n\nThis separation ensures the dataset is **broad and inclusive**, while evaluation focuses on the languages with the most immediate support and resources.\n\n---\n\n## Overview\n\n* **Source benchmark:** CAIS/MMLU (general knowledge QA benchmark).\n* **Target:** 22 Indic languages + English (translated test set).\n* **Workflow:** Machine translation → LLM-based translation enhancement → embedding comparison → Persona based LLM-as-judge Scores → Correlation between Embeddings and LLM as Judge → human/linguist rating and automated evaluation.\n* **Deliverables:** Enhanced translated `.jsonl` datasets, per-language embeddings, cosine similarity reports, teacher/linguist ratings, model evaluation scores and visualizations (radar plots).\n\n---\n\n## Key principles applied\n\n1. **Preserve semantic parity:** Questions remain semantically the same as English; only language changes.\n2. **Quality control:** Model-Based Automated checks (embeddings, math correctness) + human linguist ratings.\n3. **Reproducibility:** Dockerized inference environment, scriptable pipeline, and clear CLI commands.\n4. **Transparency:** We provide similarity metrics to demonstrate translation fidelity and teacher ratings to highlight linguistic quality.\n\n---\n\n## High-level pipeline (what we did)\n\n1. **Translate** the CAIS/MMLU test set to each Indic language using *IndicTrans2* (baseline MT).\n2. **Enhance** those translations using an open-source LLM with task-specific instructions (to correct phrasing, clarify ambiguous translations, preserve math expressions, and maintain answer choices alignment).\n3. **Embed** both the original English items and the (enhanced) translated items with the same embedding model.\n4. **Compare** embeddings using cosine similarity to measure semantic closeness.\n5. **LLM as Judge and Human review**: linguists/teachers rated samples for language quality, math correctness, and coherence.\n6. **Evaluate models**: run ~24 open-source LLMs across all languages using the lm-eval framework and visualize results (radar plots, top-3 per language).\n\n---\n\n## Flowchart Representation of the Indic MMLU Pipeline\n\nTo complement the pseudo-code, the following flowchart provides a **visual overview of the Indic MMLU pipeline**. It clearly illustrates the step-by-step workflow, starting from the English MMLU dataset and moving through translation, enhancement, evaluation, and validation:\n\n* **Translation Stage:** The English MMLU instance is translated into Indic languages using specialist MT.\n* **Enhancement Stage:** Translations are refined with large-scale LLMs for improved fluency and alignment.\n* **LLM Based Judging Phase:** Three types of evaluation are conducted — *fluency by native language experts, subject correctness by domain experts, and alignment by coherence judges*.\n* **Validation Stage:** Embedding similarity checks, n-gram metrics (BLEU, ROUGE, ChrF++), and discrepancy detection ensure quality.\n* **Final Output:** A validated Indic MMLU dataset instance ready for benchmarking.\n\n![Indic MMLU Pipeline](assets/indic-mmlu.png)\n\n# Translation Algorithm \n\n```\nInput:\n  original_mmlu\n  target_languages\n  instructions\n\nFor each language L in target_languages:\n    translated = translate(original_mmlu, tgt=L)\n    enhanced = enhance_using_llm(translated, instructions)\n\n    eng_emb = embed(original_mmlu)\n    trans_emb = embed(enhanced)\n    sim_scores = cosine_similarity(eng_emb, trans_emb)\n\n    llm_scores = LLM_as_Judge(original_mmlu, translated)\n\n    consistency = consistency_check(sim_scores, llm_scores)\n\n    If consistency is poor:\n        human_ratings = rate_sample(enhanced)\n    Else:\n        accept enhanced\n\nAggregate all scores:\n    combine sim_scores, llm_scores, human_ratings\n    compute per-language model scores\n\nProduce visualizations:\n    generate charts and plots for cross-language comparison\n\nOutput:\n    per-language evaluation scores\n    visualizations\n\n```\n\nThis visual representation makes it easier to understand how the **algorithmic steps, human evaluation, and automated checks** come together to ensure translation quality and dataset reliability.\n\n---\n\n## Reproducible commands (examples)\n\n> These are the canonical commands used in our pipeline. Replace file paths, ports, and concurrency values as appropriate for your environment.\n\n**Translate**\n\n```bash\npython3 trans_mmlu.py \\\n  --input_file_path test-00000-of-00001_with_ids.jsonl \\\n  --output_file_path mmlu_kn_in.jsonl \\\n  --tgt_lang \"kn\" \\\n  --src_lang \"en\"\n```\n\n**Enhance translations (LLM-based)**\n\n```bash\npython3 async_infr.py \\\n  --input-path mmlu_as_in.jsonl \\\n  --output-file ds_enhance_eval_hi.jsonl \\\n  --instruction-path instruction_prompts.yml \\\n  --task deepseek_enhance_instruct \\\n  --template-fields 'og_question' 'og_choices' 'question' 'answer' 'Hindi' \\\n  --max-concurrency 4096 \\\n  --backend sglang-oai-chat \\\n  --extra-request-body '{\"temperature\":0.7,\"top_p\":0.9,\"top_k\":50,\"max_tokens\":1024}' \\\n  --port 30000\n```\n\n**Generate embeddings**\n\n```bash\npython3 get_embeddings.py \\\n  --input-file test-00000-of-00001_with_ids.jsonl \\\n  --output-file mmlu_qwen3_embd.jsonl \\\n  --server-url http://127.0.0.1:30000 \\\n  --concurrency 4096\n```\n\n**Cosine similarity (English vs translated enhancements)**\n\n```bash\npython3 calculate_cosine_similarity.py \\\n  --input_dir deepseek_enhance \\\n  --output_dir mmmlu/similarity/cosine/enhance/deepseek_enhance \\\n  --english_file mmlu_eng.jsonl\n```\n\n**Linguist / teacher rating (LLM assisted orchestration)**\n\n```bash\npython3 async_infr.py \\\n  --input-path mmlu_as_in.jsonl \\\n  --output-file ds_enhance_eval_as_qwen3_instruct.jsonl \\\n  --instruction-path instruction_prompts.yml \\\n  --task rate_translated_text_linguist \\\n  --template-fields 'og_question' 'og_choices' 'enhanced_text' 'Assamese' \\\n  --max-concurrency 4096 \\\n  --backend sglang-oai-chat \\\n  --extra-request-body '{\"temperature\":0.7,\"top_p\":0.9,\"top_k\":50,\"max_tokens\":1024}' \\\n  --port 30000\n```\n\n**Model evaluation with `lm-eval`**\n\n```bash\n# Example wrapper which runs evaluation for a model snapshot against the Indic MMLU dataset\nbash lm-eval-llm.sh /path/to/model_snapshot /benchmark-result-path\n```\n\n---\n\n### Summary of LLM-as-Judge Ratings\n\nTo evaluate translation quality systematically, we employed **LLMs as judges**, operating under three carefully designed expert personas:\n\n* **Math Expert Persona** → Rated mathematical correctness, terminology, and conceptual fidelity.\n* **Linguistic Expert Persona** → Assessed fluency, grammar, idiomatic usage, and naturalness of the translation.\n* **Coherence Expert Persona** → Evaluated semantic alignment, logical flow, and clarity between English and the target language.\n\n**What we measured:**\n\n* Language quality, fluency, and linguistic suitability.\n* Mathematical correctness and fidelity of technical content.\n* Overall coherence and semantic alignment with the English source.\n\n| Language  | Avg Maths Rate | Avg Coherence Rate | Avg Linguist Rate | Total Records |\n|-----------|----------------|--------------------|-------------------|---------------|\n| Hindi\t    | 9.658029\t     | 9.722054\t          | 9.667891\t        | 14042         |\n| Bengali\t  | 9.182561\t     | 9.131505\t          | 8.202293\t        | 14042         |\n| Nepali    | 9.473793       | 9.331505           | 8.402293          | 14042         |\n| Telugu    | 9.220837       | 8.975929           | 7.986256          | 14042         |\n| Oriya     | 9.264421       | 8.944096           | 8.220268          | 14042         |\n| Punjabi   | 9.348027       | 9.049138           | 8.253454          | 14042         |\n| Assamese  | 9.147201       | 9.039097           | 8.133243          | 14042         |\n| Sanskrit  | 8.338485       | 8.112520           | 6.390115          | 14042         |\n| Kannada   | 9.322746       | 9.111736           | 8.131819          | 14042         |\n| Sindhi    | 2.070645       | 3.109956           | 2.478137          | 14042         |\n| Gujrati   | 9.260647       | 9.206523           | 8.368965          | 14042         |\n| Marathi   | 9.301951       | 9.342045           | 8.381926          | 14042         |\n| Tamil     | 9.047856       | 9.042943           | 8.071144          | 14042         |\n| Malayalam | 9.134952       | 8.998148           | 7.956488          | 14042         |\n| Maithili  | 9.263282       | 9.118146           | 8.218772          | 14042         |\n\n**How to read the table:**\n\n* Ratings are on a **1–10 scale**, where higher scores reflect better translation and enhancement quality.\n* Most languages achieved **consistently high averages (>8.9)**, showing strong quality after enhancement.\n* A few outliers suggest areas where further targeted refinement may be beneficial.\n\n---\n\n## Cosine similarity \n\n* Cosine similarity gives a quick automated signal of semantic closeness between the translated/enhanced items and their English originals.\n* Typical mean similarities range from ~0.76 to 0.85 across languages; higher means closer embeddings (better semantic preservation).\n* Use the similarity scores as a triage: low similarity → manual review; high similarity → likely semantically faithful.\n\n| Language               | Mean Similarity | Std Similarity | Min Similarity | Max Similarity |\n|-------------------------|-----------------|----------------|----------------|----------------|\n| mmlu_as_in_qwen3_embed  | 0.8045          | 0.0597         | 0.4644         | 1              |\n| mmlu_bn_in_qwen3_embed  | 0.8133          | 0.0504         | 0.5276         | 1              |\n| mmlu_gu_in_qwen3_embed  | 0.8211          | 0.0535         | 0.4016         | 1              |\n| mmlu_hi_in_qwen3_embed  | 0.8472          | 0.0489         | 0.3688         | 1              |\n| mmlu_kn_in_qwen3_embed  | 0.8106          | 0.0565         | 0.4660         | 1              |\n| mmlu_mai_in_qwen3_embed | 0.8226          | 0.0504         | 0.4936         | 1              |\n| mmlu_ml_in_qwen3_embed  | 0.8158          | 0.0531         | 0.5109         | 1              |\n| mmlu_mr_in_qwen3_embed  | 0.8129          | 0.0513         | 0.5281         | 1              |\n| mmlu_ne_in_qwen3_embed  | 0.8242          | 0.0502         | 0.4802         | 1              |\n| mmlu_or_in_qwen3_embed  | 0.8159          | 0.0555         | 0.4684         | 1              |\n| mmlu_pa_in_qwen3_embed  | 0.8246          | 0.0516         | 0.5450         | 1              |\n| mmlu_sa_in_qwen3_embed  | 0.7912          | 0.0574         | 0.4981         | 1              |\n| mmlu_sdd_in_qwen3_embed | 0.7646          | 0.0735         | 0.3633         | 0.9676         |\n| mmlu_ta_in_qwen3_embed  | 0.7964          | 0.0559         | 0.5242         | 1              |\n| mmlu_te_in_qwen3_embed  | 0.8006          | 0.0524         | 0.5379         | 1              |\n\n---\n\n## Model Evaluation Methodology\n\nWe evaluated **~24 open-source LLMs** across **16 Indic languages** (subset of the 22 total dataset languages) plus English, using the Indic MMLU benchmark.\n\nWorkflow:\n\n1. **Environment Setup:** Dockerized `vllm/vllm-openai` for reproducibility.\n2. **Framework:** `lm-eval` for standardized evaluation.\n3. **Metrics:** per-language accuracy, top-3 rankings per language, aggregate averages.\n4. **Visualization:** radar plots for each language and comparative top-3 plots.\n\n---\n\n## Important Note\n\n* **Indic MMLU Dataset = 22 Indian languages**\n* **Current Model Evaluation = 16 Indian languages + English**\n* Work is ongoing to expand the evaluation to cover all 22 languages.\n\n---\n### Indic MMLU Results \n\n| <div align=\"center\">![Assamese](assets/assamese.png)<br/>*Assamese*</div> | <div align=\"center\">![Bengali](assets/bengali.png)<br/>*Bengali*</div> |\n|---------------------------------------------------------------------------|------------------------------------------------------------------------|\n| <div align=\"center\">![English](assets/English.png)<br/>*English*</div>    | <div align=\"center\">![Gujarati](assets/Gujrati.png)<br/>*Gujarati*</div> |\n| <div align=\"center\">![Hindi](assets/Hindi.png)<br/>*Hindi*</div>          | <div align=\"center\">![Kannada](assets/kannada.png)<br/>*Kannada*</div> |\n| <div align=\"center\">![Maithili](assets/maithili.png)<br/>*Maithili*</div> | <div align=\"center\">![Malayalam](assets/Malayalam.png)<br/>*Malayalam*</div> |\n| <div align=\"center\">![Marathi](assets/marathi.png)<br/>*Marathi*</div>    | <div align=\"center\">![Nepali](assets/Nepali.png)<br/>*Nepali*</div>   |\n| <div align=\"center\">![Oriya](assets/oriya.png)<br/>*Oriya*</div>          | <div align=\"center\">![Punjabi](assets/Punjabi.png)<br/>*Punjabi*</div> |\n| <div align=\"center\">![Sanskrit](assets/sanskrit.png)<br/>*Sanskrit*</div> | <div align=\"center\">![Sindhi](assets/sindhi.png)<br/>*Sindhi*</div>   |\n| <div align=\"center\">![Tamil](assets/Tamil.png)<br/>*Tamil*</div>          | <div align=\"center\">![Telugu](assets/Telugu.png)<br/>*Telugu*</div>   |\n\n\n---\n\n## How to reproduce (quick checklist)\n\n1. Prepare the English CAIS/MMLU test set (`.jsonl` with IDs).\n2. Run `trans_mmlu.py` to produce per-language MT outputs.\n3. Run `async_infr.py` with the enhancement task to obtain human-readable improved translations.\n4. Generate embeddings for both English and enhanced translations.\n5. Run `calculate_cosine_similarity.py` to produce similarity reports.\n6. Optionally run human/linguist rating orchestration for additional quality assessment.\n7. Evaluate models with `lm-eval` and generate visualizations.\n\n---\n\n## Best practices & caveats\n\n* **Math and formatting:** preserve mathematical expressions and numerals during translation—these are critical for correctness.\n* **Choice alignment:** ensure answer choices remain in the same order and map to the same ground-truth labels.\n* **Human review:** automated metrics are helpful but not sufficient; include periodic linguist sampling.\n* **Constrained inference:** use deterministic LLM settings for enhancement (lower temperature, consistent decoding) to reduce hallucination.\n\n---\n\n## Visual assets & tables\n\nWe include per-language radar plots and a consolidated results panel in `plots/`. The README in the repo contains thumbnails; click through for full-resolution visuals. \n\n---\n"}, "6d99196f1549c0655c5580a0500e914897ade247": {"t": 1785771735.1294818, "n": 17500, "h": "18b2adfc5329f9c8", "c": "# Indic MMLU — Benchmarking Open-Source LLMs on Indian Languages\n\n**Short description:**\nA rigorously prepared, multi-language adaptation of the CAIS/MMLU test set, translated and enhanced for **22 Indic languages**, designed to evaluate open-source LLMs’ real-world ability to understand Indian languages.\nWe additionally conducted large-scale **evaluation on 16 of these Indic languages (plus English)**, producing detailed accuracy reports and per-language visualizations.\n\n# What makes this Indic MMLU unique?\nWhile there have been previous efforts to translate MMLU into Indian languages, none have released their underlying methodology. Indic MMLU is the first benchmark to provide an open recipe, open code, and translations across 22 Indian languages, making it fully transparent and reproducible.\n\n---\n\n# Table of Contents\n\n* [Indic MMLU — Benchmarking Open-Source LLMs on Indian Languages](#indic-mmlu--benchmarking-open-source-llms-on-indian-languages)\n\n  * [Why this dataset exists](#why-this-dataset-exists)\n  * [Coverage](#coverage)\n  * [Overview](#overview)\n  * [Key principles applied](#key-principles-applied)\n  * [High-level pipeline (what we did)](#high-level-pipeline-what-we-did)\n  * [Flowchart Representation of the Indic MMLU Pipeline](#flowchart-representation-of-the-indic-mmlu-pipeline)\n  * [Translation Algorithm](#translation-algorithm)\n  * [Reproducible commands (examples)](#reproducible-commands-examples)\n\n    * [Translate](#translate)\n    * [Enhance translations (LLM-based)](#enhance-translations-llm-based)\n    * [Generate embeddings](#generate-embeddings)\n    * [Cosine similarity (English vs translated enhancements)](#cosine-similarity-english-vs-translated-enhancements)\n    * [Linguist / teacher rating (LLM assisted orchestration)](#linguist--teacher-rating-llm-assisted-orchestration)\n    * [Model evaluation with lm-eval](#model-evaluation-with-lm-eval)\n  * [Summary of LLM-as-Judge Ratings](#summary-of-llm-as-judge-ratings)\n  * [Cosine similarity](#cosine-similarity)\n  * [Model Evaluation Methodology](#model-evaluation-methodology)\n  * [Important Note](#important-note)\n  * [Indic MMLU Results](#indic-mmlu-results)\n  * [How to reproduce (quick checklist)](#how-to-reproduce-quick-checklist)\n  * [Best practices & caveats](#best-practices--caveats)\n  * [Visual assets & tables](#visual-assets--tables)\n\n---\n\n## Why this dataset exists\n\nLarge language models are often benchmarked only on English datasets, which leaves open the question of whether they can *actually* understand Indian languages beyond surface-level translation.\n\nThe **Indic MMLU dataset** bridges this gap by:\n\n* Translating the widely used **[CAIS/MMLU](https://huggingface.co/datasets/cais/mmlu) benchmark** into **22 Indian languages**.\n* Enhancing those translations with high-quality open-source LLM refinement.\n* Providing a reproducible evaluation setup for **measuring open-source models across 16 Indic languages (evaluated subset)**.\n\nIf a model’s score on an Indic version is close to its score on the original English MMLU, it indicates **genuine language understanding** and strong cross-lingual generalization.\n\n---\n\n## Coverage\n\n* **Dataset scope:** 22 Indic languages (full benchmark availability).\n* **Evaluation scope:** 16 Indic languages + English (practically evaluated with ~24 open-source LLMs).\n\nThis separation ensures the dataset is **broad and inclusive**, while evaluation focuses on the languages with the most immediate support and resources.\n\n---\n\n## Overview\n\n* **Source benchmark:** CAIS/MMLU (general knowledge QA benchmark).\n* **Target:** 22 Indic languages + English (translated test set).\n* **Workflow:** Machine translation → LLM-based translation enhancement → embedding comparison → Persona based LLM-as-judge Scores → Correlation between Embeddings and LLM as Judge → human/linguist rating and automated evaluation.\n* **Deliverables:** Enhanced translated `.jsonl` datasets, per-language embeddings, cosine similarity reports, teacher/linguist ratings, model evaluation scores and visualizations (radar plots).\n\n---\n\n## Key principles applied\n\n1. **Preserve semantic parity:** Questions remain semantically the same as English; only language changes.\n2. **Quality control:** Model-Based Automated checks (embeddings, math correctness) + human linguist ratings.\n3. **Reproducibility:** Dockerized inference environment, scriptable pipeline, and clear CLI commands.\n4. **Transparency:** We provide similarity metrics to demonstrate translation fidelity and teacher ratings to highlight linguistic quality.\n\n---\n\n## High-level pipeline (what we did)\n\n1. **Translate** the CAIS/MMLU test set to each Indic language using *IndicTrans2* (baseline MT).\n2. **Enhance** those translations using an open-source LLM with task-specific instructions (to correct phrasing, clarify ambiguous translations, preserve math expressions, and maintain answer choices alignment).\n3. **Embed** both the original English items and the (enhanced) translated items with the same embedding model.\n4. **Compare** embeddings using cosine similarity to measure semantic closeness.\n5. **LLM as Judge and Human review**: linguists/teachers rated samples for language quality, math correctness, and coherence.\n6. **Evaluate models**: run ~24 open-source LLMs across all languages using the lm-eval framework and visualize results (radar plots, top-3 per language).\n\n---\n\n## Flowchart Representation of the Indic MMLU Pipeline\n\nTo complement the pseudo-code, the following flowchart provides a **visual overview of the Indic MMLU pipeline**. It clearly illustrates the step-by-step workflow, starting from the English MMLU dataset and moving through translation, enhancement, evaluation, and validation:\n\n* **Translation Stage:** The English MMLU instance is translated into Indic languages using specialist MT.\n* **Enhancement Stage:** Translations are refined with large-scale LLMs for improved fluency and alignment.\n* **LLM Based Judging Phase:** Three types of evaluation are conducted — *fluency by native language experts, subject correctness by domain experts, and alignment by coherence judges*.\n* **Validation Stage:** Embedding similarity checks, n-gram metrics (BLEU, ROUGE, ChrF++), and discrepancy detection ensure quality.\n* **Final Output:** A validated Indic MMLU dataset instance ready for benchmarking.\n\n![Indic MMLU Pipeline](assets/indic-mmlu.png)\n\n# Translation Algorithm \n\n```\nInput:\n  original_mmlu\n  target_languages\n  instructions\n\nFor each language L in target_languages:\n    translated = translate(original_mmlu, tgt=L)\n    enhanced = enhance_using_llm(translated, instructions)\n\n    eng_emb = embed(original_mmlu)\n    trans_emb = embed(enhanced)\n    sim_scores = cosine_similarity(eng_emb, trans_emb)\n\n    llm_scores = LLM_as_Judge(original_mmlu, translated)\n\n    consistency = consistency_check(sim_scores, llm_scores)\n\n    If consistency is poor:\n        human_ratings = rate_sample(enhanced)\n    Else:\n        accept enhanced\n\nAggregate all scores:\n    combine sim_scores, llm_scores, human_ratings\n    compute per-language model scores\n\nProduce visualizations:\n    generate charts and plots for cross-language comparison\n\nOutput:\n    per-language evaluation scores\n    visualizations\n\n```\n\nThis visual representation makes it easier to understand how the **algorithmic steps, human evaluation, and automated checks** come together to ensure translation quality and dataset reliability.\n\n---\n\n## Reproducible commands (examples)\n\n> These are the canonical commands used in our pipeline. Replace file paths, ports, and concurrency values as appropriate for your environment.\n\n**Translate**\n\n```bash\npython3 trans_mmlu.py \\\n  --input_file_path test-00000-of-00001_with_ids.jsonl \\\n  --output_file_path mmlu_kn_in.jsonl \\\n  --tgt_lang \"kn\" \\\n  --src_lang \"en\"\n```\n\n**Enhance translations (LLM-based)**\n\n```bash\npython3 async_infr.py \\\n  --input-path mmlu_as_in.jsonl \\\n  --output-file ds_enhance_eval_hi.jsonl \\\n  --instruction-path instruction_prompts.yml \\\n  --task deepseek_enhance_instruct \\\n  --template-fields 'og_question' 'og_choices' 'question' 'answer' 'Hindi' \\\n  --max-concurrency 4096 \\\n  --backend sglang-oai-chat \\\n  --extra-request-body '{\"temperature\":0.7,\"top_p\":0.9,\"top_k\":50,\"max_tokens\":1024}' \\\n  --port 30000\n```\n\n**Generate embeddings**\n\n```bash\npython3 get_embeddings.py \\\n  --input-file test-00000-of-00001_with_ids.jsonl \\\n  --output-file mmlu_qwen3_embd.jsonl \\\n  --server-url http://127.0.0.1:30000 \\\n  --concurrency 4096\n```\n\n**Cosine similarity (English vs translated enhancements)**\n\n```bash\npython3 calculate_cosine_similarity.py \\\n  --input_dir deepseek_enhance \\\n  --output_dir mmmlu/similarity/cosine/enhance/deepseek_enhance \\\n  --english_file mmlu_eng.jsonl\n```\n\n**Linguist / teacher rating (LLM assisted orchestration)**\n\n```bash\npython3 async_infr.py \\\n  --input-path mmlu_as_in.jsonl \\\n  --output-file ds_enhance_eval_as_qwen3_instruct.jsonl \\\n  --instruction-path instruction_prompts.yml \\\n  --task rate_translated_text_linguist \\\n  --template-fields 'og_question' 'og_choices' 'enhanced_text' 'Assamese' \\\n  --max-concurrency 4096 \\\n  --backend sglang-oai-chat \\\n  --extra-request-body '{\"temperature\":0.7,\"top_p\":0.9,\"top_k\":50,\"max_tokens\":1024}' \\\n  --port 30000\n```\n\n**Model evaluation with `lm-eval`**\n\n```bash\n# Example wrapper which runs evaluation for a model snapshot against the Indic MMLU dataset\nbash lm-eval-llm.sh /path/to/model_snapshot /benchmark-result-path\n```\n\n---\n\n### Summary of LLM-as-Judge Ratings\n\nTo evaluate translation quality systematically, we employed **LLMs as judges**, operating under three carefully designed expert personas:\n\n* **Math Expert Persona** → Rated mathematical correctness, terminology, and conceptual fidelity.\n* **Linguistic Expert Persona** → Assessed fluency, grammar, idiomatic usage, and naturalness of the translation.\n* **Coherence Expert Persona** → Evaluated semantic alignment, logical flow, and clarity between English and the target language.\n\n**What we measured:**\n\n* Language quality, fluency, and linguistic suitability.\n* Mathematical correctness and fidelity of technical content.\n* Overall coherence and semantic alignment with the English source.\n\n| Language  | Avg Maths Rate | Avg Coherence Rate | Avg Linguist Rate | Total Records |\n|-----------|----------------|--------------------|-------------------|---------------|\n| Hindi\t    | 9.658029\t     | 9.722054\t          | 9.667891\t        | 14042         |\n| Bengali\t  | 9.182561\t     | 9.131505\t          | 8.202293\t        | 14042         |\n| Nepali    | 9.473793       | 9.331505           | 8.402293          | 14042         |\n| Telugu    | 9.220837       | 8.975929           | 7.986256          | 14042         |\n| Oriya     | 9.264421       | 8.944096           | 8.220268          | 14042         |\n| Punjabi   | 9.348027       | 9.049138           | 8.253454          | 14042         |\n| Assamese  | 9.147201       | 9.039097           | 8.133243          | 14042         |\n| Sanskrit  | 8.338485       | 8.112520           | 6.390115          | 14042         |\n| Kannada   | 9.322746       | 9.111736           | 8.131819          | 14042         |\n| Sindhi    | 2.070645       | 3.109956           | 2.478137          | 14042         |\n| Gujrati   | 9.260647       | 9.206523           | 8.368965          | 14042         |\n| Marathi   | 9.301951       | 9.342045           | 8.381926          | 14042         |\n| Tamil     | 9.047856       | 9.042943           | 8.071144          | 14042         |\n| Malayalam | 9.134952       | 8.998148           | 7.956488          | 14042         |\n| Maithili  | 9.263282       | 9.118146           | 8.218772          | 14042         |\n\n**How to read the table:**\n\n* Ratings are on a **1–10 scale**, where higher scores reflect better translation and enhancement quality.\n* Most languages achieved **consistently high averages (>8.9)**, showing strong quality after enhancement.\n* A few outliers suggest areas where further targeted refinement may be beneficial.\n\n---\n\n## Cosine similarity \n\n* Cosine similarity gives a quick automated signal of semantic closeness between the translated/enhanced items and their English originals.\n* Typical mean similarities range from ~0.76 to 0.85 across languages; higher means closer embeddings (better semantic preservation).\n* Use the similarity scores as a triage: low similarity → manual review; high similarity → likely semantically faithful.\n\n| Language               | Mean Similarity | Std Similarity | Min Similarity | Max Similarity |\n|-------------------------|-----------------|----------------|----------------|----------------|\n| mmlu_as_in_qwen3_embed  | 0.8045          | 0.0597         | 0.4644         | 1              |\n| mmlu_bn_in_qwen3_embed  | 0.8133          | 0.0504         | 0.5276         | 1              |\n| mmlu_gu_in_qwen3_embed  | 0.8211          | 0.0535         | 0.4016         | 1              |\n| mmlu_hi_in_qwen3_embed  | 0.8472          | 0.0489         | 0.3688         | 1              |\n| mmlu_kn_in_qwen3_embed  | 0.8106          | 0.0565         | 0.4660         | 1              |\n| mmlu_mai_in_qwen3_embed | 0.8226          | 0.0504         | 0.4936         | 1              |\n| mmlu_ml_in_qwen3_embed  | 0.8158          | 0.0531         | 0.5109         | 1              |\n| mmlu_mr_in_qwen3_embed  | 0.8129          | 0.0513         | 0.5281         | 1              |\n| mmlu_ne_in_qwen3_embed  | 0.8242          | 0.0502         | 0.4802         | 1              |\n| mmlu_or_in_qwen3_embed  | 0.8159          | 0.0555         | 0.4684         | 1              |\n| mmlu_pa_in_qwen3_embed  | 0.8246          | 0.0516         | 0.5450         | 1              |\n| mmlu_sa_in_qwen3_embed  | 0.7912          | 0.0574         | 0.4981         | 1              |\n| mmlu_sdd_in_qwen3_embed | 0.7646          | 0.0735         | 0.3633         | 0.9676         |\n| mmlu_ta_in_qwen3_embed  | 0.7964          | 0.0559         | 0.5242         | 1              |\n| mmlu_te_in_qwen3_embed  | 0.8006          | 0.0524         | 0.5379         | 1              |\n\n---\n\n## Model Evaluation Methodology\n\nWe evaluated **~24 open-source LLMs** across **16 Indic languages** (subset of the 22 total dataset languages) plus English, using the Indic MMLU benchmark.\n\nWorkflow:\n\n1. **Environment Setup:** Dockerized `vllm/vllm-openai` for reproducibility.\n2. **Framework:** `lm-eval` for standardized evaluation.\n3. **Metrics:** per-language accuracy, top-3 rankings per language, aggregate averages.\n4. **Visualization:** radar plots for each language and comparative top-3 plots.\n\n---\n\n## Important Note\n\n* **Indic MMLU Dataset = 22 Indian languages**\n* **Current Model Evaluation = 16 Indian languages + English**\n* Work is ongoing to expand the evaluation to cover all 22 languages.\n\n---\n### Indic MMLU Results \n\n| <div align=\"center\">![Assamese](assets/assamese.png)<br/>*Assamese*</div> | <div align=\"center\">![Bengali](assets/bengali.png)<br/>*Bengali*</div> |\n|---------------------------------------------------------------------------|------------------------------------------------------------------------|\n| <div align=\"center\">![English](assets/English.png)<br/>*English*</div>    | <div align=\"center\">![Gujarati](assets/Gujrati.png)<br/>*Gujarati*</div> |\n| <div align=\"center\">![Hindi](assets/Hindi.png)<br/>*Hindi*</div>          | <div align=\"center\">![Kannada](assets/kannada.png)<br/>*Kannada*</div> |\n| <div align=\"center\">![Maithili](assets/maithili.png)<br/>*Maithili*</div> | <div align=\"center\">![Malayalam](assets/Malayalam.png)<br/>*Malayalam*</div> |\n| <div align=\"center\">![Marathi](assets/marathi.png)<br/>*Marathi*</div>    | <div align=\"center\">![Nepali](assets/Nepali.png)<br/>*Nepali*</div>   |\n| <div align=\"center\">![Oriya](assets/oriya.png)<br/>*Oriya*</div>          | <div align=\"center\">![Punjabi](assets/Punjabi.png)<br/>*Punjabi*</div> |\n| <div align=\"center\">![Sanskrit](assets/sanskrit.png)<br/>*Sanskrit*</div> | <div align=\"center\">![Sindhi](assets/sindhi.png)<br/>*Sindhi*</div>   |\n| <div align=\"center\">![Tamil](assets/Tamil.png)<br/>*Tamil*</div>          | <div align=\"center\">![Telugu](assets/Telugu.png)<br/>*Telugu*</div>   |\n\n\n---\n\n## How to reproduce (quick checklist)\n\n1. Prepare the English CAIS/MMLU test set (`.jsonl` with IDs).\n2. Run `trans_mmlu.py` to produce per-language MT outputs.\n3. Run `async_infr.py` with the enhancement task to obtain human-readable improved translations.\n4. Generate embeddings for both English and enhanced translations.\n5. Run `calculate_cosine_similarity.py` to produce similarity reports.\n6. Optionally run human/linguist rating orchestration for additional quality assessment.\n7. Evaluate models with `lm-eval` and generate visualizations.\n\n---\n\n## Best practices & caveats\n\n* **Math and formatting:** preserve mathematical expressions and numerals during translation—these are critical for correctness.\n* **Choice alignment:** ensure answer choices remain in the same order and map to the same ground-truth labels.\n* **Human review:** automated metrics are helpful but not sufficient; include periodic linguist sampling.\n* **Constrained inference:** use deterministic LLM settings for enhancement (lower temperature, consistent decoding) to reduce hallucination.\n\n---\n\n## Visual assets & tables\n\nWe include per-language radar plots and a consolidated results panel in `plots/`. The README in the repo contains thumbnails; click through for full-resolution visuals. \n\n---\n"}, "c07e117db833209475fa5ef3b008f09aa437afa7": {"t": 1785771736.3095193, "n": 17568, "h": "59428b9b5e70e15e", "c": "# Indic MMLU — Benchmarking Open-Source LLMs on Indian Languages\n\n**Short description:**\nA rigorously prepared, multi-language adaptation of the CAIS/MMLU test set, translated and enhanced for **22 Indic languages**, designed to evaluate open-source LLMs’ real-world ability to understand Indian languages.\nWe additionally conducted large-scale **evaluation on 16 of these Indic languages (plus English)**, producing detailed accuracy reports and per-language visualizations.\n\n# What makes this Indic MMLU unique?\nWhile there have been previous efforts to translate MMLU into Indian languages, none have released their underlying methodology. Indic MMLU is the first benchmark to provide an open recipe, open code, and translations across 22 Indian languages, making it fully transparent and reproducible.\n\n---\n\n# Table of Contents\n\n* [Indic MMLU — Benchmarking Open-Source LLMs on Indian Languages](#indic-mmlu--benchmarking-open-source-llms-on-indian-languages)\n\n  * [What makes Indic MMLU unique?](#what-makes-indic-mmlu-unique?)\n  * [Why this dataset exists](#why-this-dataset-exists)\n  * [Coverage](#coverage)\n  * [Overview](#overview)\n  * [Key principles applied](#key-principles-applied)\n  * [High-level pipeline (what we did)](#high-level-pipeline-what-we-did)\n  * [Flowchart Representation of the Indic MMLU Pipeline](#flowchart-representation-of-the-indic-mmlu-pipeline)\n  * [Translation Algorithm](#translation-algorithm)\n  * [Reproducible commands (examples)](#reproducible-commands-examples)\n\n    * [Translate](#translate)\n    * [Enhance translations (LLM-based)](#enhance-translations-llm-based)\n    * [Generate embeddings](#generate-embeddings)\n    * [Cosine similarity (English vs translated enhancements)](#cosine-similarity-english-vs-translated-enhancements)\n    * [Linguist / teacher rating (LLM assisted orchestration)](#linguist--teacher-rating-llm-assisted-orchestration)\n    * [Model evaluation with lm-eval](#model-evaluation-with-lm-eval)\n  * [Summary of LLM-as-Judge Ratings](#summary-of-llm-as-judge-ratings)\n  * [Cosine similarity](#cosine-similarity)\n  * [Model Evaluation Methodology](#model-evaluation-methodology)\n  * [Important Note](#important-note)\n  * [Indic MMLU Results](#indic-mmlu-results)\n  * [How to reproduce (quick checklist)](#how-to-reproduce-quick-checklist)\n  * [Best practices & caveats](#best-practices--caveats)\n  * [Visual assets & tables](#visual-assets--tables)\n\n---\n\n## Why this dataset exists\n\nLarge language models are often benchmarked only on English datasets, which leaves open the question of whether they can *actually* understand Indian languages beyond surface-level translation.\n\nThe **Indic MMLU dataset** bridges this gap by:\n\n* Translating the widely used **[CAIS/MMLU](https://huggingface.co/datasets/cais/mmlu) benchmark** into **22 Indian languages**.\n* Enhancing those translations with high-quality open-source LLM refinement.\n* Providing a reproducible evaluation setup for **measuring open-source models across 16 Indic languages (evaluated subset)**.\n\nIf a model’s score on an Indic version is close to its score on the original English MMLU, it indicates **genuine language understanding** and strong cross-lingual generalization.\n\n---\n\n## Coverage\n\n* **Dataset scope:** 22 Indic languages (full benchmark availability).\n* **Evaluation scope:** 16 Indic languages + English (practically evaluated with ~24 open-source LLMs).\n\nThis separation ensures the dataset is **broad and inclusive**, while evaluation focuses on the languages with the most immediate support and resources.\n\n---\n\n## Overview\n\n* **Source benchmark:** CAIS/MMLU (general knowledge QA benchmark).\n* **Target:** 22 Indic languages + English (translated test set).\n* **Workflow:** Machine translation → LLM-based translation enhancement → embedding comparison → Persona based LLM-as-judge Scores → Correlation between Embeddings and LLM as Judge → human/linguist rating and automated evaluation.\n* **Deliverables:** Enhanced translated `.jsonl` datasets, per-language embeddings, cosine similarity reports, teacher/linguist ratings, model evaluation scores and visualizations (radar plots).\n\n---\n\n## Key principles applied\n\n1. **Preserve semantic parity:** Questions remain semantically the same as English; only language changes.\n2. **Quality control:** Model-Based Automated checks (embeddings, math correctness) + human linguist ratings.\n3. **Reproducibility:** Dockerized inference environment, scriptable pipeline, and clear CLI commands.\n4. **Transparency:** We provide similarity metrics to demonstrate translation fidelity and teacher ratings to highlight linguistic quality.\n\n---\n\n## High-level pipeline (what we did)\n\n1. **Translate** the CAIS/MMLU test set to each Indic language using *IndicTrans2* (baseline MT).\n2. **Enhance** those translations using an open-source LLM with task-specific instructions (to correct phrasing, clarify ambiguous translations, preserve math expressions, and maintain answer choices alignment).\n3. **Embed** both the original English items and the (enhanced) translated items with the same embedding model.\n4. **Compare** embeddings using cosine similarity to measure semantic closeness.\n5. **LLM as Judge and Human review**: linguists/teachers rated samples for language quality, math correctness, and coherence.\n6. **Evaluate models**: run ~24 open-source LLMs across all languages using the lm-eval framework and visualize results (radar plots, top-3 per language).\n\n---\n\n## Flowchart Representation of the Indic MMLU Pipeline\n\nTo complement the pseudo-code, the following flowchart provides a **visual overview of the Indic MMLU pipeline**. It clearly illustrates the step-by-step workflow, starting from the English MMLU dataset and moving through translation, enhancement, evaluation, and validation:\n\n* **Translation Stage:** The English MMLU instance is translated into Indic languages using specialist MT.\n* **Enhancement Stage:** Translations are refined with large-scale LLMs for improved fluency and alignment.\n* **LLM Based Judging Phase:** Three types of evaluation are conducted — *fluency by native language experts, subject correctness by domain experts, and alignment by coherence judges*.\n* **Validation Stage:** Embedding similarity checks, n-gram metrics (BLEU, ROUGE, ChrF++), and discrepancy detection ensure quality.\n* **Final Output:** A validated Indic MMLU dataset instance ready for benchmarking.\n\n![Indic MMLU Pipeline](assets/indic-mmlu.png)\n\n# Translation Algorithm \n\n```\nInput:\n  original_mmlu\n  target_languages\n  instructions\n\nFor each language L in target_languages:\n    translated = translate(original_mmlu, tgt=L)\n    enhanced = enhance_using_llm(translated, instructions)\n\n    eng_emb = embed(original_mmlu)\n    trans_emb = embed(enhanced)\n    sim_scores = cosine_similarity(eng_emb, trans_emb)\n\n    llm_scores = LLM_as_Judge(original_mmlu, translated)\n\n    consistency = consistency_check(sim_scores, llm_scores)\n\n    If consistency is poor:\n        human_ratings = rate_sample(enhanced)\n    Else:\n        accept enhanced\n\nAggregate all scores:\n    combine sim_scores, llm_scores, human_ratings\n    compute per-language model scores\n\nProduce visualizations:\n    generate charts and plots for cross-language comparison\n\nOutput:\n    per-language evaluation scores\n    visualizations\n\n```\n\nThis visual representation makes it easier to understand how the **algorithmic steps, human evaluation, and automated checks** come together to ensure translation quality and dataset reliability.\n\n---\n\n## Reproducible commands (examples)\n\n> These are the canonical commands used in our pipeline. Replace file paths, ports, and concurrency values as appropriate for your environment.\n\n**Translate**\n\n```bash\npython3 trans_mmlu.py \\\n  --input_file_path test-00000-of-00001_with_ids.jsonl \\\n  --output_file_path mmlu_kn_in.jsonl \\\n  --tgt_lang \"kn\" \\\n  --src_lang \"en\"\n```\n\n**Enhance translations (LLM-based)**\n\n```bash\npython3 async_infr.py \\\n  --input-path mmlu_as_in.jsonl \\\n  --output-file ds_enhance_eval_hi.jsonl \\\n  --instruction-path instruction_prompts.yml \\\n  --task deepseek_enhance_instruct \\\n  --template-fields 'og_question' 'og_choices' 'question' 'answer' 'Hindi' \\\n  --max-concurrency 4096 \\\n  --backend sglang-oai-chat \\\n  --extra-request-body '{\"temperature\":0.7,\"top_p\":0.9,\"top_k\":50,\"max_tokens\":1024}' \\\n  --port 30000\n```\n\n**Generate embeddings**\n\n```bash\npython3 get_embeddings.py \\\n  --input-file test-00000-of-00001_with_ids.jsonl \\\n  --output-file mmlu_qwen3_embd.jsonl \\\n  --server-url http://127.0.0.1:30000 \\\n  --concurrency 4096\n```\n\n**Cosine similarity (English vs translated enhancements)**\n\n```bash\npython3 calculate_cosine_similarity.py \\\n  --input_dir deepseek_enhance \\\n  --output_dir mmmlu/similarity/cosine/enhance/deepseek_enhance \\\n  --english_file mmlu_eng.jsonl\n```\n\n**Linguist / teacher rating (LLM assisted orchestration)**\n\n```bash\npython3 async_infr.py \\\n  --input-path mmlu_as_in.jsonl \\\n  --output-file ds_enhance_eval_as_qwen3_instruct.jsonl \\\n  --instruction-path instruction_prompts.yml \\\n  --task rate_translated_text_linguist \\\n  --template-fields 'og_question' 'og_choices' 'enhanced_text' 'Assamese' \\\n  --max-concurrency 4096 \\\n  --backend sglang-oai-chat \\\n  --extra-request-body '{\"temperature\":0.7,\"top_p\":0.9,\"top_k\":50,\"max_tokens\":1024}' \\\n  --port 30000\n```\n\n**Model evaluation with `lm-eval`**\n\n```bash\n# Example wrapper which runs evaluation for a model snapshot against the Indic MMLU dataset\nbash lm-eval-llm.sh /path/to/model_snapshot /benchmark-result-path\n```\n\n---\n\n### Summary of LLM-as-Judge Ratings\n\nTo evaluate translation quality systematically, we employed **LLMs as judges**, operating under three carefully designed expert personas:\n\n* **Math Expert Persona** → Rated mathematical correctness, terminology, and conceptual fidelity.\n* **Linguistic Expert Persona** → Assessed fluency, grammar, idiomatic usage, and naturalness of the translation.\n* **Coherence Expert Persona** → Evaluated semantic alignment, logical flow, and clarity between English and the target language.\n\n**What we measured:**\n\n* Language quality, fluency, and linguistic suitability.\n* Mathematical correctness and fidelity of technical content.\n* Overall coherence and semantic alignment with the English source.\n\n| Language  | Avg Maths Rate | Avg Coherence Rate | Avg Linguist Rate | Total Records |\n|-----------|----------------|--------------------|-------------------|---------------|\n| Hindi\t    | 9.658029\t     | 9.722054\t          | 9.667891\t        | 14042         |\n| Bengali\t  | 9.182561\t     | 9.131505\t          | 8.202293\t        | 14042         |\n| Nepali    | 9.473793       | 9.331505           | 8.402293          | 14042         |\n| Telugu    | 9.220837       | 8.975929           | 7.986256          | 14042         |\n| Oriya     | 9.264421       | 8.944096           | 8.220268          | 14042         |\n| Punjabi   | 9.348027       | 9.049138           | 8.253454          | 14042         |\n| Assamese  | 9.147201       | 9.039097           | 8.133243          | 14042         |\n| Sanskrit  | 8.338485       | 8.112520           | 6.390115          | 14042         |\n| Kannada   | 9.322746       | 9.111736           | 8.131819          | 14042         |\n| Sindhi    | 2.070645       | 3.109956           | 2.478137          | 14042         |\n| Gujrati   | 9.260647       | 9.206523           | 8.368965          | 14042         |\n| Marathi   | 9.301951       | 9.342045           | 8.381926          | 14042         |\n| Tamil     | 9.047856       | 9.042943           | 8.071144          | 14042         |\n| Malayalam | 9.134952       | 8.998148           | 7.956488          | 14042         |\n| Maithili  | 9.263282       | 9.118146           | 8.218772          | 14042         |\n\n**How to read the table:**\n\n* Ratings are on a **1–10 scale**, where higher scores reflect better translation and enhancement quality.\n* Most languages achieved **consistently high averages (>8.9)**, showing strong quality after enhancement.\n* A few outliers suggest areas where further targeted refinement may be beneficial.\n\n---\n\n## Cosine similarity \n\n* Cosine similarity gives a quick automated signal of semantic closeness between the translated/enhanced items and their English originals.\n* Typical mean similarities range from ~0.76 to 0.85 across languages; higher means closer embeddings (better semantic preservation).\n* Use the similarity scores as a triage: low similarity → manual review; high similarity → likely semantically faithful.\n\n| Language               | Mean Similarity | Std Similarity | Min Similarity | Max Similarity |\n|-------------------------|-----------------|----------------|----------------|----------------|\n| mmlu_as_in_qwen3_embed  | 0.8045          | 0.0597         | 0.4644         | 1              |\n| mmlu_bn_in_qwen3_embed  | 0.8133          | 0.0504         | 0.5276         | 1              |\n| mmlu_gu_in_qwen3_embed  | 0.8211          | 0.0535         | 0.4016         | 1              |\n| mmlu_hi_in_qwen3_embed  | 0.8472          | 0.0489         | 0.3688         | 1              |\n| mmlu_kn_in_qwen3_embed  | 0.8106          | 0.0565         | 0.4660         | 1              |\n| mmlu_mai_in_qwen3_embed | 0.8226          | 0.0504         | 0.4936         | 1              |\n| mmlu_ml_in_qwen3_embed  | 0.8158          | 0.0531         | 0.5109         | 1              |\n| mmlu_mr_in_qwen3_embed  | 0.8129          | 0.0513         | 0.5281         | 1              |\n| mmlu_ne_in_qwen3_embed  | 0.8242          | 0.0502         | 0.4802         | 1              |\n| mmlu_or_in_qwen3_embed  | 0.8159          | 0.0555         | 0.4684         | 1              |\n| mmlu_pa_in_qwen3_embed  | 0.8246          | 0.0516         | 0.5450         | 1              |\n| mmlu_sa_in_qwen3_embed  | 0.7912          | 0.0574         | 0.4981         | 1              |\n| mmlu_sdd_in_qwen3_embed | 0.7646          | 0.0735         | 0.3633         | 0.9676         |\n| mmlu_ta_in_qwen3_embed  | 0.7964          | 0.0559         | 0.5242         | 1              |\n| mmlu_te_in_qwen3_embed  | 0.8006          | 0.0524         | 0.5379         | 1              |\n\n---\n\n## Model Evaluation Methodology\n\nWe evaluated **~24 open-source LLMs** across **16 Indic languages** (subset of the 22 total dataset languages) plus English, using the Indic MMLU benchmark.\n\nWorkflow:\n\n1. **Environment Setup:** Dockerized `vllm/vllm-openai` for reproducibility.\n2. **Framework:** `lm-eval` for standardized evaluation.\n3. **Metrics:** per-language accuracy, top-3 rankings per language, aggregate averages.\n4. **Visualization:** radar plots for each language and comparative top-3 plots.\n\n---\n\n## Important Note\n\n* **Indic MMLU Dataset = 22 Indian languages**\n* **Current Model Evaluation = 16 Indian languages + English**\n* Work is ongoing to expand the evaluation to cover all 22 languages.\n\n---\n### Indic MMLU Results \n\n| <div align=\"center\">![Assamese](assets/assamese.png)<br/>*Assamese*</div> | <div align=\"center\">![Bengali](assets/bengali.png)<br/>*Bengali*</div> |\n|---------------------------------------------------------------------------|------------------------------------------------------------------------|\n| <div align=\"center\">![English](assets/English.png)<br/>*English*</div>    | <div align=\"center\">![Gujarati](assets/Gujrati.png)<br/>*Gujarati*</div> |\n| <div align=\"center\">![Hindi](assets/Hindi.png)<br/>*Hindi*</div>          | <div align=\"center\">![Kannada](assets/kannada.png)<br/>*Kannada*</div> |\n| <div align=\"center\">![Maithili](assets/maithili.png)<br/>*Maithili*</div> | <div align=\"center\">![Malayalam](assets/Malayalam.png)<br/>*Malayalam*</div> |\n| <div align=\"center\">![Marathi](assets/marathi.png)<br/>*Marathi*</div>    | <div align=\"center\">![Nepali](assets/Nepali.png)<br/>*Nepali*</div>   |\n| <div align=\"center\">![Oriya](assets/oriya.png)<br/>*Oriya*</div>          | <div align=\"center\">![Punjabi](assets/Punjabi.png)<br/>*Punjabi*</div> |\n| <div align=\"center\">![Sanskrit](assets/sanskrit.png)<br/>*Sanskrit*</div> | <div align=\"center\">![Sindhi](assets/sindhi.png)<br/>*Sindhi*</div>   |\n| <div align=\"center\">![Tamil](assets/Tamil.png)<br/>*Tamil*</div>          | <div align=\"center\">![Telugu](assets/Telugu.png)<br/>*Telugu*</div>   |\n\n\n---\n\n## How to reproduce (quick checklist)\n\n1. Prepare the English CAIS/MMLU test set (`.jsonl` with IDs).\n2. Run `trans_mmlu.py` to produce per-language MT outputs.\n3. Run `async_infr.py` with the enhancement task to obtain human-readable improved translations.\n4. Generate embeddings for both English and enhanced translations.\n5. Run `calculate_cosine_similarity.py` to produce similarity reports.\n6. Optionally run human/linguist rating orchestration for additional quality assessment.\n7. Evaluate models with `lm-eval` and generate visualizations.\n\n---\n\n## Best practices & caveats\n\n* **Math and formatting:** preserve mathematical expressions and numerals during translation—these are critical for correctness.\n* **Choice alignment:** ensure answer choices remain in the same order and map to the same ground-truth labels.\n* **Human review:** automated metrics are helpful but not sufficient; include periodic linguist sampling.\n* **Constrained inference:** use deterministic LLM settings for enhancement (lower temperature, consistent decoding) to reduce hallucination.\n\n---\n\n## Visual assets & tables\n\nWe include per-language radar plots and a consolidated results panel in `plots/`. The README in the repo contains thumbnails; click through for full-resolution visuals. \n\n---\n"}, "9507a5406ce3f8df260db6798bb92fba46d500af": {"t": 1785771737.8523533, "n": 17563, "h": "5cd10c2e7ef83ea5", "c": "# Indic MMLU — Benchmarking Open-Source LLMs on Indian Languages\n\n**Short description:**\nA rigorously prepared, multi-language adaptation of the CAIS/MMLU test set, translated and enhanced for **22 Indic languages**, designed to evaluate open-source LLMs’ real-world ability to understand Indian languages.\nWe additionally conducted large-scale **evaluation on 16 of these Indic languages (plus English)**, producing detailed accuracy reports and per-language visualizations.\n\n# What makes Indic MMLU unique?\nWhile there have been previous efforts to translate MMLU into Indian languages, none have released their underlying methodology. Indic MMLU is the first benchmark to provide an open recipe, open code, and translations across 22 Indian languages, making it fully transparent and reproducible.\n\n---\n\n# Table of Contents\n\n* [Indic MMLU — Benchmarking Open-Source LLMs on Indian Languages](#indic-mmlu--benchmarking-open-source-llms-on-indian-languages)\n\n  * [What makes Indic MMLU unique?](#what-makes-Indic-MMLU-unique?)\n  * [Why this dataset exists](#why-this-dataset-exists)\n  * [Coverage](#coverage)\n  * [Overview](#overview)\n  * [Key principles applied](#key-principles-applied)\n  * [High-level pipeline (what we did)](#high-level-pipeline-what-we-did)\n  * [Flowchart Representation of the Indic MMLU Pipeline](#flowchart-representation-of-the-indic-mmlu-pipeline)\n  * [Translation Algorithm](#translation-algorithm)\n  * [Reproducible commands (examples)](#reproducible-commands-examples)\n\n    * [Translate](#translate)\n    * [Enhance translations (LLM-based)](#enhance-translations-llm-based)\n    * [Generate embeddings](#generate-embeddings)\n    * [Cosine similarity (English vs translated enhancements)](#cosine-similarity-english-vs-translated-enhancements)\n    * [Linguist / teacher rating (LLM assisted orchestration)](#linguist--teacher-rating-llm-assisted-orchestration)\n    * [Model evaluation with lm-eval](#model-evaluation-with-lm-eval)\n  * [Summary of LLM-as-Judge Ratings](#summary-of-llm-as-judge-ratings)\n  * [Cosine similarity](#cosine-similarity)\n  * [Model Evaluation Methodology](#model-evaluation-methodology)\n  * [Important Note](#important-note)\n  * [Indic MMLU Results](#indic-mmlu-results)\n  * [How to reproduce (quick checklist)](#how-to-reproduce-quick-checklist)\n  * [Best practices & caveats](#best-practices--caveats)\n  * [Visual assets & tables](#visual-assets--tables)\n\n---\n\n## Why this dataset exists\n\nLarge language models are often benchmarked only on English datasets, which leaves open the question of whether they can *actually* understand Indian languages beyond surface-level translation.\n\nThe **Indic MMLU dataset** bridges this gap by:\n\n* Translating the widely used **[CAIS/MMLU](https://huggingface.co/datasets/cais/mmlu) benchmark** into **22 Indian languages**.\n* Enhancing those translations with high-quality open-source LLM refinement.\n* Providing a reproducible evaluation setup for **measuring open-source models across 16 Indic languages (evaluated subset)**.\n\nIf a model’s score on an Indic version is close to its score on the original English MMLU, it indicates **genuine language understanding** and strong cross-lingual generalization.\n\n---\n\n## Coverage\n\n* **Dataset scope:** 22 Indic languages (full benchmark availability).\n* **Evaluation scope:** 16 Indic languages + English (practically evaluated with ~24 open-source LLMs).\n\nThis separation ensures the dataset is **broad and inclusive**, while evaluation focuses on the languages with the most immediate support and resources.\n\n---\n\n## Overview\n\n* **Source benchmark:** CAIS/MMLU (general knowledge QA benchmark).\n* **Target:** 22 Indic languages + English (translated test set).\n* **Workflow:** Machine translation → LLM-based translation enhancement → embedding comparison → Persona based LLM-as-judge Scores → Correlation between Embeddings and LLM as Judge → human/linguist rating and automated evaluation.\n* **Deliverables:** Enhanced translated `.jsonl` datasets, per-language embeddings, cosine similarity reports, teacher/linguist ratings, model evaluation scores and visualizations (radar plots).\n\n---\n\n## Key principles applied\n\n1. **Preserve semantic parity:** Questions remain semantically the same as English; only language changes.\n2. **Quality control:** Model-Based Automated checks (embeddings, math correctness) + human linguist ratings.\n3. **Reproducibility:** Dockerized inference environment, scriptable pipeline, and clear CLI commands.\n4. **Transparency:** We provide similarity metrics to demonstrate translation fidelity and teacher ratings to highlight linguistic quality.\n\n---\n\n## High-level pipeline (what we did)\n\n1. **Translate** the CAIS/MMLU test set to each Indic language using *IndicTrans2* (baseline MT).\n2. **Enhance** those translations using an open-source LLM with task-specific instructions (to correct phrasing, clarify ambiguous translations, preserve math expressions, and maintain answer choices alignment).\n3. **Embed** both the original English items and the (enhanced) translated items with the same embedding model.\n4. **Compare** embeddings using cosine similarity to measure semantic closeness.\n5. **LLM as Judge and Human review**: linguists/teachers rated samples for language quality, math correctness, and coherence.\n6. **Evaluate models**: run ~24 open-source LLMs across all languages using the lm-eval framework and visualize results (radar plots, top-3 per language).\n\n---\n\n## Flowchart Representation of the Indic MMLU Pipeline\n\nTo complement the pseudo-code, the following flowchart provides a **visual overview of the Indic MMLU pipeline**. It clearly illustrates the step-by-step workflow, starting from the English MMLU dataset and moving through translation, enhancement, evaluation, and validation:\n\n* **Translation Stage:** The English MMLU instance is translated into Indic languages using specialist MT.\n* **Enhancement Stage:** Translations are refined with large-scale LLMs for improved fluency and alignment.\n* **LLM Based Judging Phase:** Three types of evaluation are conducted — *fluency by native language experts, subject correctness by domain experts, and alignment by coherence judges*.\n* **Validation Stage:** Embedding similarity checks, n-gram metrics (BLEU, ROUGE, ChrF++), and discrepancy detection ensure quality.\n* **Final Output:** A validated Indic MMLU dataset instance ready for benchmarking.\n\n![Indic MMLU Pipeline](assets/indic-mmlu.png)\n\n# Translation Algorithm \n\n```\nInput:\n  original_mmlu\n  target_languages\n  instructions\n\nFor each language L in target_languages:\n    translated = translate(original_mmlu, tgt=L)\n    enhanced = enhance_using_llm(translated, instructions)\n\n    eng_emb = embed(original_mmlu)\n    trans_emb = embed(enhanced)\n    sim_scores = cosine_similarity(eng_emb, trans_emb)\n\n    llm_scores = LLM_as_Judge(original_mmlu, translated)\n\n    consistency = consistency_check(sim_scores, llm_scores)\n\n    If consistency is poor:\n        human_ratings = rate_sample(enhanced)\n    Else:\n        accept enhanced\n\nAggregate all scores:\n    combine sim_scores, llm_scores, human_ratings\n    compute per-language model scores\n\nProduce visualizations:\n    generate charts and plots for cross-language comparison\n\nOutput:\n    per-language evaluation scores\n    visualizations\n\n```\n\nThis visual representation makes it easier to understand how the **algorithmic steps, human evaluation, and automated checks** come together to ensure translation quality and dataset reliability.\n\n---\n\n## Reproducible commands (examples)\n\n> These are the canonical commands used in our pipeline. Replace file paths, ports, and concurrency values as appropriate for your environment.\n\n**Translate**\n\n```bash\npython3 trans_mmlu.py \\\n  --input_file_path test-00000-of-00001_with_ids.jsonl \\\n  --output_file_path mmlu_kn_in.jsonl \\\n  --tgt_lang \"kn\" \\\n  --src_lang \"en\"\n```\n\n**Enhance translations (LLM-based)**\n\n```bash\npython3 async_infr.py \\\n  --input-path mmlu_as_in.jsonl \\\n  --output-file ds_enhance_eval_hi.jsonl \\\n  --instruction-path instruction_prompts.yml \\\n  --task deepseek_enhance_instruct \\\n  --template-fields 'og_question' 'og_choices' 'question' 'answer' 'Hindi' \\\n  --max-concurrency 4096 \\\n  --backend sglang-oai-chat \\\n  --extra-request-body '{\"temperature\":0.7,\"top_p\":0.9,\"top_k\":50,\"max_tokens\":1024}' \\\n  --port 30000\n```\n\n**Generate embeddings**\n\n```bash\npython3 get_embeddings.py \\\n  --input-file test-00000-of-00001_with_ids.jsonl \\\n  --output-file mmlu_qwen3_embd.jsonl \\\n  --server-url http://127.0.0.1:30000 \\\n  --concurrency 4096\n```\n\n**Cosine similarity (English vs translated enhancements)**\n\n```bash\npython3 calculate_cosine_similarity.py \\\n  --input_dir deepseek_enhance \\\n  --output_dir mmmlu/similarity/cosine/enhance/deepseek_enhance \\\n  --english_file mmlu_eng.jsonl\n```\n\n**Linguist / teacher rating (LLM assisted orchestration)**\n\n```bash\npython3 async_infr.py \\\n  --input-path mmlu_as_in.jsonl \\\n  --output-file ds_enhance_eval_as_qwen3_instruct.jsonl \\\n  --instruction-path instruction_prompts.yml \\\n  --task rate_translated_text_linguist \\\n  --template-fields 'og_question' 'og_choices' 'enhanced_text' 'Assamese' \\\n  --max-concurrency 4096 \\\n  --backend sglang-oai-chat \\\n  --extra-request-body '{\"temperature\":0.7,\"top_p\":0.9,\"top_k\":50,\"max_tokens\":1024}' \\\n  --port 30000\n```\n\n**Model evaluation with `lm-eval`**\n\n```bash\n# Example wrapper which runs evaluation for a model snapshot against the Indic MMLU dataset\nbash lm-eval-llm.sh /path/to/model_snapshot /benchmark-result-path\n```\n\n---\n\n### Summary of LLM-as-Judge Ratings\n\nTo evaluate translation quality systematically, we employed **LLMs as judges**, operating under three carefully designed expert personas:\n\n* **Math Expert Persona** → Rated mathematical correctness, terminology, and conceptual fidelity.\n* **Linguistic Expert Persona** → Assessed fluency, grammar, idiomatic usage, and naturalness of the translation.\n* **Coherence Expert Persona** → Evaluated semantic alignment, logical flow, and clarity between English and the target language.\n\n**What we measured:**\n\n* Language quality, fluency, and linguistic suitability.\n* Mathematical correctness and fidelity of technical content.\n* Overall coherence and semantic alignment with the English source.\n\n| Language  | Avg Maths Rate | Avg Coherence Rate | Avg Linguist Rate | Total Records |\n|-----------|----------------|--------------------|-------------------|---------------|\n| Hindi\t    | 9.658029\t     | 9.722054\t          | 9.667891\t        | 14042         |\n| Bengali\t  | 9.182561\t     | 9.131505\t          | 8.202293\t        | 14042         |\n| Nepali    | 9.473793       | 9.331505           | 8.402293          | 14042         |\n| Telugu    | 9.220837       | 8.975929           | 7.986256          | 14042         |\n| Oriya     | 9.264421       | 8.944096           | 8.220268          | 14042         |\n| Punjabi   | 9.348027       | 9.049138           | 8.253454          | 14042         |\n| Assamese  | 9.147201       | 9.039097           | 8.133243          | 14042         |\n| Sanskrit  | 8.338485       | 8.112520           | 6.390115          | 14042         |\n| Kannada   | 9.322746       | 9.111736           | 8.131819          | 14042         |\n| Sindhi    | 2.070645       | 3.109956           | 2.478137          | 14042         |\n| Gujrati   | 9.260647       | 9.206523           | 8.368965          | 14042         |\n| Marathi   | 9.301951       | 9.342045           | 8.381926          | 14042         |\n| Tamil     | 9.047856       | 9.042943           | 8.071144          | 14042         |\n| Malayalam | 9.134952       | 8.998148           | 7.956488          | 14042         |\n| Maithili  | 9.263282       | 9.118146           | 8.218772          | 14042         |\n\n**How to read the table:**\n\n* Ratings are on a **1–10 scale**, where higher scores reflect better translation and enhancement quality.\n* Most languages achieved **consistently high averages (>8.9)**, showing strong quality after enhancement.\n* A few outliers suggest areas where further targeted refinement may be beneficial.\n\n---\n\n## Cosine similarity \n\n* Cosine similarity gives a quick automated signal of semantic closeness between the translated/enhanced items and their English originals.\n* Typical mean similarities range from ~0.76 to 0.85 across languages; higher means closer embeddings (better semantic preservation).\n* Use the similarity scores as a triage: low similarity → manual review; high similarity → likely semantically faithful.\n\n| Language               | Mean Similarity | Std Similarity | Min Similarity | Max Similarity |\n|-------------------------|-----------------|----------------|----------------|----------------|\n| mmlu_as_in_qwen3_embed  | 0.8045          | 0.0597         | 0.4644         | 1              |\n| mmlu_bn_in_qwen3_embed  | 0.8133          | 0.0504         | 0.5276         | 1              |\n| mmlu_gu_in_qwen3_embed  | 0.8211          | 0.0535         | 0.4016         | 1              |\n| mmlu_hi_in_qwen3_embed  | 0.8472          | 0.0489         | 0.3688         | 1              |\n| mmlu_kn_in_qwen3_embed  | 0.8106          | 0.0565         | 0.4660         | 1              |\n| mmlu_mai_in_qwen3_embed | 0.8226          | 0.0504         | 0.4936         | 1              |\n| mmlu_ml_in_qwen3_embed  | 0.8158          | 0.0531         | 0.5109         | 1              |\n| mmlu_mr_in_qwen3_embed  | 0.8129          | 0.0513         | 0.5281         | 1              |\n| mmlu_ne_in_qwen3_embed  | 0.8242          | 0.0502         | 0.4802         | 1              |\n| mmlu_or_in_qwen3_embed  | 0.8159          | 0.0555         | 0.4684         | 1              |\n| mmlu_pa_in_qwen3_embed  | 0.8246          | 0.0516         | 0.5450         | 1              |\n| mmlu_sa_in_qwen3_embed  | 0.7912          | 0.0574         | 0.4981         | 1              |\n| mmlu_sdd_in_qwen3_embed | 0.7646          | 0.0735         | 0.3633         | 0.9676         |\n| mmlu_ta_in_qwen3_embed  | 0.7964          | 0.0559         | 0.5242         | 1              |\n| mmlu_te_in_qwen3_embed  | 0.8006          | 0.0524         | 0.5379         | 1              |\n\n---\n\n## Model Evaluation Methodology\n\nWe evaluated **~24 open-source LLMs** across **16 Indic languages** (subset of the 22 total dataset languages) plus English, using the Indic MMLU benchmark.\n\nWorkflow:\n\n1. **Environment Setup:** Dockerized `vllm/vllm-openai` for reproducibility.\n2. **Framework:** `lm-eval` for standardized evaluation.\n3. **Metrics:** per-language accuracy, top-3 rankings per language, aggregate averages.\n4. **Visualization:** radar plots for each language and comparative top-3 plots.\n\n---\n\n## Important Note\n\n* **Indic MMLU Dataset = 22 Indian languages**\n* **Current Model Evaluation = 16 Indian languages + English**\n* Work is ongoing to expand the evaluation to cover all 22 languages.\n\n---\n### Indic MMLU Results \n\n| <div align=\"center\">![Assamese](assets/assamese.png)<br/>*Assamese*</div> | <div align=\"center\">![Bengali](assets/bengali.png)<br/>*Bengali*</div> |\n|---------------------------------------------------------------------------|------------------------------------------------------------------------|\n| <div align=\"center\">![English](assets/English.png)<br/>*English*</div>    | <div align=\"center\">![Gujarati](assets/Gujrati.png)<br/>*Gujarati*</div> |\n| <div align=\"center\">![Hindi](assets/Hindi.png)<br/>*Hindi*</div>          | <div align=\"center\">![Kannada](assets/kannada.png)<br/>*Kannada*</div> |\n| <div align=\"center\">![Maithili](assets/maithili.png)<br/>*Maithili*</div> | <div align=\"center\">![Malayalam](assets/Malayalam.png)<br/>*Malayalam*</div> |\n| <div align=\"center\">![Marathi](assets/marathi.png)<br/>*Marathi*</div>    | <div align=\"center\">![Nepali](assets/Nepali.png)<br/>*Nepali*</div>   |\n| <div align=\"center\">![Oriya](assets/oriya.png)<br/>*Oriya*</div>          | <div align=\"center\">![Punjabi](assets/Punjabi.png)<br/>*Punjabi*</div> |\n| <div align=\"center\">![Sanskrit](assets/sanskrit.png)<br/>*Sanskrit*</div> | <div align=\"center\">![Sindhi](assets/sindhi.png)<br/>*Sindhi*</div>   |\n| <div align=\"center\">![Tamil](assets/Tamil.png)<br/>*Tamil*</div>          | <div align=\"center\">![Telugu](assets/Telugu.png)<br/>*Telugu*</div>   |\n\n\n---\n\n## How to reproduce (quick checklist)\n\n1. Prepare the English CAIS/MMLU test set (`.jsonl` with IDs).\n2. Run `trans_mmlu.py` to produce per-language MT outputs.\n3. Run `async_infr.py` with the enhancement task to obtain human-readable improved translations.\n4. Generate embeddings for both English and enhanced translations.\n5. Run `calculate_cosine_similarity.py` to produce similarity reports.\n6. Optionally run human/linguist rating orchestration for additional quality assessment.\n7. Evaluate models with `lm-eval` and generate visualizations.\n\n---\n\n## Best practices & caveats\n\n* **Math and formatting:** preserve mathematical expressions and numerals during translation—these are critical for correctness.\n* **Choice alignment:** ensure answer choices remain in the same order and map to the same ground-truth labels.\n* **Human review:** automated metrics are helpful but not sufficient; include periodic linguist sampling.\n* **Constrained inference:** use deterministic LLM settings for enhancement (lower temperature, consistent decoding) to reduce hallucination.\n\n---\n\n## Visual assets & tables\n\nWe include per-language radar plots and a consolidated results panel in `plots/`. The README in the repo contains thumbnails; click through for full-resolution visuals. \n\n---\n"}, "56dab7dc93d589451abb75a4e38c4bf7e2226a05": {"t": 1785771739.1937964, "n": 17170, "h": "bc793d61f146802a", "c": "# Indic MMLU — Benchmarking Open-Source LLMs on Indian Languages\n\n**Short description:**\nA rigorously prepared, multi-language adaptation of the CAIS/MMLU test set, translated and enhanced for **22 Indic languages**, designed to evaluate open-source LLMs’ real-world ability to understand Indian languages.\nWe additionally conducted large-scale **evaluation on 16 of these Indic languages (plus English)**, producing detailed accuracy reports and per-language visualizations.\n\n---\n\n# Table of Contents\n\n* [Indic MMLU — Benchmarking Open-Source LLMs on Indian Languages](#indic-mmlu--benchmarking-open-source-llms-on-indian-languages)\n\n  * [Why this dataset exists](#why-this-dataset-exists)\n  * [Coverage](#coverage)\n  * [Overview](#overview)\n  * [Key principles applied](#key-principles-applied)\n  * [High-level pipeline (what we did)](#high-level-pipeline-what-we-did)\n  * [Flowchart Representation of the Indic MMLU Pipeline](#flowchart-representation-of-the-indic-mmlu-pipeline)\n  * [Translation Algorithm](#translation-algorithm)\n  * [Reproducible commands (examples)](#reproducible-commands-examples)\n\n    * [Translate](#translate)\n    * [Enhance translations (LLM-based)](#enhance-translations-llm-based)\n    * [Generate embeddings](#generate-embeddings)\n    * [Cosine similarity (English vs translated enhancements)](#cosine-similarity-english-vs-translated-enhancements)\n    * [Linguist / teacher rating (LLM assisted orchestration)](#linguist--teacher-rating-llm-assisted-orchestration)\n    * [Model evaluation with lm-eval](#model-evaluation-with-lm-eval)\n  * [Summary of LLM-as-Judge Ratings](#summary-of-llm-as-judge-ratings)\n  * [Cosine similarity](#cosine-similarity)\n  * [Model Evaluation Methodology](#model-evaluation-methodology)\n  * [Important Note](#important-note)\n  * [Indic MMLU Results](#indic-mmlu-results)\n  * [How to reproduce (quick checklist)](#how-to-reproduce-quick-checklist)\n  * [Best practices & caveats](#best-practices--caveats)\n  * [Visual assets & tables](#visual-assets--tables)\n\n---\n\n## Why this dataset exists\n\nLarge language models are often benchmarked only on English datasets, which leaves open the question of whether they can *actually* understand Indian languages beyond surface-level translation.\n\nThe **Indic MMLU dataset** bridges this gap by:\n\n* Translating the widely used **[CAIS/MMLU](https://huggingface.co/datasets/cais/mmlu) benchmark** into **22 Indian languages**.\n* Enhancing those translations with high-quality open-source LLM refinement.\n* Providing a reproducible evaluation setup for **measuring open-source models across 16 Indic languages (evaluated subset)**.\n\nIf a model’s score on an Indic version is close to its score on the original English MMLU, it indicates **genuine language understanding** and strong cross-lingual generalization.\n\n---\n\n## Coverage\n\n* **Dataset scope:** 22 Indic languages (full benchmark availability).\n* **Evaluation scope:** 16 Indic languages + English (practically evaluated with ~24 open-source LLMs).\n\nThis separation ensures the dataset is **broad and inclusive**, while evaluation focuses on the languages with the most immediate support and resources.\n\n---\n\n## Overview\n\n* **Source benchmark:** CAIS/MMLU (general knowledge QA benchmark).\n* **Target:** 22 Indic languages + English (translated test set).\n* **Workflow:** Machine translation → LLM-based translation enhancement → embedding comparison → Persona based LLM-as-judge Scores → Correlation between Embeddings and LLM as Judge → human/linguist rating and automated evaluation.\n* **Deliverables:** Enhanced translated `.jsonl` datasets, per-language embeddings, cosine similarity reports, teacher/linguist ratings, model evaluation scores and visualizations (radar plots).\n\n---\n\n## Key principles applied\n\n1. **Preserve semantic parity:** Questions remain semantically the same as English; only language changes.\n2. **Quality control:** Model-Based Automated checks (embeddings, math correctness) + human linguist ratings.\n3. **Reproducibility:** Dockerized inference environment, scriptable pipeline, and clear CLI commands.\n4. **Transparency:** We provide similarity metrics to demonstrate translation fidelity and teacher ratings to highlight linguistic quality.\n\n---\n\n## High-level pipeline (what we did)\n\n1. **Translate** the CAIS/MMLU test set to each Indic language using *IndicTrans2* (baseline MT).\n2. **Enhance** those translations using an open-source LLM with task-specific instructions (to correct phrasing, clarify ambiguous translations, preserve math expressions, and maintain answer choices alignment).\n3. **Embed** both the original English items and the (enhanced) translated items with the same embedding model.\n4. **Compare** embeddings using cosine similarity to measure semantic closeness.\n5. **LLM as Judge and Human review**: linguists/teachers rated samples for language quality, math correctness, and coherence.\n6. **Evaluate models**: run ~24 open-source LLMs across all languages using the lm-eval framework and visualize results (radar plots, top-3 per language).\n\n---\n\n## Flowchart Representation of the Indic MMLU Pipeline\n\nTo complement the pseudo-code, the following flowchart provides a **visual overview of the Indic MMLU pipeline**. It clearly illustrates the step-by-step workflow, starting from the English MMLU dataset and moving through translation, enhancement, evaluation, and validation:\n\n* **Translation Stage:** The English MMLU instance is translated into Indic languages using specialist MT.\n* **Enhancement Stage:** Translations are refined with large-scale LLMs for improved fluency and alignment.\n* **LLM Based Judging Phase:** Three types of evaluation are conducted — *fluency by native language experts, subject correctness by domain experts, and alignment by coherence judges*.\n* **Validation Stage:** Embedding similarity checks, n-gram metrics (BLEU, ROUGE, ChrF++), and discrepancy detection ensure quality.\n* **Final Output:** A validated Indic MMLU dataset instance ready for benchmarking.\n\n![Indic MMLU Pipeline](assets/indic-mmlu.png)\n\n# Translation Algorithm \n\n```\nInput:\n  original_mmlu\n  target_languages\n  instructions\n\nFor each language L in target_languages:\n    translated = translate(original_mmlu, tgt=L)\n    enhanced = enhance_using_llm(translated, instructions)\n\n    eng_emb = embed(original_mmlu)\n    trans_emb = embed(enhanced)\n    sim_scores = cosine_similarity(eng_emb, trans_emb)\n\n    llm_scores = LLM_as_Judge(original_mmlu, translated)\n\n    consistency = consistency_check(sim_scores, llm_scores)\n\n    If consistency is poor:\n        human_ratings = rate_sample(enhanced)\n    Else:\n        accept enhanced\n\nAggregate all scores:\n    combine sim_scores, llm_scores, human_ratings\n    compute per-language model scores\n\nProduce visualizations:\n    generate charts and plots for cross-language comparison\n\nOutput:\n    per-language evaluation scores\n    visualizations\n\n```\n\nThis visual representation makes it easier to understand how the **algorithmic steps, human evaluation, and automated checks** come together to ensure translation quality and dataset reliability.\n\n---\n\n## Reproducible commands (examples)\n\n> These are the canonical commands used in our pipeline. Replace file paths, ports, and concurrency values as appropriate for your environment.\n\n**Translate**\n\n```bash\npython3 trans_mmlu.py \\\n  --input_file_path test-00000-of-00001_with_ids.jsonl \\\n  --output_file_path mmlu_kn_in.jsonl \\\n  --tgt_lang \"kn\" \\\n  --src_lang \"en\"\n```\n\n**Enhance translations (LLM-based)**\n\n```bash\npython3 async_infr.py \\\n  --input-path mmlu_as_in.jsonl \\\n  --output-file ds_enhance_eval_hi.jsonl \\\n  --instruction-path instruction_prompts.yml \\\n  --task deepseek_enhance_instruct \\\n  --template-fields 'og_question' 'og_choices' 'question' 'answer' 'Hindi' \\\n  --max-concurrency 4096 \\\n  --backend sglang-oai-chat \\\n  --extra-request-body '{\"temperature\":0.7,\"top_p\":0.9,\"top_k\":50,\"max_tokens\":1024}' \\\n  --port 30000\n```\n\n**Generate embeddings**\n\n```bash\npython3 get_embeddings.py \\\n  --input-file test-00000-of-00001_with_ids.jsonl \\\n  --output-file mmlu_qwen3_embd.jsonl \\\n  --server-url http://127.0.0.1:30000 \\\n  --concurrency 4096\n```\n\n**Cosine similarity (English vs translated enhancements)**\n\n```bash\npython3 calculate_cosine_similarity.py \\\n  --input_dir deepseek_enhance \\\n  --output_dir mmmlu/similarity/cosine/enhance/deepseek_enhance \\\n  --english_file mmlu_eng.jsonl\n```\n\n**Linguist / teacher rating (LLM assisted orchestration)**\n\n```bash\npython3 async_infr.py \\\n  --input-path mmlu_as_in.jsonl \\\n  --output-file ds_enhance_eval_as_qwen3_instruct.jsonl \\\n  --instruction-path instruction_prompts.yml \\\n  --task rate_translated_text_linguist \\\n  --template-fields 'og_question' 'og_choices' 'enhanced_text' 'Assamese' \\\n  --max-concurrency 4096 \\\n  --backend sglang-oai-chat \\\n  --extra-request-body '{\"temperature\":0.7,\"top_p\":0.9,\"top_k\":50,\"max_tokens\":1024}' \\\n  --port 30000\n```\n\n**Model evaluation with `lm-eval`**\n\n```bash\n# Example wrapper which runs evaluation for a model snapshot against the Indic MMLU dataset\nbash lm-eval-llm.sh /path/to/model_snapshot /benchmark-result-path\n```\n\n---\n\n### Summary of LLM-as-Judge Ratings\n\nTo evaluate translation quality systematically, we employed **LLMs as judges**, operating under three carefully designed expert personas:\n\n* **Math Expert Persona** → Rated mathematical correctness, terminology, and conceptual fidelity.\n* **Linguistic Expert Persona** → Assessed fluency, grammar, idiomatic usage, and naturalness of the translation.\n* **Coherence Expert Persona** → Evaluated semantic alignment, logical flow, and clarity between English and the target language.\n\n**What we measured:**\n\n* Language quality, fluency, and linguistic suitability.\n* Mathematical correctness and fidelity of technical content.\n* Overall coherence and semantic alignment with the English source.\n\n| Language  | Avg Maths Rate | Avg Coherence Rate | Avg Linguist Rate | Total Records |\n|-----------|----------------|--------------------|-------------------|---------------|\n| Hindi\t    | 9.658029\t     | 9.722054\t          | 9.667891\t        | 14042         |\n| Bengali\t  | 9.182561\t     | 9.131505\t          | 8.202293\t        | 14042         |\n| Nepali    | 9.473793       | 9.331505           | 8.402293          | 14042         |\n| Telugu    | 9.220837       | 8.975929           | 7.986256          | 14042         |\n| Oriya     | 9.264421       | 8.944096           | 8.220268          | 14042         |\n| Punjabi   | 9.348027       | 9.049138           | 8.253454          | 14042         |\n| Assamese  | 9.147201       | 9.039097           | 8.133243          | 14042         |\n| Sanskrit  | 8.338485       | 8.112520           | 6.390115          | 14042         |\n| Kannada   | 9.322746       | 9.111736           | 8.131819          | 14042         |\n| Sindhi    | 2.070645       | 3.109956           | 2.478137          | 14042         |\n| Gujrati   | 9.260647       | 9.206523           | 8.368965          | 14042         |\n| Marathi   | 9.301951       | 9.342045           | 8.381926          | 14042         |\n| Tamil     | 9.047856       | 9.042943           | 8.071144          | 14042         |\n| Malayalam | 9.134952       | 8.998148           | 7.956488          | 14042         |\n| Maithili  | 9.263282       | 9.118146           | 8.218772          | 14042         |\n\n**How to read the table:**\n\n* Ratings are on a **1–10 scale**, where higher scores reflect better translation and enhancement quality.\n* Most languages achieved **consistently high averages (>8.9)**, showing strong quality after enhancement.\n* A few outliers suggest areas where further targeted refinement may be beneficial.\n\n---\n\n## Cosine similarity \n\n* Cosine similarity gives a quick automated signal of semantic closeness between the translated/enhanced items and their English originals.\n* Typical mean similarities range from ~0.76 to 0.85 across languages; higher means closer embeddings (better semantic preservation).\n* Use the similarity scores as a triage: low similarity → manual review; high similarity → likely semantically faithful.\n\n| Language               | Mean Similarity | Std Similarity | Min Similarity | Max Similarity |\n|-------------------------|-----------------|----------------|----------------|----------------|\n| mmlu_as_in_qwen3_embed  | 0.8045          | 0.0597         | 0.4644         | 1              |\n| mmlu_bn_in_qwen3_embed  | 0.8133          | 0.0504         | 0.5276         | 1              |\n| mmlu_gu_in_qwen3_embed  | 0.8211          | 0.0535         | 0.4016         | 1              |\n| mmlu_hi_in_qwen3_embed  | 0.8472          | 0.0489         | 0.3688         | 1              |\n| mmlu_kn_in_qwen3_embed  | 0.8106          | 0.0565         | 0.4660         | 1              |\n| mmlu_mai_in_qwen3_embed | 0.8226          | 0.0504         | 0.4936         | 1              |\n| mmlu_ml_in_qwen3_embed  | 0.8158          | 0.0531         | 0.5109         | 1              |\n| mmlu_mr_in_qwen3_embed  | 0.8129          | 0.0513         | 0.5281         | 1              |\n| mmlu_ne_in_qwen3_embed  | 0.8242          | 0.0502         | 0.4802         | 1              |\n| mmlu_or_in_qwen3_embed  | 0.8159          | 0.0555         | 0.4684         | 1              |\n| mmlu_pa_in_qwen3_embed  | 0.8246          | 0.0516         | 0.5450         | 1              |\n| mmlu_sa_in_qwen3_embed  | 0.7912          | 0.0574         | 0.4981         | 1              |\n| mmlu_sdd_in_qwen3_embed | 0.7646          | 0.0735         | 0.3633         | 0.9676         |\n| mmlu_ta_in_qwen3_embed  | 0.7964          | 0.0559         | 0.5242         | 1              |\n| mmlu_te_in_qwen3_embed  | 0.8006          | 0.0524         | 0.5379         | 1              |\n\n---\n\n## Model Evaluation Methodology\n\nWe evaluated **~24 open-source LLMs** across **16 Indic languages** (subset of the 22 total dataset languages) plus English, using the Indic MMLU benchmark.\n\nWorkflow:\n\n1. **Environment Setup:** Dockerized `vllm/vllm-openai` for reproducibility.\n2. **Framework:** `lm-eval` for standardized evaluation.\n3. **Metrics:** per-language accuracy, top-3 rankings per language, aggregate averages.\n4. **Visualization:** radar plots for each language and comparative top-3 plots.\n\n---\n\n## Important Note\n\n* **Indic MMLU Dataset = 22 Indian languages**\n* **Current Model Evaluation = 16 Indian languages + English**\n* Work is ongoing to expand the evaluation to cover all 22 languages.\n\n---\n### Indic MMLU Results \n\n| <div align=\"center\">![Assamese](assets/assamese.png)<br/>*Assamese*</div> | <div align=\"center\">![Bengali](assets/bengali.png)<br/>*Bengali*</div> |\n|---------------------------------------------------------------------------|------------------------------------------------------------------------|\n| <div align=\"center\">![English](assets/English.png)<br/>*English*</div>    | <div align=\"center\">![Gujarati](assets/Gujrati.png)<br/>*Gujarati*</div> |\n| <div align=\"center\">![Hindi](assets/Hindi.png)<br/>*Hindi*</div>          | <div align=\"center\">![Kannada](assets/kannada.png)<br/>*Kannada*</div> |\n| <div align=\"center\">![Maithili](assets/maithili.png)<br/>*Maithili*</div> | <div align=\"center\">![Malayalam](assets/Malayalam.png)<br/>*Malayalam*</div> |\n| <div align=\"center\">![Marathi](assets/marathi.png)<br/>*Marathi*</div>    | <div align=\"center\">![Nepali](assets/Nepali.png)<br/>*Nepali*</div>   |\n| <div align=\"center\">![Oriya](assets/oriya.png)<br/>*Oriya*</div>          | <div align=\"center\">![Punjabi](assets/Punjabi.png)<br/>*Punjabi*</div> |\n| <div align=\"center\">![Sanskrit](assets/sanskrit.png)<br/>*Sanskrit*</div> | <div align=\"center\">![Sindhi](assets/sindhi.png)<br/>*Sindhi*</div>   |\n| <div align=\"center\">![Tamil](assets/Tamil.png)<br/>*Tamil*</div>          | <div align=\"center\">![Telugu](assets/Telugu.png)<br/>*Telugu*</div>   |\n\n\n---\n\n## How to reproduce (quick checklist)\n\n1. Prepare the English CAIS/MMLU test set (`.jsonl` with IDs).\n2. Run `trans_mmlu.py` to produce per-language MT outputs.\n3. Run `async_infr.py` with the enhancement task to obtain human-readable improved translations.\n4. Generate embeddings for both English and enhanced translations.\n5. Run `calculate_cosine_similarity.py` to produce similarity reports.\n6. Optionally run human/linguist rating orchestration for additional quality assessment.\n7. Evaluate models with `lm-eval` and generate visualizations.\n\n---\n\n## Best practices & caveats\n\n* **Math and formatting:** preserve mathematical expressions and numerals during translation—these are critical for correctness.\n* **Choice alignment:** ensure answer choices remain in the same order and map to the same ground-truth labels.\n* **Human review:** automated metrics are helpful but not sufficient; include periodic linguist sampling.\n* **Constrained inference:** use deterministic LLM settings for enhancement (lower temperature, consistent decoding) to reduce hallucination.\n\n---\n\n## Visual assets & tables\n\nWe include per-language radar plots and a consolidated results panel in `plots/`. The README in the repo contains thumbnails; click through for full-resolution visuals. \n\n---\n"}, "26f46e2b5d1eea04031505e2a3ca6841579a6076": {"t": 1785771740.9697506, "n": 15627, "h": "0dae0fb88028b63d", "c": "# Indic MMLU — Benchmarking Open-Source LLMs on Indian Languages\n\n**Short description:**\nA rigorously prepared, multi-language adaptation of the CAIS/MMLU test set, translated and enhanced for **22 Indic languages**, designed to evaluate open-source LLMs’ real-world ability to understand Indian languages.\nWe additionally conducted large-scale **evaluation on 16 of these Indic languages (plus English)**, producing detailed accuracy reports and per-language visualizations.\n\n---\n\n## Why this dataset exists\n\nLarge language models are often benchmarked only on English datasets, which leaves open the question of whether they can *actually* understand Indian languages beyond surface-level translation.\n\nThe **Indic MMLU dataset** bridges this gap by:\n\n* Translating the widely used **[CAIS/MMLU](https://huggingface.co/datasets/cais/mmlu) benchmark** into **22 Indian languages**.\n* Enhancing those translations with high-quality open-source LLM refinement.\n* Providing a reproducible evaluation setup for **measuring open-source models across 16 Indic languages (evaluated subset)**.\n\nIf a model’s score on an Indic version is close to its score on the original English MMLU, it indicates **genuine language understanding** and strong cross-lingual generalization.\n\n---\n\n## Coverage\n\n* **Dataset scope:** 22 Indic languages (full benchmark availability).\n* **Evaluation scope:** 16 Indic languages + English (practically evaluated with ~24 open-source LLMs).\n\nThis separation ensures the dataset is **broad and inclusive**, while evaluation focuses on the languages with the most immediate support and resources.\n\n---\n\n## Overview\n\n* **Source benchmark:** CAIS/MMLU (general knowledge QA benchmark).\n* **Target:** 22 Indic languages + English (translated test set).\n* **Workflow:** Machine translation → LLM-based translation enhancement → embedding comparison → Persona based LLM-as-judge Scores → Correlation between Embeddings and LLM as Judge → human/linguist rating and automated evaluation.\n* **Deliverables:** Enhanced translated `.jsonl` datasets, per-language embeddings, cosine similarity reports, teacher/linguist ratings, model evaluation scores and visualizations (radar plots).\n\n---\n\n## Key principles applied\n\n1. **Preserve semantic parity:** Questions remain semantically the same as English; only language changes.\n2. **Quality control:** Model-Based Automated checks (embeddings, math correctness) + human linguist ratings.\n3. **Reproducibility:** Dockerized inference environment, scriptable pipeline, and clear CLI commands.\n4. **Transparency:** We provide similarity metrics to demonstrate translation fidelity and teacher ratings to highlight linguistic quality.\n\n---\n\n## High-level pipeline (what we did)\n\n1. **Translate** the CAIS/MMLU test set to each Indic language using *IndicTrans2* (baseline MT).\n2. **Enhance** those translations using an open-source LLM with task-specific instructions (to correct phrasing, clarify ambiguous translations, preserve math expressions, and maintain answer choices alignment).\n3. **Embed** both the original English items and the (enhanced) translated items with the same embedding model.\n4. **Compare** embeddings using cosine similarity to measure semantic closeness.\n5. **LLM as Judge and Human review**: linguists/teachers rated samples for language quality, math correctness, and coherence.\n6. **Evaluate models**: run ~24 open-source LLMs across all languages using the lm-eval framework and visualize results (radar plots, top-3 per language).\n\n---\n\n## Flowchart Representation of the Indic MMLU Pipeline\n\nTo complement the pseudo-code, the following flowchart provides a **visual overview of the Indic MMLU pipeline**. It clearly illustrates the step-by-step workflow, starting from the English MMLU dataset and moving through translation, enhancement, evaluation, and validation:\n\n* **Translation Stage:** The English MMLU instance is translated into Indic languages using specialist MT.\n* **Enhancement Stage:** Translations are refined with large-scale LLMs for improved fluency and alignment.\n* **LLM Based Judging Phase:** Three types of evaluation are conducted — *fluency by native language experts, subject correctness by domain experts, and alignment by coherence judges*.\n* **Validation Stage:** Embedding similarity checks, n-gram metrics (BLEU, ROUGE, ChrF++), and discrepancy detection ensure quality.\n* **Final Output:** A validated Indic MMLU dataset instance ready for benchmarking.\n\n![Indic MMLU Pipeline](assets/indic-mmlu.png)\n\n# Translation Algorithm \n\n```\nInput:\n  original_mmlu\n  target_languages\n  instructions\n\nFor each language L in target_languages:\n    translated = translate(original_mmlu, tgt=L)\n    enhanced = enhance_using_llm(translated, instructions)\n\n    eng_emb = embed(original_mmlu)\n    trans_emb = embed(enhanced)\n    sim_scores = cosine_similarity(eng_emb, trans_emb)\n\n    llm_scores = LLM_as_Judge(original_mmlu, translated)\n\n    consistency = consistency_check(sim_scores, llm_scores)\n\n    If consistency is poor:\n        human_ratings = rate_sample(enhanced)\n    Else:\n        accept enhanced\n\nAggregate all scores:\n    combine sim_scores, llm_scores, human_ratings\n    compute per-language model scores\n\nProduce visualizations:\n    generate charts and plots for cross-language comparison\n\nOutput:\n    per-language evaluation scores\n    visualizations\n\n```\n\nThis visual representation makes it easier to understand how the **algorithmic steps, human evaluation, and automated checks** come together to ensure translation quality and dataset reliability.\n\n---\n\n## Reproducible commands (examples)\n\n> These are the canonical commands used in our pipeline. Replace file paths, ports, and concurrency values as appropriate for your environment.\n\n**Translate**\n\n```bash\npython3 trans_mmlu.py \\\n  --input_file_path test-00000-of-00001_with_ids.jsonl \\\n  --output_file_path mmlu_kn_in.jsonl \\\n  --tgt_lang \"kn\" \\\n  --src_lang \"en\"\n```\n\n**Enhance translations (LLM-based)**\n\n```bash\npython3 async_infr.py \\\n  --input-path mmlu_as_in.jsonl \\\n  --output-file ds_enhance_eval_hi.jsonl \\\n  --instruction-path instruction_prompts.yml \\\n  --task deepseek_enhance_instruct \\\n  --template-fields 'og_question' 'og_choices' 'question' 'answer' 'Hindi' \\\n  --max-concurrency 4096 \\\n  --backend sglang-oai-chat \\\n  --extra-request-body '{\"temperature\":0.7,\"top_p\":0.9,\"top_k\":50,\"max_tokens\":1024}' \\\n  --port 30000\n```\n\n**Generate embeddings**\n\n```bash\npython3 get_embeddings.py \\\n  --input-file test-00000-of-00001_with_ids.jsonl \\\n  --output-file mmlu_qwen3_embd.jsonl \\\n  --server-url http://127.0.0.1:30000 \\\n  --concurrency 4096\n```\n\n**Cosine similarity (English vs translated enhancements)**\n\n```bash\npython3 calculate_cosine_similarity.py \\\n  --input_dir deepseek_enhance \\\n  --output_dir mmmlu/similarity/cosine/enhance/deepseek_enhance \\\n  --english_file mmlu_eng.jsonl\n```\n\n**Linguist / teacher rating (LLM assisted orchestration)**\n\n```bash\npython3 async_infr.py \\\n  --input-path mmlu_as_in.jsonl \\\n  --output-file ds_enhance_eval_as_qwen3_instruct.jsonl \\\n  --instruction-path instruction_prompts.yml \\\n  --task rate_translated_text_linguist \\\n  --template-fields 'og_question' 'og_choices' 'enhanced_text' 'Assamese' \\\n  --max-concurrency 4096 \\\n  --backend sglang-oai-chat \\\n  --extra-request-body '{\"temperature\":0.7,\"top_p\":0.9,\"top_k\":50,\"max_tokens\":1024}' \\\n  --port 30000\n```\n\n**Model evaluation with `lm-eval`**\n\n```bash\n# Example wrapper which runs evaluation for a model snapshot against the Indic MMLU dataset\nbash lm-eval-llm.sh /path/to/model_snapshot /benchmark-result-path\n```\n\n---\n\n### Summary of LLM-as-Judge Ratings\n\nTo evaluate translation quality systematically, we employed **LLMs as judges**, operating under three carefully designed expert personas:\n\n* **Math Expert Persona** → Rated mathematical correctness, terminology, and conceptual fidelity.\n* **Linguistic Expert Persona** → Assessed fluency, grammar, idiomatic usage, and naturalness of the translation.\n* **Coherence Expert Persona** → Evaluated semantic alignment, logical flow, and clarity between English and the target language.\n\n**What we measured:**\n\n* Language quality, fluency, and linguistic suitability.\n* Mathematical correctness and fidelity of technical content.\n* Overall coherence and semantic alignment with the English source.\n\n| Language  | Avg Maths Rate | Avg Coherence Rate | Avg Linguist Rate | Total Records |\n|-----------|----------------|--------------------|-------------------|---------------|\n| Hindi\t    | 9.658029\t     | 9.722054\t          | 9.667891\t        | 14042         |\n| Bengali\t  | 9.182561\t     | 9.131505\t          | 8.202293\t        | 14042         |\n| Nepali    | 9.473793       | 9.331505           | 8.402293          | 14042         |\n| Telugu    | 9.220837       | 8.975929           | 7.986256          | 14042         |\n| Oriya     | 9.264421       | 8.944096           | 8.220268          | 14042         |\n| Punjabi   | 9.348027       | 9.049138           | 8.253454          | 14042         |\n| Assamese  | 9.147201       | 9.039097           | 8.133243          | 14042         |\n| Sanskrit  | 8.338485       | 8.112520           | 6.390115          | 14042         |\n| Kannada   | 9.322746       | 9.111736           | 8.131819          | 14042         |\n| Sindhi    | 2.070645       | 3.109956           | 2.478137          | 14042         |\n| Gujrati   | 9.260647       | 9.206523           | 8.368965          | 14042         |\n| Marathi   | 9.301951       | 9.342045           | 8.381926          | 14042         |\n| Tamil     | 9.047856       | 9.042943           | 8.071144          | 14042         |\n| Malayalam | 9.134952       | 8.998148           | 7.956488          | 14042         |\n| Maithili  | 9.263282       | 9.118146           | 8.218772          | 14042         |\n\n**How to read the table:**\n\n* Ratings are on a **1–10 scale**, where higher scores reflect better translation and enhancement quality.\n* Most languages achieved **consistently high averages (>8.9)**, showing strong quality after enhancement.\n* A few outliers suggest areas where further targeted refinement may be beneficial.\n\n---\n\n## Cosine similarity \n\n* Cosine similarity gives a quick automated signal of semantic closeness between the translated/enhanced items and their English originals.\n* Typical mean similarities range from ~0.76 to 0.85 across languages; higher means closer embeddings (better semantic preservation).\n* Use the similarity scores as a triage: low similarity → manual review; high similarity → likely semantically faithful.\n\n| Language               | Mean Similarity | Std Similarity | Min Similarity | Max Similarity |\n|-------------------------|-----------------|----------------|----------------|----------------|\n| mmlu_as_in_qwen3_embed  | 0.8045          | 0.0597         | 0.4644         | 1              |\n| mmlu_bn_in_qwen3_embed  | 0.8133          | 0.0504         | 0.5276         | 1              |\n| mmlu_gu_in_qwen3_embed  | 0.8211          | 0.0535         | 0.4016         | 1              |\n| mmlu_hi_in_qwen3_embed  | 0.8472          | 0.0489         | 0.3688         | 1              |\n| mmlu_kn_in_qwen3_embed  | 0.8106          | 0.0565         | 0.4660         | 1              |\n| mmlu_mai_in_qwen3_embed | 0.8226          | 0.0504         | 0.4936         | 1              |\n| mmlu_ml_in_qwen3_embed  | 0.8158          | 0.0531         | 0.5109         | 1              |\n| mmlu_mr_in_qwen3_embed  | 0.8129          | 0.0513         | 0.5281         | 1              |\n| mmlu_ne_in_qwen3_embed  | 0.8242          | 0.0502         | 0.4802         | 1              |\n| mmlu_or_in_qwen3_embed  | 0.8159          | 0.0555         | 0.4684         | 1              |\n| mmlu_pa_in_qwen3_embed  | 0.8246          | 0.0516         | 0.5450         | 1              |\n| mmlu_sa_in_qwen3_embed  | 0.7912          | 0.0574         | 0.4981         | 1              |\n| mmlu_sdd_in_qwen3_embed | 0.7646          | 0.0735         | 0.3633         | 0.9676         |\n| mmlu_ta_in_qwen3_embed  | 0.7964          | 0.0559         | 0.5242         | 1              |\n| mmlu_te_in_qwen3_embed  | 0.8006          | 0.0524         | 0.5379         | 1              |\n\n---\n\n## Model Evaluation Methodology\n\nWe evaluated **~24 open-source LLMs** across **16 Indic languages** (subset of the 22 total dataset languages) plus English, using the Indic MMLU benchmark.\n\nWorkflow:\n\n1. **Environment Setup:** Dockerized `vllm/vllm-openai` for reproducibility.\n2. **Framework:** `lm-eval` for standardized evaluation.\n3. **Metrics:** per-language accuracy, top-3 rankings per language, aggregate averages.\n4. **Visualization:** radar plots for each language and comparative top-3 plots.\n\n---\n\n## Important Note\n\n* **Indic MMLU Dataset = 22 Indian languages**\n* **Current Model Evaluation = 16 Indian languages + English**\n* Work is ongoing to expand the evaluation to cover all 22 languages.\n\n---\n### Indic MMLU Results \n\n| <div align=\"center\">![Assamese](assets/assamese.png)<br/>*Assamese*</div> | <div align=\"center\">![Bengali](assets/bengali.png)<br/>*Bengali*</div> |\n|---------------------------------------------------------------------------|------------------------------------------------------------------------|\n| <div align=\"center\">![English](assets/English.png)<br/>*English*</div>    | <div align=\"center\">![Gujarati](assets/Gujrati.png)<br/>*Gujarati*</div> |\n| <div align=\"center\">![Hindi](assets/Hindi.png)<br/>*Hindi*</div>          | <div align=\"center\">![Kannada](assets/kannada.png)<br/>*Kannada*</div> |\n| <div align=\"center\">![Maithili](assets/maithili.png)<br/>*Maithili*</div> | <div align=\"center\">![Malayalam](assets/Malayalam.png)<br/>*Malayalam*</div> |\n| <div align=\"center\">![Marathi](assets/marathi.png)<br/>*Marathi*</div>    | <div align=\"center\">![Nepali](assets/Nepali.png)<br/>*Nepali*</div>   |\n| <div align=\"center\">![Oriya](assets/oriya.png)<br/>*Oriya*</div>          | <div align=\"center\">![Punjabi](assets/Punjabi.png)<br/>*Punjabi*</div> |\n| <div align=\"center\">![Sanskrit](assets/sanskrit.png)<br/>*Sanskrit*</div> | <div align=\"center\">![Sindhi](assets/sindhi.png)<br/>*Sindhi*</div>   |\n| <div align=\"center\">![Tamil](assets/Tamil.png)<br/>*Tamil*</div>          | <div align=\"center\">![Telugu](assets/Telugu.png)<br/>*Telugu*</div>   |\n\n\n---\n\n## How to reproduce (quick checklist)\n\n1. Prepare the English CAIS/MMLU test set (`.jsonl` with IDs).\n2. Run `trans_mmlu.py` to produce per-language MT outputs.\n3. Run `async_infr.py` with the enhancement task to obtain human-readable improved translations.\n4. Generate embeddings for both English and enhanced translations.\n5. Run `calculate_cosine_similarity.py` to produce similarity reports.\n6. Optionally run human/linguist rating orchestration for additional quality assessment.\n7. Evaluate models with `lm-eval` and generate visualizations.\n\n---\n\n## Best practices & caveats\n\n* **Math and formatting:** preserve mathematical expressions and numerals during translation—these are critical for correctness.\n* **Choice alignment:** ensure answer choices remain in the same order and map to the same ground-truth labels.\n* **Human review:** automated metrics are helpful but not sufficient; include periodic linguist sampling.\n* **Constrained inference:** use deterministic LLM settings for enhancement (lower temperature, consistent decoding) to reduce hallucination.\n\n---\n\n## Visual assets & tables\n\nWe include per-language radar plots and a consolidated results panel in `plots/`. The README in the repo contains thumbnails; click through for full-resolution visuals. \n\n---\n"}, "51226205dcf0683bb2eff911554bef7cce7760cd": {"t": 1785771742.5584314, "n": 156, "h": "b2db554740a504db", "c": " ### Downloading and Reproducibility Links Coming within 24 hours.\n---\n\n### Indic MMLU Translation Pipeline\n![Indic MMLU](/readme-resources/indic-mmlu.png)\n"}, "0168fc0dde6e49d417531d9bac6cb1e00f14c863": {"t": 1785771743.873519, "n": 119, "h": "bf0bcc0200d1ed91", "c": " ### Downloading and Reproducibility Links Coming within 24 hours.\n---\n![Indic MMLU](/readme-resources/indic-mmlu.png)\n"}, "88d8cc78d654cc08c402166afe68b457da6a5cb0": {"t": 1785771745.202153, "n": 148, "h": "7b1e70dd76b76da8", "c": " ### Downloading and Reproducibility Links Coming within 24 hours.\n---\n### Data Governance Pipeline\n![Indic MMLU](/readme-resources/indic-mmlu.png)\n"}}}}