AI article
Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline
No preview is available. Read the original article for the full story.
huggingface | Oct 16, 2026 | Will Jennings, Kiran Praveen, Sriharsha Niverty, Raviraj Joshi, Markus Kliegl, Yev Meyer, Utkarsh Vaidya
Automated excerpt
Native-script data for 22 languages, romanized data for twelve, and a reusable GPU-accelerated pipeline for translation, quality assessment, and repair. Four-stage Indic synthetic-data pipeline for twelve languages, implemented using NeMo Data Designer and Gemma-4-31B-it. Indic task performance and language quality across Nemotron Super generations.
Selected automatically from source text; not independently written or fact-checked. Read the original for full context.