CommonLID: language identification on noisy web data
Language identification detects which language a text uses. It supports translation routing, content moderation, dataset building, and search indexing. However, these pipelines often treat it as solved. FastText LangID, GlotLID, and OpenLID regularly score above 95% on clean benchmarks. Those scores create false confidence in performance on real web data.
Southeast Asian web text mixes languages within sentences and uses Roman spellings alongside standard scripts. Moreover, it includes phonetic spelling and social-media shorthand. Systems trained on curated text perform much worse on this data. However, existing benchmarks miss that decline because they omit the web conditions these systems face.
We re-tested fastText LangID, GlotLID, and OpenLID on CommonCrawl web text. This text included code-switching, where writers mix languages, as well as Romanised scripts and spelling variation. Then, we mapped each system’s failures across language families. We proposed stricter tests for claims about Southeast Asian languages with little available training data.
The results challenge published accuracy claims. Therefore, pipelines using language identification need further testing on real web data. This includes CommonCrawl-based datasets used to train major language models.
These failures affect content moderation, translation routing, and language model datasets across Southeast Asia. Moreover, the evaluation protocol gives future benchmarks a clear method for testing real web data. This supports tests of languages with little available training data.