SEA-VL: multicultural vision-language benchmark for Southeast Asia

Leading vision-language models interpret images and text together. However, they rely mainly on English and Western image-text data. Southeast Asia spans 11 countries and more than 700 million people. Moreover, it has hundreds of languages and distinct cultural practices. Yet training data and benchmarks underrepresent the region. Before SEA-VL, no rigorous community benchmark measured culturally grounded visual reasoning across Southeast Asia.

A valid cultural benchmark needs questions reflecting local food, festivals, buildings, scripts, and social practices. Scraped images alone cannot supply that knowledge. However, standard crowdsourcing platforms lack enough regional knowledge. Using GPT-4 or Gemini to generate their own test would also make evaluation circular. Moreover, checking quality becomes harder across 11 languages, including scripts with little available training data.

I co-led more than 100 community annotators across all 11 Southeast Asian countries. They wrote image, question, and answer sets from lived cultural knowledge. Web search could not substitute for that knowledge.

I designed and built the quality pipeline. It covered annotation rules, automated filters, human review, language checks, and cultural validation. To remove duplicate images, I compared several similarity methods against a human-checked reference set. Then, I selected the most reliable method. The final contribution included more than 10,000 image-question pairs in 11 languages.

The benchmark measures consistent cultural reasoning gaps in GPT-4V, Gemini, and Claude. Therefore, AI labs can use the results to set localisation priorities for Southeast Asian markets.