Do Multilingual VLMs Abstain Under Cross-Modal Conflict in Low-Resource Languages?
Vision-language models must choose between visual evidence and text when the two conflict. That choice matters for safety. Hundreds of millions of people use Hindi, Telugu, and other languages with limited training and red-team data. In a clinical example, an X-ray may show a fracture while its report says there is none. A model can perceive the fracture yet repeat the false report. We had little evidence on whether safe conflict handling transfers from English to lower-resource languages.
Raw text bias mixes two failures: following a false caption and failing to see the right answer at all. One average also hides model-language pairs that work in English and fail in Telugu. Abstention creates another risk. A model that refuses every question looks cautious but provides no visual value. Existing fixes require costly fine-tuning or contrastive decoding, while most earlier evaluations cover only English.
We built and released four conflict datasets in English, Hindi, and Telugu. They cover natural scenes, physics renders, remote sensing, and rendered 3D objects. NLLB-200-600M and Claude produced translations that people then checked. Each item is scored with and without a false caption. The resulting override gap and normalised override share separate caption-driven errors from basic failures of visual perception.
We ran 10,260 conflict tests on nine open multilingual models, three languages, and four datasets. We found two language-dependent failures: text capture, where the caption wins, and abstention collapse, where the model refuses everything. Override share exposed unsafe behaviour hidden by refusal rates. Qwen2.5-VL-7B had the lowest raw text bias but the highest override share at 0.81, reversing the simple ranking.
We cached each layer's residual-stream activation at the final prompt token. L2-regularised logistic probes with five-fold cross-validation then predicted whether the model followed the image or the text. The signal remained easy to decode even when behaviour worsened: peak accuracy was 0.97 in English and 0.92 in Telugu. The representation stayed strong while the model stopped acting on it.
We fit an abstain-versus-answer direction in English and injected it at middle layers through forward hooks. We varied its strength while guarding against unrelated perception errors, then transferred it to Hindi and Telugu. For models able to abstain, one English vector reduced steered text override to zero in every target language, with a 0.00 transfer gap. The method runs only at inference and needs no target-language labels. We released the MIT-licensed code and all four datasets.
This is the first multilingual audit of cross-modal conflict that controls for visual perception. It shows where grounding fails beyond English. The override gap lets deployers and regulators identify the model-language pairs most likely to follow a false caption. It also shows that an inference-time direction learned in a high-resource language can transfer to lower-resource ones. The open benchmark, steering pipeline, and datasets support direct replication and further safety work.
Cross-modal conflict is usually tested only in English. I see it as a direct safety problem. A model becomes dangerous when it ignores sound visual evidence and follows false text. The people most exposed may use languages missing from current tests. This problem connects interpretability with low-resource NLP, and both fields need to test their claims together.
Behaviour and representation separated. Models followed false text more often in lower-resource languages, yet their internal signal remained almost as clear as in English. The model still represented the visual answer but stopped using it. The model retained the knowledge and failed to elicit it.
I will first test how far an honesty direction learned in one language transfers across many others and identify where it breaks. I will also develop a standard, perception-controlled audit of missed abstention across languages. The next goal is to add coverage guarantees, turning abstention into a bounded decision with a stated guarantee.