Do Multilingual VLMs Abstain Under Cross-Modal Conflict in Low-Resource Languages?

Vision-language models use images and text together. When these sources conflict, the model must choose which evidence to follow. This choice matters for safety. Hundreds of millions of people use Hindi, Telugu, and other languages with limited training and safety-test data. For example, an X-ray may show a fracture while its report denies one. A model can see the fracture yet repeat the false report.

We had little evidence that safe handling of such conflicts transfers from English to lower-resource languages.

Raw text bias mixes caption-driven errors with failures to see the correct answer. Moreover, an average hides models that work in English and fail in Telugu. Refusing to answer introduces another risk. A model that refuses every question looks cautious yet provides no useful visual answers. Existing fixes require costly fine-tuning or decoding that compares contrasting inputs. Meanwhile, most earlier tests cover only English.

We built and released 4 conflict datasets in English, Hindi, and Telugu. They cover natural scenes, physics renders, remote sensing, and rendered 3D objects. NLLB-200-600M and Claude translated the text, which people then checked. We scored each item with and without a false caption. The override gap measures the change caused by that caption. Moreover, normalised override share separates caption-driven errors from basic failures to read images.

We ran 10,260 conflict tests across 9 open multilingual models, 3 languages, and 4 datasets. We found 2 failures that varied by language. In text capture, the model follows the caption. In abstention collapse, it refuses everything. Override share revealed unsafe behaviour hidden by refusal rates. For example, Qwen2.5-VL-7B had the lowest raw text bias, yet the highest override share at 0.81.

This reversed the simple ranking.

We saved each layer’s internal residual-stream signal at the final prompt token. Then we trained logistic classifiers to predict whether the model followed the image or text. We used L2 regularisation to limit fitted weights and 5-fold cross-validation to test the classifiers. The signal stayed readable even when behaviour worsened. Peak accuracy reached 0.97 in English and 0.92 in Telugu. Thus, the internal signal stayed strong while the model stopped using it.

We learnt an English direction in the model’s internal signals that separated abstaining from answering. Then we injected this direction into middle layers through forward hooks. These hooks change internal signals during a model run. We varied the strength while checking for unrelated perception errors. After that, we transferred the direction to Hindi and Telugu. For models able to abstain, 1 English vector reduced steered text override to zero in every target language.

The transfer gap was 0.00. The method runs only during inference and needs no target-language labels. Moreover, we released the MIT-licensed code and all 4 datasets.

Most tests of image-text conflict cover only English. I see the conflict as a direct safety problem. A model becomes dangerous when it ignores sound visual evidence and follows false text. Moreover, people using languages absent from current tests may face the greatest exposure. This problem connects work on model internals with language research where training data are scarce. Therefore, both fields need to test their claims together.

Model behaviour and internal signals diverged. Models followed false text more often in lower-resource languages. However, their internal signal stayed almost as clear as in English. The model still represented the visual answer, yet failed to use that knowledge in its answer.

First, I will test how far an honesty direction learnt in 1 language transfers across others. I will identify where that transfer breaks. Moreover, I will develop a standard multilingual audit of missed abstention that controls for visual perception. The next goal is to add coverage guarantees. These would bound error risk when the model answers or withholds an answer under stated assumptions.