Hidden-State Detection of In-Context Goal Hijacking with a Conformal False-Positive Guarantee
Most deployed LLM security tools monitor inputs and outputs. A goal-hijack attack hides an instruction in the prompt to override the user's task. It changes the model's internal computation even when the final answer remains safe. In that case, output monitoring can miss the attempt. For the Heron Fellowship work test, I asked two precise questions. Do hidden states contain a linearly readable signal of the attack? Can we bound the detector's false positive rate before deployment?
Accuracy and AUC alone hide two important failures. First, a detector may learn only that extra text appears before the task. It can then score perfectly while flagging harmless prefixes and missing the goal hijacking itself. The benchmark needs controls that expose this shortcut. Second, a raw probe score does not let an operator set an acceptable false-alarm rate in advance. The benchmark and calibration method must address both problems directly.
I built a read-only pipeline for Qwen2.5-0.5B-Instruct and ran one forward pass on each of 640 prompts. The four groups were clean prompts, attacks, harmless prefixes using attack vocabulary in safe ways, and an attack family excluded from training. I cached every layer's residual-stream state so all ablations could reuse the same pass. A logistic probe scored the last-token state at one layer. A split-conformal threshold, calibrated only on benign scores, set the decision rule at alpha = 0.05.
The harmless-prefix control exposed the shortcut. Across all ten seeds, a naive clean-versus-attack probe flagged every harmless prefixed prompt, despite perfect headline AUC. Control-aware training separated goal overrides from harmless prefixes. It removed those false alarms, raised deconfounded AUC from 0.937 to 0.998, and increased true-positive rate on the held-out attack family from 0.710 to 0.988.
I kept the original model and threshold for transfer tests. These tests covered unseen attacks, new wording without training command words, and attacks moved from the start to the end of the prompt. True-positive rate remained 1.000 in both transfer tests. This rules out simple keyword and position matching. I then repeated the benchmark on 1.5B, 3B, and 7B models in the same family. Detection improved with size and false alarms stayed near budget. Larger models followed the attack more often: about 27–49% at 0.5B and 1.5B, compared with 73–85% at 3B and 7B.
Across 200 random calibration and test splits, the mean false-positive rate was 0.037 against a 0.05 target. A second test exposed a coverage gap. A threshold calibrated only on benign prefixes misclassified 25.8% of harmless suffix prompts, despite an AUC of 0.999. Adding representative benign traffic to calibration restored the mixed-group bound at 0.045, without changing detector weights. The suffix subgroup remained at 0.087, which motivates group-conditional calibration. Twelve ablations, one behavioural check, and five unit tests ruled out layer choice, pipeline leakage, and label noise.
I compared the probe with a bag-of-words classifier on the raw prompt, using the same training and calibration protocol. Because these attacks are visible in the input, this is the correct baseline. It nearly matched the probe, with deconfounded AUC of 1.000. At the fixed threshold, it caught 0.880 of attacks in the unseen phrasing family versus 0.988 for the probe. These results do not show that hidden states are necessary for this attack class. They show that hidden-state scores transfer better to unseen wording. A stronger comparison needs attacks that cannot be read directly from input.
On one model family and attack style, a clean-versus-attack benchmark gave perfect AUC to a detector that flagged every harmless prefix. Calibration on one benign format also promised a 5% alarm rate while another format measured five times higher. Security benchmarks should therefore include confound controls, coverage audits, and simple input-only baselines by default. The behavioural results also show the limit of output monitoring. The 0.5B model followed 36–49% of injections, the 7B model followed about 80%, and the probe detected almost every attempt at each size. Greater capability helped the attacker while output monitoring missed attempts.
Most LLM security tools watch inputs and outputs while treating the model itself as a black box. They miss an attempt when the output remains safe, which happened often here. I tested the narrower claim that internal states retain an attack signal even when the answer looks clean. I also required a guaranteed false-positive rate, applying the same standard I use in my conformal prediction work.
The naive probe's failure was the most useful result. Its AUC was perfect, yet it flagged every harmless prefix. Only the benign-prefix control exposed the problem. Changing the negative training class fixed it; a larger model or more data was unnecessary. The result is a simple benchmark-design lesson that applies beyond this detector.
With hard negatives, the 0.5B layer sweep rises from chance at the embeddings, peaks in the early-middle layers, then fades near the output head. That pattern disappears at 3B and 7B, and the current attacks still come from templates. Next I will test more varied representation-hijacking attacks, a second model family, and group-conditional calibration for the suffix coverage gap. I also want to move from an external probe to a detector the model can use on itself.