Hidden-State Detection of In-Context Goal Hijacking with a Conformal False-Positive Guarantee
Deployed LLM security monitoring mostly watches inputs and outputs. An in-context goal-hijack attack embeds an instruction in the prompt that tries to override the user's task, and it changes the model's internal computation whether or not the final output changes. When the model resists the injection at the output level, which happens often, input and output monitoring records nothing, and the attempt goes unlogged. This work-test prototype for the Heron AI Security Research Fellowship asks a narrower, testable question: do a small instruction-tuned model's own hidden states carry a linearly readable signature of a hijack attempt, and can the resulting detector's false-positive rate be bounded before deployment rather than guessed at.
Activation-based attack detectors are usually reported as a bare accuracy or AUC number, which hides two failure modes. A detector trained on clean prompts versus injected-prefix prompts can reach a perfect score by learning "there is extra text before the task" rather than "the goal is being hijacked", and that shortcut also fires on completely harmless prefixed prompts, so the reported accuracy misstates real deployment behaviour unless a benchmark is built specifically to expose it. Separately, a raw probe score gives a security operator no way to pre-commit to an acceptable false-alarm rate. Both problems needed to be solved inside the benchmark and the calibration method, not patched after the fact.
Built a read-only pipeline on Qwen2.5-0.5B-Instruct, one forward pass per prompt over 640 prompts across four conditions (clean, attacked, a benign-prefix control whose hard negatives reuse the attacks' command vocabulary in benign senses, and a held-out attack family excluded from training), with residual-stream hidden states cached at every layer so the full ablation suite reuses a single model pass. A logistic-regression probe on one layer's last-token state scores each prompt, and a split-conformal threshold calibrated on benign-only scores sets the decision rule at a chosen false-positive budget, alpha = 0.05.
Designed the benign-prefix control specifically to catch the shortcut described above, and it did. The naive probe (clean vs. attacked only) flagged 100% of harmless prefixed prompts as attacks under clean-only calibration, across all ten seeds, despite reaching a perfect AUC on the headline metric. Retraining the decision boundary to separate goal-override prefixes from harmless ones (control-aware training) removed the false alarms, raised the deconfounded AUC from 0.937 to 0.998, and restored true-positive rate on the held-out attack family from 0.710 to 0.988.
Stress-tested the control-aware detector, at its original threshold and with no retraining, against attacks it had never seen in any form, paraphrases built to avoid every command word in the training templates, and attacks moved from the prefix to the suffix position. True-positive rate held at 1.000 in both cases, which rules out keyword matching and position-specific memorisation as explanations for the signal.
Verified the conformal guarantee empirically over 200 random calibration/test splits (mean false-positive rate 0.037 against a 0.05 target) and surfaced a second calibration finding: a threshold calibrated only on prefix-form benign traffic misclassified 25.8% of harmless suffix-form prompts, even though the detector's score ranking stayed near-perfect (AUC 0.999). Extending the calibration set with representative benign traffic, with detector weights unchanged, restored the bound for the mixture (0.045 combined false-positive rate; the remaining suffix-subgroup gap, 0.087, motivates group-conditional calibration). Ten ablations in total (A1 to A10) plus a behavioral check and five unit tests rule out layer-choice artifacts, pipeline leakage, and label noise.
Shows, on one small model and one attack family, that a benchmark with only clean and attacked conditions can certify a detector at perfect AUC that misclassifies every harmless prefixed prompt, and that a calibration set restricted to one benign format can certify a 5% false-alarm budget that measures five times higher on another benign format. Both findings argue for building confound controls and calibration-coverage audits into security benchmarks by default rather than treating a single accuracy number as sufficient. The behavioral check grounds the case for internal monitoring directly: injections flip the model's output only 36 to 49% of the time, while the internal probe flags almost every attempt, exactly the regime where output monitoring under-fires.
Most LLM security tooling watches the boundary, inputs and outputs, and treats the model as a black box in between. That framing misses an attack the moment the model successfully resists it at the output level, which is most of the time here. I wanted to test the narrower, falsifiable claim that the internal computation still carries the signature even when the output looks clean, and to hold myself to the same bar I would want applied to my own conformal prediction work: a false-positive rate that is guaranteed, not just measured on one favourable split.
The naive probe's failure was the most useful result in the project. It reached a perfect headline AUC and would have shipped as a working detector, and the benign-prefix control was the only thing that caught it flagging every harmless prefixed prompt. The fix was not a better model or more data, it was a different negative class in training. That is a cheap, structural lesson about benchmark design that generalises well past this one detector.
With hard negatives in the benchmark the layer sweep finally shows structure — chance at the embeddings, a peak in early-middle layers, decay toward the head — but the injections are still template-generated. The next step is a harder, semantically diverse attack suite in the style of in-context representation hijacking, where layer localisation becomes a sharper question, alongside larger models, group-conditional calibration for the subgroup coverage gap the suffix-position ablation exposed, and moving from an external probe toward a detector the model can use on itself.