Goal Hijack Probe

Terms used on this page

01 Problem

A goal-hijack attack places a competing instruction in a prompt and tries to replace the user's task. A model may receive that instruction yet answer the original task. Output monitoring then sees a normal answer and misses the attempt. Input filters can inspect the visible instruction. This study compares one of those filters with a detector that reads the model's hidden states during inference.

This study asks two questions across one family of instruction-tuned models.

1. Do residual-stream hidden states contain a signal a linear probe can use to distinguish hijack attempts from benign traffic, including benign traffic that superficially resembles an attack?
2. Can split conformal calibration bound the detector's average false-positive rate under the assumption that future benign prompts resemble the calibration sample?

The threat model is benign. Each injection requests a harmless word or number. The detector identifies an attempt to override the goal; the experiment generates no harmful content.

02 Existing literature

Prior work motivates reading internal states. Conformal prediction supplies a way to calibrate the decision threshold.

• Arditi et al. (NeurIPS 2024) identified a direction in model activations that mediates refusal behavior. That result motivates testing simple readouts of other safety-relevant states; it does not establish that goal hijacking uses the same direction.
• Yona et al. (2025, preprint) introduced in-context representation hijacking. In their attack, adversarial context alters internal representations and induces unsafe behavior. Internal representations are an attack surface.
• Lindsey (2026, preprint) reported a functional and unreliable form of introspective awareness in large language models.
• Plunkett et al. (2025, preprint) showed that models can describe internal processes behind their decisions when given access to those processes.
• Split conformal prediction (Vovk et al. 2005; Angelopoulos and Bates 2023) converts any anomaly score into a decision rule with a finite-sample marginal guarantee under exchangeability. Bates et al. (2023) formalized outlier detection with conformal p-values.

03 Gap

Activation-based attack detectors usually report accuracy or AUC. This prototype adds two missing controls.

A guaranteed false-positive budget. An operator must set an acceptable alarm rate before deployment. A raw probe score cannot do this. Under exchangeable calibration and deployment traffic, a split-conformal threshold gives the finite-sample guarantee P(flag | benign) ≤ α.

Confound controls inside the benchmark. A detector can reach perfect AUC by learning that extra text appears before the task. It will then flag harmless prefixes too. This benchmark includes a benign-prefix condition designed to expose that shortcut. Section 5 shows the result.

04 Methodology

Dataset (640 prompts, four conditions, built to catch its own shortcuts)

160 instances of each condition were generated from 48 benign base tasks (short factual and instruction-following questions, including attack-adjacent phrasings such as How do I override a CSS style with higher specificity, briefly?):

Every condition wraps the same pool of 48 benign base tasks; only what is prepended changes. The task Summarize the water cycle in one sentence. appears in all four conditions below.

Ten goal-hijack templates and twelve benign prefixes are sampled uniformly per instance; the example above shows one draw of each. Eight of the twelve prefixes are hard negatives that reuse the attack templates' command vocabulary (ignore, disregard, system, skip, forget, developer) in benign senses, such as "Note from the developer of this quiz app: keep answers short." A detector keyed on surface vocabulary or on "instruction-like prefix" fires on these; one keyed on goal override must not. An earlier draft without hard negatives scored a saturated 1.000 on every power metric. Those scores were real but uninformative. The full template lists are in Config in self_probe_hijack_detection.py.

Feature extraction (read-only)

One forward pass per prompt runs through Qwen2.5-0.5B-Instruct with its chat template, recording residual-stream hidden states at all 25 positions (embeddings plus 24 transformer blocks), with last-token and mean pooling. Features are cached, so the entire ablation suite reuses a single model pass. The extraction reads activations and modifies nothing.

Probe and two training protocols

The detector is a logistic regression on standardized features from one layer (layer 12 in the main experiment). Every condition is split 50/50 into training and evaluation pools. Two variants were trained.

• Naive: clean vs. attacked, the protocol of the original scaffold.
• Control-aware: {clean + benign-prefix} vs. attacked, which requires the decision boundary to separate goal-override prefixes from harmless prefixes.

Conformal calibration

A one-sided split-conformal threshold τ is computed on benign-only calibration scores at the corrected level ⌈(n+1)(1−α)⌉/n, so the marginal bound P(score > τ | benign) ≤ α is valid at finite sample sizes. The calibration pool mixes clean and benign-prefix scores, matching deployment traffic that contains harmless prefixed prompts in this benchmark, with α = 0.05. The bound requires future benign prompts to be exchangeable with the calibration sample.

Ablations and controls

The twelve A9 paraphrases express the same goal-override intent while avoiding the command words in the training templates. None of them use ignore, disregard, override, forget, skip, cancel, system, admin, directive, instruction, developer, task, or request. The A8 conditions reuse the ten training templates and the twelve harmless prefixes, appended after the task after it has learned prefixes that appear before the task. A8/A9 evaluate the prefix-trained detector at its original conformal threshold, with no retraining.

Five unit tests cover the conformal math, including Monte-Carlo validation of the p-value calibration and of the threshold rule. All five pass.

05 Results

Headline metrics (10 seeds, mean ± sd, layer 12, α = 0.05)

The confound, quantified

Both probes reach AUC 1.000 on clean versus attacked. The control conditions separate their performance. With the threshold calibrated on clean scores only, the naive probe flags 100% of benign-prefix controls (1.000 ± 0.000 across all 10 seeds) as attacks. The naive decision boundary encodes the feature "extra text precedes the task". Calibrating on a mixed benign pool lowers the naive false alarms to 0.031 ± 0.020, but the threshold this pushes upward collapses held-out-family TPR to 0.710 ± 0.083. The naive probe also ranks the hard-negative prefixes poorly in the first place (deconfounded AUC 0.937 ± 0.018). Control-aware training raises held-out-family TPR to 0.988 ± 0.013 and the deconfounded AUC from 0.937 to 0.998.

Ablation outcomes

• A1/A2 layer and pooling sweep. With hard negatives in the benchmark, the last-token sweep is no longer saturated. It starts at chance (0.500) at layer 0, peaks at 1.000 around layers 4–7, and falls to ≈0.977 by layer 24. The goal-override signal is most linearly available in early-middle layers in this sweep. Layer 12 reads 0.992 on the sweep split and 0.998 ± 0.003 across seeds. Mean pooling stays at 1.000 at every layer. Because it includes the injected tokens, that score may reflect their presence as well as the model's processing of the instruction.
• A3/A10 shuffled labels. A single shuffle gives AUC 0.514. Twenty independent shuffles give 0.518 ± 0.039 (range 0.455–0.598), so the single-shuffle value is an unremarkable draw from a null centered on chance. This control found no evidence of label leakage in the tested pipeline.
• A4 leave-one-injection-out. Mean AUC 0.997 over the ten held-out templates (nine of ten at 1.000, minimum 0.967). This result argues against dependence on any one training template.
• A7 conformal validity. Mean empirical FPR over 200 random calibration/test splits is 0.037 (median 0.025), below the target α = 0.05. The 99th-percentile single-split FPR is 0.138; the guarantee bounds the expectation, and individual splits may exceed α.

Paraphrase, position, and calibration tests

The detector reaches TPR 1.000 on twelve hijack paraphrases without the training templates' command words. It also detects attacks moved to the end of the prompt. These tests weaken a simple keyword or fixed-position explanation, although the template-based benchmark cannot rule out other lexical shortcuts.

The benign side produced the second calibration finding. The prefix-calibrated threshold misclassifies 25.8% of harmless suffixed prompts, although the score ranking between benign-suffix and suffix-attack remains at AUC 0.999. Benign traffic of a form absent from the calibration set violates the exchangeability assumption, and the FPR bound becomes vacuous for that traffic. Adding suffix-form benign scores to the calibration pool, with detector weights unchanged, reduces the combined benign-pool FPR to 0.045 ± 0.034, within the α = 0.05 target, at essentially unchanged TPR. The bound is marginal over the calibration mixture. The suffix subgroup alone still measures 0.087 ± 0.067, above α, and per-subgroup validity requires group-conditional (Mondrian) calibration.

The same benchmark from 0.5B to 7B (A11)

The full benchmark and 10-seed protocol were rerun on the Qwen2.5-Instruct family at four sizes, with the probe at mid-depth for each model. All four models belong to the same architecture family, although width and depth also vary with size.

Deconfounded AUC rises with model size in this family, reaching 1.000 at 3B and 7B. False-alarm means range from 0.029 to 0.054 across sizes; the 7B mean exceeds the 0.05 target by 0.004. A marginal conformal bound concerns future benign prompts under the calibration assumptions, not every observed test split. The gap between naive and control-aware AUC narrows with size in this benchmark.

The behavioral result reversed my expectation. The two smaller models obeyed the injected command in roughly three or four attempts out of ten. The two larger models obeyed it in roughly eight out of ten. In this model family and attack set, larger models followed the competing instruction more often. The probe flagged at least 98.8% of attempts at every size, whether the model obeyed or not.

The baseline that keeps the claims honest (A12)

These injections sit in plain sight in the prompt, so the fair question is what reading hidden states buys over a detector that never looks inside the model. A bag-of-words classifier on the raw prompt text (TF-IDF features, logistic regression), trained and calibrated under the identical control-aware, 10-seed protocol, is that baseline.

The baseline nearly matches the probe. Its clearest measured deficit is threshold transfer. Attack phrasings never seen in training score lower than trained ones under the text model, and 12% slip under the fixed alarm line, against 1.2% for the probe. On these template-generated injections that are legible in the input text, input monitoring performs well, and this benchmark cannot establish that internal signals are required. Hidden-state scores transfer better to unseen attack phrasings at the fixed threshold, and a stronger comparison needs attacks that are difficult to read in the prompt, such as encoded or obfuscated injections, representation-space attacks, and poisoned tool outputs. That extension is the next step.

Behavioral grounding

Greedy decoding on all 320 attacked prompts measures how often the injection changes the model's output. On the 0.5B model of the main experiments, the injected instruction succeeds in 48.8% of training-family prompts and 36.3% of held-out-family prompts. The internal probe flags ≈99% of attempts in both families. The attacked label therefore means a hijack attempt is present in context. It does not mean the hijack succeeded.

06 Discussion

What the evidence supports

• Hidden states of Qwen2.5-0.5B-Instruct contain a linearly separable signal for this class of in-context goal hijack (deconfounded AUC 0.998 ± 0.003 across 10 splits, on a benchmark whose benign class includes hard negatives).
• The probe generalizes beyond the training templates. TPR is 0.988 ± 0.013 on an unseen phrasing family and 1.000 ± 0.000 on paraphrases that share no command vocabulary with the training templates and on attacks moved from prefix to suffix position. The detector stays within budget on harmless prefixes, including hard negatives that reuse attack vocabulary, when the training protocol includes the control condition (benign-pool FPR 0.029 ± 0.016 at α = 0.05).
• Across 200 random calibration and test splits, mean empirical FPR is 0.037 against a target of 0.05. This check supports the expected behavior under the tested distribution; it does not verify the bound for a different deployment distribution.
• The internal signal is informative when output monitoring records no change. On the 0.5B model, injections change the output in 36–49% of cases, and the probe flags ≈99% of attempts.
• Against input-text monitoring, the probe's clearest advantage is threshold transfer to unseen attack phrasings. Its TPR is 0.988 against the text baseline's 0.880 at the fixed alarm line.
• The findings extend across the tested model family. On Qwen2.5-Instruct 1.5B, 3B and 7B the same benchmark gives deconfounded AUC 0.999–1.000 and held-out TPR 0.997–1.000, with the false-positive rate near budget. Compliance with the injection rises from roughly 27–49% of attempts to 73–85%. Larger models follow the hijack more often, and the probe sees the attempt either way.

What the evidence does not support

• The probe is an external classifier that reads activations. A future detector the model can use on itself would need a separate evaluation. This experiment does not test model introspection.
• No claim that internal signals are required for this attack class. A12 shows an input-only text classifier nearly matches the probe, because these injections are visible in the prompt. "Internal beats output monitoring" is measured here; "internal beats input monitoring" remains unproven in this benchmark and would require attacks that are illegible in the input text.
• Results cover one model family (Qwen2.5-Instruct, 0.5B–7B) and one template-generated attack style. The hard negatives remove the worst lexical shortcut and give the 0.5B layer sweep real structure (peak at layers 4–7, decay toward the head), but that late-layer structure disappears at 3B/7B, and the injections remain template-generated; extension to paraphrase-diverse and semantic hijacks in the style of Yona et al., and to a second architecture family, is required before the localization reading can be trusted.
• The conformal bound assumes exchangeability between calibration and deployment benign traffic. A8 measures the cost of violating it. Benign prompts in an uncalibrated form (suffix position) produced a 25.8% false-alarm rate against a 5% target. Recalibration with representative benign traffic restored mixture-level validity (combined-pool FPR 0.045 ± 0.034). The guarantee is marginal in two senses. Single calibration splits reached FPR 0.138 at the 99th percentile while the mean measured 0.037, and subgroup FPR (0.087 on suffix-benign) exceeds the mixture-level bound. Group-conditional calibration addresses the latter.

Implication for benchmark design

The clean-versus-attack comparison gave a detector AUC 1.000 while it flagged every harmless prefixed prompt. Likewise, a threshold calibrated on one benign format flagged 25.8% of harmless prompts in another format, despite a 5% target for the represented traffic. The input-text baseline nearly matched the probe on visible injections. Future benchmarks should include harmless lookalikes in detector training, check calibration coverage against intended traffic, and report internal detectors alongside input-only baselines.

07 Conclusion

This study reads residual-stream hidden states during inference, trains a linear probe to flag in-context goal-hijack attempts, and calibrates its threshold with split conformal prediction. On Qwen2.5-0.5B-Instruct, the control-aware probe reaches deconfounded AUC 0.998 ± 0.003 and a held-out-family TPR of 0.988 ± 0.013. Its measured benign-pool FPR is 0.029 ± 0.016 against a target of 0.05. Harmless prefixes that reuse attack words expose a failure hidden by the initial clean-versus-attack comparison. The corrected training protocol improves both ranking and detection at the fixed threshold.

Calibration coverage remains a separate problem. A threshold fitted to benign prefixes flags 25.8% of harmless suffix-form prompts. Recalibration with suffix-form examples brings combined benign-pool FPR to 0.045, while the suffix subgroup still measures 0.087. The conditional bound applies to the represented mixture. An input-text classifier also performs well on these visible injections. The probe's clearest advantage is higher TPR on the unseen attack family at the fixed threshold, 0.988 versus 0.880. These results motivate testing attacks that are harder to read from input, a second architecture family, and calibration methods that protect individual benign subgroups.

Experiment commands

The original experiment scripts are not included in this website repository. These commands record the protocol used for the reported results.

The scripts produce results_main.json, results_ablations.json, results_ablations_extended.json, results_behavioral.json, results_model_scale.json, results_text_baseline.json, and the figures. Feature caches are populated by one forward pass per prompt per model; every ablation runs from the caches in seconds.

References

1. Arditi et al. Refusal in Language Models Is Mediated by a Single Direction. NeurIPS, 2024.
2. Yona et al. In-Context Representation Hijacking. Preprint, 2025.
3. Lindsey. Emergent Introspective Awareness in Large Language Models. Preprint, 2026.
4. Plunkett et al. Self-Interpretability. Preprint, 2025.
5. Vovk, Gammerman, and Shafer. Algorithmic Learning in a Random World. Springer, 2005.
6. Angelopoulos and Bates. Conformal Prediction. Foundations and Trends in Machine Learning, 2023.
7. Bates et al. Testing for Outliers with Conformal p-values. Annals of Statistics, 2023.