Hidden-State Detection of In-Context Goal Hijacking with a Conformal False-Positive Guarantee

01 Problem

Most deployed security monitoring for large language models analyzes inputs and outputs. An in-context goal-hijack attack places an injected instruction in the prompt that attempts to override the user's task. The attack modifies the model's internal computation whether or not the final output changes. When the model resists the injection at the output level, input/output monitoring records nothing unusual, and the attempt goes unlogged.

This prototype tests two precise questions on a small instruction-tuned model:

1. Do residual-stream hidden states contain a linearly separable signal that distinguishes hijack attempts from benign traffic, including benign traffic that superficially resembles an attack?
2. Can the resulting detector's false-positive rate be bounded in advance, with finite-sample validity, using split conformal prediction?

The threat model is deliberately benign. Injections request harmless outputs such as a fixed word or number. The detected condition is the presence of a goal-override attempt in context. No harmful content is generated at any point.

02 Existing literature

Four results motivate reading internal states for security, and one line of statistical work supplies the guarantee mechanism.

• Arditi et al. (NeurIPS 2024) showed that refusal behavior is mediated by a small number of identifiable directions in activation space. Safety-relevant state is therefore linearly represented and readable with simple probes.
• Yona et al. (2025, preprint) introduced in-context representation hijacking: adversarial context alters internal representations and induces unsafe behavior. Internal representations are an attack surface.
• Lindsey (2025) reported a functional and unreliable form of introspective awareness in large language models.
• Plunkett et al. (2025, preprint) showed that models can describe internal processes behind their decisions when given access to those processes.
• Split conformal prediction (Vovk et al. 2005; Angelopoulos and Bates 2023) converts any anomaly score into a decision rule with a finite-sample marginal guarantee under exchangeability. Bates et al. (2023) formalized outlier detection with conformal p-values.

03 Gap

Activation-based attack detectors typically report accuracy or AUC. Two elements are usually absent, and this prototype supplies both.

A false-positive budget with a guarantee. A security operator needs to set an acceptable false-alarm rate before deployment. A raw probe score does not support this. A split-conformal threshold does: under exchangeability of calibration and deployment benign traffic, the flag rule satisfies P(flag | benign) ≤ α with finite-sample validity.

Confound controls in the benchmark itself. A detector trained on clean prompts versus prompts with an injected prefix can reach perfect AUC by encoding the feature "extra text precedes the task". That feature also matches harmless prefixed prompts, so the perfect AUC misstates deployment performance. The benchmark here includes a benign-prefix control condition designed to expose exactly this failure. Section 5 shows that it did.

04 Methodology

Dataset (640 prompts, four conditions, built to catch its own shortcuts)

160 instances of each condition were generated from 48 benign base tasks (short factual and instruction-following questions, including attack-adjacent phrasings such as How do I override a CSS style with higher specificity, briefly?):

Every condition wraps the same pool of 48 benign base tasks; only what is prepended changes. One task, Summarize the water cycle in one sentence., in all four conditions:

Ten goal-hijack templates and twelve benign prefixes are sampled uniformly per instance; the example above shows one draw of each. Eight of the twelve prefixes are hard negatives that reuse the attack templates' command vocabulary — ignore, disregard, system, skip, forget, developer — in benign senses ("Note from the developer of this quiz app: keep answers short."). A detector keyed on surface vocabulary or on "instruction-like prefix" fires on these; one keyed on goal override must not. An earlier draft without hard negatives scored a saturated 1.000 on every power metric — real, but uninformative. The full template lists are in Config in self_probe_hijack_detection.py.

Feature extraction (read-only)

One forward pass per prompt runs through Qwen2.5-0.5B-Instruct with its chat template, recording residual-stream hidden states at all 25 positions (embeddings plus 24 transformer blocks), with last-token and mean pooling. Features are cached, so the entire ablation suite reuses a single model pass. The extraction reads activations and modifies nothing.

Probe and two training protocols

The detector is a logistic regression on standardized features from one layer (layer 12 in the main experiment). Every condition is split 50/50 into training and evaluation pools. Two variants were trained:

• Naive: clean vs. attacked, the protocol of the original scaffold.
• Control-aware: {clean + benign-prefix} vs. attacked, which requires the decision boundary to separate goal-override prefixes from harmless prefixes.

Conformal calibration

A one-sided split-conformal threshold τ is computed on benign-only calibration scores at the corrected level ⌈(n+1)(1−α)⌉/n, so the marginal bound P(score > τ | benign) ≤ α is valid at finite sample sizes. The calibration pool mixes clean and benign-prefix scores, matching deployment traffic that contains harmless prefixed prompts, with α = 0.05.

Ablations and controls

The twelve A9 paraphrases express the same goal-override intent while avoiding every command word of the training templates: none of them use ignore, disregard, override, forget, skip, cancel, system, admin, directive, instruction, developer, task, or request. The A8 conditions reuse the ten training templates and the twelve harmless prefixes, appended after the task instead of before it. A8/A9 evaluate the prefix-trained detector at its original conformal threshold, with no retraining.

Five unit tests cover the conformal math, including Monte-Carlo validation of the p-value calibration and of the threshold rule. All five pass.

05 Results

Headline metrics (10 seeds, mean ± sd, layer 12, α = 0.05)

The confound, quantified

Both probes reach AUC 1.000 on clean vs. attacked. The clean-vs-attacked metric therefore cannot distinguish them. The control conditions can. With the threshold calibrated on clean scores only, the naive probe flags 100% of benign-prefix controls (1.000 ± 0.000 across all 10 seeds) as attacks. The naive decision boundary encodes the feature "extra text precedes the task". Calibrating on a mixed benign pool lowers the naive false alarms to 0.031 ± 0.020, but the threshold this pushes upward collapses held-out-family TPR to 0.710 ± 0.083 — and the naive probe ranks the hard-negative prefixes poorly in the first place (deconfounded AUC 0.937 ± 0.018). Control-aware training raises held-out-family TPR to 0.988 ± 0.013 and the deconfounded AUC from 0.937 to 0.998.

Ablation outcomes

• A1/A2 layer and pooling sweep. With hard negatives in the benchmark, the last-token sweep is no longer saturated: chance (0.500) at layer 0, a peak of 1.000 around layers 4–7, and a decay to ≈0.977 by layer 24. The goal-override signal is most linearly available in early-middle layers; the pre-registered layer 12 reads 0.992 on the sweep split and 0.998 ± 0.003 across seeds. Mean pooling stays at 1.000 at every layer, as it must: the mean includes the injected tokens themselves, so it measures lexical presence rather than the model's integrated state.
• A3/A10 shuffled labels. A single shuffle gives AUC 0.514. Twenty independent shuffles give 0.518 ± 0.039 (range 0.455–0.598), so the single-shuffle value is an unremarkable draw from a null centered on chance. No pipeline leakage.
• A4 leave-one-injection-out. Mean AUC 0.997 over the ten held-out templates (nine of ten at 1.000, minimum 0.967). The probe does not memorize template strings.
• A7 conformal validity. Mean empirical FPR over 200 random calibration/test splits: 0.037 (median 0.025), below the target α = 0.05. The 99th-percentile single-split FPR is 0.138; the guarantee bounds the expectation, and individual splits may exceed α.

Extended ablations: paraphrase, position, and a second calibration finding

The paraphrase result rules out keyword matching: the detector reaches TPR 1.000 on hijack phrasings that share no command vocabulary with its training set. The suffix result adds position generalization for attacks.

The benign side produced the second calibration finding. The prefix-calibrated threshold misclassifies 25.8% of harmless suffixed prompts, although the score ranking between benign-suffix and suffix-attack remains near-perfect (AUC 0.999). Benign traffic of a form absent from the calibration set violates the exchangeability assumption, and the FPR bound becomes vacuous for that traffic. Adding suffix-form benign scores to the calibration pool, with detector weights unchanged, reduces the combined benign-pool FPR to 0.045 ± 0.034, within the α = 0.05 target, at essentially unchanged TPR. The bound is marginal over the calibration mixture; the suffix subgroup alone still measures 0.087 ± 0.067 — above α — and per-subgroup validity requires group-conditional (Mondrian) calibration.

Behavioral grounding

Greedy decoding on all 320 attacked prompts measures how often the injection changes the model's output. The injected instruction succeeds in 48.8% of training-family prompts and 36.3% of held-out-family prompts. The internal probe flags ≈99% of attempts in both families. The attacked label therefore means a hijack attempt is present in context. It does not mean the hijack succeeded.

06 Discussion

What the evidence supports

• Hidden states of Qwen2.5-0.5B-Instruct contain a linearly separable signal for this class of in-context goal hijack (deconfounded AUC 0.998 ± 0.003 across 10 splits, on a benchmark whose benign class includes hard negatives).
• The detected feature is semantic. TPR is 0.988 ± 0.013 on an unseen phrasing family and 1.000 ± 0.000 on paraphrases that share no command vocabulary with the training templates and on attacks moved from prefix to suffix position. The detector stays within budget on harmless prefixes — including hard negatives that reuse attack vocabulary — when the training protocol includes the control condition (benign-pool FPR 0.029 ± 0.016 at α = 0.05).
• The conformal threshold satisfies its finite-sample marginal FPR bound empirically (mean FPR 0.037 over 200 splits, target 0.05).
• The internal signal is informative in the regime where output monitoring records nothing: injections change the output in 36–49% of cases, and the probe flags ≈99% of attempts.

What the evidence does not support

• The probe is an external read of activations by a separate classifier. It is the baseline that model self-use methods, the target of the fellowship project, must exceed. No claim of model introspection follows from this work.
• Results cover one 0.5B model and one template-generated attack style. The hard negatives remove the worst lexical shortcut and give the layer sweep real structure (peak at layers 4–7, decay toward the head), but the injections remain template-generated; extension to paraphrase-diverse and semantic hijacks in the style of Yona et al. is required before the localization reading can be trusted.
• The conformal bound assumes exchangeability between calibration and deployment benign traffic, and A8 measures the cost of violating it: benign prompts in an uncalibrated form (suffix position) produced a 25.8% false-alarm rate against a 5% target. Recalibration with representative benign traffic restored mixture-level validity (combined-pool FPR 0.045 ± 0.034). The guarantee is also marginal in two senses: single calibration splits reached FPR 0.138 at the 99th percentile while the mean measured 0.037, and subgroup FPR (0.087 on suffix-benign) exceeds the mixture-level bound. Group-conditional calibration addresses the latter.

Implication for benchmark design

The methodological findings are quantitative. A benchmark with only clean and attacked conditions certified a detector at AUC 1.000 that misclassified 100% of harmless prefixed prompts. A calibration set restricted to one benign format certified a 5% false-alarm budget that measured 25.8% on another benign format. Confound controls belong inside the attack benchmarks the project plans to release, detector training should include them as negative examples, and calibration sets should be audited for coverage of deployment benign traffic.

07 Conclusion

This prototype reads a model's residual-stream hidden states during inference, trains a linear probe to detect in-context goal-hijack attempts, and calibrates the decision threshold with split conformal prediction — on a benchmark deliberately salted with hard negatives so that perfect scores cannot come from surface shortcuts. On Qwen2.5-0.5B-Instruct the control-aware detector reaches deconfounded AUC 0.998 ± 0.003, holds the empirical false-positive rate to 0.029 ± 0.016 against a target of 0.05, and transfers with TPR 0.988 ± 0.013 to an injection family excluded from training — and with TPR 1.000 ± 0.000 to paraphrases with disjoint vocabulary and to suffix-position attacks, so the detected feature is the goal-override semantics rather than any keyword. Two findings came from the controls. The benign-prefix control exposed a confound that inverted the naive benchmark's conclusion. The suffix-position control measured the cost of a calibration set that under-covers benign traffic (25.8% false alarms against a 5% budget), and recalibration with representative traffic restored mixture-level validity while leaving a subgroup gap that motivates group-conditional calibration. The behavioral check showed that the internal signal detects attempts that output monitoring misses. Next steps within the fellowship: representation-level attacks, larger models, weight-space tampering, group-conditional calibration, and the transition from external probes to detectors the model itself can use.

Reproduction

Outputs: results_main.json, results_ablations.json, results_ablations_extended.json, results_behavioral.json, both figures. Two model passes (640 base prompts, 480 extended prompts) populate the feature caches; every ablation runs from the caches in seconds.

References

1. Arditi et al. Refusal in Language Models Is Mediated by a Single Direction. NeurIPS 2024.
2. Yona et al. In-Context Representation Hijacking. 2025. Preprint.
3. Lindsey. Emergent Introspective Awareness in Large Language Models. 2025.
4. Plunkett et al. Self-Interpretability: LLMs Can Describe Complex Internal Processes That Drive Their Decisions. 2025. Preprint.
5. Vovk, Gammerman, Shafer. Algorithmic Learning in a Random World. Springer, 2005.
6. Angelopoulos, Bates. Conformal Prediction: A Gentle Introduction. Foundations and Trends in Machine Learning, 2023.
7. Bates et al. Testing for Outliers with Conformal p-values. Annals of Statistics, 2023.