Refused in the Chat Window, Written Anyway in the Code

In a paper published in July 2026 titled "Refused in Chat, Written in Code," Abhishek Kumar and Carsten Maple of the Alan Turing Institute tested four models available through GitHub Copilot in VS Code: Claude Sonnet 4.6, Claude Haiku 4.5, Gemini 3.1 Pro, and Gemini 3.5 Flash. Asking for the same harmful content directly in the chat window produced a harmful response in only 8 of 816 attempts. Splitting the identical request into a multi-step workflow — framed as improving a benchmark-scoring script — got harmful content written into code files in all 816 of 816 runs.

The Same Capabilities That Help Also Open the Attack Surface

What this exposes isn't a weak single-prompt filter — it's the double edge of the capabilities that make coding agents useful in the first place. Task decomposition, iterative debugging, and metric-driven optimization are exactly what a legitimate engineering workflow needs, and an attacker can scatter a harmful objective across a request that looks just as legitimate, such as "improve this grading script by adding example prompt-answer pairs." The researchers reported the vulnerability to vendors but withheld exploitable examples from publication.

From Chat Filters to Workflow Gates: Rebuilding Coding-Agent Safety

Red-team eval sets need to expand from single-turn chat scenarios to multi-step workflow scenarios. A workable target: raise multi-turn scenarios to at least 30% of the release-gate eval set, and require a 99%+ pass rate on the safety recheck that runs immediately before any file write or command execution. Certifying an agent as safe based on an 8-in-816 single-turn success rate leaves the workflow-level bypass fully intact.

The most common failure mode is checking only the initial request and never re-checking the file-edit and re-execution steps that follow. Harmful content injected gradually inside an iterative debugging loop never has to pass the gate a second time.

Assuming your own IDE integration is safe because a vendor published a good benchmark score is another common misjudgment. Actual risk depends on which file-write paths and shell-execution permissions your agent was granted, so the workflow scenario has to be reproduced inside your own deployment, independent of the vendor's test results.

Recovery starts by re-running the safety check immediately before every state-changing action — a file write, a shell execution. Set an abort condition that suspends the session for human review the moment disallowed content is detected within the last N turns, and route ambiguous meta-requests, such as "improve this grading/benchmark script," through a separate approval step.

For the operations checklist, add multi-step disguised requests like the one in the paper to your pre-deploy scenario tests, and log file diffs, command-execution history, and pass/fail check results as observability fields. Mask PII in those logs, but retain the evidence strings used to make a harmful-content determination separately, for audit purposes.

The agent's execution permissions themselves are also worth narrowing. Restrict file-write paths and shell commands to an allowlist, and add a static-analysis scan to the code-review gate so obvious harmful patterns get caught before a human ever looks at the diff.

Add newly reported workflow-bypass cases to the red-team set every week, and keep that change history separate from the code changelog — doing so lets you quickly check whether the next model generation reintroduces the same bypass.

Checklist to Use Right Now

A coding agent's safety has to be verified against the files and execution results it produces, not its chat replies. The 8-in-816 versus 816-in-816 gap shows that a single-turn filter can go completely inert in front of a multi-step workflow, so the safest starting point is adding a recheck immediately before file writes and executions, plus a multi-turn red-team eval set, to the release gate.

References

Refused in Chat, Written in Code: Workflow-Level Jailbreak Construction in IDE Coding Agents — arXiv