Observation Is Not Defense
Watching action logs stream across a dashboard is a different layer from actually blocking a dangerous action. Roughly 88% of pilot-stage agents are reported to fail the transition to production, and a large share of that traces back to the absence of enforcement gates. Treat pass rate as a metric, but anchor your baseline on blocked-action counts and zero violations, not on observed events.
Make Five-Layer Defense the Standard
Design input validation, tool/action gating, output validation, HITL approval, and offline evals as five independent chokepoints. Only a defense-in-depth structure, where clearing one layer still triggers revalidation at the next, prevents a single prompt bypass from breaking everything. Narrow each layer's interface so it returns just three outcomes: pass, block, or escalate.
HITL Is Not a Cure-All
The assumption that routing to a human makes things safe is dangerous. In measured settings, humans actually catch only 9-26% of bad actions. When approvals pile up, reviewers drift toward bulk-approving, so HITL should not be the last line of defense. Route to it only for the small set of cases the automated gates judge ambiguous.
A Full-Page Guide: From Planning to Operations
Fix target metrics as numbers during planning. For example, specify as release criteria: automated-gate block recall of 95% or higher, HITL routing on 5% or less of total actions, approval-gate p95 latency at or under 800ms, and zero production violations. Because the EU AI Act Article 14 human-oversight obligation takes effect on 2026-08-02, work backward from that date for high-risk classified systems and lock down oversight points and log-retention policy first.
Split failure patterns into per-type recovery branches. If a tool-call parameter falls outside the schema, retry once, then block. If output validation catches PII leakage or a policy violation, substitute a safe truncated response and discard the original. Retry external API timeouts with exponential backoff up to three times, but route irreversible actions such as fund transfers or deletions straight to human confirmation with no retry. If three blocks accumulate in one session, halt the session and escalate.
Start the operations checklist with standard logging. Record request ID, gate verdict, verdict rationale, and latency as structured logs for every action, and mask PII such as email, phone, and national ID before writing the log. Aggregate each gate's block rate and false-positive rate daily, and retune the rules whenever the false-positive rate exceeds 5%. Maintain the list of irreversible actions separately and review it quarterly.
Close the continuous-improvement loop with evals. Accumulate blocked and escalated production cases into a regression set, and stop any deployment whose pre-release offline evals drop block recall below the baseline. Fold new bypass patterns into the set weekly so the gates track the real threat distribution.
Executive Summary
Observation alone will not avoid the 88% failure rate. Enforce the five layers—input, tool/action, output, HITL, and evals—as independent chokepoints, and place HITL only on a few cases given its low 9-26% catch rate. Fix your target metrics (95% block recall, zero violations, p95 800ms) as release criteria, and lock down oversight points and standard logging before Article 14 takes effect on 2026-08-02.