It Adds Up to $15,000 a Day: Designing a Crisis and Self-Harm Escalation Gate for the Callbot Loop

The Law Started Requiring a Gate First

New York's AI Companion Models law, in effect since November 5, 2025, requires operators to maintain a protocol that detects expressions of suicidal ideation or self-harm and refers users to crisis services. Violations can draw civil penalties of up to $15,000 per day from the state attorney general, and operators must remind users every three hours that they are talking to a bot, not a human, for as long as the conversation continues. The requirement isn't limited to text chatbots — in a voice callbot loop, the ASR transcript is the text the detector actually sees, so transcription lag or quality drops directly undermine the gate's reliability.

Split the Detection Layer from the Escalation Layer

Anthropic's Usage Policy requires that high-risk mental-health content be reviewed by a qualified human before it reaches the individual. Carried into a callbot loop, that principle means splitting two layers: a detection layer that runs a classifier over every turn's transcript, and an escalation layer that hands off to a human agent or crisis line. The two layers need separate thresholds and separate failure-recovery paths — merge them into one module and neither can be tuned independently.

False Positives and Misses Don't Share a Metric

An over-sensitive classifier piles up alerts until agents start tuning them out; an under-sensitive one lets a real crisis pass through untouched. Because the two failure modes recover differently, recall and false-positive rate need to be tracked as separate targets, not one blended score.

From Design to Handoff: Operating the Crisis Escalation Gate

Fix the numbers before writing any code. Set the crisis classifier's recall target at 95% or higher, cap the false-positive rate at 2%, and require that handoff to a human agent complete within 30 seconds of an escalation trigger. Where the law has already fixed a number — New York's three-hour re-notification rule, for instance — move it straight into a timer in the code rather than renegotiating it.

The first failure pattern shows up when ASR transcript confidence is low and the keyword or classifier layer reacts late. When transcript confidence drops below threshold, don't rely on text matching alone — cross-check secondary voice signals such as pause length and speech rate to cut down on missed detections.

The second is an over-sensitive classifier flagging figurative language in ordinary conversation as a crisis, building alert fatigue. Adding a second-pass check — re-reading the last three turns of context after the first classifier fires — cuts the cases where agents start ignoring real alerts.

The third is escalation stalling when the human-agent queue is saturated. If handoff to a person doesn't complete within 30 seconds, the code needs a hard-coded fallback that transfers directly to a pre-registered crisis line (988 in the US, 1393 in Korea) instead of waiting.

Before deployment, measure recall and false-positive rate against a synthetic set of crisis-expression scenarios, and separately check for false positives against ordinary conversation recordings. Logs should capture session ID, trigger timestamp, handoff duration, and final outcome — but never the raw triggering utterance; store only a category and a hash to keep PII out of the log.

Each week, roll flagged false positives and misses back into the classifier's retraining set, and check legislative developments every quarter — if the re-notification interval or handoff bar changes, the timer values in the code are the first thing to update.

What to Check Before Handoff

A crisis detection gate is never just one classifier. Split detection from escalation, hard-code the 95% recall, 2% false-positive, and 30-second handoff numbers, and back every automatic escalation with a direct line to a crisis service when the human handoff stalls — that's what it takes to meet the bar New York's law has already set.

References

Usage Policy — Anthropic

Assembly Bill A6767 — The New York State Senate