Offline evaluation alone is not enough
Even after passing offline evaluation before deployment, quality appears different under the real user distribution. Online evaluation samples production traffic and scores it in real time, so when the score drops you immediately get a signal that something changed. For self-changing systems like automatic loops, this real-time signal matters especially.
The key is to monitor quality continuously without a human reviewing every interaction.
A background judge
A background LLM judge asynchronously samples a fraction of daily sessions and grades them against the same rubric used offline. This produces a continuous quality dashboard and scores production traces in real time to catch the moment scores drop. Offline and online using the same rubric is the key to consistency.
Full guide: from planning to operations. In planning, define online-evaluation goals as numbers. For example, set the daily sampling rate, an alert threshold on the quality-score moving average, and a detection-to-alert delay ceiling. Sample not only randomly but stratified toward high-risk segments so you do not miss quality on important slices. Apply the bias corrections covered earlier to the online judge so the judge itself does not distort the signal.
A common failure pattern is segment regression hiding behind the overall average. Average quality holds while a specific user group or request type drops sharply. To prevent this, decompose scores by segment and fire an alert when a segment breaches a threshold. For recovery, when online scores drop, nominate recent changes as candidates and roll back, and add that segment's failure cases into the offline eval set. A large online-offline gap signals the offline eval set does not reflect reality.
On the operations checklist, keep scoring reproducible. Log which session was scored when with which rubric version, alongside the judge version, in standard fields. As observability fields, use sampling coverage, per-segment scores, online-offline gap, and alert frequency. Always apply personal-data masking when using production data for scoring.
The continuous improvement loop absorbs regressions detected only online into the offline eval set weekly. Add failure types online catches often to the pre-deployment gate too so the same problem does not ship again. Online evaluation should be a system that evolves with offline evaluation by absorbing new regression types, not a fixed dashboard.
Key takeaways
In short, offline evaluation alone cannot guarantee production quality. Sample traffic with a background judge and score in real time against the offline rubric, detect regressions by segment, and absorb online failures into the offline eval set. Observe the online-offline gap so the eval set keeps reflecting reality.