Defining the self-improvement loop
Recursive self-improvement (RSI) is a structure in which an agent observes its own prompts, policies, and tool usage, produces improvement proposals, and applies them again. Approaching it only because performance goes up is risky. Left uncontrolled, self-improvement tends to optimize metrics rather than quality.
So the core of RSI is not the ability to improve but the ability to verify and revert improvements. You must nail down, in advance, what can and cannot be changed by the agent itself.
A three-stage propose-verify-approve structure
A safe RSI loop separates proposal, verification, and approval. The improvement agent only proposes changes and never applies them directly. Verification compares the previous and new versions on a fixed evaluation set, and approval lets a human or a separate conservative judge decide whether to deploy. Not letting the same actor both propose and apply is the first guard against runaway behavior.
Full guide: from planning to operations
In planning, define improvement success as numbers. For example, promote only when an eval-set gain of 2% or more, zero safety-rule violations, and zero regressions are all satisfied at once. Improvements that raise a single metric often sacrifice others, so always check multiple metrics together. Also restrict the improvement agent's reach to prompts and routing rules, and lock the evaluation criteria and safety policy themselves so they cannot be self-edited.
The most dangerous failure pattern is eval-set overfitting. As self-improvement repeats, the agent gets finely tuned to pass only the eval set and may get worse on real user inputs. To prevent this, split the eval set into a fixed set and a rotating set, and inject fresh real cases into the rotating set periodically. If a proposal improves only on the fixed set and worsens on the rotating set, reject it immediately. For recovery, keep every change versioned so a single command rolls back to the last stable version.
On the operations checklist, state the limits of self-improvement: a cap on consecutive promotions, a cap on daily changes, and a circuit breaker that automatically halts improvement when metrics wobble. Bundling the change proposal, evaluation result, approver, and apply time into one audit log makes post-hoc tracing easy. In particular, if even one safety-policy violation is found, the whole proposal must be discarded regardless of score gains.
The continuous improvement loop reviews both promoted and rejected changes weekly. Seeing which proposal types get rejected often gives grounds to tune the improvement agent's own prompt. That is, RSI is sustainable only when run as a double loop that improves not just the agent but the improvement process itself.
Key takeaways
In short, RSI safety comes from separating proposal and application, preventing eval-set overfitting, and one-step rollback. With simultaneous multi-metric thresholds, a locked safety policy, and a circuit breaker, self-improvement accumulates quality without runaway. A double loop that periodically inspects the improvement process itself creates sustainability.