A score rise is not improvement
Just because a self-improvement change raised the evaluation score does not guarantee it is a real improvement. With a small sample, random variation looks like improvement, and applying several changes at once hides which one had the effect. So improvement needs judgment.
The key is to isolate changes and compare them one at a time, then confirm with statistics that the difference is not chance.
Experiment tracking and significance
Experiment tracking records which change was applied when and to whom, attributing results to changes precisely. Statistical significance confirms whether the observed difference is large enough not to be explained by sampling chance. Without both, the illusion of improvement ships as-is.
Full guide: from planning to operations
In planning, define judgment criteria as numbers. For example, fix the minimum sample size, a 5% significance level, the minimum detectable effect size, and a maximum experiment duration in advance. Setting criteria after the fact leads to interpreting toward a desired conclusion, so fix success conditions before the experiment starts. When viewing several metrics at once, apply multiple-comparison correction to filter out chance significance.
A common failure pattern is early peeking. Repeatedly checking results before the experiment ends and stopping when favorable spikes false positives. To prevent this, defer judgment until a fixed sample is reached, or use sequential testing that allows repeated checks while statistically controlling the early-stop criterion. For recovery, if real metrics after deployment deviate greatly from the experiment's prediction, auto-roll back and re-examine the experiment design.
On the operations checklist, keep the full experiment history. Record the hypothesis, sample size, significance level, result, judgment, and deployment status in standard fields for reproducibility. Use per-experiment sample count, estimated false-positive rate, early-stop ratio, and post-deployment prediction error as observability fields. Keep masking rules so experiment data contains no personal information.
The continuous improvement loop analyzes experiments judged significant but ineffective after deployment weekly. For types with a large prediction-reality gap, reinforce sample criteria or metric definitions. The experiment system should be a living process that keeps hardening criteria to reflect wrong judgments, not a fixed procedure.
Key takeaways
In short, a score rise and real improvement differ. Isolate changes and compare by A/B, fix significance level and sample criteria in advance, and control early peeking with sequential testing. Observe post-deployment prediction error to keep hardening experiment criteria so the illusion of improvement does not reach deployment.