What to Merge, and When

Compressing a streaming log of turns and actions into a roughly 4k-token budget comes down to two decisions: a merge rule that decides whether an incoming chunk joins an existing cluster ("atom") or starts a new one (a cosine threshold τ), and a placement rule for how retrieved clusters land in the prompt (grouped under cluster headers vs. flattened into a ranked list). A recent arXiv preprint (2609.04915) isolates these two factors with a 2×3 factorial design and validates them on AMA-Bench and RealMem.

The result cuts against the usual instinct. The retrieval-ranking rule's own contribution disappeared once clustering and packing were held fixed (p=.40), while the merge rule explained +5.7pp over K-Means and cluster-grouped packing added +5.02±1.00pp over a flat list. Teams that spend their first engineering cycle polishing the retrieval algorithm are, on this data, optimizing the wrong lever.

Where the Gain Runs Out

At a 4k-token budget, the method reproduced 83% of Full-Context quality at 32% of its tokens (0.311/0.373). Averaged over four seeds it beat Online K-Means by +3.5-6.0pp and Streaming-PCA (Oja) by +4.6-9.5pp. But the domain-level breakdown tells a different story: it won on 4 of 6 domains (OPENWORLD-QA, WEB, TEXT2SQL, SOFTWARE), while EMBODIED-AI (-2.68pp, p=.042) and Game favored K-Means or showed no difference.

That reversal sits only in the appendix — the abstract and conclusion don't surface it. The authors themselves note the compression gain shows up only in streams with "sufficiently repeated latent structure," and the two losing domains read as exactly the cases that fall outside that condition. Judging rollout readiness off a single benchmark average misses this boundary entirely.

From Design to Operations: A Compact-Memory Clustering Rollout Gate

(a) Planning starts by pinning the target numbers before any code ships: at a 2-5k token budget, quality at or above 80% of Full-Context, a per-domain delta of at least +2pp over the incumbent retrieval method (K-Means, BM25-RAG, etc.), and 100% pre-rollout evaluation coverage across every production domain. Exclude any segment that can already afford 12k+ tokens — comparing it against a compressed baseline distorts the whole picture.

(b) Four failure patterns recur. First, budgeting engineering time for retrieval-ranking tuning while leaving the merge threshold and packing scheme at their defaults. Second, rolling out to every domain off a single benchmark average and missing the quality drop in domains like EMBODIED-AI or Game. Third, validating against a single embedding model and a single judge model, so the result doesn't reproduce once you swap either stack. Fourth, failing to screen for data characteristics — like those in LoCoMo or LongMemEval — where the compressed-memory advantage weakens or vanishes.

(b') When a domain's pre-rollout delta comes back negative, route it automatically to Full-Context or the existing K-Means path instead of compressed memory. Implement that as a routing branch, not a deployment rollback, so adding a new domain later means adding one row to the same table.

(c) The operations checklist puts merge-threshold tuning ahead of retrieval-rule work, runs an A/B between cluster-grouped packing and flat listing to measure the gap directly, and builds a holdout set of at least six representative domains before launch to catch reversals early. Logs need a domain tag, token usage, merge threshold, and packing mode — without them, the next quarter's comparison has nothing to compare against.

(d) Re-measure the per-domain delta table every quarter and check whether any domain entered or left the fallback list. Whenever the embedding model or judge model changes, treat the entire table as due for revalidation — an advantage measured on one stack is not guaranteed to survive a swap to another.

Takeaways You Can Ship

Compact-memory clustering can hit 83% of Full-Context quality at 32% of the tokens in the 2-5k range, but most of that advantage comes from the merge threshold and cluster packing, not the retrieval rule. Skip the per-domain pre-rollout check and the negative-delta fallback, and domains like EMBODIED-AI or Game can end up worse off than before.

References

Compact-Memory LLM Agents via Online Max-Member Clustering and Atom-Aware Packing — arXiv

sunny34.com Research: RSM-full compact-memory agent review — 83% quality at 32% tokens