r/ContextEngineering • u/MattAlreadyHere • 3h ago
Does your context compression survive the prompt cache? I built a deterministic compressor for Claude Code and Codex
Context question first, repo at the end.
We spent a while compressing coding-agent context and hit something that isn't obvious until you look at a bill: on agent traffic about 93% of input tokens are cache reads, at roughly a tenth of the fresh price. So if your compressor turns the same tool output into slightly different bytes next turn, the prefix is dead and the whole conversation gets re-priced. You compress harder and pay more.
That one fact decided everything else for us.
What we ended up with
Same input, same bytes, every turn. No clock, no randomness, and it never looks at the rest of the conversation. Scoring a tool result against the question that prompted it does compress better - and it also compresses differently on every turn that result survives into. We gave that up on purpose.
It de-duplicates, it doesn't summarise. The model always keeps at least one real copy of every distinct line. Repeats fold into a marker like [×37 ref:k_Qm9v…]. The handle is a hash of the content that was removed, so the same fragment gets the same handle on any machine, in any later run. The proxy hands the model a retrieve tool and runs that tool itself, so it works even if your client has never heard of us.
Only machine-generated tool output over ~512 chars. System, developer and assistant messages are never touched, and neither are blocks carrying cache breakpoints.
No ML in the request path. Detection is fifteen rules in order over the first megabyte, and source code goes through nine tree-sitter grammars compiled into the binary. Nothing gets downloaded at runtime, nothing varies between machines.
Fails open. If the result isn't smaller it gets thrown away, and any error forwards the request unchanged.
What it bought
All measured on real provider bills with the cache discount already in, not on removed tokens priced at list rate.
Codex on GPT-5.5, Terminal-Bench 2.1, per run over the 87 tasks every arm finished: 62.8% of eligible tool output removed in a replay of 6,255 real outputs (34.2M chars), $107.89 -> $83.40, input 96.1M -> 72.2M tokens, solve rate 82.2% -> 81.1% (not statistically significant). Requests over 200K tokens went from 1.9% to 0.13%, and that mattered to me more than the money.
Claude Code on Opus 5.5 turned out to be a different problem, and this is the part I'd like this sub's take on. It marks its conversation with explicit cache breakpoints, so 92-95% of billed prompt is already cache reads and there's barely anything left to compress - tail-only compression removes 4.7-8.0% of what the engine even sees. But a cache write there costs 25x a read, and a default entry lives five minutes. A training run or a long build idles past that, and when the agent comes back the whole prefix gets written again at full price: 927K tokens per run.
So compression stopped being the thing to fix. We keep the entry alive instead - re-send the last request body every 4.5 minutes with max_tokens: 1, stop at the agent's next request or after 11 pings. Tokens lost to cache expiry dropped 99%, the Anthropic bill fell 20% ($35.35 -> $28.30 per run, solve rate 89.9% -> 90.6%), and that one mechanism was worth about 11% of it. Adaptive one-hour TTL on top: always using 1h entries actually cost 2.3% more, switching only after a conversation's first long idle gap saved 7.7%.
Where it does nothing. Short reasoning tasks. The median output there is 270 chars, under our floor, and the bill moved 8%. Prose is left whole on purpose. The wins sit in ML training, builds from source and data pipelines.
One thing worth knowing if you measure this yourself: gross removal over-counts badly, because every turn re-compresses the same growing history and credits the same bytes again. And every retrieval puts the fragment back in the prompt, so net savings can go negative.
MIT, one Rust binary. kompressor wrap codex, kompressor wrap claude, or kompressor serve for anything speaking the OpenAI or Anthropic API:
https://github.com/hystax/optscale-kompressor
Write-ups with the task-by-task tables:
Codex: https://optscale.ai/blog/context-compression-coding-agents
Claude Code: https://optscale.ai/blog/claude-code-prompt-caching
What I'm curious about: is anyone here compressing query-aware, and have you checked what it does to your cache hit rate? That's the tradeoff I keep going back and forth on.
And if it turns out useful on your setup, stars are welcome.










