r/OpenSourceAI • u/AsianKidTV • 2d ago
I built open-source tooling to evaluate agent setups and see where agents and subagents waste time and money
I run Claude Code and Codex all day, often as multi-agent workflows with dozens of subagents. I could see each session but never the whole picture: what a run actually cost, where agents sat idle, whether a change to my setup helped. So I built coding.sh, an open-source tool for evaluating agent setups, and pointed it at my own runs.
Experiments: does that change actually help? I A/B tested a popular "be concise" plugin: 10 tasks, 6 runs each, paired, with 95% intervals.
Replies: 17% shorter
Output tokens: −8%
Cost per run: +11% [+7%, +15%]
Pass rate and clarity: no change
The plugin adds its style guide to every request, and that costs more than the shorter output saves. coding.sh compare and exp run your own tasks with and without a change (a rule, a skill, a plugin, a model) and tell you whether the difference is real or noise.
Workflows and subagents Across 57 workflow runs (666 subagents, ~$2,500 API-equivalent):
Broken prompt caches cost $106 extra, about 4% of the spend.
41 of those breaks were agents sitting idle at a barrier, waiting for siblings, until their cache expired. That's $21 for doing nothing.
40 subagents failed outright.
coding.sh workflows stats shows the critical path, concurrency and idle time at barriers. The viewer draws each run as a timeline, one bar per agent.
Your own sessions Mining 3,600 of my sessions turned up a find-and-replace typo in my AGENTS.md that sent Codex looking for a folder that doesn't exist, in 92 sessions.
What it is
Hooks into Claude Code, Codex, Gemini CLI, OpenCode, Copilot CLI, Aider and more
Stores prompts, tool calls, diffs, test runs, tokens and cost in SQLite on your machine
coding.sh improvesays what to fix first and whether your last AGENTS.md change helpedMCP server, so your agent can look up its own past sessions
Free, MIT, no account, nothing leaves your machine. npx coding.sh to try it. GitHub: https://github.com/jackulau/coding.sh