disclosure up front: i built the thing i'm describing. not selling anything, it's free — posting because i want to know if the approach is useful or if i've built something nobody needs.
been lurking here a while, finally have something worth posting.
context: about 4 months ago someone in a thread mentioned that at their company, >90% of CI failures had nothing to do with the change being tested. eng team went 100 → 300 people, CI couldn't scale, and the policy was you had to do root cause analysis before retriggering. so everyone came in at 9am to three red builds and spent the morning reading logs to figure out if it was even their fault. half the team eventually got laid off. that story lived in my head rent free.
my first attempt was a bot that read the failure and tried to auto-fix the code. built the whole thing. then i reread that 90% stat and realized a bot that patches the code under test is patching the wrong file 9 times out of 10. threw it out.
what i built instead just answers one question: is this red build actually your fault?
the rules are boring on purpose:
failing file isn't in your diff at all → probably not you
dependency version changed between last green run and now, and your PR didn't touch the lockfile → blame the version
failure is in the install step before any of your code ran → dependency
runner OOM'd / network died / "lost communication with the server" → infra
assertion failed in a file your PR actually modified → yeah, that one's yours
can't tell confidently → says UNKNOWN and shows you the log instead of guessing
then for the ones that are provably not your code, it reruns them automatically. and if the rerun passes, it says so. if the rerun fails identically, it says that too and escalates — because a failure that reproduces on the same commit isn't flaky, it's real.
things i deliberately didn't do, mostly from arguing with people in threads here:
it never touches your code. only comments and reruns. can't delete an assertion, widen a timeout, or add a retry to make things green. the "pipeline is green but the bug shipped" failure mode is worse than a red build
no LLM in the classification. it's rules you can read and audit. an LLM that sounds confident and an LLM that's correct look identical in a PR comment
UNKNOWN is a first-class answer. saying "i don't know, here's the log" beats a confident wrong verdict, and one wrong "not your fault" on something that WAS your fault would kill any trust in it forever
honest limits: full test-level classification is pytest-only right now (js/go would need their own parsers). infra + dependency + the rerun logic work on any github actions repo. and it's only as good as its rules — real world workflows are messier than my test cases, i fully expect edge cases i haven't seen.
i built a repo with 5 seeded PRs, one per failure class, so you can see it call each one live instead of taking my word: https://github.com/chaturvediabhinav692-lgtm/koredex-gauntlet/pulls
it's a github app, free while i'm figuring out if it's actually useful to anyone but me. mostly posting because i want to know if the "is the failing file even in your diff" check would actually save you time, or if that's obvious to everyone and i've built something nobody needs.