r/Python • u/Stunning-Sherbet1853 • 3d ago
Discussion 112 CodeQL alerts on my python app, 14 were real
HI, i run a long-lived python backend on windows and mac!
it only listens on 127.0.0.1 and almost every route needs an access token. last week i turned on the default CodeQL scan for the first time and got 112 alerts ( 112 → 1 critical, 87 high, 24 medium).
i went through them one by one with a single question: can this value come from text someone else wrote (a web page, a downloaded file, model output)? where an alert described an attack input, i actually fed it in and timed it.
the first one that surprised me was the regex that checks whether a downloaded file is html. `<!--\` followed by \`--><!--` 28 times kept it busy for 27 seconds. two more regexes that read text we didn't write did the same thing (22s and 11s). six alerts in that family were real. after the fix the html check runs 5,000 times in 0.003s.
the one i'm most embarrassed about: to see whether a login had finished, i checked whether the url contained the site name. so `evil.example/?next=naver.com` passed as a finished login, and `evilnaver.com` counted as the same site. seven alerts across different files. now one small helper pulls out the hostname and checks "equal, or ends with `.domain`", and all seven places use it. the last real one was a script tag filter that let `</script foo>` through.
every fix got a test, and i checked that the test fails when the fix is taken out.
the other 98 i dismissed with a one-line reason each. 42 of them were in my own tests, lines that just write down the url a test expects. most of the rest were path injection or stack trace exposure on routes that need the token anyway. i only traced two of the path ones all the way, so i might be wrong about a couple.
the same "incomplete url substring" rule was noise all 42 times in tests and a real hole in the login check. asking where the value comes from got me through it faster than reading the alert type.
how do you deal with this on a localhost-only or solo project? i dismissed with reasons but i'm not sure that scales.
1
u/Delicious_Break5937 3d ago
asking where the value can come from is a good way to sort them. did you do anything to stop new code from doing the url substring check again, like a lint rule, or just the helper
0
u/Stunning-Sherbet1853 3d ago
just the helper and its tests for now, nothing stops new code from doing `"google.com" in url` again. a lint rule would also fire on all 42 test lines, so i'd have to exclude tests. have you found one that doesn't?
0
u/mhmd2520 2d ago
You might not need a separate lint rule. CodeQL already is the rule. The query that caught your 7 (py/incomplete-url-substring-sanitization) will catch the next
google.comin url too, as long as it runs on pull requests and not just as a one-off scan. On a PR it only annotates alerts whose lines are all in the diff, so you see new ones without the old 112 coming back.For the test noise, you've got two options. GitHub already labels alerts in test files as "Test", and the "Only alerts in application code" filter (or autofilter:true in the search box) hides them. If you'd rather tests weren't analysed at all, put paths-ignore with your tests dir in a codeql-config.yml. That works for Python because it isn't compiled. As far as I can tell the config file also works with default setup now through a repo property, so you don't have to switch to an advanced workflow just for this.
The catch with paths-ignore is that it's all or nothing per path. To keep tests out of one query only, you'd need a second tool.
If you want something that fails before you even push, a small semgrep rule does it. Match
"$S" in $URL, add a metavariable-regex on $S so it only fires when the string looks like a domain, and use the rule's own paths: exclude for tests. That way the exclusion is per rule, not global. Ruff won't help here because it doesn't support custom rules yet.On dismiss-with-reason scaling: I'd move as many as possible out of dismissals and into config. A dismissal is a click in the UI that nobody reviews in a diff. A paths-ignore line or a query-filters exclude by rule id sits in the repo with a commit message explaining why.
1
u/Stunning-Sherbet1853 2d ago
I appreciate your kindness :)
It already runs on every PR here, the python analysis shows up in the checks on the last one. So the helper plus codeql on PRs probably covers "someone writes it again", and semgrep would only buy me failing before the push.
Right now all 98 are UI dismissals. Moving the 42 test ones into paths-ignore and keeping the rest as dismissals feels like the honest split, Since the tests folder is the only place where the rule is wrong for every line in it.
0
u/mhmd2520 2d ago
One correction to what I said before. The config file on default setup works through a custom repository property, and GitHub's docs say those have to be created at the organization level. So if this repo is under your personal account and not an org, I don't think that route is open to you.
In that case paths-ignore means turning default setup off and switching to advanced setup, so the workflow's init step points at your config file. You can't run both side by side, because default setup blocks advanced uploads while it's on.
If the repo is in an org, you just set github-codeql-config-file on the repo to the yml path, and default setup merges it in on the next scan.
The split makes sense to me too. Tests are the only folder where that rule is wrong for every line.
0
u/Stunning-Sherbet1853 1d ago
The repo sits under an org, so the config-file route should be open without turning default setup off. Thanks!
0
u/Delicious_Break5937 2d ago
the semgrep rule in the reply below is what i'd do, its paths: exclude keeps tests out for just that rule
1
u/AleDev99 3d ago
14 real out of 112 is honestly a pretty normal ratio for a first CodeQL run. Two things that helped me keep it manageable:
Put the exclusions in a codeql-config.yml (query-filters, or paths-ignore for tests and vendored code) instead of dismissing alerts one by one in the UI. Dismissals are invisible to the next person; the config is reviewed in a PR and you can leave a comment on why each rule is off.
Be careful with "it only listens on 127.0.0.1" as the reason to dismiss things. Any web page open in a browser on that machine can still send requests to localhost, and with DNS rebinding it can even read the responses. If the token is sent as a header (not a cookie) and you reject requests whose Host header isn't 127.0.0.1/localhost, most of that goes away, but I wouldn't dismiss path-injection or SSRF alerts just because of the bind address.
Out of curiosity, which rule were most of the false positives? A common one is py/path-injection on paths that are already normalized.
0
u/Stunning-Sherbet1853 2d ago
by count it was the url substring rule in tests, 42 of them. path-injection came second with 23 and none were real, but i only traced two all the way, so your normalized-path guess could cover most of the rest.
the token is a custom header, not a cookie, so a rebound page can't attach it. what i don't do is check Host, and a handful of routes skip the token (health and the pairing start). that's the part of your comment i'm keeping.
0
u/AleDev99 2d ago
makes sense, the custom header kills the rebinding angle for the token routes. for the ones that skip it I'd just reject any Host that isn't localhost or 127.0.0.1, it's a few lines and covers health and pairing start too. good to know the url substring rule was mostly test noise, that matches what I usually see.
0
u/SteveJeffersonBots 2d ago
For the "dismiss-with-reason doesn't scale" worry, batch dismissals into categories instead of reasons. Stamp every closed alert with one of three buckets: test-code, guarded (token-gated routes only you on the machine can hit), and real input surface. You end up with a handful of bucket counts instead of 112 individual reasons, and you can actually see which category is growing.
That smaller set is next time's working set. The automation that turns those buckets into something you don't re-review is already covered: a codeql-config.yml with paths-ignore for tests and query-filters for chronically noisy rules on your token-gated routes, plus CodeQL on PRs so only diff-intersecting alerts surface. Config commits are reviewed in PRs while UI dismissals disappear, and for solo occasional runs a single commit message per bucket noting which rules you dismissed and why is the commit-message equivalent of your per-alert reason — something future-you can actually audit.
One concrete thing for your specific case: for the six ReDoS regexes, add a small property check next to each one asserting re.search on a fixed catastrophic input returns within a threshold (time the call in an assert or a test). That way the "text we didn't write can hang us" pattern is caught by a test without depending on CodeQL, which is notoriously spotty on ReDoS.
Smallest set of automations that actually reduces re-review: codeql-config.yml (paths-ignore for tests, query-filters for noisy rules), CodeQL on PRs for diff-scoping, and a tiny semgrep rule for the "$S in $URL" pattern with a per-rule paths: exclude for tests. That handles the substring-check lint problem Delicious_Break5937 surfaced per-rule without hitting your 42 test lines again.
13
u/Financial-Grass6753 3d ago
in general, you just use industry-wide solutions in 99% of cases (Pydantic, FastAPI, eh whatever you need) and just follow their advice. Reinventing the wheel comes at a cost of security and maintenance and you don't really won't to follow this path unless absolutely needed to