r/Python • u/Kiuhnm • Mar 07 '24
Discussion Tests should have 100% coverage
No, I'm not saying what you probably think I am. Keep reading: I'll be brief.
Here's what I found today in one of the tests:
for bad_fp in _bad_fps:
with pytest.raises(BadArgumentError):
follow_path(bad_fp, '.')
follow_path('.', bad_fp)The correct code should've been
for bad_fp in _bad_fps:
with pytest.raises(BadArgumentError):
follow_path(bad_fp, '.')
with pytest.raises(BadArgumentError):
follow_path('.', bad_fp)Is that a stupid mistake? Yes, it is, but stupid mistakes do happen, unfortunately.
How did I catch it? The coverage of the testing file wasn't 100% like it usually is! I got suspicious and I saw that there was a glaring red line (vscode, coverage gutter) on the second call to follow_path.
P.S. In case you're wondering, the code uses a custom hierarchy of exceptions instead of reusing the standard ones. The code is internal and we like it that way :)
32
u/GraphicH Mar 07 '24
High percent coverage is much more critical in dynamically typed / scripting languages than in compiled languages where the compiler watches your back. Its one of the hardest things for people who worked primarily in compiled languages to get their head around when writing "production grade" python code. Its also why I think the testing and linting utilities for these languages seem a lot more superior to the stuff you find for compiled languages. And yeah if you have test code that never fires, that begs the question, why is it written, we do coverage reports on test code as much as production code.
23
u/lukewhale Mar 07 '24
Yall writing tests ? đđđ /s
9
u/badjayplaness Mar 08 '24
I used to only practice the âpush to masterâ driven development method. Apparently theyâre other methods.
4
1
u/bartenderandthethief Mar 08 '24
Question, do you think it is good practice to only have one line within the pytest raises context?
2
u/Kiuhnm Mar 09 '24
It is according to https://docs.astral.sh/ruff/rules/pytest-raises-with-multiple-statements/
It certainly helps avoid these kinds of mistakes, but looking at the coverage is more general, IMHO.
1
u/ArtOfWarfare Mar 08 '24
I donât think your actual code had 100% branch coverage. If it did, then that extra line in your test is superfluous and should just be removed.
Also, check your mutation coverage. If youâve not seen it before, it tends to expose 50% of tests as being useless garbage.
1
u/Kiuhnm Mar 08 '24 edited Mar 08 '24
There's the testing code and the tested code. Usually, code coverage helps to detect untested parts of the tested code.
Here I'm saying that code coverage of the testing code helped me detect parts of the testing code (as opposed to the tested code) that weren't being executed.
The code
with pytest.raises(BadArgumentError): follow_path(bad_fp, '.') follow_path('.', bad_fp)is wrong because the first call to
follow_pathis expected to throw, but that way, the second call is never executed and tested. One needs to have a separatewithstatement for the second call.1
u/ArtOfWarfare Mar 08 '24
Yes, I understand how your test is wrong, but Iâm saying your actual code either isnât actually fully covered (youâve missed branches) or it is fully covered and this incorrect test is just redundant.
But also, check your mutation coverage. Thatâs far more useful than line coverage. Itâll expose all the bugs that your test suite donât. And if youâve never checked your mutation coverage before, youâll probably find your tests donât even find half the potential bugs in your code despite having full line and branch coverage.
1
u/Kiuhnm Mar 08 '24
I don't have and I don't try to have 100% coverage of my code. I was just talking about the coverage of my testing code. Now I understand what you meant.
1
u/foarsitter Mar 09 '24
Found some problems with my asyncio code because of this. The test was succeeding but the coverage wasn't 100% so I can recommend going for the extra mile.
Another thing I can recommend is flake8-pytest-style / ruff, they report this behavior: https://docs.astral.sh/ruff/rules/pytest-raises-with-multiple-statements/
1
u/Kiuhnm Mar 09 '24
That's weird: I'm using ruff in vscode through the official extension, but it didn't report the problem.
2
u/foarsitter Mar 10 '24
The PT rules are not enabled by default. Did you enable them in your configuration?
1
6
u/Ouitos Mar 07 '24
Speaking of coverage, I wonder if there exists a tool to compute not only coverage in line hit but also in type hit, when a function is type hinted as accepting multiple types.
Would that be useful? Would it be a lot of work to make it work ?
4
u/ToddBradley Mar 08 '24
That's a very interesting idea. I know about branch coverage, but this is something different. Most of my knowledge of test coverage comes from strongly typed languages, so I've never seen such a thing as what you're describing.
6
2
Mar 08 '24 edited Mar 08 '24
I've been working on a Django app (a small animal shelter management web app) and have been trying to get into the habit of writing tests - it's been a real downer for me, though. I'm not really trained in writing code and figured it out over years automating stuff as a network engineer - but man, i swear writing test cases literally costs me more time than writing the app and manually testing combined. Some of the stuff, like testing views requiring data input into a form, just kinda confuses me and I end up spending hours trying to get the test set up correctly.
Is there like a general rule for how big an app needs to be to justify rigorous testing? I do this on the side for fun but have spent like 10 hours trying to sort out my testing and still don't have any really useful tests besides a few basic views. Testing in Flask made way more sense to me but the rest of the app writing makes way more sense in Django, lol.
I'm using pytest with pytest-django and factory-boy for handling instantiating my models, but it's the construction of the tests I struggle with. Even the ones I have right now are looking for specific static info from the page so I know it lands on the right url, but this feels pretty fragile and so I'm just questioning everything I'm doing with tests now lol.
5
u/billsil Mar 08 '24
Coverage without the right checks means nothing. Â I could just removing my type checking and improve coverage. Â Yeah that check may fail while building the right exception message, but at least itâs failing in the right spot.Â
 100% coverage is not an effective use of time and doesnât catch edge cases.  Actively trying to break the function and not worrying about coverage actually makes code robust.
Not everyone needs to have bulletproof code either. Â The run once, sit on the shelf for 6 months, add new feature and run again mode does not need the same level of quality as Amazonâs backend.
7
2
u/legobmw99 Mar 08 '24
The comments here completely missed the point
2
u/eveninghighlight Mar 08 '24
Makes sense, the sub is mostly people who want to or have just done a 30 day python coding challengeÂ
1
-5
u/wineblood Mar 07 '24
Don't assert two different things in a single test.
4
u/jer1uc Mar 07 '24
Maybe just a difference in our terminology, but I read the problem OP expressed as the inverse: two tests with only one assertion.
Another way to express this might be:
assert (is_odd(3) or is_odd(4))The issue here being that you could be convinced that both scenarios work because the test only ever asserts once.
3
u/if_username_is_None Mar 07 '24
I think you're agreeing; OP says the "correct" version has two tests (two function calls) and two assertions (neither function call raises BadArgument)
I would refactor your example and OP's into two separate tests. "is odd works for odd case" & "is odd works for even case"
5
u/Kiuhnm Mar 08 '24
OP says the "correct" version has two tests (two function calls) and two assertions (neither function call raises BadArgument)
It's the opposite: both function calls must raise BadArgumentError. The problem is that when the first call throws, the second one is just skipped.
29
u/ToddBradley Mar 08 '24
Also, I think you (or someone else) would've noticed this bug sooner if there was only one assertion in the test and you fed the scenarios in via
pytest.mark.parametrize.