r/Python • • Mar 07 '24

Discussion Tests should have 100% coverage

No, I'm not saying what you probably think I am. Keep reading: I'll be brief.

Here's what I found today in one of the tests:

for bad_fp in _bad_fps:
    with pytest.raises(BadArgumentError):
        follow_path(bad_fp, '.')
        follow_path('.', bad_fp)

The correct code should've been

for bad_fp in _bad_fps:
    with pytest.raises(BadArgumentError):
        follow_path(bad_fp, '.')
    with pytest.raises(BadArgumentError):
        follow_path('.', bad_fp)

Is that a stupid mistake? Yes, it is, but stupid mistakes do happen, unfortunately.

How did I catch it? The coverage of the testing file wasn't 100% like it usually is! I got suspicious and I saw that there was a glaring red line (vscode, coverage gutter) on the second call to follow_path.

P.S. In case you're wondering, the code uses a custom hierarchy of exceptions instead of reusing the standard ones. The code is internal and we like it that way :)

0 Upvotes

37 comments sorted by

29

u/ToddBradley Mar 08 '24

Also, I think you (or someone else) would've noticed this bug sooner if there was only one assertion in the test and you fed the scenarios in via pytest.mark.parametrize.

3

u/Kiuhnm Mar 08 '24

In this case, it would've certainly prevented the bug, but, in general, I'm not sure having small and specific tests is without downsides. We have no strict guidelines about this.

For instance, my code is usually not so straightforward to test because it involves algorithms on graphs, so I end up generating random graphs and manipulating them while asserting that everything is behaving as it should. This results in test functions that can have up to 20 assertions each. (I also write formal proofs of course.)

The 1-assertion-per-test limitation would make my life much harder and, I think, my tests less reliable. Fixtures do help, but are not enough.

Let me know if you have any suggestions!

9

u/ToddBradley Mar 08 '24

"generating random graphs" and "everything is behaving as it should" makes me wonder if you're writing unit tests or something higher on the test automation pyramid.

1

u/Kiuhnm Mar 08 '24

I still consider them unit tests because there's nothing below them, but a unit may be a small class instead of a single function or method.

If I'm trying to prove 10 different behaviors of a class, I'll write 10 different test functions.

I'm under the impression that many programmers want to know what's wrong with their test just by looking at the logs. I'm just happy to know that there's a problem with a particular graph and sequence of operations. Then I analyze the whole thing (we're still talking of a single class) and find out what's wrong.

I guess different problems require different approaches.

The mistake I talked about wasn't mine :), but I dislike relying on "advanced" pytest features as well. I feel that they're too limited for my taste. For instance, pytest.mark.parametrize only supports a single sequence of args. Repeating the decorator results in the cartesian product of the args, which is not what I want.

Why should I try to combine big lego blocks to get what I want when I can just use Python itself?

Also, if I change testing library, most of what I wrote will still work.

I'm probably the only guy who organizes his notes as a tree of small txt files, so maybe it's just me. I like to keep things simple. Luckily for me, I have lots of freedom.

2

u/wyldstallionesquire Mar 08 '24

Hypothesis might be nice in your case.

2

u/Kiuhnm Mar 08 '24

Thank you. I'll port some of my biggest tests to Hypothesis and see how it goes. I'm not too optimistic, though, because my tests are carefully crafted to be as fast as possible while still retaining most of the "reliability". I'm certainly willing to pay something for the gain in productivity, but not too much. We'll see.

1

u/Kiuhnm Mar 08 '24

As I suspected, the problem with similar tools is that they can't generate good examples when there are lots of assumptions involved. My graphs must have certain shapes and no automatic general-purpose library can possibly generate them as efficiently as my code.

1

u/eveninghighlight Mar 08 '24

I think you're right about the purpose of tests, in my mind the ideal testing dashboard is a big green button if they all pass and then a big red button if any fail

then you have to do some digging to find out what failed

2

u/Kiuhnm Mar 08 '24

"Something bad happened! Come home!" and hangs up.

I'm just joking :)

Some dashboards organize tests in a tree so that, when it's completely collapsed, you get your single big green/red button.

2

u/eveninghighlight Mar 08 '24

Yeah in my mind you could click on the button to see more, find which test failed, the logs etc but the headline message is that something broke

32

u/GraphicH Mar 07 '24

High percent coverage is much more critical in dynamically typed / scripting languages than in compiled languages where the compiler watches your back. Its one of the hardest things for people who worked primarily in compiled languages to get their head around when writing "production grade" python code. Its also why I think the testing and linting utilities for these languages seem a lot more superior to the stuff you find for compiled languages. And yeah if you have test code that never fires, that begs the question, why is it written, we do coverage reports on test code as much as production code.

23

u/lukewhale Mar 07 '24

Yall writing tests ? 😂😂😂 /s

9

u/badjayplaness Mar 08 '24

I used to only practice the “push to master” driven development method. Apparently they’re other methods.

4

u/crawl_dht Mar 08 '24

No QA team can beat testing in production.

1

u/bartenderandthethief Mar 08 '24

Question, do you think it is good practice to only have one line within the pytest raises context?

2

u/Kiuhnm Mar 09 '24

It is according to https://docs.astral.sh/ruff/rules/pytest-raises-with-multiple-statements/

It certainly helps avoid these kinds of mistakes, but looking at the coverage is more general, IMHO.

1

u/ArtOfWarfare Mar 08 '24

I don’t think your actual code had 100% branch coverage. If it did, then that extra line in your test is superfluous and should just be removed.

Also, check your mutation coverage. If you’ve not seen it before, it tends to expose 50% of tests as being useless garbage.

1

u/Kiuhnm Mar 08 '24 edited Mar 08 '24

There's the testing code and the tested code. Usually, code coverage helps to detect untested parts of the tested code.

Here I'm saying that code coverage of the testing code helped me detect parts of the testing code (as opposed to the tested code) that weren't being executed.

The code

with pytest.raises(BadArgumentError):
        follow_path(bad_fp, '.')
        follow_path('.', bad_fp)

is wrong because the first call to follow_path is expected to throw, but that way, the second call is never executed and tested. One needs to have a separate with statement for the second call.

1

u/ArtOfWarfare Mar 08 '24

Yes, I understand how your test is wrong, but I’m saying your actual code either isn’t actually fully covered (you’ve missed branches) or it is fully covered and this incorrect test is just redundant.

But also, check your mutation coverage. That’s far more useful than line coverage. It’ll expose all the bugs that your test suite don’t. And if you’ve never checked your mutation coverage before, you’ll probably find your tests don’t even find half the potential bugs in your code despite having full line and branch coverage.

1

u/Kiuhnm Mar 08 '24

I don't have and I don't try to have 100% coverage of my code. I was just talking about the coverage of my testing code. Now I understand what you meant.

1

u/foarsitter Mar 09 '24

Found some problems with my asyncio code because of this. The test was succeeding but the coverage wasn't 100% so I can recommend going for the extra mile.

Another thing I can recommend is flake8-pytest-style / ruff, they report this behavior: https://docs.astral.sh/ruff/rules/pytest-raises-with-multiple-statements/

1

u/Kiuhnm Mar 09 '24

That's weird: I'm using ruff in vscode through the official extension, but it didn't report the problem.

2

u/foarsitter Mar 10 '24

The PT rules are not enabled by default. Did you enable them in your configuration?

1

u/Kiuhnm Mar 10 '24

Now, it works. Thanks!

6

u/Ouitos Mar 07 '24

Speaking of coverage, I wonder if there exists a tool to compute not only coverage in line hit but also in type hit, when a function is type hinted as accepting multiple types.

Would that be useful? Would it be a lot of work to make it work ?

4

u/ToddBradley Mar 08 '24

That's a very interesting idea. I know about branch coverage, but this is something different. Most of my knowledge of test coverage comes from strongly typed languages, so I've never seen such a thing as what you're describing.

6

u/Wh00ster Mar 08 '24

Why write tests when you have production?

2

u/[deleted] Mar 08 '24 edited Mar 08 '24

I've been working on a Django app (a small animal shelter management web app) and have been trying to get into the habit of writing tests - it's been a real downer for me, though. I'm not really trained in writing code and figured it out over years automating stuff as a network engineer - but man, i swear writing test cases literally costs me more time than writing the app and manually testing combined. Some of the stuff, like testing views requiring data input into a form, just kinda confuses me and I end up spending hours trying to get the test set up correctly.

Is there like a general rule for how big an app needs to be to justify rigorous testing? I do this on the side for fun but have spent like 10 hours trying to sort out my testing and still don't have any really useful tests besides a few basic views. Testing in Flask made way more sense to me but the rest of the app writing makes way more sense in Django, lol.

I'm using pytest with pytest-django and factory-boy for handling instantiating my models, but it's the construction of the tests I struggle with. Even the ones I have right now are looking for specific static info from the page so I know it lands on the right url, but this feels pretty fragile and so I'm just questioning everything I'm doing with tests now lol.

5

u/billsil Mar 08 '24

Coverage without the right checks means nothing.  I could just removing my type checking and improve coverage.  Yeah that check may fail while building the right exception message, but at least it’s failing in the right spot. 

 100% coverage is not an effective use of time and doesn’t catch edge cases.  Actively trying to break the function and not worrying about coverage actually makes code robust.

Not everyone needs to have bulletproof code either.  The run once, sit on the shelf for 6 months, add new feature and run again mode does not need the same level of quality as Amazon’s backend.

7

u/ToddBradley Mar 08 '24

I'm not convinced you understood the post.

2

u/legobmw99 Mar 08 '24

The comments here completely missed the point

2

u/eveninghighlight Mar 08 '24

Makes sense, the sub is mostly people who want to or have just done a 30 day python coding challenge 

1

u/ToddBradley Mar 08 '24

the sub is mostly people

Posters, commenters, or readers?

-5

u/wineblood Mar 07 '24

Don't assert two different things in a single test.

4

u/jer1uc Mar 07 '24

Maybe just a difference in our terminology, but I read the problem OP expressed as the inverse: two tests with only one assertion.

Another way to express this might be:

assert (is_odd(3) or is_odd(4))

The issue here being that you could be convinced that both scenarios work because the test only ever asserts once.

3

u/if_username_is_None Mar 07 '24

I think you're agreeing; OP says the "correct" version has two tests (two function calls) and two assertions (neither function call raises BadArgument)

I would refactor your example and OP's into two separate tests. "is odd works for odd case" & "is odd works for even case"

5

u/Kiuhnm Mar 08 '24

OP says the "correct" version has two tests (two function calls) and two assertions (neither function call raises BadArgument)

It's the opposite: both function calls must raise BadArgumentError. The problem is that when the first call throws, the second one is just skipped.