r/rust • • 2d ago

🛠️ project I diffed uutils coreutils against GNU with generated black-box tests: 43 bug reports, 50 fix PRs, 34 merged

Edit: rewrote this to focus on what I learned and the actual bugs, per the mod note.

I spent the last few weeks running black-box differential tests of uutils coreutils against GNU coreutils 9.12: same inputs, then compare stdout, stderr, exit code and the files written. Some things I learned:

  1. The bugs are almost never in the main path. Normal input matched GNU nearly every time. The differences were in edge cases: odd file names, directories passed where a file is expected, empty arguments, option values that look like options.

  2. Some real bugs it found, all fixed upstream with a test:

  • tail 007 reported the missing file as '7', because the name went through number formatting in the message. The same bug was in tac, nl, md5sum, sort and shred.

  • sort -c dir exited 0 with no message instead of failing.

  • readlink -f '' resolved to the current directory and exited 0, GNU fails.

  • fmt -w 20 could print a line longer than 20 columns.

  • md5sum -c rejected the escaped lines that md5sum itself writes for names with a backslash or newline.

  • mv -n z z failed with "are the same file", GNU just skips it.

  1. Pin the reference version. A lot of my early "bugs" were really GNU changing between 9.4, 9.7 and 9.12 (quoting in tail -v headers, date modifiers). uutils targets the newest GNU, so recording from an older one gives false positives.

  2. Exit codes and error messages matter to scripts more than you would think. A wrong exit code on a read error is a silent bug in a pipeline.

  3. Small PRs with one test each get reviewed fast. The uutils maintainers were quick and precise.

The tool I wrote for this is open source if anyone wants to try it on their own rewrite: https://github.com/emirmrkaljevic/behave

Happy to answer questions about the approach.

0 Upvotes

6 comments sorted by

•

u/rust-ModTeam 2d ago

TL;DR: We ask that posts & articles submitted to r/rust abide by the No Code Dumps rules.


Your post, or the article it links to, appears not to abide by the No Code Dumps rules.

We invite you to double-check the rules, as well as their Q&A.


For any question or dispute, please send us a modmail.

4

u/InternationalFee3911 2d ago

This does partly violate the rule, though promoting their tool is only an explanation of how they did. As to what they did, this seems legit. It’s a huge boost to a project that’s becoming widely used through Ubuntu and that’s been all over Rust news. So it deserves some spotlight here!

1

u/emirdev_mo 1d ago

Thanks, fair point. I rewrote the post to focus on what I learned and the actual bugs.