r/learnpython • • 7h ago

Variability in langdetect output

Hi all,
I’m using Python to filter out rows that contains non-English text from a noisy 10,000-row Excel dataset, and the final count varies slightly between runs (for example, keeping 7,522 vs. 7,508 cells—a difference of less than 0.001% of total cells). Is that normal for noisy data, and is that level of variance acceptable for research?

Thanks all!

3 Upvotes

7 comments sorted by

2

u/Brian 6h ago

It maybe depends on how you're doing it, but I'd kind of expect something like that to be deterministic, at least if using a locally running method. It might not perfectly detect the correct language every time, but I'd expect the same result from run to run unless the data or code itself changes.

It might vary if it's doing something like hitting a network service for language detection, but otherwise it may be something going wrong (eg. not handling errors correctly, or a race condition (if done in parallel), so may be worth debugging a bit.

1

u/AR3126 6h ago

Thanks for the detailed answer. From my readings it's likely non determinists and hence why the different results

1

u/FontanJanneice18 3h ago

yeah for actual research reproducibility you definitely want the exact same output every run, even if the accuracy itself isn't 100%

1

u/egotherapy 6h ago

In my experience, when strings are really short, langdetect has problems detecting languages correctly. I'd expect that you've filtered out really short/long/nonsense/corrupt text, but if not, that may account for some of the variability.

1

u/AR3126 4h ago

Thank you so much! This is helpful to know!

•

u/lakseol 32m ago

The langdetect page in PyPi has this:

Language detection algorithm is non-deterministic, which means that if you
try to run it on a text which is either too short or too ambiguous, you might
get different results everytime you run it.

To enforce consistent results, call following code before the first language
detection:

    from langdetect import DetectorFactory
    DetectorFactory.seed = 0

You will have to experiment a bit to see if this helps in your case.