r/learnpython • u/AR3126 • 7h ago
Variability in langdetect output
Hi all,
I’m using Python to filter out rows that contains non-English text from a noisy 10,000-row Excel dataset, and the final count varies slightly between runs (for example, keeping 7,522 vs. 7,508 cells—a difference of less than 0.001% of total cells). Is that normal for noisy data, and is that level of variance acceptable for research?
Thanks all!
1
u/egotherapy 6h ago
In my experience, when strings are really short, langdetect has problems detecting languages correctly. I'd expect that you've filtered out really short/long/nonsense/corrupt text, but if not, that may account for some of the variability.
•
u/lakseol 32m ago
The langdetect page in PyPi has this:
Language detection algorithm is non-deterministic, which means that if you
try to run it on a text which is either too short or too ambiguous, you might
get different results everytime you run it.
To enforce consistent results, call following code before the first language
detection:
from langdetect import DetectorFactory
DetectorFactory.seed = 0You will have to experiment a bit to see if this helps in your case.
2
u/Brian 6h ago
It maybe depends on how you're doing it, but I'd kind of expect something like that to be deterministic, at least if using a locally running method. It might not perfectly detect the correct language every time, but I'd expect the same result from run to run unless the data or code itself changes.
It might vary if it's doing something like hitting a network service for language detection, but otherwise it may be something going wrong (eg. not handling errors correctly, or a race condition (if done in parallel), so may be worth debugging a bit.