r/Python • u/Paddy3118 • Jul 10 '22
Discussion Raw Python vs Python & SQLite vs GNU Linux command line utilities!
https://paddy3118.blogspot.com/2022/07/raw-python-vs-python-sqlite-vs-gnu.html3
u/arthurno1 Jul 11 '22
This should be obviously I/O bound. Reading 9 gig data, and then storing same 9 gig in a database should be twice the I/O which your test seem to confirm. I have just seen the original post and all the expert advices upvoted by people who probably didn't try but were just theory crafting.
Anyway, It would be interesting to see a pure bash solution on your data. I would lilke to compare bash vs python. Could you run this:
#!/bin/env bash
arr=()
declare -A map
while IFS= read -r line
do
((map[$line]++))
done < "$1"
for word in ${!map[*]}
do
arr[ ${map[$word]} ]=$word
done
index=(${!arr[@]})
for ((i=${#index[@]}-1 ; i>=0 ; i--))
do
word=${arr[index[i]]}
printf "%s %s\n" $word ${map[$word]}
doneSave it in some file, say "words-per-file.sh", chmod +x ./words-per-file.sh, and use as:
./words-per-file.sh your_words_file > words.outI don't have 9 gig file to create, nor am I willing to create one just to compare. I guess Python should be faster, but would be interesting to see by how much.
2
u/zurtex Jul 10 '22 edited Jul 10 '22
If you're trying to eek out performance with Python you should read the files in binary, then use
defaultdict(int)instead ofas your collection, and then sort, and finally encode them at the end.Encoding is surprisingly expensive and
Counteris pure Python and pretty slow at times. I bet if you could probably equal your Datamash results, and probably surpass it in Python 3.11.