r/madeinpython • • Oct 17 '20

Program to easily search through thousands of papers

Hi,

I am an undergrad, who has to constantly write different scientific reports for university.

Because english is my second language, I sometimes struggle to properly express myself, especially in "scientific english". Furthermore, I cant really wrap my head around the english punctuation.

To help me with this, I wrote a small pyhton script, which will look through up to 200.000 papers for a specific phrase or expression.

If it finds a paper in which the expression was used, it will print out the corresponding paragraph, so you have some context.

The program really helped me a lot during my last report, so I thought I would share it.

You can download it, along with instructions how to install it here:

https://github.com/nickhir/PhraseBase

32 Upvotes

9 comments sorted by

4

u/Nerdy_Shoes Oct 18 '20

That’s really cool, where did you get all the papers from?

4

u/shalash96 Oct 18 '20

From Kaggle, the data set is approximately 21GB.

2

u/nhaus111 Oct 18 '20

Exactly! However, you don't need every paper for the program to work. You can simply stop the unzipping earlier. I have never searched through more than 10.000 papers myself.

1

u/ImaginingAlchemist Oct 18 '20

This is a cool idea. Two suggestions/questions though:

Have you thought of using an html requests module or urllib3 to do a Google search for the phrase instead? This would let you run your program without having to predownload the papers.

But if you want to keep it how it is, why not just ask the user for the folder location rather than modifying the script?

1

u/nhaus111 Oct 18 '20

I decided to modify the script, because I usually run the script very often. For example, I try a certain phrase with or without a comma.

If you would have to type in the search location each time that would be very tiresome.

However, I agree that this is not an optimal solution.

I will probably change that, so the user has to modify a txt file and the actual script will ready this file. Then the user cant accidently mess with the script

1

u/hak122hak Oct 18 '20

İf i were you i prefer to ocr all papers, then load all ocr to mssql finaly i can be search the phrases. But this is long way and really struggle to me.

İ gues, Your way is better, but Ram is a really big problem. How did you solve this problem‽

1

u/nhaus111 Oct 18 '20

Im not sure what you mean. The papers are in a json format, so the files are super small (~>100kb). The whole program is not memory intensive at all.

1

u/hak122hak Oct 18 '20

İ mean i will go to use tesseract program to Ocr (Optical Character Recognition) to get words from digital pages and then i will use any database to save and search.

İ thought you are getting whole pages in memory but i was wrong.

Anyway your work is better. Thanks bro

1

u/SirDerpington660 Oct 24 '20

very cool! i have a ton of storage on my pc so i’m gonna download the papers and think of something i can do with these. maybe like find the sum of all the numbers that appear in the papers. i’d prolly have to have that program run overnight or smth lol