r/learnpython • u/bn_sj2020 • Dec 11 '20
My first project: Made a COVID 19 bot using selenium, pytesseract and tweepy
Posting here to get general advice, coding tips, and how to further improve my skills
Hello! I am a self taught "programmer" been learning on and off these past 7 months. Finally, after the basics, learning version control and using APIs I was inspired to do a twitter bot that tweets COVID 19 data specifically for my city.
This is my first proper project using Git and GitHub (my previous ones were just one .py files for practice)
This is my webscraper bot to get data from this website/tableau dashboard. I have been up several nights trying to figure out how to do a requests.get but failing every time since I have recently found out this is a dynamic website using js, which I am totally unfamiliar with. Thanks to this sub and r/learnprogramming by posting questions there I tried to learn selenium and take a screenshot of the data I need then using pytesseract, an optical character recognition module in python, to convert the image to a string so that I can tweet the data using the Twitter API.
Currently, I am very fascinated with the ease of use of APIs and I have been day dreaming of future projects using APIs. :)
EDIT: Thank you for the awards! Means a lot and gives me motivation to keep going. Currently, I have added more functionality to the bot where I can check the data before tweeting and being able to edit it. Also it can retweet now. Hehe.
2
Dec 11 '20
I am loving all the COVID twitter bots that people are making! I've been learning about 5 months and was at a loss for my next big girl project and these are an inspiration :)
1
u/bn_sj2020 Dec 12 '20
Keep it up! As long as you love what you are doing you will enjoy the journey!
2
u/PandaIceBear Dec 11 '20
Hello! Great work :D I'd like to let you know about this python package
requests-html
This package allows you to make requests from the website & render some javascript. As you mentioned you used selenium cause the site is dynamic and it's with some javascript so probably this might help you just incase you must not have a 3rd party software. Keep it up!
1
u/bn_sj2020 Dec 12 '20
I was reading the docs and could do the examples given but doing it on the website that I was trying to scrape was really difficult for me and could not do it :( When I get better Ill tackle this problem again ehhe
2
u/newtocoding153 Dec 11 '20
Interesting! From PH myself. Currently doing FCC Python course and self-taught. What resources did you use to learn? Anyway, awesome work!
3
u/bn_sj2020 Dec 12 '20 edited Dec 12 '20
Hello!
GitHub was very confusing for me at the start, I made an account in August 2020 and only recently that I partly understood the basics.
These two videos have helped me a lot. Take note I recall I kept watching and going back to these videos more than a couple of times while understanding GitHub.
For APIs, I learned about them about a week ago and it took me a couple of tries and reading docs. This video also helped me a lot.
Simple walkthrough/Steps how I built my project:
Find a website to get data from
Learned how to webscrape from books and videos (requests module and selenium)
Now that I have the data, format it so that it can be tweeted in a pretty way
Read this article about twitter API TweepyTutorial)
I used task scheduler in Windows 10 to make it run every day at 8am. Note: My PC needs to be on for this to work. Right now I'm studying how to make my code run on a server so that I dont have to turn on my computer everyday.
2
u/Both-Ad6729 Dec 11 '20
Perdón por hablar en español pero te recomiendo lo siguiente: si selenium lo usas solamente para scrapear información html, te recomiendo usar requests y BeatifoulSouo que es más rápido, práctico y fácil. Cualquier duda que tengas escribime por discord
xJamp#3238
2
u/bn_sj2020 Dec 11 '20
Perdón por hablar en español pero te recomiendo lo siguiente: si selenium lo usas solamente para scrapear información html, te recomiendo usar requests y BeatifoulSouo que es más rápido, práctico y fácil. Cualquier duda que tengas escribime por discord
Hello! Sorry if there will be a misunderstanding, what I did is copy pasted this on google translate! Haha. Yep! I did do that at first but this is a dynamic JS website, the data is difficult to scrape through HTML using requests or beautifulsoup, most of the time my scripts return None or the wrong thing and CTRL+F the data that I need does not pop up. Upon further research this website is built dynamically and plain scraping will not work or just difficult. But, big but here, it was difficult for me and I really tried and did my best and made a work around with selenium. If someone can do it with just requests and beautiful soup I would be in awe and would ask that person to be my sensei haha!
0
Dec 11 '20
[deleted]
1
u/bn_sj2020 Dec 11 '20
Thank you! It feels rushed and I keep editing and committing stuff everyday on github making small tweaks and changes to improve quality of life features for myself haha
0
Dec 11 '20
[deleted]
1
u/bn_sj2020 Dec 11 '20
Mañana podés hacer llamada por discord? Tengo cosas interesantes que decirte
Hello! I have discord you can message me if you want first hehe
2
16
u/ConflictedJew Dec 11 '20
FYI - it looks like the DOH posts a data drop on a public Google Drive.
Incorporating those data drops would be a nice progression of this project. Nice work!