r/Python • u/preetisw • Nov 29 '22
Tutorial Python and Selenium are better for scraping data
Headless browsers in Python and Selenium are better for scraping data from web pages where scrapers have been blocked.
We followed step-by-step advice for scraping data from a web page using a headless chrome in Selenium Python. It's fast, easy, requires less resources (so it can run on older laptops), and can deal with JavaScript perfectly.
49
u/CKtalon Nov 29 '22
Slow as fuck compared to good ol’ requests. Only use it when antiscraping measures are prevalent.
1
u/pconwell Feb 23 '23
Dumb question - I don't do much scraping. Is captcha considered an anti-scraping measure? I am working on a project to download a bunch of csv files (I guess not technically web scraping), but the data is behind a username/password + captcha page.
I would assume requests would not work well with captcha, and something like selnium would be better?
1
1
u/greatestbaker Mar 30 '23
What do you mean by good ol' request? I want to read more about it. What keyword should I search?
1
45
u/colei_canis Nov 29 '22
It’s a very useful tool but I’d only use it for scraping if the site I was trying to scrape was aggressively trying to avoid it because it’s also slow as swimming in treacle.
40
Nov 29 '22
As someone that's built a couple scrappers in his job, just use Scrapy.
It handles pretty much everything a production grade webscraper should. And if it doesn't theres probably an extension or middleware that does.
7
u/dalepo Nov 29 '22
I've been doing data scraping for years and no tool has been close to scrapy in terms of efficiency. I've got some old bots that has been running for more than 6 years
The only thing I dislike about the scrapy ecosystem is splash, which in my case switched to puppeteer + node.
5
u/someone383726 Nov 29 '22
I’ve been meaning to give scrapy a shot. I mostly just use requests and beautiful soup, but will use selenium for accessing JavaScript sites.
4
Nov 29 '22 edited Nov 29 '22
In my experience, rendering JS pages is rarely a must and will slow down your scrapping significantly.
Most often the rendered JS is simply requesting data from a backend API which should be pretty easy to replicate in your code. Just check what requests are being fired by the page and what they contain by using your browser's developer tools (in particular the Network tab).
There has only been a single time in which I've had to render JS. It was a page with heavily obfuscated JS and APIs, which I didn't have the time, patience or experience to make sense of (not a fan of JS).
5
u/sageknight Nov 29 '22
My site uses JS to log in. I couldn't get around that as the Network tab does not show the post form that they use to log in.
3
Nov 29 '22
I actually have the same problem. Currently I'm using selenium to log in to a site and then I use selenium-requests to send my requests from the browser's session. It's been running for a couple years now and unlike when it was automated using only selenium, it doesn't break every couple of months when something in the frontend changes.
I did manage to capture and examine the log in request once, using Wireshark, but I haven't gotten around to trying to replicate it yet.
2
u/someone383726 Nov 29 '22
Thanks for the info. I actually have one JavaScript page that is a form that I have to submit to get the data I need that I’m using selenium for. I’ll have to take a closer look again. Before when I tried requests I never got a good response when working with it.
2
u/alcalde Nov 30 '22
Most often the rendered JS is simply requesting data from a backend API which should be pretty easy to replicate in your code. Just check what requests are being fired by the page and what they contain by using your browser's developer tools (in particular the Network tab).
That sounds like a lot more work than just using Selenium/beautifulsoup.
3
3
3
u/Mrblabbles Nov 29 '22
The speed of selenium is pretty apparent when your trying to mass scrape a website. I'm running selenium right now for scraping top tiktok views and hashtags and it only calls twice a day and the speed doesn't matter all the much. If I was scraping more data and more frequently I'd probably switch to beautifulsoup. Though I will say selenium is so easy to use, especially with Xpaths.
3
u/liko Nov 29 '22
Hard pass. I’ll stick with the soup.
1
u/alcalde Nov 30 '22
Why not both? Selenium to navigate the javascript-riddled web pages, then you have selenium pass the HTML to beautifulsoup. That's what I've done to parse Amazon wish lists.
2
u/ZachVorhies Nov 29 '22 edited Nov 30 '22
I want to point out that setting up selenium can be a giant pain and if people want to get a bot up on Max/Win/Linux and running as quickly as possible they should try out my tool install selenium web driver for you:
https://github.com/zackees/open-webdriver
It downloads everything automatically and uses sane defaults. It’s just a few lines of code to get running. It also does things like headless by default, disables SSL certs and has a tested path to compile a native C++ binary using nuitka. There is an example dockerfile to run in a container.
1
u/Dasher38 Nov 30 '22
I've never done any webdev but I was wondering why certificates are turned off by default ? Does it just ignore certificates when using https ?
3
u/ZachVorhies Nov 30 '22
I added this because when I was scraping certain cites, like gab.com, they had a bad cert that selenium wouldn’t accept. Im not that concerned about fake sites for a bot scraper so I turned it off.
2
u/MeNotSanta Nov 29 '22
Stop talking about the website like you are not one of the people that own it.
3
u/betazoid_one Nov 29 '22
This becomes difficult when most modern websites are dynamically loaded content
1
u/Chilling_Home_1001 Nov 29 '22
Has anyone ever done a scraper that uses the finished format of the page like an OCR system instead of the DOM structure? I use companies that scrape but I'm curious about this because I assume it would be more stable to minor changes in the HTML
1
1
1
5
Nov 29 '22
[removed] — view removed comment
1
u/Ordinary_Price_2189 Nov 29 '22
Can you refer some tutorial or guide? I have always used selenium for dynamic websites and it is slow slow.
3
u/Ihtmlelement Nov 29 '22
Your best bet would be to run fiddler while using the website. Narrow down which http traffic matters and script them with requests. For complicated stuff I sometimes grab the cURL data and use postman to build the python code. Basically just a game of finding what headers and cookies are required to unlock the next door.
90
u/jimtk Nov 29 '22
This guy has been promoting his site, mostly filled with inane articles, for the last 2 months.