r/javascript • • Aug 08 '22

Web Scraping with JavaScript and NodeJS

https://www.scrapingbee.com/blog/web-scraping-javascript/
590 Upvotes

11 comments sorted by

View all comments

19

u/c_eliacheff Aug 08 '22

As much as I love JS/Typescript, I would still go with (python) Scrapy for scraping (or at least a scraping framework). The numbers of functionalities out of the box is too good (automatic paging, automatic detection and following of links, retry strategies, switchable exporters with no config (json, csv, db, whatever), automatic map to entities, easy config for proxies (rotations, random ones, ...), plus the awesome ecosystem for data processing (pandas, numpy, ...). I don't want to reinvent the wheel for scraping anymore.

4

u/Secret-Plant-1542 JavaScript yabbascript Aug 09 '22

I did a lot of web scraping in 2015 using python.

I'm a purely full stack JavaScript developer, and haven't touched Python in years.

I agree. Doing web scraping with JavaScript is such a weird song and dance. Even all the web scraping libs (and I've used a bunch) aren't as easy as it was during my python days.