As much as I love JS/Typescript, I would still go with (python) Scrapy for scraping (or at least a scraping framework). The numbers of functionalities out of the box is too good (automatic paging, automatic detection and following of links, retry strategies, switchable exporters with no config (json, csv, db, whatever), automatic map to entities, easy config for proxies (rotations, random ones, ...), plus the awesome ecosystem for data processing (pandas, numpy, ...). I don't want to reinvent the wheel for scraping anymore.
I'm a purely full stack JavaScript developer, and haven't touched Python in years.
I agree. Doing web scraping with JavaScript is such a weird song and dance. Even all the web scraping libs (and I've used a bunch) aren't as easy as it was during my python days.
19
u/c_eliacheff Aug 08 '22
As much as I love JS/Typescript, I would still go with (python) Scrapy for scraping (or at least a scraping framework). The numbers of functionalities out of the box is too good (automatic paging, automatic detection and following of links, retry strategies, switchable exporters with no config (json, csv, db, whatever), automatic map to entities, easy config for proxies (rotations, random ones, ...), plus the awesome ecosystem for data processing (pandas, numpy, ...). I don't want to reinvent the wheel for scraping anymore.