r/javascript • u/pijora • Aug 08 '22

Web Scraping with JavaScript and NodeJS

https://www.scrapingbee.com/blog/web-scraping-javascript/
590 Upvotes

96% Upvoted

11 comments sorted by

28

u/badsyntax Aug 08 '22

No mention of Playwright? It'd be my first choice.

19

u/c_eliacheff Aug 08 '22

As much as I love JS/Typescript, I would still go with (python) Scrapy for scraping (or at least a scraping framework). The numbers of functionalities out of the box is too good (automatic paging, automatic detection and following of links, retry strategies, switchable exporters with no config (json, csv, db, whatever), automatic map to entities, easy config for proxies (rotations, random ones, ...), plus the awesome ecosystem for data processing (pandas, numpy, ...). I don't want to reinvent the wheel for scraping anymore.

4

u/Secret-Plant-1542 JavaScript yabbascript Aug 09 '22

I did a lot of web scraping in 2015 using python.

I'm a purely full stack JavaScript developer, and haven't touched Python in years.

I agree. Doing web scraping with JavaScript is such a weird song and dance. Even all the web scraping libs (and I've used a bunch) aren't as easy as it was during my python days.

4

u/[deleted] Aug 08 '22

and multi-threading

0

u/[deleted] Aug 08 '22

[deleted]

1

u/AegonThe241st Aug 08 '22

No it is not

2

u/vlevi Aug 08 '22

Try puppeteer for nodejs

0

u/[deleted] Aug 08 '22

[deleted]

29

u/SpeedDart1 Aug 08 '22

Definitely not faster. Might have a better developer experience but that’s a matter of opinion.

Edit: aight looked at your comment history and you’re a troll. Everyone ignore this guy.

0

u/[deleted] Aug 08 '22

[deleted]

1

u/SpeedDart1 Aug 09 '22 edited Aug 09 '22

Because he said it was faster.

He’s also posting bait in r/ruby about ruby being bad!

He insulted someone’s project on r/programming.