r/aws • • Jan 09 '20

article Serverless Web Scraping with AWS Lambda

https://www.scrapingbee.com/blog/serverless-web-scraping-with-aws-lambda-and-java/
42 Upvotes

5 comments sorted by

3

u/chaizus Jan 10 '20

Nice! I used aws chalice to make a http get request with python and works great with my web scrapping logic

3

u/TommyF-17 Jan 10 '20

In the article, it talks about "Web Scraping is I/O bound, most of the time is spent waiting for HTTP responses, so we don’t need high-end CPU servers." and then goes on to use Lambda - arguably one of the more expensive-per-unit forms of compute.

Lambda is suited to sparodic requests of an unknown load, as it scales to meet said load. Scraping looks to be more of a batch process where you kick off a request and it does things. There are more cost efficient serverless solutions for doing something like this if they can be made to work. Fargate, for example, should work for a use case like this as far as I can tell.

Now, it does also depend on how this is to be consumed. If you keep under the free tier, then fine. If you are providing this as a service with sporadic instantiations and uncertain load, then probably fine also (you would have to do some usage analysis to see if it's worth finding a more efficient implementation). But if you are doing a daily 'scrape all teh things' then probably not so much.

I guess this adds to the complicated area that I have seen discussed over the last few days - misusue of serverless.

It depends...

1

u/mr_timmaah Jan 10 '20

This. I have a side hustle that does a lot of web scraping. Lambda was cool but it was $200 vs $5 Digital Ocean droplet. You are paying for all the time it waiting on responses. Didn't make sense.

1

u/jonathantn Jan 10 '20

Lambda + Puppeteer