r/aws • u/farnulfo • Jan 09 '20
article Serverless Web Scraping with AWS Lambda
https://www.scrapingbee.com/blog/serverless-web-scraping-with-aws-lambda-and-java/3
u/TommyF-17 Jan 10 '20
In the article, it talks about "Web Scraping is I/O bound, most of the time is spent waiting for HTTP responses, so we don’t need high-end CPU servers." and then goes on to use Lambda - arguably one of the more expensive-per-unit forms of compute.
Lambda is suited to sparodic requests of an unknown load, as it scales to meet said load. Scraping looks to be more of a batch process where you kick off a request and it does things. There are more cost efficient serverless solutions for doing something like this if they can be made to work. Fargate, for example, should work for a use case like this as far as I can tell.
Now, it does also depend on how this is to be consumed. If you keep under the free tier, then fine. If you are providing this as a service with sporadic instantiations and uncertain load, then probably fine also (you would have to do some usage analysis to see if it's worth finding a more efficient implementation). But if you are doing a daily 'scrape all teh things' then probably not so much.
I guess this adds to the complicated area that I have seen discussed over the last few days - misusue of serverless.
It depends...
1
u/mr_timmaah Jan 10 '20
This. I have a side hustle that does a lot of web scraping. Lambda was cool but it was $200 vs $5 Digital Ocean droplet. You are paying for all the time it waiting on responses. Didn't make sense.
1
3
u/chaizus Jan 10 '20
Nice! I used aws chalice to make a http get request with python and works great with my web scrapping logic