r/scrapingtheweb • • Oct 05 '25

$1000 for someone who really knows their web-scraping stuff (for real)

127 Upvotes

I will send you a $1000 and a job even, if you can tell me how and show me how to scrape LinkedIn posts for keywords/phrases at scale (hopefully not very expensively), bonus $100 if you can also how I can find similar phrases in other websites too. Ps:- I know about Apify but it can only scrape posts when you already have a profile in mind, I want to search phrases/keywords across all posts.


r/scrapingtheweb • • Dec 25 '25

Unpopular opinion: If it's on the public web, it's scrapeable. Change my mind.

57 Upvotes

I've been in the web scraping community for a while now, and I keep seeing the same debate play out: where's the actual line between ethical scraping and crossing into shady territory?

I've watched people get torn apart for admitting they scraped public data, while others openly discuss scraping massive sites with zero pushback. The rules seem... made up.

Here's the take that keeps coming up (and dividing people):
If data is on the public web (no login, no paywall, indexed by Google), it's already public. Using a script instead of manually copying it 10,000 times is just automation, not theft.

Where most people seem to draw the line:
✅ robots.txt - Some read it as gospel, others treat it like a suggestion. It's not legally binding either way.
✅ Rate limiting - Don't DOS the site, but also don't crawl at "1 page per minute" when you need scale.
❌ Login walls - Don't scrape behind auth. That's clearly unauthorized access.
❌ PII - Personal emails, phone numbers, addresses = hard no without consent.
⚠️ ToS - If you never clicked "I agree," is it actually binding? Legal experts disagree.

The questions that expose the real tension:

  1. Google scrapes the entire web and makes billions. Why is that okay but individual scrapers get vilified?

  2. If I manually copy 10,000 listings into a spreadsheet, that's fine. But automate it and suddenly I'm a criminal?

  3. Companies publish data publicly, then act shocked when people use it. Why make it public then?

Where do YOU draw the line?

  • Is robots.txt sacred or just a suggestion?

  • Is scraping "public" data theft, fair use, or something in between?

  • Does commercial use change the ethics? (Scraping for research vs selling datasets)

  • If a site's ToS says "no scraping" but you never agreed to it, does it apply?

I'm not looking for the "correct" answer—I want to know where you actually draw the line when nobody's watching. Not the LinkedIn-safe version.

Change my mind


r/scrapingtheweb • • Nov 24 '25

Why is Home Depot blocking literally everything? Puppeteer, Selenium, Playwright, real browsers… all get “Oops!! Something went wrong.”

57 Upvotes

I’ve been trying to scrape some product pages from Home Depot for a project, and I’m hitting a wall I can’t get around. No matter what I use — Puppeteer, Playwright, Selenium, undetected-chromedriver but the site eventually returns the same thing: “Oops!! Something went wrong.” It doesn’t matter whether I run Chrome, Chromium, Firefox, or Edge.They still flag it.

At this point it feels like Home Depot is running some extremely aggressive bot-detection system that triggers on anything unusual. Either that or their anti-scraping heuristics basically assume every visit is a bot unless proven human.

Has anyone here actually found a reliable way to fetch HTML from Home Depot product pages without immediately running into their block page? Is there something specific they look for? Any tricks that actually work? Curious what’s worked for others, because right now every approach — even ones that work on much harder sites — just face-plants on Home Depot. (Btw I’m just a beginner)


r/scrapingtheweb • • 13d ago

I made a Python Library for scheduling recurring webscraping

33 Upvotes

I made LaptopScheduler, a Python scheduling library for Windows laptops.

It’s for recurring work that can wait until your laptop is available: reports, backups, data imports, and similar tasks. You can make sure your scraping runs consistently without renting out a webserver. You register your own Python functions, and it records their runs, handles retries, and applies your chosen catch-up policy after downtime.

Install: pip install laptopscheduler

https://github.com/seaweed8888/laptop-schedueler


r/scrapingtheweb • • May 30 '24

Best Oxylabs alternatives for residential proxies and web scraping?

31 Upvotes

Are there any alternatives to Oxylabs on the residential proxy front that don't get as many issues with captcha or IP bans? I have the budget but need something more reliable.


r/scrapingtheweb • • Apr 05 '26

Discussion 104K Github⭐️ for Firecrawl😳. Never used it. Am I missing something?

26 Upvotes

Of course, I heard about it, but I never heard anyone going all gaga over it. I see the repository says that it's open-source. Really? I did research, seems more like open-source + commercial business on top of it ... the ususal path. I see people are ranting about it a lot, especially mocking its open-source version, and calling out its (excessively) expensive pricing. Just curious, if I should try it out. What's driving so much interest? Have you used it? What's unique? Why this craze behind it?

Scrapy sits at 61K+. Crawlee at 22K. I've used these. Enough for my scraping use cases. How would you position firecrawl against these, Orange and Roses? Or is it fair comparison?

By the way, I feel, I'm emotional about web scraping (because it has been my bread and butter during tough times), and so, I'm very happy to see a scraping library so wildly popular, hence, regardless of whether you praise or rant about it, of course, I'm going to try it ... and already reading the docs :P but thought let's see what the community thinks of it. First impression is "I've not missed anything."


r/scrapingtheweb • • Jul 19 '24

[Best proxy sites?] Oxylabs vs Bright data vs IPRoyal comparison. What should I try first?

26 Upvotes

Hey folks. This is my first journey into paying for enterprise residential proxy plans for data scraping as a side gig. What's considered the gold standard proxies these days? My current vendor only provides data center proxies and those get flagged up every few days.

What do you all suggest I battle test first?

13 votes, Jul 26 '24
2 Oxylabs
8 Bright data
0 IPRoyal
3 Other

r/scrapingtheweb • • Feb 12 '25

Best Residential Proxy Providers if just a single IP Adress is needed?

25 Upvotes

I'm trying to access the TikTok Rewards Program, which is only available in select countries, including Germany.

I’ve looked into providers like Bright Data, IPRoyal, and Smartproxy, but their pricing models are a bit confusing. Many of them seem to require purchasing IPs in bulk, which isn’t ideal for me.

Since I only need to imitate a real TikTok user, I just need a single residential IP (deticated or sticky, not changing to often within a short timeframe).

Does anyone have recommendations for a provider that offers a single residential IPs without requiring bulk purchases?

(I know this subreddit is mostly for web scraping, but r/proxies seems inactive, so I figured this would be the best place to ask.)


r/scrapingtheweb • • Jun 17 '26

Free MCP That Scrapes Literally Anything

Post image
21 Upvotes

r/scrapingtheweb • • May 28 '26

Help How to perform web scrapping using Claude?

21 Upvotes

Guys, I have a digital marketing agency and I am looking for first client. I need to perform web scrapping for outreach through cold emails or WhatsApp. How can I do it with Claude? Is there a skill or a connector that can make this process efficient? Guys please help a first client would mean a lot for me!


r/scrapingtheweb • • Apr 17 '26

300M Scraping Fine 😳 But they will get 00M 🤣

20 Upvotes

Not supporting illegal or unethical scraping, but Spotify vs Anna's Archive case is a little disheartening. Yes, they are data pirates, and until they are caught the ruling from the courts is completely useless. I don't see a point of such cases then. Isn't it just to thwart the scraping? Irony is that Spotify has no problem in using the scraped web data (hard work of writers and other creative people) to create huge libraries of AI-generated music, but they have a problem when someone feasts upon their database. Honestly, the lines between ethical and unethical data scraping is blurring.


r/scrapingtheweb • • 15d ago

How do you find out your scraper is silently returning bad data?

19 Upvotes

Not talking about crashes — those are easy. I mean the runs that finish "successfully" but the site changed a class name and now `price` is empty in 90% of rows, or the item count quietly drops from 500 to 40.

The last time this happened to me, I only found out when someone downstream asked why a dashboard looked off. By then it had been broken for a couple of days.

Curious what everyone does:

- Do you validate output against a schema after every run?
- Track item counts / null rates over time?
- Diff the HTML to catch layout changes?
- Or just wait for someone to complain? 😅

If you maintain a lot of scrapers (10+), how much time a week goes into noticing and fixing breakages?

Would love to hear setups — homegrown scripts, Great Expectations, Grafana alerts, whatever works.


r/scrapingtheweb • • 4d ago

Discussion Open source vs commercial scraping tools: what actually survived production for you?

17 Upvotes

There are plenty of “best scraping tool” lists, but I am more interested in real experience after using them for a while.

For complex scraping projects, what are you actually using today?

  • Python/JS + Playwright/Scrapy/Crawlee

  • Open-source/stealth browsers

  • Scraping APIs or cloud browsers

  • Low-code/AI scraping platforms

  • Enterprise scraping software

  • Managed Data Services

What worked well, what became painful, and what made you switch?

If possible, mention roughly what scale you were running and whether the biggest issue was blocking, browser management, maintenance, cost or data quality.

Please share genuine experience, not paid/affiliate recommendations. If you are associated with a tool, just mention it so others can judge the feedback fairly.

The goal is to collect honest experiences that could actually help someone choosing a production setup.


r/scrapingtheweb • • Oct 03 '26

Remote job opening

17 Upvotes

We’re looking for a strong developer to work with us on a project

The work is hands-on and primarily focused on web crawling, catalog discovery, extraction, transformation, and structured data.

Tech stack / experience we’re looking for:
• Python
• Playwright / web crawling and scraping
• ETL and data pipelines
• REST APIs
• JSON / JSON-LD
•C#, java
• Catalog discovery and extraction across platforms such as Coursedog, Acalog,catalog systems
• Regex, HTML parsing, URL discovery, caching, and data transformation
• Git / GitHub
• Cloud storage experience is helpful
• Experience working with LLM-assisted extraction pipelines is a plus

You should be comfortable understanding an existing codebase quickly, debugging extraction issues, improving discovery logic, and working independently.

Commitment: approximately 5–6 hours per day
Budget: up to $2,000
Work mode: Remote

We’re looking for someone who can start quickly and contribute directly to the codebase rather than someone who needs extensive onboarding.

If interested, please DM me with:

  1. Your GitHub / portfolio

  2. Relevant Python + scraping/ETL experience

  3. education-data experience

  4. Your availability


r/scrapingtheweb • • Jan 10 '26

how do you guys actually choose proxy providers?

17 Upvotes

hey everyone, currently a student trying to get into webscraping for a data project and honestly... im completely lost lol. thought the hard part would be writing the code but nah its actually finding decent proxies that dont suck

every provider i look at has these insane landing pages saying "99.9% success rates!!" and "millions of clean ips!!" but when i look around a bit these all seem to be overhyped marketing bs. the more i read the more confused i get about whats actually real:

  • the reseller thing - is it actually true that most "new" providers are just reselling from the same massive pools?? like if thats the case arent those ips already burnt before i even use them

  • big players vs niche players - should i go with the big names who seem to have literally everyone using their pools, or niche players with actual private pools... but then again are there even any real private pools out there??

  • testing proxies - when it comes to testing what factors should i even look for?? heard something about fraud scores floating around, is that something i should actually check

  • hybrid proxies - also heard about this hybrid proxy thing, do they actually work on tough sites like cloudflare and akamai or is it just another gimmick

at this point i just want to learn from actual scrapers who've been doing this for a while (no marketing bs please). when youre selecting a provider what should i look out for in proxy testing?? which factors do you actually consider before committing to one

any advice would be super helpful, feeling pretty overwhelmed rn 😅 and no fake claims from proxy sellers here please


r/scrapingtheweb • • Oct 07 '26

Tools / Library I built a YouTube scraper for Python (no API key, no browser)

16 Upvotes

r/scrapingtheweb • • May 23 '26

Scrapy Honest Brave Search API review after 3 months of using it in production

16 Upvotes

Spoiler it went pretty bad. We use Brave Search API as the retrieval layer for a research agent for about three months. Most reviews I found were either surface-level or written by someone who used it for a weekend, so I decided to put down a few notes on it

The independence from google and Bing is valuable right now especially after Microsoft killed the bing api I last year, Brave is one of very few options left that crawls its own index

Where it frustrated me

400 character query limit becomes a problem when your agent generates verbose searches - that impacted us pretty quickly and we saw our free tier going from 5,000 queries/month to $5 in credits which is fine for production but very annoying when you're burning through queries during eval runs

For bigger structural limitation for my use case, Brave gives you search results and not full page content so if your agent needs to read the article or report, you need a separate scraping layer on top. I’m trying to test solutions and turnarounds to this, as well as any alternative that may be more pertaining to my use case.

Would appreciate any input!

Quick update: we still use brave as a search source, but we replace the scraping layer with Firecrawl and that solved most of the issues we were running into with fetching and parsing pages, and the whole pipeline is much simpler


r/scrapingtheweb • • May 18 '26

Open sourced CLI to reverse engineer APIs from any website

16 Upvotes

I’m open-sourcing Opensteer -- a lightweight CLI runtime that lets AI agents control your real browser, inspect network requests, and turn browser workflows and reverse engineered APIs into reusable agent tools.

The idea is that an agent can use Opensteer to drive your local Chrome through DOM automation, observe how a site loads data, and then write repeatable scripts or functions for the parts that should be deterministic. Those functions live in your project directory, so future agents running from that directory can call them like tools.

It runs locally, connects to your actual browser session, and is meant for workflows where plain HTTP scraping is not enough: logged-in pages, JS-heavy apps, visual debugging, API discovery, and repeatable browser automation.

Feel free to try it out, and would appreciate any feedback.


r/scrapingtheweb • • Dec 22 '25

What's your go-to web scraper for production in 2025?

16 Upvotes

Some libraries/tool options:

  1. Scrapy

  2. Playwright/Puppeteer

  3. Selenium

  4. BeautifulSoup + Requests

  5. Custom scripts

  6. Commercial tools (Apify, Bright Data, etc.)

  7. Other


r/scrapingtheweb • • Sep 13 '26

Scraping for Newbies?

15 Upvotes

I am brand new to scraping Elder-mellennial here (almost gen x, but don't hold it against me)... I realize I am going to be left behind in my field it I don't learn how to use Ai to my advantage. Any tips for newbies? I am looking at Apify and feeling overwhelmed at the number of actors who can do what I want them to do. How do I know which one to choose? Are there reviews like amazon? Help me out Please?


r/scrapingtheweb • • Jun 09 '26

Tools / Library ChromiumFish - A fingerprint-hardened Browser for Scraping

15 Upvotes

Okay, as you might already have guessed, I'm a professional web scraper. I've pretty much exhausted most open-source stealth browsers, and the paid ones aren't a very economical solution when you want to scrape at scale and have to deal with heavy bot mitigation.

The issue I repeatedly faced with most tools is that almost all of them run on Linux and try to pretend they aren't, which might work for a while, but given enough samples, it's pretty obvious to large bot-mitigation infrastructures, and they start blocking or at least push a bit harder.

So I tried to build a browser myself, which does a few things:

  • Randomize profiling based on the given seed

  • Match the timezone based on the proxy exit IP

  • Can outsource canvas to a different machine ( from the target OS )

This canvas outsourcing is what really makes it different, because pretending isn't enough. Some of the toughest bot-mitigation infrastructures probe deep into the canvas, a depth that a simple patch can't bypass on a different OS. But this is optional and should only be done if everything else fails.

Repo: https://github.com/arman-bd/chromiumfish

This is available for both Python and JavaScript via pip and npm, with a Playwright wrapper. For other tools, feel free to download the browser binaries from the release or build them yourself.

Right now, this is an early release. Any feedback is highly appreciated 🙌


r/scrapingtheweb • • 12d ago

Medicine data collected from website through crawler

14 Upvotes

Is there anyone need medicine related database I have informations of 50k medicines....thinking of what to do with them.. basically I scraped them from websites with own created crawller..........


r/scrapingtheweb • • May 05 '26

Help Best Alternatives to Brave Search API?

14 Upvotes

Hello community, need some inputs. I’ve been using Brave API since 2022 but after the recent updates it feels less reliable and a bit annoying to work with. I’m currently redesigning my search layer for a new app and debating whether to stick with APIs or move toward a more custom setup with caching and controlled queries. What’s working well for you guys right now?


r/scrapingtheweb • • Jun 30 '26

Web Scraping freelance

13 Upvotes

Hi everyone! Basically, I’d like to give freelancing a go. I learnt Python a while back and was thinking of brushing up on my skills and heading in that direction.
What are your thoughts on this in general? Is it still a viable option, is there plenty of work, and which programming language should I use?


r/scrapingtheweb • • Feb 24 '26

How do you manage proxies and avoid IP bans for web scraping?

13 Upvotes

Looking for recommendations on tools or libraries that make proxy management less of a headache in web scraping.

Ideally something that:

  • rotates proxies automatically with sane retry/backoff

  • supports residential IPs and sticky sessions for logged‑in stuff

  • has at least basic stats (success rate, status codes, captcha hits, etc.)

  • isn’t completely sketchy from a legal/compliance angle

What are people actually using these days that works well for you?