r/scrapingtheweb • • 5h ago

Help Easiest platform for scraping tasks

2 Upvotes

Other than freelancer and upwork

What's the best platform after them in popularity, easy to start from the beginning and has automation and scraping tasks periodically?


r/scrapingtheweb • • 5h ago

Discussion Did a one time pull of 8,000 Google results for a research project. Was paying for an API the right call?

1 Upvotes

Needed Google results for about 8,000 queries for a class project, and scraping it myself with requests and then headless Chrome hit captchas within the first few dozen, so I paid for a SERP API and the whole pull came in under $20. I've got a second project coming that could be 50,000 queries. At what volume does running your own setup start to beat paying per query?


r/scrapingtheweb • • 2d ago

Playwright / Puppeteer Free eBay Scraper, Thoughts?

7 Upvotes

Hi Guys. I am new to this sub and hope I don't get banned but a few months ago I posted a free github link to an eBay scraper and just made a new one that uses stealth puppeteer, discord alerts, and minor price logic for what I have used to be a very effective scraper if your trying to snipe something at a certain price. Here's the link and let me know if you have any questions.

https://github.com/emmanuel-a-7/ebaydealfinder-1.1

P.S. Yes I did use AI assistance but I feel as its warranted these days


r/scrapingtheweb • • 3d ago

I want to scrape Insta reels for a project (Help needed 🙏)

8 Upvotes

Hi everyone,

I am working on a project where I need to collect public Instagram Reels and transcribe their audio into text.
Since Instagram is aggressive with rate limiting and account bans, I want to make sure I set up a clean, reliable pipeline without risking my personal account or IP address.
So plzz suggest some methods or tools that i can use to scrape without my account getting banned 🙏🙏.
It would be a great help .


r/scrapingtheweb • • 3d ago

I just released an easy web scraper : It also works with social medias -> some testers here ?

Thumbnail
youtu.be
0 Upvotes

r/scrapingtheweb • • 4d ago

Free X image grabber?

3 Upvotes

I'm looking for a way to mass download someones media imagine posts on x, all of the grabbers i could find online either don't work, or only let you grab like 10 imgs then ask for a ton of money. does anyone know one that works and is free?


r/scrapingtheweb • • 4d ago

I built an API that extracts contact information from websites

2 Upvotes

Hey everyone!

I'm a backend developer, and I recently built and published a Website Email and Contact Scraper API on RapidAPI.

The idea is simple: provide a website URL and the API extracts publicly available contact information, including:

  • Email addresses

  • Phone numbers

  • Physical addresses

  • Social media links

I built it with Python and FastAPI, with MongoDB caching and an asynchronous job-processing system.

I'm now looking for developers and businesses who might find this useful. I'd particularly appreciate feedback on the API's functionality, response structure, and pricing.

You can check it out here:

https://rapidapi.com/malik78479/api/website-email-and-contact-scraper

I'm the developer behind this project, so I'd genuinely appreciate honest feedback!


r/scrapingtheweb • • 5d ago

Discussion Open source vs commercial scraping tools: what actually survived production for you?

18 Upvotes

There are plenty of “best scraping tool” lists, but I am more interested in real experience after using them for a while.

For complex scraping projects, what are you actually using today?

  • Python/JS + Playwright/Scrapy/Crawlee

  • Open-source/stealth browsers

  • Scraping APIs or cloud browsers

  • Low-code/AI scraping platforms

  • Enterprise scraping software

  • Managed Data Services

What worked well, what became painful, and what made you switch?

If possible, mention roughly what scale you were running and whether the biggest issue was blocking, browser management, maintenance, cost or data quality.

Please share genuine experience, not paid/affiliate recommendations. If you are associated with a tool, just mention it so others can judge the feedback fairly.

The goal is to collect honest experiences that could actually help someone choosing a production setup.


r/scrapingtheweb • • 5d ago

Python Understanding SHEIN's Bot Protection and an Open-Source Scraper NSFW Spoiler

6 Upvotes

SHEIN has two different types of redirects you should understand before scraping its product pages. We noticed this while testing our Python scraper: treating them as the same problem leads to unnecessary CAPTCHA-solving attempts.

Understanding SHEIN's bot protection

SHEIN uses two main risk redirects:

  • /risk/challenge:: an interactive CAPTCHA challenge. It starts with an "I am human" checkbox and can escalate into either a 3×3 image grid or a puzzle requiring users to click icons in a specific order.

  • /risk/action/limit: a rate-limit response. This isn't a CAPTCHA, so there's nothing to solve.

The scraper handles these situations differently. For /risk/challenge, it clicks the initial checkbox (free) and, if necessary, sends the puzzle to 2Captcha. It supports both GridTask and CoordinatesTask challenges.

The --risk-challenge-rounds option controls challenge rounds, while --max-solves limits paid CAPTCHA solutions per run (default: 8). If SHEIN returns /risk/action/limit, the scraper exits with code 3 instead of wasting money on a solver. With --rate-limit-cooldown 300 it waits five minutes and retries once.

How well it works: on 2026-09-30 we tested five fresh US browser profiles. Three passed the challenge, each on its third round, because SHEIN usually rejects the first correct answer or two. The other two were rejected on every answer, including correct ones: SHEIN had flagged the profile itself, and only a different profile helped.

Where SHEIN product data comes from

Search and category pages expose product data through window.gbRawData, while individual product pages use Schema.org JSON-LD. The scraper uses different parsing methods depending on the page.

You get one row per product: title, brand, current and original price, discount, rating, review count and stock flags. It supports product searches, category listings and individual product URLs. It collects review counts and ratings, but doesn't extract individual customer review texts.

Installing and running the scraper

The implementation uses Python 3.9+ and Playwright with Chromium. Clone the SHEIN scraper repository on GitHub (https://github.com/2scraper/shein-scraper) and install the dependencies:

git clone https://github.com/2scraper/shein-scraper.git
cd shein-scraper

python3 -m venv .venv
source .venv/bin/activate

pip install -r requirements-playwright.txt
playwright install chromium

cp .env.example .env

For regular runs, set SHEIN_CDP_ENDPOINT and TWOCAPTCHA_KEY in your .env file.

To search for products:

python3 playwright_scraper.py \
  --query "summer dress" \
  --max-results 20

You can also scrape an individual product page:

python3 playwright_scraper.py \
  --url "https://us.shein.com/dsbayvkj-p-33704388.html"

Exporting and comparing results

Results are saved as JSON by default (add --format csv --out products.csv for CSV), together with a .meta.json file containing run information, including status and CAPTCHA solves spent.

There's also a diff_runs.py script for comparing two JSON exports:

python3 diff_runs.py previous.json current.json

It identifies newly added, removed and repriced products, which is handy for tracking prices over time.

Limitations

A few things to consider:

  • Currently targets us.shein.com only; other regional domains haven't been verified. Prices are in USD.

  • Only about the first 20 products of a listing are verified. Whether scrolling loads more hasn't been confirmed.

  • sku is shared across colours and sizes, so you get one row per product. On a product page the price is the lowest variant price.

  • --sort is recorded in the results but not sent to SHEIN yet.

  • Scraper API mode doesn't support scrolling and retrieves only one page per run.

  • Selenium runs a local Chrome only: it can't use a remote browser profile or a proxy with a password.

I'm curious how others handle e-commerce scraping when a site uses both interactive CAPTCHA challenges and profile-based rate limiting.

Disclosure: I work with the team behind this scraper (2scraper (https://2scraper.com/scrapers/shein-scraper/)) and 2Captcha.


r/scrapingtheweb • • 5d ago

Sharing 4 Social Media scrapers I built for Telegram, Facebook, Instagram and TikTok any suggestions welcome

Thumbnail
reddit.com
1 Upvotes

r/scrapingtheweb • • 5d ago

30% off A-Parser and Premium Mix for the 11.11 sale

Post image
1 Upvotes

r/scrapingtheweb • • 6d ago

Help Help needed!

Thumbnail reddit.com
2 Upvotes

Im vibecoding my project and im having a little trouble scraping one website in particular. Seems like they are using DataDome + Imperva.

Claude code and ChatGPT are not being able to get past it.

Any one knows how to do it?

Thanks


r/scrapingtheweb • • 6d ago

Proxy / IP Issue Mobile proxies on jobs that didnt need them

1 Upvotes

Went through our usage and most of the mobile traffic was catalog pages

Catalog pages!! Residential eats those all day

Somebody set mobile as the default in a config months ago and no one questioned it. Now mobile only comes out when a target clearly hates home ranges or ad ops wants to see what a phone network gets served. Both pools sit on the same 2extract account so the switch was literally one line, which is kinda embarrassing given how long it took us to notice


r/scrapingtheweb • • 6d ago

Python If I can scrape any website where the best place to find clients?

9 Upvotes

r/scrapingtheweb • • 7d ago

Help Where can I find companies or people looking to buy custom datasets?

Thumbnail reddit.com
4 Upvotes

r/scrapingtheweb • • 9d ago

Help Collecting and Comparing Scraper APIs

9 Upvotes

Hi all!

I'm learning about various web scrape API services to use for building an open-model Art Historian agent which can identify artwork, artists, artistic movements and periods, and converse identification AI agent. I've leaned heavily on free-tier triels that provide enpoints to Google reverse-image and lens-search, including ScrapeBadger, Tavily, Apify, and OxyLabs.

What has everyone else been using, and what has been your experience with it? What are your recommendations for long-term value?


r/scrapingtheweb • • 8d ago

Scrapers to identify new businesses and expanding businesses

5 Upvotes

Hey Scraper community,

I have Apify connected via MCP and I'm looking to see if it's possible to put together several actors that allow me to see companies that recently launched new advertising campaigns on social media and google. In conjunction I would also like a secondary search to scrape google maps to identify new locations or physical businesses opening.

The ideal businesses I am trying to find are those that are starting new online marketing campaigns AND opening new physical locations.

Has anyone ever done this before?


r/scrapingtheweb • • 10d ago

Discussion Do companies still pay for ready-made datasets in the AI era?

7 Upvotes

I was looking at sites like:

https://www.scrapehero.com/store/shop/
https://locationlists.com/data

They sell ready-made datasets for things like store locations, eCommerce, pricing etc.

With AI making scraping much easier now, I was wondering what actually makes a dataset worth paying for today.

Is it freshness, accuracy, regular updates, historical data or simply saving the time of collecting and maintaining it yourself?

Would you still pay $50-$100 for a good one-time dataset, or is the real value now in continuously updated data?


r/scrapingtheweb • • 10d ago

Benchmarking Zero-DOM HTTP/2 vs Headless Chrome for Contact & Tech Stack Scraping (1,500 domains/min in 128 MB RAM)

2 Upvotes

Hey everyone,

When scraping contact data (emails, direct phones, social profiles, ad pixels) from thousands of business websites, the default approach is usually launching Puppeteer/Playwright or Selenium.

While headless browsers work, they hit severe scaling bottlenecks:

  • Memory footprint: Headless Chrome easily consumes 200–500 MB RAM per page, meaning a 10-concurrency runner needs 4–8 GB RAM.

  • Throughput: Between JS rendering, CSS parsing, and font downloads, speeds rarely exceed 5–15 domains/minute.

  • Unreachable sites: Dead or expired domains tie up browser contexts and cause CPU spikes on timeouts.

The Zero-DOM Alternative

I built and tested a pure HTTP/2 streaming pipeline that operates under a strict 128 MB RAM budget with an active ~5 MB V8 heap footprint: 👉 Open on Apify Store: https://apify.com/Galhali/smart-contact-tech-enricher

Architectural Breakdown:

  1. HTTP/2 Streaming Pipeline: Instead of buffering full multi-megabyte HTML documents into DOM trees (cheerio/JSDOM), responses are processed through chunked regex stream scanners. It carries over 4 KB buffers across chunk boundaries so regex matches are never split in half.

  2. DNS & MX Verification Before Crawling: Before firing HTTP requests, we resolve domain MX records (aspmx.l.google.com, etc.). If a domain is an NXDOMAIN or dead host, it halts immediately without wasting bandwidth or billing compute time.

  3. Cloudflare Email Deobfuscation: Extracts data-cfemail hex payloads and decodes them via XOR keying without needing any browser execution.

  4. 100-Point Confidence Scoring: Raw regexes on web pages pull in noise (image assets like [email protected], retina subdomains, spam-trap mailtos). We implemented a 100-point heuristic scoring system that rewards matching root domains and penalties for CDN multipliers and generic webmail.

  5. USA State Corporate Registry Links (sosUrl): Generates direct search URLs for all 50 US State Secretary of State databases (Sunbiz Florida, California BizFile, Texas CPA, Delaware, NY DOS) alongside CMS Modulo-10 NPI numbers for healthcare leads.

  6. 1-Click Chaining with Datasets: Can be chained directly with raw Google Maps scraper outputs (e.g. compass/crawler-google-places) by passing the datasetId—no CSV export/import needed.

Curious if others here have moved away from headless browsers for contact enrichment, or if you're still relying on Puppeteer clusters? Happy to discuss edge cases and throughput optimizations!


r/scrapingtheweb • • 10d ago

Open-source guide to identifying and handling 38 CAPTCHA types

2 Upvotes

r/scrapingtheweb • • 11d ago

help with google link scraping

7 Upvotes

I'm trying to build a Python scraper in Jupyter Notebook to collect Hyderabad IT company leads and export them to Excel.

What I need:

  • LinkedIn company profile URLs only (linkedin.com/company/...)

  • Company name

  • Category

  • Employee range

  • Company website

  • Hyderabad-based companies

My current setup:

  • Provider: SerpApi

  • Search engine: Google

  • Access: API

  • Python + Jupyter Notebook

  • Excel export using pandas/openpyxl

My exact search query:

site:linkedin.com/company Hyderabad technology company "11-50 employees"

This exact query works manually in Google. My browser shows actual company pages such as Zodeck Technologies, Zeominds IT Solutions, DE FINGERS, and Techeduxon, with company size and Hyderabad location in the snippets.

However, when I send the same query through SerpApi, I get unrelated results like RevenueBase, Glassdoor, Crunchbase, Facebook posts, and YouTube videos.

The API returns HTTP 200, status Success, and no search error. It reports 51 results, but the first 10 results are mostly irrelevant.

I also tried:

  • Google through SerpApi

  • Bing manually

  • Different queries and employee ranges

  • Filtering results to LinkedIn company URLs

  • Checking the API response metadata

Filtering doesn't solve the problem because the actual company pages I need aren't appearing in the API results.


r/scrapingtheweb • • 11d ago

I built a tool that turns "I need this data from that website" into a dataset with a receipt for every value

Thumbnail reddit.com
1 Upvotes

r/scrapingtheweb • • 12d ago

LinkedIn job posting scrapper which also gets job poster's name

Thumbnail reddit.com
3 Upvotes

r/scrapingtheweb • • 12d ago

How to bulk-check DMARC, SPF and DKIM across a domain list (approach + gotchas)

1 Upvotes

If you're scraping lead lists or doing B2B outreach, email security records are a useful signal per domain. Here's the approach I use to check them at scale:

  1. Normalise inputs first. Lists arrive as full URLs, emails, mailto: links and bare domains. Strip each to the host, lowercase it, drop www, and dedupe before doing any lookups.

  2. DMARC: TXT lookup at _dmarc.<domain>. Parse p=, sp=, pct= and rua=/ruf=. rua can hold several comma-separated mailto: URIs, sometimes with size suffixes like !10m. No record at all is a finding, not an error.

  3. SPF: TXT at the root starting v=spf1. Watch for more than one SPF record (that's a permerror and breaks SPF entirely), ~all vs -all, and the 10-DNS-lookup limit from nested includes.

  4. DKIM: you can't list selectors, so you have to guess. Common ones: google, selector1/selector2 (Microsoft 365), k1 (Mailchimp), s1/s2, default, mail, plus host-specific ones like hostingermail. Checking around 60 common selectors catches most setups.

  5. Retry on SERVFAIL and timeouts, ideally across more than one resolver, or you'll log false "missing record" results.

Gotchas I hit: duplicate SPF records are more common than you'd think, p=none with no rua is DMARC in name only, and a few domains return different TXT answers from different resolvers.

If you'd rather not build it yourself, I packaged this as an Apify Actor (pay per domain; invalid and duplicate inputs are free): https://apify.com/ausbizstack/b2b-email-security-dmarc-spf-auditor

Happy to answer questions on edge cases.