r/scrapingtheweb • • Jan 03 '26

HIRING - Bot Detection Engineer

5 Upvotes

POSITION IS FILLED, THANKS FOR YOUR INTEREST

Hi all

We're looking for a bot detection expert to join our company.

This is a remote position, work whatever hours you want, whenever you want.

The expectation is you do what you say you're going to do, and deliver excellent work.

We're a nice company, and will treat you well. We expect the same in return.

Please contact me by DM to discuss. Also happy to answer any questions here.

Thanks.


r/scrapingtheweb • • Jan 04 '26

The hidden ChatGPT API

Post image
0 Upvotes

r/scrapingtheweb • • Sep 13 '26

Help How to scraping 25.000 links daily

0 Upvotes

Hi, everyone I'm new to doing web scraping thinks I was asking Which is the Best way to do 25.000 links daily from the same website to get specific information from every link? I will thank you for any help or ideas, thanks.


r/scrapingtheweb • • Jan 26 '26

Best scraper for finding illegal image violation

1 Upvotes

Hi guys i'm building a software for finding an illegal image violation.

I mean, if someone pubblish a photo and someone stole this photo, the software find this violation.

I want more power and I need some scraper who can scan at least 100.000 website.

Any suggestion?


r/scrapingtheweb • • Jan 02 '26

Looking for Strong Web Scraper to Build an Early-Stage Product (Equity-Based, Startup / Entrepreneur Interest)

0 Upvotes

Hi everyone,

I’m a Full-Stack Developer working on building a real product with the goal of starting a company and entering the entrepreneur / startup journey.

I already have a clear idea and product development has started. To move faster and build this properly, I’m looking to collaborate with strong technical people who are interested in building a product from scratch and learning startup execution hands-on.

This is not a paid job.

This is an equity-based collaboration for people who genuinely want to build a real product and be part of a startup journey from the beginning.

Who I’m Looking For

  1. Data Web Scraper

Strong experience in web scraping

Able to build reliable, maintainable scraping systems

Understands data accuracy, consistency, and real-world challenges

Thinks beyond quick scripts and hacks, proxy , ip rotation

What This Collaboration Is About

Building a real product, not just discussing ideas

Working together as early team members

Learning and executing in a startup / entrepreneur environment

Shared ownership and equity-based growth

High responsibility and hands-on contribution


r/scrapingtheweb • • Oct 15 '25

I need help! I bypassed an iPhone 13 P.Mx that I found only that now it won't let me access Apple accounts, they told me that there were proxies for that :/ someone help me!! (Use Iremoval pro)

0 Upvotes

r/scrapingtheweb • • Dec 28 '25

I Can bypass akamai datadome and cloudflare with my solution

Post image
0 Upvotes

DnD intuitive builder , automatically injects mitigation codes.

I'm selling or offering freelancer work if any one needs


r/scrapingtheweb • • Aug 21 '26

Tools / Library I built an API after seeing users make the same search repeatedly across TikTok, Instagram, and YouTube

0 Upvotes

I work on SocialCrawl, a unified API for pulling data from social platforms.

I realised the same problem kept coming up with our heavy users: they wanted to run the same search on TikTok, Instagram, and YouTube, etc., but had to call each platform separately.

The search itself was OK but repeating the requests, managing pagination, and normalising different response shapes was still very inefficient.

So I built Prism to handle that repetitive part. You send one API query, and it runs across relevant social platforms and returns a single structured response, instead of making you repeatedly wire together the same calls.

I’m curious whether you've encountered a similar problem when scraping social data. Would appreciate any feedback as well. Details: https://www.socialcrawl.dev/prism


r/scrapingtheweb • • Jul 29 '26

I got tired of LLMs hallucinating on complex HTML tables, so I built a smarter Python parser (handles rowspan/colspan)

Thumbnail
reddit.com
0 Upvotes

r/scrapingtheweb • • Aug 03 '26

Would you pay for a visual editor with code capability?

0 Upvotes

Hey everyone,

Before I start, I don't want to advertise anything. Just asking for feedback and see if people like my idea.

I'm working on a visual web automation tool focused on web scraping and web automation, and I wanted to get some honest feedback on the idea and the pricing.

The basic premise is a local desktop app where you can build robust scrapers visually instead of coding everything from scratch. It has a script recorder similar to Playwright's codegen, and another tool that handles things like asynchronous banners, random pop-ups, and shifting page layouts.

You aren't restricted to just the UI, either. There's a full text editor built in if you need to write custom logic. The app just compiles all your visual instructions directly into standard Playwright code, for you to use it anywhere, not only inside the platform. It also handles the standard CSV imports and exports for your data.

My main question is: does this solve a big enough headache in your workflow that you'd actually pay for it? If so, would you consider $149 a fair price for an unlimited, lifetime license?


r/scrapingtheweb • • May 22 '26

GitHub - h34351449-del/somesites: It is a html code cracker it get html codes

Thumbnail github.com
0 Upvotes

Somesites code (ONLY ON TERMUX)


r/scrapingtheweb • • Mar 25 '26

Vibe hack the web and reverse engineer website APIs from inside your browser

0 Upvotes

Most AI web agents click through pages like a human would. That works, but it's slow and expensive when you need data at scale.

We built on the core insight that websites are just API wrappers. So we took a different approach: our agent monitors network traffic and then writes a script to pull that data directly in seconds and one LLM call.

The data layer is cleaner than anything you'd get from DOM parsing not to mention the improved speed, cost and constant scaling unlocked.

The hard part of raw HTTP scraping was always (1) finding the endpoints and (2) recreating auth headers. Your browser already handles both. So we built Vibe Hacking inside rtrvr.ai's browser extension for users to unlock this agentic reverse-engineering in seconds and for free that would normally take a professional developer hours.

Now you can turn any webpage into your personal database with just prompting!


r/scrapingtheweb • • Aug 07 '26

Scrapy How hard is it to scrap eBay last sold data

0 Upvotes

Hello everyone I am working on an utility tool (tcg scanner like collectr and price charting) and I need to fetch daily prices of 8000 cards on eBay with with different variations such as Raw , PSA 8 9 10 .. which makes it 40000 request , did anyone work on something like this or is there any services that gets data from eBay with this amount ?


r/scrapingtheweb • • Jan 11 '26

First of a kind vibe scraping platform leveraging Extension to control cloud browsers

0 Upvotes

Most of us have a list of URLs we need data from (government listings, local business info, pdf directories). Usually, that means hiring a freelancer or paying for an expensive, rigid SaaS.

We built rtrvr.ai to make "Vibe Scraping" a thing.

How it works:

  1. Upload a Google Sheet with your URLs.

  2. Type: "Find the email, phone number, and their top 3 services."

  3. Watch the AI agents open 50+ browsers at once and fill your sheet in real-time.

It’s powered by a multi-agent system that can take actions, upload files, and crawl through paginations.

Web Agent technology built from the ground:

  • 𝗘𝗻𝗱-𝘁𝗼-𝗘𝗻𝗱 𝗔𝗴𝗲𝗻𝘁: we built a resilient agentic harness with 20+ specialized sub-agents that transforms a single prompt into a complete end-to-end workflow. Turn any prompt into an end to end workflow, and on any site changes the agent adapts.

  • 𝗗𝗢𝗠 𝗜𝗻𝘁𝗲𝗹𝗹𝗶𝗴𝗲𝗻𝗰𝗲: we perfected a DOM-only web agent approach that represents any webpage as semantic trees guaranteeing zero hallucinations and leveraging the underlying semantic reasoning capabilities of LLMs.

  • 𝗡𝗮𝘁𝗶𝘃𝗲 𝗖𝗵𝗿𝗼𝗺𝗲 𝗔𝗣𝗜𝘀: we built a Chrome Extension to control cloud browsers that runs in the same process as the browser to avoid the bot detection and failure rates of CDP. We further solved the hard problems of interacting with the Shadow DOM and other DOM edge cases.

Cost: We engineered the cost down to $10/mo but you can bring your own Gemini key and proxies to use for nearly FREE. Compare that to the $200+/mo some lead gen tools charge.

Use the free browser extension for login walled sites like LinkedIn locally, or the cloud platform for scale on the public web.

Curious to hear if this would make your dataset generation, scraping, or automation easier or is it missing the mark?


r/scrapingtheweb • • Jun 07 '26 All Seeing Upvote

Webclaw - Open source Rust scraper with protocol-level TLS/JA4 bypass

2 Upvotes

hey,
wanted to share webclaw, an open source rust scraping engine designed for speed and bypass. we just crossed 1,300 stars on github and wanted to get some feedback from the community

features:

  • protocol-level TLS/JA4 bypass: handles advanced bot detection without headless browser overhead

  • rust-powered: memory safe and fast for high concurrency jobs

  • protocol mimicry: precise control over headers, TLS fingerprints, and HTTP/2 settings

  • zero fluff: just a direct utility for devs scraping at scale

if you're tired of getting blocked by cloudflare/akamai or need better performance than python wrappers, check out the code

https://github.com/0xMassi/webclaw


r/scrapingtheweb • • Aug 24 '26

I built an automated pipeline tracking tax deed & sheriff auctions across all 50 states. Here’s the tech stack & how we unmask LLC owners in <800ms

0 Upvotes

Hey r/SideProject,

I spent the last 6 months building **PropertyIntel** ([https://property.vectorfeedhq.com](https://property.vectorfeedhq.com/)), an automated data pipeline in Python that monitors 3,143 US counties and 8 major auction portals for off-market tax deed auctions, sheriff sales, and municipal code liens.

💡 The Problem We're Solving

Legacy data providers (CoreLogic / ATTOM) charge $500–$2,000/mo on rigid 12-month annual contracts for raw, unformatted data dumps. Meanwhile, mid-tier tools (PropStream, BatchLeads) charge $99/mo+nickel−and−dime users with $99/*mo*\+$0.12–$0.18 per skip-trace credit.

When a distressed property is owned by an LLC (e.g. `SUNBELT HOLDINGS LLC`), traditional skip-tracers return dead consumer data because they search consumer credit headers, not state corporate registries.

We automated the entire public records ingestion fleet to deliver biweekly formatted Excel rosters starting at $25/mo.

🛠️ The Tech Stack

* **Scraper Fleet:** Python 3.13 + headless Playwright (for dynamic JS tables like Bid4Assets/GovEase) + `pdfplumber` (for raw 90-page county PDFs). * **Address Normalization:** Custom tokenizer following **USPS Publication 28** standards (`Avenue` \-> `AVE`, `Suite` \-> `STE`). * **Deduplication:** Deterministic **SHA-256 hashing** (`county + normalized_address + apn`) stored in SQLite / PostgreSQL (`ON CONFLICT DO UPDATE`). * **OSINT Enrichment:** Async scrapers querying state Secretary of State registries to unmask LLC registered agents and managing officers in <800ms at zero per-record cost. * **50-State Statutory Redemption Engine:** Automatically computes statutory redemption periods, penalty yields (e.g. Texas 25% penalty return, Florida 0-day instant deed finality), and IRS 120-day compliance flags. * **Frontend & Landing Page:** Modern Glassmorphism CSS + vanilla JS (hosted on Vercel). * **Monetization & Billing:** Stripe Checkout + HMAC-SHA256 verified webhooks ($25 Local *$*40 Regional $99 National).

🚀 Key Takeaway on Bootstrapping Data Products

You don’t need an enterprise data team to compete with legacy data giants. By automating scrapers in Python, standardizing data via USPS Pub 28, and delivering via automated biweekly email rosters, our ongoing infrastructure cost is under $40/mo.

You can inspect our live catalog and download a free 10-state sample deal sheet here: 👉 [**https://property.vectorfeedhq.com**](https://property.vectorfeedhq.com/)

Would love feedback from fellow builders on the landing page UI, pricing tiers, or data pipeline architecture!


r/scrapingtheweb • • Apr 29 '26

I give up... I just can’t scrape Pages Jaunes with vibe coding

3 Upvotes

I’ve been trying to scrape Pages Jaunes for a small local business research project, but I’m honestly stuck.

I’m not a developer. I tried the whole “vibe coding” thing with ChatGPT and Claude, and I did get a script running at some point, but it keeps breaking or returning messy results.

What I need is pretty basic:

- business name
- phone number
- address
- website if available
- maybe email or company details if possible

At this point, I don’t really want to maintain code, proxies, retries, or random errors.

I just need a no-code tool that can handle Pages Jaunes searches and give me clean CSV results.

Has anyone used a reliable and affordable no-code Pages Jaunes scraper?


r/scrapingtheweb • • Jan 27 '26

Building a scraper that keeps hitting 403s?(We are looking for interested testers)

0 Upvotes

It feels like Cloudflare and Akamai tightened their grip significantly in the last few weeks.A lot of my usual go-to datacenter proxies are getting flagged instantly.

We've been working on a new rotation logic at Thordata using a fresh pool of residential IPs,and so far, Currently, our pool contains approximately 60 million ethical IP addresses.

it's bypassing the new challenges pretty well in our tests.

I want to see if it holds up in the wild.

If anyone here is currently struggling with a specific target site (E-commerce, Social Media, SERP, etc.) and wants to test if our IPs can get through:
I’m giving away free trial data to anyone willing to test specific use cases.

No strings attached, no CC needed. Just looking for validation on which sites we are crushing and which ones we need to optimize.

We are recruiting honest feedback providers. If you are interested, please send a short message explaining how you plan to use it and the expected traffic volume.

Our recruitment target is limited.


r/scrapingtheweb • • Aug 17 '26

Build a Semantic Job Search Engine with Bright Data, LanceDB, and Cohere

1 Upvotes

Job boards often search by exact words, so the right role can stay hidden when your phrasing doesn’t match the job posting.

Semantic search matches on meaning instead.

I built a semantic job search engine end to end, then compared the search modes to see which one actually performed better instead of assuming the most complex approach would win.

Full guide: https://brightdata.com/blog/ai/semantic-job-search-engine-with-bright-data-lancedb-and-cohere


r/scrapingtheweb • • Jan 04 '26

Scraping selenium

1 Upvotes

I want to create a script for scrap the gmail profiles and also for getting imei number info with validation of checking imei number is valid or not.

can anyone has any idea in it please share with me.i appreciate a lot.


r/scrapingtheweb • • Jun 13 '26

Python Scraping dati locali: alternative a Google Places API?

1 Upvotes

Ciao a tutti,

sto lavorando a un progetto personale e sto cercando di capire quali siano le migliori alternative per ottenere dati locali (es. "parrucchieri a Torino", "ristoranti a Milano", ecc.) in modo affidabile, legale e tecnicamente sostenibile.

Niente di aggressivo. Ricerche sporadiche per me.

Inizialmente stavo valutando l'estrazione di informazioni senza a l'uso di API Google. Poi in realtà per quello che mi serve mi bastano i crediti. Ma volevo provare prima a farlo senza queste ultime

Quello che mi interessa è ottenere dati come:

- nome attività

- categoria

- indirizzo

- telefono

- sito web

- recensioni/valutazioni (se disponibili)

- eventuali orari di apertura

Le domande che ho sono:

  1. Quali piattaforme consigliate come alternativa a Google?

  2. Esistono motori di ricerca o servizi che utilizzano dati simili a quelli di Google ma con API o accesso più adatto agli sviluppatori?

  3. DuckDuckGo, Bing, OpenStreetMap, Yelp, Foursquare, TomTom, Here o altri servizi possono essere una valida alternativa?

  4. Se doveste costruire oggi un sistema per cercare attività locali (es. "parrucchieri a Torino"), quale stack di dati usereste?

  5. Quali soluzioni offrono il miglior compromesso tra qualità dei dati, costi, stabilità e conformità alle regole d'uso?

Domanda tecnica separata: quando faccio una ricerca Google e nella console vedo risorse tipo search?q=..., è tecnicamente possibile leggerne l’HTML con uno script per uso didattico? Oppure è meglio evitarlo del tutto perché fragile/non conforme ai termini? Non sto cercando bypass, proxy o automazioni massive: voglio solo capire qual è il modo corretto di affrontare il problema.

Grazie!


r/scrapingtheweb • • Aug 28 '26

Help Any api scraping You would recommend ?

1 Upvotes

currently pulling instagram and tiktok data on a weekly schedule. the api scraping tool i'm using works most of the time but i'm getting maybe a 70-80% success rate on runs which means i'm always doing some manual cleanup before i can send anything over.

curious what others are using for these 2 social platforms ?


r/scrapingtheweb • • Jun 19 '26

ChromiumFish Update: Stealth + AI Workflow Browser for Playwright

4 Upvotes

Hey r/scrapingtheweb,

A few days ago, I posted that I built ChromiumFish, a fingerprint-hardened Chromium fork with drop-in Playwright support for Python and Node. I have now updated it to include built-in AI Agent support at the C++ level.

I know most of you are gonna smell the chromium, but rest assured, this fork tries its best to pretend like a Chrome running on a Windows machine.

The reason I built this is that when you are running your browser on a Linux machine, there are lots of telltale signs that it's Chrome running on Linux. That's what I tried to prevent on here.

Then there's the AI Agent. But how is this different from other AI agents? The main difference is that some critical AI tooling runs at the C++ level in the browser, making it much harder to detect than at the CDP/Playwright layer. You can use any OpenAI compliant provider and model that supports tool calling.

Repo: https://github.com/arman-bd/chromiumfish

It's available for both Python and Node via npm and pip. If you want, you can use the built binary with other frameworks as well, or build your own. It's completely open-source and MIT-licensed.


r/scrapingtheweb • • Dec 25 '25

Unpopular opinion: If it's on the public web, it's scrapeable. Change my mind.

57 Upvotes

I've been in the web scraping community for a while now, and I keep seeing the same debate play out: where's the actual line between ethical scraping and crossing into shady territory?

I've watched people get torn apart for admitting they scraped public data, while others openly discuss scraping massive sites with zero pushback. The rules seem... made up.

Here's the take that keeps coming up (and dividing people):
If data is on the public web (no login, no paywall, indexed by Google), it's already public. Using a script instead of manually copying it 10,000 times is just automation, not theft.

Where most people seem to draw the line:
✅ robots.txt - Some read it as gospel, others treat it like a suggestion. It's not legally binding either way.
✅ Rate limiting - Don't DOS the site, but also don't crawl at "1 page per minute" when you need scale.
❌ Login walls - Don't scrape behind auth. That's clearly unauthorized access.
❌ PII - Personal emails, phone numbers, addresses = hard no without consent.
⚠️ ToS - If you never clicked "I agree," is it actually binding? Legal experts disagree.

The questions that expose the real tension:

  1. Google scrapes the entire web and makes billions. Why is that okay but individual scrapers get vilified?

  2. If I manually copy 10,000 listings into a spreadsheet, that's fine. But automate it and suddenly I'm a criminal?

  3. Companies publish data publicly, then act shocked when people use it. Why make it public then?

Where do YOU draw the line?

  • Is robots.txt sacred or just a suggestion?

  • Is scraping "public" data theft, fair use, or something in between?

  • Does commercial use change the ethics? (Scraping for research vs selling datasets)

  • If a site's ToS says "no scraping" but you never agreed to it, does it apply?

I'm not looking for the "correct" answer—I want to know where you actually draw the line when nobody's watching. Not the LinkedIn-safe version.

Change my mind


r/scrapingtheweb • • Aug 05 '26

How to avoid false monitoring when monitoring a page?

2 Upvotes

Anyone here with prior experience detecting web page changes, for example, monitoring and detecting when a pricing page is updated?

My approach currently:

Scrape the page, strip headers/footers.
Convert the content to markdown.
Provide the md to llm to extract data.
Use backend to determine the changes and dispatch change alerts.

Does this approach sound reasonable?