After countless hours spent automating tasks only to get blocked by Cloudflare, rage-quitting over reCAPTCHA v3 (why is there no button to click?), and nearly throwing my laptop out the window, I built PyDoll.

GitHub: https://github.com/thalissonvs/pydoll/

It’s not magic, but it solves what matters:

• Native bypass for reCAPTCHA v3 & Cloudflare Turnstile (just click in the checkbox).

• 100% async – because nobody has time to wait for requests.

• Currently running in a critical project at work (translation: if it breaks, I get fired).

FAQ (For the Skeptical):

• “Is this illegal?” → No, but I’m not your lawyer.

• “Does it actually work?” → It’s been in production for 3 months, and I’m still employed.

• “Why open-source?” → Because I suffered through building it, so you don’t have to (or you can help make it better).

For those struggling with hCAPTCHA, native support is coming soon – drop a star ⭐ to support the cause

Just uncovered something that hit far closer to home than expected, even as an experienced scraper. I’d appreciate any insight from others in the scraping community.

I’ve been in large-scale data automation for years. Most of my projects involve tens of millions of data points. I rely heavily on proxy infrastructure and routinely use thousands of IPs per project, primarily residential.

Last week, in what initially seemed unrelated, I needed to install some niche video plugins on my 11-year-old son’s Windows 11 laptop. Normally, I’d use something like MPC-HC with LAV Filters, but he wanted something quick and easy to install. Since I’ve used K-Lite Codec Pack off and on since the late 1990s without issue, I sent him the download link from their official site.

A few days later, while monitoring network traffic for a separate home project, I noticed his laptop was actively pushing outbound traffic on ports 4444 and 4650. Closer inspection showed nearly 25GB of data transferred in just a couple of days. There was no UI, no tray icon, and nothing suspicious in Task Manager. Antivirus came up clean.

I eventually traced the activity to an executable associated with a company called Infatica. But it didn’t stop there. After discovering the proxyware on my son’s laptop, I checked another relative’s computer who I had previously recommended K-Lite to and found it had been silently bundled with a different proxyware client, this time from a company named Digital Pulse. Digital Pulse has been definitively linked to massive botnets (one article estimated more than 400,000 infected devices at the time). These compromised systems are apparently a major source used to build out their residential proxy pools.

After looking into Infatica further, I was somewhat surprised to find that the company has flown mostly under the radar. They operate a polished website and market themselves as just another legitimate proxy provider, promoting “ethical practices” and claiming access to “millions of real IPs.” But if this were truly the case, I doubt their client would be pushing 25GB of outbound traffic with no disclosure, no UI, and no user awareness. My suspicion is that, like Digital Pulse, silent installs are a core part of how they build out the residential proxy pool they advertise.

As a scraper, I’ve occasionally questioned how proxy providers can offer such large-scale, reliable coverage so cheaply while still claiming to be ethically sourced. Rightly or wrongly (yes, I know, wrongly), I used to dismiss those concerns by telling myself I only use “reputable” providers. Having my own kid’s laptop and our home IP silently turned into someone else’s proxy node was a quick cure for that cognitive dissonance.

I’ve always assumed the shady side of proxy sourcing happened mostly at the wholesale level, with sketchy aggregators reselling to front-end services that appeared more legitimate. But in this case, companies like Digital Pulse and Infatica appear to directly distribute and operate their own proxy clients under their own brand. And in my case, the bandwidth usage was anything but subtle.

Are companies like these outliers or is this becoming standard practice now (or has it been for a while)? Is there really any way to ensure that using unsuspecting 11-year-old kids' laptops is the exception rather than the norm?

Thanks to everyone for any insight or perspectives!

EDIT: Following up on a comment below in case it helps someone else... the main file involved was Infatica-Service-App.exe located in C:\Program Files (x86)\Infatica P2B. I removed it using Revo Uninstaller, which handled most of the cleanup, but there were still a few leftover registry keys and temp files/directories that needed to be removed manually.

Hello,

I am learning web scrapping and tried beautifulsoup and selenium to scrape. With bot detection and resources, I realized they aren't the most efficient ones and I can try using API calls instead to get the data. I, however, noticed that big companies like Amazon hide their API calls unlike small companies where I can see the JSON file from the request.

I have looked at a few post, and some mentioned about encryption. How does it work? Is there any way to get around this? If so, how do I do that? I would appreciate if you could also point me out to any articles to improve my understanding on this matter.

Thank you.

UPDATE: I built a tool to make this super easy: https://eddie.surf - no more scraping/AI headaches!

1. Don't try putting scraping tools in Lambda. Just admit defeat!

2. Selenium is cool and talked about a lot, but Playwright/Puppeteer/hrequests are new and better.

3. Don't feel like you have to go with Python. The Node.JS scraping community is huge! And more modern advice than Selenium.

4. AI will likely teach you old tricks because it's trained on a lot of old data. Use Medium/google search with timeframe < 1 year.

5. Scraping is about new tricks, as Cloudflare, etc block a lot of scraping tactics.

6. Playwright is super cool! A lot of MS coders brought on from Puppeteer, from what I heard. The stealth plugin doesn't work, however (most stealth plugins don't, in fact!)

7. Find out YOUR browser headers

8. Don't worry about fancy proxies, etc if you're scraping lots of sites at scale. Worry if you're scraping lots of data from one site, or regular data scraping from one site.

9. If you're going to use proxies, use residential ones! (Update: people have suggested using mobile proxies. I would suggest using data center, then residential, then mobile as a waterfall-like fallback to keep costs down.)

10. Find out what your browser headers are (user agent, etc) and mimic the same settings in Playwright!

11. Use checker tools like "Am I Headless" to find out some detection.

12. Don't try putting things in Lambda! If you like happiness and a work/life balance.

13. Don't learn scraping avoidance techniques from scraping sites. Learn from the sites that teach detecting these!

14. Put a random delay between requests, 800ms-2s. If the scraping errors, back off a little more and retry a few more seconds later.

15. Browser pools are great! A small EC2 instance will happily run about 5 at a time.

Hey everyone,

I published my 3rd pypi lib and it's open source. It's called stealthkit - requests on steroids. Good for those who want to send http requests to websites that might not allow it through programming - like amazon, yahoo finance, stock exchanges, etc.

What My Project Does

• User-Agent Rotation: Automatically rotates user agents from Chrome, Edge, and Safari across different OS platforms (Windows, MacOS, Linux).

• Random Referer Selection: Simulates real browsing behavior by sending requests with randomized referers from search engines.

• Cookie Handling: Fetches and stores cookies from specified URLs to maintain session persistence.

• Proxy Support: Allows requests to be routed through a provided proxy.

• Retry Logic: Retries failed requests up to three times before giving up.

• RESTful Requests: Supports GET, POST, PUT, and DELETE methods with automatic proxy integration.

Why did I create it?

In 2020, I created a yahoo finance lib and it required me to tweak python's requests module heavily - like session, cookies, headers, etc.

In 2022, I worked on my django project which required it to fetch amazon product data; again I needed requests workaround.

This year, I created second pypi - amzpy. And I soon understood that all of my projects evolve around web scraping and data processing. So I created a separate lib which can be used in multiple projects. And I am working on another stock exchange python api wrapper which uses this module at its core.

It's open source, and anyone can fork and add features and use the code as s/he likes.

If you're into it, please let me know if you liked it.

Pypi: https://pypi.org/project/stealthkit/

Github: https://github.com/theonlyanil/stealthkit

Target Audience

Developers who scrape websites blocked by anti-bot mechanisms.

Comparison

So far I don't know of any pypi packages that does it better and with such simplicity.

🚀 Excited to announce Scrapling v0.3 - The most significant update yet!

After months of development, we've completely rebuilt Scrapling from the ground up with revolutionary features that change how we approach web scraping:

🤖 AI-Powered Web Scraping: Built-in MCP Server integrates directly with Claude, ChatGPT, and other AI chatbots. Now you can scrape websites conversationally with smart CSS selector targeting and automatic content extraction.

🛡️ Advanced Anti-Bot Capabilities:

• Automatic Cloudflare Turnstile solver

• Real browser fingerprint impersonation with TLS matching

• Enhanced stealth mode for protected sites

🏗️ Session-Based Architecture: Persistent browser sessions, concurrent tab management, and async browser automation that keep contexts alive across requests.

⚡ Massive Performance Gains:

• 60% faster dynamic content scraping

• 50% speed boost in core selection methods

• and more...

📱 Terminal commands for scraping without programming

🐚 Interactive Web Scraping shell:

• Interactive IPython shell with smart shortcuts

• Direct curl-to-request conversion from DevTools

And this is just the tip of the iceberg; there are many changes in this release

This update represents 4 months of intensive development and community feedback. We've maintained backward compatibility while delivering these game-changing improvements.

Ideal for data engineers, researchers, automation specialists, and anyone working with large-scale web data.

📖 Full release notes: https://github.com/D4Vinci/Scrapling/releases/tag/v0.3

🔧 Get started: https://scrapling.readthedocs.io/en/latest/

Scrapling v0.4 is here — the biggest update yet 🕷️

Scrapling is an adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl, and it's free!

Its parser learns from website changes and automatically relocates your elements when pages update. Its fetchers bypass anti-bot systems like Cloudflare Turnstile out of the box. And its spider framework lets you scale up to concurrent, multi-session crawls with pause/resume and automatic proxy rotation — all in a few lines of Python. One library, zero compromises.

Blazing fast crawls with real-time stats and streaming. Built by Web Scrapers for Web Scrapers and regular users, there's something for everyone.

Below, we talk about some of the new stuff:

New: Async Spider Framework A full crawling framework with a Scrapy-like API — define a Spider, set your URLs, and go.

from scrapling.spiders import Spider

class MySpider(Spider):
    name = "demo"
    start_urls = ["https://example.com/"]

    async def parse(self, response):
        for item in response.css('.product'):
            yield {"title": item.css('h2::text').get()}

MySpider().start()

• Concurrent crawling with per-domain throttling

• Mix HTTP, headless, and stealth browser sessions in one spider

• Pause with Ctrl+C, resume later from checkpoint

• Stream items in real-time with async for.

• Blocked request detection and automatic retries

• Built-in JSON/JSONL export

• Detailed crawl stats and lifecycle hooks

• uvloop support for faster execution

New: Proxy Rotation: Thread-safe ProxyRotator with custom rotation strategies. Works with all fetchers and spider sessions. Override per-request anytime.

Browser Fetcher Improvements:

• Block requests to specific domains with blocked_domains

• Automatic retries with proxy-aware error detection

• Response metadata tracking across requests

• Response.follow() for easy link-following

Bug Fixes:

• Parser optimized for repeated operations

• Fixed browser not closing on error pages

• Fixed Playwright loop leak on CDP connection failure

• Full mypy/pyright compliance

Upgrade: pip install scrapling --upgrade. Full release notes: github.com/D4Vinci/Scrapling/releases/tag/v0.4 There is a brand new website design too, with improved docs: https://scrapling.readthedocs.io/

This update took a lot of time and effort. Please try it out and let me know what you think!

Hey everyone, just a web scraping enthusiast here. I see a lot of people struggling with slow headless browsers or getting blocked by anti-bots.

Before writing a heavy script, take 1 minute to do this:

1. Hit F12 and go to the Network tab.

2. Filter by Fetch/XHR.

3. Refresh the page or click a few buttons.

Most modern sites fetch their data from a clean JSON API in the background. Hitting that endpoint directly using requests is 100x faster, bypasses basic UI bot-protection, and often gives you more data than what's on the screen.

Wish you all the best! ✌️

As the title says, I've spent the past few days creating a free proxy pricing comparison tool. You all know how hard it can be to compare prices from different providers, so I tried my best and this is the result: https://proxyprice.thewebscraping.club/

I hope you don't flag it as spam or self-promotion, I just wanted to share something useful.

EDIT: it's still an alpha version, so any feedback is welcome. I'm filling it with more companies in these days.

Hey everyone, I've been working on a setup to tackle two of the biggest problems in large-scale scraping: speed and getting blocked. I wanted to share a proof-of-concept that can hit ~20,000 requests/sec, which is fast enough to scrape millions of pages a day.

After a lot of tuning, I managed to get a stable ~20,000 requests/second from a single client machine.

Here's 10 million requests submitted at once:

The code itself is based on asyncio and a library called rnet A key reason I used the rnet library is that its underlying Rust core has a robust TLS configuration, which is much better at bypassing WAFs like Cloudflare than standard Python libraries. This lets me get the developer-friendly syntax of Python with the raw speed of Rust for the actual networking.

The most interesting part wasn't the code, but the OS tuning. The default kernel settings on Linux are nowhere near ready for this kind of load. The application would fail instantly without these changes.

Here are the most critical settings I had to change on both the client and server:

• Increased Max File Descriptors: Every socket is a file. The default limit of 1024 is the first thing you'll hit.ulimit -n 65536

• Expanded Ephemeral Port Range: The client needs a large pool of ports to make outgoing connections from.net.ipv4.ip_local_port_range = 1024 65535

• Increased Connection Backlog: The server needs a bigger queue to hold incoming connections before they are accepted. The default is tiny.net.core.somaxconn = 65535

• Enabled TIME_WAIT Reuse: This is huge. It allows the kernel to quickly reuse sockets that are in a TIME_WAIT state, which is essential when you're opening/closing thousands of connections per second.net.ipv4.tcp_tw_reuse = 1

I've open-sourced the entire test setup, including the client code, a simple server, and the full tuning scripts for both machines. You can find it all here if you want to replicate it or just look at the code:

GitHub Repo: https://github.com/lafftar/requestSpeedTest

Blog Post (I go in a little more detail): https://tjaycodes.com/pushing-python-to-20000-requests-second/

On an 8-core machine, this setup hit ~15k req/s, and it scaled to ~20k req/s on a 32-core machine. Interestingly, the CPU was never fully maxed out, so the bottleneck likely lies somewhere else in the stack.

I'll be hanging out in the comments to answer any questions. Let me know what you think!

Hello everyone,

It has been some time since I wrote here :)

A lot has happened since my last post, and we are now almost at 1.5M downloads.

I wanted to share this update with you here, as many people have asked for these features since the first release.

Introducing the new spiders templates feature to make generic spiders easier. Now you have a CrawlSpider and a SitemapSpider. Both offer many options and controls. We also added tools to make it easier, like LinkExtractor and CrawlRule, which will help you build your generic spider. Stop rewriting "follow links matching X" boilerplate in every parse() :D

Check the new page for them on the website here: https://scrapling.readthedocs.io/en/latest/spiders/generic-templates.html

Also, we have improved the adaptive feature so it's now much better, fixed some bugs, improved the docs, and updated browsers/fingerprints

The full release notes are here: https://github.com/D4Vinci/Scrapling/releases/tag/v0.4.8

Let me know your thoughts and what I should add next :)

Out of curiosity, how did OpenAI *scrape the entire Internet for training ChatGPT?

Calling anybody with a large and complex scraping setup…

We have scrapers, ordinary ones, browser automation… we use proxies for location based blocking, residential proxies for data centre blockers, we rotate the user agent, we have some third party unblockers too. But often, we still get captchas, and CloudFlare can get in the way too.

I heard about browser fingerprinting - a system where machine learning can identify your browsing behaviour and profile as robotic, and then block your IP.

Has anybody got any advice about what else we can do to avoid being ‘identified’ while scraping?

Also, I heard about something called phone farms (see image), as a means of scraping… anybody using that?

https://github.com/Redrrx/browser-js-dumper

After reading this sub for a while, it looks like there's plenty of people who are scraping millions of pages every month with minimal costs - meaning dozens of $ per month (excluding servers, database, etc).

I am still new to this, but I get confused by that figure. If I want to reliably (meaning with relatively high success rate) scrape websites, I probably should residential proxies. These are not cheap - the prices are going from roughly $0.50/1GB of bandwidth to almost $10 in some cases.

There are web scraping API services on the web that handle headless browsers, proxies, CAPTCHAs etc, which costs starts from around ~$150/month for 1M requests (no bandwidth limits). At glance, it looks like the residential proxies are way cheaper than the API solutions, but because of bandwidth, the price starts to quickly add up and it can actually get more expensive than the API solutions.

Back to my first paragraph, to the people who scrape data very cheaply - how do they do it? Are they scraping without proxies (but that would likely mean they would get banned soon)? Or am I missing anything obvious here?

Hey r/webscraping!

If you’re tired of getting IP-banned or waiting ages for proxy validation, I’ve got news for you: I just released v2.0.0 of my Python library, swiftshadow, and it’s now 15x faster thanks to async magic! 🚀

What’s New?

⚡ 15x Speed Boost: Rewrote proxy validation with aiohttp – dropped from ~160s to ~10s for 100 proxies.
🌐 8 New Providers: Added sources like KangProxy, GoodProxy, and Anonym0usWork1221 for more reliable IPs.
📦 Proxy Class: Use Proxy.as_requests_dict() to plug directly into requests or httpx.
🗄️ Faster Caching: Switched to pickle – no more JSON slowdowns.

Why It Matters for Scraping

• Avoid Bans: Rotate proxies seamlessly during large-scale scraping.

• Speed: Validate hundreds of proxies in seconds, not minutes.

• Flexibility: Filter by country/protocol (HTTP/HTTPS) to match your target site.

Get Started

pip install swiftshadow  

Basic usage:

from swiftshadow import ProxyInterface  

# Fetch and auto-rotate proxies  
proxy_manager = ProxyInterface(autoRotate=True)  
proxy = proxy_manager.get()  

# Use with requests  
import requests  
response = requests.get("https://example.com", proxies=proxy.as_requests_dict())  

Benchmark Comparison

Why Use This Over Alternatives?

Most free proxy tools are slow, unreliable, or lack async support. swiftshadow focuses on:

• Speed: Async-first design for large-scale scraping.

• Simplicity: No complex setup – just import and go.

• Transparency: Open-source with type hints for easy debugging.

Try It & Feedback Welcome!

GitHub: github.com/sachin-sankar/swiftshadow

Let me know how it works for your projects! If you hit issues or have ideas, open a GitHub ticket. Stars ⭐ are appreciated too!

TL;DR: Async proxy validation = 15x faster scraping. Avoid bans, save time, and scrape smarter. 🕷️💻

Scrapling is an Undetectable, high-performance, intelligent Web scraping library for Python 3 to make Web Scraping easy!

Scrapling isn't only about making undetectable requests or fetching pages under the radar!

It has its own parser that adapts to website changes and provides many element selection/querying options other than traditional selectors, powerful DOM traversal API, and many other features while significantly outperforming popular parsing alternatives.

Scrapling is built from the ground up by Web scraping experts for beginners and experts. The goal is to provide powerful features while maintaining simplicity and minimal boilerplate code.

After a long wait (and a battle with perfectionism), I’m excited to finally launch the official documentation website for Scrapling 🚀

Why this matters:

• Scrapling has grown greatly, and the old README wasn’t enough.

• The new site includes detailed documentation with rich examples — especially for Fetchers — to help both beginners and advanced users.

• It also features helpful articles like how to migrate from BeautifulSoup to Scrapling.

• Plus, an auto-generated reference section from the library’s source code makes exploring internal functions much easier.

This has been long overdue, but I wanted it to reflect the level of quality I’m proud of. Now that it’s live, I can fully focus on building v3, which will be a game-changer 👀

Link: https://scrapling.readthedocs.io/en/latest/

Thanks for the support! ❤️

Speaking in the way our beloved LLMs like to talk:

Let data, not opinions, do the talking.

Good Afternoon Everyone,

This is not promotion, this is a genuine post, and feel free to ignore if this does not concern your web scraping activities.

Some of you might remember adultdatalink from the pornstar stock market index post I made here a while back. It had scrapers that turned 20+ sites into clean JSON, performer and keyword analytics, scraping projects, and some video tools.

After about two years I'm shutting the hosted service down.

Instead of letting the code rot on a hard drive, I'm making it open source under MIT: https://github.com/adultdatalink/adultdatalink-public

First, I owe some people an apology. The API went down at the end of April and I never said anything.

Some of you noticed, and there's even a thread on webdev asking whether it was dead.

I stepped away from it without a proper shutdown notice, and anyone who had built something on top of it deserved better. This post is the notice I should have written back then. Since the code is now public, anyone who depended on it can run it themselves.

That thread also said the user numbers I posted before were probably fake, and that most people just signed up and connected without really using it. That's not correct. 1,635 people signed up, and 1,008 of them generated API keys. Signups started slow, with 17 in March 2025, peaked at 193 in December 2025, and stayed somewhere between 100 and 190 a month after that.

Our own request logs captured about 2.45 million API calls. Almost 2 million of those came between December and April alone, from 303 different API keys, so roughly 430K requests a month. The busiest single day was 25,935 requests. 97.8% of requests came back successful, with a median response time of about a second, which I'm pretty happy with considering every call was scraping a live third party site.

On the Cloudflare side, the peak was 3.9 million requests and 36 GB in one 30 day window back in March 2025, from almost 100 countries. A month and a half before that, the same window showed 195K requests. When I added the video tools, bandwidth went up to 407 GB in a single month, and Brazil, Germany, Canada and Spain suddenly became some of the biggest users. Server errors stayed around 0.05% the whole time. Pornhub was by far the most requested source, followed by Eporner, XVideos, xHamster and XNXX.

So why shut it down?

It just didn't make much money & sunk cost fallacy.

I spent so much time building this that I didn't want to just walk away from it fully, but there comes a where you cut losses on bad investment.

The usage was real but very concentrated. In the latest logs, one integration made about 80% of all requests, and the top 10 keys made 94%. A lot of people tried it, and a handful actually built on it. In this niche, payment processors, ad networks and app stores all make things harder than they need to be, and the scrapers need constant fixing every time a site changes something. Between hosting and my time, there wasn't much left over. I could keep it going as a small side thing, but I'd rather put the code out there and let people build stuff with it than keep it to myself.

The other thing I want to talk about is how much AI has changed this.

I built and ran the whole thing basically by myself. That was twelve FastAPI services covering the scraping API, an analytics engine, accounts, payments, logging, a CDN, storage and an ops dashboard, plus the scrapers, a YOLO annotation pipeline, and the stock style indexes.

A few years ago that would have taken a small team. AI coding tools have made the barrier to entry for SWE alot lower. Building it is not really the hard part anymore, finding people who will pay for it, getting it in front of them, and keeping it maintained is the hard part, and that's where this one came up short.

And also its really not hard for someone to just copy your shit, so software moats are very difficult to maintain IMO nowadays unless paired with hardware.

THE REPO

The repo has 2 pieces. The first is the adult content API & analytics engine.

Everything is configured with environment variable and it runs with pip and uvicorn.

It's archived and I won't be maintaining it, so fork it and make it your own. Some scrapers will break as the sites change, that's just how it goes.

Please read the README part about terms of service and not exposing the proxy routes to the public internet. And obviously it's 18+ only.

Huge thank you to everyone who signed up, built something on the API, sent feedback, reported bugs. I learned a ton doing this and had a lot of fun with it.