r/webscraping • u/Lafftar • Oct 07 '25
A 20,000 req/s Python setup for large-scale scraping (full code & notes on bypassing blocks).
Hey everyone, I've been working on a setup to tackle two of the biggest problems in large-scale scraping: speed and getting blocked. I wanted to share a proof-of-concept that can hit ~20,000 requests/sec, which is fast enough to scrape millions of pages a day.
After a lot of tuning, I managed to get a stable ~20,000 requests/second from a single client machine.
Here's 10 million requests submitted at once:

The code itself is based on asyncio and a library called rnet A key reason I used the rnet library is that its underlying Rust core has a robust TLS configuration, which is much better at bypassing WAFs like Cloudflare than standard Python libraries. This lets me get the developer-friendly syntax of Python with the raw speed of Rust for the actual networking.
The most interesting part wasn't the code, but the OS tuning. The default kernel settings on Linux are nowhere near ready for this kind of load. The application would fail instantly without these changes.
Here are the most critical settings I had to change on both the client and server:
Increased Max File Descriptors: Every socket is a file. The default limit of 1024 is the first thing you'll hit.ulimit -n 65536
Expanded Ephemeral Port Range: The client needs a large pool of ports to make outgoing connections from.net.ipv4.ip_local_port_range = 1024 65535
Increased Connection Backlog: The server needs a bigger queue to hold incoming connections before they are accepted. The default is tiny.net.core.somaxconn = 65535
Enabled TIME_WAIT Reuse: This is huge. It allows the kernel to quickly reuse sockets that are in a TIME_WAIT state, which is essential when you're opening/closing thousands of connections per second.net.ipv4.tcp_tw_reuse = 1
I've open-sourced the entire test setup, including the client code, a simple server, and the full tuning scripts for both machines. You can find it all here if you want to replicate it or just look at the code:
GitHub Repo: https://github.com/lafftar/requestSpeedTest
Blog Post (I go in a little more detail): https://tjaycodes.com/pushing-python-to-20000-requests-second/
On an 8-core machine, this setup hit ~15k req/s, and it scaled to ~20k req/s on a 32-core machine. Interestingly, the CPU was never fully maxed out, so the bottleneck likely lies somewhere else in the stack.
I'll be hanging out in the comments to answer any questions. Let me know what you think!
11
u/innovasior Oct 07 '25
Crazy. Thats a denial of service attack 😅😂
9
u/Lafftar Oct 07 '25
That's why I hit my own server 😂😂😂
Don't wanna give anyone any wrong ideas haha.
2
7
u/Repeat_Status Oct 07 '25
Ok, but as far as I understand this doesn't include DNS resolving times, and no real life amount of data transferred? You'd need 1GBit connection to download 1k page/sec with average size 100kB, so 20k/s requests would need 20GBit link... For me it looks more like ping or DDOS test than real life scraping usability test...
3
u/Lafftar Oct 07 '25
Just returning a few bytes from the server, but yeah that's something I hadn't considered! Ofc the speed of the servers would be an issue lol. I'm not sure we'll hit that bottle neck for a couple 100k r/s tho.
Also it includes the entire time the request/response takes, everything's included.
3
3
2
2
2
u/Busy_Sugar5183 Oct 07 '25
how does it deal with reCAPTCHA? Does rnet bypasses it or are you using CAPTCHA solving services
2
2
u/Crafty_Disk_7026 Oct 11 '25
Dude just switch to rust or go and you will easily 10x that. Learn the right tool for the job the Python GIL is not your friend here
1
1
u/Boring-Ad-5924 Oct 12 '25
Created a script to check a list of over 300k websites for something using Deno.js few years back was able to do about 80-100k req/sec depending on how good my proxies were. Only used residential ones
0
Oct 07 '25
[deleted]
7
u/Lafftar Oct 07 '25
Yes, the event loop for python (even uvloop) is a great addition to the library but is still lacking when compared to golang or even TS. I actually don't understand why, I need to do more research there.
And no worries man you're good! You may have actually given me the topic to my next video haha. (A look at how 2 static langs vs 2 interpreted langs do concurrency.)
There's a lot to learn and a lot to try, and I'm excited to expand my knowledge 😁
5
u/saintpetejackboy Oct 08 '25
Go is an amazing language , and there are not going to be a lot of setups that do this particular task "faster" than properly configured Go. Arguably, some "faster" languages like Rust, could easily be implemented in such a way as to end up slower.
If you do videos on stuff like C or Fortran or ASM or Rust, you will get a torrent of people saying "well, if you would have (arcane esoteric methodology) it would have been 604% faster" - in other words, most of us would never be able to pull off a fully "proper" implementation. Just because a language can be faster than another doesn't mean we are skilled enough with it to juice that level of performance out of it.
Go is kind of the best of all worlds in that aspect - you get low cost abstractions and high concurrency without having to break your brain open. The trade-off is that there is still GC going on and Go isn't even really probably in the Top 5 "fastest" languages.
Then you also have stuff like NIM, Zig, D and trusty Java, .NET/C# style JITted languages where, arguably, an expert can make those languages just as performant or more performant than Go. Or close enough to not really be significant.
The other poster above knows what they are talking about - the bottlenecks you end up hitting with scraping are going to be based a lot on DNS/network and other stuff you can't easily control.
A very simple and "dumb" script might also be super fast and able to scrape tons of content, but not actually parse it or handle situations where it is trying to scrape a downed server or from a downed connection internally. Just one example, but there are layers of complexity that you might need later that will also drag down the overall speed because, in the real world, servers go down. They become unresponsive. They ban your IP. So now you're juggling IP with each connection and of course the amount of records you can scrape every second takes a hit.
You'll likely start to bump up against other barriers: while we focus heavily on CPU and RAM, a common bottleneck in all software ends up being the database. Are you checking for duplicates? Well, that is easy with 500k records, but becomes more of a burden at 20m+ records - at the point that the entirety of the database can no longer fit inside memory, you will start taking a beating on queries.
In some areas / regions, you may even hit bandwidth or other caps that just were not considered early on.
Why I am saying this is because, you can hop around languages all day seeing which one is "fastest" by certain measurements, but the real world is a dangerous jungle full of unexpected consequences.
Second, as I mentioned earlier up, almost all code could be refactored and optimized. No matter how stellar your implementation is in each language, somebody is bound to come along and say "why did you do this foolish thing! You messed up (x) and (y) or (z)".
Don't worry about either of those things. Those people got to that spot today by doing what you are doing now, yesterday.
People might sound snarky when they say your implementation isn't the fastest, but they are often speaking from their own experience and mistakes.
1
Oct 08 '25
[removed] — view removed comment
1
u/Lafftar Oct 08 '25
That's just 200 requests a sec.
Don't know what kind of work you'll do after you get the response but 1 VPS should be enough.
The main thing I'm thinking of is proxy cost, that could run you run you ~$15,000/day in resi cost, unless you manage to get ISPs/Datacenters at bulk for a good price, you could be shafted there.
1
u/polygraph-net Oct 08 '25
I also have to send 17 million requests per day to a datadome protected website.
This is fascinating. Can you share why the need for so many requests?
1
Oct 08 '25
[removed] — view removed comment
1
u/polygraph-net Oct 08 '25
Is it a DDoS sort of thing or they have a genuine business need for 17 million requests?
13
u/Gojo_dev Oct 07 '25
20k req/sec that’s wild, man. I’ve used asyncio and thread pools but those numbers are crazy. How many ports did you use? And if we need to deal with bot protections or multi‑step scraping, does that rate still hold? What was your benchmark setup? I can check the repo, but I’d love to hear your personal details.