I run a side project that scores and roasts GitHub profiles (Next.js on Vercel, Turso + Upstash Redis as the data layer). In early July, things started looking *too* good: traffic charts climbing, a sudden "SEO explosion" in the referrer dashboard, big spikes in function invocations.
Digging in, the real picture was this:
- **~50% of all traffic came from large-scale scrapers** on rotating datacenter proxies (Oxylabs, Datacamp, M247, and friends). These aren't dumb curl loops — they run headless browsers, **execute JavaScript**, and fire my analytics events. Vercel Analytics, Speed Insights, and GA4 were all polluted.
- A burst of requests from Alibaba Cloud IPs **exhausted my database connection pool** and took the API down with 500s.
- The scrapers started **faking `referrer=www.google.com`** — from 57/day to 15,000/day overnight — making the SEO dashboard read like a rocket launch. Actual Google clicks from residential ISPs: about **280/day**.
- My heaviest pages (crawlable leaderboards) were `force-dynamic`, so every one of ~16k daily crawler hits was a full function invocation and a cache MISS. I was paying compute for every bot.
The naive response is "block all bots." But for this product that's wrong: AI assistants and crawlers that *identify themselves* (ChatGPT, Claude, Perplexity, search engines) are a distribution channel — I *want* to be citable in AI answers, and agents are literally a customer segment for the API. The goal isn't zero bots. It's:
Stop paying (in compute, database connections, and money) for anonymous scraper traffic.
Stop letting it pollute the numbers I make decisions with.
Keep — and even monetize — the bots that identify themselves.
Everything below follows one principle:
> **Classify traffic by the cost the sender pays, not by what it claims to be.**
>
> User-Agents are free to fake. Referrers are free to fake. Executing JS is cheap. But residential IP space, verified bot signatures, and cryptographic attestation are not. Trust signals that cost something; ignore signals that don't.
## Layer 1: robots.txt — block training, keep retrieval (Day 1)
First, policy. Not all AI crawlers are equal, and lumping them together throws away real value:
-
**Allowed:**
search/answer/user-triggered crawlers — `OAI-SearchBot`, `ChatGPT-User`, `Claude-User`, `Claude-SearchBot`, `PerplexityBot`, `Applebot`. These send users to me. When someone asks an AI "is this GitHub account legit?", I want to be the citation.
-
**Blocked:**
training crawlers — `GPTBot`, `ClaudeBot`, `CCBot`, `Google-Extended`, `Applebot-Extended`, `meta-externalagent`, `Amazonbot`, `Bytespider`. They consume the corpus and send nothing back.
- Plus `Content-Signal: ai-train=no` for crawlers that honor it.
Is robots.txt enforceable? No. But well-funded crawlers
*do*
respect it, and it establishes a clean policy baseline:
*retrieval yes, training no*
. The ones that ignore robots.txt entirely get handled by the layers below.
## Layer 2: API hardening — the parts that fell over under load (Day 2)
The Alibaba Cloud burst exposed three real bugs in my main scan endpoint. All three are generic enough that you probably have at least one of them:
**1. Rate-limit
*before*
the cache lookup, not after.**
My per-IP rate limiter ran after the cache check, on the theory that cached responses are cheap. But "cheap" wasn't free — cached hits still wrote analytics rows to the database. Bots hammering the same popular usernames bypassed the limiter entirely while holding database connections. The limiter now runs first, and even cached responses carry standard `RateLimit-*` headers.
**2. Put a Redis `SET NX` shield in front of hot database writes.**
Recording "who looked up which account" used a database transaction as its dedup gate. Under a burst, thousands of identical (username, IP) pairs each opened a connection just to discover they were duplicates. Now a Redis `SET NX` with a 24h TTL sits in front: repeats cost one Redis round-trip instead of a database connection. The database constraint stays as the source of truth; Redis is just the cheap bouncer at the door.
**3. Validate body types like an adversary wrote them.**
`{"username": 12345}` — a number, not a string — hit `.trim()` and crashed with an uncaught TypeError → 500. Bots fuzz your API whether you invited them or not. Unknown input shapes now return a clean 400.
## Layer 3: ISR — make the hits you can't block cost nothing (Day 3)
My leaderboard/facet pages were the single heaviest crawler target:
**~16k function invocations per day, every one a cache MISS**
, because the route was `force-dynamic`.
The fix was converting them to ISR (`revalidate: 600`) — but two details were non-obvious and worth sharing:
- The pages had a `?u=<username>` query param that pinned a user's card to the top. Query-string reads make a route dynamic, so I moved that logic
**client-side**
(a small component under `Suspense` reading `useSearchParams`). The canonical URLs crawlers hit are now fully static; the personalized pin renders in the browser. Same UX, zero server cost per crawl.
-
**Upstash's Redis client defaults to `cache: no-store` on its fetch calls**
, and in Next 15 an uncached fetch inside an ISR page silently flips the whole route from static back to dynamic at runtime (it 500s under `next start`). Passing `cache: "default"` to the client fixes it — Upstash REST calls are POSTs which Next never data-caches anyway, so there's no staleness risk. This one took a while to find; if you use Upstash inside ISR pages, check it.
Result: crawler hits on those boards now cost roughly nothing — they hit the static cache instead of invoking a function that queried Redis 50 times.
And since agents keep crawling these boards anyway, the pages now end with a pitch:
*"vetting accounts in bulk? There's an API — here are the docs."*
If a bot is going to read my HTML 16,000 times a day, it might as well read the sales copy.
## Layer 4: the invisible gate — humans and honest bots pass, impersonators get a 403 with directions (Day 3)
This is the interesting one. These scrapers execute JS, so classic client-side detection was already beaten, and I refused to put a CAPTCHA in front of the core product experience.
The expensive endpoint (LLM-backed, costs me actual API credits per call) now gates its free path with
**Vercel BotID**
— an invisible, no-interaction check that classifies clients at the edge. The decision table:
| Client | Result |
|---|---|
| Human in a real browser | ✅ pass, no friction |
| API caller with a Bearer key | ✅ pass (they're paying) |
| Caller with their own LLM key (BYO) | ✅ pass (costs me nothing) |
| Verified bots — googlebot, ChatGPT-User, Claude, etc. | ✅ pass |
| Unverified headless browser impersonating a human | ❌ 403 |
And the 403 body isn't a wall — it's a
**signpost**
: "this endpoint requires verification; documented API and MCP endpoints are here." If someone built an agent that's scraping me anonymously, the failure message itself tells them the legitimate (and rate-limited, and monetizable) way in. Some fraction of hostile traffic converts to API users.
The auth logic got extracted into a shared module, so the scan endpoint (which uses a Turnstile flow for browsers) and the roast endpoint enforce identical machine-auth rules.
**Cleaning the analytics, same day:**
Vercel Analytics and Speed Insights get a `beforeSend` that drops events when `navigator.webdriver` is set; GA4 config is skipped under the same condition. Is `navigator.webdriver` spoofable? Absolutely. But these scrapers apparently don't bother, and my dashboards went from "fiction" to "usable."
## Layer 5: trust no referrer — verify traffic sources against ASN (Day 4)
Once the bot check went up, the fake-Google-referrer trick became the remaining data problem: 15k/day of `referrer=www.google.com` from datacenter IP ranges would convince anyone their SEO had exploded.
Since the dashboards can't be trusted, I wrote a small ops script over Vercel's observability metrics API that splits traffic into three layers:
-
**Scraper layer**
— anonymous requests from a curated blocklist of proxy-provider ASNs (Oxylabs, Datacamp, M247, …). Background noise; ignore in all decisions.
-
**Agent layer**
— self-declared crawlers/AI assistants by bot signature. Good traffic; this is the monetization funnel.
-
**Human layer**
— what remains, anchored by
*action*
endpoints (scan/roast/search API calls) rather than pageviews, because actions are much harder to fake at scale than pageviews.
The script also cross-checks every google-referred request against the proxy ASN list: `proxy ASN + google referrer = fake`. Ironically,
**the spoofing backfired**
— faking Google referrers from datacenter IPs is a self-signed confession, and the fake traffic exposed 8 new proxy providers I hadn't blocklisted yet.
One nuance for anyone building an ASN blocklist:
**VPN egress ranges have real users behind them.**
I deliberately excluded the VPN/proxy providers my actual users route through (in my case, providers popular with Chinese users behind the GFW). Blanket-blocking "datacenter ASN = bot" would have cut off a chunk of my real audience. Curate the list; don't import someone else's.
