hey yall, I'm trying to build a scrapper for data gathering, but my agents is driving me nuts, I use to be able to scrape anything, now too many guardrails. http://www.barreau.qc.ca/fr/trouver-un-avocat/resultats/?p=1. (there is like 30 profiles per pages) approx 1200-1300 pages.

I need the data for each profile, I can use headless browser, I have plenty of proxies, but right now the only thing I manage to do is get 73 profiles a day... which would take me years. Any guidance?

Hello, I'm new here.

Recently some of the forums I visit just disappared (closing, nobody to pay hosting, etc.) Lots of knowledge has been lost.

How could I scrape forums to get their offline versions before they close down?

Take odds of India 1.14 and Australia 5.60. Convert each to implied probability (1/odds) and you get 87.72% and 17.86%, which adds up to 105.58%. That extra 5.58% is the bookmaker's margin. I wanted to watch that number move live, so I wrote a small scraper for Parimatch.

How it fits together:

• A Flask app runs a background Selenium thread per event. It pulls the odds and keeps the latest values in memory.

• A GET /odds?event=... endpoint returns those odds plus implied probabilities. If nobody is watching that event yet, it starts a new thread and tells you to retry in a few seconds.

• A Streamlit page polls the API and plots odds, probabilities and margin over time.

• A separate script checks whether 1/a + 1/b drops below 1. That condition is what defines an arbitrage window.

The weak spots are real. The scraper just grabs the first market on the page, because I haven't added market selection. Error handling for site changes is thin. It only covers one bookmaker. The arbitrage check is also simplified. It compares a fixed initial bet against the current opposing odds, which is not proper simultaneous betting across bookmakers.

Mostly the project taught me how odds encode the house edge. It's educational only, since betting carries real financial risk.

Small project, but the scraping side was the interesting bit.

I wanted to follow every NFT link in the 250 biggest collections on Stacks, from the contract to the metadata to the image, and see which still load. The marketplace API and most IPFS gateways wouldn't answer a normal script, so the checker runs as code in a browser tab. Browsers throttle tabs you aren't looking at, so a background worker keeps it ticking. Each file gets tried on several gateways before it's marked dead.

Result: 34 of 241 art collections can't reach their files. Own servers 9 of 16 broken, IPFS 4 of 203.

How would you have handled the gateway blocking? Write-up: https://xtrata.xyz/a/3654

I build Xtrata, the on-chain storage layer the write-up is inscribed with, so read it knowing that.

First time on this subreddit, I am publicizing a full reverse of Akamai, all documentation is in this repository. Fully native, the throughput should be fast on any machine and scale pretty well. You might have to add some of your own fingerprints though. Maybe I'll add more later.

https://github.com/9xz/reverseakamai

(NO SANDBOXING, FULLY NATIVE) !!!!

Enjoy!

CF WAF & Turnstiles will be next!

Hi everyone,

I'm looking for someone who can help me scrape data from Marktplaats.nl (Dutch marketplace).

I have a list of Marktplaats seller accounts and would like to collect information about each seller, specifically:

• Seller account/profile

• Number of active advertisements/listings associated with each seller

I have a list of seller accounts that I want to process, potentially in large quantities.

Let me know if interested.

Thanks.

https://github.com/maximilianfeix/proxy-scraper

Something I measured this week that might save you some proxies: a lot of the ones your checker marks dead aren't dead, they're flaky. In my runs about 350 proxies an hour passed the first request and then timed out on the second, even with an immediate retry 1.5 s later. I now give those a second try at the end of the run, at least 20 seconds later. In an A/B run (two GitHub runners started at the same time, 150k candidates each) 296 of 828 came back, and the list went from 768 to 986.

The other change was the timeout. About 10k candidates per run connected but needed more than 6 s to answer. With 10 s and the second try the list went to 1,177. 15 s added only 6 % more, and slower ones, so I stayed at 10.

What didn't help: removing the 150k candidate limit (1,111 vs 1,095) and adding six new self-checking lists (+3, their proxies were already in the other lists).

Plain lists per protocol and country: https://github.com/maximilianfeix/free-proxy-list

I'm looking for a way to mass download someones media imagine posts on x, all of the grabbers i could find online either don't work, or only let you grab like 10 imgs then ask for a ton of money. does anyone know one that works and is free?

Proxies are fine, your headers match a real browser, and you still get blocked. Usually the site isn't checking one thing. It looks at the whole picture your browser gives off: OS, GPU, fonts, language, screen and more. Here's what's in that picture.

Layer 1: what the server sees before any JS runs

• IP. Geolocation, and whether it's an ISP, datacenter or residential address. WebRTC can expose a different IP if you don't handle it.

• User-Agent. Browser, version, OS. Easy to change, so on its own it means little.

• Client Hints (Sec-CH-UA*). Chromium sends a structured version of the same info. It has to match the User-Agent.

• Other headers. Accept-Language, Accept-Encoding, and sometimes even header order, which differs between engines.

Layer 2: basic values read with JS

• Screen size, color depth, pixel ratio.

• Languages and timezone.

• navigator.platform, CPU threads, a coarse memory value in Chromium.

• navigator.webdriver and other automation markers.

• Some extensions (like blockers) can be detected indirectly.

Each one is weak alone, since lots of people share the same screen or language. The combo is what matters.

Layer 3: how your device renders things

This is harder to fake. The page asks your browser to draw or compute something, then reads the result:

• Canvas: text and shapes drawn on a hidden canvas. OS text rendering, GPU and drivers change the pixels slightly.

• WebGL: the renderer string (often names the GPU) and a profile of its limits.

• Audio: a generated signal run through the Web Audio engine.

• Fonts: which ones are installed, inferred by measuring text.

Layer 4: the ID

A few dozen values get hashed into a short ID (FingerprintJS does this). The site stores it and matches it on the next visit, so a new IP with the same fingerprint can still look like the same client, and a clean fingerprint on a flagged IP can still get blocked. Older studies like Panopticlick and AmIUnique found a large share of browsers were unique, while larger-scale ones found lower numbers. So think of it as one probabilistic signal that sites mix with IP reputation, cookies and behavior, not a perfect ID.

Sites also ask "does this even make sense?", not just "who is this?". Common mismatches:

• Windows User-Agent, but navigator.platform says something else.

• Chrome User-Agent with no Client Hints.

• Timezone or language that doesn't match the IP.

• Windows User-Agent with a software GPU renderer or a Linux-style font list.

• Hardware values that don't exist on real devices.

What to do with this

1. Keep values stable within a session.

2. Values for the same device (OS, platform, Client Hints, fonts) should match each other.

3. Build one coherent fingerprint for the specific task (target site, region, device type) and stick with it, instead of patching values one by one.

I am looking for an expert automation developer to build a fast, concurrent slot-booking system managed via a private Telegram Bot.

Target URL & Schedule:

https://www.goethe.de/ins/dz/fr/m/spr/prf.html

The registration dates are publicly listed on the page, and slots open at exactly 12:00 PM Algeria Time (GMT+1). Due to massive concurrent traffic, slots run out in seconds.

System Requirements:

1. Telegram Bot Interface: Allows me to input and queue multiple candidates before registration day (MeinGoethe login credentials, exam level A1-C2, modules, First Name, Last Name, Birth Date).

2. Pre-Login & Session Warmup: 10 minutes before 12:00 PM (GMT+1), the backend logs into all candidate accounts using isolated residential proxies (1 proxy per candidate) to maintain active sessions.

3. Concurrent Checkout Execution: At 12:00:00 PM, the bot monitors the target URL, detects when the booking button activates, selects the exam modules (.cs-checkbox__input), selects "book for myself", fills the required inputs, and submits the reservation.

4. No Payment Gateway Needed: The portal does not require online payment during checkout; the process ends at the final order confirmation screen.

5. Proof of Completion: The bot captures a screenshot of the final confirmation receipt for each candidate and sends it directly to the Telegram chat.

Please apply if you have prior experience with high-concurrency checkout bots, Playwright/Puppeteer stealth, or direct HTTP request automation.

if you have an old script pulling substack archives, check it. /api/v1/archive?offset=... returns 400 now. it took me a while to figure out what changed

what works now:

let cursor = ''
while (true) {
  const url = \https://NAME.substack.com/api/v1/archive?sort=new&limit=12${cursor ? '&cursor=' + cursor : ''}`  const { posts, nextCursor } = await (await fetch(url)).json()  // do something with posts  if (!nextCursor) break  cursor = encodeURIComponent(nextCursor)}`

/api/v1/archive?sort=new&limit=12 gives you {posts, nextCursor}, then pass &cursor=<nextCursor> for the next page
limit is capped at 12, asking for more doesnt give you more

the old search endpoint is gone too, substack.com/api/v1/top/search?query=... is the one that works, the publications are inside the profile results

/feed (rss) still works if you only need the last ~20 posts

also small thing, post.wordCount and the actual body length don’t always match, so dont rely on it for filtering

posting because i found a bunch of old threads with scripts that break on exactly this

Im vibecoding my project and im having a little trouble scraping one website in particular. Seems like they are using DataDome + Imperva.

Claude code and ChatGPT are not being able to get past it.

Any one knows how to do it?

Thanks

I am currently working on a project that requires extracting text, tables, and layouts from scanned images. I have tried using standard OCR tools, but the accuracy has been inadequate.

I am operating in a CPU-only environment and would prefer not to deploy heavy machine learning models. Are there any alternative approaches, lightweight tools, or specific techniques that can reliably achieve an extraction accuracy of 95% or higher under these constraints?

I have used ocr : Tesseract, Rapid Ocr, paddle ocr, doctr, easyocr ocrmypdf

Hi everyone,

I'm trying to learn more about residential IP addresses and network configuration.

I have a few questions:

1. Is it possible to assign a unique residential IP address to each browser profile?

2. How can I ensure that each profile maintains an independent network connection?

3. Can different IP ranges be assigned to separate profiles?

4. What are the technical considerations when managing approximately 50 separate network connections?

I'm a beginner in this area and would appreciate any explanations or educational resources that could help me understand how this works.

Thank you!

So I've been doing a goodreads scraping project for awhile. I'm looking to compare notes with other goodreads scraper folks. the below numbers give a flavor of what I've gotten so far. I just recently discovered the site map - so seems like there are 9.5 million author_id and 150,000 lists.

( UPDATE: after making this post, I started publishing details of my scrape at https://philmcrew.com/goodreads_zeitgeist/per-field.html)

The short of it is I have

• 18m book_id

• 400k detailed books (every work with more than 1000 ratings and more)

• 1.1m author_id

• 5.7m work_id

• 4087 distinct book tags

• 75k lists with 91,235 distinct list tags

Books

• ----------------------------------------------------------------------

• … books: field totals…

• … books: distinct counts…

• total : 17,987,973 100.0% ✓ complete

• author_id : 17,938,309 99.7% (49,664 missing) · 1,333,260 distinct

• ratings : 17,987,973 100.0% ✓ complete

• avg_rating : 17,978,021 99.9% (9,952 missing) · 415 distinct

• published : 4,081,236 22.7% (13,906,737 missing)

• pages : 381,066 2.1% (17,606,907 missing) · 1,920 distinct

• series_pos : 1,796,442 10.0% (16,191,531 missing) · 7,189 distinct

• genres : 360,926 2.0% (17,627,047 missing)

• tags : 753,422 4.2% (17,234,551 missing)

• work_id : 6,372,169 35.4% (11,615,804 missing) · 5,790,484 distinct

• ratings_zero : 11,563,215 · books with exactly 0 ratings

• … books: scanning genres/tags JSON (can take a while)…

• genres : distinct genre labels 1,410

• tags : distinct tag/shelf names 4,079

Book Details

• total : 394,895 100.0% ✓ complete

• publisher : 389,552 98.6% (5,343 missing) · 52,603 distinct

• isbn13 : 278,329 70.5% (116,566 missing)

• isbn10 : 270,280 68.4% (124,615 missing)

• asin : 388,574 98.4% (6,321 missing)

• format : 393,267 99.6% (1,628 missing) · 346 distinct

• language : 385,411 97.6% (9,484 missing) · 97 distinct

• description : 390,975 99.0% (3,920 missing)

• series : 204,121 51.7% (190,774 missing)

• reviews_count : 394,610 99.9% (285 missing)

• ratings_dist : 394,610 99.9% (285 missing)

• currently_reading : 392,966 99.5% (1,929 missing)

• to_read : 394,203 99.8% (692 missing)

• editions_count : 368,108 93.2% (26,787 missing)

Authors

• total : 1,084,031 100.0% ✓ complete

• average_rating : 792,753 73.1% (291,278 missing)

• num_ratings : 730,873 67.4% (353,158 missing)

• num_reviews : 621,310 57.3% (462,721 missing)

• num_shelves : 768,262 70.9% (315,769 missing)

• slug : 930,956 85.9% (153,075 missing) · 930,956 distinct

• catalog_pages : 847,609 78.2% (236,422 missing)

• source : 144,435 13.3% (939,596 missing) · 10 distinct

Book Tags

• total rows : 4,572,460

• distinct tags : 4,087

• distinct book IDs : 781,592

• distinct harvests : 6,937

• position : 4,572,357 100.0% (103 missing)

• shelved : 4,572,460 100.0% ✓ complete

• shelved min : 0 max 3,165,909 avg 210.6

What I should do in a website after having logged in is:

-Search for a file
-Download it

for a couple hundred files of the same type.

I don't have previous experience with webscraping. Is it possible to automate this process and if yes, is it possible to do it for free?

Before I set bids on a new Google Search + Shopping test, I do an "auction read". For each keyword I record who is advertising, whether their ads sit at the top or bottom, their headlines and offers, and which Shopping listings show and at what price.

My list is about 150 keywords, US only.

What I'm doing now:

Ad Preview and Diagnosis tool, set to United States / English / mobile. My campaigns are paused, and the preview inside the tool shows no ads while they're paused. So I open the preview frame as its own page, which does show the ads.

Checking the live Google results page for Shopping ads.

The problems:

It's slow. I'm only about 70 keywords in after more than a day doing it myself

I hit a Google captcha roughly every 15 searches, even at one search every 5 minutes when I use claude.

The live results page stops showing Shopping ads after about 3 searches from the same browser.

Auction Insights doesn't help, because nothing is running yet.

How do you read the competition for a new keyword list before launch? Is checking every keyword overkill?

Any tools that pull US paid results (text ads and Shopping) in bulk?

Is Ad Preview a fair picture of the live auction? It shows zero top-of-page ads on every keyword I've checked, and I'm not sure if that's real or a quirk of the tool.

Any workflow tips appreciated.

I have tried Tools that can download the webpages or help you download the webpage automatically by just pasting the link (like youzim.it) but I am really frustrated with internet, like there are many things that I wanted to download and then read them because I don't have that much of internet (my internet speed is like 50-100 kb and I can't even load a simple webpage)

As for the title, I need some recommendations/ tools that could help me to download webpages.

Thank you

IM PRETTY DESPERATE.

I'm hoping someone here with more experience can point me in the right direction because I've been going in circles trying to figure this out.

I'm working on a personal music discovery project. I want to search approximately five years of discussions in a particular subreddit and build a massive list of artists and bands that people have recommended over the years.

The biggest challenge is that I don't want to rely on Reddit's limited search results or just collect the most popular posts. I'm trying to dig as deep as possible, including older discussions, overlooked threads, obscure recommendations, and names buried in comment replies.

Ideally, I'd like to:

- Search hundreds or thousands of historical threads using multiple keywords.

- Retrieve complete comment sections, including nested replies.

- Identify recommendation-related discussions while filtering out unrelated posts.

- Export the collected information into CSV files for further processing.

- Extract and deduplicate artist names to create one comprehensive list.

Here's what I've already tried:

I installed Python, PRAW, and Selenium on my Windows computer.

Unfortunately, Reddit requires API approval for my intended use, and my application was denied. I also tried Selenium with Firefox, but Reddit returned a network security block.

I've looked into a few open-source projects, but I'm unsure which ones are reliable or currently working.

What I'm looking for is a completely free solution that can handle a large amount of historical data. I don't mind letting a script run for several hours if necessary.

Has anyone successfully collected several years of Reddit posts and comments recently?

Are there any open-source tools, public Reddit archives, downloadable datasets, or other legitimate approaches you'd recommend?

I'm particularly interested in methods that can retrieve older discussions and their full comment trees rather than stopping after the first few hundred results.

This is strictly for personal music discovery, not commercial use or collecting personal information.

I'd really appreciate hearing what has actually worked for people rather than just theoretical suggestions.

Thanks in advance!

hi!

i have working on a cdp (chrome devtools protocol) library in rust

(cdp is the protocol used to manage chrome browsers programmatically)

initially, it started like a direct rewrite of a python implementation of the protocol but then i started trying to do the implementation in a proper rust-idiomatic way

here is the repo (rscdp)

any commentary, code reviews, or general feedback are really welcome!

I'm currently doing some data analysis on public tweets and I'm struggling to find the location info for individual tweets.

I know that the location field in the user's profile is just user-input text and can be fake.

I've heard about the new "About this account" feature and the account_based_in field, but that's for the account level, not for a specific tweet, and it requires an extra GraphQL request.

My question is: Is there any hidden field or reliable API endpoint in the current X web GraphQL that actually returns the real country (based on IP or device) for a single tweet? Or is it completely removed/abandoned by X?

I've searched previous threads but couldn't find a clear answer. Any insights would be greatly appreciated!

So at my company we’ve got about 10,000 SKUs. So far by using a mix of 5-6 OpenAI Luna calls for each, I can get 65-75% of the data I need for 0.50c a product…

However, that last 30% is quite difficult and requires a more persistent lookup which isn’t economically viable.

What approaches would people here recommend? I know deepseek has a really cheap web search api, but I’m wondering if some combination of manufacturer brand page search + non web search Luna call would suffice?

Curious to hear others thoughts!

Hello. I don't have too much experience with webscraping but i'm building a antidetect browser and future use, i would like to add some plugins for webscraping. If anyone has knowledge about this could you share the knowledge please?

If doing webscraping with antidetect browser is a waste of effort and there are better ways for beginners. I'm open to any kind of information.

Thanks in advance 🤝