r/learnpython • • Dec 05 '19

Python Scraping - Ignoring Loading Page

Hi All,

I am using Python and Beautiful Soup to scrape the following page: https://www.willhaben.at/iad/immobilien/immobilien/angebote?rows=100&areaId=900&AD_TYPE=1

Every now and then the page gives a "Loading" page instead of the actual page, which causes the script to bug. I try/catch the error, but occasionally it continues displaying the unwanted page.

How might I skip the Loading page? (waiting a couple of seconds after the page request opens the full page)

Thanks for any advice!

(This is what the loading page looks like: https://pastebin.com/UMpLBFaj)

123 Upvotes

20 comments sorted by

View all comments

1

u/Financial_Farm4326 23h ago

youre hitting a bot gate, not slow javascript. standard http libraries grab raw markup once and disconnect, meaning pausing your script accomplishes zero. fwiw, the free workaround is checking the body text for the blocker, sleeping briefly, then retrying inside a persistent session object so cookies carry over. otherwise spin up a headless browser using playwright or nodriver and pause until a product selector appears. when sites still block you based on tls fingerprints, Scrape-fly handles it upstream so your beautifulsoup code stays untouched. dig into the raw source for embedded json data first though, since scraping tags is fragile as hell.