r/webscraping • u/godz_ares • Apr 26 '25
Scraping coordinates, tried everything. ChatGPT even failed
Hi all,
Context:
I am creating a data engineering project. The aim is to create a tool where rock climbing crags (essentially a set of climbable rocks) are paired with weather data so someone could theoretically use this to plan which crags to climb in the next five days depending on the weather.
There are no publicly available APIs and most websites such as UKC and theCrag have some sort of protection like Cloudflare. Because of this I am scraping a website called Crag27.
Because this is my first scraping project I am scraping page by page, starting from the end point 'routes' and ending with the highest level 'continents'. After this, I want to adapt the code to create a fully working web crawler.
The Problem:
I want to scrape the coordinates of the crag. This is important as I can use the coordinates as an argument when I use the weather API. That way I can pair the correct weather data with the correct crags.
However, this is proving to be insanely difficulty.
I started with Scrapy and used XPath notation: //div[@class="description"]/text() and my code looked like this:
import scrapy
from scrapy.crawler import CrawlerProcess
import csv
import os
import pandas as pd
class CragScraper(scrapy.Spider):
name = 'crag_scraper'
def start_requests(self):
yield scrapy.Request(url='https://27crags.com/crags/brimham/topos/atlantis-31159', callback=self.parse)
def parse(self, response):
sector = response.xpath('//*[@id="sectors-dropdown"]/span[1]/text()').get()
self.save_sector([sector]) # Changed to list to match save_routes method
def save_sector(self, sectors): # Renamed to match the call in parse method
with open('sectors.csv', 'w', newline='') as f:
writer = csv.writer(f)
writer.writerow(['sector'])
for sector in sectors:
writer.writerow([sector])
# Create a CrawlerProcess instance to run the spider
process = CrawlerProcess()
process.crawl(CragScraper)
process.start()
# Read the saved routes from the CSV file
sectors_df = pd.read_csv('sectors.csv')
print(sectors_df) # Corrected variable nameHowever, this didn't work. Being new and I out of ideas I asked ChatGPT what was wrong with the code and it bought me down a winding passage of using playwright, simulating a browser and intercepting an API call. Even after all the prompting in the world, ChatGPT gave up and recommended hard coding the coordinates.
This all goes beyond my current understanding of scraping but I really want to do this project.
This his how my code looks now:
from playwright.sync_api import sync_playwright
import json
import csv
import pandas as pd
from pathlib import Path
def scrape_sector_data():
with sync_playwright() as p:
browser = p.chromium.launch(headless=False) # Show browser
context = browser.new_context()
page = context.new_page()
# Intercept all network requests
sector_data = {}
def handle_response(response):
if 'graphql' in response.url:
try:
json_response = response.json()
if 'data' in json_response:
# Look for 'topo' inside GraphQL data
if 'topo' in json_response['data']:
print("✅ Found topo data!")
sector_data.update(json_response['data']['topo'])
except Exception as e:
pass # Ignore non-JSON responses
page.on('response', handle_response)
# Go to the sector page
page.goto('https://27crags.com/crags/brimham/topos/atlantis-31159', wait_until="domcontentloaded", timeout=60000)
# Give Playwright a few seconds to capture responses
page.wait_for_timeout(5000)
if sector_data:
# Save sector data
topo_name = sector_data.get('name', 'Unknown')
crag_name = sector_data.get('place', {}).get('name', 'Unknown')
lat = sector_data.get('place', {}).get('lat', 0)
lon = sector_data.get('place', {}).get('lon', 0)
print(f"Topo Name: {topo_name}")
print(f"Crag Name: {crag_name}")
print(f"Latitude: {lat}")
print(f"Longitude: {lon}")
with open('sectors.csv', 'w', newline='') as f:
writer = csv.writer(f)
writer.writerow(['topo_name', 'crag_name', 'latitude', 'longitude'])
writer.writerow([topo_name, crag_name, lat, lon])
else:
print("❌ Could not capture sector data from network requests.")
browser.close()
# Run the scraper
scrape_sector_data()
# Read and display CSV if created
csv_path = Path('sectors.csv')
if csv_path.exists():
sectors_df = pd.read_csv(csv_path)
print("\nScraped Sector Data:")
print(sectors_df)
else:
print("\nCSV file was not created because no sector data was found.")Can anyone lend me some help?
