r/CloudandCode • u/yourclouddude Founder | YourCloudDude • 29d ago
Guide [Python] A small website checker that logs HTTP errors and timeouts to CSV
If you're comfortable with Python loops, try this as your next project: a URL checker that records each result in a CSV and keeps going when a request fails.
You'll practise exceptions and file handling. The whole exercise runs locally, so you don't need an AWS account.
1. Save the checker
Install Requests in the Python environment you'll use for the script:
python -m pip install requestsSave this as checker.py:
import csv
import sys
from datetime import datetime, timezone
from time import perf_counter
import requests
urls = sys.argv[1:] or ["https://example.com"]
with requests.Session() as session, open(
"health_checks.csv", "a", newline="", encoding="utf-8"
) as output:
writer = csv.writer(output)
if output.tell() == 0:
writer.writerow(["checked_at_utc", "url", "status", "elapsed_ms", "error"])
for url in urls:
checked_at = datetime.now(timezone.utc).isoformat()
started = perf_counter()
status, error = "", ""
try:
with session.get(
url, timeout=(3, 5), stream=True, allow_redirects=False
) as response:
status = response.status_code
if status >= 400:
error = f"HTTP_{status}"
except requests.RequestException as exc:
error = type(exc).__name__
elapsed_ms = round((perf_counter() - started) * 1000, 1)
writer.writerow([checked_at, url, status, elapsed_ms, error])
output.flush()
print(url, status or "no response", elapsed_ms, error or "headers received")Pass URLs as command-line arguments. If you leave them out, python checker.py checks example.com once. Each run adds rows to health_checks.csv in your current folder.
2. Make it fail on purpose
Create an empty practice folder and start a local server there:
python -m http.server 8000 --bind 127.0.0.1Leave it running. Open a second terminal in the folder containing checker.py and run:
python checker.py http://127.0.0.1:8000/ http://127.0.0.1:8000/missing-pageOpen the CSV. You should see one row for each request: 200 for the directory page, then 404 with HTTP_404 for the missing path.
Now stop the server with Ctrl+C and run the checker again. The new rows should have an empty status and ConnectionError. The earlier 404 was a reply from the server; this time, the checker didn't receive an HTTP response.
3. Read the results
timeout=(3, 5)sets connect and read timeouts. The request doesn't have a guaranteed eight-second deadline.This script reads the response headers and closes the response without downloading the full body.
elapsed_msincludes that work; it isn't a full page-load measurement.Redirects are logged without being followed. A
302row doesn't tell you whether the destination works.A
200response doesn't tell you whether login, checkout or a database query works. A403or429could mean an access restriction or rate limit, so neither automatically means the whole site is down.
Local checks covered 200, 404, 503, an unfollowed 302, a delayed-header timeout, a refused connection and an invalid URL. A second run also appended rows without repeating the CSV header.
For the next version, give each URL an expected status and log any mismatch. If a URL is supposed to redirect, what else would you check before calling it healthy?
References: Requests timeouts and streaming responses.