Replace urlopen automatic redirect following with custom HTTPRedirectHandler
that raises on 3xx redirects. Each redirect hop is validated (scheme, host,
port) before the request is made using _validate_url_before_io().
Key changes:
- _StrictRedirectHandler intercepts 301/302/303/307/308 responses
- _extract_redirect_url() extracts Location header from redirect responses
- fetch_html() manually follows redirects with hop count limit (MAX_REDIRECT_HOPS=5)
- Relative redirect URLs resolved with urljoin() before validation
- All redirect targets validated against ALLOWED_HOSTS, ALLOWED_PORTS, HTTPS-only
Tests:
- test_fetch_html_redirect_to_disallowed_host_rejected (mocked redirect)
- test_fetch_html_redirect_chain_limit (exceeds MAX_REDIRECT_HOPS)
- test_extract_redirect_url_from_headers (Location/location headers)
- test_urljoin_resolves_relative_redirects (relative URL resolution)
- Single exclusive lock covers read-check-write in one critical section
- Lockfile pattern ensures cross-process mutual exclusion
- Atomic write via temp file + rename after unlock
- Flush + fsync before unlock to prevent data loss
- Real multi-process test: 3 concurrent processes get exactly 1 reservation
- 111 Python tests pass (+2 new tests)
- _write_state: write to temp file, fsync, rename atomically
- Acquire exclusive lock before any file operations
- Flush and fsync before unlock to prevent data loss
- Remove stale .tmp file after successful write
- Add test for atomic write behavior
- 109 Python tests pass
- Add _validate_url_host() to check allowlist before urlopen()
- Validate both original URL and redirect target
- Reject localhost, internal IPs, and non-allowlisted hosts
- Add 2 unit tests for disallowed host rejection
- Prevents SSRF attacks via malicious source URLs
ik