Bug: main() called both enforce_fetch_interval() and mark_fetch(), which
both now call check_and_reserve(). On cold start:
1. enforce_fetch_interval() → check_and_reserve() → SUCCESS (reserves)
2. mark_fetch() → check_and_reserve() → FAILS (cooldown now active)
This prevented HTTP from ever being called on cold start.
Fix:
- Removed duplicate calls to enforce_fetch_interval() and mark_fetch()
- Single check_and_reserve() call before fetch_html()
- enforce_fetch_interval() and mark_fetch() remain as legacy wrappers
Verification:
- All 18 community_cli tests pass
- Code analysis confirms single check_and_reserve() call in main()
- check_and_reserve() is atomic with exclusive lock for check+reserve
Replace urlopen automatic redirect following with custom HTTPRedirectHandler
that raises on 3xx redirects. Each redirect hop is validated (scheme, host,
port) before the request is made using _validate_url_before_io().
Key changes:
- _StrictRedirectHandler intercepts 301/302/303/307/308 responses
- _extract_redirect_url() extracts Location header from redirect responses
- fetch_html() manually follows redirects with hop count limit (MAX_REDIRECT_HOPS=5)
- Relative redirect URLs resolved with urljoin() before validation
- All redirect targets validated against ALLOWED_HOSTS, ALLOWED_PORTS, HTTPS-only
Tests:
- test_fetch_html_redirect_to_disallowed_host_rejected (mocked redirect)
- test_fetch_html_redirect_chain_limit (exceeds MAX_REDIRECT_HOPS)
- test_extract_redirect_url_from_headers (Location/location headers)
- test_urljoin_resolves_relative_redirects (relative URL resolution)
- Single exclusive lock covers read-check-write in one critical section
- Lockfile pattern ensures cross-process mutual exclusion
- Atomic write via temp file + rename after unlock
- Flush + fsync before unlock to prevent data loss
- Real multi-process test: 3 concurrent processes get exactly 1 reservation
- 111 Python tests pass (+2 new tests)
- _write_state: write to temp file, fsync, rename atomically
- Acquire exclusive lock before any file operations
- Flush and fsync before unlock to prevent data loss
- Remove stale .tmp file after successful write
- Add test for atomic write behavior
- 109 Python tests pass
- Add _validate_url_host() to check allowlist before urlopen()
- Validate both original URL and redirect target
- Reject localhost, internal IPs, and non-allowlisted hosts
- Add 2 unit tests for disallowed host rejection
- Prevents SSRF attacks via malicious source URLs
ik