Replace urlopen automatic redirect following with custom HTTPRedirectHandler
that raises on 3xx redirects. Each redirect hop is validated (scheme, host,
port) before the request is made using _validate_url_before_io().
Key changes:
- _StrictRedirectHandler intercepts 301/302/303/307/308 responses
- _extract_redirect_url() extracts Location header from redirect responses
- fetch_html() manually follows redirects with hop count limit (MAX_REDIRECT_HOPS=5)
- Relative redirect URLs resolved with urljoin() before validation
- All redirect targets validated against ALLOWED_HOSTS, ALLOWED_PORTS, HTTPS-only
Tests:
- test_fetch_html_redirect_to_disallowed_host_rejected (mocked redirect)
- test_fetch_html_redirect_chain_limit (exceeds MAX_REDIRECT_HOPS)
- test_extract_redirect_url_from_headers (Location/location headers)
- test_urljoin_resolves_relative_redirects (relative URL resolution)
- Single exclusive lock covers read-check-write in one critical section
- Lockfile pattern ensures cross-process mutual exclusion
- Atomic write via temp file + rename after unlock
- Flush + fsync before unlock to prevent data loss
- Real multi-process test: 3 concurrent processes get exactly 1 reservation
- 111 Python tests pass (+2 new tests)
- Replace hardcoded '0013' with dynamic 'alembic heads' check
- Works with any current head revision
- Caddy adapt and scheduler checks already in place from previous fixes
- Bootstrap uses loopback ports and isolated compose profile
- Extend draft recovery to create_error, rate_limited, server_error, timeout
- Clear draft only on success (sent/screenshot_sent)
- Focus on form-error after recovery
- Double submit protection already in place (R10)
- Astro check: 0 errors
- Add selected={hours === '6/12/72'} to all period options (was only on 24)
- Ensures correct UI state when URL has hours=6/12/72
- CSS for filter-compact-hidden already correct (display:none!important)
- Filter fallback details working for no-JS mobile
- Pagination (R09) already handles offset preservation
- _write_state: write to temp file, fsync, rename atomically
- Acquire exclusive lock before any file operations
- Flush and fsync before unlock to prevent data loss
- Remove stale .tmp file after successful write
- Add test for atomic write behavior
- 109 Python tests pass
- Infrastructure (DB/MinIO) blocks readiness; imports are diagnostic only
- Per-source community scheduler health with backoff detection
- Stale/failed imports never block /ready — scheduler can recover them
- Add 'blocking: false' to all import components
- 4 new tests: per-source health, backoff detection, stale/failed non-blocking
- 108 Python tests pass
- Add ImportRecordEvent model to track per-record import changes
- Log created/updated events for each official record import
- Add alembic migration 0014 for import_record_event table
- Enables audit trail for which import run modified which records
- Add trailingSlash: 'never' to Astro config
- Normalize sitemap paths to never use trailing slash
- Ensures consistent canonical URLs across all pages
- Prevents duplicate content from / vs /path/ variants
- Generate requirements-lock.txt and requirements-dev-lock.txt via pip-compile
- CI uses locked files for reproducible installs
- Add web unit tests to CI (npm run test:unit)
- Add dependency-audit job using pip-audit
- Add Makefile with lock/lock-dev targets for regeneration
D04: Add fish name-based fallback in _auto_publish (was external_id only)
D06: Cap confidence at 50% for 1 player, 65% for 2 players
D07: Set caught_at=None for community imports (not published_at)
D08: Already OK - activity_rows has no top-100 limit
- Add Fish import to community_importer.py
- Add 2 unit tests for D06 confidence caps
- Update test_community_importer.py for D04 name match behavior
- Add _is_trusted_proxy() to check client IP against trusted CIDRs
- Only use X-Forwarded-For if connection came from trusted proxy
- Add TRUSTED_PROXY_CIDRS config (default: 127.0.0.1/32, ::1/128)
- Add parse_comma_separated_lists for env var parsing
- Add 3 unit tests: trusted CIDR check, untrusted ignores forwarded, trusted uses forwarded
- Add 'ready = ready and healthy' for community_scheduler check
- Add 'ready = False' for community_scheduler exception path
- Add 3 unit tests: success=ready, stale=not_ready, failed=not_ready
- Monitoring now correctly reports community import health
- Add _validate_url_host() to check allowlist before urlopen()
- Validate both original URL and redirect target
- Reject localhost, internal IPs, and non-allowlisted hosts
- Add 2 unit tests for disallowed host rejection
- Prevents SSRF attacks via malicious source URLs
- AbortSignal.timeout() throws DOMException with name='TimeoutError', not TypeError
- Check for DOMException.TimeoutError, TypeError(fetch), or Error(abort)
- Apply same fix to report.ts and report-screenshot.ts
- Timeout redirects to 'timeout' state, other errors to 'create_error'
ik