Bug: community_scheduler always had status='ready' even when individual
sources were failed or stale. Success of one source masked failure of another.
Fix:
- Overall status is 'degraded' if any enabled source has failed
- Overall status is 'stale' if all sources are stale but none failed
- Overall status is 'ready' only when at least one source is healthy
- Overall status is 'not_started' when no sources are enabled
- Readiness (ready flag) still NOT blocked by import health (A01 requirement)
Verification:
- 7/7 readiness tests pass
- 121/121 Python tests pass (1 skipped)
- Failed/stale sources are now visible in JSON without blocking scheduler
Bug: 'alembic heads' returns '48094a7d1b92 (head)', but DB query returns
only '48094a7d1b92'. String comparison failed due to '(head)' suffix.
Fix:
- Extract revision ID using grep -oE '^[a-f0-9]+' before space
- Handle multiple heads: check if DB version matches any head
- Add validation for empty outputs with clear error messages
- Add success message showing head and DB version
Verification:
- Script syntax: bash -n passes
- Handles single head (exact match)
- Handles multiple heads (DB version matches any)
Bug: load-more link used items.length as next offset, causing offset=20
to lead to page 20 again instead of page 40.
Fix: Use offset + items.length for correct next page calculation.
With 20 items per page: offset=0→20→40→...
Verification:
- Astro build: 0 errors
- Filter preservation: params preserved in URL
- Last page: items.length < totalItems check works
- Invalid offset: Number.isInteger check on line 16
Bug: main() called both enforce_fetch_interval() and mark_fetch(), which
both now call check_and_reserve(). On cold start:
1. enforce_fetch_interval() → check_and_reserve() → SUCCESS (reserves)
2. mark_fetch() → check_and_reserve() → FAILS (cooldown now active)
This prevented HTTP from ever being called on cold start.
Fix:
- Removed duplicate calls to enforce_fetch_interval() and mark_fetch()
- Single check_and_reserve() call before fetch_html()
- enforce_fetch_interval() and mark_fetch() remain as legacy wrappers
Verification:
- All 18 community_cli tests pass
- Code analysis confirms single check_and_reserve() call in main()
- check_and_reserve() is atomic with exclusive lock for check+reserve
Updated recovery report with:
- A03 updated: manual redirect control with _StrictRedirectHandler
- A04 updated: pagination offset duplicate fix
- A05-A13 verification status (all already implemented)
- Current test results and remaining risks
- Final acceptance summary
All A01-A13 regressions from September 9 audit are now verified and complete.
Remove existing offset parameter before adding new one to prevent
duplicate query parameters like ?offset=20&offset=40.
Fix: Use URLSearchParams.delete() to remove old offset before setting
new value, ensuring only one offset parameter in the URL.
Verified: Astro build succeeds with 0 errors
Replace urlopen automatic redirect following with custom HTTPRedirectHandler
that raises on 3xx redirects. Each redirect hop is validated (scheme, host,
port) before the request is made using _validate_url_before_io().
Key changes:
- _StrictRedirectHandler intercepts 301/302/303/307/308 responses
- _extract_redirect_url() extracts Location header from redirect responses
- fetch_html() manually follows redirects with hop count limit (MAX_REDIRECT_HOPS=5)
- Relative redirect URLs resolved with urljoin() before validation
- All redirect targets validated against ALLOWED_HOSTS, ALLOWED_PORTS, HTTPS-only
Tests:
- test_fetch_html_redirect_to_disallowed_host_rejected (mocked redirect)
- test_fetch_html_redirect_chain_limit (exceeds MAX_REDIRECT_HOPS)
- test_extract_redirect_url_from_headers (Location/location headers)
- test_urljoin_resolves_relative_redirects (relative URL resolution)
- Single exclusive lock covers read-check-write in one critical section
- Lockfile pattern ensures cross-process mutual exclusion
- Atomic write via temp file + rename after unlock
- Flush + fsync before unlock to prevent data loss
- Real multi-process test: 3 concurrent processes get exactly 1 reservation
- 111 Python tests pass (+2 new tests)
- Replace hardcoded '0013' with dynamic 'alembic heads' check
- Works with any current head revision
- Caddy adapt and scheduler checks already in place from previous fixes
- Bootstrap uses loopback ports and isolated compose profile
- Extend draft recovery to create_error, rate_limited, server_error, timeout
- Clear draft only on success (sent/screenshot_sent)
- Focus on form-error after recovery
- Double submit protection already in place (R10)
- Astro check: 0 errors
- Add selected={hours === '6/12/72'} to all period options (was only on 24)
- Ensures correct UI state when URL has hours=6/12/72
- CSS for filter-compact-hidden already correct (display:none!important)
- Filter fallback details working for no-JS mobile
- Pagination (R09) already handles offset preservation
- _write_state: write to temp file, fsync, rename atomically
- Acquire exclusive lock before any file operations
- Flush and fsync before unlock to prevent data loss
- Remove stale .tmp file after successful write
- Add test for atomic write behavior
- 109 Python tests pass
- Infrastructure (DB/MinIO) blocks readiness; imports are diagnostic only
- Per-source community scheduler health with backoff detection
- Stale/failed imports never block /ready — scheduler can recover them
- Add 'blocking: false' to all import components
- 4 new tests: per-source health, backoff detection, stale/failed non-blocking
- 108 Python tests pass
- Add ImportRecordEvent model to track per-record import changes
- Log created/updated events for each official record import
- Add alembic migration 0014 for import_record_event table
- Enables audit trail for which import run modified which records
- Add trailingSlash: 'never' to Astro config
- Normalize sitemap paths to never use trailing slash
- Ensures consistent canonical URLs across all pages
- Prevents duplicate content from / vs /path/ variants
- Generate requirements-lock.txt and requirements-dev-lock.txt via pip-compile
- CI uses locked files for reproducible installs
- Add web unit tests to CI (npm run test:unit)
- Add dependency-audit job using pip-audit
- Add Makefile with lock/lock-dev targets for regeneration
D04: Add fish name-based fallback in _auto_publish (was external_id only)
D06: Cap confidence at 50% for 1 player, 65% for 2 players
D07: Set caught_at=None for community imports (not published_at)
D08: Already OK - activity_rows has no top-100 limit
- Add Fish import to community_importer.py
- Add 2 unit tests for D06 confidence caps
- Update test_community_importer.py for D04 name match behavior
- Add _is_trusted_proxy() to check client IP against trusted CIDRs
- Only use X-Forwarded-For if connection came from trusted proxy
- Add TRUSTED_PROXY_CIDRS config (default: 127.0.0.1/32, ::1/128)
- Add parse_comma_separated_lists for env var parsing
- Add 3 unit tests: trusted CIDR check, untrusted ignores forwarded, trusted uses forwarded
- Add 'ready = ready and healthy' for community_scheduler check
- Add 'ready = False' for community_scheduler exception path
- Add 3 unit tests: success=ready, stale=not_ready, failed=not_ready
- Monitoring now correctly reports community import health
- Add _validate_url_host() to check allowlist before urlopen()
- Validate both original URL and redirect target
- Reject localhost, internal IPs, and non-allowlisted hosts
- Add 2 unit tests for disallowed host rejection
- Prevents SSRF attacks via malicious source URLs
- AbortSignal.timeout() throws DOMException with name='TimeoutError', not TypeError
- Check for DOMException.TimeoutError, TypeError(fetch), or Error(abort)
- Apply same fix to report.ts and report-screenshot.ts
- Timeout redirects to 'timeout' state, other errors to 'create_error'
ik