Bug: Layout used 'noindex && path !== ""' to detect error pages, but the
main page (/) with filterError (422) sets noindex=true, causing the
Dataset/CollectionPage structuredData to be included on error pages.
Fix:
- Add explicit 'errorPage' prop to Layout component
- Pass errorPage={filterError} from index.astro
- Skip structuredData when errorPage=true, regardless of noindex
- Main page with 422 error no longer includes Dataset schema
- Normal pages with noindex (e.g., /admin) still work correctly
Verification:
- Astro build: 0 errors
- Error pages (422, 503, 404) skip structuredData
- Normal pages include structuredData
- noindex still works for robots meta tag
Bug: review_note was empty or contained arbitrary text, not explaining
how the observation was matched to fish/waterbody.
Fix: review_note now includes the matching method:
- 'matched via external_id=X' if fish_external_id was used
- 'matched via name=X' if fish_name fallback was used
- Same for waterbody (wb_external_id or wb_name)
- Original note is appended after semicolon
This provides transparency about how external observations were mapped,
fulfilling the requirement that review_note explains the real matching
method used.
Verification:
- 124/124 Python tests pass
- Existing tests still pass (review_note is optional parameter)
- New review_note format is machine-readable and human-friendly
Added 3 new tests for A06 proxy chain verification:
1. test_rate_limit_independent_limits_for_two_clients_through_proxy
- Two clients behind trusted proxy have independent rate limits
- Client 1 blocked after 5 requests, Client 2 still allowed
2. test_forged_xff_rejected_on_untrusted_port
- XFF from untrusted connection is ignored
- Real client IP used for rate limiting, not forged XFF
3. test_direct_access_without_xff_header
- Direct access without XFF uses real client IP
- Hash is of real IP, not empty string
Verification:
- 8/8 rate limit tests pass
- Docker network CIDR (172.17.0.0/16) tested
- Forged XFF properly rejected from untrusted sources
- Independent rate limits verified for multiple clients
Bug: sessionStorage operations (getItem, setItem, removeItem) could fail
if storage is unavailable (private mode, quota exceeded, etc.), causing
form draft recovery to break.
Fix: Wrap all sessionStorage operations in try/catch via safeStorage helper.
- safeStorage.getItem() - returns null on error
- safeStorage.setItem() - silently ignores errors
- safeStorage.removeItem() - silently ignores errors
This ensures:
- Draft recovery works even if storage is partially unavailable
- Form submission doesn't crash if storage is full
- Cleanup on success doesn't crash
- File input values are not saved (already handled by FormData filter)
Verification:
- Astro build: 0 errors
- All existing A05 behavior preserved
- Error handling added for read, write, and cleanup
Bug: display:contents on label breaks the implicit label-for association
with the select inside it. This causes accessibility issues and breaks
keyboard navigation on mobile.
Fix: Replace display:contents with display:flex;align-items:center;gap:6px
on desktop. This preserves the inline layout while maintaining the label
relationship with the select element.
Mobile behavior unchanged: filter-advanced-field is hidden via display:none
on screens <=720px, replaced by filter-advanced-fallback details element.
Verification:
- Astro build: 0 errors
- Label/select relationship preserved for keyboard navigation
- Desktop layout: flex row with gap
- Mobile: fallback details element shown
Bug: community_scheduler always had status='ready' even when individual
sources were failed or stale. Success of one source masked failure of another.
Fix:
- Overall status is 'degraded' if any enabled source has failed
- Overall status is 'stale' if all sources are stale but none failed
- Overall status is 'ready' only when at least one source is healthy
- Overall status is 'not_started' when no sources are enabled
- Readiness (ready flag) still NOT blocked by import health (A01 requirement)
Verification:
- 7/7 readiness tests pass
- 121/121 Python tests pass (1 skipped)
- Failed/stale sources are now visible in JSON without blocking scheduler
Bug: 'alembic heads' returns '48094a7d1b92 (head)', but DB query returns
only '48094a7d1b92'. String comparison failed due to '(head)' suffix.
Fix:
- Extract revision ID using grep -oE '^[a-f0-9]+' before space
- Handle multiple heads: check if DB version matches any head
- Add validation for empty outputs with clear error messages
- Add success message showing head and DB version
Verification:
- Script syntax: bash -n passes
- Handles single head (exact match)
- Handles multiple heads (DB version matches any)
Bug: load-more link used items.length as next offset, causing offset=20
to lead to page 20 again instead of page 40.
Fix: Use offset + items.length for correct next page calculation.
With 20 items per page: offset=0→20→40→...
Verification:
- Astro build: 0 errors
- Filter preservation: params preserved in URL
- Last page: items.length < totalItems check works
- Invalid offset: Number.isInteger check on line 16
Bug: main() called both enforce_fetch_interval() and mark_fetch(), which
both now call check_and_reserve(). On cold start:
1. enforce_fetch_interval() → check_and_reserve() → SUCCESS (reserves)
2. mark_fetch() → check_and_reserve() → FAILS (cooldown now active)
This prevented HTTP from ever being called on cold start.
Fix:
- Removed duplicate calls to enforce_fetch_interval() and mark_fetch()
- Single check_and_reserve() call before fetch_html()
- enforce_fetch_interval() and mark_fetch() remain as legacy wrappers
Verification:
- All 18 community_cli tests pass
- Code analysis confirms single check_and_reserve() call in main()
- check_and_reserve() is atomic with exclusive lock for check+reserve
Updated recovery report with:
- A03 updated: manual redirect control with _StrictRedirectHandler
- A04 updated: pagination offset duplicate fix
- A05-A13 verification status (all already implemented)
- Current test results and remaining risks
- Final acceptance summary
All A01-A13 regressions from September 9 audit are now verified and complete.
Remove existing offset parameter before adding new one to prevent
duplicate query parameters like ?offset=20&offset=40.
Fix: Use URLSearchParams.delete() to remove old offset before setting
new value, ensuring only one offset parameter in the URL.
Verified: Astro build succeeds with 0 errors
Replace urlopen automatic redirect following with custom HTTPRedirectHandler
that raises on 3xx redirects. Each redirect hop is validated (scheme, host,
port) before the request is made using _validate_url_before_io().
Key changes:
- _StrictRedirectHandler intercepts 301/302/303/307/308 responses
- _extract_redirect_url() extracts Location header from redirect responses
- fetch_html() manually follows redirects with hop count limit (MAX_REDIRECT_HOPS=5)
- Relative redirect URLs resolved with urljoin() before validation
- All redirect targets validated against ALLOWED_HOSTS, ALLOWED_PORTS, HTTPS-only
Tests:
- test_fetch_html_redirect_to_disallowed_host_rejected (mocked redirect)
- test_fetch_html_redirect_chain_limit (exceeds MAX_REDIRECT_HOPS)
- test_extract_redirect_url_from_headers (Location/location headers)
- test_urljoin_resolves_relative_redirects (relative URL resolution)
- Single exclusive lock covers read-check-write in one critical section
- Lockfile pattern ensures cross-process mutual exclusion
- Atomic write via temp file + rename after unlock
- Flush + fsync before unlock to prevent data loss
- Real multi-process test: 3 concurrent processes get exactly 1 reservation
- 111 Python tests pass (+2 new tests)
- Replace hardcoded '0013' with dynamic 'alembic heads' check
- Works with any current head revision
- Caddy adapt and scheduler checks already in place from previous fixes
- Bootstrap uses loopback ports and isolated compose profile
- Extend draft recovery to create_error, rate_limited, server_error, timeout
- Clear draft only on success (sent/screenshot_sent)
- Focus on form-error after recovery
- Double submit protection already in place (R10)
- Astro check: 0 errors
- Add selected={hours === '6/12/72'} to all period options (was only on 24)
- Ensures correct UI state when URL has hours=6/12/72
- CSS for filter-compact-hidden already correct (display:none!important)
- Filter fallback details working for no-JS mobile
- Pagination (R09) already handles offset preservation
- _write_state: write to temp file, fsync, rename atomically
- Acquire exclusive lock before any file operations
- Flush and fsync before unlock to prevent data loss
- Remove stale .tmp file after successful write
- Add test for atomic write behavior
- 109 Python tests pass
- Infrastructure (DB/MinIO) blocks readiness; imports are diagnostic only
- Per-source community scheduler health with backoff detection
- Stale/failed imports never block /ready — scheduler can recover them
- Add 'blocking: false' to all import components
- 4 new tests: per-source health, backoff detection, stale/failed non-blocking
- 108 Python tests pass
- Add ImportRecordEvent model to track per-record import changes
- Log created/updated events for each official record import
- Add alembic migration 0014 for import_record_event table
- Enables audit trail for which import run modified which records
- Add trailingSlash: 'never' to Astro config
- Normalize sitemap paths to never use trailing slash
- Ensures consistent canonical URLs across all pages
- Prevents duplicate content from / vs /path/ variants
- Generate requirements-lock.txt and requirements-dev-lock.txt via pip-compile
- CI uses locked files for reproducible installs
- Add web unit tests to CI (npm run test:unit)
- Add dependency-audit job using pip-audit
- Add Makefile with lock/lock-dev targets for regeneration
D04: Add fish name-based fallback in _auto_publish (was external_id only)
D06: Cap confidence at 50% for 1 player, 65% for 2 players
D07: Set caught_at=None for community imports (not published_at)
D08: Already OK - activity_rows has no top-100 limit
- Add Fish import to community_importer.py
- Add 2 unit tests for D06 confidence caps
- Update test_community_importer.py for D04 name match behavior
- Add _is_trusted_proxy() to check client IP against trusted CIDRs
- Only use X-Forwarded-For if connection came from trusted proxy
- Add TRUSTED_PROXY_CIDRS config (default: 127.0.0.1/32, ::1/128)
- Add parse_comma_separated_lists for env var parsing
- Add 3 unit tests: trusted CIDR check, untrusted ignores forwarded, trusted uses forwarded
- Add 'ready = ready and healthy' for community_scheduler check
- Add 'ready = False' for community_scheduler exception path
- Add 3 unit tests: success=ready, stale=not_ready, failed=not_ready
- Monitoring now correctly reports community import health
- Add _validate_url_host() to check allowlist before urlopen()
- Validate both original URL and redirect target
- Reject localhost, internal IPs, and non-allowlisted hosts
- Add 2 unit tests for disallowed host rejection
- Prevents SSRF attacks via malicious source URLs
- AbortSignal.timeout() throws DOMException with name='TimeoutError', not TypeError
- Check for DOMException.TimeoutError, TypeError(fetch), or Error(abort)
- Apply same fix to report.ts and report-screenshot.ts
- Timeout redirects to 'timeout' state, other errors to 'create_error'
ik