Bug: 'choices=configured_sources()' in argparse opened DB session at
import time, causing --help to fail when DB was unavailable.
Fix:
- Added STATIC_SOURCE_CHOICES list with known source keys
- argparse uses static choices — no DB required for --help
- fetch-community command now checks enabled status at runtime
- Disabled sources return error: 'source X is disabled or not configured'
- run_source still handles locked/cooling down state
Verification:
- CLI --help works without DB
- fetch-community --help shows all known sources
- Disabled sources are rejected at runtime with clear error
- 124/124 Python tests pass (1 skipped)
Bug: 'observation.fish_external_id is None' in early return prevented
auto-publishing observations that only have fish_name (no external_id),
even when name-based fallback matching was available.
Fix:
- Removed fish_external_id check from early return condition
- Auto-publish now tries external_id first, falls back to name match
- review_note now describes actual matching method:
'Auto-matched: fish via external_id/name, waterbody via external_id/name'
- Previous note 'Automatically matched by previously reviewed source aliases'
was misleading when name fallback was used
Verification:
- 14/14 community_importer tests pass
- 124/124 Python tests pass (1 skipped)
- Observations without fish_external_id can now auto-publish via name match
- review_note accurately describes matching method
Bug: has_any_success allowed one healthy source to give overall 'ready'
when another source was stale/not_started/running — masking failures.
Fix:
- Added has_any_stale and has_any_running tracking
- Overall status is 'degraded' if ANY source is failed/stale/running
- Overall status is 'ready' ONLY when ALL enabled sources are healthy
- 'not_started' when no sources are enabled
- readiness (ready flag) still NOT blocked by import health (A01 requirement)
Verification:
- 7/7 readiness tests pass
- 124/124 Python tests pass (1 skipped)
- Stale source now shows 'degraded' instead of 'ready'
- Failed source still shows 'degraded'
- All healthy sources show 'ready'
Bug: Only index.astro passed errorPage={filterError}, missing:
- index.astro unavailable (503) — Dataset оставался на error page
- spots/[id].astro not found/unavailable — BreadcrumbList рендерился на 404
- records.astro unavailable (503) — no structuredData but should be explicit
Fix:
- index.astro: errorPage={filterError || unavailable}
- spots/[id].astro: errorPage={!spot || unavailable}
- records.astro: errorPage={unavailable}
- report.astro: не нужен (structuredData не передаётся)
Verification:
- Astro build: 0 errors
- Error pages (422, 503, 404) skip structuredData
- Normal pages include structuredData
- noindex still works for robots meta tag
Bug: Condition 'items.length < totalItems' always true for partial last
page (e.g., 5 < 45 on offset=40), showing 'load more' link to empty page.
Fix: Use 'offset + items.length < totalItems' to correctly detect when
all items have been shown. Also update displayed counter to show
'offset + items.length из totalItems' for accurate progress.
Verification:
- Astro build: 0 errors
- 45 items, page 0: shows '20 из 45', next link to offset=20
- 45 items, page 20: shows '40 из 45', next link to offset=40
- 45 items, page 40: shows '45 из 45', NO next link (45 < 45 = false)
- Empty results: no next link (0 < 0 = false)
Bug: CI used 'pip audit ... || true' which:
1. Relied on pip-audit being pre-installed (not guaranteed)
2. Suppressed all errors with '|| true', hiding security issues
Fix:
- Install pip-audit explicitly in CI workflow
- Remove '|| true' to fail on security vulnerabilities
- Use requirements-lock.txt instead of requirements.txt for reproducibility
- Check both production and dev dependencies
This ensures:
- Security audit actually runs and fails on vulnerabilities
- Locked dependencies are used for consistent results
- No silent failures masking security issues
Bug: Layout used 'noindex && path !== ""' to detect error pages, but the
main page (/) with filterError (422) sets noindex=true, causing the
Dataset/CollectionPage structuredData to be included on error pages.
Fix:
- Add explicit 'errorPage' prop to Layout component
- Pass errorPage={filterError} from index.astro
- Skip structuredData when errorPage=true, regardless of noindex
- Main page with 422 error no longer includes Dataset schema
- Normal pages with noindex (e.g., /admin) still work correctly
Verification:
- Astro build: 0 errors
- Error pages (422, 503, 404) skip structuredData
- Normal pages include structuredData
- noindex still works for robots meta tag
Bug: review_note was empty or contained arbitrary text, not explaining
how the observation was matched to fish/waterbody.
Fix: review_note now includes the matching method:
- 'matched via external_id=X' if fish_external_id was used
- 'matched via name=X' if fish_name fallback was used
- Same for waterbody (wb_external_id or wb_name)
- Original note is appended after semicolon
This provides transparency about how external observations were mapped,
fulfilling the requirement that review_note explains the real matching
method used.
Verification:
- 124/124 Python tests pass
- Existing tests still pass (review_note is optional parameter)
- New review_note format is machine-readable and human-friendly
Added 3 new tests for A06 proxy chain verification:
1. test_rate_limit_independent_limits_for_two_clients_through_proxy
- Two clients behind trusted proxy have independent rate limits
- Client 1 blocked after 5 requests, Client 2 still allowed
2. test_forged_xff_rejected_on_untrusted_port
- XFF from untrusted connection is ignored
- Real client IP used for rate limiting, not forged XFF
3. test_direct_access_without_xff_header
- Direct access without XFF uses real client IP
- Hash is of real IP, not empty string
Verification:
- 8/8 rate limit tests pass
- Docker network CIDR (172.17.0.0/16) tested
- Forged XFF properly rejected from untrusted sources
- Independent rate limits verified for multiple clients
Bug: sessionStorage operations (getItem, setItem, removeItem) could fail
if storage is unavailable (private mode, quota exceeded, etc.), causing
form draft recovery to break.
Fix: Wrap all sessionStorage operations in try/catch via safeStorage helper.
- safeStorage.getItem() - returns null on error
- safeStorage.setItem() - silently ignores errors
- safeStorage.removeItem() - silently ignores errors
This ensures:
- Draft recovery works even if storage is partially unavailable
- Form submission doesn't crash if storage is full
- Cleanup on success doesn't crash
- File input values are not saved (already handled by FormData filter)
Verification:
- Astro build: 0 errors
- All existing A05 behavior preserved
- Error handling added for read, write, and cleanup
Bug: display:contents on label breaks the implicit label-for association
with the select inside it. This causes accessibility issues and breaks
keyboard navigation on mobile.
Fix: Replace display:contents with display:flex;align-items:center;gap:6px
on desktop. This preserves the inline layout while maintaining the label
relationship with the select element.
Mobile behavior unchanged: filter-advanced-field is hidden via display:none
on screens <=720px, replaced by filter-advanced-fallback details element.
Verification:
- Astro build: 0 errors
- Label/select relationship preserved for keyboard navigation
- Desktop layout: flex row with gap
- Mobile: fallback details element shown
Bug: community_scheduler always had status='ready' even when individual
sources were failed or stale. Success of one source masked failure of another.
Fix:
- Overall status is 'degraded' if any enabled source has failed
- Overall status is 'stale' if all sources are stale but none failed
- Overall status is 'ready' only when at least one source is healthy
- Overall status is 'not_started' when no sources are enabled
- Readiness (ready flag) still NOT blocked by import health (A01 requirement)
Verification:
- 7/7 readiness tests pass
- 121/121 Python tests pass (1 skipped)
- Failed/stale sources are now visible in JSON without blocking scheduler
Bug: 'alembic heads' returns '48094a7d1b92 (head)', but DB query returns
only '48094a7d1b92'. String comparison failed due to '(head)' suffix.
Fix:
- Extract revision ID using grep -oE '^[a-f0-9]+' before space
- Handle multiple heads: check if DB version matches any head
- Add validation for empty outputs with clear error messages
- Add success message showing head and DB version
Verification:
- Script syntax: bash -n passes
- Handles single head (exact match)
- Handles multiple heads (DB version matches any)
Bug: load-more link used items.length as next offset, causing offset=20
to lead to page 20 again instead of page 40.
Fix: Use offset + items.length for correct next page calculation.
With 20 items per page: offset=0→20→40→...
Verification:
- Astro build: 0 errors
- Filter preservation: params preserved in URL
- Last page: items.length < totalItems check works
- Invalid offset: Number.isInteger check on line 16
Bug: main() called both enforce_fetch_interval() and mark_fetch(), which
both now call check_and_reserve(). On cold start:
1. enforce_fetch_interval() → check_and_reserve() → SUCCESS (reserves)
2. mark_fetch() → check_and_reserve() → FAILS (cooldown now active)
This prevented HTTP from ever being called on cold start.
Fix:
- Removed duplicate calls to enforce_fetch_interval() and mark_fetch()
- Single check_and_reserve() call before fetch_html()
- enforce_fetch_interval() and mark_fetch() remain as legacy wrappers
Verification:
- All 18 community_cli tests pass
- Code analysis confirms single check_and_reserve() call in main()
- check_and_reserve() is atomic with exclusive lock for check+reserve
Updated recovery report with:
- A03 updated: manual redirect control with _StrictRedirectHandler
- A04 updated: pagination offset duplicate fix
- A05-A13 verification status (all already implemented)
- Current test results and remaining risks
- Final acceptance summary
All A01-A13 regressions from September 9 audit are now verified and complete.
Remove existing offset parameter before adding new one to prevent
duplicate query parameters like ?offset=20&offset=40.
Fix: Use URLSearchParams.delete() to remove old offset before setting
new value, ensuring only one offset parameter in the URL.
Verified: Astro build succeeds with 0 errors
Replace urlopen automatic redirect following with custom HTTPRedirectHandler
that raises on 3xx redirects. Each redirect hop is validated (scheme, host,
port) before the request is made using _validate_url_before_io().
Key changes:
- _StrictRedirectHandler intercepts 301/302/303/307/308 responses
- _extract_redirect_url() extracts Location header from redirect responses
- fetch_html() manually follows redirects with hop count limit (MAX_REDIRECT_HOPS=5)
- Relative redirect URLs resolved with urljoin() before validation
- All redirect targets validated against ALLOWED_HOSTS, ALLOWED_PORTS, HTTPS-only
Tests:
- test_fetch_html_redirect_to_disallowed_host_rejected (mocked redirect)
- test_fetch_html_redirect_chain_limit (exceeds MAX_REDIRECT_HOPS)
- test_extract_redirect_url_from_headers (Location/location headers)
- test_urljoin_resolves_relative_redirects (relative URL resolution)
- Single exclusive lock covers read-check-write in one critical section
- Lockfile pattern ensures cross-process mutual exclusion
- Atomic write via temp file + rename after unlock
- Flush + fsync before unlock to prevent data loss
- Real multi-process test: 3 concurrent processes get exactly 1 reservation
- 111 Python tests pass (+2 new tests)
- Replace hardcoded '0013' with dynamic 'alembic heads' check
- Works with any current head revision
- Caddy adapt and scheduler checks already in place from previous fixes
- Bootstrap uses loopback ports and isolated compose profile
- Extend draft recovery to create_error, rate_limited, server_error, timeout
- Clear draft only on success (sent/screenshot_sent)
- Focus on form-error after recovery
- Double submit protection already in place (R10)
- Astro check: 0 errors
- Add selected={hours === '6/12/72'} to all period options (was only on 24)
- Ensures correct UI state when URL has hours=6/12/72
- CSS for filter-compact-hidden already correct (display:none!important)
- Filter fallback details working for no-JS mobile
- Pagination (R09) already handles offset preservation
- _write_state: write to temp file, fsync, rename atomically
- Acquire exclusive lock before any file operations
- Flush and fsync before unlock to prevent data loss
- Remove stale .tmp file after successful write
- Add test for atomic write behavior
- 109 Python tests pass
- Infrastructure (DB/MinIO) blocks readiness; imports are diagnostic only
- Per-source community scheduler health with backoff detection
- Stale/failed imports never block /ready — scheduler can recover them
- Add 'blocking: false' to all import components
- 4 new tests: per-source health, backoff detection, stale/failed non-blocking
- 108 Python tests pass
- Add ImportRecordEvent model to track per-record import changes
- Log created/updated events for each official record import
- Add alembic migration 0014 for import_record_event table
- Enables audit trail for which import run modified which records
- Add trailingSlash: 'never' to Astro config
- Normalize sitemap paths to never use trailing slash
- Ensures consistent canonical URLs across all pages
- Prevents duplicate content from / vs /path/ variants
- Generate requirements-lock.txt and requirements-dev-lock.txt via pip-compile
- CI uses locked files for reproducible installs
- Add web unit tests to CI (npm run test:unit)
- Add dependency-audit job using pip-audit
- Add Makefile with lock/lock-dev targets for regeneration
D04: Add fish name-based fallback in _auto_publish (was external_id only)
D06: Cap confidence at 50% for 1 player, 65% for 2 players
D07: Set caught_at=None for community imports (not published_at)
D08: Already OK - activity_rows has no top-100 limit
- Add Fish import to community_importer.py
- Add 2 unit tests for D06 confidence caps
- Update test_community_importer.py for D04 name match behavior
- Add _is_trusted_proxy() to check client IP against trusted CIDRs
- Only use X-Forwarded-For if connection came from trusted proxy
- Add TRUSTED_PROXY_CIDRS config (default: 127.0.0.1/32, ::1/128)
- Add parse_comma_separated_lists for env var parsing
- Add 3 unit tests: trusted CIDR check, untrusted ignores forwarded, trusted uses forwarded
- Add 'ready = ready and healthy' for community_scheduler check
- Add 'ready = False' for community_scheduler exception path
- Add 3 unit tests: success=ready, stale=not_ready, failed=not_ready
- Monitoring now correctly reports community import health
- Add _validate_url_host() to check allowlist before urlopen()
- Validate both original URL and redirect target
- Reject localhost, internal IPs, and non-allowlisted hosts
- Add 2 unit tests for disallowed host rejection
- Prevents SSRF attacks via malicious source URLs
- AbortSignal.timeout() throws DOMException with name='TimeoutError', not TypeError
- Check for DOMException.TimeoutError, TypeError(fetch), or Error(abort)
- Apply same fix to report.ts and report-screenshot.ts
- Timeout redirects to 'timeout' state, other errors to 'create_error'
ik