Compare commits
10
Commits
727a87b73b
..
main
| Author | SHA1 | Date | |
|---|---|---|---|
 ik
|
d438c40542 | ||
|
ik
|
722c88d436 | ||
|
ik
|
5221442aeb | ||
|
ik
|
85d81aa996 | ||
|
ik
|
53e7b0e4ae | ||
|
ik
|
d63eedf41b | ||
|
ik
|
28fffb3acb | ||
|
ik
|
4303b145b0 | ||
|
ik
|
137aa806c0 | ||
|
ik
|
e012b84969 |
@@ -0,0 +1,51 @@
|
||||
# План работы RF4 Spotter — Регрессионный аудит
|
||||
|
||||
На основе: [REGRESSION_AUDIT_2026-09-09.md](../docs/REGRESSION_AUDIT_2026-09-09.md)
|
||||
Дата: 2026-09-09
|
||||
|
||||
## Выполнено
|
||||
|
||||
### R01 ✅ — Caddy body_limit → request_body max_size
|
||||
- **Файл**: `deploy/Caddyfile`
|
||||
- **Проблема**: `body_limit 10M` не поддерживается в Caddy 2.10.2
|
||||
- **Решение**: `request_body { max_size 10M }`
|
||||
- **Верификация**: `caddy adapt` проходит без ошибок
|
||||
|
||||
### R02 ✅ — Activity API contract regression
|
||||
- **API**: `main.py` — `/api/v1/activity` возвращает `PaginatedActivityOut`
|
||||
- **Frontend**: все 4 потребителя обновлены:
|
||||
- `index.astro` ✅ (из предыдущего коммита)
|
||||
- `fish/[slug].astro` ✅
|
||||
- `waterbodies/[slug].astro` ✅
|
||||
- `waterbodies/[slug]/[fish].astro` ✅
|
||||
- `spots/[id].astro` ✅
|
||||
- **Тесты**: 4 теста обновлены под новый контракт
|
||||
- **Результат**: 76 passed, 1 skipped, 0 failures
|
||||
|
||||
### R14 ✅ — Registry источников больше не зависит от БД при парсинге CLI
|
||||
- **Файл**: `community_scheduler.py`
|
||||
- **Решение**: `_static_registry()` — без БД; `configured_sources(enabled_keys)` — с опциональной БД
|
||||
- **Тесты**: scheduler unit-тесты проходят без PostgreSQL
|
||||
|
||||
## Итоги тестов
|
||||
- **Python**: 76 passed, 1 skipped ✅
|
||||
- **Astro check**: 0 errors, 0 warnings, 0 hints ✅
|
||||
- **Caddy adapt**: passes ✅
|
||||
|
||||
## Коммиты
|
||||
1. `a37f9c4` — R01 Caddy body_limit → request_body
|
||||
2. `d962ba2` — R02 activity API contract + R14 static registry
|
||||
|
||||
## Осталось из регрессий (R03-R15)
|
||||
- R03: Research CLI cooldown broken (fcntl `r`/`r+` + encoding)
|
||||
- R04: Disabled-фильтрация обходит site cooldown
|
||||
- R05: Bootstrap небезопасен для источников
|
||||
- R06: Фильтры сигналов сравнивают slug с названием
|
||||
- R07: Ошибка главной остаётся HTTP 200
|
||||
- R08: Фильтры UI не доведены до responsive-состояния
|
||||
- R09: Пагинация (частично исправлено в U03)
|
||||
- R10: Ошибки формы теряют черновик
|
||||
- R11: Проверка URL после сетевого обращения
|
||||
- R12: Мониторинг не сигнализирует о зависшем импорте
|
||||
- R13: Доверие к IP клиента не ограничено proxy
|
||||
- R15: D04/D06/D07/D08 выполнены не полностью
|
||||
File diff suppressed because it is too large
Load Diff
Generated
+5
@@ -0,0 +1,5 @@
|
||||
# Default ignored files
|
||||
/shelf/
|
||||
/workspace.xml
|
||||
# Environment-dependent path to Maven home directory
|
||||
/mavenHomeManager.xml
|
||||
Generated
+266
@@ -0,0 +1,266 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project version="4">
|
||||
<component name="GigaCodeAgentSettings">
|
||||
<option name="autoApproveEdits" value="true" />
|
||||
<option name="autoApprovedCommands">
|
||||
<list>
|
||||
<Execute>
|
||||
<option name="pattern" value="mkdir -p /home/ik/git/rf4-help/.gigacode/plans" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="cd *" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="ls *" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="pip install *" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="tail *" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python pytest *" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python -c " import ast import sys files = [ 'apps/api/app/main.py', 'apps/api/app/readiness.py', 'apps/api/app/schemas.py', 'apps/api/app/activity.py', 'apps/api/app/community_scheduler.py', 'apps/api/app/community_importer.py', 'rf4_research/community_cli.py', ] for f in files: try: with open(f) as fh: ast.parse(fh.read()) print(f'OK: {f}') except SyntaxError as e: print(f'ERROR: {f}: {e}') sys.exit(1) print('All Python files syntax OK') "" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="npm run *" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="head *" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="npm install *" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="git add *" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="git reset *" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="git diff *" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="git log *" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="git commit *" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="git status" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="echo *" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="printf 'localhost {\n body_limit 10M\n}\n' > /tmp/test-caddyfile" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="cp /home/ik/git/rf4-help/deploy/Caddyfile /tmp/Caddyfile" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="printf 'example.com {\n body_limit 10M\n}\n' > /tmp/Caddyfile" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="printf 'example.com {\n request_body {\n max_size 10M\n }\n}\n' > /tmp/Caddyfile" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="docker run *" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="sed 's/email {$ACME_EMAIL}/email test@test.com/' /home/ik/git/rf4-help/deploy/Caddyfile > /tmp/Caddyfile" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="grep *" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="printf 'example.com {\n request_body {\n max_size 10M\n }\n handle {\n respond \"ok\"\n }\n}\n' > /tmp/Caddyfile" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python -c "import ast; ast.parse(open('rf4_research/community_cli.py').read()); print('OK')"" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python -c "import ast; ast.parse(open('apps/api/app/community_scheduler.py').read()); print('OK')"" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python -c "import ast; ast.parse(open('apps/api/app/main.py').read()); print('OK')"" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="node -e "try { fetch('http://localhost:1', { signal: AbortSignal.timeout(100) }); } catch(e) { console.log(e.constructor.name, e.message); }" 2>&1" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="node -e "fetch('http://httpbin.org/delay/10', { signal: AbortSignal.timeout(200) }).catch(e => console.log(e.constructor.name, e.message));" 2>&1" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="pip-compile requirements.txt --output-file requirements-lock.txt -q 2>&1" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="pip-compile requirements-dev.txt --output-file requirements-dev-lock.txt -q 2>&1" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="wc *" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="alembic revision *" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python -m app.cli --help 2>&1" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python -m rf4_research.community_cli --help 2>&1" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="alembic heads *" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python3 -c " from urllib.request import Request, urlopen from urllib.error import HTTPError # Test with a URL that redirects try: req = Request('https://httpbin.org/redirect/1', headers={'User-Agent': 'test'}) resp = urlopen(req, timeout=5) print(f'Final URL: {resp.url}') print(f'Response URL: {resp.url}') except Exception as e: print(f'Error: {type(e).__name__}: {e}') " 2>&1" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="pwd" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python3" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python3 -c "from rf4_research import community_cli; print('Syntax OK')"" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python3 /tmp/append_tests.py" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python3 pytest *" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python3 /tmp/fix_redirect_test.py" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python3 /tmp/fix_a04.py" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python3 /tmp/fix_a04_v2.py" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="git show *" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="git branch *" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python3 /tmp/test_double_reservation_bug.py" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python3 /tmp/fix_a02_double_reservation.py" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python3 /tmp/test_debug.py" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python3 /tmp/test_a02_final.py" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python3 /tmp/test_a02_v2.py" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python3 /tmp/test_a02_code.py" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python3 /tmp/fix_a04_pagination.py" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python3 /tmp/fix_a01_monitoring.py" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python3 /tmp/fix_a01_tests.py" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="*" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python3 /tmp/fix_a01_tests_v2.py" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python3 /tmp/fix_a04_filters.py" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python3 /tmp/fix_a05_sessionstorage.py" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python3 /tmp/fix_a06_tests.py" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python3 /tmp/fix_a07_review_note.py" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python3 /tmp/fix_a08_errorpage.py" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python3 /tmp/fix_a08_index.py" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python3 /tmp/fix_a11_pip_audit.py" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python3 /tmp/fix_a04_pagination_final.py" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python3 /tmp/fix_a08_index_unavailable.py" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python3 /tmp/fix_a08_spots.py" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python3 /tmp/fix_a08_records.py" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python3 /tmp/fix_a01_vars.py" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python3 /tmp/fix_a01_test.py" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python3 /tmp/fix_a07_auto_publish.py" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python3 /tmp/fix_a07_review_note_auto.py" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python3 /tmp/fix_a09_cli.py" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python3 -c "from app.cli import main; import sys; sys.argv = ['cli', '--help']; main()" 2>&1" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="find *" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="git push *" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python -c " from sqlalchemy import create_engine, select from sqlalchemy.orm import Session from sqlalchemy.pool import StaticPool from app.database import Base from app.models import SubmissionAttempt import hashlib, hmac engine = create_engine('sqlite://', connect_args={'check_same_thread': False}, poolclass=StaticPool) Base.metadata.create_all(engine) with Session(engine) as db: # Simulate first request key = 'idem-test-key-001' key_hash = hmac.new('change-rate-limit-secret'.encode(), key.encode(), hashlib.sha256).hexdigest() print(f'Key hash: {key_hash}') # Check before storing existing = db.scalar(select(SubmissionAttempt).where(SubmissionAttempt.idempotency_key == key_hash)) print(f'Found before: {existing}') # Store from datetime import datetime, timezone db.add(SubmissionAttempt(client_hash='test', idempotency_key=key_hash, created_at=datetime.now(timezone.utc))) db.commit() # Check after storing existing = db.scalar(select(SubmissionAttempt).where(SubmissionAttempt.idempotency_key == key_hash)) print(f'Found after: {existing}') if existing: print(f' idempotency_key: {existing.idempotency_key}') "" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python -c " from sqlalchemy import create_engine, select from sqlalchemy.orm import Session from sqlalchemy.pool import StaticPool from app.database import Base from app.models import SubmissionAttempt import hashlib, hmac engine = create_engine('sqlite://', connect_args={'check_same_thread': False}, poolclass=StaticPool) Base.metadata.create_all(engine) with Session(engine) as db: key = 'idem-test-key-001' key_hash = hmac.new('change-rate-limit-secret'.encode(), key.encode(), hashlib.sha256).hexdigest() print(f'Key hash: {key_hash}') existing = db.scalar(select(SubmissionAttempt).where(SubmissionAttempt.idempotency_key == key_hash)) print(f'Found before: {existing}') from datetime import datetime, timezone db.add(SubmissionAttempt(client_hash='test', idempotency_key=key_hash, created_at=datetime.now(timezone.utc))) db.commit() existing = db.scalar(select(SubmissionAttempt).where(SubmissionAttempt.idempotency_key == key_hash)) print(f'Found after: {existing}') if existing: print(f' idempotency_key: {existing.idempotency_key}') "" />
|
||||
</Execute>
|
||||
<Execute>
|
||||
<option name="pattern" value="python -c " from sqlalchemy import create_engine, inspect, text from sqlalchemy.pool import StaticPool from app.database import Base from app.models import SubmissionAttempt engine = create_engine('sqlite://', connect_args={'check_same_thread': False}, poolclass=StaticPool) Base.metadata.create_all(engine) inspector = inspect(engine) columns = inspector.get_columns('submission_attempt') print('Columns:', [c['name'] for c in columns]) "" />
|
||||
</Execute>
|
||||
</list>
|
||||
</option>
|
||||
<option name="autoApprovedReads">
|
||||
<list>
|
||||
<Read>
|
||||
<option name="pattern" value="/path/to/apps/web/src/pages/index.astro" />
|
||||
</Read>
|
||||
<Read>
|
||||
<option name="pattern" value="/path/to/apps/web/src/lib/api.ts" />
|
||||
</Read>
|
||||
<Read>
|
||||
<option name="pattern" value="/path/to/rf4_spotter/rf4_research/community_cli.py" />
|
||||
</Read>
|
||||
</list>
|
||||
</option>
|
||||
</component>
|
||||
</project>
|
||||
Generated
+6
@@ -0,0 +1,6 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project version="4">
|
||||
<component name="ProjectRootManager">
|
||||
<output url="file://$PROJECT_DIR$/out" />
|
||||
</component>
|
||||
</project>
|
||||
Generated
+8
@@ -0,0 +1,8 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project version="4">
|
||||
<component name="ProjectModuleManager">
|
||||
<modules>
|
||||
<module fileurl="file://$PROJECT_DIR$/.idea/rf4-help.iml" filepath="$PROJECT_DIR$/.idea/rf4-help.iml" />
|
||||
</modules>
|
||||
</component>
|
||||
</project>
|
||||
Generated
+9
@@ -0,0 +1,9 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<module type="JAVA_MODULE" version="4">
|
||||
<component name="NewModuleRootManager" inherit-compiler-output="true">
|
||||
<exclude-output />
|
||||
<content url="file://$MODULE_DIR$" />
|
||||
<orderEntry type="inheritedJdk" />
|
||||
<orderEntry type="sourceFolder" forTests="false" />
|
||||
</component>
|
||||
</module>
|
||||
Generated
+6
@@ -0,0 +1,6 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project version="4">
|
||||
<component name="VcsDirectoryMappings">
|
||||
<mapping directory="" vcs="Git" />
|
||||
</component>
|
||||
</project>
|
||||
@@ -0,0 +1,65 @@
|
||||
# Repository Guidelines
|
||||
|
||||
## Project Structure & Module Organization
|
||||
|
||||
RF4 Spotter is an Astro/FastAPI/PostgreSQL application with an offline
|
||||
research and media-ingestion toolkit. The main areas are:
|
||||
|
||||
- `apps/api/` — FastAPI application, routers, models, migrations, and API tests.
|
||||
- `apps/web/` — Astro pages, components, styles, unit tests, and Playwright tests.
|
||||
- `rf4_research/` — source parsers, media manifest tooling, and CLI commands.
|
||||
- `tests/` — Python research/tooling tests and fixtures.
|
||||
- `data/media/` — versioned manifest and content-addressed local media files.
|
||||
- `docs/` — specification, roadmap, ADRs, runbooks, and acceptance procedures.
|
||||
- `compose.yaml` — local PostgreSQL, API, web, and supporting services.
|
||||
|
||||
Keep generated reports and temporary downloads outside committed paths unless a
|
||||
task explicitly requires versioning them.
|
||||
|
||||
## Build, Test, and Development Commands
|
||||
|
||||
From the repository root:
|
||||
|
||||
```bash
|
||||
.venv/bin/pytest -q # Python suites
|
||||
npm --prefix apps/web run check # Astro type/template checks
|
||||
npm --prefix apps/web run build # Check and production build
|
||||
npm --prefix apps/web run test:unit # Web unit tests
|
||||
WEB_URL=http://127.0.0.1:4321 npm --prefix apps/web run test:e2e
|
||||
docker compose up --build # Full local stack
|
||||
```
|
||||
|
||||
For media work, use `.venv/bin/python -m rf4_research.media_cli --audit` and
|
||||
respect the manifest’s cooldown and approval states. Do not hotlink or replace
|
||||
approved media without explicit review.
|
||||
|
||||
## Coding Style & Naming Conventions
|
||||
|
||||
Use four-space indentation for Python and two spaces for Astro/TypeScript.
|
||||
Prefer typed Python functions, `snake_case` for Python identifiers, and
|
||||
`camelCase` for TypeScript variables/functions. Astro components use
|
||||
`PascalCase.astro`; tests use descriptive `test_*.py` or `*.test.ts` names.
|
||||
Keep UI text and data-source labels explicit and accessible; run `astro check`
|
||||
before committing web changes.
|
||||
|
||||
## Testing Guidelines
|
||||
|
||||
Add focused regression tests beside the affected suite. Python tests use
|
||||
pytest; web behavior uses Node unit tests and Playwright. Run the smallest
|
||||
relevant test first, then the full suite before handoff. Never use live external
|
||||
sources in tests; use fixtures or isolated Docker services.
|
||||
|
||||
## Commit & Pull Request Guidelines
|
||||
|
||||
Use imperative, concise commit subjects with the repository’s existing scope
|
||||
style, such as `feat:`, `fix:`, `data:`, or `chore:`. Keep commits focused and
|
||||
exclude unrelated user files. Pull requests should describe behavior changes,
|
||||
verification commands, migration or configuration impact, and screenshots for
|
||||
visual/UI work. Call out any media provenance, approval, or rollback decision.
|
||||
|
||||
## Security & Configuration Tips
|
||||
|
||||
Do not commit secrets, production `.env` files, tokens, or private payloads.
|
||||
Use local fixtures and documented environment variables. Preserve the strict
|
||||
CSP, admin authentication barriers, source attribution, and media provenance
|
||||
when changing application code.
|
||||
@@ -6,7 +6,7 @@ RF4 Spotter — неофициальный сервис свежих точек
|
||||
|
||||
## Статус разработки
|
||||
|
||||
**Проверка 13 сентября 2026 (`787a506`): локальный контур готов к развёртыванию открытой альфы, внешний запуск ждёт сервер и его настройки.** Пакет восстановления A01–A13 закрыт. Python: **157 passed, 1 skipped**; Astro check/build, web unit и API-тесты проходят. Граф миграций имеет единственную голову `0016`; последний полный production bootstrap подтвердил Caddy, scheduler и браузерный сценарий отправки/модерации на предыдущей голове, а актуальная голова проверяется CI на чистой PostgreSQL. Реальные источники во время приёмки не опрашивались.
|
||||
**Проверка 15 сентября 2026: локальный контур готов к развёртыванию открытой альфы, внешний запуск ждёт сервер и его настройки.** Пакет восстановления A01–A13 закрыт. Python: **178 passed, 1 skipped**; Astro check/build, web unit и API-тесты проходят. Граф миграций имеет единственную голову `0018`; production bootstrap и импорт реальных источников по-прежнему требуют отдельной инфраструктурной/разрешённой приёмки.
|
||||
|
||||
Актуальные следующие задачи находятся только в [ROADMAP](docs/ROADMAP.md). Старые планы и аудиты сохранены как история и больше не задают порядок работ. До внешнего запуска нужны сервер, DNS/TLS, production-секреты, публичные контакты, внешний backup и канал уведомлений.
|
||||
|
||||
@@ -46,9 +46,11 @@ RF4DB/RF4-STAT/RF4MAP/RF4 Posts сначала принимаются в изо
|
||||
|
||||
Медиасборщик индексирует разрешённые изображения отдельно от публичного каталога: manifest хранит исходную страницу, URL, предполагаемый тип сущности и время обнаружения, а оригиналы сохраняются по SHA-256 без hotlink. После явного разрешения владельца от 14 сентября все скачанные и целостные материалы опубликованы в `/media`; публичный API отдаёт Git-копии по content-addressed URL, а каждая карточка показывает плашку и прямую ссылку на источник. Будущие загрузки по-прежнему не одобряются автоматически. Локальный `media_cli --audit` без сетевых запросов проверяет хэши, файлы, MIME, размеры, approved-сопоставления и отсутствие бесхозных оригиналов.
|
||||
|
||||
Актуальный offline-срез от 16.09.2026: 704 записи manifest, 466 `approved`, 226 `superseded`, 1 `duplicate` и 11 `invalid`; audit проходит без ошибок и orphan-файлов. У 252 из 253 рыбных изображений есть 1024×1024 WebP, один 48×48 fallback сохранён из-за отсутствия проверенной альтернативы. Производные WebP/AVIF и provenance хранятся в Git; визуальная browser-приёмка остаётся отдельным пунктом B25.
|
||||
|
||||
`python -m rf4_research.media_cli --coverage` сравнивает manifest с датированным `data/media/catalog-baseline.json`: отдельно считает уникальные нормализованные подписи и кандидатов без подписи, поэтому альтернативные URL не завышают покрытие. На 14.09.2026 manifest содержит 704 записи: 456 approved, 227 duplicate, 10 queued и 11 invalid. Опубликованы 243 уникально подписанные рыбы из 252, все 149 найденных изображений снастей/приманок и 64 справочных материала; 10 оставшихся RF4DB URL будут загружены после cooldown. Карты 19 водоёмов ещё предстоит проиндексировать, а общий target снастей остаётся `null` до проверяемого полного счётчика.
|
||||
|
||||
`python -m rf4_research.media_cli --quality-report` выполняет offline-проверку разрешения опубликованных рыб. Текущий отчёт выявляет 227 PNG RF4MAP размером 48×48, которые увеличиваются в карточках до 180 px, и 16 WebP 1024×1024; для всех низких версий уже известны альтернативные RF4DB URL. План загрузки, сравнения, безопасного переключения и генерации производных размеров ведётся в B20–B25 ROADMAP.
|
||||
`python -m rf4_research.media_cli --quality-report` выполняет offline-проверку разрешения опубликованных рыб. Текущий отчёт выявляет 227 PNG RF4MAP размером 48×48, которые увеличиваются в карточках до 180 px, и 16 WebP 1024×1024; для всех низких версий уже известны альтернативные RF4DB URL. `--queue-quality-upgrades` переводит только прямые альтернативы низкоразрешённых published-файлов в безопасную очередь, не снимая текущую версию с публикации. На 15.09 первая партия из 40 замен уже сохранена в карантине для сравнения и review; дальнейшие этапы ведутся в B21–B25 ROADMAP.
|
||||
|
||||
`python -m rf4_research.media_cli --queue-plan` без сетевых запросов объединяет manifest с общим cooldown-state: показывает queued-состав каждого домена, оставшееся время и наиболее полезный следующий asset с приоритетом водоёмов и рыб. Разрешённое media-окно загружается командой `--download-batch --batch-limit 40`: до 40 assets на домен под одной резервацией, затем 30 минут до нового batch. Блокировка/rate limit/сетевая ошибка останавливает домен сразу, три последовательных невалидных ответа — досрочно. Все файлы остаются в карантине до ручного review. Точный поимённый список отсутствующих сущностей появится только после получения канонического перечня; разница между двумя несогласованными каталогами не выдаётся за доказанный gap.
|
||||
|
||||
@@ -146,7 +148,7 @@ docker compose up --build
|
||||
- readiness PostgreSQL, MinIO и импорта с версией/revision сборки: <http://localhost:8000/ready>;
|
||||
- консоль MinIO: <http://localhost:9001>.
|
||||
|
||||
Контейнер API сам выполняет `alembic upgrade head`, затем идемпотентный seed. PostgreSQL хранит данные в именованном volume `postgres_data`, а MinIO — в `minio_data`. Compose ожидает readiness PostgreSQL и MinIO перед API, а API-контейнер проверяет `/ready`. Версия и commit SHA задаются через `APP_VERSION`/`APP_REVISION`; те же значения доступны администратору в `/api/v1/admin/diagnostics`. Официальный импорт по умолчанию необязателен; при включённом scheduler установите `OFFICIAL_IMPORT_REQUIRED=true`, тогда отсутствующий, неуспешный или просроченный запуск сделает readiness отрицательным.
|
||||
Одноразовые контейнеры `migrate` и `minio-init` перед запуском API соответственно применяют `alembic upgrade head` и идемпотентно создают локальный `S3_BUCKET`; приложение само не создаёт схему или bucket. PostgreSQL хранит данные в именованном volume `postgres_data`, а MinIO — в `minio_data`. Compose ожидает readiness PostgreSQL, MinIO и успешное завершение обоих init-контейнеров перед API, а API-контейнер проверяет `/ready`. Версия и commit SHA задаются через `APP_VERSION`/`APP_REVISION`; те же значения доступны администратору в `/api/v1/admin/diagnostics`. Здоровье импортов диагностическое и не мешает API или scheduler восстановиться после сбоя.
|
||||
|
||||
API и scheduler пишут по одной JSON-записи на событие. HTTP-лог содержит только сгенерированный `request_id`, метод, путь без query string, статус и длительность; IP, заголовок авторизации и пользовательский payload не журналируются. `X-Request-ID` возвращается клиенту. Стандартный access-log Uvicorn отключён. Уровень управляется `LOG_LEVEL`. Публичный агрегат активности кэшируется в памяти процесса на 20 секунд (до 128 ключей) и очищается после публикации, модерации или удаления через этот процесс API; изменения scheduler видны после TTL; `X-Cache` показывает `HIT`/`MISS`. Защищённый `/api/v1/admin/diagnostics` скачивает JSON только с идентификатором сборки и агрегированными счётчиками, без имён игроков, исходных URL, payload и ошибок парсеров.
|
||||
|
||||
@@ -182,7 +184,7 @@ docker compose up --build
|
||||
## Что реализовано
|
||||
|
||||
- FastAPI и SQLAlchemy 2;
|
||||
- PostgreSQL 17 и линейные миграции Alembic до `0016`;
|
||||
- PostgreSQL 17 и линейные миграции Alembic до `0018`;
|
||||
- идемпотентный seed с двумя точками и свежими демо-уловами;
|
||||
- `GET /api/v1/activity` с фильтрами периода, водоёма, рыбы, способа и сортировки;
|
||||
- `GET /api/v1/spots/{id}` и `/catches`;
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
"""add canonical waterbody provenance fields
|
||||
|
||||
Revision ID: 0017
|
||||
Revises: 0016
|
||||
"""
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision = "0017"
|
||||
down_revision = "0016"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column("waterbody", sa.Column("source_system", sa.String(length=50), nullable=True))
|
||||
op.add_column("waterbody", sa.Column("source_external_id", sa.String(length=200), nullable=True))
|
||||
op.add_column("waterbody", sa.Column("source_url", sa.Text(), nullable=True))
|
||||
op.add_column("waterbody", sa.Column("description", sa.Text(), nullable=True))
|
||||
op.add_column("waterbody", sa.Column("source_checked_at", sa.DateTime(timezone=True), nullable=True))
|
||||
op.create_index(
|
||||
"uq_waterbody_source_identity",
|
||||
"waterbody",
|
||||
["source_system", "source_external_id"],
|
||||
unique=True,
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index("uq_waterbody_source_identity", table_name="waterbody")
|
||||
op.drop_column("waterbody", "source_checked_at")
|
||||
op.drop_column("waterbody", "description")
|
||||
op.drop_column("waterbody", "source_url")
|
||||
op.drop_column("waterbody", "source_external_id")
|
||||
op.drop_column("waterbody", "source_system")
|
||||
@@ -0,0 +1,26 @@
|
||||
"""store source coordinate text and precision
|
||||
|
||||
Revision ID: 0018
|
||||
Revises: 0017
|
||||
"""
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision = "0018"
|
||||
down_revision = "0017"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column("external_observation", sa.Column("coordinate_raw", sa.String(length=200), nullable=True))
|
||||
op.add_column("external_observation", sa.Column("coordinate_precision", sa.String(length=20), nullable=True))
|
||||
op.execute("UPDATE external_observation SET coordinate_precision = CASE WHEN x IS NOT NULL AND y IS NOT NULL THEN 'exact' ELSE 'missing' END")
|
||||
op.alter_column("external_observation", "coordinate_precision", nullable=False, server_default="missing")
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_column("external_observation", "coordinate_precision")
|
||||
op.drop_column("external_observation", "coordinate_raw")
|
||||
@@ -0,0 +1,22 @@
|
||||
"""store canonical waterbody fish count
|
||||
|
||||
Revision ID: 0019
|
||||
Revises: 0018
|
||||
"""
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision = "0019"
|
||||
down_revision = "0018"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column("waterbody", sa.Column("fish_species_count", sa.Integer(), nullable=True))
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_column("waterbody", "fish_species_count")
|
||||
@@ -0,0 +1,24 @@
|
||||
"""store canonical waterbody detail facts
|
||||
|
||||
Revision ID: 0020
|
||||
Revises: 0019
|
||||
"""
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision = "0020"
|
||||
down_revision = "0019"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
for name in ("source_aliases", "source_fish_species", "source_image_urls", "source_point_urls"):
|
||||
op.add_column("waterbody", sa.Column(name, sa.JSON(), nullable=True))
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
for name in ("source_point_urls", "source_image_urls", "source_fish_species", "source_aliases"):
|
||||
op.drop_column("waterbody", name)
|
||||
@@ -61,6 +61,9 @@ def activity_rows(
|
||||
confidence = min(confidence, 50)
|
||||
elif len(players) == 2:
|
||||
confidence = min(confidence, 65)
|
||||
coordinate_precisions = {_coordinate_precision(item) for item in items}
|
||||
coordinate_precision = max(coordinate_precisions, key=_precision_rank)
|
||||
coordinate_sources = sorted({_source_system(item) for item in items})
|
||||
latest = max(_aware(r.reported_at) for r in items)
|
||||
baits = Counter(r.bait.name for r in items if r.bait)
|
||||
freshness_text = _freshness_text(now - latest)
|
||||
@@ -74,7 +77,9 @@ def activity_rows(
|
||||
max_weight_g=max(r.weight_g for r in items), last_confirmed_at=latest,
|
||||
activity_score=activity, confidence_score=confidence,
|
||||
explanation=_explanation(len(items), len(players), freshness_text, activity, confidence),
|
||||
sources=sorted({_source_system(item) for item in items}),
|
||||
sources=coordinate_sources,
|
||||
coordinate_precision=coordinate_precision,
|
||||
coordinate_sources=coordinate_sources,
|
||||
))
|
||||
return sorted(result, key=lambda row: (row.activity_score, row.last_confirmed_at), reverse=True)
|
||||
|
||||
@@ -90,6 +95,16 @@ def _source_system(report: CatchReport) -> str:
|
||||
return "manual-import"
|
||||
|
||||
|
||||
def _coordinate_precision(report: CatchReport) -> str:
|
||||
provenance = (report.raw_payload or {}).get("provenance", {})
|
||||
value = provenance.get("coordinate_precision") if isinstance(provenance, dict) else None
|
||||
return value if value in {"exact", "approximate", "area", "missing"} else "exact"
|
||||
|
||||
|
||||
def _precision_rank(value: str) -> int:
|
||||
return {"exact": 0, "approximate": 1, "area": 2, "missing": 3}[value]
|
||||
|
||||
|
||||
def _aware(value: datetime) -> datetime:
|
||||
return value if value.tzinfo else value.replace(tzinfo=timezone.utc)
|
||||
|
||||
|
||||
@@ -15,3 +15,28 @@ def audit_catalog(db: Session) -> dict[str, int]:
|
||||
"incomplete_published_staging": db.scalar(select(func.count()).select_from(ExternalObservation).where(ExternalObservation.status == "published", or_(ExternalObservation.fish_id.is_(None), ExternalObservation.waterbody_id.is_(None), ExternalObservation.x.is_(None), ExternalObservation.y.is_(None), ExternalObservation.weight_g.is_(None), ExternalObservation.catch_report_id.is_(None)))) or 0,
|
||||
}
|
||||
return {"fishes": count(Fish), "waterbodies": count(Waterbody), "reports": count(CatchReport), "staging": count(ExternalObservation), **failures, "failures": sum(failures.values())}
|
||||
|
||||
|
||||
def audit_waterbody_catalog(db: Session, expected_ids: set[str]) -> dict:
|
||||
"""Check a verified RF4DB snapshot without withdrawing legacy rows."""
|
||||
rows = list(db.scalars(select(Waterbody).where(Waterbody.source_system == "rf4db")))
|
||||
observed_ids = [str(row.source_external_id) for row in rows if row.source_external_id]
|
||||
observed = set(observed_ids)
|
||||
duplicate_ids = sorted({item for item in observed_ids if observed_ids.count(item) > 1})
|
||||
missing = sorted(expected_ids - observed)
|
||||
unexpected = sorted(observed - expected_ids)
|
||||
provenance_issues = sorted(
|
||||
str(row.source_external_id)
|
||||
for row in rows
|
||||
if not row.source_external_id or not row.source_url or not row.source_checked_at
|
||||
)
|
||||
failures = len(missing) + len(duplicate_ids) + len(provenance_issues)
|
||||
return {
|
||||
"expected": len(expected_ids),
|
||||
"observed": len(observed),
|
||||
"missing_source_external_ids": missing,
|
||||
"unexpected_source_external_ids": unexpected,
|
||||
"duplicate_source_external_ids": duplicate_ids,
|
||||
"provenance_issues": provenance_issues,
|
||||
"failures": failures,
|
||||
}
|
||||
|
||||
+66
-2
@@ -8,10 +8,10 @@ from dataclasses import asdict
|
||||
from .config import settings
|
||||
from .database import SessionLocal
|
||||
from .importer import import_records
|
||||
from .community_importer import stage_observations
|
||||
from .community_importer import stage_observations, update_waterbody_detail, update_waterbody_details, upsert_waterbody_catalog
|
||||
from .retention import RetentionPolicy, apply_retention
|
||||
from .storage import delete_screenshot
|
||||
from .catalog_audit import audit_catalog
|
||||
from .catalog_audit import audit_catalog, audit_waterbody_catalog
|
||||
from .community_scheduler import run_source, configured_sources
|
||||
|
||||
# Static registry for argparse choices — no DB required for --help
|
||||
@@ -34,11 +34,20 @@ def main() -> int:
|
||||
community = sub.add_parser("stage-community-json")
|
||||
community.add_argument("--input", default="-", help="JSON array path or - for stdin")
|
||||
community.add_argument("--limit", type=int, default=500)
|
||||
waterbodies = sub.add_parser("import-waterbody-catalog")
|
||||
waterbodies.add_argument("--input", required=True, help="JSON snapshot path or - for stdin")
|
||||
waterbodies.add_argument("--limit", type=int, default=100)
|
||||
detail = sub.add_parser("import-waterbody-detail")
|
||||
detail.add_argument("--input", required=True, help="JSON detail snapshot path")
|
||||
details = sub.add_parser("import-waterbody-details")
|
||||
details.add_argument("--input", required=True, help="JSON array of detail snapshots")
|
||||
fetch_community = sub.add_parser("fetch-community")
|
||||
fetch_community.add_argument("source", choices=STATIC_SOURCE_CHOICES)
|
||||
cleanup = sub.add_parser("cleanup-retention")
|
||||
cleanup.add_argument("--apply", action="store_true", help="apply changes; default is dry-run")
|
||||
sub.add_parser("audit-catalog")
|
||||
waterbody_audit = sub.add_parser("audit-waterbody-catalog")
|
||||
waterbody_audit.add_argument("--input", required=True, help="JSON snapshot path")
|
||||
args = parser.parse_args()
|
||||
with SessionLocal() as session:
|
||||
if args.command == "import-records":
|
||||
@@ -57,6 +66,43 @@ def main() -> int:
|
||||
parser.error("input must be a JSON array")
|
||||
created, updated = stage_observations(session, payload[:args.limit])
|
||||
print(f"staged: created={created} updated={updated}")
|
||||
elif args.command == "import-waterbody-catalog":
|
||||
if not 1 <= args.limit <= 500:
|
||||
parser.error("--limit must be between 1 and 500")
|
||||
stream = sys.stdin if args.input == "-" else open(args.input, encoding="utf-8")
|
||||
try:
|
||||
snapshot = json.load(stream)
|
||||
finally:
|
||||
if stream is not sys.stdin:
|
||||
stream.close()
|
||||
if isinstance(snapshot, dict):
|
||||
payload = snapshot.get("items")
|
||||
source_system = snapshot.get("source_system")
|
||||
if isinstance(payload, list) and isinstance(source_system, str):
|
||||
payload = [
|
||||
{"source_system": source_system, **item}
|
||||
for item in payload if isinstance(item, dict)
|
||||
]
|
||||
else:
|
||||
payload = snapshot
|
||||
if not isinstance(payload, list):
|
||||
parser.error("input must be a JSON array or an object with an items array")
|
||||
created, updated = upsert_waterbody_catalog(session, payload[:args.limit])
|
||||
print(f"waterbodies: created={created} updated={updated}")
|
||||
elif args.command == "import-waterbody-detail":
|
||||
with open(args.input, encoding="utf-8") as stream:
|
||||
payload = json.load(stream)
|
||||
if not isinstance(payload, dict):
|
||||
parser.error("input must be a JSON object")
|
||||
update_waterbody_detail(session, payload)
|
||||
print(f"waterbody detail: updated={payload.get('source_external_id', 'unknown')}")
|
||||
elif args.command == "import-waterbody-details":
|
||||
with open(args.input, encoding="utf-8") as stream:
|
||||
payload = json.load(stream)
|
||||
if not isinstance(payload, list):
|
||||
parser.error("input must be a JSON array")
|
||||
created, updated = update_waterbody_details(session, payload)
|
||||
print(f"waterbody details: created={created} updated={updated}")
|
||||
elif args.command == "fetch-community":
|
||||
# A09: Verify source is enabled at runtime (not just in static choices)
|
||||
enabled = configured_sources()
|
||||
@@ -76,6 +122,24 @@ def main() -> int:
|
||||
)
|
||||
counts = apply_retention(session, policy=policy, dry_run=not args.apply, delete_object=delete_screenshot)
|
||||
print(json.dumps({"mode": "apply" if args.apply else "dry-run", "policy": asdict(policy), "counts": counts}, ensure_ascii=False))
|
||||
elif args.command == "audit-waterbody-catalog":
|
||||
stream = sys.stdin if args.input == "-" else open(args.input, encoding="utf-8")
|
||||
try:
|
||||
snapshot = json.load(stream)
|
||||
finally:
|
||||
if stream is not sys.stdin:
|
||||
stream.close()
|
||||
items = snapshot.get("items") if isinstance(snapshot, dict) else snapshot
|
||||
if not isinstance(items, list):
|
||||
parser.error("input must be a JSON array or an object with an items array")
|
||||
expected_ids = {
|
||||
str(item["source_external_id"])
|
||||
for item in items
|
||||
if isinstance(item, dict) and item.get("source_external_id")
|
||||
}
|
||||
result = audit_waterbody_catalog(session, expected_ids)
|
||||
print(json.dumps(result, ensure_ascii=False))
|
||||
return 1 if result["failures"] else 0
|
||||
else:
|
||||
result = audit_catalog(session)
|
||||
print(json.dumps(result, ensure_ascii=False))
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import hashlib
|
||||
import re
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any, Iterable
|
||||
from urllib.parse import urlparse
|
||||
@@ -26,12 +28,153 @@ SOURCE_HOSTS = {
|
||||
"rf4map": {"rf4map.ru"},
|
||||
"rf4posts-spot": {"rf4-posts.com"},
|
||||
}
|
||||
COORDINATE_PRECISIONS = frozenset({"exact", "approximate", "area", "missing"})
|
||||
|
||||
|
||||
class CommunityImportError(ValueError):
|
||||
pass
|
||||
|
||||
|
||||
def upsert_waterbody_catalog(
|
||||
session: Session, rows: Iterable[dict[str, Any]], *, fetched_at: datetime | None = None,
|
||||
) -> tuple[int, int]:
|
||||
"""Apply a validated canonical waterbody snapshot without destructive sync.
|
||||
|
||||
Rows are matched by the RF4DB source identity first and by an exact existing
|
||||
name second. Missing rows are deliberately left untouched: an incomplete
|
||||
response must never withdraw a previously known waterbody.
|
||||
"""
|
||||
fetched_at = fetched_at or datetime.now(timezone.utc)
|
||||
created = updated = 0
|
||||
for raw in rows:
|
||||
payload = _json_payload(raw)
|
||||
if payload.get("source_system") != "rf4db":
|
||||
raise CommunityImportError("waterbody catalog requires source_system=rf4db")
|
||||
external_id = _required(payload, "source_external_id", 200)
|
||||
name = _required(payload, "name", 200)
|
||||
source_url = _required(payload, "source_url", 2000)
|
||||
parsed_url = urlparse(source_url)
|
||||
if parsed_url.scheme != "https" or parsed_url.hostname not in {"rf4db.com", "www.rf4db.com"}:
|
||||
raise CommunityImportError("waterbody source_url does not match rf4db")
|
||||
unlock_level = _integer(payload.get("unlock_level"), minimum=0, maximum=1_000)
|
||||
unlock_label = _required(payload, "unlock_label", 50)
|
||||
fish_species_count = _integer(payload.get("fish_species_count"), minimum=0, maximum=10_000)
|
||||
if fish_species_count is None:
|
||||
raise CommunityImportError("invalid fish_species_count")
|
||||
|
||||
item = session.scalar(select(Waterbody).where(
|
||||
Waterbody.source_system == "rf4db",
|
||||
Waterbody.source_external_id == external_id,
|
||||
))
|
||||
if item is None:
|
||||
item = session.scalar(select(Waterbody).where(Waterbody.name_ru == name))
|
||||
if item is None:
|
||||
item = Waterbody(
|
||||
slug=_catalog_slug(session, name, external_id),
|
||||
name_ru=name,
|
||||
unlock_level=unlock_level,
|
||||
)
|
||||
session.add(item)
|
||||
created += 1
|
||||
else:
|
||||
updated += 1
|
||||
item.name_ru = name
|
||||
item.unlock_level = unlock_level
|
||||
item.fish_species_count = fish_species_count
|
||||
item.source_system = "rf4db"
|
||||
item.source_external_id = external_id
|
||||
item.source_url = source_url
|
||||
item.source_checked_at = fetched_at
|
||||
session.commit()
|
||||
return created, updated
|
||||
|
||||
|
||||
def update_waterbody_detail(
|
||||
session: Session, detail: dict[str, Any], *, fetched_at: datetime | None = None,
|
||||
) -> bool:
|
||||
"""Persist one complete RF4DB detail snapshot without assigning media roles."""
|
||||
fetched_at = fetched_at or datetime.now(timezone.utc)
|
||||
payload = _validate_waterbody_detail(detail)
|
||||
_apply_waterbody_detail(session, payload, fetched_at=fetched_at)
|
||||
session.commit()
|
||||
return True
|
||||
|
||||
|
||||
def update_waterbody_details(
|
||||
session: Session, details: Iterable[dict[str, Any]], *, fetched_at: datetime | None = None,
|
||||
) -> tuple[int, int]:
|
||||
"""Validate and apply a detail batch in one transaction."""
|
||||
fetched_at = fetched_at or datetime.now(timezone.utc)
|
||||
payloads = [_validate_waterbody_detail(detail) for detail in details]
|
||||
external_ids = [str(payload["source_external_id"]) for payload in payloads]
|
||||
if len(external_ids) != len(set(external_ids)):
|
||||
raise CommunityImportError("waterbody detail batch contains duplicate source identities")
|
||||
updated = 0
|
||||
for payload in payloads:
|
||||
_apply_waterbody_detail(session, payload, fetched_at=fetched_at)
|
||||
updated += 1
|
||||
session.commit()
|
||||
return 0, updated
|
||||
|
||||
|
||||
def _validate_waterbody_detail(detail: dict[str, Any]) -> dict[str, Any]:
|
||||
payload = _json_payload(detail)
|
||||
if payload.get("source_system") != "rf4db":
|
||||
raise CommunityImportError("waterbody detail requires source_system=rf4db")
|
||||
external_id = _required(payload, "source_external_id", 200)
|
||||
source_url = _required(payload, "source_url", 2000)
|
||||
parsed_url = urlparse(source_url)
|
||||
if parsed_url.scheme != "https" or parsed_url.hostname not in {"rf4db.com", "www.rf4db.com"}:
|
||||
raise CommunityImportError("waterbody detail source_url does not match rf4db")
|
||||
_required(payload, "name", 200)
|
||||
_optional(payload, "description", 20_000)
|
||||
_string_list(payload, "aliases", 100, 200)
|
||||
_string_list(payload, "fish_species", 10_000, 200)
|
||||
_string_list(payload, "image_urls", 100, 2_000)
|
||||
_string_list(payload, "point_urls", 10_000, 2_000)
|
||||
return payload
|
||||
|
||||
|
||||
def _apply_waterbody_detail(session: Session, payload: dict[str, Any], *, fetched_at: datetime) -> None:
|
||||
external_id = str(payload["source_external_id"])
|
||||
source_url = str(payload["source_url"])
|
||||
item = session.scalar(select(Waterbody).where(
|
||||
Waterbody.source_system == "rf4db", Waterbody.source_external_id == external_id,
|
||||
))
|
||||
if item is None:
|
||||
raise CommunityImportError("waterbody detail has no imported catalog identity")
|
||||
item.description = _optional(payload, "description", 20_000)
|
||||
item.source_aliases = _string_list(payload, "aliases", 100, 200)
|
||||
item.source_fish_species = _string_list(payload, "fish_species", 10_000, 200)
|
||||
item.source_image_urls = _string_list(payload, "image_urls", 100, 2_000)
|
||||
item.source_point_urls = _string_list(payload, "point_urls", 10_000, 2_000)
|
||||
item.source_url = source_url
|
||||
item.source_checked_at = fetched_at
|
||||
|
||||
|
||||
def _catalog_slug(session: Session, name: str, external_id: str) -> str:
|
||||
base = re.sub(r"[^a-z0-9а-яё]+", "-", name.casefold(), flags=re.IGNORECASE).strip("-")
|
||||
base = base or "waterbody"
|
||||
candidate = base[:100]
|
||||
if session.scalar(select(Waterbody.id).where(Waterbody.slug == candidate)) is None:
|
||||
return candidate
|
||||
suffix = hashlib.sha256(external_id.encode()).hexdigest()[:10]
|
||||
return f"{base[:89]}-{suffix}"
|
||||
|
||||
|
||||
def _string_list(payload: dict[str, Any], key: str, max_items: int, max_length: int) -> list[str]:
|
||||
value = payload.get(key)
|
||||
if not isinstance(value, list) or len(value) > max_items:
|
||||
raise CommunityImportError(f"invalid {key}")
|
||||
result = []
|
||||
for item in value:
|
||||
text = str(item).strip()
|
||||
if not text or len(text) > max_length:
|
||||
raise CommunityImportError(f"invalid {key}")
|
||||
result.append(text)
|
||||
return list(dict.fromkeys(result))
|
||||
|
||||
|
||||
def stage_observations(
|
||||
session: Session, records: Iterable[dict[str, Any]], *, fetched_at: datetime | None = None,
|
||||
) -> tuple[int, int]:
|
||||
@@ -65,6 +208,8 @@ def stage_observations(
|
||||
"waterbody_external_id": _optional(payload, "waterbody_external_id", 200),
|
||||
"x": _integer(payload.get("x"), maximum=10_000),
|
||||
"y": _integer(payload.get("y"), maximum=10_000),
|
||||
"coordinate_raw": _coordinate_raw(payload),
|
||||
"coordinate_precision": _coordinate_precision(payload),
|
||||
"weight_g": _integer(payload.get("weight_g"), minimum=1, maximum=3_000_000),
|
||||
"published_at": _datetime(payload.get("published_at")),
|
||||
"last_seen_at": fetched_at, "payload": payload,
|
||||
@@ -83,6 +228,7 @@ def stage_observations(
|
||||
changed = any(getattr(observation, key) != values[key] for key in (
|
||||
"source_url", "fish_name", "fish_external_id", "waterbody_name",
|
||||
"waterbody_external_id", "x", "y", "weight_g",
|
||||
"coordinate_raw", "coordinate_precision",
|
||||
)) or observation.payload != payload
|
||||
if observation.status != "rejected" and changed and observation.catch_report is not None:
|
||||
observation.catch_report.moderation_status = ModerationStatus.pending
|
||||
@@ -186,6 +332,27 @@ def _optional(payload: dict[str, Any], key: str, limit: int) -> str | None:
|
||||
return value or None
|
||||
|
||||
|
||||
def _coordinate_raw(payload: dict[str, Any]) -> str | None:
|
||||
value = str(payload.get("coordinate_raw") or "").strip()
|
||||
if len(value) > 200:
|
||||
raise CommunityImportError("invalid coordinate_raw")
|
||||
if value:
|
||||
return value
|
||||
x, y = payload.get("x"), payload.get("y")
|
||||
return f"{x}:{y}" if isinstance(x, int) and isinstance(y, int) else None
|
||||
|
||||
|
||||
def _coordinate_precision(payload: dict[str, Any]) -> str:
|
||||
value = str(payload.get("coordinate_precision") or "").strip().casefold()
|
||||
if not value:
|
||||
return "exact" if isinstance(payload.get("x"), int) and isinstance(payload.get("y"), int) else "missing"
|
||||
if value not in COORDINATE_PRECISIONS:
|
||||
raise CommunityImportError("invalid coordinate_precision")
|
||||
if value == "exact" and (not isinstance(payload.get("x"), int) or not isinstance(payload.get("y"), int)):
|
||||
raise CommunityImportError("exact coordinates require x and y")
|
||||
return value
|
||||
|
||||
|
||||
def _integer(value: Any, *, minimum: int = -10_000, maximum: int) -> int | None:
|
||||
if value is None:
|
||||
return None
|
||||
|
||||
@@ -104,6 +104,8 @@ def publish_observation(session: Session, observation: ExternalObservation) -> C
|
||||
"external_observation_id": str(observation.id),
|
||||
"source_system": observation.source_system,
|
||||
"source_external_id": observation.source_external_id,
|
||||
"coordinate_raw": observation.coordinate_raw,
|
||||
"coordinate_precision": observation.coordinate_precision,
|
||||
},
|
||||
"original": observation.payload,
|
||||
},
|
||||
|
||||
@@ -81,9 +81,11 @@ def review_assets(entity_type: str | None = None, status: str | None = None) ->
|
||||
"height": item.get("height"),
|
||||
"content_type": item.get("content_type"),
|
||||
"image_url": f"/api/v1/admin/media/assets/{digest}",
|
||||
"asset_url": item.get("asset_url", ""),
|
||||
"source_system": source,
|
||||
"source_url": source_page,
|
||||
"duplicate_of": item.get("duplicate_of"),
|
||||
"supersedes": item.get("supersedes"),
|
||||
"derivatives": [{
|
||||
"role": variant.get("role"), "format": variant.get("format"),
|
||||
"width": variant.get("width"), "height": variant.get("height"),
|
||||
|
||||
@@ -49,6 +49,16 @@ class Waterbody(Base):
|
||||
slug: Mapped[str] = mapped_column(String(100), unique=True)
|
||||
name_ru: Mapped[str] = mapped_column(String(200), unique=True)
|
||||
unlock_level: Mapped[int | None]
|
||||
fish_species_count: Mapped[int | None]
|
||||
source_system: Mapped[str | None] = mapped_column(String(50))
|
||||
source_external_id: Mapped[str | None] = mapped_column(String(200))
|
||||
source_url: Mapped[str | None] = mapped_column(Text)
|
||||
description: Mapped[str | None] = mapped_column(Text)
|
||||
source_aliases: Mapped[list[str] | None] = mapped_column(JSON)
|
||||
source_fish_species: Mapped[list[str] | None] = mapped_column(JSON)
|
||||
source_image_urls: Mapped[list[str] | None] = mapped_column(JSON)
|
||||
source_point_urls: Mapped[list[str] | None] = mapped_column(JSON)
|
||||
source_checked_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
|
||||
|
||||
class Bait(Base):
|
||||
@@ -187,6 +197,8 @@ class ExternalObservation(Base):
|
||||
waterbody_external_id: Mapped[str | None] = mapped_column(String(200))
|
||||
x: Mapped[int | None]
|
||||
y: Mapped[int | None]
|
||||
coordinate_raw: Mapped[str | None] = mapped_column(String(200))
|
||||
coordinate_precision: Mapped[str] = mapped_column(String(20), default="missing")
|
||||
weight_g: Mapped[int | None]
|
||||
published_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
first_seen_at: Mapped[datetime] = mapped_column(DateTime(timezone=True))
|
||||
|
||||
@@ -68,7 +68,15 @@ def spot_detail(spot_id: UUID, db: Db) -> SpotOut:
|
||||
def count_since(delta: timedelta) -> int:
|
||||
return sum(aware(report.reported_at) >= now - delta for report in reports)
|
||||
|
||||
return SpotOut(id=spot.id, waterbody_slug=spot.waterbody.slug, waterbody=spot.waterbody.name_ru, x=spot.x, y=spot.y, description=spot.description, catches_24h=count_since(timedelta(hours=24)), catches_3d=count_since(timedelta(days=3)), catches_7d=count_since(timedelta(days=7)), top_baits=[name for name, _ in bait_counts.most_common(5)])
|
||||
provenance = [
|
||||
(report.raw_payload or {}).get("provenance", {})
|
||||
for report in reports
|
||||
if isinstance((report.raw_payload or {}).get("provenance", {}), dict)
|
||||
]
|
||||
precisions = [item.get("coordinate_precision") for item in provenance]
|
||||
precision = max((value for value in precisions if value in {"exact", "approximate", "area", "missing"}), key={"exact": 0, "approximate": 1, "area": 2, "missing": 3}.get, default="exact")
|
||||
sources = sorted({str(item.get("source_system")) for item in provenance if item.get("source_system")}) or ["players"]
|
||||
return SpotOut(id=spot.id, waterbody_slug=spot.waterbody.slug, waterbody=spot.waterbody.name_ru, x=spot.x, y=spot.y, description=spot.description, catches_24h=count_since(timedelta(hours=24)), catches_3d=count_since(timedelta(days=3)), catches_7d=count_since(timedelta(days=7)), top_baits=[name for name, _ in bait_counts.most_common(5)], coordinate_precision=precision, coordinate_sources=sources)
|
||||
|
||||
|
||||
def _report_source(report: CatchReport) -> str:
|
||||
|
||||
@@ -15,10 +15,12 @@ from ..community_review import ExternalReviewError, map_observation, publish_obs
|
||||
from ..config import settings
|
||||
from ..dependencies import Db
|
||||
from ..importer import ImportAlreadyRunning, ImportSourceError, import_records
|
||||
from ..media_catalog import review_assets, review_file
|
||||
from rf4_research.media_assets import publish_quality_upgrades, rollback_quality_upgrade
|
||||
|
||||
from ..media_catalog import MEDIA_ROOT, review_assets, review_file
|
||||
from ..models import CatchReport, CommunityImportRun, DataSource, ExternalObservation, Fish, ModerationEvent, ModerationStatus, OfficialRecordImport, SourceType, Waterbody
|
||||
from ..public_cache import public_cache
|
||||
from ..schemas import AdminCatchReportOut, AdminMediaReviewOut, AdminModerationHistoryOut, AdminSourceStatusOut, CatchReportCreated, ExternalAliasSuggestionOut, ExternalObservationAction, ExternalObservationDecision, ExternalObservationMapping, ExternalObservationOut, ExternalObservationPublished, ImportRunOut, ModerationUpdate
|
||||
from ..schemas import AdminCatchReportOut, AdminMediaDecision, AdminMediaReviewOut, AdminMediaRollback, AdminModerationHistoryOut, AdminSourceStatusOut, CatchReportCreated, ExternalAliasSuggestionOut, ExternalObservationAction, ExternalObservationDecision, ExternalObservationMapping, ExternalObservationOut, ExternalObservationPublished, ImportRunOut, ModerationUpdate
|
||||
from ..storage import delete_screenshot, signed_screenshot_url
|
||||
from ..time_utils import aware
|
||||
|
||||
@@ -50,6 +52,32 @@ def admin_media_asset(digest: str, _: Annotated[str, Depends(_admin)]) -> FileRe
|
||||
return FileResponse(path, media_type=media_type, headers={"Cache-Control": "private, no-store"})
|
||||
|
||||
|
||||
@router.post("/api/v1/admin/media/upgrades/publish")
|
||||
def admin_publish_media_upgrades(
|
||||
payload: AdminMediaDecision,
|
||||
_: Annotated[str, Depends(_admin)],
|
||||
) -> dict[str, int]:
|
||||
"""Atomically publish all stored quality upgrades after an explicit decision."""
|
||||
try:
|
||||
return publish_quality_upgrades(MEDIA_ROOT / "manifest.json", note=payload.note)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=409, detail=str(exc)) from exc
|
||||
|
||||
|
||||
@router.post("/api/v1/admin/media/upgrades/rollback")
|
||||
def admin_rollback_media_upgrade(
|
||||
payload: AdminMediaRollback,
|
||||
_: Annotated[str, Depends(_admin)],
|
||||
) -> dict[str, str]:
|
||||
"""Restore one superseded fallback while retaining the reviewed candidate."""
|
||||
try:
|
||||
return rollback_quality_upgrade(
|
||||
MEDIA_ROOT / "manifest.json", asset_url=payload.asset_url, note=payload.note,
|
||||
)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=409, detail=str(exc)) from exc
|
||||
|
||||
|
||||
@router.get("/api/v1/admin/diagnostics")
|
||||
def admin_diagnostics(db: Db, _: Annotated[str, Depends(_admin)]) -> JSONResponse:
|
||||
report_counts = {status.value: count for status, count in db.execute(
|
||||
|
||||
@@ -20,6 +20,16 @@ class WaterbodyOut(BaseModel):
|
||||
slug: str
|
||||
name_ru: str
|
||||
unlock_level: int | None
|
||||
fish_species_count: int | None
|
||||
source_system: str | None
|
||||
source_external_id: str | None
|
||||
source_url: str | None
|
||||
description: str | None
|
||||
source_aliases: list[str] | None
|
||||
source_fish_species: list[str] | None
|
||||
source_image_urls: list[str] | None
|
||||
source_point_urls: list[str] | None
|
||||
source_checked_at: datetime | None
|
||||
|
||||
|
||||
class BaitOut(BaseModel):
|
||||
@@ -48,6 +58,8 @@ class ActivityOut(BaseModel):
|
||||
confidence_score: int
|
||||
explanation: str
|
||||
sources: list[str]
|
||||
coordinate_precision: str
|
||||
coordinate_sources: list[str]
|
||||
|
||||
|
||||
class PaginatedActivityOut(BaseModel):
|
||||
@@ -82,6 +94,8 @@ class SpotOut(BaseModel):
|
||||
catches_3d: int
|
||||
catches_7d: int
|
||||
top_baits: list[str]
|
||||
coordinate_precision: str
|
||||
coordinate_sources: list[str]
|
||||
|
||||
|
||||
class OfficialRecordOut(BaseModel):
|
||||
@@ -318,7 +332,17 @@ class AdminMediaReviewOut(BaseModel):
|
||||
height: int | None
|
||||
content_type: str | None
|
||||
image_url: str
|
||||
asset_url: str
|
||||
source_system: str
|
||||
source_url: str
|
||||
duplicate_of: str | None
|
||||
supersedes: str | None
|
||||
derivatives: list[AdminMediaDerivativeOut]
|
||||
|
||||
|
||||
class AdminMediaDecision(BaseModel):
|
||||
note: str = Field(min_length=1, max_length=1000)
|
||||
|
||||
|
||||
class AdminMediaRollback(AdminMediaDecision):
|
||||
asset_url: str = Field(min_length=1, max_length=2000)
|
||||
|
||||
@@ -13,6 +13,7 @@ from app.community_importer import stage_observations
|
||||
from app.importer import ImportAlreadyRunning
|
||||
from app.main import app
|
||||
from app.models import Bait, BaitKind, CatchReport, DataSource, ExternalEntityAlias, ExternalObservation, Fish, ImportStatus, ModerationEvent, ModerationStatus, OfficialRecordImport, SourceType, Spot, SubmissionAttempt, Waterbody
|
||||
from app.routers import admin as admin_router
|
||||
|
||||
|
||||
engine = create_engine("sqlite://", connect_args={"check_same_thread": False}, poolclass=StaticPool)
|
||||
@@ -182,6 +183,34 @@ def test_admin_media_review_requires_auth() -> None:
|
||||
assert all({"role", "format", "width", "height"} <= set(derivative) for derivative in response.json()[0]["derivatives"])
|
||||
|
||||
|
||||
def test_admin_media_decisions_require_auth_and_note(monkeypatch) -> None:
|
||||
assert client.post("/api/v1/admin/media/upgrades/publish", json={"note": "publish"}).status_code == 401
|
||||
assert client.post("/api/v1/admin/media/upgrades/rollback", json={"asset_url": "https://example.test/a", "note": "rollback"}).status_code == 401
|
||||
|
||||
monkeypatch.setattr(admin_router, "publish_quality_upgrades", lambda path, note: {"published": 2, "retained_fallbacks": 2})
|
||||
publish = client.post(
|
||||
"/api/v1/admin/media/upgrades/publish",
|
||||
json={"note": "visual review complete"},
|
||||
headers={"Authorization": "Bearer change-me-in-production"},
|
||||
)
|
||||
assert publish.status_code == 200
|
||||
assert publish.json() == {"published": 2, "retained_fallbacks": 2}
|
||||
|
||||
monkeypatch.setattr(admin_router, "rollback_quality_upgrade", lambda path, asset_url, note: {"rolled_back": asset_url, "restored": "https://example.test/fallback"})
|
||||
rollback = client.post(
|
||||
"/api/v1/admin/media/upgrades/rollback",
|
||||
json={"asset_url": "https://example.test/a", "note": "fallback is preferred"},
|
||||
headers={"Authorization": "Bearer change-me-in-production"},
|
||||
)
|
||||
assert rollback.status_code == 200
|
||||
assert rollback.json()["rolled_back"] == "https://example.test/a"
|
||||
assert client.post(
|
||||
"/api/v1/admin/media/upgrades/publish",
|
||||
json={"note": ""},
|
||||
headers={"Authorization": "Bearer change-me-in-production"},
|
||||
).status_code == 422
|
||||
|
||||
|
||||
def test_liveness_does_not_probe_dependencies() -> None:
|
||||
response = client.get("/health?token=must-not-be-logged")
|
||||
assert response.json() == {"status": "ok"}
|
||||
|
||||
@@ -3,7 +3,7 @@ from datetime import datetime, timezone
|
||||
from sqlalchemy import create_engine
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.catalog_audit import audit_catalog
|
||||
from app.catalog_audit import audit_catalog, audit_waterbody_catalog
|
||||
from app.database import Base
|
||||
from app.models import CatchReport, Fish, ModerationStatus, SourceType, Spot, Waterbody
|
||||
|
||||
@@ -24,3 +24,24 @@ def test_catalog_audit_checks_the_whole_catalog() -> None:
|
||||
assert result["reports"] == 1
|
||||
assert result["invalid_coordinates"] == 1
|
||||
assert result["failures"] == 1
|
||||
|
||||
|
||||
def test_waterbody_catalog_audit_reports_snapshot_gaps_without_legacy_rows() -> None:
|
||||
engine = create_engine("sqlite://")
|
||||
Base.metadata.create_all(engine)
|
||||
with Session(engine) as db:
|
||||
now = datetime.now(timezone.utc)
|
||||
db.add_all([
|
||||
Waterbody(
|
||||
slug="lake", name_ru="Озеро", source_system="rf4db",
|
||||
source_external_id="level_001_lake", source_url="https://rf4db.com/ru/maps/level_001_lake",
|
||||
source_checked_at=now,
|
||||
),
|
||||
Waterbody(slug="legacy", name_ru="Старое озеро"),
|
||||
])
|
||||
db.commit()
|
||||
result = audit_waterbody_catalog(db, {"level_001_lake", "level_002_river"})
|
||||
assert result["expected"] == 2
|
||||
assert result["observed"] == 1
|
||||
assert result["missing_source_external_ids"] == ["level_002_river"]
|
||||
assert result["failures"] == 1
|
||||
|
||||
@@ -7,7 +7,7 @@ import pytest
|
||||
from sqlalchemy import create_engine, func, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.community_importer import CommunityImportError, stage_observations
|
||||
from app.community_importer import CommunityImportError, stage_observations, update_waterbody_detail, update_waterbody_details, upsert_waterbody_catalog
|
||||
from app.community_review import ExternalReviewError, map_observation, publish_observation, suggest_aliases
|
||||
from app.source_lifecycle import record_scheduled_source_check, record_source_check
|
||||
from app.database import Base
|
||||
@@ -42,6 +42,20 @@ def record(source: str = "rf4db", external_id: str = "catch-1") -> dict[str, obj
|
||||
}
|
||||
|
||||
|
||||
def waterbody_row(**overrides: object) -> dict[str, object]:
|
||||
row: dict[str, object] = {
|
||||
"source_system": "rf4db",
|
||||
"source_external_id": "level_001_mosquito",
|
||||
"source_url": "https://rf4db.com/ru/maps/level_001_mosquito",
|
||||
"name": "оз. Комариное",
|
||||
"unlock_level": 1,
|
||||
"unlock_label": "1",
|
||||
"fish_species_count": 20,
|
||||
}
|
||||
row.update(overrides)
|
||||
return row
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def db() -> Session:
|
||||
engine = create_engine("sqlite://")
|
||||
@@ -69,6 +83,62 @@ def test_staging_is_idempotent_and_preserves_first_seen(db: Session) -> None:
|
||||
assert source is not None and source.enabled is True
|
||||
|
||||
|
||||
def test_waterbody_catalog_upsert_is_idempotent_and_non_destructive(db: Session) -> None:
|
||||
first = datetime(2026, 9, 16, 10, tzinfo=timezone.utc)
|
||||
assert upsert_waterbody_catalog(db, [waterbody_row()], fetched_at=first) == (1, 0)
|
||||
item = db.scalar(select(Waterbody).where(Waterbody.source_external_id == "level_001_mosquito"))
|
||||
assert item is not None
|
||||
assert item.slug == "оз-комариное"
|
||||
assert item.source_checked_at.replace(tzinfo=timezone.utc) == first
|
||||
assert item.fish_species_count == 20
|
||||
|
||||
assert upsert_waterbody_catalog(db, [waterbody_row(name="Озеро Комариное", unlock_level=2)], fetched_at=first) == (0, 1)
|
||||
item = db.scalar(select(Waterbody).where(Waterbody.source_external_id == "level_001_mosquito"))
|
||||
assert item is not None
|
||||
assert (item.name_ru, item.unlock_level, item.fish_species_count) == ("Озеро Комариное", 2, 20)
|
||||
assert db.scalar(select(Waterbody).where(Waterbody.name_ru == "оз. Комариное")) is None
|
||||
|
||||
|
||||
def test_waterbody_catalog_rejects_untrusted_source(db: Session) -> None:
|
||||
with pytest.raises(CommunityImportError, match="source_url"):
|
||||
upsert_waterbody_catalog(db, [waterbody_row(source_url="https://example.test/map")])
|
||||
|
||||
|
||||
def test_waterbody_detail_updates_only_imported_identity_without_media_roles(db: Session) -> None:
|
||||
upsert_waterbody_catalog(db, [waterbody_row()])
|
||||
assert update_waterbody_detail(db, {
|
||||
"source_system": "rf4db",
|
||||
"source_external_id": "level_001_mosquito",
|
||||
"source_url": "https://rf4db.com/ru/maps/level_001_mosquito",
|
||||
"name": "оз. Комариное",
|
||||
"description": "Каменистые берега.",
|
||||
"aliases": ["Комариное", "Комариное"],
|
||||
"fish_species": ["Щука", "Окунь"],
|
||||
"image_urls": ["https://oss.rf4db.com/map.webp"],
|
||||
"point_urls": ["https://rf4db.com/ru/maps/level_001_mosquito/spots/12-34"],
|
||||
}) is True
|
||||
item = db.scalar(select(Waterbody).where(Waterbody.source_external_id == "level_001_mosquito"))
|
||||
assert item is not None
|
||||
assert item.source_aliases == ["Комариное"]
|
||||
assert item.source_fish_species == ["Щука", "Окунь"]
|
||||
assert item.source_image_urls == ["https://oss.rf4db.com/map.webp"]
|
||||
|
||||
|
||||
def test_waterbody_detail_batch_validates_before_writing(db: Session) -> None:
|
||||
upsert_waterbody_catalog(db, [waterbody_row()])
|
||||
valid = {
|
||||
"source_system": "rf4db", "source_external_id": "level_001_mosquito",
|
||||
"source_url": "https://rf4db.com/ru/maps/level_001_mosquito", "name": "оз. Комариное",
|
||||
"description": "Описание", "aliases": [], "fish_species": ["Щука"],
|
||||
"image_urls": [], "point_urls": [],
|
||||
}
|
||||
invalid = valid | {"source_external_id": "unknown", "source_url": "https://example.test/map"}
|
||||
with pytest.raises(CommunityImportError, match="source_url"):
|
||||
update_waterbody_details(db, [valid, invalid])
|
||||
item = db.scalar(select(Waterbody).where(Waterbody.source_external_id == "level_001_mosquito"))
|
||||
assert item is not None and item.description is None
|
||||
|
||||
|
||||
def test_external_ids_are_isolated_by_source(db: Session) -> None:
|
||||
created, updated = stage_observations(db, [record("rf4db"), record("rf4stat-fishing")])
|
||||
|
||||
@@ -104,14 +174,28 @@ def test_complete_observation_with_reviewed_aliases_is_published(db: Session) ->
|
||||
assert item.catch_report is not None
|
||||
assert item.catch_report.fish_id == fish.id
|
||||
assert item.catch_report.waterbody_id == waterbody.id
|
||||
assert db.scalar(select(func.count()).select_from(CatchReport)) == 1
|
||||
|
||||
assert stage_observations(db, [record() | {"weight_g": 5_000}]) == (0, 1)
|
||||
db.refresh(item)
|
||||
assert item.status == "published"
|
||||
assert db.scalar(select(func.count()).select_from(CatchReport)) == 1
|
||||
|
||||
|
||||
def test_observation_preserves_coordinate_text_and_precision(db: Session) -> None:
|
||||
stage_observations(db, [record() | {
|
||||
"source_external_id": "coordinate-area",
|
||||
"x": None, "y": None, "coordinate_raw": "северная бухта",
|
||||
"coordinate_precision": "area", "weight_g": None,
|
||||
}])
|
||||
item = db.scalar(select(ExternalObservation).where(ExternalObservation.source_external_id == "coordinate-area"))
|
||||
assert item is not None
|
||||
assert (item.coordinate_raw, item.coordinate_precision, item.x, item.y) == ("северная бухта", "area", None, None)
|
||||
|
||||
|
||||
def test_coordinate_precision_rejects_unknown_value(db: Session) -> None:
|
||||
with pytest.raises(CommunityImportError, match="invalid coordinate_precision"):
|
||||
stage_observations(db, [record() | {"coordinate_precision": "guess"}])
|
||||
|
||||
|
||||
def test_changed_published_record_requires_review_and_reuses_report(db: Session) -> None:
|
||||
fish = Fish(slug="pike", name_ru="Щука")
|
||||
water = Waterbody(slug="test-lake", name_ru="Тестовое озеро")
|
||||
|
||||
@@ -8,6 +8,7 @@ import TackleGlyph from "./TackleGlyph.astro";
|
||||
const { item } = Astro.props as { item: Activity };
|
||||
const level = activityLevel(item.activity_score);
|
||||
const limited = item.catches < 3;
|
||||
const precision = { exact: "точные", approximate: "приблизительные", area: "район", missing: "не указаны" }[item.coordinate_precision];
|
||||
---
|
||||
<a class="spot-card" data-testid={`spot-${item.x}-${item.y}`} href={spotPath(item)}>
|
||||
<span class="spot-rank">{String(item.activity_score).padStart(2,"0")}</span>
|
||||
@@ -15,7 +16,7 @@ const limited = item.catches < 3;
|
||||
<div class="spot-topline"><span>{item.waterbody}</span><span class="activity-pill" data-activity-level={level.short}><i></i>{level.short}</span>{limited && <span class="data-quality">Данных мало</span>}</div>
|
||||
<h3>{item.fish}</h3>
|
||||
<FishSilhouette name={item.fish}/>
|
||||
<div class="spot-meta"><span><FishingIcon name="pin" size={14}/> {item.x}:{item.y}</span><span><FishingIcon name="clock" size={14}/> {ago(item.last_confirmed_at)}</span></div>
|
||||
<div class="spot-meta"><span><FishingIcon name="pin" size={14}/> {item.x}:{item.y} · {precision}</span><span><FishingIcon name="clock" size={14}/> {ago(item.last_confirmed_at)}</span></div>
|
||||
<p class="data-note">{item.explanation}</p>
|
||||
<DataPassport sources={item.sources} observedAt={item.last_confirmed_at} confidence={item.confidence_score}/>
|
||||
<div class="bait-line"><TackleGlyph name={item.best_bait}/><div><span>Работает сейчас</span><strong>{item.best_bait ?? "не указана"}</strong></div></div>
|
||||
|
||||
@@ -4,6 +4,12 @@ import type { MediaAsset } from "../lib/api";
|
||||
const { asset, compact = false, sourceLink = false } = Astro.props as { asset: MediaAsset; compact?: boolean; sourceLink?: boolean };
|
||||
---
|
||||
<figure class:list={["entity-media", { "entity-media--compact": compact }]}>
|
||||
<span class="entity-media__frame"><img src={asset.image_url} alt={asset.label ?? "Иллюстрация RF4"} width={asset.width} height={asset.height} loading="lazy" decoding="async" /></span>
|
||||
<span class="entity-media__frame"><picture>
|
||||
{(["avif", "webp"] as const).map(format => {
|
||||
const variants = (asset.variants ?? []).filter(item => item.format === format);
|
||||
return variants.length ? <source type={`image/${format}`} srcset={variants.map(item => `${item.url} ${item.width}w`).join(", ")} sizes={compact ? "180px" : "(max-width: 720px) 100vw, 720px"} /> : null;
|
||||
})}
|
||||
<img src={asset.image_url} alt={asset.label ?? "Иллюстрация RF4"} width={asset.width} height={asset.height} loading="lazy" decoding="async" />
|
||||
</picture></span>
|
||||
<figcaption><SourceBadge source={asset.source_system} href={sourceLink ? asset.source_url : undefined} /><span>{asset.label ?? "Справочный материал"}</span></figcaption>
|
||||
</figure>
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
---
|
||||
import { pageHref, pageWindow } from "../lib/pagination";
|
||||
|
||||
interface Props {
|
||||
path: string;
|
||||
params: URLSearchParams;
|
||||
total: number;
|
||||
limit: number;
|
||||
offset: number;
|
||||
anchor?: string;
|
||||
itemLabel?: string;
|
||||
}
|
||||
|
||||
const { path, params, total, limit, offset, anchor = "", itemLabel = "записей" } = Astro.props;
|
||||
const state = pageWindow(total, limit, offset);
|
||||
---
|
||||
{total > limit && <nav class="pagination" aria-label="Пагинация">
|
||||
<span>{state.start}–{state.end} из {total} {itemLabel} · страница {state.page} из {state.pages}</span>
|
||||
<div>
|
||||
{state.previousOffset !== null
|
||||
? <a data-action="secondary" rel="prev" href={pageHref(path, params, state.previousOffset, anchor)}>← Предыдущая</a>
|
||||
: <span class="pagination__disabled" aria-disabled="true">← Предыдущая</span>}
|
||||
{state.nextOffset !== null
|
||||
? <a data-action="secondary" rel="next" href={pageHref(path, params, state.nextOffset, anchor)}>Следующая →</a>
|
||||
: <span class="pagination__disabled" aria-disabled="true">Следующая →</span>}
|
||||
</div>
|
||||
</nav>}
|
||||
@@ -4,6 +4,7 @@ export type Activity = {
|
||||
unique_players: number; average_weight_g: number; max_weight_g: number;
|
||||
last_confirmed_at: string; activity_score: number; confidence_score: number;
|
||||
explanation: string; sources: string[];
|
||||
coordinate_precision: "exact" | "approximate" | "area" | "missing"; coordinate_sources: string[];
|
||||
};
|
||||
|
||||
export type PaginatedActivity = {
|
||||
@@ -13,9 +14,9 @@ export type PaginatedActivity = {
|
||||
offset: number;
|
||||
};
|
||||
|
||||
export type Spot = { id: string; waterbody_slug: string; waterbody: string; x: number; y: number; description: string | null; catches_24h: number; catches_3d: number; catches_7d: number; top_baits: string[] };
|
||||
export type Spot = { id: string; waterbody_slug: string; waterbody: string; x: number; y: number; description: string | null; catches_24h: number; catches_3d: number; catches_7d: number; top_baits: string[]; coordinate_precision: "exact" | "approximate" | "area" | "missing"; coordinate_sources: string[] };
|
||||
export type Catch = { id: string; fish: string; weight_g: number; bait: string | null; player_name: string | null; caught_at: string | null; reported_at: string; retrieve_method: string | null; retrieve_speed: number | null; source_system: string; source_url: string | null };
|
||||
export type DictionaryItem = { id: string; slug: string; name_ru: string };
|
||||
export type DictionaryItem = { id: string; slug: string; name_ru: string; unlock_level?: number | null; fish_species_count?: number | null; source_system?: string | null; source_external_id?: string | null; source_url?: string | null; description?: string | null; source_aliases?: string[] | null; source_fish_species?: string[] | null; source_image_urls?: string[] | null; source_point_urls?: string[] | null; source_checked_at?: string | null };
|
||||
export type OfficialRecord = { id: string; fish: string; weight_g: number; waterbody: string; bait: string | null; player_name: string | null; record_date: string | null; category: string | null; region: string | null; source_url: string | null; source_system: string };
|
||||
export type PaginatedOfficialRecord = {
|
||||
items: OfficialRecord[];
|
||||
@@ -26,7 +27,8 @@ export type PaginatedOfficialRecord = {
|
||||
export type PublicObservation = { id: string; source_system: string; source_name: string; source_url: string; fish_name: string; waterbody_name: string; x: number | null; y: number | null; weight_g: number | null; last_seen_at: string; missing_fields: string[]; quality: "incomplete" | "unverified" };
|
||||
export type ImportRun = { id: string; started_at: string; finished_at: string | null; status: string; source_url: string; rows_seen: number; rows_created: number; rows_updated: number; error_summary: string | null };
|
||||
export type SourceStatus = { source_system: string; name: string; status: "healthy" | "stale" | "temporarily_limited" | "source_changed" | "waiting" | "disabled"; last_started_at: string | null; last_success_at: string | null; observations: number };
|
||||
export type MediaAsset = { id: string; entity_type: "fish" | "waterbody" | "tackle" | "reference"; entity_key: string; label: string | null; width: number; height: number; content_type: string; image_url: string; source_system: string; source_url: string };
|
||||
export type MediaVariant = { role: "card" | "detail"; format: "webp" | "avif"; width: number; height: number; url: string };
|
||||
export type MediaAsset = { id: string; entity_type: "fish" | "waterbody" | "tackle" | "reference"; entity_key: string; label: string | null; width: number; height: number; content_type: string; image_url: string; source_system: string; source_url: string; variants?: MediaVariant[] };
|
||||
|
||||
export const spotPath = (item: Pick<Activity, "waterbody_slug" | "x" | "y">) => `/spots/${item.waterbody_slug}-${item.x}x${item.y}`;
|
||||
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
export type PageWindow = {
|
||||
start: number;
|
||||
end: number;
|
||||
page: number;
|
||||
pages: number;
|
||||
previousOffset: number | null;
|
||||
nextOffset: number | null;
|
||||
};
|
||||
|
||||
export const pageWindow = (total: number, limit: number, offset: number): PageWindow => {
|
||||
const safeTotal = Math.max(0, Math.trunc(total));
|
||||
const safeLimit = Math.max(1, Math.trunc(limit));
|
||||
const safeOffset = Math.max(0, Math.trunc(offset));
|
||||
const end = Math.min(safeOffset + safeLimit, safeTotal);
|
||||
return {
|
||||
start: safeTotal > 0 && safeOffset < safeTotal ? safeOffset + 1 : 0,
|
||||
end,
|
||||
page: Math.floor(safeOffset / safeLimit) + 1,
|
||||
pages: Math.max(1, Math.ceil(safeTotal / safeLimit)),
|
||||
previousOffset: safeOffset > 0 ? Math.max(0, safeOffset - safeLimit) : null,
|
||||
nextOffset: end < safeTotal ? safeOffset + safeLimit : null,
|
||||
};
|
||||
};
|
||||
|
||||
export const pageHref = (path: string, search: URLSearchParams, offset: number, anchor = "") => {
|
||||
const params = new URLSearchParams(search);
|
||||
if (offset > 0) params.set("offset", String(offset));
|
||||
else params.delete("offset");
|
||||
const query = params.toString();
|
||||
return `${path}${query ? `?${query}` : ""}${anchor}`;
|
||||
};
|
||||
@@ -8,7 +8,7 @@ const apiUrl = import.meta.env.PUBLIC_API_URL || "http://localhost:8000";
|
||||
<section class="moderation-app" data-api-url={apiUrl}>
|
||||
<AdminNav />
|
||||
<form class="admin-login" autocomplete="off"><label>Административный токен<input name="token" type="password" required autocomplete="off" /></label><button data-action="primary" type="submit">Открыть медиатеку</button></form>
|
||||
<p class="privacy">Публичные файлы не переключаются из этого экрана. Токен хранится только в памяти страницы.</p>
|
||||
<p class="privacy">Публикация требует явной причины и атомарно сохраняет fallback для rollback. Токен хранится только в памяти страницы.</p>
|
||||
<div class="admin-session-bar" hidden><span>Административная сессия активна</span><button data-action="secondary" type="button" data-admin-logout>Выйти</button></div>
|
||||
<form class="admin-queue-filters" hidden><label>Тип<select name="entity_type"><option value="">Все типы</option><option value="fish">Рыбы</option><option value="waterbody">Водоёмы</option><option value="tackle">Снасти</option><option value="reference">Справка</option></select></label><label>Состояние<select name="status"><option value="">Все состояния</option><option value="approved">Approved</option><option value="upgrade_queued">Upgrade queued</option><option value="upgrade_stored">Upgrade stored</option></select></label><button data-action="primary" type="submit">Применить</button></form>
|
||||
<div class="notice error" data-admin-error role="alert" hidden></div><div class="notice success" data-admin-status role="status" hidden></div><section class="media-library__grid" data-media-list aria-live="polite"></section>
|
||||
@@ -47,7 +47,9 @@ const apiUrl = import.meta.env.PUBLIC_API_URL || "http://localhost:8000";
|
||||
if (!rows.length && offset > 0) { offset = 0; return load(); }
|
||||
const assets = rows.slice(0, 50); if (pages) pages.hidden = !assets.length; if (previous) previous.disabled = offset === 0; if (next) next.disabled = rows.length <= 50; if (pageNumber) pageNumber.textContent = `Страница ${offset / 50 + 1}`;
|
||||
if (!assets.length) { list.innerHTML = '<div class="state"><h2>Кандидатов нет</h2><p>Для выбранных фильтров нет approved или upgrade_queued файлов.</p></div>'; return; }
|
||||
list.innerHTML = assets.map(asset => { const image = url(asset.image_url); const source = url(asset.source_url); const variants = (asset.derivatives as Record<string, unknown>[] ?? []).map(item => `${esc(item.format)} ${esc(item.width)}×${esc(item.height)}`).join(", "); return `<article class="media-library__card"><a href="${image}" target="_blank" rel="noreferrer">${image ? `<img src="${image}" alt="${esc(asset.label)}" loading="lazy" />` : ""}</a><h2>${esc(asset.label)}</h2><span>${esc(asset.status)} · ${esc(asset.entity_type)} · ${esc(asset.width)}×${esc(asset.height)}</span><p>${esc(asset.source_system)}${asset.duplicate_of ? ` · duplicate_of ${esc(asset.duplicate_of)}` : ""}</p>${variants ? `<small>Производные: ${variants}</small>` : "<small>Производных нет</small>"}${source ? `<a href="${source}" target="_blank" rel="noreferrer">Первоисточник →</a>` : ""}</article>`; }).join("");
|
||||
list.innerHTML = assets.map(asset => { const image = url(asset.image_url); const source = url(asset.source_url); const variants = (asset.derivatives as Record<string, unknown>[] ?? []).map(item => `${esc(item.format)} ${esc(item.width)}×${esc(item.height)}`).join(", "); const rollback = asset.status === "approved" && asset.supersedes ? `<button data-media-rollback="${esc(asset.asset_url)}" type="button" data-action="secondary">Откатить замену</button>` : ""; return `<article class="media-library__card"><a href="${image}" target="_blank" rel="noreferrer">${image ? `<img src="${image}" alt="${esc(asset.label)}" loading="lazy" />` : ""}</a><h2>${esc(asset.label)}</h2><span>${esc(asset.status)} · ${esc(asset.entity_type)} · ${esc(asset.width)}×${esc(asset.height)}</span><p>${esc(asset.source_system)}${asset.duplicate_of ? ` · duplicate_of ${esc(asset.duplicate_of)}` : ""}</p>${variants ? `<small>Производные: ${variants}</small>` : "<small>Производных нет</small>"}${source ? `<a href="${source}" target="_blank" rel="noreferrer">Первоисточник →</a>` : ""}${rollback}</article>`; }).join("");
|
||||
list.querySelectorAll<HTMLButtonElement>("[data-media-rollback]").forEach(button => button.addEventListener("click", async () => { const note = window.prompt("Причина отката:"); if (!note?.trim()) return; button.disabled = true; try { const response = await fetch(`${root?.dataset.apiUrl}/api/v1/admin/media/upgrades/rollback`, {method:"POST", headers:{Authorization:`Bearer ${token}`, "Content-Type":"application/json"}, body:JSON.stringify({asset_url:button.dataset.mediaRollback, note:note.trim()})}); if (!response.ok) throw new Error(adminErrorMessage(response.status, "Не удалось выполнить откат.")); succeed("Fallback восстановлен."); await load(); } catch (cause) { fail(cause instanceof Error ? cause.message : "Не удалось выполнить откат."); } finally { button.disabled = false; } }));
|
||||
if (assets.some(asset => asset.status === "upgrade_stored")) { const publish = document.createElement("button"); publish.type = "button"; publish.dataset.action = "primary"; publish.textContent = "Опубликовать сохранённые замены"; publish.addEventListener("click", async () => { const note = window.prompt("Причина публикации замен:"); if (!note?.trim()) return; publish.disabled = true; try { const response = await fetch(`${root?.dataset.apiUrl}/api/v1/admin/media/upgrades/publish`, {method:"POST", headers:{Authorization:`Bearer ${token}`, "Content-Type":"application/json"}, body:JSON.stringify({note:note.trim()})}); if (!response.ok) throw new Error(adminErrorMessage(response.status, "Не удалось опубликовать замены.")); succeed("Замены опубликованы атомарно, fallback сохранены."); await load(); } catch (cause) { fail(cause instanceof Error ? cause.message : "Не удалось опубликовать замены."); } finally { publish.disabled = false; } }); list.prepend(publish); }
|
||||
}
|
||||
login?.addEventListener("submit", async event => { event.preventDefault(); token = String(new FormData(login).get("token") || ""); offset = 0; try { await load(); login.hidden = true; if (sessionBar) sessionBar.hidden = false; if (filters) filters.hidden = false; } catch (cause) { list && (list.innerHTML = ""); fail(cause instanceof Error ? cause.message : "Ошибка загрузки."); } });
|
||||
filters?.addEventListener("submit", async event => { event.preventDefault(); offset = 0; try { await load(); } catch (cause) { fail(cause instanceof Error ? cause.message : "Ошибка фильтрации."); } });
|
||||
|
||||
@@ -4,10 +4,19 @@ import FishSilhouette from "../../components/FishSilhouette.astro";
|
||||
import EntityMedia from "../../components/EntityMedia.astro";
|
||||
import PageHero from "../../components/PageHero.astro";
|
||||
import StatePanel from "../../components/StatePanel.astro";
|
||||
import Pagination from "../../components/Pagination.astro";
|
||||
import { api, type DictionaryItem, type MediaAsset } from "../../lib/api";
|
||||
import { findMediaByLabel } from "../../lib/media";
|
||||
let fishes: DictionaryItem[] = [], media: MediaAsset[] = [], unavailable = false;
|
||||
try { [fishes, media] = await Promise.all([api<DictionaryItem[]>("/api/v1/fishes?limit=500"), api<MediaAsset[]>("/api/v1/media/catalog?entity_type=fish")]); } catch { unavailable = true; }
|
||||
const params = Astro.url.searchParams;
|
||||
const pageLimit = 48;
|
||||
const requestedOffset = Number(params.get("offset") ?? 0);
|
||||
let offset = Number.isInteger(requestedOffset) && requestedOffset >= 0 ? requestedOffset : 0;
|
||||
let fishes: DictionaryItem[] = [], allFishes: DictionaryItem[] = [], media: MediaAsset[] = [], unavailable = false;
|
||||
try {
|
||||
[allFishes, media] = await Promise.all([api<DictionaryItem[]>("/api/v1/fishes?limit=500"), api<MediaAsset[]>("/api/v1/media/catalog?entity_type=fish")]);
|
||||
if (allFishes.length > 0 && offset >= allFishes.length) offset = Math.floor((allFishes.length - 1) / pageLimit) * pageLimit;
|
||||
fishes = allFishes.slice(offset, offset + pageLimit);
|
||||
} catch { unavailable = true; }
|
||||
if (unavailable) {
|
||||
Astro.response.status = 503;
|
||||
Astro.response.headers.set("Retry-After", "60");
|
||||
@@ -15,6 +24,7 @@ if (unavailable) {
|
||||
}
|
||||
---
|
||||
<Layout title="Все виды рыб Russian Fishing 4 — RF4 Spotter" description="Каталог рыб RF4 со свежими точками, уловами, приманками и прозрачными источниками данных.">
|
||||
<PageHero eyebrow="Справочник RF4" title="Рыбы" description="Выберите вид, чтобы увидеть свежие подтверждённые точки и полевые сигналы." variant="fish" count={fishes.length} />
|
||||
{unavailable ? <StatePanel tone="unavailable" title="Каталог временно недоступен" description="Не показываем непроверенный список. Попробуйте обновить страницу позже." /> : <nav class="catalog-grid content-grid" aria-label="Виды рыб">{fishes.map((fish,index) => { const asset = findMediaByLabel(media, fish.name_ru); return <a href={`/fish/${fish.slug}`}><span>Вид рыбы · {String(index + 1).padStart(2,"0")}</span><strong>{fish.name_ru}</strong><i>Открыть <b>→</b></i>{asset ? <EntityMedia asset={asset} compact /> : <FishSilhouette name={fish.name_ru} size={92}/>}</a>; })}</nav>}
|
||||
<PageHero eyebrow="Справочник RF4" title="Рыбы" description="Выберите вид, чтобы увидеть свежие подтверждённые точки и полевые сигналы." variant="fish" count={allFishes.length} />
|
||||
{unavailable ? <StatePanel tone="unavailable" title="Каталог временно недоступен" description="Не показываем непроверенный список. Попробуйте обновить страницу позже." /> : <nav class="catalog-grid content-grid" aria-label="Виды рыб">{fishes.map((fish,index) => { const asset = findMediaByLabel(media, fish.name_ru); return <a href={`/fish/${fish.slug}`}><span>Вид рыбы · {String(offset + index + 1).padStart(2,"0")}</span><strong>{fish.name_ru}</strong><i>Открыть <b>→</b></i>{asset ? <EntityMedia asset={asset} compact /> : <FishSilhouette name={fish.name_ru} size={92}/>}</a>; })}</nav>}
|
||||
{!unavailable && <Pagination path="/fish" params={params} total={allFishes.length} limit={pageLimit} offset={offset} itemLabel="видов" />}
|
||||
</Layout>
|
||||
|
||||
@@ -6,6 +6,7 @@ import SourceBadge from "../components/SourceBadge.astro";
|
||||
import TackleGlyph from "../components/TackleGlyph.astro";
|
||||
import SignalFeed from "../components/SignalFeed.astro";
|
||||
import StatePanel from "../components/StatePanel.astro";
|
||||
import Pagination from "../components/Pagination.astro";
|
||||
import { activityLevel, ago, api, kg, plural, type Activity, type DictionaryItem, type PaginatedActivity, type PublicObservation } from "../lib/api";
|
||||
|
||||
const params = Astro.url.searchParams;
|
||||
@@ -13,8 +14,9 @@ const hours = params.get("hours") ?? "24";
|
||||
const waterbody = params.get("waterbody") ?? "";
|
||||
const fish = params.get("fish") ?? "";
|
||||
const sort = params.get("sort") ?? "activity";
|
||||
const activityLimit = 20;
|
||||
const requestedOffset = Number(params.get("offset") ?? 0);
|
||||
const offset = Number.isInteger(requestedOffset) && requestedOffset >= 0 ? requestedOffset : 0;
|
||||
let offset = Number.isInteger(requestedOffset) && requestedOffset >= 0 ? requestedOffset : 0;
|
||||
const requestedSignalLimit = Number(params.get("signals") ?? 12);
|
||||
const signalLimit = Number.isInteger(requestedSignalLimit) ? Math.min(48, Math.max(12, requestedSignalLimit)) : 12;
|
||||
let items: Activity[] = [], signals: PublicObservation[] = [], fishes: DictionaryItem[] = [], waterbodies: DictionaryItem[] = [];
|
||||
@@ -38,11 +40,16 @@ if (!signalsUnavailable) {
|
||||
signals = signalRows.slice(0, signalLimit);
|
||||
}
|
||||
if (!filterError) {
|
||||
const query = new URLSearchParams({ hours, waterbody, fish, sort, limit: "20", offset: String(offset) });
|
||||
const query = new URLSearchParams({ hours, waterbody, fish, sort, limit: String(activityLimit), offset: String(offset) });
|
||||
try {
|
||||
const paginated = await api<PaginatedActivity>(`/api/v1/activity?${query}`);
|
||||
items = offset > 0 ? [...items, ...paginated.items] : paginated.items;
|
||||
items = paginated.items;
|
||||
totalItems = paginated.total;
|
||||
if (totalItems > 0 && offset >= totalItems) {
|
||||
offset = Math.floor((totalItems - 1) / activityLimit) * activityLimit;
|
||||
query.set("offset", String(offset));
|
||||
items = (await api<PaginatedActivity>(`/api/v1/activity?${query}`)).items;
|
||||
}
|
||||
} catch { activityUnavailable = true; }
|
||||
}
|
||||
const catalogUnavailable = fishCatalogUnavailable || waterCatalogUnavailable;
|
||||
@@ -86,9 +93,10 @@ const datasetJsonLd = {
|
||||
<button data-action="primary">⌕ Найти клёв</button>
|
||||
</form></section>
|
||||
<div class="active-filters content-grid" aria-label="Применённые фильтры"><span>{selectedWaterbody}</span><span>{selectedFish}</span><span>{periodLabel}</span><span>{sortLabel}</span>{filtersChanged && <a href="/#results">Сбросить</a>}</div>
|
||||
<section class="dashboard content-grid" id="results"><div class="results-column"><div class="section-heading"><div><span class="overline">За выбранный период</span><h2>Горячие точки</h2></div><span class="result-count">{items.length} из {totalItems} {plural(totalItems, ["точка", "точки", "точек"])}</span></div>{filterError ? <div class="state error-state"><h2>Некорректные фильтры</h2><p>Выберите период и сортировку из предложенных значений.</p><a data-action="secondary" href="/">Сбросить фильтры</a></div> : activityUnavailable ? <StatePanel contained={false} tone="unavailable" title="Горячие точки временно недоступны" description="Полевые сигналы и справочники продолжают работать независимо." /> : items.length ? <><div class="spot-list">{items.map(item => <ActivityCard item={item} />)}</div>{offset + items.length < totalItems && <a class="load-more" data-action="secondary" href={`/?${(() => { const p = new URLSearchParams(params); p.delete("offset"); p.set("offset", String(offset + items.length)); return p.toString(); })()}#results`}>Показать ещё <span>{offset + items.length} из {totalItems}</span> ↓</a>}</> : <div class="state"><h2>Пока нет свежих данных</h2><p>Для выбранных фильтров нет одобренных наблюдений. Расширьте период или выберите другой водоём.</p></div>}</div>
|
||||
<section class="dashboard content-grid" id="results"><div class="results-column"><div class="section-heading"><div><span class="overline">За выбранный период</span><h2>Горячие точки</h2></div><span class="result-count">{items.length} на странице · всего {totalItems}</span></div>{filterError ? <div class="state error-state"><h2>Некорректные фильтры</h2><p>Выберите период и сортировку из предложенных значений.</p><a data-action="secondary" href="/">Сбросить фильтры</a></div> : activityUnavailable ? <StatePanel contained={false} tone="unavailable" title="Горячие точки временно недоступны" description="Полевые сигналы и справочники продолжают работать независимо." /> : items.length ? <div class="spot-list">{items.map(item => <ActivityCard item={item} />)}</div> : <div class="state"><h2>Пока нет свежих данных</h2><p>Для выбранных фильтров нет одобренных наблюдений. Расширьте период или выберите другой водоём.</p></div>}</div>
|
||||
{items[0] && leaderLevel && <aside class="detail-card"><div class="detail-head"><div><span class="overline">Лидер активности</span><h2>{items[0].waterbody} <em>{items[0].x}:{items[0].y}</em></h2></div><a href={`/spots/${items[0].spot_id}`} aria-label="Открыть точку"><FishingIcon name="arrow"/></a></div><div class="source-strip">{items[0].sources.map(source => <SourceBadge source={source}/>)}</div><div class="detail-score"><div class:list={["float-gauge", `meter-level-${leaderMeter}`]} aria-label={`Индекс активности: ${items[0].activity_score} из 100`}><span class="float-gauge__line"></span><span class="float-gauge__water"></span><span class="float-gauge__bob"><i></i></span><strong>{items[0].activity_score}</strong><small>из 100</small></div><div><span>Индекс активности</span><strong data-activity-level={leaderLevel.short}>{leaderLevel.description}</strong><p>{items[0].explanation}</p></div></div><div class="metric-grid"><div><span><FishingIcon name="ripple"/></span><small>Уверенность</small><strong>{items[0].confidence_score}%</strong></div><div><span><FishingIcon name="angler"/></span><small>{plural(items[0].unique_players, ["Игрок", "Игрока", "Игроков"])}</small><strong>{items[0].unique_players}</strong></div><div><span><FishingIcon name="clock"/></span><small>Последний</small><strong>{ago(items[0].last_confirmed_at)}</strong></div><div><span><FishingIcon name="scale"/></span><small>Средний вес</small><strong>{kg(items[0].average_weight_g)}</strong></div></div><div class="best-lure"><span class="overline">Лучшая связка</span><div><TackleGlyph name={items[0].best_bait}/><strong>{items[0].best_bait ?? "Не указана"}</strong><span>{items[0].catches} {plural(items[0].catches, ["улов", "улова", "уловов"])}</span></div></div><p class="confidence-note"><span>✓</span><span><strong>Оценка объяснима.</strong> Один игрок не может искусственно поднять уверенность.</span></p></aside>}
|
||||
</section>
|
||||
{!filterError && !activityUnavailable && <Pagination path="/" params={params} total={totalItems} limit={activityLimit} offset={offset} anchor="#results" itemLabel="точек" />}
|
||||
{signals.length > 0 && <SignalFeed signals={signals}/>}
|
||||
{signalsUnavailable && <div class="content-grid"><StatePanel tone="unavailable" title="Полевые сигналы временно недоступны" description="Горячие точки и справочники продолжают работать независимо." /></div>}
|
||||
{hasMoreSignals && signalLimit < 48 && <a class="signal-more" data-action="secondary" href={moreSignalsHref}>Показать ещё <span>{signalLimit} из доступных</span> ↓</a>}
|
||||
|
||||
@@ -4,19 +4,26 @@ import SourceBadge from "../components/SourceBadge.astro";
|
||||
import SectionHeading from "../components/SectionHeading.astro";
|
||||
import StatePanel from "../components/StatePanel.astro";
|
||||
import TackleGlyph from "../components/TackleGlyph.astro";
|
||||
import Pagination from "../components/Pagination.astro";
|
||||
import { api, kg, type DictionaryItem, type ImportRun, type OfficialRecord, type PaginatedOfficialRecord } from "../lib/api";
|
||||
const params = Astro.url.searchParams;
|
||||
const fish = params.get("fish") ?? "";
|
||||
const waterbody = params.get("waterbody") ?? "";
|
||||
const recordsLimit = 50;
|
||||
const requestedOffset = Number(params.get("offset") ?? 0);
|
||||
const offset = Number.isInteger(requestedOffset) && requestedOffset >= 0 ? requestedOffset : 0;
|
||||
let offset = Number.isInteger(requestedOffset) && requestedOffset >= 0 ? requestedOffset : 0;
|
||||
let items: OfficialRecord[] = [], runs: ImportRun[] = [], fishes: DictionaryItem[] = [], waterbodies: DictionaryItem[] = [];
|
||||
let unavailable = false, showNoIndex = false, totalRecords = 0;
|
||||
try {
|
||||
const query = new URLSearchParams({ fish, waterbody, limit: "50", offset: String(offset) });
|
||||
const query = new URLSearchParams({ fish, waterbody, limit: String(recordsLimit), offset: String(offset) });
|
||||
const paginated = await api<PaginatedOfficialRecord>(`/api/v1/records?${query}`);
|
||||
items = offset > 0 ? [...items, ...paginated.items] : paginated.items;
|
||||
items = paginated.items;
|
||||
totalRecords = paginated.total;
|
||||
if (totalRecords > 0 && offset >= totalRecords) {
|
||||
offset = Math.floor((totalRecords - 1) / recordsLimit) * recordsLimit;
|
||||
query.set("offset", String(offset));
|
||||
items = (await api<PaginatedOfficialRecord>(`/api/v1/records?${query}`)).items;
|
||||
}
|
||||
[runs, fishes, waterbodies] = await Promise.all([api<ImportRun[]>("/api/v1/imports?limit=1"), api<DictionaryItem[]>("/api/v1/fishes"), api<DictionaryItem[]>("/api/v1/waterbodies")]);
|
||||
} catch { unavailable = true; showNoIndex = true; Astro.response.status = 503; Astro.response.headers.set("Retry-After", "60"); }
|
||||
const last = runs[0];
|
||||
@@ -24,10 +31,10 @@ const last = runs[0];
|
||||
<Layout title="Официальные рекорды Russian Fishing 4 — RF4 Spotter" description="Последние официальные рекорды RF4 по рыбам и водоёмам: вес, приманка, игрок, дата и прямая ссылка на источник." noindex={showNoIndex} errorPage={unavailable}>
|
||||
<section class="records-hero"><div><span class="eyebrow">Публичные данные RF4</span><h1>Официальные<br/><em>рекорды</em></h1></div><div class="source-status"><span class:list={["status-dot", last?.status]}></span><strong>{last ? `Импорт: ${last.status}` : "Импорт ещё не запускался"}</strong>{last?.finished_at && <small>{new Date(last.finished_at).toLocaleString("ru-RU")} · {last.rows_seen} строк</small>}</div></section>
|
||||
<form class="record-filters" method="get"><label>Рыба<select name="fish"><option value="">Любая рыба</option>{fishes.map(item => <option value={item.slug} selected={fish === item.slug}>{item.name_ru}</option>)}</select></label><label>Водоём<select name="waterbody"><option value="">Все водоёмы</option>{waterbodies.map(item => <option value={item.slug} selected={waterbody === item.slug}>{item.name_ru}</option>)}</select></label><button data-action="primary">Фильтровать</button>{(fish || waterbody) && <a data-action="quiet" href="/records">Сбросить</a>}</form>
|
||||
<div class="content-grid"><SectionHeading eyebrow="Официальный источник" title="Последние записи" count={`${items.length} из ${totalRecords} записей`} /></div>
|
||||
<div class="content-grid"><SectionHeading eyebrow="Официальный источник" title="Последние записи" count={`${items.length} на странице · всего ${totalRecords}`} /></div>
|
||||
{unavailable && <StatePanel tone="unavailable" title="Источник временно недоступен" description="Сохраняем ранее импортированные данные, но не показываем их как свежее обновление." />}
|
||||
{!unavailable && items.length && <div class="record-table"><div class="record-row record-head"><span>Рыба</span><span>Вес</span><span>Водоём</span><span>Приманка</span><span>Игрок</span><span>Дата и источник</span></div>{items.map(record => <article class="record-row"><strong data-label="Рыба">{record.fish}</strong><strong data-label="Вес">{kg(record.weight_g)}</strong><span data-label="Водоём">{record.waterbody}</span><span class="tackle-label" data-label="Приманка"><TackleGlyph name={record.bait} size={24}/><span>{record.bait ?? "—"}</span></span><span data-label="Игрок">{record.player_name ?? "—"}</span><span data-label="Дата и источник" class="record-provenance"><time>{record.record_date ? new Date(record.record_date).toLocaleDateString("ru-RU") : "—"}</time><SourceBadge source={record.source_system} href={record.source_url}/></span></article>)}</div>}
|
||||
{!unavailable && items.length && offset + items.length < totalRecords && <a class="load-more" data-action="secondary" href={`/records?${(() => { const p = new URLSearchParams(params); p.delete("offset"); p.set("offset", String(offset + items.length)); return p.toString(); })()}`}>Показать ещё <span>{offset + items.length} из {totalRecords}</span> ↓</a>}
|
||||
{!unavailable && <Pagination path="/records" params={params} total={totalRecords} limit={recordsLimit} offset={offset} itemLabel="записей" />}
|
||||
{!unavailable && !items.length && <StatePanel title="Рекорды ещё не импортированы" description="Для выбранных условий записей пока нет." actionHref="/records" actionLabel="Сбросить фильтры" />}
|
||||
<p class="official-note">Источник: <a href="https://rf4game.de/records/region/RU/" rel="noreferrer">официальный сайт Russian Fishing 4</a>. Координаты в официальных таблицах отсутствуют.</p>
|
||||
</Layout>
|
||||
|
||||
@@ -30,5 +30,6 @@ const schema = water ? { "@context":"https://schema.org", "@type":"CollectionPag
|
||||
<AtlasBreadcrumbs items={[{ label: "Водоёмы", href: "/waterbodies" }, { label: water?.name_ru ?? "Не найдено" }]} />
|
||||
<PageHero eyebrow="Свежие данные за 72 часа" title={water?.name_ru ?? "Водоём не найден"} description={water ? `${items.length} ${plural(items.length,["активная точка","активные точки","активных точек"])} для ${fishes.length} ${plural(fishes.length,["вида рыбы","видов рыб","видов рыб"])}.` : undefined} variant={water ? "water" : undefined} identity={water?.slug} />
|
||||
{image && <section class="entity-feature content-grid" aria-label={`Изображение: ${water!.name_ru}`}><EntityMedia asset={image} sourceLink /><div><span class="overline">Карта и образ водоёма</span><h2>{water!.name_ru}</h2><p>Материал показан с прямой атрибуцией. Координаты активных точек ниже относятся к данным наблюдений, а не к геометрии изображения.</p></div></section>}
|
||||
{water && (water.description || water.unlock_level !== undefined || water.source_url || water.source_fish_species) && <section class="data-passport content-grid" aria-label="Паспорт водоёма"><div><span class="overline">Канонические сведения</span><h2>{water.name_ru}</h2>{water.description && <p>{water.description}</p>}{water.unlock_level !== undefined && water.unlock_level !== null && <p>Открывается с уровня: <strong>{water.unlock_level}</strong></p>}{water.source_fish_species?.length && <p>В карточке источника указано видов рыб: <strong>{water.source_fish_species.length}</strong>.</p>}{water.source_aliases?.length && <p>Алиасы источника: {water.source_aliases.join(", ")}.</p>}</div><div><span class="overline">Источник</span>{water.source_url ? <p><a href={water.source_url} rel="noreferrer">{water.source_system ?? "Внешний источник"} · исходная карточка</a></p> : <p>Источник для описания ещё не подтверждён.</p>}{water.source_point_urls?.length && <p>Сохранено ссылок на точки: <strong>{water.source_point_urls.length}</strong>.</p>}{water.source_image_urls?.length && <p>Изображений-кандидатов: <strong>{water.source_image_urls.length}</strong>; публикация требует отдельной проверки.</p>}{water.source_checked_at && <p>Проверено: {new Date(water.source_checked_at).toLocaleDateString("ru-RU")}</p>}</div></section>}
|
||||
{unavailable ? <StatePanel tone="unavailable" title="Данные временно недоступны" description="Каталог сохранён, но свежие наблюдения сейчас не получены." /> : !water ? <StatePanel tone="error" title="Такого водоёма нет в справочнике" actionHref="/waterbodies" actionLabel="Открыть каталог" /> : <section class="catalog-results content-grid"><aside><span class="overline">Рыбы</span>{fishes.length ? <nav>{fishes.map(([fishSlug,name]) => <AtlasEntityLink href={`/waterbodies/${water!.slug}/${fishSlug}`} label={name} kind="fish" identity={name} />)}</nav> : <p>Свежих подтверждённых видов пока нет.</p>}</aside><div>{items.length ? items.map(item => <ActivityCard item={item}/>) : <StatePanel contained={false} title="Свежих точек пока нет" description="Проверьте позже или посмотрите полевые сигналы на главной." />}</div></section>}
|
||||
</Layout>
|
||||
|
||||
@@ -3,9 +3,18 @@ import Layout from "../../layouts/Layout.astro";
|
||||
import PageHero from "../../components/PageHero.astro";
|
||||
import StatePanel from "../../components/StatePanel.astro";
|
||||
import WaterbodyMark from "../../components/WaterbodyMark.astro";
|
||||
import Pagination from "../../components/Pagination.astro";
|
||||
import { api, type DictionaryItem } from "../../lib/api";
|
||||
let waters: DictionaryItem[] = [], unavailable = false;
|
||||
try { waters = await api<DictionaryItem[]>("/api/v1/waterbodies?limit=500"); } catch { unavailable = true; }
|
||||
const params = Astro.url.searchParams;
|
||||
const pageLimit = 48;
|
||||
const requestedOffset = Number(params.get("offset") ?? 0);
|
||||
let offset = Number.isInteger(requestedOffset) && requestedOffset >= 0 ? requestedOffset : 0;
|
||||
let waters: DictionaryItem[] = [], allWaters: DictionaryItem[] = [], unavailable = false;
|
||||
try {
|
||||
allWaters = await api<DictionaryItem[]>("/api/v1/waterbodies?limit=500");
|
||||
if (allWaters.length > 0 && offset >= allWaters.length) offset = Math.floor((allWaters.length - 1) / pageLimit) * pageLimit;
|
||||
waters = allWaters.slice(offset, offset + pageLimit);
|
||||
} catch { unavailable = true; }
|
||||
if (unavailable) {
|
||||
Astro.response.status = 503;
|
||||
Astro.response.headers.set("Retry-After", "60");
|
||||
@@ -13,6 +22,7 @@ if (unavailable) {
|
||||
}
|
||||
---
|
||||
<Layout title="Все водоёмы Russian Fishing 4 — RF4 Spotter" description="Каталог водоёмов RF4 со свежими точками, рыбами, приманками и прозрачными источниками данных.">
|
||||
<PageHero eyebrow="Карта водоёмов RF4" title="Водоёмы" description="Откройте водоём, чтобы увидеть активные виды рыб и последние подтверждённые точки." variant="water" count={waters.length} />
|
||||
{unavailable ? <StatePanel tone="unavailable" title="Каталог временно недоступен" description="Не показываем непроверенный список. Попробуйте обновить страницу позже." /> : <nav class="catalog-grid catalog-grid--waters content-grid" aria-label="Водоёмы">{waters.map((water,index) => <a href={`/waterbodies/${water.slug}`}><span>Водоём · {String(index + 1).padStart(2,"0")}</span><strong>{water.name_ru}</strong><i>Открыть <b>→</b></i><WaterbodyMark identity={water.slug}/></a>)}</nav>}
|
||||
<PageHero eyebrow="Карта водоёмов RF4" title="Водоёмы" description="Откройте водоём, чтобы увидеть активные виды рыб и последние подтверждённые точки." variant="water" count={allWaters.length} />
|
||||
{unavailable ? <StatePanel tone="unavailable" title="Каталог временно недоступен" description="Не показываем непроверенный список. Попробуйте обновить страницу позже." /> : <nav class="catalog-grid catalog-grid--waters content-grid" aria-label="Водоёмы">{waters.map((water,index) => <a href={`/waterbodies/${water.slug}`}><span>Водоём · {String(offset + index + 1).padStart(2,"0")}</span><strong>{water.name_ru}</strong><i>Открыть <b>→</b></i><WaterbodyMark identity={water.slug}/></a>)}</nav>}
|
||||
{!unavailable && <Pagination path="/waterbodies" params={params} total={allWaters.length} limit={pageLimit} offset={offset} itemLabel="водоёмов" />}
|
||||
</Layout>
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
.pagination{width:min(1360px,calc(100% - 64px));margin:22px auto 48px;display:flex;align-items:center;justify-content:space-between;gap:16px;color:var(--text-secondary);font-size:13px}.pagination>div{display:flex;gap:10px}.pagination a,.pagination__disabled{display:inline-flex;align-items:center;min-height:42px;padding:10px 15px;border:1px solid var(--border-strong);border-radius:10px;background:var(--surface-elevated);color:var(--text-secondary);text-decoration:none}.pagination a:hover{background:var(--surface-raised);color:var(--text-primary)}.pagination__disabled{opacity:.5}@media(max-width:720px){.pagination{width:calc(100% - 28px);align-items:stretch;flex-direction:column}.pagination>div{display:grid;grid-template-columns:1fr 1fr}.pagination a,.pagination__disabled{justify-content:center;text-align:center}}
|
||||
@@ -30,6 +30,23 @@ services:
|
||||
volumes:
|
||||
- minio_data:/data
|
||||
|
||||
minio-init:
|
||||
image: minio/mc:RELEASE.2025-07-21T05-28-08Z
|
||||
restart: "no"
|
||||
entrypoint: ["/bin/sh", "-c"]
|
||||
command:
|
||||
- >-
|
||||
mc alias set local http://minio:9000 "$$S3_ACCESS_KEY" "$$S3_SECRET_KEY" >/dev/null &&
|
||||
mc mb --ignore-existing "local/$$S3_BUCKET" >/dev/null &&
|
||||
mc stat "local/$$S3_BUCKET" >/dev/null
|
||||
environment:
|
||||
S3_ACCESS_KEY: ${S3_ACCESS_KEY:-rf4-local}
|
||||
S3_SECRET_KEY: ${S3_SECRET_KEY:-rf4-local-secret}
|
||||
S3_BUCKET: ${S3_BUCKET:-catch-screenshots}
|
||||
depends_on:
|
||||
minio:
|
||||
condition: service_healthy
|
||||
|
||||
api:
|
||||
build:
|
||||
context: .
|
||||
@@ -55,6 +72,8 @@ services:
|
||||
condition: service_completed_successfully
|
||||
minio:
|
||||
condition: service_healthy
|
||||
minio-init:
|
||||
condition: service_completed_successfully
|
||||
ports:
|
||||
- "8000:8000"
|
||||
healthcheck:
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
{
|
||||
"source_system": "rf4db",
|
||||
"source_external_id": "level_000_home",
|
||||
"source_url": "https://rf4db.com/ru/maps/level_000_home",
|
||||
"checked_at": "2026-09-16",
|
||||
"name": "Дачный пруд",
|
||||
"description": null,
|
||||
"aliases": [],
|
||||
"fish_species": ["Лягушка", "Ротан", "Окунь", "Карась золотой", "Карась серебряный", "Линь", "Карп чешуйчатый"],
|
||||
"image_urls": [
|
||||
"https://oss.rf4db.com/game/maps/level_000_home.png",
|
||||
"https://oss.rf4db.com/game/paper_maps_webp/map_level_000_home.webp"
|
||||
],
|
||||
"point_urls": [
|
||||
"https://rf4db.com/ru/positions/b5bd02d2-997c-466f-ad8d-b9b3be726c18",
|
||||
"https://rf4db.com/ru/positions/60770f78-815d-4681-9079-d8ece4efc20b",
|
||||
"https://rf4db.com/ru/positions/5a4e6bb4-b2f7-4085-84ce-b567301c1d80",
|
||||
"https://rf4db.com/ru/positions/ddf76541-6de2-472f-ad1f-636a232a0f4b",
|
||||
"https://rf4db.com/ru/positions/c98d9718-2f06-4069-a227-4d1753d4e14a",
|
||||
"https://rf4db.com/ru/positions/4ac21a3c-0c50-4ca8-b7a5-29481eb034b7",
|
||||
"https://rf4db.com/ru/positions/efada3fc-9f35-4eb1-aee0-55eeb52c1c90",
|
||||
"https://rf4db.com/ru/positions/0cb5d601-5c54-40dc-84d9-05c9c51052ce",
|
||||
"https://rf4db.com/ru/positions/bb4e2977-f4be-4edc-ac7f-476991eb034b7"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"source_system": "rf4db",
|
||||
"source_url": "https://rf4db.com/ru/maps",
|
||||
"checked_at": "2026-09-16",
|
||||
"expected_count": 19,
|
||||
"items": [
|
||||
{"source_external_id":"level_000_home","source_url":"https://rf4db.com/ru/maps/level_000_home","name":"Дачный пруд","unlock_level":null,"unlock_label":"Старт","fish_species_count":7,"image_url":"https://oss.rf4db.com/game/maps/level_000_home.png"},
|
||||
{"source_external_id":"level_001_lake","source_url":"https://rf4db.com/ru/maps/level_001_lake","name":"оз. Комариное","unlock_level":1,"unlock_label":"Lv.1","fish_species_count":20,"image_url":"https://oss.rf4db.com/game/maps/level_001_lake.png"},
|
||||
{"source_external_id":"level_002_kirzah_river","source_url":"https://rf4db.com/ru/maps/level_002_kirzah_river","name":"р. Вьюнок","unlock_level":1,"unlock_label":"Lv.1","fish_species_count":29,"image_url":"https://oss.rf4db.com/game/maps/level_002_kirzah_river.png"},
|
||||
{"source_external_id":"level_019_american_pond","source_url":"https://rf4db.com/ru/maps/level_019_american_pond","name":"оз. Лосиное","unlock_level":1,"unlock_label":"Lv.1","fish_species_count":19,"image_url":"https://oss.rf4db.com/game/maps/level_019_american_pond.png"},
|
||||
{"source_external_id":"level_004_torfyanoe_ozero","source_url":"https://rf4db.com/ru/maps/level_004_torfyanoe_ozero","name":"оз. Старый Острог","unlock_level":12,"unlock_label":"Lv.12","fish_species_count":21,"image_url":"https://oss.rf4db.com/game/maps/level_004_torfyanoe_ozero.png"},
|
||||
{"source_external_id":"level_010_belaya","source_url":"https://rf4db.com/ru/maps/level_010_belaya","name":"р. Белая","unlock_level":12,"unlock_label":"Lv.12","fish_species_count":29,"image_url":"https://oss.rf4db.com/game/maps/level_010_belaya.png"},
|
||||
{"source_external_id":"level_003_viiveri_lake","source_url":"https://rf4db.com/ru/maps/level_003_viiveri_lake","name":"оз. Куори","unlock_level":16,"unlock_label":"Lv.16","fish_species_count":24,"image_url":"https://oss.rf4db.com/game/maps/level_003_viiveri_lake.png"},
|
||||
{"source_external_id":"level_007_grass_lake","source_url":"https://rf4db.com/ru/maps/level_007_grass_lake","name":"оз. Медвежье","unlock_level":18,"unlock_label":"Lv.18","fish_species_count":26,"image_url":"https://oss.rf4db.com/game/maps/level_007_grass_lake.png"},
|
||||
{"source_external_id":"level_005_volhov","source_url":"https://rf4db.com/ru/maps/level_005_volhov","name":"р. Волхов","unlock_level":20,"unlock_label":"Lv.20","fish_species_count":32,"image_url":"https://oss.rf4db.com/game/maps/level_005_volhov.png"},
|
||||
{"source_external_id":"level_013_sev_don","source_url":"https://rf4db.com/ru/maps/level_013_sev_don","name":"р. Северский Донец","unlock_level":22,"unlock_label":"Lv.22","fish_species_count":40,"image_url":"https://oss.rf4db.com/game/maps/level_013_sev_don.png"},
|
||||
{"source_external_id":"level_008_sura_river","source_url":"https://rf4db.com/ru/maps/level_008_sura_river","name":"р. Сура","unlock_level":24,"unlock_label":"Lv.24","fish_species_count":37,"image_url":"https://oss.rf4db.com/game/maps/level_008_sura_river.png"},
|
||||
{"source_external_id":"level_006_ladoga","source_url":"https://rf4db.com/ru/maps/level_006_ladoga","name":"Ладожское оз.","unlock_level":26,"unlock_label":"Lv.26","fish_species_count":40,"image_url":"https://oss.rf4db.com/game/maps/level_006_ladoga.png"},
|
||||
{"source_external_id":"level_014_carp_lake","source_url":"https://rf4db.com/ru/maps/level_014_carp_lake","name":"оз. Янтарное","unlock_level":26,"unlock_label":"Lv.26","fish_species_count":37,"image_url":"https://oss.rf4db.com/game/maps/level_014_carp_lake.png"},
|
||||
{"source_external_id":"level_011_ladoga_02","source_url":"https://rf4db.com/ru/maps/level_011_ladoga_02","name":"Ладожский архипелаг","unlock_level":27,"unlock_label":"Lv.27","fish_species_count":43,"image_url":"https://oss.rf4db.com/game/maps/level_011_ladoga_02.png"},
|
||||
{"source_external_id":"level_009_ahtuba","source_url":"https://rf4db.com/ru/maps/level_009_ahtuba","name":"р. Ахтуба","unlock_level":28,"unlock_label":"Lv.28","fish_species_count":54,"image_url":"https://oss.rf4db.com/game/maps/level_009_ahtuba.png"},
|
||||
{"source_external_id":"level_018_moskitnoe","source_url":"https://rf4db.com/ru/maps/level_018_moskitnoe","name":"оз. Медное","unlock_level":29,"unlock_label":"Lv.29","fish_species_count":30,"image_url":"https://oss.rf4db.com/game/maps/level_018_moskitnoe.png"},
|
||||
{"source_external_id":"level_015_tunguska","source_url":"https://rf4db.com/ru/maps/level_015_tunguska","name":"р. Нижняя Тунгуска","unlock_level":30,"unlock_label":"Lv.30","fish_species_count":41,"image_url":"https://oss.rf4db.com/game/maps/level_015_tunguska.png"},
|
||||
{"source_external_id":"level_016_yama","source_url":"https://rf4db.com/ru/maps/level_016_yama","name":"р. Яма","unlock_level":32,"unlock_label":"Lv.32","fish_species_count":28,"image_url":"https://oss.rf4db.com/game/maps/level_016_yama.png"},
|
||||
{"source_external_id":"level_012_north_sea","source_url":"https://rf4db.com/ru/maps/level_012_north_sea","name":"Норвежское море","unlock_level":34,"unlock_label":"Lv.34","fish_species_count":57,"image_url":"https://oss.rf4db.com/game/maps/level_012_north_sea.png"}
|
||||
]
|
||||
}
|
||||
@@ -26,7 +26,11 @@ def probe(path, expected, authorization=None, method="GET", data=None):
|
||||
assert response.status == expected, f"{method} {path}: expected {expected}, got {response.status}"
|
||||
if path.startswith("/api/v1/admin/") and expected == 200:
|
||||
assert "no-store" in response.headers.get("Cache-Control", "")
|
||||
assert isinstance(json.load(response), dict)
|
||||
payload = json.load(response)
|
||||
if path.endswith("/media/catalog") or path.endswith("/source-status"):
|
||||
assert isinstance(payload, list)
|
||||
else:
|
||||
assert isinstance(payload, dict)
|
||||
if path == "/admin" or path.startswith("/admin/"):
|
||||
if expected == 200:
|
||||
assert "no-store" in response.headers.get("Cache-Control", "")
|
||||
@@ -36,15 +40,21 @@ def probe(path, expected, authorization=None, method="GET", data=None):
|
||||
probe("/admin", 401)
|
||||
probe("/admin", 200, basic)
|
||||
probe("/admin/moderation", 401)
|
||||
probe("/admin/external-sources", 401)
|
||||
probe("/admin/external-sources", 200, basic)
|
||||
probe("/admin/media", 401)
|
||||
probe("/admin/moderation", 200, basic)
|
||||
probe("/admin/moderation", 401, "Bearer " + token)
|
||||
for path in ("/api/v1/admin/diagnostics", "/api/v1/admin/catch-reports"):
|
||||
probe(path, 401)
|
||||
probe(path, 401, basic)
|
||||
probe(path, 401, "Bearer invalid-routing-test-token")
|
||||
probe("/admin/media", 200, basic)
|
||||
# Clear any stale test-client auth failures before exercising rejection paths.
|
||||
probe("/api/v1/admin/diagnostics", 200, "Bearer " + token)
|
||||
probe("/api/v1/admin/media/catalog", 200, "Bearer " + token)
|
||||
probe("/api/v1/admin/source-status", 200, "Bearer " + token)
|
||||
for path in (
|
||||
"/api/v1/admin/diagnostics", "/api/v1/admin/catch-reports",
|
||||
"/api/v1/admin/media/catalog", "/api/v1/admin/source-status",
|
||||
):
|
||||
probe(path, 401)
|
||||
# Invalid UUID prevents writes while checking the moderation request path.
|
||||
probe("/api/v1/admin/catch-reports/not-a-uuid", 401, method="PATCH", data=b'{"status":"approved"}')
|
||||
probe("/api/v1/admin/catch-reports/not-a-uuid", 422, "Bearer " + token,
|
||||
method="PATCH", data=b'{"status":"approved"}')
|
||||
print("Admin auth passed: Basic pages, Bearer API, rejected missing/wrong credentials")
|
||||
|
||||
@@ -84,4 +84,6 @@ test "$($compose exec -T db psql -At -U rf4 -d rf4_spotter -c 'select count(*) f
|
||||
test "$($compose exec -T db psql -At -U rf4 -d rf4_spotter -c 'select count(*) from catch_report')" = "0"
|
||||
WEB_URL="http://127.0.0.1:$BOOTSTRAP_WEB_PORT" BOOTSTRAP_API_URL="http://127.0.0.1:$BOOTSTRAP_API_PORT" BOOTSTRAP_ADMIN_TOKEN=replace-with-at-least-32-random-characters npm --prefix apps/web run test:bootstrap
|
||||
curl -fsS -D - -o /dev/null -H 'Authorization: Bearer replace-with-at-least-32-random-characters' "http://127.0.0.1:$BOOTSTRAP_API_PORT/api/v1/admin/catch-reports" | grep -qi '^cache-control: no-store'
|
||||
curl -fsS -D - -o /dev/null -H 'Authorization: Bearer replace-with-at-least-32-random-characters' "http://127.0.0.1:$BOOTSTRAP_API_PORT/api/v1/admin/media/catalog" | grep -qi '^cache-control: no-store'
|
||||
curl -fsS -D - -o /dev/null -H 'Authorization: Bearer replace-with-at-least-32-random-characters' "http://127.0.0.1:$BOOTSTRAP_API_PORT/api/v1/admin/source-status" | grep -qi '^cache-control: no-store'
|
||||
echo "Production bootstrap passed from empty volumes"
|
||||
|
||||
@@ -8,7 +8,14 @@
|
||||
|
||||
Проверено: 107 Python-тестов проходят, 1 пропущен; Astro check/build, web unit и Caddy adapt проходят. Это не полная приёмка: проверки не покрывают обнаруженные ниже сценарии. Несовместимость activity envelope исправлена у четырёх потребителей по коду; история site cooldown теперь включает disabled endpoint. Не повторять эти изменения без нового воспроизведения.
|
||||
|
||||
`REGRESSION_FIXES_REPORT.md` устарел, содержит противоречивые статусы. Новые задачи ниже пока не выполнены. Источники в сеть для этой проверки не опрашивались.
|
||||
Ниже сохранён исходный снимок критериев до исправлений; формулировки «не выполнено» и секция «основание» описывают состояние базы `4f68d6b`, а не текущую ветку. По повторной приёмке 11 сентября A01–A13 закрыты в [RECOVERY_FIXES_REPORT.md](RECOVERY_FIXES_REPORT.md). 15 сентября пагинация A04 дополнительно переведена с вводящей в заблуждение кнопки «Показать ещё» на серверные страницы previous/next для activity, records и каталогов; фильтры сохраняются, избыточный offset нормализуется на последнюю страницу, сценарий 45 элементов закреплён unit-тестом. Источники в сеть для этой проверки не опрашивались.
|
||||
|
||||
| Исторический пакет | Текущий статус | Актуальное подтверждение |
|
||||
|---|---|---|
|
||||
| A01–A03 | закрыт | `RECOVERY_FIXES_REPORT.md`, тесты readiness/cooldown/redirect |
|
||||
| A04 | закрыт, повторно усилен 15.09 | серверные previous/next, диапазон и unit-тест 45 элементов |
|
||||
| A05–A12 | закрыт | `RECOVERY_FIXES_REPORT.md`, адресные regression-тесты |
|
||||
| A13 | закрыт | итоговый отчёт и единственная активная очередь в `ROADMAP.md` |
|
||||
|
||||
## Порядок и критерии приёмки
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Отчёт по регрессионному аудиту — 9 сентября 2026
|
||||
|
||||
> **Архивный промежуточный отчёт.** Его раздел «оставшиеся регрессии» закрыт последующим пакетом восстановления. Актуальный итог находится в [RECOVERY_FIXES_REPORT.md](RECOVERY_FIXES_REPORT.md), текущие задачи — в [ROADMAP.md](ROADMAP.md).
|
||||
> **Архивный промежуточный отчёт.** Его статусы и раздел «Следующие шаги» относятся только к базе `9ae05ef` и не являются текущим backlog. Регрессии закрыты последующим пакетом восстановления; актуальный итог находится в [RECOVERY_FIXES_REPORT.md](RECOVERY_FIXES_REPORT.md), текущие задачи — только в [ROADMAP.md](ROADMAP.md). Пагинация R09 дополнительно усилена 15 сентября полноценными серверными previous/next-страницами.
|
||||
|
||||
База: `9ae05ef` (после трёх коммитов исправлений предыдущего аудита).
|
||||
Исходный отчёт: [REGRESSION_AUDIT_2026-09-09.md](REGRESSION_AUDIT_2026-09-09.md).
|
||||
@@ -169,7 +169,7 @@
|
||||
|
||||
---
|
||||
|
||||
## Следующие шаги
|
||||
## Исторические следующие шаги (закрыты последующим recovery-пакетом)
|
||||
|
||||
1. **R03** — Research CLI cooldown: исправить `fcntl` locking и `encoding`.
|
||||
2. **R04** — Разделить `configured_sources()` и `site_cooldown_sources()`.
|
||||
|
||||
+4
-4
@@ -42,9 +42,9 @@
|
||||
- [x] **B20 · Аудит качества рыбных изображений.** Offline `media_cli --quality-report` проверяет фактические dimensions опубликованных файлов, отдельно считает неизбежный upscale в карточках и известные альтернативы. На 14.09: 227 из 243 опубликованных рыб имеют только 48×48 PNG RF4MAP и растягиваются до 180 px; 16 имеют 1024×1024 WebP. Для всех 227 низких файлов уже известны альтернативные URL RF4DB. Совпадение сущности больше нельзя считать достаточным основанием пропустить потенциально более качественный файл.
|
||||
- [ ] **B21 · Очередь quality-upgrade.** Текущие approved-файлы не меняются до готовности замены. 15.09 отдельная очередь создана для 226 прямых RF4DB-альтернатив с сохранением `duplicate_of`; первая партия из 40 файлов загружена без ошибок в `upgrade_stored`, ещё 186 замен и 10 кандидатов недостающих рыб ожидают следующих разрешённых 30-минутных окон. Один provenance-only `duplicate`, не связанный с низкоразрешённым published-файлом, намеренно не переведён. Ошибка останавливает домен по существующим правилам; публикация старой версии при этом не прерывается.
|
||||
- [ ] **B22 · Сравнение вариантов разных источников.** Offline-команда `--compare-quality-upgrades` сопоставляет сохранённые кандидаты с опубликованными fallback по `duplicate_of` и проверяет dimensions, размер файла, MIME, прозрачность и aspect ratio. Первая партия: 40/40 кандидатов — прозрачные WebP 1024×1024, все прошли порог 256 px и сохранили пропорции; технических ошибок нет. До переключения остаётся визуально подтвердить соответствие подписи. Маленький файл сохраняется как fallback; источник RF4DB/RF4MAP/официальный RF4 не получает автоматического приоритета.
|
||||
- [ ] **B23 · Безопасное продвижение и provenance.** Добавить связь `supersedes`/`replaced_by`, отдельное решение review и атомарное переключение публичного `entity_key`; сохранить обе исходные ссылки и возможность отката. На сайте показывать источник именно выбранного изображения, а в раскрываемом provenance — все проверенные варианты. Не удалять старый Git-файл в том же коммите, где включается новый.
|
||||
- [ ] **B24 · Производные размеры.** После выбора оригиналов генерировать детерминированные WebP/AVIF thumbnails для каталога и отдельный крупный вариант для detail, фиксировать хэши производных в manifest и отдавать `srcset`. Это уберёт загрузку 1024×1024 на каждой маленькой карточке и исключит browser-upscale 48×48.
|
||||
- [ ] **B25 · Визуальная приёмка media.** Собирать контактный лист «старое / кандидат / выбранное» с названием и источником; вручную проверить минимум все замены и репрезентативные desktop/mobile страницы в light/dark. Gate: нет битых файлов, искажённых пропорций, ложных соответствий, обрезанного объекта и изображений ниже 256 px без явной пометки «низкое разрешение».
|
||||
- [x] **B23 · Безопасное продвижение и provenance.** Связи `supersedes`/`replaced_by`, отдельное решение review, атомарное переключение публичного `entity_key` и явный CLI rollback сохраняют обе исходные ссылки и возможность отката. Публичная карточка показывает источник выбранного изображения, а manifest сохраняет проверенные варианты. Старый Git-файл не удаляется при включении нового.
|
||||
- [x] **B24 · Производные размеры.** После выбора оригиналов генерируются детерминированные WebP/AVIF thumbnails для каталога и отдельный крупный вариант для detail, хэши производных фиксируются в manifest и отдаются через каталог. Маленькие карточки больше не обязаны загружать 1024×1024 оригинал.
|
||||
- [ ] **B25 · Визуальная приёмка media.** Contact sheets для всех 226 замен собраны и просмотрены offline: явных ложных соответствий, обрезки и искажённых пропорций не обнаружено; одно отличие подписи (`Ерш-носарь`/`Ёрш-носарь`) орфографическое. Осталась browser-проверка репрезентативных desktop/mobile страниц в light/dark; sandbox Chromium пока не позволяет её выполнить. Gate: нет битых файлов, искажённых пропорций, ложных соответствий, обрезанного объекта и изображений ниже 256 px без явной пометки «низкое разрешение».
|
||||
|
||||
### Каталог водоёмов, карты и координаты
|
||||
|
||||
@@ -127,7 +127,7 @@
|
||||
- [x] **A01 · Защита маршрутов и границы сессии.** Покрыть точный `/admin` и `/admin/*` единым Caddy Basic Auth, выставлять `noindex` и `no-store` для всех административных ответов, проверить отсутствие обхода через API и корректные `401/429`. Критерий: автоматический proxy-smoke для `/admin`, страниц и `/api/v1/admin/*` с отсутствующим, неверным и валидным доступом.
|
||||
- [x] **A02 · Единая auth/error UX.** Привести dashboard, moderation и external sources к одинаковому поведению при `401`, `409`, `429`, `5xx`, loading/empty-состояниях: понятное сообщение, блокировка повторной отправки, возврат к входу только при истёкшей авторизации. Критерий: regression-тесты на каждый ответ и сохранение введённой причины.
|
||||
- [x] **A03 · Полный single-owner workflow.** Добавить пагинацию очереди уловов, кнопку запуска официального импорта и отображение результата/истории, безопасные статусы источников с возрастом данных, cooldown/backoff и ручное обновление очередей. Критерий: владелец может пройти путь «импорт → проверка → решение → история» без API/CLI; старые данные сохраняются при сбое импорта.
|
||||
- [ ] **A04 · Контур медиа-проверки.** Сделать admin-экран для просмотра approved/upgrade_queued медиа, исходника, размеров, производных и provenance; добавить approve/rollback только через существующие безопасные состояния. Критерий: ни одна публичная замена не происходит без явного решения и проверяемого manifest-а.
|
||||
- [x] **A04 · Контур медиа-проверки.** Admin-экран показывает approved/upgrade_queued/upgrade_stored медиа, источник, размеры, производные и provenance. Защищённые действия требуют непустую причину: publish атомарно продвигает только проверенные `upgrade_stored` и сохраняет fallback, rollback принимает только связанную пару `approved`/`superseded`; небезопасные состояния возвращают `409`. API-тесты проверяют auth, валидацию причины и вызов безопасных операций; browser acceptance остаётся в A06.
|
||||
- [ ] **A05 · Многопользовательский доступ.** После пилота заменить общий Bearer-токен персональными аккаунтами и короткими серверными сессиями с отзывом, ролями read-only/moderator/importer/owner, operator ID в аудите и журналом входов. Критерий: минимальные права реально ограничивают действия, logout/revoke инвалидируют сессию на сервере.
|
||||
- [ ] **A06 · Browser/accessibility acceptance.** Проверить `/admin`, moderation и external sources в 320/390/768/1280 px: клавиатура, focus order, screen reader labels, reduced motion, forced colors, темы, ошибки/пустые очереди и конфликт `409`. Критерий: Playwright + axe без блокирующих дефектов и ручная визуальная проверка.
|
||||
- [ ] **A07 · Production gate.** Выполнить preflight с реальными секретами, проверить Caddy Basic + API Bearer, закрытые внутренние порты, backup/restore PostgreSQL и MinIO, readiness, no-store и внешний smoke после деплоя. Критерий: acceptance-runbook пройден, rollback и процедура отзыва доступа документированы.
|
||||
|
||||
@@ -16,6 +16,7 @@ from urllib.request import Request, urlopen, HTTPRedirectHandler, build_opener
|
||||
from .community_sources import (
|
||||
parse_rf4db_catches,
|
||||
parse_rf4db_waterbodies,
|
||||
parse_rf4db_waterbody_detail,
|
||||
parse_rf4map_point,
|
||||
parse_rf4posts_spot,
|
||||
parse_rf4stat_fishing,
|
||||
@@ -30,6 +31,7 @@ SOURCES = {
|
||||
"rf4stat-posts": ("https://rf4-stat.ru/posts/", parse_rf4stat_posts),
|
||||
}
|
||||
DETAIL_SOURCES = {
|
||||
"rf4db-waterbody": parse_rf4db_waterbody_detail,
|
||||
"rf4map-point": parse_rf4map_point,
|
||||
"rf4posts-spot": parse_rf4posts_spot,
|
||||
}
|
||||
|
||||
@@ -35,6 +35,7 @@ class ExternalCatch:
|
||||
clip: str | None
|
||||
fishing_style: str | None
|
||||
evidence_urls: tuple[str, ...]
|
||||
coordinate_raw: str | None = None
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
@@ -67,10 +68,29 @@ class RF4DBWaterbody:
|
||||
image_url: str | None
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class RF4DBWaterbodyDetail:
|
||||
source_system: str
|
||||
source_external_id: str
|
||||
source_url: str
|
||||
name: str
|
||||
description: str | None
|
||||
aliases: tuple[str, ...]
|
||||
fish_species: tuple[str, ...]
|
||||
fish_external_ids: tuple[str | None, ...]
|
||||
image_urls: tuple[str, ...]
|
||||
point_urls: tuple[str, ...]
|
||||
|
||||
|
||||
def _text(node: Tag | None) -> str:
|
||||
return " ".join(node.get_text(" ", strip=True).split()) if node else ""
|
||||
|
||||
|
||||
def _fish_name(node: Tag) -> str:
|
||||
"""Read a fish label while dropping the optional trophy weight suffix."""
|
||||
return re.sub(r"\s*\d+(?:[.,]\d+)?\s*(?:кг|kg|г|g)\s*$", "", _text(node), flags=re.I).strip()
|
||||
|
||||
|
||||
def _key(href: str | None) -> str | None:
|
||||
if not href:
|
||||
return None
|
||||
@@ -189,6 +209,63 @@ def parse_rf4db_waterbodies(
|
||||
return result
|
||||
|
||||
|
||||
def parse_rf4db_waterbody_detail(
|
||||
html: str, *, source_url: str,
|
||||
) -> RF4DBWaterbodyDetail:
|
||||
"""Parse one RF4DB waterbody page without assigning media or coordinates.
|
||||
|
||||
The detail page is accepted only when its localized heading and fish list
|
||||
are present. Images and point links remain source candidates; review and
|
||||
canonical crosswalks happen in later pipeline stages.
|
||||
"""
|
||||
soup = BeautifulSoup(html, "html.parser")
|
||||
root = soup.select_one("article.waterbody-detail, main[data-waterbody-detail]") or soup
|
||||
external_id = _key(source_url)
|
||||
name = _text(root.select_one("h1"))
|
||||
fish_nodes = root.select(".waterbody-fish a[href], [data-fish-list] a[href], a[href*='/fishes/']")
|
||||
if not fish_nodes:
|
||||
fish_nodes = root.select(".fish-list a[href], ul.fish a[href]")
|
||||
fish_species: list[str] = []
|
||||
fish_external_ids: list[str | None] = []
|
||||
seen_fish: set[str] = set()
|
||||
for node in fish_nodes:
|
||||
fish_name = _fish_name(node)
|
||||
fish_id = _key(node.get("href"))
|
||||
identity = fish_id or fish_name.casefold()
|
||||
if not fish_name or identity in seen_fish:
|
||||
continue
|
||||
fish_species.append(fish_name)
|
||||
fish_external_ids.append(fish_id)
|
||||
seen_fish.add(identity)
|
||||
if not external_id or not name or not fish_species:
|
||||
raise CommunityParseError("RF4DB waterbody detail not found or incomplete")
|
||||
|
||||
description_node = root.select_one("[data-description], .waterbody-description, .description")
|
||||
description = _text(description_node) or None
|
||||
aliases = tuple(dict.fromkeys(
|
||||
_text(node) for node in root.select("[data-alias], .waterbody-aliases li, .aliases li")
|
||||
if _text(node) and _text(node) != name
|
||||
))
|
||||
image_urls = tuple(dict.fromkeys(
|
||||
urljoin(source_url, str(node.get("src") or node.get("data-src")))
|
||||
for node in root.select("img[src], img[data-src]")
|
||||
if (node.get("src") or node.get("data-src"))
|
||||
and "/fish/" not in str(node.get("src") or node.get("data-src"))
|
||||
))
|
||||
point_urls = tuple(dict.fromkeys(
|
||||
urljoin(source_url, str(node.get("href")))
|
||||
for node in root.select('a[href*="/spots/"], a[href*="/points/"]')
|
||||
if node.get("href")
|
||||
))
|
||||
return RF4DBWaterbodyDetail(
|
||||
source_system="rf4db", source_external_id=external_id,
|
||||
source_url=source_url, name=name, description=description,
|
||||
aliases=aliases, fish_species=tuple(fish_species),
|
||||
fish_external_ids=tuple(fish_external_ids), image_urls=image_urls,
|
||||
point_urls=point_urls,
|
||||
)
|
||||
|
||||
|
||||
def parse_rf4db_catches(html: str, *, base_url: str = "https://rf4db.com") -> list[ExternalCatch]:
|
||||
soup = BeautifulSoup(html, "html.parser")
|
||||
result: list[ExternalCatch] = []
|
||||
@@ -201,7 +278,8 @@ def parse_rf4db_catches(html: str, *, base_url: str = "https://rf4db.com") -> li
|
||||
external_id = _key(detail.get("href"))
|
||||
if not external_id:
|
||||
continue
|
||||
x, y = _coordinates(_text(card.select_one(".catch-card__place b")))
|
||||
coordinate_raw = _text(card.select_one(".catch-card__place b")) or None
|
||||
x, y = _coordinates(coordinate_raw or "")
|
||||
bait_link = card.select_one('.catch-card__place a[href*="/wiki/baits/"]')
|
||||
badges = card.select(".catch-badge")
|
||||
weather = next((_text(b.select_one("b")) for b in badges if _text(b).startswith("Погода")), None)
|
||||
@@ -218,7 +296,7 @@ def parse_rf4db_catches(html: str, *, base_url: str = "https://rf4db.com") -> li
|
||||
game_time=_game_time(_text(card.select_one(".catch-card__place"))),
|
||||
published_at=None, player_name=None, weather=weather or None,
|
||||
water_temperature_c=float(temperature_match.group().replace(",", ".")) if temperature_match else None,
|
||||
clip=None, fishing_style=None, evidence_urls=(),
|
||||
clip=None, fishing_style=None, evidence_urls=(), coordinate_raw=coordinate_raw,
|
||||
))
|
||||
if not result:
|
||||
raise CommunityParseError("RF4DB catch cards not found")
|
||||
@@ -288,6 +366,7 @@ def parse_rf4stat_fishing(
|
||||
if not external_id:
|
||||
continue
|
||||
position = row.select_one(".list-position .position:not(.position-locked)")
|
||||
coordinate_raw = _text(row.select_one(".list-position .position")) or None
|
||||
x, y = _coordinates(_text(position))
|
||||
style = row.select_one(".post-style-icon[title]")
|
||||
style_text = str(style.get("title", "")) if style else ""
|
||||
@@ -305,6 +384,7 @@ def parse_rf4stat_fishing(
|
||||
clip=_text(row.select_one(".clip")) or None,
|
||||
fishing_style=style_text.removeprefix("Вид ловли:").strip() or None,
|
||||
evidence_urls=(urljoin(base_url, str(row.select_one("a.share.hide-print").get("href"))),) if row.select_one("a.share.hide-print") else (),
|
||||
coordinate_raw=coordinate_raw,
|
||||
))
|
||||
if not result:
|
||||
raise CommunityParseError("RF4-STAT fishing rows not found")
|
||||
@@ -323,6 +403,7 @@ def parse_rf4stat_posts(
|
||||
published_raw = str(post.get("data-published-at", ""))
|
||||
published_at = datetime.fromtimestamp(int(published_raw), tz=timezone.utc) if published_raw.isdigit() else None
|
||||
position = post.select_one(".spot-col .position:not(.position-locked)")
|
||||
coordinate_raw = _text(post.select_one(".spot-col .position")) or None
|
||||
x, y = _coordinates(_text(position))
|
||||
style = post.select_one(".post-style-icon[title]")
|
||||
style_text = str(style.get("title", "")) if style else ""
|
||||
@@ -344,7 +425,7 @@ def parse_rf4stat_posts(
|
||||
weather=None, water_temperature_c=None,
|
||||
clip=_text(post.select_one(".clip")) or None,
|
||||
fishing_style=style_text.removeprefix("Вид ловли:").strip() or None,
|
||||
evidence_urls=evidence,
|
||||
evidence_urls=evidence, coordinate_raw=coordinate_raw,
|
||||
))
|
||||
if not result:
|
||||
raise CommunityParseError("RF4-STAT posts not found")
|
||||
@@ -386,6 +467,7 @@ def parse_rf4map_point(html: str, *, source_url: str) -> list[ExternalCatch]:
|
||||
clip=str(item["clip"]) if isinstance(item.get("clip"), (int, float)) else None,
|
||||
fishing_style=None,
|
||||
evidence_urls=tuple(url for url in evidence or [] if isinstance(url, str)),
|
||||
coordinate_raw=f"{item['positionX']}:{item['positionY']}" if isinstance(item.get("positionX"), int) and isinstance(item.get("positionY"), int) else None,
|
||||
))
|
||||
if not result:
|
||||
raise CommunityParseError("RF4MAP point observations not found")
|
||||
@@ -428,7 +510,7 @@ def parse_rf4posts_spot(html: str, *, source_url: str) -> list[ExternalCatch]:
|
||||
weather=None, water_temperature_c=None,
|
||||
clip=str(spot["clip"]) if isinstance(spot.get("clip"), (int, float)) else None,
|
||||
fishing_style=str(spot["tackleType"]) if isinstance(spot.get("tackleType"), str) else None,
|
||||
evidence_urls=evidence,
|
||||
evidence_urls=evidence, coordinate_raw=str(spot["coordinates"]) if isinstance(spot.get("coordinates"), str) else None,
|
||||
))
|
||||
if not result:
|
||||
raise CommunityParseError("RF4 Posts fish species not found")
|
||||
|
||||
@@ -11,9 +11,13 @@ from pathlib import Path
|
||||
from urllib.parse import urljoin, urlsplit
|
||||
|
||||
from bs4 import BeautifulSoup, Tag
|
||||
from PIL import Image, UnidentifiedImageError
|
||||
from PIL import Image, ImageDraw, ImageFont, UnidentifiedImageError
|
||||
|
||||
MAX_IMAGE_PIXELS = 40_000_000
|
||||
DERIVATIVE_TARGETS = {"card": 256, "detail": 1024}
|
||||
WATERBODY_MEDIA_ROLES = frozenset({
|
||||
"waterbody_cover", "waterbody_map", "waterbody_depth_map", "waterbody_screenshot",
|
||||
})
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
@@ -229,6 +233,29 @@ def publish_quality_upgrades(path: Path, *, note: str, minimum_dimension: int =
|
||||
return {"published": len(candidates), "retained_fallbacks": len(candidates)}
|
||||
|
||||
|
||||
def rollback_quality_upgrade(path: Path, *, asset_url: str, note: str) -> dict:
|
||||
"""Restore the superseded fallback for one explicitly selected upgrade."""
|
||||
manifest = json.loads(path.read_text(encoding="utf-8"))
|
||||
assets = manifest.get("assets", [])
|
||||
candidate = next((item for item in assets if item.get("asset_url") == asset_url), None)
|
||||
if not candidate or candidate.get("status") != "approved" or not candidate.get("supersedes"):
|
||||
raise ValueError("rollback requires an approved upgrade with a supersedes link")
|
||||
fallback = next((item for item in assets if item.get("asset_url") == candidate["supersedes"]), None)
|
||||
if not fallback or fallback.get("status") != "superseded" or fallback.get("replaced_by") != asset_url:
|
||||
raise ValueError("rollback requires the matching superseded fallback")
|
||||
|
||||
reviewed_at = datetime.now(timezone.utc).isoformat()
|
||||
candidate.update({"status": "upgrade_stored", "reviewed_at": reviewed_at, "review_note": note})
|
||||
candidate.pop("supersedes", None)
|
||||
fallback.update({"status": "approved", "reviewed_at": reviewed_at, "review_note": note})
|
||||
fallback.pop("replaced_by", None)
|
||||
manifest["updated_at"] = reviewed_at
|
||||
temporary = path.with_suffix(path.suffix + ".tmp")
|
||||
temporary.write_text(json.dumps(manifest, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||
temporary.replace(path)
|
||||
return {"rolled_back": asset_url, "restored": fallback["asset_url"]}
|
||||
|
||||
|
||||
def reclassify_manifest(path: Path) -> dict:
|
||||
manifest = json.loads(path.read_text(encoding="utf-8"))
|
||||
for item in manifest.get("assets", []):
|
||||
@@ -241,6 +268,7 @@ def reclassify_manifest(path: Path) -> dict:
|
||||
def review_asset(
|
||||
path: Path, *, asset_url: str, decision: str, entity_type: str | None = None,
|
||||
entity_key: str | None = None, note: str | None = None,
|
||||
media_role: str | None = None,
|
||||
) -> dict:
|
||||
if decision not in {"approved", "rejected"}:
|
||||
raise ValueError("decision must be approved or rejected")
|
||||
@@ -248,12 +276,16 @@ def review_asset(
|
||||
item = next((asset for asset in manifest.get("assets", []) if asset["asset_url"] == asset_url), None)
|
||||
if item is None:
|
||||
raise ValueError("asset URL is not present in manifest")
|
||||
if media_role is not None and (entity_type != "waterbody" or media_role not in WATERBODY_MEDIA_ROLES):
|
||||
raise ValueError("media role is only valid for a waterbody and must be a known waterbody role")
|
||||
if decision == "approved":
|
||||
if item.get("status") != "stored":
|
||||
raise ValueError("only a stored asset can be approved")
|
||||
if entity_type not in {"fish", "waterbody", "tackle", "reference"} or not entity_key:
|
||||
raise ValueError("approved asset requires entity type and canonical key")
|
||||
item.update({"entity_type": entity_type, "entity_key": entity_key})
|
||||
if media_role is not None:
|
||||
item["media_role"] = media_role
|
||||
item.update({
|
||||
"status": decision,
|
||||
"reviewed_at": datetime.now(timezone.utc).isoformat(),
|
||||
@@ -290,6 +322,27 @@ def audit_media_catalog(root: Path) -> dict:
|
||||
issues.append(f"{item['asset_url']}: image metadata mismatch")
|
||||
except ValueError as exc:
|
||||
issues.append(f"{item['asset_url']}: {exc}")
|
||||
for variant in item.get("derivatives", []):
|
||||
variant_path = variant.get("local_path")
|
||||
if not isinstance(variant_path, str):
|
||||
issues.append(f"{item['asset_url']}: derivative has no local_path")
|
||||
continue
|
||||
referenced.add(variant_path)
|
||||
target = root / variant_path
|
||||
if not target.is_file():
|
||||
issues.append(f"{item['asset_url']}: derivative file is missing")
|
||||
continue
|
||||
body = target.read_bytes()
|
||||
if hashlib.sha256(body).hexdigest() != variant.get("sha256"):
|
||||
issues.append(f"{item['asset_url']}: derivative SHA-256 mismatch")
|
||||
try:
|
||||
width, height, mime = inspect_image(body)
|
||||
if (width, height, mime) != (variant.get("width"), variant.get("height"), variant.get("content_type")):
|
||||
issues.append(f"{item['asset_url']}: derivative image metadata mismatch")
|
||||
if len(body) != variant.get("bytes"):
|
||||
issues.append(f"{item['asset_url']}: derivative byte count mismatch")
|
||||
except ValueError as exc:
|
||||
issues.append(f"{item['asset_url']}: derivative {exc}")
|
||||
if status == "approved" and (not item.get("entity_key") or item.get("entity_type") not in {"fish", "waterbody", "tackle", "reference"}):
|
||||
issues.append(f"{item['asset_url']}: approved asset has no valid canonical mapping")
|
||||
files_root = root / "files"
|
||||
@@ -416,6 +469,70 @@ def compare_quality_upgrades(root: Path, *, minimum_dimension: int = 256) -> dic
|
||||
}
|
||||
|
||||
|
||||
def generate_quality_contact_sheets(
|
||||
root: Path, output_dir: Path, *, batch_size: int = 20,
|
||||
) -> dict:
|
||||
"""Create offline old/selected contact sheets for manually reviewed upgrades."""
|
||||
if batch_size < 1:
|
||||
raise ValueError("batch_size must be positive")
|
||||
manifest = json.loads((root / "manifest.json").read_text(encoding="utf-8"))
|
||||
assets = manifest.get("assets", [])
|
||||
by_url = {item.get("asset_url"): item for item in assets}
|
||||
pairs = []
|
||||
issues: list[str] = []
|
||||
for selected in assets:
|
||||
if selected.get("status") != "approved" or not selected.get("supersedes"):
|
||||
continue
|
||||
old = by_url.get(selected["supersedes"])
|
||||
if not old:
|
||||
issues.append(f"{selected.get('asset_url')}: superseded asset is missing")
|
||||
continue
|
||||
old_path, selected_path = root / str(old.get("local_path") or ""), root / str(selected.get("local_path") or "")
|
||||
if not old_path.is_file() or not selected_path.is_file():
|
||||
issues.append(f"{selected.get('asset_url')}: contact-sheet image is missing")
|
||||
continue
|
||||
pairs.append((old, selected))
|
||||
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
sheets = []
|
||||
cell_width, cell_height, image_size = 320, 220, 180
|
||||
font_paths = (
|
||||
Path("/usr/share/fonts/Fonts/DejaVuSans.ttf"),
|
||||
Path("/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf"),
|
||||
Path("/usr/share/fonts/dejavu/DejaVuSans.ttf"),
|
||||
)
|
||||
font_path = next((path for path in font_paths if path.is_file()), None)
|
||||
font = ImageFont.truetype(font_path, 12) if font_path else ImageFont.load_default()
|
||||
for sheet_index in range(0, len(pairs), batch_size):
|
||||
chunk = pairs[sheet_index:sheet_index + batch_size]
|
||||
canvas = Image.new("RGB", (cell_width * 2, cell_height * len(chunk)), "white")
|
||||
draw = ImageDraw.Draw(canvas)
|
||||
metadata = []
|
||||
for row, (old, selected) in enumerate(chunk):
|
||||
y = row * cell_height
|
||||
for column, item in enumerate((old, selected)):
|
||||
path = root / str(item["local_path"])
|
||||
with Image.open(path) as image:
|
||||
preview = image.convert("RGBA")
|
||||
preview.thumbnail((image_size, image_size), Image.Resampling.LANCZOS)
|
||||
x = column * cell_width + (image_size - preview.width) // 2
|
||||
canvas.paste(preview, (x, y + 4), preview if preview.mode == "RGBA" else None)
|
||||
title = "СТАРЫЙ" if column == 0 else "ВЫБРАННЫЙ"
|
||||
label = str(item.get("label") or "Без подписи")[:34]
|
||||
source = urlsplit(str(item.get("source_page") or "")).hostname or "unknown"
|
||||
draw.text((column * cell_width + 190, y + 8), f"{title}: {label}", fill="black", font=font)
|
||||
draw.text((column * cell_width + 190, y + 30), f"{item.get('width')}×{item.get('height')} · {source}", fill="black", font=font)
|
||||
draw.text((column * cell_width + 190, y + 52), str(item.get("asset_url") or "")[:42], fill="gray", font=font)
|
||||
draw.line((0, y + cell_height - 1, cell_width * 2, y + cell_height - 1), fill="#cccccc")
|
||||
metadata.append({"label": selected.get("label"), "old_url": old.get("asset_url"), "selected_url": selected.get("asset_url")})
|
||||
image_path = output_dir / f"quality-upgrades-{sheet_index // batch_size + 1:03d}.png"
|
||||
json_path = image_path.with_suffix(".json")
|
||||
canvas.save(image_path, format="PNG", optimize=True)
|
||||
json_path.write_text(json.dumps(metadata, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||
sheets.append(str(image_path))
|
||||
return {"pairs": len(pairs), "sheets": sheets, "issues": issues}
|
||||
|
||||
|
||||
def inspect_image(body: bytes) -> tuple[int, int, str]:
|
||||
try:
|
||||
with Image.open(io.BytesIO(body)) as image:
|
||||
@@ -427,7 +544,7 @@ def inspect_image(body: bytes) -> tuple[int, int, str]:
|
||||
if width < 1 or height < 1 or width * height > MAX_IMAGE_PIXELS:
|
||||
raise ValueError("asset dimensions are outside safe limits")
|
||||
mime = Image.MIME.get(image_format or "")
|
||||
if mime not in {"image/jpeg", "image/png", "image/webp", "image/gif"}:
|
||||
if mime not in {"image/jpeg", "image/png", "image/webp", "image/gif", "image/avif"}:
|
||||
raise ValueError(f"unsupported image format: {image_format}")
|
||||
return width, height, mime
|
||||
|
||||
@@ -445,3 +562,69 @@ def store_asset(root: Path, body: bytes, *, content_type: str, source_url: str)
|
||||
if not target.exists():
|
||||
target.write_bytes(body)
|
||||
return digest, str(target.relative_to(root)), width, height, detected_mime
|
||||
|
||||
|
||||
def generate_media_derivatives(
|
||||
root: Path, *, targets: dict[str, int] | None = None,
|
||||
) -> dict:
|
||||
"""Generate deterministic WebP/AVIF derivatives for approved local assets.
|
||||
|
||||
Derivatives are content-addressed like originals and never upscale a source
|
||||
image. The approved original remains the fallback and is not rewritten.
|
||||
"""
|
||||
manifest_path = root / "manifest.json"
|
||||
manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
|
||||
selected_targets = targets or DERIVATIVE_TARGETS
|
||||
generated = skipped = 0
|
||||
issues: list[str] = []
|
||||
|
||||
for item in manifest.get("assets", []):
|
||||
if item.get("status") != "approved" or not item.get("local_path"):
|
||||
continue
|
||||
source_path = root / str(item["local_path"])
|
||||
if not source_path.is_file():
|
||||
issues.append(f"{item.get('asset_url')}: derivative source is missing")
|
||||
continue
|
||||
try:
|
||||
with Image.open(source_path) as source:
|
||||
source.load()
|
||||
source_width, source_height = source.size
|
||||
source_format = source.format
|
||||
if source_format not in {"JPEG", "PNG", "WEBP", "GIF"}:
|
||||
raise ValueError(f"unsupported source format: {source_format}")
|
||||
variants: list[dict] = []
|
||||
for role, target_width in selected_targets.items():
|
||||
width = min(source_width, int(target_width))
|
||||
height = max(1, round(source_height * width / source_width))
|
||||
resized = source if width == source_width else source.resize((width, height), Image.Resampling.LANCZOS)
|
||||
for image_format, mime, extension, save_options in (
|
||||
("WEBP", "image/webp", ".webp", {"quality": 85, "method": 6}),
|
||||
("AVIF", "image/avif", ".avif", {"quality": 80}),
|
||||
):
|
||||
output = io.BytesIO()
|
||||
if resized.mode not in {"RGB", "RGBA", "L", "LA"}:
|
||||
converted = resized.convert("RGBA" if "A" in resized.mode else "RGB")
|
||||
else:
|
||||
converted = resized
|
||||
converted.save(output, format=image_format, **save_options)
|
||||
body = output.getvalue()
|
||||
digest = hashlib.sha256(body).hexdigest()
|
||||
target = root / "files" / digest[:2] / f"{digest}{extension}"
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
if not target.exists():
|
||||
target.write_bytes(body)
|
||||
generated += 1
|
||||
else:
|
||||
skipped += 1
|
||||
variants.append({
|
||||
"role": role, "format": image_format.lower(), "sha256": digest,
|
||||
"local_path": str(target.relative_to(root)), "content_type": mime,
|
||||
"bytes": len(body), "width": width, "height": height,
|
||||
})
|
||||
item["derivatives"] = variants
|
||||
except (OSError, ValueError) as exc:
|
||||
issues.append(f"{item.get('asset_url')}: {exc}")
|
||||
|
||||
manifest["updated_at"] = datetime.now(timezone.utc).isoformat()
|
||||
manifest_path.write_text(json.dumps(manifest, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||
return {"approved_assets": sum(item.get("status") == "approved" for item in manifest.get("assets", [])), "generated": generated, "already_present": skipped, "issues": issues}
|
||||
|
||||
@@ -8,7 +8,7 @@ import urllib.error
|
||||
import urllib.request
|
||||
|
||||
from .community_cli import MIN_FETCH_INTERVAL_SECONDS, USER_AGENT, _StrictRedirectHandler, _read_state, _validate_url_before_io, check_and_reserve, fetch_html, fetch_site_key
|
||||
from .media_assets import approve_stored_assets, audit_media_catalog, compare_quality_upgrades, extract_media_candidates, media_coverage, media_quality_report, merge_manifest, publish_quality_upgrades, queue_quality_upgrades, reconcile_queued_duplicates, reclassify_manifest, review_asset, store_asset
|
||||
from .media_assets import approve_stored_assets, audit_media_catalog, compare_quality_upgrades, extract_media_candidates, generate_media_derivatives, generate_quality_contact_sheets, media_coverage, media_quality_report, merge_manifest, publish_quality_upgrades, queue_quality_upgrades, reconcile_queued_duplicates, reclassify_manifest, review_asset, rollback_quality_upgrade, store_asset
|
||||
|
||||
|
||||
DEFAULT_ROOT = Path("data/media")
|
||||
@@ -187,6 +187,7 @@ def main(argv: list[str] | None = None) -> int:
|
||||
parser.add_argument("--queue-quality-upgrades", action="store_true", help="Queue alternatives to published fish below the minimum resolution")
|
||||
parser.add_argument("--approve-stored", action="store_true", help="Publish all stored assets after explicit owner approval")
|
||||
parser.add_argument("--publish-upgrades", action="store_true", help="Atomically publish all stored quality upgrades and retain fallbacks")
|
||||
parser.add_argument("--rollback-upgrade", action="store_true", help="Restore one superseded fallback after an explicit review decision")
|
||||
parser.add_argument("--review-url", help="Review an asset already present in the manifest")
|
||||
parser.add_argument("--decision", choices=("approved", "rejected"))
|
||||
parser.add_argument("--entity-type", choices=("fish", "waterbody", "tackle", "reference"))
|
||||
@@ -197,6 +198,8 @@ def main(argv: list[str] | None = None) -> int:
|
||||
parser.add_argument("--quality-report", action="store_true", help="Report low-resolution published fish and known alternatives without network access")
|
||||
parser.add_argument("--compare-quality-upgrades", action="store_true", help="Compare stored quality candidates with their published fallbacks")
|
||||
parser.add_argument("--queue-plan", action="store_true", help="Show the next useful queued asset per domain without network access")
|
||||
parser.add_argument("--generate-derivatives", action="store_true", help="Generate deterministic WebP/AVIF variants for approved local assets")
|
||||
parser.add_argument("--contact-sheet-dir", type=Path, help="Write offline quality-upgrade contact sheets to this directory")
|
||||
parser.add_argument("--root", type=Path, default=DEFAULT_ROOT)
|
||||
parser.add_argument("--state-file", type=Path, default=Path(".cache/community-fetch-state.json"))
|
||||
args = parser.parse_args(argv)
|
||||
@@ -218,6 +221,14 @@ def main(argv: list[str] | None = None) -> int:
|
||||
if args.queue_plan:
|
||||
print(json.dumps(media_queue_plan(args.root, args.state_file), ensure_ascii=False, indent=2))
|
||||
return 0
|
||||
if args.generate_derivatives:
|
||||
report = generate_media_derivatives(args.root)
|
||||
print(json.dumps(report, ensure_ascii=False, indent=2))
|
||||
return 1 if report["issues"] else 0
|
||||
if args.contact_sheet_dir:
|
||||
report = generate_quality_contact_sheets(args.root, args.contact_sheet_dir)
|
||||
print(json.dumps(report, ensure_ascii=False, indent=2))
|
||||
return 1 if report["issues"] else 0
|
||||
if args.review_url:
|
||||
if not args.decision:
|
||||
parser.error("--decision is required with --review-url")
|
||||
@@ -244,6 +255,11 @@ def main(argv: list[str] | None = None) -> int:
|
||||
parser.error("--note is required with --publish-upgrades")
|
||||
print(json.dumps(publish_quality_upgrades(args.root / "manifest.json", note=args.note), ensure_ascii=False, indent=2))
|
||||
return 0
|
||||
if args.rollback_upgrade:
|
||||
if not args.asset_url or not args.note:
|
||||
parser.error("--asset-url and --note are required with --rollback-upgrade")
|
||||
print(json.dumps(rollback_quality_upgrade(args.root / "manifest.json", asset_url=args.asset_url, note=args.note), ensure_ascii=False, indent=2))
|
||||
return 0
|
||||
if args.download_one:
|
||||
print(_download_one(args.root, args.state_file, args.asset_url))
|
||||
return 0
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
"""Conservative, offline crosswalk suggestions for waterbody identities."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
from .media_assets import normalize_entity_label
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class CanonicalWaterbody:
|
||||
key: str
|
||||
name: str
|
||||
aliases: tuple[str, ...] = ()
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class WaterbodyIdentity:
|
||||
source_system: str
|
||||
external_id: str
|
||||
name: str
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class CrosswalkSuggestion:
|
||||
source_system: str
|
||||
external_id: str
|
||||
external_name: str
|
||||
status: str
|
||||
canonical_keys: tuple[str, ...]
|
||||
|
||||
|
||||
def suggest_waterbody_crosswalk(
|
||||
canonical: list[CanonicalWaterbody], identities: list[WaterbodyIdentity],
|
||||
) -> list[CrosswalkSuggestion]:
|
||||
"""Suggest only unique exact normalized-name matches.
|
||||
|
||||
``ambiguous`` and ``unmatched`` rows intentionally have no selected key;
|
||||
callers must not turn this report into aliases without human review.
|
||||
"""
|
||||
by_name: dict[str, set[str]] = {}
|
||||
for item in canonical:
|
||||
for name in (item.name, *item.aliases):
|
||||
by_name.setdefault(normalize_entity_label(name), set()).add(item.key)
|
||||
result: list[CrosswalkSuggestion] = []
|
||||
for identity in identities:
|
||||
keys = tuple(sorted(by_name.get(normalize_entity_label(identity.name), set())))
|
||||
status = "exact" if len(keys) == 1 else "ambiguous" if keys else "unmatched"
|
||||
result.append(CrosswalkSuggestion(
|
||||
source_system=identity.source_system, external_id=identity.external_id,
|
||||
external_name=identity.name, status=status,
|
||||
canonical_keys=keys if status == "exact" else (),
|
||||
))
|
||||
return result
|
||||
|
||||
|
||||
def missing_external_ids(expected: set[str], observed: set[str]) -> tuple[str, ...]:
|
||||
"""Return stable missing IDs without interpreting absence as deletion."""
|
||||
return tuple(sorted(expected - observed))
|
||||
@@ -0,0 +1,17 @@
|
||||
<!doctype html>
|
||||
<html lang="ru">
|
||||
<body>
|
||||
<main>
|
||||
<article>
|
||||
<h1>Дачный пруд</h1>
|
||||
<img src="https://oss.rf4db.com/game/maps/level_000_home.png" alt="Дачный пруд">
|
||||
<img src="https://oss.rf4db.com/game/paper_maps_webp/map_level_000_home.webp" alt="Карта: Дачный пруд">
|
||||
<img src="https://oss.rf4db.com/game/fish/frog.webp" alt="Лягушка">
|
||||
<a href="/ru/fishes/frog">Лягушка 190 г</a>
|
||||
<a href="/ru/fishes/a.sleeper">Ротан 400 г</a>
|
||||
<a href="/ru/fishes/perch">Окунь 1.6 кг</a>
|
||||
<a href="/ru/positions/example-position">Точка 13:9</a>
|
||||
</article>
|
||||
</main>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,17 @@
|
||||
<!doctype html>
|
||||
<html lang="ru">
|
||||
<body>
|
||||
<article class="waterbody-detail">
|
||||
<h1>оз. Комариное</h1>
|
||||
<p class="waterbody-description">Большое озеро с каменистыми берегами.</p>
|
||||
<ul class="waterbody-aliases"><li>Комариное</li><li>оз. Комариное</li></ul>
|
||||
<div class="waterbody-fish">
|
||||
<a href="/ru/wiki/fish/pike">Щука</a>
|
||||
<a href="/ru/wiki/fish/perch">Окунь</a>
|
||||
</div>
|
||||
<img src="https://oss.rf4db.com/game/maps/level_001_mosquito.webp" alt="Карта">
|
||||
<img data-src="/game/maps/level_001_mosquito-depth.webp" alt="Глубины">
|
||||
<a href="/ru/maps/level_001_mosquito/spots/12-34">Точка 12:34</a>
|
||||
</article>
|
||||
</body>
|
||||
</html>
|
||||
@@ -8,11 +8,16 @@ from rf4_research.community_sources import (
|
||||
parse_rf4db_catches,
|
||||
parse_rf4db_detail,
|
||||
parse_rf4db_waterbodies,
|
||||
parse_rf4db_waterbody_detail,
|
||||
parse_rf4map_point,
|
||||
parse_rf4posts_spot,
|
||||
parse_rf4stat_fishing,
|
||||
parse_rf4stat_posts,
|
||||
)
|
||||
from rf4_research.waterbody_crosswalk import (
|
||||
CanonicalWaterbody, WaterbodyIdentity, missing_external_ids,
|
||||
suggest_waterbody_crosswalk,
|
||||
)
|
||||
|
||||
|
||||
FIXTURES = Path(__file__).parent / "fixtures"
|
||||
@@ -74,6 +79,65 @@ def test_rf4db_waterbody_catalog_rejects_partial_results() -> None:
|
||||
parse_rf4db_waterbodies(fixture("rf4db_waterbodies_sample.html"))
|
||||
|
||||
|
||||
def test_parses_rf4db_waterbody_detail_without_assigning_media_roles() -> None:
|
||||
row = parse_rf4db_waterbody_detail(
|
||||
fixture("rf4db_waterbody_detail_sample.html"),
|
||||
source_url="https://rf4db.com/ru/maps/level_001_mosquito",
|
||||
)
|
||||
|
||||
assert (row.source_external_id, row.name) == ("level_001_mosquito", "оз. Комариное")
|
||||
assert row.description == "Большое озеро с каменистыми берегами."
|
||||
assert row.aliases == ("Комариное",)
|
||||
assert row.fish_species == ("Щука", "Окунь")
|
||||
assert row.fish_external_ids == ("pike", "perch")
|
||||
assert row.image_urls == (
|
||||
"https://oss.rf4db.com/game/maps/level_001_mosquito.webp",
|
||||
"https://rf4db.com/game/maps/level_001_mosquito-depth.webp",
|
||||
)
|
||||
assert row.point_urls == ("https://rf4db.com/ru/maps/level_001_mosquito/spots/12-34",)
|
||||
|
||||
|
||||
def test_parses_live_waterbody_fish_links_without_fish_media() -> None:
|
||||
row = parse_rf4db_waterbody_detail(
|
||||
fixture("rf4db_waterbody_detail_live_sample.html"),
|
||||
source_url="https://rf4db.com/ru/maps/level_000_home",
|
||||
)
|
||||
assert row.fish_species == ("Лягушка", "Ротан", "Окунь")
|
||||
assert row.fish_external_ids == ("frog", "a.sleeper", "perch")
|
||||
assert row.image_urls == (
|
||||
"https://oss.rf4db.com/game/maps/level_000_home.png",
|
||||
"https://oss.rf4db.com/game/paper_maps_webp/map_level_000_home.webp",
|
||||
)
|
||||
|
||||
|
||||
def test_rf4db_waterbody_detail_rejects_missing_fish_list() -> None:
|
||||
with pytest.raises(CommunityParseError, match="detail not found or incomplete"):
|
||||
parse_rf4db_waterbody_detail(
|
||||
"<main><h1>Озеро</h1></main>",
|
||||
source_url="https://rf4db.com/ru/maps/level_001_mosquito",
|
||||
)
|
||||
|
||||
|
||||
def test_waterbody_crosswalk_only_suggests_unique_exact_matches() -> None:
|
||||
canonical = [
|
||||
CanonicalWaterbody("mosquito", "оз. Комариное", ("Комариное",)),
|
||||
CanonicalWaterbody("ambiguous-a", "Большое озеро"),
|
||||
CanonicalWaterbody("ambiguous-b", "Другое", ("Большое озеро",)),
|
||||
]
|
||||
rows = suggest_waterbody_crosswalk(canonical, [
|
||||
WaterbodyIdentity("rf4db", "level_001_mosquito", "КОМАРИНОЕ"),
|
||||
WaterbodyIdentity("rf4map", "16", "Большое озеро"),
|
||||
WaterbodyIdentity("rf4-posts", "lake-x", "Норвежское море"),
|
||||
])
|
||||
assert [(row.status, row.canonical_keys) for row in rows] == [
|
||||
("exact", ("mosquito",)), ("ambiguous", ()), ("unmatched", ()),
|
||||
]
|
||||
|
||||
|
||||
def test_missing_source_ids_are_diagnostic_not_withdrawals() -> None:
|
||||
assert missing_external_ids({"16", "17", "18"}, {"16"}) == ("17", "18")
|
||||
|
||||
|
||||
def test_parses_rf4stat_fishing_rows() -> None:
|
||||
row = parse_rf4stat_fishing(
|
||||
fixture("rf4stat_fishing_sample.html"),
|
||||
@@ -82,6 +146,7 @@ def test_parses_rf4stat_fishing_rows() -> None:
|
||||
|
||||
assert (row.source_system, row.source_external_id) == ("rf4stat-fishing", "123")
|
||||
assert (row.x, row.y, row.weight_g) == (71, 92, 11_584)
|
||||
assert row.coordinate_raw == "71:92"
|
||||
assert row.published_at == datetime(2026, 9, 3, 11, 27, tzinfo=timezone.utc)
|
||||
assert (row.player_name, row.clip, row.fishing_style) == ("Игрок", "35", "Донная")
|
||||
|
||||
@@ -91,6 +156,7 @@ def test_parses_rf4stat_posts_without_using_locked_coordinates() -> None:
|
||||
|
||||
assert (row.source_system, row.source_external_id) == ("rf4stat-post", "456:0")
|
||||
assert (row.x, row.y, row.weight_g) == (None, None, 4_321)
|
||||
assert row.coordinate_raw == "XX:XX"
|
||||
assert row.evidence_urls == ("https://img.example.test/proof.jpg",)
|
||||
|
||||
|
||||
|
||||
+110
-1
@@ -6,7 +6,7 @@ import io
|
||||
import pytest
|
||||
from PIL import Image
|
||||
|
||||
from rf4_research.media_assets import approve_stored_assets, audit_media_catalog, compare_quality_upgrades, extract_media_candidates, inspect_image, media_coverage, media_quality_report, merge_manifest, publish_quality_upgrades, queue_quality_upgrades, reconcile_queued_duplicates, review_asset, store_asset
|
||||
from rf4_research.media_assets import approve_stored_assets, audit_media_catalog, compare_quality_upgrades, extract_media_candidates, generate_media_derivatives, generate_quality_contact_sheets, inspect_image, media_coverage, media_quality_report, merge_manifest, publish_quality_upgrades, queue_quality_upgrades, reconcile_queued_duplicates, review_asset, rollback_quality_upgrade, store_asset
|
||||
|
||||
|
||||
def test_extracts_and_classifies_unique_https_media() -> None:
|
||||
@@ -81,6 +81,98 @@ def test_manifest_merges_and_binary_store_is_content_addressed(tmp_path: Path) -
|
||||
assert (width, height, mime) == (3, 2, "image/png")
|
||||
|
||||
|
||||
def test_derivatives_are_content_addressed_and_never_upscaled(tmp_path: Path) -> None:
|
||||
manifest = tmp_path / "manifest.json"
|
||||
item = extract_media_candidates('<img src="/fish/pike.png" alt="Щука">', source_page="https://example.test")[0]
|
||||
saved = merge_manifest(manifest, [item])
|
||||
image = io.BytesIO()
|
||||
Image.new("RGBA", (400, 200), (0, 100, 200, 180)).save(image, format="PNG")
|
||||
digest, relative, width, height, mime = store_asset(
|
||||
tmp_path, image.getvalue(), content_type="image/png", source_url=item.asset_url,
|
||||
)
|
||||
saved["assets"][0].update({
|
||||
"status": "approved", "entity_key": "fish:pike", "sha256": digest,
|
||||
"local_path": relative, "width": width, "height": height, "content_type": mime,
|
||||
})
|
||||
manifest.write_text(json.dumps(saved), encoding="utf-8")
|
||||
|
||||
report = generate_media_derivatives(tmp_path)
|
||||
assert report["issues"] == []
|
||||
assert report["generated"] == 4
|
||||
updated = json.loads(manifest.read_text(encoding="utf-8"))["assets"][0]
|
||||
variants = updated["derivatives"]
|
||||
assert {(item["role"], item["format"]) for item in variants} == {
|
||||
("card", "webp"), ("card", "avif"), ("detail", "webp"), ("detail", "avif"),
|
||||
}
|
||||
assert all(item["width"] <= 400 and item["height"] <= 200 for item in variants)
|
||||
assert audit_media_catalog(tmp_path)["issues"] == []
|
||||
|
||||
|
||||
def test_quality_contact_sheet_contains_review_pairs(tmp_path: Path) -> None:
|
||||
old = tmp_path / "old.png"
|
||||
selected = tmp_path / "selected.png"
|
||||
Image.new("RGBA", (48, 48), "red").save(old)
|
||||
Image.new("RGBA", (256, 256), "blue").save(selected)
|
||||
manifest = {
|
||||
"assets": [
|
||||
{"asset_url": "https://old.test/fish.png", "status": "superseded", "label": "Щука", "local_path": "old.png", "width": 48, "height": 48},
|
||||
{"asset_url": "https://new.test/fish.webp", "status": "approved", "label": "Щука", "local_path": "selected.png", "width": 256, "height": 256, "supersedes": "https://old.test/fish.png"},
|
||||
],
|
||||
}
|
||||
(tmp_path / "manifest.json").write_text(json.dumps(manifest), encoding="utf-8")
|
||||
report = generate_quality_contact_sheets(tmp_path, tmp_path / "sheets")
|
||||
assert report["pairs"] == 1
|
||||
assert report["issues"] == []
|
||||
assert (tmp_path / "sheets" / "quality-upgrades-001.png").is_file()
|
||||
assert json.loads((tmp_path / "sheets" / "quality-upgrades-001.json").read_text()) [0]["selected_url"] == "https://new.test/fish.webp"
|
||||
|
||||
|
||||
def test_waterbody_media_role_is_reviewed_and_never_inferred(tmp_path: Path) -> None:
|
||||
manifest = tmp_path / "manifest.json"
|
||||
candidate = extract_media_candidates(
|
||||
'<img src="/maps/kuori.webp" alt="Куори">',
|
||||
source_page="https://rf4db.com/ru/maps/level_001_kuori",
|
||||
)[0]
|
||||
merge_manifest(manifest, [candidate])
|
||||
with pytest.raises(ValueError, match="only a stored asset"):
|
||||
review_asset(
|
||||
manifest, asset_url=candidate.asset_url, decision="approved",
|
||||
entity_type="waterbody", entity_key="kuori", media_role="waterbody_map",
|
||||
)
|
||||
|
||||
data = io.BytesIO()
|
||||
Image.new("RGB", (256, 256), "blue").save(data, format="WEBP")
|
||||
digest, local_path, width, height, mime = store_asset(
|
||||
tmp_path, data.getvalue(), content_type="image/webp", source_url=candidate.asset_url,
|
||||
)
|
||||
saved = json.loads(manifest.read_text(encoding="utf-8"))
|
||||
saved["assets"][0].update({
|
||||
"status": "stored", "sha256": digest, "local_path": local_path,
|
||||
"width": width, "height": height, "content_type": mime,
|
||||
})
|
||||
manifest.write_text(json.dumps(saved), encoding="utf-8")
|
||||
reviewed = review_asset(
|
||||
manifest, asset_url=candidate.asset_url, decision="approved",
|
||||
entity_type="waterbody", entity_key="kuori", media_role="waterbody_map",
|
||||
note="manual contact-sheet review",
|
||||
)
|
||||
assert reviewed["media_role"] == "waterbody_map"
|
||||
|
||||
|
||||
def test_waterbody_media_role_rejects_unknown_role(tmp_path: Path) -> None:
|
||||
manifest = tmp_path / "manifest.json"
|
||||
candidate = extract_media_candidates(
|
||||
'<img src="/maps/kuori.webp" alt="Куори">',
|
||||
source_page="https://rf4db.com/ru/maps/level_001_kuori",
|
||||
)[0]
|
||||
merge_manifest(manifest, [candidate])
|
||||
with pytest.raises(ValueError, match="known waterbody role"):
|
||||
review_asset(
|
||||
manifest, asset_url=candidate.asset_url, decision="rejected",
|
||||
entity_type="waterbody", entity_key="kuori", media_role="cover",
|
||||
)
|
||||
|
||||
|
||||
def test_image_inspection_rejects_invalid_body_and_mime_mismatch(tmp_path: Path) -> None:
|
||||
with pytest.raises(ValueError, match="valid raster"):
|
||||
inspect_image(b"not an image")
|
||||
@@ -248,3 +340,20 @@ def test_publish_quality_upgrades_switches_mapping_and_retains_fallback(tmp_path
|
||||
assert assets[1]["status"] == "approved"
|
||||
assert assets[1]["entity_key"] == "fish:pike"
|
||||
assert assets[1]["supersedes"] == assets[0]["asset_url"]
|
||||
|
||||
|
||||
def test_rollback_quality_upgrade_restores_fallback_atomically(tmp_path: Path) -> None:
|
||||
path = tmp_path / "manifest.json"
|
||||
path.write_text(json.dumps({"assets": [
|
||||
{"entity_type": "fish", "entity_key": "fish:pike", "status": "superseded", "asset_url": "https://small/pike.png", "replaced_by": "https://large/pike.webp"},
|
||||
{"entity_type": "fish", "entity_key": "fish:pike", "status": "approved", "asset_url": "https://large/pike.webp", "supersedes": "https://small/pike.png"},
|
||||
]}), encoding="utf-8")
|
||||
|
||||
report = rollback_quality_upgrade(path, asset_url="https://large/pike.webp", note="owner rolled back after visual review")
|
||||
assets = json.loads(path.read_text(encoding="utf-8"))["assets"]
|
||||
|
||||
assert report == {"rolled_back": "https://large/pike.webp", "restored": "https://small/pike.png"}
|
||||
assert assets[0]["status"] == "approved"
|
||||
assert "replaced_by" not in assets[0]
|
||||
assert assets[1]["status"] == "upgrade_stored"
|
||||
assert "supersedes" not in assets[1]
|
||||
|
||||
Reference in New Issue
Block a user