Backend API: auth, projects/images/annotations, R2 presign, tracing, thumbnails, pagination, tests, docs

This commit is contained in:
hailt 2026-11-12 15:12:26 +07:00
parent 5c5deed2f5
commit f8ed08b854
49 changed files with 2626 additions and 0 deletions

133
README.md Normal file
View file

@ -0,0 +1,133 @@
# danable backend
Backend API for danable, a computer-vision data labeling platform
(classification, object detection, segmentation). Metadata lives in
PostgreSQL; image bytes, datasets, models and exports live in Cloudflare R2
(S3-compatible). A Redis-backed action-event pipeline records every user
operation for tracelog and replay.
## Tech stack
- Python 3.10+, FastAPI, SQLAlchemy 2 (asyncio) + asyncpg, Alembic (reserved)
- PostgreSQL 16, Redis 7, Celery (reserved for heavy workers)
- boto3 for R2 (S3-compatible), PyJWT + bcrypt for auth, Pillow for thumbnails
- pytest + pytest-asyncio + aiosqlite for tests
## Quickstart
```bash
cd backend
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env # then fill R2 keys (see Configuration)
docker compose up -d postgres redis
uvicorn src.main:app --host 0.0.0.0 --port 8000 --reload
curl -X POST localhost:8000/api/v1/admin/init-db
```
`init-db` creates all tables and seeds the dev admin:
- email: `admin@danable.local`
- password: `Admin123!`
Change or remove this account before any shared deployment. API docs (Swagger):
`http://localhost:8000/docs`.
Detached runs (servers survive the shell):
```bash
bash restart_api.sh # kill + start API on :8000, health-checks itself
```
## Configuration
Secrets live ONLY in `.env` (gitignored). Everything structural lives in
`config.yaml`. The application never reads secrets from yaml, and never reads
shape/thresholds from env.
| `.env` key | Purpose | Default |
|---|---|---|
| `DATABASE_URL` | Postgres DSN (asyncpg) | `postgresql+asyncpg://danable:danable@localhost:5432/danable` |
| `REDIS_URL` | Redis for the event queue | `redis://localhost:6379/0` |
| `JWT_SECRET` | Signs access/refresh tokens | `change-me` (must override) |
| `JWT_EXPIRE_MINUTES` | Access token lifetime | `720` |
| `R2_ENDPOINT` | `https://<account>.r2.cloudflarestorage.com` | account endpoint |
| `R2_ACCESS_KEY_ID` / `R2_SECRET_ACCESS_KEY` | R2 API token | empty (R2 calls return 503 until set) |
| `R2_BUCKET` | Main private bucket | `danagble` |
| `R2_PUBLIC_BASE_URL` | Optional public/CDN base for reads | empty (presigned URLs only) |
`config.yaml` holds: API prefix and page sizes, R2 key layout
(`datasets/{project}/...`, `models/...`, `artifacts/...`), presigned TTLs,
per-task settings, the AI model registry, traffic budgets, and thumbnail
settings (`enabled`, `max_side: 512`, `quality: 70`, micro `128px`).
## Project structure
```text
backend/
config.yaml / config.example.yaml # non-secret shape (see above)
docker-compose.yml # postgres + redis + api
requirements.txt
models/ # SQLAlchemy tables (Postgres metadata only)
shared/ # cross-cutting: logger, tracing, traffic, r2, keys,
# thumbnails, events, schemas
src/
main.py # app wiring, middleware, lifespan (event drainer)
core/ # config (.env + yaml), db engine, security (JWT/bcrypt)
api/v1/ # thin routers: auth, users, projects, images,
# annotations, events, activities, system
services/ # business logic (routers never touch the DB directly)
stores/ # local dev mirror of the R2 layout (see stores/README.md)
tests/ # pytest suite (sqlite-backed, no external services)
```
Layering rule: routers parse/validate and call `src/services/*`; shared code
never imports routers. Background work goes through the Redis event queue,
never inline in requests.
## Usage workflows
- Label flow: `POST /projects` -> `POST .../images/presign` -> PUT bytes
straight to R2 -> `POST .../images` (confirm mints 512px + 128px thumbs)
-> `PUT .../annotation` per image -> `PATCH .../images/{id}?status=`.
- Image lists are paginated (`limit` <= 200, `offset`, `total` in response)
with server-side `status` / `search` / `assignee` filters. The UI never
fetches the whole dataset.
- Backfill thumbnails for pre-existing rows:
`POST /api/v1/admin/backfill-thumbnails?limit=100`.
- Every mutating call emits an action event (Redis -> `action_events`
table); the activity feed reads that table. Client-only actions arrive via
`POST /api/v1/events` with a `session_id`.
- Tracing: send `X-Session-ID` (one per visit) and the `Authorization` header;
every response echoes `X-Session-ID` / `X-Trace-ID` and JSON logs carry
`session_id`, `trace_id`, `user_id` for replay.
Full endpoint reference: [docs/API.md](docs/API.md).
## Testing
```bash
.venv/bin/python -m pytest tests/ -q
```
Tests are sqlite-backed and need no running services. Current coverage:
auth (hash/token/route protection), annotations (server defaults, 404),
tracing middleware + event ingest, thumbnail resize bounds, pagination
(totals, filters, limit clamp).
## Notes and limits
- `POST /admin/*` endpoints are dev-only: protect or remove them in any
shared deployment.
- Thumbnails are minted inline at confirm time (fail-open; the UI falls back
to the full image). Move to a worker when uploads spike.
- R2 free tier is account-wide: watch Class A/B request counts once many
annotators browse thousand-image projects; the UI paginates (50/page) and
caches immutable thumbs for a year to stay inside it.
## License
Apache-2.0, see [LICENSE](LICENSE).