No description
Find a file
2026-11-12 21:38:54 +07:00
.github/workflows feat: add backend CI/CD deployment 2026-11-12 21:32:31 +07:00
docs feat: add backend CI/CD deployment 2026-11-12 21:32:31 +07:00
models feat: add resumable dataset uploads 2026-11-12 17:31:42 +07:00
shared feat: add backend CI/CD deployment 2026-11-12 21:32:31 +07:00
src feat: add backend CI/CD deployment 2026-11-12 21:32:31 +07:00
stores Backend API: auth, projects/images/annotations, R2 presign, tracing, thumbnails, pagination, tests, docs 2026-11-12 15:12:26 +07:00
tests test: isolate storage presign tests 2026-11-12 21:38:54 +07:00
.dockerignore feat: add backend CI/CD deployment 2026-11-12 21:32:31 +07:00
.env.example feat: add backend CI/CD deployment 2026-11-12 21:32:31 +07:00
.gitignore Backend API: auth, projects/images/annotations, R2 presign, tracing, thumbnails, pagination, tests, docs 2026-11-12 15:12:26 +07:00
config.example.yaml feat: add backend CI/CD deployment 2026-11-12 21:32:31 +07:00
config.yaml feat: add backend CI/CD deployment 2026-11-12 21:32:31 +07:00
docker-compose.prod.yml feat: add backend CI/CD deployment 2026-11-12 21:32:31 +07:00
docker-compose.yml feat: add backend CI/CD deployment 2026-11-12 21:32:31 +07:00
Dockerfile Backend API: auth, projects/images/annotations, R2 presign, tracing, thumbnails, pagination, tests, docs 2026-11-12 15:12:26 +07:00
LICENSE Initial commit 2026-11-12 08:57:03 +07:00
README.md feat: add backend CI/CD deployment 2026-11-12 21:32:31 +07:00
requirements.txt Backend API: auth, projects/images/annotations, R2 presign, tracing, thumbnails, pagination, tests, docs 2026-11-12 15:12:26 +07:00
restart_api.sh feat: add backend CI/CD deployment 2026-11-12 21:32:31 +07:00

danagble backend

Backend API for danagble, a computer-vision data labeling platform (classification, object detection, segmentation). Metadata lives in PostgreSQL; image bytes, datasets, models and exports live in Cloudflare R2 (S3-compatible). A Redis-backed action-event pipeline records every user operation for tracelog and replay.

Tech stack

  • Python 3.10+, FastAPI, SQLAlchemy 2 (asyncio) + asyncpg, Alembic (reserved)
  • PostgreSQL 16, Redis 7, Celery (reserved for heavy workers)
  • boto3 for R2 (S3-compatible), PyJWT + bcrypt for auth, Pillow for thumbnails
  • pytest + pytest-asyncio + aiosqlite for tests

Quickstart

cd backend
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

cp .env.example .env   # then fill R2 keys (see Configuration)
docker compose up -d postgres redis

uvicorn src.main:app --host 0.0.0.0 --port 8000 --reload
curl -X POST localhost:8000/api/v1/admin/init-db

init-db creates all tables and seeds the dev admin:

  • email: admin@danagble.local
  • password: Admin123!

Change or remove this account before any shared deployment. API docs (Swagger): http://localhost:8000/docs.

Detached runs (servers survive the shell):

bash restart_api.sh        # kill + start API on :8000, health-checks itself

Configuration

Secrets live ONLY in .env (gitignored). Everything structural lives in config.yaml. The application never reads secrets from yaml, and never reads shape/thresholds from env.

.env key Purpose Default
DATABASE_URL Postgres DSN (asyncpg) postgresql+asyncpg://danagble:danagble@localhost:5432/danagble
REDIS_URL Redis for the event queue redis://localhost:6379/0
JWT_SECRET Signs access/refresh tokens change-me (must override)
JWT_EXPIRE_MINUTES Access token lifetime 720
R2_ENDPOINT https://<account>.r2.cloudflarestorage.com account endpoint
R2_ACCESS_KEY_ID / R2_SECRET_ACCESS_KEY R2 API token empty (R2 calls return 503 until set)
R2_BUCKET Main private bucket danagble
R2_PUBLIC_BASE_URL Optional public/CDN base for reads empty (presigned URLs only)

config.yaml holds: API prefix and page sizes, R2 key layout (datasets/{project}/..., models/..., artifacts/...), presigned TTLs, per-task settings, the AI model registry, traffic budgets, and thumbnail settings (enabled, max_side: 512, quality: 70, micro 128px).

Project structure

backend/
  config.yaml / config.example.yaml  # non-secret shape (see above)
  docker-compose.yml                 # postgres + redis + api
  requirements.txt
  models/            # SQLAlchemy tables (Postgres metadata only)
  shared/            # cross-cutting: logger, tracing, traffic, r2, keys,
                     #   thumbnails, events, schemas
  src/
    main.py          # app wiring, middleware, lifespan (event drainer)
    core/            # config (.env + yaml), db engine, security (JWT/bcrypt)
    api/v1/          # thin routers: auth, users, projects, images,
                     #   annotations, events, activities, system
    services/        # business logic (routers never touch the DB directly)
  stores/            # local dev mirror of the R2 layout (see stores/README.md)
  tests/             # pytest suite (sqlite-backed, no external services)

Layering rule: routers parse/validate and call src/services/*; shared code never imports routers. Background work goes through the Redis event queue, never inline in requests.

Usage workflows

  • Label flow: POST /projects -> POST .../images/presign -> PUT bytes straight to R2 -> POST .../images (confirm mints 512px + 128px thumbs) -> PUT .../annotation per image -> PATCH .../images/{id}?status=.
  • Image lists are paginated (limit <= 200, offset, total in response) with server-side status / search / assignee filters. The UI never fetches the whole dataset.
  • Backfill thumbnails for pre-existing rows: POST /api/v1/admin/backfill-thumbnails?limit=100.
  • Every mutating call emits an action event (Redis -> action_events table); the activity feed reads that table. Client-only actions arrive via POST /api/v1/events with a session_id.
  • Tracing: send X-Session-ID (one per visit) and the Authorization header; every response echoes X-Session-ID / X-Trace-ID and JSON logs carry session_id, trace_id, user_id for replay.

Full endpoint reference: docs/API.md.

Testing

.venv/bin/python -m pytest tests/ -q

Tests are sqlite-backed and need no running services. Current coverage: auth (hash/token/route protection), annotations (server defaults, 404), tracing middleware + event ingest, thumbnail resize bounds, pagination (totals, filters, limit clamp).

Notes and limits

  • POST /admin/* endpoints are dev-only: protect or remove them in any shared deployment.
  • Thumbnails are minted inline at confirm time (fail-open; the UI falls back to the full image). Move to a worker when uploads spike.
  • R2 free tier is account-wide: watch Class A/B request counts once many annotators browse thousand-image projects; the UI paginates (50/page) and caches immutable thumbs for a year to stay inside it.

License

Apache-2.0, see LICENSE.