133 lines
5.2 KiB
Markdown
133 lines
5.2 KiB
Markdown
# danagble backend
|
|
|
|
Backend API for danagble, a computer-vision data labeling platform
|
|
(classification, object detection, segmentation). Metadata lives in
|
|
PostgreSQL; image bytes, datasets, models and exports live in Cloudflare R2
|
|
(S3-compatible). A Redis-backed action-event pipeline records every user
|
|
operation for tracelog and replay.
|
|
|
|
## Tech stack
|
|
|
|
- Python 3.10+, FastAPI, SQLAlchemy 2 (asyncio) + asyncpg, Alembic (reserved)
|
|
- PostgreSQL 16, Redis 7, Celery (reserved for heavy workers)
|
|
- boto3 for R2 (S3-compatible), PyJWT + bcrypt for auth, Pillow for thumbnails
|
|
- pytest + pytest-asyncio + aiosqlite for tests
|
|
|
|
## Quickstart
|
|
|
|
```bash
|
|
cd backend
|
|
python3 -m venv .venv
|
|
source .venv/bin/activate
|
|
pip install -r requirements.txt
|
|
|
|
cp .env.example .env # then fill R2 keys (see Configuration)
|
|
docker compose up -d postgres redis
|
|
|
|
uvicorn src.main:app --host 0.0.0.0 --port 8000 --reload
|
|
curl -X POST localhost:8000/api/v1/admin/init-db
|
|
```
|
|
|
|
`init-db` creates all tables and seeds the dev admin:
|
|
|
|
- email: `admin@danagble.local`
|
|
- password: `Admin123!`
|
|
|
|
Change or remove this account before any shared deployment. API docs (Swagger):
|
|
`http://localhost:8000/docs`.
|
|
|
|
Detached runs (servers survive the shell):
|
|
|
|
```bash
|
|
bash restart_api.sh # kill + start API on :8000, health-checks itself
|
|
```
|
|
|
|
## Configuration
|
|
|
|
Secrets live ONLY in `.env` (gitignored). Everything structural lives in
|
|
`config.yaml`. The application never reads secrets from yaml, and never reads
|
|
shape/thresholds from env.
|
|
|
|
| `.env` key | Purpose | Default |
|
|
|---|---|---|
|
|
| `DATABASE_URL` | Postgres DSN (asyncpg) | `postgresql+asyncpg://danagble:danagble@localhost:5432/danagble` |
|
|
| `REDIS_URL` | Redis for the event queue | `redis://localhost:6379/0` |
|
|
| `JWT_SECRET` | Signs access/refresh tokens | `change-me` (must override) |
|
|
| `JWT_EXPIRE_MINUTES` | Access token lifetime | `720` |
|
|
| `R2_ENDPOINT` | `https://<account>.r2.cloudflarestorage.com` | account endpoint |
|
|
| `R2_ACCESS_KEY_ID` / `R2_SECRET_ACCESS_KEY` | R2 API token | empty (R2 calls return 503 until set) |
|
|
| `R2_BUCKET` | Main private bucket | `danagble` |
|
|
| `R2_PUBLIC_BASE_URL` | Optional public/CDN base for reads | empty (presigned URLs only) |
|
|
|
|
`config.yaml` holds: API prefix and page sizes, R2 key layout
|
|
(`datasets/{project}/...`, `models/...`, `artifacts/...`), presigned TTLs,
|
|
per-task settings, the AI model registry, traffic budgets, and thumbnail
|
|
settings (`enabled`, `max_side: 512`, `quality: 70`, micro `128px`).
|
|
|
|
## Project structure
|
|
|
|
```text
|
|
backend/
|
|
config.yaml / config.example.yaml # non-secret shape (see above)
|
|
docker-compose.yml # postgres + redis + api
|
|
requirements.txt
|
|
models/ # SQLAlchemy tables (Postgres metadata only)
|
|
shared/ # cross-cutting: logger, tracing, traffic, r2, keys,
|
|
# thumbnails, events, schemas
|
|
src/
|
|
main.py # app wiring, middleware, lifespan (event drainer)
|
|
core/ # config (.env + yaml), db engine, security (JWT/bcrypt)
|
|
api/v1/ # thin routers: auth, users, projects, images,
|
|
# annotations, events, activities, system
|
|
services/ # business logic (routers never touch the DB directly)
|
|
stores/ # local dev mirror of the R2 layout (see stores/README.md)
|
|
tests/ # pytest suite (sqlite-backed, no external services)
|
|
```
|
|
|
|
Layering rule: routers parse/validate and call `src/services/*`; shared code
|
|
never imports routers. Background work goes through the Redis event queue,
|
|
never inline in requests.
|
|
|
|
## Usage workflows
|
|
|
|
- Label flow: `POST /projects` -> `POST .../images/presign` -> PUT bytes
|
|
straight to R2 -> `POST .../images` (confirm mints 512px + 128px thumbs)
|
|
-> `PUT .../annotation` per image -> `PATCH .../images/{id}?status=`.
|
|
- Image lists are paginated (`limit` <= 200, `offset`, `total` in response)
|
|
with server-side `status` / `search` / `assignee` filters. The UI never
|
|
fetches the whole dataset.
|
|
- Backfill thumbnails for pre-existing rows:
|
|
`POST /api/v1/admin/backfill-thumbnails?limit=100`.
|
|
- Every mutating call emits an action event (Redis -> `action_events`
|
|
table); the activity feed reads that table. Client-only actions arrive via
|
|
`POST /api/v1/events` with a `session_id`.
|
|
- Tracing: send `X-Session-ID` (one per visit) and the `Authorization` header;
|
|
every response echoes `X-Session-ID` / `X-Trace-ID` and JSON logs carry
|
|
`session_id`, `trace_id`, `user_id` for replay.
|
|
|
|
Full endpoint reference: [docs/API.md](docs/API.md).
|
|
|
|
## Testing
|
|
|
|
```bash
|
|
.venv/bin/python -m pytest tests/ -q
|
|
```
|
|
|
|
Tests are sqlite-backed and need no running services. Current coverage:
|
|
auth (hash/token/route protection), annotations (server defaults, 404),
|
|
tracing middleware + event ingest, thumbnail resize bounds, pagination
|
|
(totals, filters, limit clamp).
|
|
|
|
## Notes and limits
|
|
|
|
- `POST /admin/*` endpoints are dev-only: protect or remove them in any
|
|
shared deployment.
|
|
- Thumbnails are minted inline at confirm time (fail-open; the UI falls back
|
|
to the full image). Move to a worker when uploads spike.
|
|
- R2 free tier is account-wide: watch Class A/B request counts once many
|
|
annotators browse thousand-image projects; the UI paginates (50/page) and
|
|
caches immutable thumbs for a year to stay inside it.
|
|
|
|
## License
|
|
|
|
Apache-2.0, see [LICENSE](LICENSE).
|