Backend API: auth, projects/images/annotations, R2 presign, tracing, thumbnails, pagination, tests, docs
This commit is contained in:
parent
5c5deed2f5
commit
f8ed08b854
49 changed files with 2626 additions and 0 deletions
133
README.md
Normal file
133
README.md
Normal file
|
|
@ -0,0 +1,133 @@
|
|||
# danable backend
|
||||
|
||||
Backend API for danable, a computer-vision data labeling platform
|
||||
(classification, object detection, segmentation). Metadata lives in
|
||||
PostgreSQL; image bytes, datasets, models and exports live in Cloudflare R2
|
||||
(S3-compatible). A Redis-backed action-event pipeline records every user
|
||||
operation for tracelog and replay.
|
||||
|
||||
## Tech stack
|
||||
|
||||
- Python 3.10+, FastAPI, SQLAlchemy 2 (asyncio) + asyncpg, Alembic (reserved)
|
||||
- PostgreSQL 16, Redis 7, Celery (reserved for heavy workers)
|
||||
- boto3 for R2 (S3-compatible), PyJWT + bcrypt for auth, Pillow for thumbnails
|
||||
- pytest + pytest-asyncio + aiosqlite for tests
|
||||
|
||||
## Quickstart
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
python3 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
pip install -r requirements.txt
|
||||
|
||||
cp .env.example .env # then fill R2 keys (see Configuration)
|
||||
docker compose up -d postgres redis
|
||||
|
||||
uvicorn src.main:app --host 0.0.0.0 --port 8000 --reload
|
||||
curl -X POST localhost:8000/api/v1/admin/init-db
|
||||
```
|
||||
|
||||
`init-db` creates all tables and seeds the dev admin:
|
||||
|
||||
- email: `admin@danable.local`
|
||||
- password: `Admin123!`
|
||||
|
||||
Change or remove this account before any shared deployment. API docs (Swagger):
|
||||
`http://localhost:8000/docs`.
|
||||
|
||||
Detached runs (servers survive the shell):
|
||||
|
||||
```bash
|
||||
bash restart_api.sh # kill + start API on :8000, health-checks itself
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
Secrets live ONLY in `.env` (gitignored). Everything structural lives in
|
||||
`config.yaml`. The application never reads secrets from yaml, and never reads
|
||||
shape/thresholds from env.
|
||||
|
||||
| `.env` key | Purpose | Default |
|
||||
|---|---|---|
|
||||
| `DATABASE_URL` | Postgres DSN (asyncpg) | `postgresql+asyncpg://danable:danable@localhost:5432/danable` |
|
||||
| `REDIS_URL` | Redis for the event queue | `redis://localhost:6379/0` |
|
||||
| `JWT_SECRET` | Signs access/refresh tokens | `change-me` (must override) |
|
||||
| `JWT_EXPIRE_MINUTES` | Access token lifetime | `720` |
|
||||
| `R2_ENDPOINT` | `https://<account>.r2.cloudflarestorage.com` | account endpoint |
|
||||
| `R2_ACCESS_KEY_ID` / `R2_SECRET_ACCESS_KEY` | R2 API token | empty (R2 calls return 503 until set) |
|
||||
| `R2_BUCKET` | Main private bucket | `danagble` |
|
||||
| `R2_PUBLIC_BASE_URL` | Optional public/CDN base for reads | empty (presigned URLs only) |
|
||||
|
||||
`config.yaml` holds: API prefix and page sizes, R2 key layout
|
||||
(`datasets/{project}/...`, `models/...`, `artifacts/...`), presigned TTLs,
|
||||
per-task settings, the AI model registry, traffic budgets, and thumbnail
|
||||
settings (`enabled`, `max_side: 512`, `quality: 70`, micro `128px`).
|
||||
|
||||
## Project structure
|
||||
|
||||
```text
|
||||
backend/
|
||||
config.yaml / config.example.yaml # non-secret shape (see above)
|
||||
docker-compose.yml # postgres + redis + api
|
||||
requirements.txt
|
||||
models/ # SQLAlchemy tables (Postgres metadata only)
|
||||
shared/ # cross-cutting: logger, tracing, traffic, r2, keys,
|
||||
# thumbnails, events, schemas
|
||||
src/
|
||||
main.py # app wiring, middleware, lifespan (event drainer)
|
||||
core/ # config (.env + yaml), db engine, security (JWT/bcrypt)
|
||||
api/v1/ # thin routers: auth, users, projects, images,
|
||||
# annotations, events, activities, system
|
||||
services/ # business logic (routers never touch the DB directly)
|
||||
stores/ # local dev mirror of the R2 layout (see stores/README.md)
|
||||
tests/ # pytest suite (sqlite-backed, no external services)
|
||||
```
|
||||
|
||||
Layering rule: routers parse/validate and call `src/services/*`; shared code
|
||||
never imports routers. Background work goes through the Redis event queue,
|
||||
never inline in requests.
|
||||
|
||||
## Usage workflows
|
||||
|
||||
- Label flow: `POST /projects` -> `POST .../images/presign` -> PUT bytes
|
||||
straight to R2 -> `POST .../images` (confirm mints 512px + 128px thumbs)
|
||||
-> `PUT .../annotation` per image -> `PATCH .../images/{id}?status=`.
|
||||
- Image lists are paginated (`limit` <= 200, `offset`, `total` in response)
|
||||
with server-side `status` / `search` / `assignee` filters. The UI never
|
||||
fetches the whole dataset.
|
||||
- Backfill thumbnails for pre-existing rows:
|
||||
`POST /api/v1/admin/backfill-thumbnails?limit=100`.
|
||||
- Every mutating call emits an action event (Redis -> `action_events`
|
||||
table); the activity feed reads that table. Client-only actions arrive via
|
||||
`POST /api/v1/events` with a `session_id`.
|
||||
- Tracing: send `X-Session-ID` (one per visit) and the `Authorization` header;
|
||||
every response echoes `X-Session-ID` / `X-Trace-ID` and JSON logs carry
|
||||
`session_id`, `trace_id`, `user_id` for replay.
|
||||
|
||||
Full endpoint reference: [docs/API.md](docs/API.md).
|
||||
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
.venv/bin/python -m pytest tests/ -q
|
||||
```
|
||||
|
||||
Tests are sqlite-backed and need no running services. Current coverage:
|
||||
auth (hash/token/route protection), annotations (server defaults, 404),
|
||||
tracing middleware + event ingest, thumbnail resize bounds, pagination
|
||||
(totals, filters, limit clamp).
|
||||
|
||||
## Notes and limits
|
||||
|
||||
- `POST /admin/*` endpoints are dev-only: protect or remove them in any
|
||||
shared deployment.
|
||||
- Thumbnails are minted inline at confirm time (fail-open; the UI falls back
|
||||
to the full image). Move to a worker when uploads spike.
|
||||
- R2 free tier is account-wide: watch Class A/B request counts once many
|
||||
annotators browse thousand-image projects; the UI paginates (50/page) and
|
||||
caches immutable thumbs for a year to stay inside it.
|
||||
|
||||
## License
|
||||
|
||||
Apache-2.0, see [LICENSE](LICENSE).
|
||||
Loading…
Add table
Add a link
Reference in a new issue