feat: add resumable dataset uploads

This commit is contained in:
hailt 2026-11-12 17:31:42 +07:00
parent f8ed08b854
commit 8d40cff227
23 changed files with 913 additions and 21 deletions

View file

@ -24,6 +24,7 @@ Interactive docs: `http://localhost:8000/docs`.
|---|---|---|
| GET | `/health` | Liveness incl. DB check, `{ ok: true }` |
| GET | `/metrics` | Phase-1 traffic counters per route |
| GET | `/storage/usage` | Real bucket `{ bytes, objects, quota_bytes }` (10-min cache) |
| GET | `/` (root, no prefix) | Name, docs and health links |
| POST | `/admin/init-db` | Create tables, seed dev admin. Dev-only |
| POST | `/admin/backfill-thumbnails?limit=100` | Mint missing thumbs. Dev-only |
@ -97,6 +98,44 @@ GET /projects/{id}/images?status=all&search=img_&assignee=all&limit=50&offset=0
`PATCH /projects/{id}/images/{imageId}?status=completed` — status transition.
`POST /projects/{id}/images/batch-presign` — one call for ≤200 files:
`[{ fileName, contentType }]` → `[{ fileName, putUrl, r2Key }]`
(keys carry unique tmp ids, so duplicate filenames never collide).
`GET /projects/{id}/images/by-id/{imageId}` — `{ offset, total }` in list
order (deterministic `created_at DESC, id DESC`); backs `?imageId=` deep
links into thousand-image datasets.
## Resumable multipart uploads (files ~100MB and up)
Small files keep the single-PUT presign path. Big single files
(GB zips, TIFF, video) go multipart, straight to R2:
1. `POST /projects/{id}/uploads/multipart` `{ fileName, contentType }` →
`{ key, uploadId, partSize: 8388608 }`.
2. `POST .../uploads/multipart/parts` `{ key, uploadId, numbers: [1..N] }`
→ `{ urls: { "1": "...", ... } }` (1h TTL each).
3. PUT every part (parallel, per-part retry; store ETags client-side —
the upload resumes after network drops instead of restarting).
4. `POST .../uploads/multipart/complete` `{ key, uploadId,
parts: [{ ETag, PartNumber }] }` → `{ key, etag }`.
5. `POST .../uploads/multipart/abort` to cancel and free the upload.
ETag response headers are exposed via the bucket CORS policy, which the
frontend needs to read them.
## Server-side dataset imports (GB-scale zips)
`POST /projects/{id}/imports/local-zip` `{ path }` (absolute server path)
or `POST /projects/{id}/imports/r2-zip` `{ r2_key }` (zip already uploaded,
e.g. via multipart) → `202 { id, status: queued, ... }`. The worker streams
the archive (one image in RAM at a time), uploads originals + 512/128px
thumbs concurrently, commits DB rows in batches, then bulk-upserts COCO
annotations. Re-running skips files already present.
`GET /projects/{id}/imports` and `GET /projects/{id}/imports/{jobId}` poll
`{ status, total_files, done_files, saved_annotations, error }`.
## Annotations (auth)
`GET /projects/{pid}/images/{iid}/annotation` — one row per image:
@ -109,6 +148,10 @@ GET /projects/{id}/images?status=all&search=img_&assignee=all&limit=50&offset=0
"visible": true, "bbox": { "x": 10, "y": 20, "width": 100, "height": 80 } }] }
```
`PUT /projects/{id}/annotations/batch` with `{ [imageId]: { classificationLabelId?, flags?, shapes[] } }`
upserts up to 500 images in one commit (COCO/YOLO import path) →
`{ saved, missing: [...] }`.
`PUT` same path with `{ classificationLabelId?, flags?, shapes[] }` upserts
(one row per image, 404 when the image does not exist) and syncs
`object_count` / flags onto the image row. Also emits an `annotation.save`