feat: add resumable dataset uploads
This commit is contained in:
parent
f8ed08b854
commit
8d40cff227
23 changed files with 913 additions and 21 deletions
43
docs/API.md
43
docs/API.md
|
|
@ -24,6 +24,7 @@ Interactive docs: `http://localhost:8000/docs`.
|
|||
|---|---|---|
|
||||
| GET | `/health` | Liveness incl. DB check, `{ ok: true }` |
|
||||
| GET | `/metrics` | Phase-1 traffic counters per route |
|
||||
| GET | `/storage/usage` | Real bucket `{ bytes, objects, quota_bytes }` (10-min cache) |
|
||||
| GET | `/` (root, no prefix) | Name, docs and health links |
|
||||
| POST | `/admin/init-db` | Create tables, seed dev admin. Dev-only |
|
||||
| POST | `/admin/backfill-thumbnails?limit=100` | Mint missing thumbs. Dev-only |
|
||||
|
|
@ -97,6 +98,44 @@ GET /projects/{id}/images?status=all&search=img_&assignee=all&limit=50&offset=0
|
|||
|
||||
`PATCH /projects/{id}/images/{imageId}?status=completed` — status transition.
|
||||
|
||||
`POST /projects/{id}/images/batch-presign` — one call for ≤200 files:
|
||||
`[{ fileName, contentType }]` → `[{ fileName, putUrl, r2Key }]`
|
||||
(keys carry unique tmp ids, so duplicate filenames never collide).
|
||||
|
||||
`GET /projects/{id}/images/by-id/{imageId}` — `{ offset, total }` in list
|
||||
order (deterministic `created_at DESC, id DESC`); backs `?imageId=` deep
|
||||
links into thousand-image datasets.
|
||||
|
||||
## Resumable multipart uploads (files ~100MB and up)
|
||||
|
||||
Small files keep the single-PUT presign path. Big single files
|
||||
(GB zips, TIFF, video) go multipart, straight to R2:
|
||||
|
||||
1. `POST /projects/{id}/uploads/multipart` `{ fileName, contentType }` →
|
||||
`{ key, uploadId, partSize: 8388608 }`.
|
||||
2. `POST .../uploads/multipart/parts` `{ key, uploadId, numbers: [1..N] }`
|
||||
→ `{ urls: { "1": "...", ... } }` (1h TTL each).
|
||||
3. PUT every part (parallel, per-part retry; store ETags client-side —
|
||||
the upload resumes after network drops instead of restarting).
|
||||
4. `POST .../uploads/multipart/complete` `{ key, uploadId,
|
||||
parts: [{ ETag, PartNumber }] }` → `{ key, etag }`.
|
||||
5. `POST .../uploads/multipart/abort` to cancel and free the upload.
|
||||
|
||||
ETag response headers are exposed via the bucket CORS policy, which the
|
||||
frontend needs to read them.
|
||||
|
||||
## Server-side dataset imports (GB-scale zips)
|
||||
|
||||
`POST /projects/{id}/imports/local-zip` `{ path }` (absolute server path)
|
||||
or `POST /projects/{id}/imports/r2-zip` `{ r2_key }` (zip already uploaded,
|
||||
e.g. via multipart) → `202 { id, status: queued, ... }`. The worker streams
|
||||
the archive (one image in RAM at a time), uploads originals + 512/128px
|
||||
thumbs concurrently, commits DB rows in batches, then bulk-upserts COCO
|
||||
annotations. Re-running skips files already present.
|
||||
|
||||
`GET /projects/{id}/imports` and `GET /projects/{id}/imports/{jobId}` poll
|
||||
`{ status, total_files, done_files, saved_annotations, error }`.
|
||||
|
||||
## Annotations (auth)
|
||||
|
||||
`GET /projects/{pid}/images/{iid}/annotation` — one row per image:
|
||||
|
|
@ -109,6 +148,10 @@ GET /projects/{id}/images?status=all&search=img_&assignee=all&limit=50&offset=0
|
|||
"visible": true, "bbox": { "x": 10, "y": 20, "width": 100, "height": 80 } }] }
|
||||
```
|
||||
|
||||
`PUT /projects/{id}/annotations/batch` with `{ [imageId]: { classificationLabelId?, flags?, shapes[] } }`
|
||||
upserts up to 500 images in one commit (COCO/YOLO import path) →
|
||||
`{ saved, missing: [...] }`.
|
||||
|
||||
`PUT` same path with `{ classificationLabelId?, flags?, shapes[] }` upserts
|
||||
(one row per image, 404 when the image does not exist) and syncs
|
||||
`object_count` / flags onto the image row. Also emits an `annotation.save`
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue