retoor/devplacepy
1
0
Fork 1
Code Issues 44 Pull Requests 4 Actions Packages Projects Releases Wiki Activity

fix: switch blob sharding from uuid7 leading hex to random tail in attachments, project_files, and zip_service

Browse Source
This commit is contained in:
retoor
2026-06-13 19:12:54 +00:00
parent 1294c95bed
commit 5c199474f3
5 changed files with 9 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+3 -1
View File
@@ -51,6 +51,8 @@ devplace migrate-data # relocate legacy runtime files into data/ (idempote
Every runtime/user-generated artifact lives under one root, `config.DATA_DIR` (`DEVPLACE_DATA_DIR`, default `<repo>/data`). `config.py` derives every runtime path from it and lists them in the `DATA_PATHS` registry; `ensure_data_dirs()` (called at the top of `startup()` and from `database.init_db`, plus a guard at `database.py` import before `dataset.connect`) creates the whole tree before anything is written. **No module computes a runtime path from scratch** - they import the constants (`UPLOADS_DIR`, `ATTACHMENTS_DIR`, `PROJECT_FILES_DIR`, `ZIPS_DIR`, `ZIP_STAGING_DIR`, `FORK_STAGING_DIR`, `KEYS_DIR`, `BOT_DIR`, `LOCKS_DIR`, `CONTAINER_WORKSPACES_DIR`, `DEVII_TASKS_DB`, `DEVII_LESSONS_DB`, `VAPID_*_FILE`, `SERVICE_LOCK_FILE`, `INIT_LOCK_FILE`). Uploads moved physically from `devplacepy/static/uploads/` to `data/uploads/` but are still **served at the unchanged `/static/uploads/` URL** (FastAPI `UploadStaticFiles` mount -> `config.UPLOADS_DIR`; nginx `alias /data/uploads/` from a mounted volume). Stored DB values for attachments/project files are relative (`directory` + `stored_name`), so **no DB rewrite is needed**. `devplace migrate-data` relocates an older scattered install (copy -> verify CRC -> atomic replace -> unlink; idempotent).
**Blob sharding (`attachments._directory_for`, the single shard function reused by `attachments`, `project_files`, and `zip_service`):** blobs are spread into a two-level `xx/yy` directory tree to keep any one directory small. The shard is taken from the **random tail** of the uuid7 (`tail = uid.replace("-", "")`, `f"{tail[-2:]}/{tail[-4:-2]}"`), NOT its leading bytes. This is load-bearing: a uuid7's first 48 bits are a millisecond timestamp, so the leading hex (`uid[:2]`/`uid[2:4]`) advances only every ~35 years / ~50 days respectively - prefix-sharding a time-ordered id funnels every contemporaneous write into ONE bucket (the legacy `data/uploads/attachments/01/9e/` pile is exactly this defect). The last 62 bits are random, so the tail gives a uniform 256x256 fan-out from the first write. Any new shard helper MUST shard on a high-entropy field (uuid tail or a hash), never the head. The change is forward-only: every consumer persists the computed `directory`/`local_path` and reads it back, so pre-existing rows keep resolving from their stored value and only new writes use the new layout (no migration required; `migrate-data` could rebalance the old pile if ever desired).
Tests run on port 10501 with a tempfile SQLite DB and a dedicated `DEVPLACE_DATA_DIR`, and `DEVPLACE_DISABLE_SERVICES=1` so background services don't start. `make test` runs **serially, one test at a time, in a single process**: one isolated DB + data dir + uvicorn subprocess + Chromium, shared across the whole session, so session fixtures (`seeded_db`) and intra-file ordering hold. Serial execution is enforced centrally in `pyproject.toml` (`[tool.pytest.ini_options]` `addopts = "--tb=line -p no:xdist"`); pytest-xdist is no longer a dependency and `-n` is rejected, so the suite can never run concurrently.
## Environment variables
@@ -143,7 +145,7 @@ Batch helpers (`get_users_by_uids`, `get_comment_counts_by_post_uids`, `get_vote
To add a service: extend `BaseService`, override `async def run_once(self) -> None`, register in `main.py`'s startup with `service_manager.register(YourService())`. It auto-appears on the services page.
### Async job services (`devplacepy/services/jobs/`)
The **standard** way to run heavy, blocking work off the request path and return a result URL. A shared `jobs` table is the queue (discriminated by a `kind` column; "default" fields plus `payload`/`result` JSON, lifecycle timestamps, `retry_count`, `last_accessed_at`/`expires_at`, and `bytes_in`/`bytes_out`/`item_count` stat columns). `services/jobs/queue.py` (`enqueue`, `get_job`, `touch_job`, `list_jobs`) is pure DB and callable from **any** worker - the row is the queue. `JobService(BaseService)` runs only in the lock owner; each `run_once` reaps finished in-flight tasks, recovers orphaned `running` rows (left by a dead owner) back to `pending` with bounded `retry_count`, refills up to `max_concurrent` oldest pending jobs (uuid7 = FIFO) as `asyncio` tasks, and **sweeps expired artifacts** via the per-kind `cleanup()` hook (retention is built in, default 7 days, admin-configurable; no separate reaper). A new kind subclasses `JobService`, sets `kind`, and implements `async process(self, job) -> dict` and `cleanup(self, job)`. **`ZipService`** (kind `zip`) is the first consumer: `process` materializes a project subtree via `project_files.export_to_dir` (staging under `config.DATA_DIR/zip_staging/{uid}`), compresses it in a **subprocess** (`python -m devplacepy.services.jobs.zip_worker`, stdlib-only, prints stats JSON incl. crc32), names the output `{crc32}.{slug}.zip` under `config.DATA_DIR/zips/{uid[:2]}/{uid[2:4]}/`, and returns stats. (Runtime artifacts live in `DATA_DIR` = `data/` by default, OUTSIDE the package and NOT under `/static`; the download is served by the `/zips/{uid}/download` route via `FileResponse`, not the static mount.) Enqueue endpoints own authz (`POST /projects/{slug}/zip`, `POST /projects/{slug}/files/zip?path=`); `GET /zips/{uid}` (status `ZipJobOut`) and `GET /zips/{uid}/download` (FileResponse, extends expiry) are **capability URLs** scoped only by the unguessable uuid7, not by owner. Frontend `app.zipDownloader` (`static/js/ZipDownloader.js`) auto-wires any `data-zip-download` element plus the files context menu: POST -> poll `/zips/{uid}` -> trigger download. Devii tools `zip_project`/`zip_status`; CLI `devplace zips prune|clear`. **`ForkService`** (kind `fork`, `services/jobs/fork_service.py`) is the second consumer and forks a project: `process` creates the destination project (via `content.create_content_item`, owned by the forking user), copies the whole virtual FS through `project_files.export_to_dir` -> `config.DATA_DIR/fork_staging/{uid}` -> `import_from_dir(..., skip_names=set())` (exact copy), and records the directional relation via `database.record_fork(source_uid, forked_uid, forked_by_uid)` into the `project_forks` table; on any failure it **rolls back** the half-created project (delete files + relation + row) and re-raises. **Critical difference from zip: the artifact is a permanent project, so `cleanup()` is a no-op on the project** - the retention sweep only removes the job tracking row, never the fork. Enqueue `POST /projects/{slug}/fork` (`require_user` + `can_view_project`, body `ForkForm{title}`); `GET /forks/{uid}` returns `ForkJobOut` whose `project_url` is set once done. Frontend `app.projectForker` (`static/js/ProjectForker.js`) wires `data-fork-project`: prompt for a name -> POST -> poll `/forks/{uid}` -> redirect to `project_url`. The forked project's detail page shows a "Forked from X" link (`database.get_fork_parent`). Devii tools `fork_project`/`fork_status`; CLI `devplace forks prune|clear` (job rows only).
The **standard** way to run heavy, blocking work off the request path and return a result URL. A shared `jobs` table is the queue (discriminated by a `kind` column; "default" fields plus `payload`/`result` JSON, lifecycle timestamps, `retry_count`, `last_accessed_at`/`expires_at`, and `bytes_in`/`bytes_out`/`item_count` stat columns). `services/jobs/queue.py` (`enqueue`, `get_job`, `touch_job`, `list_jobs`) is pure DB and callable from **any** worker - the row is the queue. `JobService(BaseService)` runs only in the lock owner; each `run_once` reaps finished in-flight tasks, recovers orphaned `running` rows (left by a dead owner) back to `pending` with bounded `retry_count`, refills up to `max_concurrent` oldest pending jobs (uuid7 = FIFO) as `asyncio` tasks, and **sweeps expired artifacts** via the per-kind `cleanup()` hook (retention is built in, default 7 days, admin-configurable; no separate reaper). A new kind subclasses `JobService`, sets `kind`, and implements `async process(self, job) -> dict` and `cleanup(self, job)`. **`ZipService`** (kind `zip`) is the first consumer: `process` materializes a project subtree via `project_files.export_to_dir` (staging under `config.DATA_DIR/zip_staging/{uid}`), compresses it in a **subprocess** (`python -m devplacepy.services.jobs.zip_worker`, stdlib-only, prints stats JSON incl. crc32), names the output `{crc32}.{slug}.zip` under `config.DATA_DIR/zips/{uid[-2:]}/{uid[-4:-2]}/` (sharded on the random tail of the uuid7 via `attachments._directory_for`, not its time-ordered head - see note below), and returns stats. (Runtime artifacts live in `DATA_DIR` = `data/` by default, OUTSIDE the package and NOT under `/static`; the download is served by the `/zips/{uid}/download` route via `FileResponse`, not the static mount.) Enqueue endpoints own authz (`POST /projects/{slug}/zip`, `POST /projects/{slug}/files/zip?path=`); `GET /zips/{uid}` (status `ZipJobOut`) and `GET /zips/{uid}/download` (FileResponse, extends expiry) are **capability URLs** scoped only by the unguessable uuid7, not by owner. Frontend `app.zipDownloader` (`static/js/ZipDownloader.js`) auto-wires any `data-zip-download` element plus the files context menu: POST -> poll `/zips/{uid}` -> trigger download. Devii tools `zip_project`/`zip_status`; CLI `devplace zips prune|clear`. **`ForkService`** (kind `fork`, `services/jobs/fork_service.py`) is the second consumer and forks a project: `process` creates the destination project (via `content.create_content_item`, owned by the forking user), copies the whole virtual FS through `project_files.export_to_dir` -> `config.DATA_DIR/fork_staging/{uid}` -> `import_from_dir(..., skip_names=set())` (exact copy), and records the directional relation via `database.record_fork(source_uid, forked_uid, forked_by_uid)` into the `project_forks` table; on any failure it **rolls back** the half-created project (delete files + relation + row) and re-raises. **Critical difference from zip: the artifact is a permanent project, so `cleanup()` is a no-op on the project** - the retention sweep only removes the job tracking row, never the fork. Enqueue `POST /projects/{slug}/fork` (`require_user` + `can_view_project`, body `ForkForm{title}`); `GET /forks/{uid}` returns `ForkJobOut` whose `project_url` is set once done. Frontend `app.projectForker` (`static/js/ProjectForker.js`) wires `data-fork-project`: prompt for a name -> POST -> poll `/forks/{uid}` -> redirect to `project_url`. The forked project's detail page shows a "Forked from X" link (`database.get_fork_parent`). Devii tools `fork_project`/`fork_status`; CLI `devplace forks prune|clear` (job rows only).
### Bug tracker (`devplacepy/services/gitea/`)
The `/bugs` feature is a **full Gitea integration with no local issue store** - the listing and detail views read issues straight from Gitea (default repo `retoor/devplacepy`) with live status. `services/gitea/` holds `config.py` (the admin-editable `CONFIG_FIELDS` for base URL/owner/repo/token + AI model, surfaced on the `BugTrackerService` config at `/admin/services`; `gitea_config()` -> `GiteaConfig`), `client.py` (async `GiteaClient` over the REST API, `Authorization: token`, `GiteaError`), `fake.py` (`FakeGiteaClient` test double), `runtime.py` (`get_client`/`set_client` swap), `enhance.py` (`enhance_ticket` rewrites a raw report into a consistent markdown ticket via the **internal AI gateway**, fail-soft to a deterministic template), `store.py` (the only DB state: `bug_tickets` and `bug_comment_authors`, both in `SOFT_DELETE_TABLES` - thin mappings of Gitea number/comment-id -> DevPlace author for attribution + cached `last_status`/`last_comment_count` for update detection), and `service.py` (`BugTrackerService`, `default_enabled=False`, polls tracked tickets and notifies the **reporter** on developer replies / status changes). Filing is an **async job** (`BugCreateService`, kind `bug_create`): enhance -> create Gitea issue -> record mapping -> notify reporter; `POST /bugs/create` enqueues, frontend `app.bugReporter` polls `/bugs/jobs/{uid}`. Comments (`POST /bugs/{number}/comment`, notifies **all admins**) and admin status changes (`POST /bugs/{number}/status`, open/closed) are synchronous single Gitea calls. All Gitea actions use one shared bot token; per-user identity is shown by storing DevPlace author names locally, never per-user Gitea accounts.