Fix container sync races that leaked orphan blobs; add a system-prune CLI command

sync_workspace (user-triggered) and the reconciler's sync_bidirectional_sync
could run concurrently for the same project, and store_upload's read-then-
write on a changed path meant two racing imports each wrote their own blob
while only one ever got referenced - the loser leaked forever. Combined with
no build-artifact exclusion, an actively-compiling workspace hit this
constantly and leaked 5.9M orphan blobs (~96GB) in production before it was
caught.

Closes it at the root: api._sync_dir_bidirectional_locked serializes both
call sites per-project (non-blocking - a project already mid-sync is simply
skipped until the next tick), and IMPORT_SKIP_NAMES/IMPORT_SKIP_EXTENSIONS
keep build output (build/, dist/, *.o, *.pyc, ...) out of the walk entirely.

Recovering what already leaked is a separate concern: a new CLI subcommand
(plus matching make targets) sweeps soft-deleted attachment/project-file
blobs and any blob with zero DB reference at all, plus orphaned container
workspace directories. run_maintenance_cleanup.sh wraps the existing
prune/clear commands for routine disk upkeep.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BWJy6PrMMt5hwWxQwia2rd
This commit is contained in:
2026-09-08 03:43:49 +02:00
co-authored by Claude Sonnet 5
parent 85bd8fad47
commit 7880bf4b31
19 changed files with 1068 additions and 19 deletions
+6 -3
View File
@@ -18,6 +18,7 @@ def paginate(
order=None,
cursor_field="created_at",
viewer_uid=None,
limit=PAGE_SIZE,
**filters,
):
order = order or ["-" + cursor_field]
@@ -30,9 +31,9 @@ def paginate(
clauses.append(table.table.columns.user_uid.notin_(blocked))
if before:
clauses.append(table.table.columns[cursor_field] < before)
rows = list(table.find(*clauses, **filters, order_by=order, _limit=PAGE_SIZE + 1))
has_more = len(rows) > PAGE_SIZE
rows = rows[:PAGE_SIZE]
rows = list(table.find(*clauses, **filters, order_by=order, _limit=limit + 1))
has_more = len(rows) > limit
rows = rows[:limit]
next_cursor = rows[-1][cursor_field] if has_more and rows else None
return rows, next_cursor
@@ -64,6 +65,7 @@ def paginate_diverse(
cursor_field="created_at",
uid_key="user_uid",
viewer_uid=None,
limit=PAGE_SIZE,
**filters,
):
rows, next_cursor = paginate(
@@ -73,6 +75,7 @@ def paginate_diverse(
order=order,
cursor_field=cursor_field,
viewer_uid=viewer_uid,
limit=limit,
**filters,
)
return interleave_by_author(rows, uid_key=uid_key), next_cursor