Fix container sync races that leaked orphan blobs; add a system-prune CLI command
sync_workspace (user-triggered) and the reconciler's sync_bidirectional_sync could run concurrently for the same project, and store_upload's read-then- write on a changed path meant two racing imports each wrote their own blob while only one ever got referenced - the loser leaked forever. Combined with no build-artifact exclusion, an actively-compiling workspace hit this constantly and leaked 5.9M orphan blobs (~96GB) in production before it was caught. Closes it at the root: api._sync_dir_bidirectional_locked serializes both call sites per-project (non-blocking - a project already mid-sync is simply skipped until the next tick), and IMPORT_SKIP_NAMES/IMPORT_SKIP_EXTENSIONS keep build output (build/, dist/, *.o, *.pyc, ...) out of the walk entirely. Recovering what already leaked is a separate concern: a new CLI subcommand (plus matching make targets) sweeps soft-deleted attachment/project-file blobs and any blob with zero DB reference at all, plus orphaned container workspace directories. run_maintenance_cleanup.sh wraps the existing prune/clear commands for routine disk upkeep. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BWJy6PrMMt5hwWxQwia2rd
This commit is contained in:
@@ -18,6 +18,7 @@ def paginate(
|
||||
order=None,
|
||||
cursor_field="created_at",
|
||||
viewer_uid=None,
|
||||
limit=PAGE_SIZE,
|
||||
**filters,
|
||||
):
|
||||
order = order or ["-" + cursor_field]
|
||||
@@ -30,9 +31,9 @@ def paginate(
|
||||
clauses.append(table.table.columns.user_uid.notin_(blocked))
|
||||
if before:
|
||||
clauses.append(table.table.columns[cursor_field] < before)
|
||||
rows = list(table.find(*clauses, **filters, order_by=order, _limit=PAGE_SIZE + 1))
|
||||
has_more = len(rows) > PAGE_SIZE
|
||||
rows = rows[:PAGE_SIZE]
|
||||
rows = list(table.find(*clauses, **filters, order_by=order, _limit=limit + 1))
|
||||
has_more = len(rows) > limit
|
||||
rows = rows[:limit]
|
||||
next_cursor = rows[-1][cursor_field] if has_more and rows else None
|
||||
return rows, next_cursor
|
||||
|
||||
@@ -64,6 +65,7 @@ def paginate_diverse(
|
||||
cursor_field="created_at",
|
||||
uid_key="user_uid",
|
||||
viewer_uid=None,
|
||||
limit=PAGE_SIZE,
|
||||
**filters,
|
||||
):
|
||||
rows, next_cursor = paginate(
|
||||
@@ -73,6 +75,7 @@ def paginate_diverse(
|
||||
order=order,
|
||||
cursor_field=cursor_field,
|
||||
viewer_uid=viewer_uid,
|
||||
limit=limit,
|
||||
**filters,
|
||||
)
|
||||
return interleave_by_author(rows, uid_key=uid_key), next_cursor
|
||||
|
||||
Reference in New Issue
Block a user