Fix container sync races that leaked orphan blobs; add a system-prune CLI command

sync_workspace (user-triggered) and the reconciler's sync_bidirectional_sync
could run concurrently for the same project, and store_upload's read-then-
write on a changed path meant two racing imports each wrote their own blob
while only one ever got referenced - the loser leaked forever. Combined with
no build-artifact exclusion, an actively-compiling workspace hit this
constantly and leaked 5.9M orphan blobs (~96GB) in production before it was
caught.

Closes it at the root: api._sync_dir_bidirectional_locked serializes both
call sites per-project (non-blocking - a project already mid-sync is simply
skipped until the next tick), and IMPORT_SKIP_NAMES/IMPORT_SKIP_EXTENSIONS
keep build output (build/, dist/, *.o, *.pyc, ...) out of the walk entirely.

Recovering what already leaked is a separate concern: a new CLI subcommand
(plus matching make targets) sweeps soft-deleted attachment/project-file
blobs and any blob with zero DB reference at all, plus orphaned container
workspace directories. run_maintenance_cleanup.sh wraps the existing
prune/clear commands for routine disk upkeep.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BWJy6PrMMt5hwWxQwia2rd
This commit is contained in:
2026-09-08 03:43:49 +02:00
co-authored by Claude Sonnet 5
parent 85bd8fad47
commit 7880bf4b31
19 changed files with 1068 additions and 19 deletions
+41
View File
@@ -161,3 +161,44 @@ def test_main_dispatches_subcommand(local_db, monkeypatch, capsys):
monkeypatch.setattr("sys.argv", ["devplace", "role", "get", username])
cli.main()
assert capsys.readouterr().out.strip() == "admin"
def test_system_prune_reclaims_orphans_across_subsystems(local_db, tmp_path, capsys, monkeypatch):
from devplacepy import attachments as att
from devplacepy import project_files as pf
monkeypatch.setattr(att, "ATTACHMENTS_DIR", tmp_path / "attachments")
monkeypatch.setattr(pf, "PROJECT_FILES_DIR", tmp_path / "project_files")
monkeypatch.setattr("devplacepy.config.CONTAINER_WORKSPACES_DIR", tmp_path / "workspaces")
attachments_dir = tmp_path / "attachments" / "ab" / "cd"
attachments_dir.mkdir(parents=True)
(attachments_dir / "orphan.png").write_bytes(b"a" * 10)
project_files_dir = tmp_path / "project_files" / "ab" / "cd"
project_files_dir.mkdir(parents=True)
(project_files_dir / "orphan.o").write_bytes(b"b" * 20)
(tmp_path / "workspaces").mkdir()
(tmp_path / "workspaces" / "orphan-project").mkdir()
cli.cmd_system_prune(argparse.Namespace(dry_run=True))
dry_output = capsys.readouterr().out
assert "DRY RUN" in dry_output
assert (attachments_dir / "orphan.png").exists()
assert (project_files_dir / "orphan.o").exists()
assert (tmp_path / "workspaces" / "orphan-project").exists()
cli.cmd_system_prune(argparse.Namespace(dry_run=False))
real_output = capsys.readouterr().out
assert "DRY RUN" not in real_output
assert not (attachments_dir / "orphan.png").exists()
assert not (project_files_dir / "orphan.o").exists()
assert not (tmp_path / "workspaces" / "orphan-project").exists()
def test_system_prune_registered_in_parser():
parser = cli.build_parser()
args = parser.parse_args(["system", "prune", "--dry-run"])
assert args.func is cli.cmd_system_prune
assert args.dry_run is True