## Implementation Plan: Backup Schedule & Disk Cleanup Gaps Based on investigation, the original ticket contains several inaccuracies. The actual four gaps are: 1. **`seed_default_schedule()` failure silently swallowed** – if the first service init races with DB readiness, no schedule is created and no warning is raised. 2. **No proactive disk alerting** – disk warnings are shown on the dashboard but never pushed via email or Devii message. 3. **No database growth rate monitoring** – size tracked but not trended or alerted. 4. **No bulk sweep of stopped containers for hard-deleted instances** – `_reconcile` handles `ST_REMOVING` rows, but if the instance row is gone, the Docker container is orphaned indefinitely. None of these changes require adding a new backup schedule or rotation policy (those already exist). The plan below addresses only the true gaps. ### Change 1: Log and propagate `seed_default_schedule()` failures **File:** `devplacepy/services/backup/service.py` - In `_init_backend()` (around line 38–40), change the silent `except` to log the exception and optionally re-raise after initialisation completes (so the service still starts but the error is visible). - Add a `logger.warning(...)` call when `seed_default_schedule` returns `False` or raises. ```python # Before (approx line 38-40): try: seed_default_schedule() except Exception: pass # After: try: seed_default_schedule() except Exception as exc: logger.warning("Failed to seed default backup schedule: %s", exc) ``` ### Change 2: Proactive disk alerting **File:** `devplacepy/services/backup/store.py` (the `compute_storage_stats` function or a new method) - After computing disk usage, if `percent_used > config.DISK_WARNING_THRESHOLD` (default 85), call a new helper `_send_disk_alert(percent_used)`. - The helper should write an audit event (`"disk.usage.critical"`) and, if a Devii notification channel is available, send a message to the admin user(s). For now, use the existing `audit.log_event` and add a simple in-app notification via the existing `NotificationService` (if available; otherwise just an audit event and a log entry). - To keep it minimal, trigger the check at the end of every scheduled backup (inside `execute_schedule()`). **File:** `devplacepy/services/backup/service.py` - In `execute_schedule()` (around line 110), after backup and rotation completes, call `store.check_disk_and_alert()`. ### Change 3: Database growth rate monitoring **File:** `devplacepy/services/backup/store.py` - Add a small CSV or simple in-memory/DB table to record database size at each backup. - On each scheduled backup, compute `(current_size - last_size) / days_since_last`. If the daily growth exceeds a threshold (e.g., 10% of last size per day), log a warning and write an audit event `"backup.db_growth_high"`. - Threshold can be a config constant `DB_GROWTH_ALERT_PCT` default 0.1 (10% daily). - The historical data is stored as a new key in the existing `state.json` (or a separate JSON file in the backup directory). Use the same pattern as the backup state. ### Change 4: Bulk sweep of orphaned containers from hard-deleted instances **File:** `devplacepy/services/containers/service.py` - In `_reconcile()` (around line 170–175), after handling `ST_REMOVING` instances, add a new method `_sweep_orphaned_containers()` that: 1. Lists all Docker containers with label `managed_by=devplace` (if such label exists; otherwise query by known naming pattern). 2. For each container, check if a corresponding record exists in `instances` table. 3. If not, stop and remove the container (with graceful timeout). - Call this method once per tick (every 5 seconds) but with a frequency limiter (e.g., only every 60th tick) to avoid excessive Docker API calls. - Add logging for each removed orphan. **Note:** Ensure labels are set when creating containers (check existing `_create_container` – likely already has labels). If not, add label `managed_by=devplace` to simplify sweep. --- ## Definition of Done - [ ] Change 1 implemented: `seed_default_schedule` failure is logged at `WARNING` level in `devplacepy/services/backup/service.py`. - [ ] Change 2 implemented: disk alert triggers audit event `disk.usage.critical` and (if available) a Devii notification when usage > 85% after a scheduled backup. - [ ] Change 3 implemented: database size is recorded on each backup; daily growth >10% triggers audit event `backup.db_growth_high` and a warning log. - [ ] Change 4 implemented: orphaned Docker containers (no matching `instances` row) are stopped and removed at most once per 60 ticks in `devplacepy/services/containers/service.py`. - [ ] All changes are covered by existing unit tests; no new test failures introduced. - [ ] `pip install -e '.[dev]' -q && make test-unit` passes with zero failures. - [ ] No new lint violations introduced beyond those fixable with `ruff --fix` (if any).