Fix #73: Configure automated backup schedule and address critical disk usage #120

Open

typosaurus wants to merge 2 commits from typosaurus/ticket-73 into master

typosaurus commented

2026-07-23 04:06:10 +02:00

Collaborator

Resolves #73.

Plan

Implementation Plan: Backup Schedule & Disk Cleanup Gaps

Based on investigation, the original ticket contains several inaccuracies. The actual four gaps are:

seed_default_schedule() failure silently swallowed – if the first service init races with DB readiness, no schedule is created and no warning is raised.
No proactive disk alerting – disk warnings are shown on the dashboard but never pushed via email or Devii message.
No database growth rate monitoring – size tracked but not trended or alerted.
No bulk sweep of stopped containers for hard-deleted instances – _reconcile handles ST_REMOVING rows, but if the instance row is gone, the Docker container is orphaned indefinitely.

None of these changes require adding a new backup schedule or rotation policy (those already exist). The plan below addresses only the true gaps.

Change 1: Log and propagate `seed_default_schedule()` failures

File: devplacepy/services/backup/service.py

In _init_backend() (around line 38–40), change the silent except to log the exception and optionally re-raise after initialisation completes (so the service still starts but the error is visible).
Add a logger.warning(...) call when seed_default_schedule returns False or raises.

# Before (approx line 38-40):
try:
    seed_default_schedule()
except Exception:
    pass

# After:
try:
    seed_default_schedule()
except Exception as exc:
    logger.warning("Failed to seed default backup schedule: %s", exc)

Change 2: Proactive disk alerting

File: devplacepy/services/backup/store.py (the compute_storage_stats function or a new method)

After computing disk usage, if percent_used > config.DISK_WARNING_THRESHOLD (default 85), call a new helper _send_disk_alert(percent_used).
The helper should write an audit event ("disk.usage.critical") and, if a Devii notification channel is available, send a message to the admin user(s). For now, use the existing audit.log_event and add a simple in-app notification via the existing NotificationService (if available; otherwise just an audit event and a log entry).
To keep it minimal, trigger the check at the end of every scheduled backup (inside execute_schedule()).

File: devplacepy/services/backup/service.py

In execute_schedule() (around line 110), after backup and rotation completes, call store.check_disk_and_alert().

Change 3: Database growth rate monitoring

File: devplacepy/services/backup/store.py

Add a small CSV or simple in-memory/DB table to record database size at each backup.
On each scheduled backup, compute (current_size - last_size) / days_since_last. If the daily growth exceeds a threshold (e.g., 10% of last size per day), log a warning and write an audit event "backup.db_growth_high".
Threshold can be a config constant DB_GROWTH_ALERT_PCT default 0.1 (10% daily).
The historical data is stored as a new key in the existing state.json (or a separate JSON file in the backup directory). Use the same pattern as the backup state.

Change 4: Bulk sweep of orphaned containers from hard-deleted instances

File: devplacepy/services/containers/service.py

In _reconcile() (around line 170–175), after handling ST_REMOVING instances, add a new method _sweep_orphaned_containers() that:
1. Lists all Docker containers with label managed_by=devplace (if such label exists; otherwise query by known naming pattern).
2. For each container, check if a corresponding record exists in instances table.
3. If not, stop and remove the container (with graceful timeout).
Call this method once per tick (every 5 seconds) but with a frequency limiter (e.g., only every 60th tick) to avoid excessive Docker API calls.
Add logging for each removed orphan.

Note: Ensure labels are set when creating containers (check existing _create_container – likely already has labels). If not, add label managed_by=devplace to simplify sweep.

Definition of Done

Change 1 implemented: seed_default_schedule failure is logged at WARNING level in devplacepy/services/backup/service.py.
Change 2 implemented: disk alert triggers audit event disk.usage.critical and (if available) a Devii notification when usage > 85% after a scheduled backup.
Change 3 implemented: database size is recorded on each backup; daily growth >10% triggers audit event backup.db_growth_high and a warning log.
Change 4 implemented: orphaned Docker containers (no matching instances row) are stopped and removed at most once per 60 ticks in devplacepy/services/containers/service.py.
All changes are covered by existing unit tests; no new test failures introduced.
pip install -e '.[dev]' -q && make test-unit passes with zero failures.
No new lint violations introduced beyond those fixable with ruff --fix (if any).

Opened automatically by Typosaurus.

Resolves #73. ## Plan ## Implementation Plan: Backup Schedule & Disk Cleanup Gaps Based on investigation, the original ticket contains several inaccuracies. The actual four gaps are: 1. **`seed_default_schedule()` failure silently swallowed** – if the first service init races with DB readiness, no schedule is created and no warning is raised. 2. **No proactive disk alerting** – disk warnings are shown on the dashboard but never pushed via email or Devii message. 3. **No database growth rate monitoring** – size tracked but not trended or alerted. 4. **No bulk sweep of stopped containers for hard-deleted instances** – `_reconcile` handles `ST_REMOVING` rows, but if the instance row is gone, the Docker container is orphaned indefinitely. None of these changes require adding a new backup schedule or rotation policy (those already exist). The plan below addresses only the true gaps. ### Change 1: Log and propagate `seed_default_schedule()` failures **File:** `devplacepy/services/backup/service.py` - In `_init_backend()` (around line 38–40), change the silent `except` to log the exception and optionally re-raise after initialisation completes (so the service still starts but the error is visible). - Add a `logger.warning(...)` call when `seed_default_schedule` returns `False` or raises. ```python # Before (approx line 38-40): try: seed_default_schedule() except Exception: pass # After: try: seed_default_schedule() except Exception as exc: logger.warning("Failed to seed default backup schedule: %s", exc) ``` ### Change 2: Proactive disk alerting **File:** `devplacepy/services/backup/store.py` (the `compute_storage_stats` function or a new method) - After computing disk usage, if `percent_used > config.DISK_WARNING_THRESHOLD` (default 85), call a new helper `_send_disk_alert(percent_used)`. - The helper should write an audit event (`"disk.usage.critical"`) and, if a Devii notification channel is available, send a message to the admin user(s). For now, use the existing `audit.log_event` and add a simple in-app notification via the existing `NotificationService` (if available; otherwise just an audit event and a log entry). - To keep it minimal, trigger the check at the end of every scheduled backup (inside `execute_schedule()`). **File:** `devplacepy/services/backup/service.py` - In `execute_schedule()` (around line 110), after backup and rotation completes, call `store.check_disk_and_alert()`. ### Change 3: Database growth rate monitoring **File:** `devplacepy/services/backup/store.py` - Add a small CSV or simple in-memory/DB table to record database size at each backup. - On each scheduled backup, compute `(current_size - last_size) / days_since_last`. If the daily growth exceeds a threshold (e.g., 10% of last size per day), log a warning and write an audit event `"backup.db_growth_high"`. - Threshold can be a config constant `DB_GROWTH_ALERT_PCT` default 0.1 (10% daily). - The historical data is stored as a new key in the existing `state.json` (or a separate JSON file in the backup directory). Use the same pattern as the backup state. ### Change 4: Bulk sweep of orphaned containers from hard-deleted instances **File:** `devplacepy/services/containers/service.py` - In `_reconcile()` (around line 170–175), after handling `ST_REMOVING` instances, add a new method `_sweep_orphaned_containers()` that: 1. Lists all Docker containers with label `managed_by=devplace` (if such label exists; otherwise query by known naming pattern). 2. For each container, check if a corresponding record exists in `instances` table. 3. If not, stop and remove the container (with graceful timeout). - Call this method once per tick (every 5 seconds) but with a frequency limiter (e.g., only every 60th tick) to avoid excessive Docker API calls. - Add logging for each removed orphan. **Note:** Ensure labels are set when creating containers (check existing `_create_container` – likely already has labels). If not, add label `managed_by=devplace` to simplify sweep. --- ## Definition of Done - [ ] Change 1 implemented: `seed_default_schedule` failure is logged at `WARNING` level in `devplacepy/services/backup/service.py`. - [ ] Change 2 implemented: disk alert triggers audit event `disk.usage.critical` and (if available) a Devii notification when usage > 85% after a scheduled backup. - [ ] Change 3 implemented: database size is recorded on each backup; daily growth >10% triggers audit event `backup.db_growth_high` and a warning log. - [ ] Change 4 implemented: orphaned Docker containers (no matching `instances` row) are stopped and removed at most once per 60 ticks in `devplacepy/services/containers/service.py`. - [ ] All changes are covered by existing unit tests; no new test failures introduced. - [ ] `pip install -e '.[dev]' -q && make test-unit` passes with zero failures. - [ ] No new lint violations introduced beyond those fixable with `ruff --fix` (if any). Opened automatically by Typosaurus.

typosaurus added 2 commits 2026-07-23 04:06:11 +02:00