Fix circular import, primary-admin NULL trap, and add gateway quota reset

Restores a working import graph and closes two data-correctness bugs, plus adds a reset for the AI gateway's rolling 24h spend. Circular import: database/__init__ -> engagement -> content -> utils -> database made the package unimportable. get_project_devlog moves out of database/engagement.py into content.py, where enrich_items already lives. Primary administrator: _can_hold_primary_admin read is_active with bool(row.get("is_active")), so an admin row whose is_active column is SQL NULL (any row predating the column) was treated as deactivated and skipped. Every other site defaults an unknown is_active to active; this one now does too. Profile JSON: xp_next_level and xp_progress_pct were computed but only put on the top-level context, never on profile_user, so they serialised as null even though UserOut declares them and the API docs document them as embedded there. Gateway quota reset: a cap previously lifted only with the passage of time. quota.reset upserts a watermark row into gateway_quota_resets, scoped by the same three nullable dimensions as a quota rule, and spent_24h sums from max(24h cutoff, watermark). No ledger row is deleted, so the cost analytics on /admin/ai-usage stay intact. Reaches every surface: POST /admin/gateway/quota-resets, a per-rule Reset spend button, the Devii tool gateway_quota_reset (confirm-gated), devplace gateway quota reset, and the API docs. Admin's Reset all quotas now stamps a global gateway watermark too, which is what a caller stuck on "AI gateway daily quota exceeded" needed. Startup: _backfill_gamification swept every xp=0 user on every boot in every worker and could never converge, since a user with no content earns no XP. It now intersects pending users with _milestone_candidates(). db.tables is a live reflection, so it is hoisted out of the loops that probed it per row. Docker: the dependency layer now depends on pyproject.toml only, so a source edit no longer reinstalls every dependency and re-downloads Chromium. Adds start_interval so the healthcheck probes during the start period, and a docker-reload target, since docker-up does not restart an unchanged container. Adds events.md, the audit event catalogue that README, CLAUDE.md, the quiz docs and the tooling all referenced but which never existed: 288 keys across 28 categories, including the families built from a variable at the call site. Test fixes: both devlog helpers dated post 0 as the newest while the tests assumed post 2 was; a profile login posted username= to a form that takes email=; a devlog assertion matched six buttons under strict mode; and the primary-admin tests seeded founders newer than the back-dated fixture admin, so they only passed without the api tier. Full suite: 2989 passed, 1 skipped.
2026-07-27 11:17:48 +02:00
parent b185574760
commit 571a0485c5
33 changed files with 1290 additions and 92 deletions
@@ -73,7 +73,11 @@ The gateway records one row per upstream call (chat, vision, passthrough) and su

 **429 body never carries a dollar figure**, admin or not (`{"detail": "AI gateway daily quota exceeded"}`) - mirrors Devii's own over-limit WS message, which likewise never states a number. The admin-only services log line and the `ai.quota.exceeded` audit row (`GatewayService._audit_quota_exceeded`, reusing `usage.audit_actor_for`) do carry the spend/limit/matched-rule-uid, since those are admin-only surfaces.

-**CRUD.** Admin JSON at `/admin/gateway/quota-rules` (`routers/admin/gateway_configs.py`, list returns each rule's live `spent_24h_usd` plus the Layer A defaults for context), audited `gateway.quota_rule.update`/`gateway.quota_rule.delete` (category `ai`, both already in `events.md`), rendered in the **Quota rules** section of `/admin/gateway` (`GatewayAdmin.js`, mirrors the providers/models CRUD tables). Devii tools `gateway_quota_rules`/`gateway_quota_rule_set`/`gateway_quota_rule_delete` (`requires_admin=True`, delete is `CONFIRM_REQUIRED`) proxy the same endpoints via `handler="http"`, same as the provider/model tools. CLI: `devplace gateway quota list|set|delete`.
+**Resetting the counted spend (`gateway_quota_resets`).** A cap is only lifted by *time* otherwise, so there is a reset that clears what has been counted **without deleting any ledger row** - `gateway_usage_ledger` is the cost-analytics source for `/admin/ai-usage`, so a reset must never truncate it. `quota.reset(QuotaResetIn, created_by=)` upserts one watermark row into `gateway_quota_resets` (same `ensure_tables()`/`"gateway_quota"` cache-version/hard-CRUD shape as the rules table, same two indexes) scoped by the SAME three nullable dimensions as a rule, and `quota.spent_24h` sums from `max(24h cutoff, reset_watermark(scope))`. A reset row applies to a queried scope when each of its non-null dimensions equals that scope's - so an all-null reset clears everyone, while a reset scoped to one app deliberately does NOT clear a broader per-user-all-apps scope (clearing a narrower window can only over-credit). Spend recorded after the reset counts again immediately against the same limit. `QuotaScopeIn` is the shared base holding the three dimensions and their validators; `QuotaRuleIn` and `QuotaResetIn` both extend it, so scope parsing exists once.
+
+**The two AI quotas are separate systems and the reset surfaces must say so.** `/admin/ai-usage`'s *Reset all quotas* clears the Devii `devii_usage_ledger` AND now also stamps a global gateway watermark, because a caller hitting `429 AI gateway daily quota exceeded` had no reset at all before and the button looked global. *Reset guest quotas* stays Devii-only (guest gateway calls ride the shared internal key, so there is no per-guest gateway scope to clear).
+
+**CRUD.** Admin JSON at `/admin/gateway/quota-rules` (`routers/admin/gateway_configs.py`, list returns each rule's live `spent_24h_usd` plus the Layer A defaults for context), audited `gateway.quota_rule.update`/`gateway.quota_rule.delete` (category `ai`, both already in `events.md`), rendered in the **Quota rules** section of `/admin/gateway` (`GatewayAdmin.js`, mirrors the providers/models CRUD tables). Devii tools `gateway_quota_rules`/`gateway_quota_rule_set`/`gateway_quota_rule_delete` (`requires_admin=True`, delete is `CONFIRM_REQUIRED`) proxy the same endpoints via `handler="http"`, same as the provider/model tools. Reset is `POST /admin/gateway/quota-resets` (same file, `_payload`/`ValidationError` shape as the rule CRUD), audited `gateway.quota.reset` (category `ai`), surfaced as a per-rule **Reset spend** button in the Quota rules table (`GatewayAdmin.js`), and exposed as the Devii tool `gateway_quota_reset` (`requires_admin=True`, in `CONFIRM_REQUIRED` with a declared `confirm` param, like the other quota-lifting admin resets). CLI: `devplace gateway quota list|set|delete|reset`.

 ## Image generation

@@ -16,6 +16,7 @@ from devplacepy.database import bump_cache_version, db, get_table, sync_local_ca
 logger = logging.getLogger(__name__)

 RULES_TABLE = "gateway_quota_rules"
+RESETS_TABLE = "gateway_quota_resets"
 CACHE_NAME = "gateway_quota"
 APP_REFERENCE_PATTERN = re.compile(r"^[a-zA-Z0-9_.-]{1,30}$")
 OWNER_KINDS = ("internal", "key", "user", "admin", "anonymous")
@@ -62,22 +63,38 @@ def ensure_tables() -> None:
        )
    except Exception as exc:
        logger.warning("gateway quota rule index creation failed: %s", exc)
+    db.query(
+        "CREATE TABLE IF NOT EXISTS "
+        + RESETS_TABLE
+        + " (id INTEGER PRIMARY KEY, uid TEXT, owner_kind TEXT, owner_id TEXT, "
+        "app_reference TEXT, reset_at TEXT, created_by TEXT)"
+    )
+    try:
+        db.query(
+            "CREATE UNIQUE INDEX IF NOT EXISTS idx_gateway_quota_resets_uid ON "
+            + RESETS_TABLE
+            + " (uid)"
+        )
+        db.query(
+            "CREATE INDEX IF NOT EXISTS idx_gateway_quota_resets_lookup ON "
+            + RESETS_TABLE
+            + " (owner_kind, owner_id, app_reference)"
+        )
+    except Exception as exc:
+        logger.warning("gateway quota reset index creation failed: %s", exc)


-class QuotaRuleIn(BaseModel):
+class QuotaScopeIn(BaseModel):
    owner_kind: Optional[str] = None
    owner_id: Optional[str] = Field(default=None, max_length=64)
    app_reference: Optional[str] = Field(default=None, max_length=30)
-    limit_usd: float = Field(default=0.0, ge=0)
-    is_active: bool = True
-    label: str = Field(default="", max_length=200)

    @field_validator("owner_kind")
    @classmethod
    def _clean_owner_kind(cls, value: Optional[str]) -> Optional[str]:
+        value = (value or "").strip().lower()
        if not value:
            return None
-        value = value.strip().lower()
        if value not in OWNER_KINDS:
            raise ValueError(f"owner_kind must be one of {', '.join(OWNER_KINDS)}")
        return value
@@ -85,20 +102,24 @@ class QuotaRuleIn(BaseModel):
    @field_validator("owner_id")
    @classmethod
    def _clean_owner_id(cls, value: Optional[str]) -> Optional[str]:
-        if not value:
-            return None
-        return value.strip()
+        return (value or "").strip() or None

    @field_validator("app_reference")
    @classmethod
    def _clean_app_reference(cls, value: Optional[str]) -> Optional[str]:
+        value = (value or "").strip()
        if not value:
            return None
-        value = value.strip()
        if not APP_REFERENCE_PATTERN.match(value):
            raise ValueError("app_reference must match ^[a-zA-Z0-9_.-]{1,30}$")
        return value

+
+class QuotaRuleIn(QuotaScopeIn):
+    limit_usd: float = Field(default=0.0, ge=0)
+    is_active: bool = True
+    label: str = Field(default="", max_length=200)
+
    @field_validator("label")
    @classmethod
    def _clean_label(cls, value: str) -> str:
@@ -114,6 +135,10 @@ class QuotaRuleIn(BaseModel):
        return self


+class QuotaResetIn(QuotaScopeIn):
+    pass
+
+
@dataclass(frozen=True)
 class QuotaRule:
    uid: str
@@ -250,6 +275,83 @@ class QuotaRuleStore:
 quota_rule_store = QuotaRuleStore()


+def _load_resets() -> list[dict]:
+    sync_local_cache(CACHE_NAME, _QUOTA_CACHE)
+    if "resets" not in _QUOTA_CACHE:
+        resets: list[dict] = []
+        try:
+            if RESETS_TABLE in db.tables:
+                for row in get_table(RESETS_TABLE).all():
+                    if row.get("reset_at"):
+                        resets.append(
+                            {
+                                "owner_kind": row.get("owner_kind") or None,
+                                "owner_id": row.get("owner_id") or None,
+                                "app_reference": row.get("app_reference") or None,
+                                "reset_at": str(row["reset_at"]),
+                            }
+                        )
+        except Exception as exc:
+            logger.warning("gateway quota reset load failed: %s", exc)
+        _QUOTA_CACHE["resets"] = resets
+    return _QUOTA_CACHE["resets"]
+
+
+def _reset_applies(reset: dict, scope: dict) -> bool:
+    for field in ("owner_kind", "owner_id", "app_reference"):
+        wanted = reset.get(field)
+        if wanted is None:
+            continue
+        if scope.get(field) is None or scope[field] != wanted:
+            return False
+    return True
+
+
+def reset_watermark(
+    owner_kind: Optional[str], owner_id: Optional[str], app_reference: Optional[str]
+) -> str:
+    scope = {
+        "owner_kind": owner_kind,
+        "owner_id": owner_id,
+        "app_reference": app_reference,
+    }
+    stamps = [r["reset_at"] for r in _load_resets() if _reset_applies(r, scope)]
+    return max(stamps) if stamps else ""
+
+
+def reset(payload: Optional[QuotaResetIn] = None, *, created_by: str = "") -> dict:
+    ensure_tables()
+    payload = payload or QuotaResetIn()
+    table = get_table(RESETS_TABLE)
+    scope = {
+        "owner_kind": payload.owner_kind,
+        "owner_id": payload.owner_id,
+        "app_reference": payload.app_reference,
+    }
+    stamp = _now()
+    existing = table.find_one(**scope)
+    if existing:
+        table.update({"id": existing["id"], "reset_at": stamp, "created_by": created_by}, ["id"])
+        uid = existing.get("uid") or uuid.uuid4().hex
+    else:
+        uid = uuid.uuid4().hex
+        table.insert({**scope, "uid": uid, "reset_at": stamp, "created_by": created_by})
+    bump_cache_version(CACHE_NAME)
+    _QUOTA_CACHE.clear()
+    return {**scope, "uid": uid, "reset_at": stamp}
+
+
+def scope_label(scope: dict, fallback: str = "") -> str:
+    parts = []
+    if scope.get("owner_kind"):
+        parts.append(f"role={scope['owner_kind']}")
+    if scope.get("owner_id"):
+        parts.append(f"user={scope['owner_id']}")
+    if scope.get("app_reference"):
+        parts.append(f"app={scope['app_reference']}")
+    return ", ".join(parts) or fallback
+
+
 def default_limit(owner_kind: str, cfg: dict) -> float:
    field = _DEFAULT_FIELD_BY_KIND.get(owner_kind, FIELD_DEFAULT_USER)
    return float(cfg.get(field, 0.0) or 0.0)
@@ -294,10 +396,10 @@ def spent_24h(

    if GATEWAY_LEDGER not in db.tables:
        return 0.0
+    cutoff = (datetime.now(timezone.utc) - timedelta(hours=24)).isoformat()
+    watermark = reset_watermark(owner_kind, owner_id, app_reference)
    clauses = ["created_at >= :cutoff"]
-    params: dict = {
-        "cutoff": (datetime.now(timezone.utc) - timedelta(hours=24)).isoformat()
-    }
+    params: dict = {"cutoff": max(cutoff, watermark) if watermark else cutoff}
    if owner_kind is not None:
        clauses.append("owner_kind = :owner_kind")
        params["owner_kind"] = owner_kind