Fix circular import, primary-admin NULL trap, and add gateway quota reset
DevPlace CI / test (push) Failing after 58m59s

Restores a working import graph and closes two data-correctness bugs, plus
adds a reset for the AI gateway's rolling 24h spend.

Circular import: database/__init__ -> engagement -> content -> utils ->
database made the package unimportable. get_project_devlog moves out of
database/engagement.py into content.py, where enrich_items already lives.

Primary administrator: _can_hold_primary_admin read is_active with
bool(row.get("is_active")), so an admin row whose is_active column is SQL
NULL (any row predating the column) was treated as deactivated and skipped.
Every other site defaults an unknown is_active to active; this one now does
too.

Profile JSON: xp_next_level and xp_progress_pct were computed but only put on
the top-level context, never on profile_user, so they serialised as null even
though UserOut declares them and the API docs document them as embedded there.

Gateway quota reset: a cap previously lifted only with the passage of time.
quota.reset upserts a watermark row into gateway_quota_resets, scoped by the
same three nullable dimensions as a quota rule, and spent_24h sums from
max(24h cutoff, watermark). No ledger row is deleted, so the cost analytics on
/admin/ai-usage stay intact. Reaches every surface: POST
/admin/gateway/quota-resets, a per-rule Reset spend button, the Devii tool
gateway_quota_reset (confirm-gated), devplace gateway quota reset, and the API
docs. Admin's Reset all quotas now stamps a global gateway watermark too,
which is what a caller stuck on "AI gateway daily quota exceeded" needed.

Startup: _backfill_gamification swept every xp=0 user on every boot in every
worker and could never converge, since a user with no content earns no XP.
It now intersects pending users with _milestone_candidates(). db.tables is a
live reflection, so it is hoisted out of the loops that probed it per row.

Docker: the dependency layer now depends on pyproject.toml only, so a source
edit no longer reinstalls every dependency and re-downloads Chromium.
Adds start_interval so the healthcheck probes during the start period, and a
docker-reload target, since docker-up does not restart an unchanged container.

Adds events.md, the audit event catalogue that README, CLAUDE.md, the quiz
docs and the tooling all referenced but which never existed: 288 keys across
28 categories, including the families built from a variable at the call site.

Test fixes: both devlog helpers dated post 0 as the newest while the tests
assumed post 2 was; a profile login posted username= to a form that takes
email=; a devlog assertion matched six buttons under strict mode; and the
primary-admin tests seeded founders newer than the back-dated fixture admin,
so they only passed without the api tier.

Full suite: 2989 passed, 1 skipped.
This commit is contained in:
2026-07-27 11:17:48 +02:00
parent b185574760
commit 571a0485c5
33 changed files with 1290 additions and 92 deletions
+11
View File
@@ -105,6 +105,17 @@ if "comments" not in db.tables:
`init_db()` ends with `_refresh_query_planner_stats()`: a one-time `ANALYZE` when `sqlite_stat1` is absent, then `PRAGMA optimize` every boot (both inside `with db:`). Validate any index change with `EXPLAIN QUERY PLAN` against `data/devplace.db` (or a copy) - a correct plan reads `SEARCH ... USING INDEX <name>` with no temp b-tree.
## Startup backfills must converge (hard rule)
`init_db()` runs inside `lifespan` under an exclusive `init_lock()`, **before the worker accepts a single request**, and every uvicorn worker runs it in turn. Time-to-serving is therefore `workers x init_db`, so anything added there is paid N times on every boot and every deploy. Two rules follow:
- **A backfill must be able to finish.** A backfill selects the rows that still need migrating and must leave them *not* selected afterwards. If a row can stay in the candidate set after a successful pass, the "one-time migration" is really a permanent per-boot scan that grows with the table. `_backfill_gamification` had exactly this bug: it selected `users.find(xp=0)` and then ran the full milestone sweep over *every* one of them, but a user with no content is awarded no XP, so they stayed at `xp=0` and were re-swept forever. At 7814 such users that was 36s of the 37.5s boot - about 140k queries that provably could not award anything, on every worker, on every restart.
- **Never fan a per-row query out over a whole table at boot.** Compute the candidate set with a few set-based `GROUP BY`/`DISTINCT` queries first, then do per-row work only for rows that survive. `_milestone_candidates()` is the pattern: one `SELECT DISTINCT` per milestone source table (`MILESTONE_SOURCES`), unioned into a set, intersected with the pending users. A user absent from all of those tables scores 0 on every milestone metric and the lowest threshold is 5, so skipping them cannot change any award - verified by diffing the full `badges` table between the full sweep and the narrowed one (identical, 9800 rows, 22.6s -> 0.9s).
Watch for the same shape in `dataset` internals: `db.tables` is a live SQLAlchemy reflection, not a cached attribute. `get_user_stars` does one `in db.tables` check per `STAR_TARGETS` entry, so a per-user loop calling it re-reflects the whole table list on every iteration - 78667 reflections costing 17.7s in the profile above. Hoist `db.tables` into a local when looping.
Profile with the real database before and after any change here (`cProfile` around `init_db()` against a copy of `data/devplace.db`); a synthetic or empty DB hides every one of these costs.
## Project-wide soft delete (hard rule)
Attachments (see "Profile media gallery and soft-deleted attachments" below) are one instance of a platform-wide model: **every removal is a soft delete; only garbage collection is a hard delete.** Every removable row carries two columns, `deleted_at` (ISO timestamp) and `deleted_by` (actor uid, or `system`). A live row has `deleted_at = NULL`, and every list/count read filters `deleted_at IS NULL`.
+1 -2
View File
@@ -8,7 +8,7 @@ from .users import get_users_by_uids, _admins_cache, invalidate_admins_cache, ge
from .relations import _relations_cache, get_user_relations, get_blocked_uids, get_muted_uids, get_silenced_uids, invalidate_user_relations
from .pagination import PAGE_SIZE, paginate, interleave_by_author, paginate_diverse, get_user_post_count, clear_user_post_count, build_pagination
from .soft_delete import SOFT_DELETE_TABLES, ensure_soft_delete_columns, soft_delete, soft_delete_in, restore, purge, list_deleted, count_deleted, restore_event, purge_event
from .engagement import _comment_count_cache, get_comment_counts_by_post_uids, get_post_counts_by_user_uids, get_project_devlog, get_vote_counts, get_user_votes, get_reactions_by_targets, get_user_bookmarks, get_polls_by_post_uids, get_poll_for_post
from .engagement import _comment_count_cache, get_comment_counts_by_post_uids, get_post_counts_by_user_uids, get_vote_counts, get_user_votes, get_reactions_by_targets, get_user_bookmarks, get_polls_by_post_uids, get_poll_for_post
from .usage import _add_usage, _get_usage, add_correction_usage, get_correction_usage, add_modifier_usage, get_modifier_usage, NEWS_USAGE_KEY, add_news_usage, get_news_usage, ISSUE_USAGE_KEY, add_issue_usage, get_issue_usage, SEO_USAGE_KEY, add_seo_usage, get_seo_usage, AWARD_USAGE_KEY, add_award_usage, get_award_usage
from .awards import (
AWARDS_PER_PAGE,
@@ -115,7 +115,6 @@ __all__ = [
"_comment_count_cache",
"get_comment_counts_by_post_uids",
"get_post_counts_by_user_uids",
"get_project_devlog",
"get_vote_counts",
"get_user_votes",
"get_reactions_by_targets",
+1 -24
View File
@@ -1,9 +1,6 @@
# retoor <retoor@molodetz.nl>
from .core import TTLCache, _in_clause, db, defaultdict, get_table
from .pagination import paginate
from devplacepy.content import enrich_items
from .users import get_users_by_uids
from .core import TTLCache, _in_clause, db, defaultdict
_comment_count_cache = TTLCache(ttl=15, max_size=10000)
@@ -190,23 +187,3 @@ def get_poll_for_post(post_uid, user=None):
return get_polls_by_post_uids([post_uid], user).get(post_uid)
def get_project_devlog(project_uid: str, before: str | None = None, viewer: dict | None = None) -> tuple[list, str | None]:
posts_table = get_table("posts")
posts, next_cursor = paginate(
posts_table,
before=before,
viewer_uid=viewer["uid"] if viewer else None,
project_uid=project_uid,
)
if not posts:
return [], None
authors = get_users_by_uids([p["user_uid"] for p in posts])
counts = get_comment_counts_by_post_uids([p["uid"] for p in posts])
result = enrich_items(
posts, "post", authors, {"comment_count": counts}, user=viewer
)
return result, next_cursor
+12 -9
View File
@@ -32,12 +32,13 @@ def _ranked_authors() -> list:
cached = _authors_cache.get("ranked")
if cached is not None:
return cached
tables = db.tables
sources = [
(target_type, table_name)
for target_type, table_name in VOTABLE_TARGETS.items()
if table_name in db.tables
if table_name in tables
]
if "votes" not in db.tables or not sources:
if "votes" not in tables or not sources:
_authors_cache.set("ranked", [])
return []
target_union = " UNION ALL ".join(
@@ -101,12 +102,13 @@ def get_user_stars(user_uid: str) -> int:
cached = _stars_cache.get(user_uid)
if cached is not None:
return cached
if "votes" not in db.tables:
tables = db.tables
if "votes" not in tables:
return 0
target_union = " UNION ALL ".join(
f"SELECT uid, '{target_type}' AS target_type FROM {table_name} WHERE user_uid = :u AND deleted_at IS NULL"
for target_type, table_name in VOTABLE_TARGETS.items()
if table_name in db.tables
if table_name in tables
)
if not target_union:
return 0
@@ -154,7 +156,8 @@ def delete_engagement(target_type: str, target_uids: list) -> None:
uids = [uid for uid in (target_uids or []) if uid]
if not uids:
return
if "reactions" in db.tables:
tables = db.tables
if "reactions" in tables:
placeholders, params = _in_clause(uids)
params["tt"] = target_type
with db:
@@ -162,7 +165,7 @@ def delete_engagement(target_type: str, target_uids: list) -> None:
f"DELETE FROM reactions WHERE target_type=:tt AND target_uid IN ({placeholders})",
**params,
)
if "bookmarks" in db.tables:
if "bookmarks" in tables:
placeholders, params = _in_clause(uids)
params["tt"] = target_type
with db:
@@ -170,12 +173,12 @@ def delete_engagement(target_type: str, target_uids: list) -> None:
f"DELETE FROM bookmarks WHERE target_type=:tt AND target_uid IN ({placeholders})",
**params,
)
if target_type == "post" and "polls" in db.tables:
if target_type == "post" and "polls" in tables:
for uid in uids:
for poll in db["polls"].find(post_uid=uid):
if "poll_votes" in db.tables:
if "poll_votes" in tables:
db["poll_votes"].delete(poll_uid=poll["uid"])
if "poll_options" in db.tables:
if "poll_options" in tables:
db["poll_options"].delete(poll_uid=poll["uid"])
db["polls"].delete(post_uid=uid)
+30 -2
View File
@@ -1761,6 +1761,29 @@ def backfill_api_keys() -> int:
return updated
MILESTONE_SOURCES = (
("posts", "user_uid"),
("comments", "user_uid"),
("projects", "user_uid"),
("gists", "user_uid"),
("follows", "follower_uid"),
("follows", "following_uid"),
("user_activity", "user_uid"),
)
def _milestone_candidates() -> set:
tables = db.tables
candidates = set()
for table, column in MILESTONE_SOURCES:
if table not in tables:
continue
for row in db.query(f"SELECT DISTINCT {column} AS uid FROM {table}"):
if row["uid"]:
candidates.add(row["uid"])
return candidates
def _backfill_gamification():
if "users" not in db.tables:
return
@@ -1824,7 +1847,12 @@ def _backfill_gamification():
)
_authors_cache.clear()
for user in pending:
candidates = _milestone_candidates()
checked = [user for user in pending if user["uid"] in candidates]
for user in checked:
check_milestone_badges(user["uid"])
logger.info(f"Gamification backfill processed {len(pending)} users")
logger.info(
f"Gamification backfill processed {len(pending)} users, "
f"{len(checked)} with milestone-eligible activity"
)
+4 -1
View File
@@ -77,7 +77,10 @@ def get_online_users(cutoff_iso: str, limit: int = 30) -> list:
def _can_hold_primary_admin(row, tracks_active):
if row.get("deleted_at"):
return False
return not tracks_active or bool(row.get("is_active"))
if not tracks_active:
return True
is_active = row.get("is_active")
return is_active is None or bool(is_active)
def get_primary_admin_uid():