diff --git a/CLAUDE.md b/CLAUDE.md index 932c21cd..51d53e08 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -119,6 +119,7 @@ Tests run on port 10501 with a tempfile SQLite DB and a dedicated `DEVPLACE_DATA | `DEVPLACE_WEB_WORKERS` / `--workers` | `nproc` (prod) | Uvicorn worker count; `make prod WEB_WORKERS=N` to override. | | `DEVPLACE_PRESENCE_TIMEOUT_SECONDS` | `60` | How long after a user's last activity they still count as online. `config.PRESENCE_WRITE_SECONDS` (half of it) throttles `last_seen` writes per worker. | | `DEVPLACE_PRESENCE_ONLINE_LIMIT` | `30` | Max avatars in the feed's live "Online now" panel. | +| `DEVPLACE_PRESENCE_TRACK_LIMIT` | `500` | Size of the online set the presence relay tracks and publishes as the authority for every avatar dot. `PRESENCE_ONLINE_LIMIT` only caps how many of them the feed panel *displays*. | | `DEVPLACE_PRESENCE_ONLINE_MARGIN_SECONDS` | `20` | Grace margin (hysteresis) before an online user is dropped, kills dot/roster flicker at the boundary. | | `DEVPLACE_DATA_DIR` | `/data` | Single root for ALL runtime/user-generated data OUTSIDE the package and OUTSIDE `/static`. Point at a volume in prod. | | `DEVPLACE_OUTBOUND_PROXY_URL` | unset | Fallback for the `outbound_proxy_url` site setting (below) when the DB/settings row is unavailable (early CLI contexts). Prefer configuring the setting via `/admin/settings` - it applies live with no restart. | diff --git a/README.md b/README.md index dcaacc0c..53c3b36b 100644 --- a/README.md +++ b/README.md @@ -227,6 +227,7 @@ The log is **administrator-only**. `/admin/audit-log` is a paginated, filterable | `DEVPLACE_PRESENCE_TIMEOUT_SECONDS` | `60` | Online-presence window: a user counts as online for this many seconds after their last activity. `last_seen` is refreshed by a throttled in-place update at most once per half this interval per worker (no per-load inserts, no data growth) | | `DEVPLACE_PRESENCE_ONLINE_LIMIT` | `30` | Maximum avatars shown in the feed's live "Online now" panel (ordered alphabetically by username) | | `DEVPLACE_PRESENCE_ONLINE_MARGIN_SECONDS` | `20` | Grace margin before an online user drops to offline (hysteresis): online at the timeout, offline only after timeout + this. Prevents online/offline flicker for users hovering at the boundary | +| `DEVPLACE_PRESENCE_TRACK_LIMIT` | `500` | Size of the online set the presence relay tracks and publishes on `public.presence.roster`. That one set is the single source of truth behind every avatar presence dot on every page; `DEVPLACE_PRESENCE_ONLINE_LIMIT` only caps how many of them the feed's Online now panel displays | ### Runtime settings diff --git a/devplacepy/config.py b/devplacepy/config.py index d3ba4432..85607809 100644 --- a/devplacepy/config.py +++ b/devplacepy/config.py @@ -50,6 +50,7 @@ SITE_URL = environ.get("DEVPLACE_SITE_URL", "").rstrip("/") PRESENCE_TIMEOUT_SECONDS = int(environ.get("DEVPLACE_PRESENCE_TIMEOUT_SECONDS", "60")) PRESENCE_WRITE_SECONDS = max(1, PRESENCE_TIMEOUT_SECONDS // 2) PRESENCE_ONLINE_LIMIT = int(environ.get("DEVPLACE_PRESENCE_ONLINE_LIMIT", "30")) +PRESENCE_TRACK_LIMIT = int(environ.get("DEVPLACE_PRESENCE_TRACK_LIMIT", "500")) PRESENCE_ONLINE_MARGIN_SECONDS = int( environ.get("DEVPLACE_PRESENCE_ONLINE_MARGIN_SECONDS", "20") ) diff --git a/devplacepy/database/CLAUDE.md b/devplacepy/database/CLAUDE.md index 3f70304e..dc9e7cd8 100644 --- a/devplacepy/database/CLAUDE.md +++ b/devplacepy/database/CLAUDE.md @@ -28,6 +28,8 @@ _index(db, "news", "idx_news_status", ["status"]) # now safe - column exists Do this with `get_table(name)` (NOT `if name in db.tables`): `get_table` + `create_column_by_example` creates the table if it is absent, so the schema is guaranteed regardless of which process or which insert touches the table first. Symptom to watch for: a page renders fine in isolation but shows empty data or 500s only after a CLI/maintenance test runs earlier in the suite (it created the table partially). +**This applies to `users` too, and an `if "users" not in db.tables: return` guard silently defeats it.** `backfill_api_keys()` is the `users` ensure-block (every non-signup column: `api_key`, `last_seen`, the AI correction/modifier settings, `avatar_seed`, `award_count`/`last_award_at`/`last_award_slug`/`last_award_uid`, ...). It used to early-return when `users` was absent, which is exactly the state on a **brand-new database**: `init_db()` runs before the first signup, so the ensure-block was skipped entirely, and the first `/auth/signup` then created `users` with only the columns of that INSERT. Every ensured-but-unwritten column was therefore missing from the long-running server's reflected metadata, so `users.find_one(...)` returned rows without them **for the whole process lifetime** - the feature reading them looked simply switched off (this is what made the awards tab, the prominent-award banner, and the avatar award badge invisible on a fresh DB, and it is invisible in production only because the columns happen to exist from an older boot). It now calls `get_table("users")` unconditionally, so the table is born with the full ensured column set. Never reintroduce a `db.tables` guard in front of a column-ensure block. + The `_index(...)` helper supports `where=` (partial) and `unique=` indexes; every table with a `uid` column gets a UNIQUE `idx__uid`, soft-delete tables get a PARTIAL `idx_
_trash` (`WHERE deleted_at IS NOT NULL`) and NEVER a bare `deleted_at` index (it mis-steers the planner on live reads), and "live newest-first" listings need a composite/live-partial index that carries the sort column (see "Indexing conventions" below). `init_db()` finishes with `ANALYZE`/`PRAGMA optimize`. Verify any index change with `EXPLAIN QUERY PLAN`. **SQLite is synchronous by design and will never be made async (hard rule).** The `dataset`/SQLite layer is synchronous and is called directly inside the async route handlers. This is intentional and more than fast enough for this platform: the database is a local file tuned with WAL, `synchronous=NORMAL`, a 30s busy timeout, an 8MB page cache, and a 256MB mmap, so queries are sub-millisecond and never meaningfully stall the event loop. Do NOT convert the database layer to async, do NOT wrap DB calls in a threadpool/`run_in_executor`/`to_thread`, and do NOT raise the synchronous DB access as a performance, blocking-I/O, or "requests aren't fully async" concern. This is a settled, sophisticated design choice and is not open for revisiting. diff --git a/devplacepy/database/schema.py b/devplacepy/database/schema.py index 5295c4a7..8f8eb7e8 100644 --- a/devplacepy/database/schema.py +++ b/devplacepy/database/schema.py @@ -1720,9 +1720,7 @@ def migrate_ai_gateway_settings() -> None: def backfill_api_keys() -> int: - if "users" not in db.tables: - return 0 - users = db["users"] + users = get_table("users") if not users.has_column("api_key"): users.create_column_by_example("api_key", "") if not users.has_column("created_at"): diff --git a/devplacepy/services/CLAUDE.md b/devplacepy/services/CLAUDE.md index 3104b5df..5f3633eb 100644 --- a/devplacepy/services/CLAUDE.md +++ b/devplacepy/services/CLAUDE.md @@ -179,12 +179,12 @@ Online status is a single **`users.last_seen`** UTC-ISO column (ensured in `data **Read path (any worker):** `presence.is_online(user_row)` = `now - last_seen < PRESENCE_TIMEOUT_SECONDS` (env `DEVPLACE_PRESENCE_TIMEOUT_SECONDS`, default 60). Profile (`routers/profile/index.py` -> `profile_online`) and messages (`routers/messages.py` seed) read `last_seen` off the user row they already loaded - no extra query. Exposed as the Jinja global `is_online(user)` (`templating.py`), on `UserOut.last_seen` and `ProfileOut.profile_online`. This is the **only** cross-worker-correct approach here because pub/sub is in-process. -**Live path (lock owner only), change-only + hysteresis:** `PresenceRelayService` (`services/presence_relay.py`, `BaseService`, default-enabled, 2s tick, registered in `main.py`) is a sibling of `NotificationRelayService`/`LiveViewRelayService`. Each tick it recomputes ONE global online set `self._online` (dot-subscribed uids batch-read via `get_users_by_uids` + roster candidates) and drives BOTH the per-user dots and the feed roster from that one set, so they can never disagree. The set uses **hysteresis** via `presence.stays_online(elapsed, was_online)`: a user becomes online at `PRESENCE_TIMEOUT_SECONDS` but only drops after `+ PRESENCE_ONLINE_MARGIN_SECONDS` (env `DEVPLACE_PRESENCE_ONLINE_MARGIN_SECONDS`, default 20) - **quick to go online, slow (grace margin) to go offline** - which kills boundary flicker for a user hovering near the timeout. It publishes `{online, last_seen}` to `public.presence.{uid}` **only when a topic's `online` bool changed OR the topic is newly subscribed (first-seen)** - never on a fixed interval, so a steady page emits nothing after the initial frame (`self._published[topic] -> bool`, pruned to active topics). `public.presence.{uid}` is subscribable by any logged-in user (`pubsub/policy.py` allows `public.*`); guests fall back to the server-rendered initial state. The one-directional grace also means dots and roster stay consistent across viewers (the shared `self._online` is the single authority). To keep it lightweight the relay reads all due users in **one batched `get_users_by_uids`** per tick. +**Live path (lock owner only), ONE set on ONE topic, change-only + hysteresis:** `PresenceRelayService` (`services/presence_relay.py`, `BaseService`, default-enabled, 2s tick, registered in `main.py`) is a sibling of `NotificationRelayService`/`LiveViewRelayService` and is **the single source of truth for live online status**. Each tick, only while the roster topic has subscribers, it reads the online population in ONE indexed query (`presence.online_candidates()`, capped at `config.PRESENCE_TRACK_LIMIT`, env `DEVPLACE_PRESENCE_TRACK_LIMIT`, default 500) and recomputes ONE set `self._online` with **hysteresis** via `presence.stays_online(elapsed, was_online)`: a user becomes online at `PRESENCE_TIMEOUT_SECONDS` but only drops after `+ PRESENCE_ONLINE_MARGIN_SECONDS` (env `DEVPLACE_PRESENCE_ONLINE_MARGIN_SECONDS`, default 20) - **quick to go online, slow (grace margin) to go offline** - which kills boundary flicker. It publishes that set on the ONE shared topic `public.presence.roster` (`roster_payload`: `{count, online: [uid...], users: [display rows]}`) **only when the set of online uids changes** (a `frozenset` compare, so reordering never republishes), never on a fixed interval, so an idle site emits nothing. `online` is the authority for EVERY avatar dot; `users` is the same set trimmed to `PRESENCE_ONLINE_LIMIT` for the feed's avatar panel, and `count` matches it. **There are no per-user `public.presence.{uid}` topics** - they were removed precisely because their candidate population differed from the roster's, so dots and roster could disagree (the /messages-vs-feed bug). One set, one topic, one frame. `public.presence.roster` is subscribable by any logged-in user (`pubsub/policy.py` allows `public.*`); guests keep the server-rendered initial state. -**Frontend:** `static/js/PresenceManager.js` (`app.presence`, constructed with `this.pubsub` in `Application.js`, mirroring `LocalTime`/`CounterManager`) scans `[data-presence-uid]` elements, subscribes each uid to `public.presence.{uid}` (deduped per uid, so a repeated author is one subscription), and treats each frame's `online` flag as **authoritative** (`entry.online`), toggling the `online` class + (for `data-presence-label` elements) the "online / last seen X / offline" text. Because the relay is change-only and authoritative, a live-subscribed dot is **not** expired by the client clock (no false-offline flicker for an active user whose `last_seen` the client cannot see advancing); the 20s `last_seen` staleness timer only applies to entries that never received a frame (guests / degraded). The window is read from ``. Both the profile `.profile-presence` dot and the messages `#messages-presence` span carry `data-presence-uid`/`data-presence-last-seen`. +**Frontend:** `static/js/PresenceManager.js` (`app.presence`, constructed with `this.pubsub` in `Application.js`, mirroring `LocalTime`/`CounterManager`) makes ONE subscription to `public.presence.roster` and keeps the pushed `online` uid set. It scans `[data-presence-uid]` elements (on load and via a `MutationObserver`, so dynamically inserted markup is covered) and renders each one purely as membership of that set - toggling the `online` class and, for `data-presence-label` elements, the "online / last seen X / offline" text, where the relative time is a `