ticket #71 attempt 2

.gitignore

@@ -32,3 +32,10 @@ var/
 .coverage
 .coverage.*
 htmlcov/
+
+# local environments and scratch
+.venv/
+tmp/
+*.log
+*.bak
+test.db

CLAUDE.md

@@ -2,6 +2,8 @@
 
 This file provides guidance to Claude Code when working with code in this repository. It holds only what applies regardless of which part of the codebase is being touched. Deep, subsystem-specific detail lives in nested `CLAUDE.md` files placed inside the relevant directory - Claude Code auto-loads a nested file only when a file under that directory is read or edited, so the always-loaded cost of this repository stays proportional to this file alone. See "Subsystem map" below for the full list.
 
+It is a big project, whatever you are implementing, it is probably done before. You should look it up and match the implementation structurely and visually. For inconsistency there is zero tolerance policy. Develop dry, kiss, re-usable code, consistent with existing implementation. Literally always try to find relatable examples before making a modification. If no-example exists, explain to user what is the case and let user decide what to do and how to continue. 
+
 ## Project
 
 DevPlace is a server-rendered social network for developers. FastAPI backend serves Jinja2 templates with pure ES6 module JavaScript on the frontend. SQLite via the `dataset` library (auto-syncs schema). No JS framework, no NPM, no JWT.
@@ -99,6 +101,7 @@ Tests run on port 10501 with a tempfile SQLite DB and a dedicated `DEVPLACE_DATA
 | `DEVPLACE_PRESENCE_ONLINE_LIMIT` | `30` | Max avatars in the feed's live "Online now" panel. |
 | `DEVPLACE_PRESENCE_ONLINE_MARGIN_SECONDS` | `20` | Grace margin (hysteresis) before an online user is dropped, kills dot/roster flicker at the boundary. |
 | `DEVPLACE_DATA_DIR` | `<repo>/data` | Single root for ALL runtime/user-generated data OUTSIDE the package and OUTSIDE `/static`. Point at a volume in prod. |
+| `DEVPLACE_OUTBOUND_PROXY_URL` | unset | Fallback for the `outbound_proxy_url` site setting (below) when the DB/settings row is unavailable (early CLI contexts). Prefer configuring the setting via `/admin/settings` - it applies live with no restart. |
 
 ## Subsystem map
 
@@ -152,7 +155,7 @@ Routers in `devplacepy/routers/` are organised as a **directory tree that mirror
 | `/auth` | auth/ package |
 | `/feed`, `/posts`, `/comments` | flat files |
 | `/projects`, `/projects/{slug}/files`, `/projects/{slug}/containers` | projects/ package - see `routers/projects/CLAUDE.md` |
-| `/profile` | profile/ package (customization, notifications, ai-correction, ai-modifier, telegram, usage) |
+| `/profile` | profile/ package (customization, notifications, ai-correction, ai-modifier, interactions, telegram, usage) |
 | `/messages` | messages.py - see `services/messaging/CLAUDE.md` |
 | `/notifications`, `/votes`, `/reactions`, `/bookmarks`, `/polls`, `/avatar`, `/follow`, `/leaderboard` | flat files |
 | (none) | relations.py - block/mute |
@@ -190,7 +193,9 @@ The CLIENT pipeline (`ContentRenderer.js`, `dp-content`/`dp-title`) is retained
 
 **Emoji shortcodes are the full GitHub/Discord `:name:` set** (~4869 names), generated once from the `emoji` library by `rendering.py` `build_emoji_shortcodes()`; the frontend gets the identical map via the generated `static/js/emoji-shortcodes.js` (regenerate with `devplace emoji-sync` after bumping the dependency, never hand-edit).
 
-**Template em-dash normalization:** the shared `templates.env.template_class` runs every rendered template's final HTML through `normalize_em_dash` - the em-dash character and its HTML entity forms all become a plain hyphen, application-wide. One hook; never re-strip em-dash per template.
+**Email anonymization:** both `_render_content` and `_render_title` mask email addresses to prevent doxing. `_mask_emails` in `rendering.py` runs on rendered text nodes only (inside `_transform_text`, `_MediaProcessor`, and `_InlineFilter` - never before mistune, or the `*` mask characters would be parsed as emphasis) and stars ~80% of the local part (keeps a leading 20%, minimum one visible char). Addresses on `molodetz.nl` (and its subdomains) are exempt and render verbatim.
+
+**Em-dash normalization:** `_normalize_dashes` in `rendering.py` replaces all forms (em dash `\u2014`, en dash `\u2013`, and their HTML entities) with a hyphen BEFORE mistune processes the text. Runs inside `_render_content`/`_render_title` which are `@lru_cache`d, so each unique text is normalized once. No per-template overhead.
 
 ### Auth
 
@@ -235,7 +240,7 @@ Devii is also reachable over **Telegram** (one supervised long-poller subprocess
 
 - **No comments, no docstrings in source.** Code is self-documenting.
 - **Forbidden variable name patterns:** `_new`, `_old`, `_temp`, `_v2`, `better_`, `my_`, `the_`.
-- **Outbound HTTP goes through the stealth client only.** Never instantiate a bare `httpx.AsyncClient(...)`/`httpx.Client(...)` for outbound traffic - build it via `devplacepy.stealth.stealth_async_client(...)`/`stealth_sync_client(...)`. The async factory routes bytes through **`curl_cffi` (curl-impersonate, BoringSSL)** behind an `httpx` transport adapter, so TLS JA3/JA4 and the HTTP/2 fingerprint match Chrome 146 byte-for-byte; falls back to a pure-httpx Chrome-aligned transport if `curl_cffi` is missing. Applies to news, gateway upstream, Devii/DeepSearch/SEO fetch, gitea, push, attachments. SSRF-guarded fetches via `net_guard.guarded_async_client(...)` get the fingerprint automatically. Sole exception: `routers/proxy.py` relays the user's own headers verbatim. **Load-bearing gotcha:** cleartext `http://` is forced to HTTP/1.1 (`curl_transport.http_version_for`) - the Chrome-impersonation profile defaults to HTTP/2, but a plaintext origin has no TLS/ALPN to negotiate h2 and uvicorn (HTTP/1.1 only) returns `400 "Invalid HTTP request received."` once a cleartext HTTP/2 body crosses ~128KB. Every internal AI consumer calls `INTERNAL_GATEWAY_URL` = `http://localhost:<port>/openai/v1/...`, so without this override a large Devii turn (big system prompt + tool list) fails on every request. `https://` keeps the HTTP/2 default. `stealth_sync_client` stays pure-httpx Chrome-aligned (no curl_cffi sync adapter).
+- **Outbound HTTP goes through the stealth client only.** Never instantiate a bare `httpx.AsyncClient(...)`/`httpx.Client(...)` for outbound traffic - build it via `devplacepy.stealth.stealth_async_client(...)`/`stealth_sync_client(...)`. The async factory routes bytes through **`curl_cffi` (curl-impersonate, BoringSSL)** behind an `httpx` transport adapter, so TLS JA3/JA4 and the HTTP/2 fingerprint match Chrome 146 byte-for-byte; falls back to a pure-httpx Chrome-aligned transport if `curl_cffi` is missing. Applies to news, gateway upstream, Devii/DeepSearch/SEO fetch, gitea, push, attachments. SSRF-guarded fetches via `net_guard.guarded_async_client(...)` get the fingerprint automatically. Sole exception: `routers/proxy.py` relays the user's own headers verbatim. **Load-bearing gotcha:** cleartext `http://` is forced to HTTP/1.1 (`curl_transport.http_version_for`) - the Chrome-impersonation profile defaults to HTTP/2, but a plaintext origin has no TLS/ALPN to negotiate h2 and uvicorn (HTTP/1.1 only) returns `400 "Invalid HTTP request received."` once a cleartext HTTP/2 body crosses ~128KB. Every internal AI consumer calls `INTERNAL_GATEWAY_URL` = `http://localhost:<port>/openai/v1/...`, so without this override a large Devii turn (big system prompt + tool list) fails on every request. `https://` keeps the HTTP/2 default. `stealth_sync_client` stays pure-httpx Chrome-aligned (no curl_cffi sync adapter). **Never bolt a hand-written browser-header dict (UA/`sec-ch-ua`/`Sec-Fetch-*`) onto a `stealth_async_client`/`stealth_sync_client` call** - curl_cffi's impersonation already emits an internally-consistent Chrome 146 identity across TLS, UA, and Client Hints; a caller-supplied header set claiming a different Chrome version or platform creates a UA-vs-TLS-vs-Client-Hints mismatch that is a textbook anti-bot tripwire (this was a real bug in Devii's `fetch_url`/`http_request` tool, fixed by deleting its hardcoded Chrome-131 header block). `stealth.detect_consent_gate(html)` recognizes the common JS-redirect cookie-consent wall pattern (DPG Media's `nu.nl` and sibling sites, and generically any page gated behind a `decodeURIComponent(...)` callback referencing a "privacy gate"/`consent.js`) - reuse it wherever a fetched page's real content sits behind that gate rather than special-casing a domain. `stealth_transport`/`stealth_async_client`/`stealth_sync_client` all accept a `proxy=` override, defaulting to `stealth.configured_proxy_url()` (the admin-editable `outbound_proxy_url` site setting, falling back to `DEVPLACE_OUTBOUND_PROXY_URL` when the setting is unset) - this is the only real fix for a destination that blocks by IP/ASN reputation (e.g. reddit.com returns an identical "blocked by network security" page across every TLS/header fingerprint tried, confirmed by testing curl_cffi's full impersonation list - it is blocking the datacenter ASN outright, not fingerprinting the client).
 - **Form validation is Pydantic-native.** Routers declare `data: Annotated[SomeForm, Form()]` (models in `models.py`); the global `RequestValidationError` handler re-renders auth pages with messages (400) or redirects other routes. Read raw `await request.form()` only when also reading an uploaded file alongside a model.
 - **`RedirectResponse(url=..., status_code=302)`** - always pass `status_code` as an integer literal, never `status.HTTP_302_FOUND`.
 - **Ownership checks:** compare `resource["user_uid"] == user["uid"]` before edit/delete (use `content.is_owner`). **Deletes additionally allow any admin:** `is_owner(...) or is_admin(user)` on every content delete endpoint, so an administrator may delete any member's content while members are limited to their own. Edits stay owner-only. Deletes are **soft** and cascade under one shared stamp - see `devplacepy/database/CLAUDE.md`.

Dockerfile

@@ -31,6 +31,7 @@ RUN pip install --no-cache-dir ".[bots]" \
 EXPOSE 10500
 
 ENV DEVPLACE_WEB_WORKERS=2
+ENV DEVPLACE_TEMPLATE_AUTO_RELOAD=0
 
 HEALTHCHECK --interval=30s --timeout=10s --retries=3 --start-period=10s \
     CMD curl -f http://localhost:10500/ || exit 1

README.md

@@ -109,6 +109,8 @@ Member progression is driven by activity and peer recognition.
 
 - **AI modifier.** Enabled by default and applied synchronously by default. It works like AI content correction, except it runs **only** where the prose you author contains an inline `@ai <instruction>` directive: the configured prompt tells the model to execute that instruction and replace the marked part, removing the `@ai` marker. Text with no `@ai ...` directive is left exactly as written. It is **context-aware**: the model is given a grounding summary of who is asking (your username, role, level, stars, post count, rank, followers, and bio), the current date, and where the directive sits - the post a comment replies to, the conversation a direct message belongs to, the gist's language and code, and so on - so directives like `@ai answer the question above`, `@ai write my bio from my stats`, or `@ai reply to this` work. It uses your own API key for per-user attribution, is fail-soft (the original is kept on any error), and applies across the web UI, the REST and devRant APIs, and Devii, on the same prose fields as correction (posts, projects, gists, comments, direct messages, and your bio). Code and source files are never touched. In direct messages it runs live: typing `@ai <instruction>` in a message executes it and the resolved result appears in the chat for both participants without a reload. You can switch the apply mode to background or disable it on your profile or via the Devii `ai_modifier_set` tool; the settings are saved at `POST /profile/{username}/ai-modifier`. The default instruction is "Execute what is behind `@ai` (the prompt) and replace that part including `@ai`". Successful modifications accumulate per-user running totals - modifications, token counts, cost, and timing/performance (average latency, average speed in tokens per second, and total processing time) - shown on the profile page; token, call, and performance figures are visible to the member, while the dollar figures (total and average cost) are shown to administrators only.
 
+- **Devii interactive widgets.** Administrators set the site default on the Devii service (`devii_interactions_default`, default on). Guests always use that default. Signed-in members inherit it until they override it on their profile or via the Devii `interactions_set` tool (`POST /profile/{username}/interactions`; owner or admin). When enabled, Devii may present decisions with channel-aware controls (`ui_prompt`); when disabled, it falls back to plain numbered menus.
+
 Every AI gateway response (`/openai/v1/*`) also returns per-call `X-Gateway-*` headers with the full token breakdown and the dollar cost of that call, so any client can read its own usage.
 
 ## Code Farm

devplacepy/attachments.py

@@ -444,12 +444,13 @@ def link_attachments(uids, target_type, target_uid):
         return
     placeholders = ",".join(f":p{i}" for i in range(len(flat)))
     params = {f"p{i}": uid for i, uid in enumerate(flat)}
-    db.query(
-        f"UPDATE attachments SET target_type=:tt, target_uid=:tu WHERE uid IN ({placeholders})",
-        tt=target_type,
-        tu=target_uid,
-        **params,
-    )
+    with db:
+        db.query(
+            f"UPDATE attachments SET target_type=:tt, target_uid=:tu WHERE uid IN ({placeholders})",
+            tt=target_type,
+            tu=target_uid,
+            **params,
+        )
 
 
 def set_gitea_asset_id(uid, asset_id):
@@ -617,7 +618,8 @@ def delete_attachments_for(target_type, target_uids):
     for row in rows:
         _unlink_attachment_files(row)
     ids = ",".join(str(row["id"]) for row in rows)
-    db.query(f"DELETE FROM attachments WHERE id IN ({ids})")
+    with db:
+        db.query(f"DELETE FROM attachments WHERE id IN ({ids})")
 
 
 def get_attachments(target_type, target_uid):

devplacepy/awards/images.py

@@ -0,0 +1,33 @@
+# retoor <retoor@molodetz.nl>
+
+from io import BytesIO
+
+from PIL import Image
+
+
+def enforce_rgba_png(file_bytes: bytes) -> bytes:
+    img = Image.open(BytesIO(file_bytes)).convert("RGBA")
+    width, height = img.size
+    if width > 1 and height > 1:
+        corner = img.getpixel((0, 0))
+        if len(corner) == 4 and corner[3] == 255:
+            bg = corner[:3]
+            data = img.getdata()
+            cleaned = []
+            for pixel in data:
+                if pixel[:3] == bg:
+                    cleaned.append((pixel[0], pixel[1], pixel[2], 0))
+                else:
+                    cleaned.append(pixel)
+            img.putdata(cleaned)
+    buf = BytesIO()
+    img.save(buf, format="PNG")
+    return buf.getvalue()
+
+
+def resize_award_png(source: bytes, size: int) -> bytes:
+    img = Image.open(BytesIO(source)).convert("RGBA")
+    img = img.resize((size, size), Image.LANCZOS)
+    buf = BytesIO()
+    img.save(buf, format="PNG")
+    return buf.getvalue()

devplacepy/cli/devii.py

@@ -1,13 +1,11 @@
 # retoor <retoor@molodetz.nl>
 
 import sys
-from devplacepy.database import get_table
+from devplacepy.database import db, get_table
 from devplacepy.cli._shared import _audit_cli
 
 
 def cmd_devii_reset_quota(args):
-    from devplacepy.database import db
-
     table_name = "devii_usage_ledger"
     if table_name not in db.tables:
         print(f"Table '{table_name}' does not exist, nothing to reset")
@@ -48,6 +46,67 @@ def cmd_devii_reset_quota(args):
     print(f"Reset AI quota for '{args.username}' ({count} ledger rows deleted)")
 
 
+def _active_count() -> int:
+    if "devii_lessons" not in db.tables:
+        return 0
+    return db["devii_lessons"].count(deleted_at=None)
+
+
+def _soft_deleted_count() -> int:
+    if "devii_lessons" not in db.tables:
+        return 0
+    return db["devii_lessons"].count(deleted_at={"!=": None})
+
+
+def cmd_devii_lessons_count(args):
+    active = _active_count()
+    deleted = _soft_deleted_count()
+    print(f"Lessons: {active} active, {deleted} soft-deleted ({active + deleted} total)")
+
+
+def cmd_devii_lessons_clear(args):
+    from devplacepy.services.devii.agentic.lessons import TABLE
+
+    if TABLE not in db.tables:
+        print("No devii_lessons table exists")
+        return
+    active = _active_count()
+    deleted = _soft_deleted_count()
+    total = active + deleted
+    if not args.force:
+        print(f"Will delete {total} lesson(s) ({active} active, {deleted} soft-deleted). Pass --force to confirm.")
+        return
+    db[TABLE].delete()
+    _audit_cli("cli.devii.lessons.clear", "CLI cleared all devii_lessons", metadata={"active": active, "soft_deleted": deleted})
+    print(f"Deleted {total} lesson(s)")
+
+
+def cmd_devii_lessons_prune(args):
+    from devplacepy.services.devii.agentic.lessons import LessonStore, _read_retention_settings
+
+    if "devii_lessons" not in db.tables:
+        print("No devii_lessons table exists")
+        return
+    active_before = _active_count()
+    if args.all_owners:
+        _, max_age = _read_retention_settings(db)
+        store = LessonStore(db, "_global", "_global")
+        pruned = store.prune_all_owners(max_age)
+    elif args.username:
+        user = get_table("users").find_one(username=args.username)
+        if not user:
+            print(f"User '{args.username}' not found")
+            sys.exit(1)
+        _, max_age = _read_retention_settings(db)
+        store = LessonStore(db, "user", user["uid"])
+        pruned = store.prune(max_age)
+    else:
+        print("Provide --all-owners, or --username USER")
+        sys.exit(1)
+    _audit_cli("cli.devii.lessons.prune", "CLI pruned devii_lessons", metadata={"pruned": pruned, "active_before": active_before})
+    print(f"Pruned {pruned} lesson(s) (active before: {active_before}, now: {_active_count()})")
+
+
 def register_devii(subparsers):
     devii = subparsers.add_parser("devii", help="Devii assistant management")
     devii_sub = devii.add_subparsers(title="action", dest="action")
@@ -64,3 +123,18 @@ def register_devii(subparsers):
         "--all", action="store_true", help="Reset every quota (users and guests)"
     )
     devii_reset.set_defaults(func=cmd_devii_reset_quota)
+
+    devii_lessons = devii_sub.add_parser("lessons", help="Manage persisted Devii lesson data")
+    lessons_sub = devii_lessons.add_subparsers(title="sub-action", dest="sub_action")
+
+    lessons_count = lessons_sub.add_parser("count", help="Count active and soft-deleted lessons")
+    lessons_count.set_defaults(func=cmd_devii_lessons_count)
+
+    lessons_prune = lessons_sub.add_parser("prune", help="Soft-delete lessons older than the configured max age")
+    lessons_prune.add_argument("--all-owners", action="store_true", help="Prune across every owner")
+    lessons_prune.add_argument("--username", help="Prune for a specific user")
+    lessons_prune.set_defaults(func=cmd_devii_lessons_prune)
+
+    lessons_clear = lessons_sub.add_parser("clear", help="Hard-delete every devii_lessons row")
+    lessons_clear.add_argument("--force", action="store_true", help="Required to confirm hard deletion")
+    lessons_clear.set_defaults(func=cmd_devii_lessons_clear)

devplacepy/config.py

@@ -68,6 +68,21 @@ INTERNAL_GATEWAY_URL = f"{INTERNAL_BASE_URL}/openai/v1/chat/completions"
 INTERNAL_EMBED_URL = f"{INTERNAL_BASE_URL}/openai/v1/embeddings"
 INTERNAL_MODEL = "molodetz"
 INTERNAL_EMBED_MODEL = "molodetz~embed"
+INTERNAL_IMAGE_MODEL = "molodetz-img-small"
+
+AWARD_GIVE_COOLDOWN_HOURS_DEFAULT = 24
+AWARD_RECEIVE_COOLDOWN_HOURS_DEFAULT = 24
+AWARD_DISPLAY_HOURS_DEFAULT = 24
+AWARD_DESCRIPTION_MAX = 125
+AWARD_IMAGE_MODEL_DEFAULT = "molodetz-img-small"
+AWARD_IMAGE_SIZE_DEFAULT = "512x512"
+AWARD_GENERATION_TIMEOUT_SECONDS = 120.0
+AWARD_IMAGE_PROMPT_DEFAULT = (
+    "Generate a single decorative developer award emblem/badge as a PNG with a fully "
+    "transparent background (alpha channel). No rectangular backdrop, no drop shadow "
+    "plate, no text labels rendered in the image. Center one stylized trophy/medal "
+    "icon that visually matches this message:"
+)
 DEFAULT_CORRECTION_PROMPT = "Leave literary as is, only do punctuation and casing"
 DEFAULT_MODIFIER_PROMPT = (
     "Execute what is behind `@ai` (the prompt) and replace that part including `@ai`"

devplacepy/content.py

@@ -13,6 +13,7 @@ from devplacepy.database import (
     resolve_by_slug,
     get_users_by_uids,
     get_vote_counts,
+    STAR_TARGETS,
     get_user_votes,
     get_reactions_by_targets,
     get_user_bookmarks,
@@ -20,6 +21,7 @@ from devplacepy.database import (
     get_poll_for_post,
     update_target_stars,
     clear_user_stars,
+    clear_user_post_count,
     get_target_owner_uid,
     resolve_object_url,
     soft_delete,
@@ -162,6 +164,8 @@ def create_content_item(
             **fields,
         }
     )
+    if table_name == "posts":
+        clear_user_post_count(user["uid"])
     if table_name == "projects":
         from devplacepy.templating import clear_user_projects_cache
 
@@ -615,6 +619,8 @@ def delete_content_item(
     soft_delete_engagement(target_type, [item["uid"]], actor)
     if comment_uids:
         soft_delete_engagement("comment", comment_uids, actor)
+    if target_type == "post":
+        clear_user_post_count(item["user_uid"])
     if target_type == "project":
         from devplacepy.project_files import soft_delete_all_project_files
         from devplacepy.templating import clear_user_projects_cache
@@ -647,7 +653,11 @@ def load_detail(
     if user and item["user_uid"] in get_blocked_uids(user["uid"]):
         return None
     author = get_users_by_uids([item["user_uid"]]).get(item["user_uid"])
-    ups, downs = get_vote_counts([item["uid"]])
+    if target_type in STAR_TARGETS:
+        star_count = item.get("stars") or 0
+    else:
+        ups, downs = get_vote_counts([item["uid"]])
+        star_count = ups.get(item["uid"], 0) - downs.get(item["uid"], 0)
     reactions = (
         get_reactions_by_targets(target_type, [item["uid"]], user).get(
             item["uid"], {"counts": {}, "mine": []}
@@ -664,7 +674,7 @@ def load_detail(
         "item": item,
         "author": author,
         "is_owner": bool(user and user["uid"] == item["user_uid"]),
-        "star_count": ups.get(item["uid"], 0) - downs.get(item["uid"], 0),
+        "star_count": star_count,
         "my_vote": get_user_votes(user["uid"], [item["uid"]]).get(item["uid"], 0)
         if user
         else 0,

devplacepy/curl_transport.py

@@ -73,10 +73,17 @@ class CurlResponseStream(httpx.AsyncByteStream):
 
 
 class CurlTransport(httpx.AsyncBaseTransport):
-    def __init__(self, *, impersonate: str = IMPERSONATE_TARGET, verify: bool = True) -> None:
+    def __init__(
+        self,
+        *,
+        impersonate: str = IMPERSONATE_TARGET,
+        verify: bool = True,
+        proxy: str | None = None,
+    ) -> None:
         self._session = AsyncSession()
         self._impersonate = impersonate
         self._verify = verify
+        self._proxy = proxy
 
     async def handle_async_request(self, request: httpx.Request) -> httpx.Response:
         headers = {
@@ -97,6 +104,7 @@ class CurlTransport(httpx.AsyncBaseTransport):
                 data=body or None,
                 impersonate=self._impersonate,
                 verify=self._verify,
+                proxy=self._proxy,
                 stream=True,
                 allow_redirects=False,
                 timeout=resolve_timeout(request),

devplacepy/customization.py

@@ -36,6 +36,18 @@ def owner_for(request: Request) -> tuple[str, str] | None:
 
 
 def _overrides_for(request: Request) -> dict:
+    cached = getattr(request.state, "_custom_overrides", None)
+    if cached is not None:
+        return cached
+    overrides = _resolve_overrides(request)
+    try:
+        request.state._custom_overrides = overrides
+    except Exception:
+        pass
+    return overrides
+
+
+def _resolve_overrides(request: Request) -> dict:
     if get_setting("customization_enabled", "1") != "1":
         return {"css": "", "js": ""}
     owner = owner_for(request)

devplacepy/database/CLAUDE.md

@@ -99,7 +99,7 @@ if "comments" not in db.tables:
 
 - **NEVER index the bare `deleted_at` column - use a PARTIAL trash index `WHERE deleted_at IS NOT NULL`.** `ensure_soft_delete_columns` creates `idx_<table>_trash ON (deleted_at) WHERE deleted_at IS NOT NULL` (and drops any legacy full `idx_<table>_deleted`). A full `deleted_at` index is a planner hazard: the column is one giant `NULL` bucket plus many unique delete-timestamps, so `sqlite_stat1` mis-estimates `deleted_at IS NULL` as returning ~2 rows and the planner picks that index for live reads, then `USE TEMP B-TREE FOR ORDER BY` to sort the whole live set (the global feed was doing exactly this, with 82% of posts soft-deleted). The partial index serves the admin Trash view (`deleted_at IS NOT NULL`) cheaply and stops poisoning live `IS NULL` queries.
 
-- **For "live, newest-first" listings add a composite or live-partial index that includes the sort column.** A `WHERE deleted_at IS NULL ORDER BY created_at` query needs the ordering in the index or it filesorts. Posts use a partial `idx_posts_live_created ON (created_at) WHERE deleted_at IS NULL` (feed) plus `idx_posts_user_created (user_uid, created_at)` (profile); comments use `idx_comments_target_created (target_type, target_uid, created_at)`; votes use `idx_votes_user_target (user_uid, target_uid)` (the per-user "my_vote" check on every card); notifications/gists/projects use `(user_uid, created_at)`. All were verified to drop the `USE TEMP B-TREE FOR ORDER BY` step.
+- **For "live, newest-first" listings add a composite or live-partial index that includes the sort column.** A `WHERE deleted_at IS NULL ORDER BY created_at` query needs the ordering in the index or it filesorts. Posts use a partial `idx_posts_live_created ON (created_at) WHERE deleted_at IS NULL` (feed) plus `idx_posts_user_created (user_uid, created_at)` (profile); comments use `idx_comments_target_created (target_type, target_uid, created_at)`; votes use `idx_votes_user_target (user_uid, target_uid)` (the per-user "my_vote" check on every card); notifications/gists/projects use `(user_uid, created_at)`; follows use `idx_follows_follower_created (follower_uid, created_at)` + `idx_follows_following_created (following_uid, created_at)` (the followers/following tabs sort newest-first; the legacy single-column follower/following indexes were dropped as redundant prefixes). All were verified to drop the `USE TEMP B-TREE FOR ORDER BY` step.
 
 - **Index the non-`uid` lookup keys too, not just the sort/owner columns.** A demand-vs-supply audit added the last missing single-key lookups: the `resolve_by_slug` hot path filters `slug` on content detail pages, so posts/gists/news/projects each get `idx_<table>_slug (slug)`; `get_setting`/`set_setting` filter `key`, so `idx_site_settings_key (key)`; the container store's `find_one(slug=)`/`find_one(name=)` fallbacks get `idx_instances_slug`/`idx_instances_name`. The DM thread load `find(sender_uid=, receiver_uid=)` gets the covering composites `idx_messages_conversation (sender_uid, receiver_uid)` + `idx_messages_conversation_rev (receiver_uid, sender_uid)` (the read-flag `UPDATE` uses the reverse); the badge-has check gets `idx_badges_user_name (user_uid, badge_name)`; the admin user list `ORDER BY -created_at` gets `idx_users_created_at (created_at)` (the existing `(role, created_at)` cannot serve a full-table created_at sort). All are non-unique so `_index` always creates them even if legacy duplicate data exists. Column sets already resolved to ~1 row by an existing prefix index (votes `+target_type`, game_quests `+kind`, poll_options `position`) are intentionally left uncovered - a trailing column there only adds write cost.
 
@@ -180,7 +180,7 @@ Site settings are seeded on startup (`site_settings` table):
 | `customization_js_enabled` | `"1"` | When `"0"`, custom CSS still serves but custom JS is suppressed |
 | `extra_head` | `""` | Raw HTML emitted verbatim into every page `<head>` by `templating.extra_head_tag()`; site-wide trusted-admin input, not sanitized |
 
-Besides the site/news/upload keys above, the **Operational** group is admin-editable at `/admin/settings`: `site_url` (public origin for absolute links incl. container ingress; resolved by `seo.public_base_url()` = setting -> `DEVPLACE_SITE_URL` env -> request origin), `rate_limit_per_minute`, `rate_limit_window_seconds`, `news_service_interval`, `session_max_age_days`, `session_remember_days`, `registration_open`, `maintenance_mode`, `maintenance_message`, `docs_search_mode` (`agent`|`bm25`, default `agent` - picks the `/docs/search.html` surface: the in-page Devii chat or the classic BM25 list; a viewer over their daily AI limit, or a guest when Devii is disabled, auto-falls-back to BM25 via `routers/docs/views.py` `_agent_search_state`). The **Custom Code** key `extra_head` is the sole key in the settings handler's `CLEARABLE_SETTINGS` set, so saving an empty textarea removes it (the default loop skips empty values).
+Besides the site/news/upload keys above, the **Operational** group is admin-editable at `/admin/settings`: `site_url` (public origin for absolute links incl. container ingress; resolved by `seo.public_base_url()` = setting -> `DEVPLACE_SITE_URL` env -> request origin), `rate_limit_per_minute`, `rate_limit_window_seconds`, `news_service_interval`, `session_max_age_days`, `session_remember_days`, `registration_open`, `maintenance_mode`, `maintenance_message`, `docs_search_mode` (`agent`|`bm25`, default `agent` - picks the `/docs/search.html` surface: the in-page Devii chat or the classic BM25 list; a viewer over their daily AI limit, or a guest when Devii is disabled, auto-falls-back to BM25 via `routers/docs/views.py` `_agent_search_state`), `outbound_proxy_url` (empty by default - when set, every `stealth.stealth_async_client`/`stealth_sync_client` call across the whole app routes through it via `stealth.configured_proxy_url()`; validated as `http(s)://`/`socks5(h)://` with a host in `AdminSettingsForm`; falls back to `DEVPLACE_OUTBOUND_PROXY_URL` when unset - see the "Outbound HTTP" note in the root `CLAUDE.md`). The **Custom Code** key `extra_head` is the sole key in the settings handler's `CLEARABLE_SETTINGS` set, so saving an empty textarea removes it (the default loop skips empty values).
 
 The seed block in `database.py` is guarded by `if "site_settings" in tables:` - on a brand-new DB the table does not exist yet (dataset creates tables lazily on first insert), so none of these rows are written until the table exists. Correct runtime behavior therefore relies on every consumer passing the production default to `get_setting`/`get_int_setting`, not on the seed.
 

devplacepy/database/__init__.py

@@ -5,10 +5,27 @@ from .core import refresh_snapshot, _local_cache_versions, _cache_version_cache,
 from .settings import _settings_cache, get_setting, get_int_setting, set_setting, clear_settings_cache, internal_gateway_key
 from .users import get_users_by_uids, _admins_cache, invalidate_admins_cache, get_admin_uids, set_user_timezone, set_last_seen, get_online_users, get_primary_admin_uid, search_users_by_username
 from .relations import _relations_cache, get_user_relations, get_blocked_uids, get_muted_uids, get_silenced_uids, invalidate_user_relations
-from .pagination import PAGE_SIZE, paginate, interleave_by_author, paginate_diverse, get_user_post_count, build_pagination
+from .pagination import PAGE_SIZE, paginate, interleave_by_author, paginate_diverse, get_user_post_count, clear_user_post_count, build_pagination
 from .soft_delete import SOFT_DELETE_TABLES, ensure_soft_delete_columns, soft_delete, soft_delete_in, restore, purge, list_deleted, count_deleted, restore_event, purge_event
 from .engagement import _comment_count_cache, get_comment_counts_by_post_uids, get_post_counts_by_user_uids, get_vote_counts, get_user_votes, get_reactions_by_targets, get_user_bookmarks, get_polls_by_post_uids, get_poll_for_post
-from .usage import _add_usage, _get_usage, add_correction_usage, get_correction_usage, add_modifier_usage, get_modifier_usage, NEWS_USAGE_KEY, add_news_usage, get_news_usage, ISSUE_USAGE_KEY, add_issue_usage, get_issue_usage, SEO_USAGE_KEY, add_seo_usage, get_seo_usage
+from .usage import _add_usage, _get_usage, add_correction_usage, get_correction_usage, add_modifier_usage, get_modifier_usage, NEWS_USAGE_KEY, add_news_usage, get_news_usage, ISSUE_USAGE_KEY, add_issue_usage, get_issue_usage, SEO_USAGE_KEY, add_seo_usage, get_seo_usage, AWARD_USAGE_KEY, add_award_usage, get_award_usage
+from .awards import (
+    AWARDS_PER_PAGE,
+    award_display_hours,
+    award_give_cooldown_hours,
+    award_receive_cooldown_hours,
+    award_is_prominent,
+    can_give_award,
+    can_receive_award,
+    count_published_awards,
+    enrich_award,
+    get_prominent_award,
+    get_user_awards,
+    has_giver_cooldown,
+    has_receiver_cooldown,
+    recompute_user_award_stats,
+    revoke_award,
+)
 from .seo_meta import SEO_META_TYPES, get_seo_metadata, get_seo_metadata_batch, has_fresh_seo_metadata, upsert_seo_metadata, mark_seo_metadata_stale
 from .activity import record_activity, record_unique_activity, get_user_activity, _activity_cache, _ACTIVITY_TABLES, get_activity_calendar, _activity_level, get_first_activity_date, HEATMAP_WEEKS, get_activity_heatmap, get_activity_months, get_streaks
 from .customization import CUSTOMIZATION_GLOBAL_SCOPE, CUSTOMIZATION_LANGS, _customizations_cache, _customization_key, CUSTOMIZATION_PREF_COLUMNS, get_customization_prefs, set_customization_pref, get_custom_overrides, get_custom_override, list_custom_overrides, set_custom_override, delete_custom_override
@@ -19,7 +36,7 @@ from .follows import get_follow_counts, get_follow_list, get_following_among
 from .deepsearch import _ds_now, create_deepsearch_session, update_deepsearch_session, get_deepsearch_session, add_deepsearch_message, get_deepsearch_messages, get_cached_deepsearch_url, upsert_deepsearch_url_cache
 from .ranking import VOTABLE_TARGETS, STAR_TARGETS, _authors_cache, _ranked_authors, _rank_map, get_top_authors, get_leaderboard, get_user_rank, get_user_stars, clear_user_stars, update_target_stars, soft_delete_engagement, delete_engagement, get_target_owner_uid
 from .comments import _drop_blocked, _build_comment_items, load_comments, get_recent_comments_by_target_uids, get_recent_comments_by_post_uids, load_comments_by_target_uids
-from .content import resolve_by_slug, resolve_object_url, get_uids_by_username_match, text_search_clause, get_daily_topic, get_featured_news
+from .content import resolve_by_slug, resolve_object_url, get_uids_by_username_match, text_search_clause, get_daily_topic, get_featured_news, get_trending_topics
 from .attachments_data import get_attachments, get_attachments_by_type, get_news_images_by_uids, delete_attachment_record, delete_attachments, _delete_attachment_file, get_user_media, get_deleted_media
 from .stats import _stats_cache, get_site_stats, _analytics_cache, get_platform_analytics, _gist_languages_cache, get_gist_languages
 from .schema import BUG_TABLE_RENAMES, migrate_bug_tables_to_issue_tables, init_db, _refresh_query_planner_stats, OLD_GATEWAY_URL, migrate_ai_gateway_settings, backfill_api_keys, _backfill_gamification
@@ -81,6 +98,7 @@ __all__ = [
     "interleave_by_author",
     "paginate_diverse",
     "get_user_post_count",
+    "clear_user_post_count",
     "build_pagination",
     "SOFT_DELETE_TABLES",
     "ensure_soft_delete_columns",
@@ -208,6 +226,7 @@ __all__ = [
     "text_search_clause",
     "get_daily_topic",
     "get_featured_news",
+    "get_trending_topics",
     "get_attachments",
     "get_attachments_by_type",
     "get_news_images_by_uids",

devplacepy/database/awards.py

@@ -0,0 +1,206 @@
+# retoor <retoor@molodetz.nl>
+
+from datetime import datetime, timedelta, timezone
+
+from devplacepy.cache import TTLCache
+from devplacepy.config import (
+    AWARD_DISPLAY_HOURS_DEFAULT,
+    AWARD_GIVE_COOLDOWN_HOURS_DEFAULT,
+    AWARD_RECEIVE_COOLDOWN_HOURS_DEFAULT,
+)
+from .core import db
+from .pagination import build_pagination
+from .settings import get_int_setting
+from .core import get_table, _now_iso
+from .users import get_users_by_uids
+from .content import resolve_by_slug
+from .soft_delete import soft_delete, soft_delete_in
+
+AWARDS_PER_PAGE = 12
+
+
+def _awards_table():
+    return get_table("awards")
+
+
+def award_give_cooldown_hours() -> int:
+    return max(1, get_int_setting("award_give_cooldown_hours", AWARD_GIVE_COOLDOWN_HOURS_DEFAULT))
+
+
+def award_receive_cooldown_hours() -> int:
+    return max(
+        1, get_int_setting("award_receive_cooldown_hours", AWARD_RECEIVE_COOLDOWN_HOURS_DEFAULT)
+    )
+
+
+def award_display_hours() -> int:
+    return max(1, get_int_setting("award_display_hours", AWARD_DISPLAY_HOURS_DEFAULT))
+
+
+def _cooldown_cutoff(hours: int) -> str:
+    return (datetime.now(timezone.utc) - timedelta(hours=hours)).isoformat()
+
+
+def has_giver_cooldown(giver_uid: str) -> bool:
+    if not giver_uid or "awards" not in db.tables:
+        return False
+    cutoff = _cooldown_cutoff(award_give_cooldown_hours())
+    row = _awards_table().find_one(
+        giver_uid=giver_uid, deleted_at=None, created_at={">=": cutoff}
+    )
+    return row is not None
+
+
+def has_receiver_cooldown(receiver_uid: str) -> bool:
+    if not receiver_uid or "awards" not in db.tables:
+        return False
+    cutoff = _cooldown_cutoff(award_receive_cooldown_hours())
+    row = _awards_table().find_one(
+        receiver_uid=receiver_uid, deleted_at=None, created_at={">=": cutoff}
+    )
+    return row is not None
+
+
+def can_receive_award(receiver_uid: str) -> bool:
+    return not has_receiver_cooldown(receiver_uid)
+
+
+def can_give_award(giver_uid: str, receiver_uid: str) -> bool:
+    if not giver_uid or not receiver_uid or giver_uid == receiver_uid:
+        return False
+    return not has_giver_cooldown(giver_uid) and not has_receiver_cooldown(receiver_uid)
+
+
+def _published_filter():
+    return {"deleted_at": None, "generated_at": {">": ""}}
+
+
+def count_published_awards(receiver_uid: str) -> int:
+    if not receiver_uid or "awards" not in db.tables:
+        return 0
+    return _awards_table().count(receiver_uid=receiver_uid, **_published_filter())
+
+
+def _latest_published(receiver_uid: str):
+    if not receiver_uid or "awards" not in db.tables:
+        return None
+    rows = list(
+        _awards_table().find(
+            receiver_uid=receiver_uid,
+            deleted_at=None,
+            generated_at={">": ""},
+            order_by=["-generated_at"],
+            _limit=1,
+        )
+    )
+    return rows[0] if rows else None
+
+
+def recompute_user_award_stats(receiver_uid: str) -> None:
+    if not receiver_uid or "users" not in db.tables:
+        return
+    count = count_published_awards(receiver_uid)
+    latest = _latest_published(receiver_uid)
+    users = get_table("users")
+    payload = {
+        "uid": receiver_uid,
+        "award_count": count,
+        "last_award_at": latest.get("generated_at") if latest else None,
+        "last_award_slug": latest.get("slug") if latest else None,
+        "last_award_uid": latest.get("uid") if latest else None,
+    }
+    users.update(payload, ["uid"])
+
+
+_prominence_cache = TTLCache(ttl=15, max_size=500)
+
+
+def award_is_prominent(user: dict | None) -> bool:
+    if not user or not user.get("last_award_at") or not user.get("last_award_uid"):
+        return False
+    cached = _prominence_cache.get(user["last_award_uid"])
+    if cached is not None:
+        return cached
+    prominent = _compute_prominence(user["last_award_uid"])
+    _prominence_cache.set(user["last_award_uid"], prominent)
+    return prominent
+
+
+def _compute_prominence(award_uid: str) -> bool:
+    award = resolve_by_slug(_awards_table(), award_uid)
+    if not award or not award.get("generated_at"):
+        return False
+    try:
+        published = datetime.fromisoformat(award["generated_at"])
+        if published.tzinfo is None:
+            published = published.replace(tzinfo=timezone.utc)
+    except (ValueError, TypeError):
+        return False
+    window = timedelta(hours=award_display_hours())
+    return datetime.now(timezone.utc) - published <= window
+
+
+def enrich_award(row: dict, givers: dict | None = None) -> dict:
+    item = dict(row)
+    giver_uid = row.get("giver_uid", "")
+    giver = (givers or {}).get(giver_uid) or get_users_by_uids([giver_uid]).get(giver_uid)
+    item["giver"] = giver
+    item["image_url"] = f"/awards/{row.get('slug', '')}/256"
+    item["thumb_url"] = f"/awards/{row.get('slug', '')}/64"
+    return item
+
+
+def get_user_awards(receiver_uid: str, page: int = 1, per_page: int = AWARDS_PER_PAGE):
+    if not receiver_uid or "awards" not in db.tables:
+        return [], build_pagination(page, 0, per_page)
+    table = _awards_table()
+    total = table.count(receiver_uid=receiver_uid, **_published_filter())
+    offset = max(0, (page - 1) * per_page)
+    rows = list(
+        table.find(
+            receiver_uid=receiver_uid,
+            deleted_at=None,
+            generated_at={">": ""},
+            order_by=["-generated_at"],
+            _limit=per_page,
+            _offset=offset,
+        )
+    )
+    giver_uids = [row.get("giver_uid") for row in rows if row.get("giver_uid")]
+    givers = get_users_by_uids(giver_uids)
+    items = [enrich_award(row, givers) for row in rows]
+    return items, build_pagination(page, total, per_page)
+
+
+def get_prominent_award(profile_user: dict) -> dict | None:
+    if not award_is_prominent(profile_user):
+        return None
+    award = resolve_by_slug(_awards_table(), profile_user.get("last_award_uid", ""))
+    if not award:
+        return None
+    return enrich_award(award)
+
+
+def revoke_award(award_uid: str, admin_uid: str) -> dict | None:
+    table = _awards_table()
+    row = table.find_one(uid=award_uid)
+    if not row or row.get("deleted_at"):
+        return None
+    stamp = _now_iso()
+    attachment_uids = [
+        uid
+        for uid in (
+            row.get("attachment_uid_512"),
+            row.get("attachment_uid_256"),
+            row.get("attachment_uid_64"),
+        )
+        if uid
+    ]
+    soft_delete("awards", admin_uid, stamp=stamp, uid=award_uid)
+    from devplacepy.attachments import soft_delete_attachments_for
+
+    soft_delete_attachments_for("award", [award_uid], admin_uid)
+    if attachment_uids:
+        soft_delete_in("attachments", "uid", attachment_uids, admin_uid, stamp=stamp)
+    recompute_user_award_stats(row.get("receiver_uid", ""))
+    return row

devplacepy/database/comments.py

@@ -126,7 +126,7 @@ def load_comments_by_target_uids(target_type, target_uids, user=None):
     params["tt"] = target_type
     raw = list(
         db.query(
-            f"SELECT * FROM comments WHERE target_type=:tt AND target_uid IN ({placeholders}) ORDER BY created_at",
+            f"SELECT * FROM comments WHERE target_type=:tt AND target_uid IN ({placeholders}) AND deleted_at IS NULL ORDER BY created_at",
             **params,
         )
     )

devplacepy/database/content.py

@@ -1,7 +1,14 @@
 # retoor <retoor@molodetz.nl>
 
+from collections import Counter
+
+from devplacepy.cache import TTLCache
+
 from .core import db, get_table, or_
 
+_daily_topic_cache = TTLCache(ttl=60, max_size=1)
+_trending_cache = TTLCache(ttl=15, max_size=1)
+
 
 def resolve_by_slug(table, slug, include_deleted=False):
     has_soft_delete = table.has_column("deleted_at")
@@ -40,6 +47,12 @@ def resolve_object_url(target_type: str, target_uid: str) -> str:
             comment.get("target_uid") or comment.get("post_uid", ""),
         )
         return f"{parent_url}#comment-{target_uid}"
+    if target_type == "award":
+        award = resolve_by_slug(get_table("awards"), target_uid)
+        if award:
+            receiver = get_table("users").find_one(uid=award.get("receiver_uid", ""))
+            if receiver:
+                return f"/profile/{receiver['username']}?tab=awards#award-{award.get('slug', '')}"
     return "/feed"
 
 
@@ -71,6 +84,15 @@ def text_search_clause(
 
 
 def get_daily_topic():
+    cached = _daily_topic_cache.get("topic")
+    if cached is not None:
+        return cached
+    topic = _load_daily_topic()
+    _daily_topic_cache.set("topic", topic)
+    return topic
+
+
+def _load_daily_topic():
     if "news" in db.tables:
         article = db["news"].find_one(
             status="published", deleted_at=None, order_by=["-synced_at"]
@@ -122,3 +144,24 @@ def get_featured_news(limit=5):
             }
         )
     return articles
+
+
+def get_trending_topics(limit: int = 6) -> list[dict]:
+    cached = _trending_cache.get("topics")
+    if cached is not None:
+        return cached[:limit]
+    if "posts" not in db.tables or "topic" not in db["posts"].columns:
+        return []
+    rows = db.query(
+        "SELECT topic FROM posts WHERE deleted_at IS NULL "
+        "AND topic IS NOT NULL AND topic != '' "
+        "ORDER BY created_at DESC LIMIT 200"
+    )
+    counter: Counter[str] = Counter()
+    for row in rows:
+        topic = (row["topic"] or "").strip()
+        if topic:
+            counter[topic] += 1
+    topics = [{"topic": t, "count": c} for t, c in counter.most_common(limit)]
+    _trending_cache.set("topics", topics)
+    return topics

devplacepy/database/core.py

@@ -61,10 +61,11 @@ def _ensure_cache_state() -> None:
     global _cache_state_ready
     if _cache_state_ready:
         return
-    db.query(
-        "CREATE TABLE IF NOT EXISTS cache_state "
-        "(name TEXT PRIMARY KEY, version INTEGER NOT NULL DEFAULT 0)"
-    )
+    with db:
+        db.query(
+            "CREATE TABLE IF NOT EXISTS cache_state "
+            "(name TEXT PRIMARY KEY, version INTEGER NOT NULL DEFAULT 0)"
+        )
     _cache_state_ready = True
 
 
@@ -74,18 +75,15 @@ def get_cache_version(name: str) -> int:
         return cached
     try:
         _ensure_cache_state()
-        row = next(
-            iter(
-                db.query(
-                    "SELECT version FROM cache_state WHERE name = :name", name=name
-                )
-            ),
-            None,
-        )
-        version = int(row["version"]) if row else 0
+        with db:
+            rows = list(db.query("SELECT name, version FROM cache_state"))
+        versions = {row["name"]: int(row["version"]) for row in rows}
     except Exception as e:
         logger.warning(f"Could not read cache version {name}: {e}")
         return 0
+    for key, version in versions.items():
+        _cache_version_cache.set(key, version)
+    version = versions.get(name, 0)
     _cache_version_cache.set(name, version)
     return version
 
@@ -93,16 +91,15 @@ def get_cache_version(name: str) -> int:
 def bump_cache_version(name: str) -> None:
     try:
         _ensure_cache_state()
-        db.query(
-            "INSERT OR IGNORE INTO cache_state (name, version) VALUES (:name, 0)",
-            name=name,
-        )
-        db.query(
-            "UPDATE cache_state SET version = version + 1 WHERE name = :name", name=name
-        )
-        connection = db.executable
-        if connection.in_transaction():
-            connection.commit()
+        with db:
+            db.query(
+                "INSERT OR IGNORE INTO cache_state (name, version) VALUES (:name, 0)",
+                name=name,
+            )
+            db.query(
+                "UPDATE cache_state SET version = version + 1 WHERE name = :name",
+                name=name,
+            )
         _cache_version_cache.pop(name)
     except Exception as e:
         logger.warning(f"Could not bump cache version {name}: {e}")

devplacepy/database/notifications.py

@@ -17,6 +17,7 @@ NOTIFICATION_TYPES = [
     {"key": "issue", "label": "Issue tracker", "description": "Updates on issue reports you filed"},
     {"key": "reminder", "label": "Reminders", "description": "A reminder or scheduled task you asked Devii to run fires"},
     {"key": "harvest_stolen", "label": "Farm raids", "description": "Someone steals a ready build from your Code Farm"},
+    {"key": "award", "label": "Awards", "description": "Someone gives you an award on your profile"},
 ]
 
 

devplacepy/database/pagination.py

@@ -1,5 +1,6 @@
 # retoor <retoor@molodetz.nl>
 
+from devplacepy.cache import TTLCache
 from .core import db, get_table
 from .relations import get_blocked_uids
 
@@ -7,6 +8,9 @@ from .relations import get_blocked_uids
 PAGE_SIZE = 25
 
 
+_user_post_count_cache = TTLCache(ttl=15, max_size=2000)
+
+
 def paginate(
     table,
     *clauses,
@@ -74,10 +78,19 @@ def paginate_diverse(
     return interleave_by_author(rows, uid_key=uid_key), next_cursor
 
 
+def clear_user_post_count(user_uid: str) -> None:
+    _user_post_count_cache.pop(user_uid)
+
+
 def get_user_post_count(user_uid: str) -> int:
+    cached = _user_post_count_cache.get(user_uid)
+    if cached is not None:
+        return cached
     if "posts" not in db.tables:
         return 0
-    return get_table("posts").count(user_uid=user_uid, deleted_at=None)
+    count = get_table("posts").count(user_uid=user_uid, deleted_at=None)
+    _user_post_count_cache.set(user_uid, count)
+    return count
 
 
 def build_pagination(page, total, per_page=25):

devplacepy/database/ranking.py

@@ -16,7 +16,7 @@ VOTABLE_TARGETS: dict[str, str] = {
 STAR_TARGETS: set[str] = {"post", "project", "gist"}
 
 
-_authors_cache = TTLCache(ttl=15, max_size=200)
+_authors_cache = TTLCache(ttl=60, max_size=200)
 
 
 _stars_cache = TTLCache(ttl=15, max_size=2000)
@@ -151,17 +151,19 @@ def delete_engagement(target_type: str, target_uids: list) -> None:
     if "reactions" in db.tables:
         placeholders, params = _in_clause(uids)
         params["tt"] = target_type
-        db.query(
-            f"DELETE FROM reactions WHERE target_type=:tt AND target_uid IN ({placeholders})",
-            **params,
-        )
+        with db:
+            db.query(
+                f"DELETE FROM reactions WHERE target_type=:tt AND target_uid IN ({placeholders})",
+                **params,
+            )
     if "bookmarks" in db.tables:
         placeholders, params = _in_clause(uids)
         params["tt"] = target_type
-        db.query(
-            f"DELETE FROM bookmarks WHERE target_type=:tt AND target_uid IN ({placeholders})",
-            **params,
-        )
+        with db:
+            db.query(
+                f"DELETE FROM bookmarks WHERE target_type=:tt AND target_uid IN ({placeholders})",
+                **params,
+            )
     if target_type == "post" and "polls" in db.tables:
         for uid in uids:
             for poll in db["polls"].find(post_uid=uid):

devplacepy/database/remote.py

@@ -0,0 +1,348 @@
+# retoor <retoor@molodetz.nl>
+
+import inspect
+import os
+
+import httpx
+
+from devplacepy.cache import TTLCache
+from devplacepy_services.base.db_codec import (
+    decode_value,
+    encode_args,
+    is_write,
+    is_write_sql,
+)
+
+_SERVICE_URL = os.environ.get("DEVPLACE_DB_SERVICE_URL", "http://127.0.0.1:10601").rstrip("/")
+_INTERNAL_KEY = os.environ.get("DEVPLACE_GATEWAY_INTERNAL_KEY", "").strip()
+_CLIENT: httpx.Client | None = None
+
+# Section 7.3: settings reads tolerate up to 5s staleness. patch_module()
+# generically RPCs every devplacepy.database call, bypassing the local
+# TTL cache get_setting/get_int_setting had in-process - without this,
+# every settings read (rate limiting, maintenance mode, admin dashboards)
+# pays a full HTTP round trip to the database broker.
+_SETTINGS_CACHE_TTL_SECONDS = 5
+_SETTINGS_CACHE = TTLCache(ttl=_SETTINGS_CACHE_TTL_SECONDS, max_size=512)
+_CACHED_SETTINGS_FNS = frozenset({"get_setting", "get_int_setting"})
+
+
+def _service_url() -> str:
+    return os.environ.get("DEVPLACE_DB_SERVICE_URL", _SERVICE_URL).rstrip("/")
+
+
+def _headers() -> dict[str, str]:
+    headers: dict[str, str] = {}
+    key = os.environ.get("DEVPLACE_GATEWAY_INTERNAL_KEY", _INTERNAL_KEY).strip()
+    if key:
+        headers["X-Internal-Key"] = key
+    return headers
+
+
+def _client() -> httpx.Client:
+    global _CLIENT
+    if _CLIENT is None:
+        _CLIENT = httpx.Client(timeout=30.0)
+    return _CLIENT
+
+
+def _post(path: str, body: dict) -> object:
+    response = _client().post(
+        f"{_service_url()}/{path.lstrip('/')}",
+        json=body,
+        headers=_headers(),
+    )
+    if response.status_code >= 400:
+        payload = response.json() if response.content else {}
+        message = payload.get("error", "Database service request failed")
+        raise RuntimeError(message)
+    if not response.content:
+        return None
+    return decode_value(response.json())
+
+
+def _invoke_cached(fn_name: str, args, kwargs):
+    cache_key = f"{fn_name}:{args!r}:{sorted(kwargs.items())!r}"
+    cached = _SETTINGS_CACHE.get(cache_key)
+    if cached is not None:
+        return cached
+    value = _invoke(fn_name, args, kwargs, write=False)
+    _SETTINGS_CACHE.set(cache_key, value)
+    return value
+
+
+def _invoke(fn_name: str, args, kwargs, *, write: bool = False):
+    encoded_args, encoded_kwargs = encode_args(args, kwargs)
+    payload = {
+        "fn": fn_name,
+        "args": encoded_args,
+        "kwargs": encoded_kwargs,
+        "write": write,
+    }
+    result = _post("internal/invoke", payload)
+    if isinstance(result, dict) and "result" in result:
+        return result["result"]
+    return result
+
+
+class RemoteSearchClause:
+    def __init__(self, term, fields, author_field=None):
+        self.term = term.strip()
+        self.fields = tuple(fields)
+        self.author_field = author_field
+
+
+class RemoteUidInClause:
+    def __init__(self, field, uids):
+        self.field = field
+        self.uids = frozenset(uids)
+
+
+class RemoteTable:
+    def __init__(self, db: "RemoteDb", name: str) -> None:
+        self._db = db
+        self._name = name
+        self._column_cache = None
+
+    def __getattr__(self, name: str):
+        def caller(*args, **kwargs):
+            return self._db._table_op(self._name, name, args, kwargs)
+
+        return caller
+
+    def has_column(self, name: str) -> bool:
+        cache = self._column_cache
+        if cache is None:
+            sample = self.find(_limit=1)
+            row = next(iter(sample), None)
+            cache = set(row.keys()) if row else set()
+            self._column_cache = cache
+        return name in cache
+
+    def count(self, **kwargs):
+        return self._db._table_op(self._name, "count", [], kwargs)
+
+    @property
+    def table(self):
+        return self
+
+    @property
+    def exists(self) -> bool:
+        return self._name in self._db.tables
+
+class RemoteDb:
+    def __init__(self) -> None:
+        self._tables_cache: list[str] | None = None
+
+    @property
+    def tables(self) -> list[str]:
+        if self._tables_cache is None:
+            result = _post("internal/db-op", {"op": "tables"})
+            self._tables_cache = list(result or [])
+        return self._tables_cache
+
+    def __getitem__(self, name: str) -> RemoteTable:
+        return RemoteTable(self, name)
+
+    def query(self, sql: str, **params):
+        encoded_args, encoded_kwargs = encode_args((sql,), params)
+        result = _post(
+            "internal/db-op",
+            {
+                "op": "query",
+                "args": encoded_args,
+                "kwargs": encoded_kwargs,
+                "write": is_write_sql(sql),
+            },
+        )
+        return result or []
+
+    def _table_op(self, table: str, method: str, args, kwargs, *, write: bool = False):
+        encoded_args, encoded_kwargs = encode_args(args, kwargs)
+        result = _post(
+            "internal/db-op",
+            {
+                "op": "table_op",
+                "table": table,
+                "method": method,
+                "args": encoded_args,
+                "kwargs": encoded_kwargs,
+                "write": write,
+            },
+        )
+        if method in {"insert", "update", "delete"}:
+            self._tables_cache = None
+        return result
+
+    @property
+    def executable(self):
+        return self
+
+    @property
+    def in_transaction(self) -> bool:
+        return False
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, exc_type, exc, tb):
+        return False
+
+
+_LOCAL_REMOTE = frozenset(
+    {
+        "get_table",
+        "refresh_snapshot",
+        "_in_clause",
+        "_now_iso",
+        "text_search_clause",
+    }
+)
+
+
+def _remote_text_search_clause(
+    table, search, fields=("title", "description"), author_field=None
+):
+    term = (search or "").strip()
+    if not term:
+        return None
+    if type(table).__name__ == "RemoteTable":
+        return RemoteSearchClause(term, fields, author_field)
+    from devplacepy.database.content import text_search_clause as local_clause
+
+    return local_clause(table, search, fields, author_field=author_field)
+
+
+def _remote_get_table(name: str):
+    import devplacepy.database.core as core
+
+    return core.db[name]
+
+
+def _remote_refresh_snapshot() -> None:
+    return None
+
+
+def patch_module(module) -> None:
+    import devplacepy.database as db_module
+
+    for name in db_module.__all__:
+        if name in _LOCAL_REMOTE:
+            continue
+        target = getattr(module, name, None)
+        if target is None or not callable(target):
+            continue
+        if inspect.isclass(target):
+            continue
+
+        def make_wrapper(fn_name: str, fn_write: bool):
+            if fn_name in _CACHED_SETTINGS_FNS:
+
+                def wrapper(*args, **kwargs):
+                    return _invoke_cached(fn_name, args, kwargs)
+
+                wrapper.__name__ = fn_name
+                return wrapper
+
+            def wrapper(*args, **kwargs):
+                return _invoke(fn_name, args, kwargs, write=fn_write)
+
+            wrapper.__name__ = fn_name
+            return wrapper
+
+        setattr(module, name, make_wrapper(name, is_write(name)))
+
+
+def activate() -> None:
+    import devplacepy.database.core as core
+
+    core.db = RemoteDb()
+    import devplacepy.database as db_module
+
+    patch_module(db_module)
+    for submodule_name in (
+        "settings",
+        "users",
+        "relations",
+        "pagination",
+        "soft_delete",
+        "engagement",
+        "usage",
+        "awards",
+        "seo_meta",
+        "activity",
+        "customization",
+        "email",
+        "notifications",
+        "forks",
+        "follows",
+        "deepsearch",
+        "ranking",
+        "comments",
+        "content",
+        "attachments_data",
+        "stats",
+        "schema",
+    ):
+        try:
+            submodule = __import__(
+                f"devplacepy.database.{submodule_name}",
+                fromlist=[submodule_name],
+            )
+        except ImportError:
+            continue
+        patch_module(submodule)
+    for external_name in (
+        "devplacepy.services.statistics.tracking",
+        "devplacepy.services.base",
+        "devplacepy.attachments",
+        "devplacepy.project_files",
+    ):
+        try:
+            external = __import__(external_name, fromlist=[external_name.split(".")[-1]])
+        except ImportError:
+            continue
+        if hasattr(external, "db"):
+            external.db = RemoteDb()
+    db_module.db = core.db
+    db_module.get_table = _remote_get_table
+    core.get_table = _remote_get_table
+    db_module.refresh_snapshot = _remote_refresh_snapshot
+    core.refresh_snapshot = _remote_refresh_snapshot
+    db_module.text_search_clause = _remote_text_search_clause
+    import devplacepy.database.content as content_module
+
+    content_module.text_search_clause = _remote_text_search_clause
+    for submodule_name in (
+        "settings",
+        "users",
+        "relations",
+        "pagination",
+        "soft_delete",
+        "engagement",
+        "usage",
+        "awards",
+        "seo_meta",
+        "activity",
+        "customization",
+        "email",
+        "notifications",
+        "forks",
+        "follows",
+        "deepsearch",
+        "ranking",
+        "comments",
+        "content",
+        "attachments_data",
+        "stats",
+        "schema",
+    ):
+        try:
+            submodule = __import__(
+                f"devplacepy.database.{submodule_name}",
+                fromlist=[submodule_name],
+            )
+        except ImportError:
+            continue
+        if hasattr(submodule, "db"):
+            submodule.db = core.db

devplacepy/database/schema.py

@@ -1,6 +1,6 @@
 # retoor <retoor@molodetz.nl>
 
-from .core import DEFAULT_CORRECTION_PROMPT, DEFAULT_MODIFIER_PROMPT, INTERNAL_GATEWAY_URL, _index, _uid_index, db, defaultdict, get_table, logger
+from .core import DEFAULT_CORRECTION_PROMPT, DEFAULT_MODIFIER_PROMPT, INTERNAL_GATEWAY_URL, _drop_index, _index, _uid_index, db, defaultdict, get_table, logger
 from .settings import get_setting, set_setting
 from .soft_delete import SOFT_DELETE_TABLES, ensure_soft_delete_columns
 from .ranking import _authors_cache
@@ -183,8 +183,10 @@ def init_db():
     )
     _index(db, "badges", "idx_badges_user", ["user_uid"])
     _index(db, "badges", "idx_badges_user_name", ["user_uid", "badge_name"])
-    _index(db, "follows", "idx_follows_follower", ["follower_uid"])
-    _index(db, "follows", "idx_follows_following", ["following_uid"])
+    _drop_index(db, "idx_follows_follower")
+    _drop_index(db, "idx_follows_following")
+    _index(db, "follows", "idx_follows_follower_created", ["follower_uid", "created_at"])
+    _index(db, "follows", "idx_follows_following_created", ["following_uid", "created_at"])
     user_relations = get_table("user_relations")
     for column, example in (
         ("uid", ""),
@@ -363,9 +365,10 @@ def init_db():
         if not conversations.has_column("channel"):
             conversations.create_column_by_example("channel", "main")
         try:
-            db.query(
-                "UPDATE devii_conversations SET channel='main' WHERE channel IS NULL"
-            )
+            with db:
+                db.query(
+                    "UPDATE devii_conversations SET channel='main' WHERE channel IS NULL"
+                )
         except Exception as e:  # noqa: BLE001
             logger.warning(f"Could not backfill devii_conversations.channel: {e}")
     _index(
@@ -391,6 +394,7 @@ def init_db():
         db, "devii_tasks", "idx_devii_tasks_due", ["enabled", "status", "next_run_at"]
     )
     _index(db, "devii_lessons", "idx_devii_lessons_owner", ["owner_kind", "owner_id"])
+    _index(db, "devii_lessons", "idx_devii_lessons_owner_created", ["owner_kind", "owner_id", "created_at"])
     _index(
         db, "devii_virtual_tools", "idx_devii_vtools_owner", ["owner_kind", "owner_id"]
     )
@@ -430,6 +434,12 @@ def init_db():
         "idx_gw_usage_endpoint_time",
         ["endpoint", "created_at"],
     )
+    _index(
+        db,
+        "gateway_usage_ledger",
+        "idx_gw_usage_appref_time",
+        ["app_reference", "created_at"],
+    )
     _index(db, "gateway_concurrency_samples", "idx_gw_conc_time", ["created_at"])
     jobs_table = get_table("jobs")
     for column, example in (
@@ -559,10 +569,11 @@ def init_db():
             correction_usage.create_column_by_example(column, example)
     try:
         if "correction_usage" in db.tables:
-            db.query(
-                "CREATE UNIQUE INDEX IF NOT EXISTS idx_correction_usage_user "
-                "ON correction_usage (user_uid)"
-            )
+            with db:
+                db.query(
+                    "CREATE UNIQUE INDEX IF NOT EXISTS idx_correction_usage_user "
+                    "ON correction_usage (user_uid)"
+                )
     except Exception as e:
         logger.warning(f"Could not create unique index on correction_usage: {e}")
 
@@ -582,10 +593,11 @@ def init_db():
             modifier_usage.create_column_by_example(column, example)
     try:
         if "modifier_usage" in db.tables:
-            db.query(
-                "CREATE UNIQUE INDEX IF NOT EXISTS idx_modifier_usage_user "
-                "ON modifier_usage (user_uid)"
-            )
+            with db:
+                db.query(
+                    "CREATE UNIQUE INDEX IF NOT EXISTS idx_modifier_usage_user "
+                    "ON modifier_usage (user_uid)"
+                )
     except Exception as e:
         logger.warning(f"Could not create unique index on modifier_usage: {e}")
 
@@ -605,10 +617,11 @@ def init_db():
             news_usage.create_column_by_example(column, example)
     try:
         if "news_usage" in db.tables:
-            db.query(
-                "CREATE UNIQUE INDEX IF NOT EXISTS idx_news_usage_user "
-                "ON news_usage (user_uid)"
-            )
+            with db:
+                db.query(
+                    "CREATE UNIQUE INDEX IF NOT EXISTS idx_news_usage_user "
+                    "ON news_usage (user_uid)"
+                )
     except Exception as e:
         logger.warning(f"Could not create unique index on news_usage: {e}")
 
@@ -628,10 +641,11 @@ def init_db():
             issue_usage.create_column_by_example(column, example)
     try:
         if "issue_usage" in db.tables:
-            db.query(
-                "CREATE UNIQUE INDEX IF NOT EXISTS idx_issue_usage_user "
-                "ON issue_usage (user_uid)"
-            )
+            with db:
+                db.query(
+                    "CREATE UNIQUE INDEX IF NOT EXISTS idx_issue_usage_user "
+                    "ON issue_usage (user_uid)"
+                )
     except Exception as e:
         logger.warning(f"Could not create unique index on issue_usage: {e}")
 
@@ -651,13 +665,77 @@ def init_db():
             seo_usage.create_column_by_example(column, example)
     try:
         if "seo_usage" in db.tables:
-            db.query(
-                "CREATE UNIQUE INDEX IF NOT EXISTS idx_seo_usage_user "
-                "ON seo_usage (user_uid)"
-            )
+            with db:
+                db.query(
+                    "CREATE UNIQUE INDEX IF NOT EXISTS idx_seo_usage_user "
+                    "ON seo_usage (user_uid)"
+                )
     except Exception as e:
         logger.warning(f"Could not create unique index on seo_usage: {e}")
 
+    award_usage = get_table("award_usage")
+    for column, example in (
+        ("user_uid", ""),
+        ("calls", 0),
+        ("prompt_tokens", 0),
+        ("completion_tokens", 0),
+        ("total_tokens", 0),
+        ("cost_usd", 0.0),
+        ("upstream_latency_ms", 0.0),
+        ("total_latency_ms", 0.0),
+        ("updated_at", ""),
+    ):
+        if not award_usage.has_column(column):
+            award_usage.create_column_by_example(column, example)
+    try:
+        if "award_usage" in db.tables:
+            with db:
+                db.query(
+                    "CREATE UNIQUE INDEX IF NOT EXISTS idx_award_usage_user "
+                    "ON award_usage (user_uid)"
+                )
+    except Exception as e:
+        logger.warning(f"Could not create unique index on award_usage: {e}")
+
+    awards = get_table("awards")
+    for column, example in (
+        ("uid", ""),
+        ("slug", ""),
+        ("description", ""),
+        ("giver_uid", ""),
+        ("receiver_uid", ""),
+        ("attachment_uid_512", ""),
+        ("attachment_uid_256", ""),
+        ("attachment_uid_64", ""),
+        ("generated_at", ""),
+        ("created_at", ""),
+        ("job_uid", ""),
+        ("deleted_at", ""),
+        ("deleted_by", ""),
+    ):
+        if not awards.has_column(column):
+            awards.create_column_by_example(column, example)
+    try:
+        if "awards" in db.tables:
+            with db:
+                db.query(
+                    "CREATE UNIQUE INDEX IF NOT EXISTS idx_awards_slug ON awards (slug)"
+                )
+                db.query(
+                    "CREATE INDEX IF NOT EXISTS idx_awards_receiver_created "
+                    "ON awards (receiver_uid, created_at)"
+                )
+                db.query(
+                    "CREATE INDEX IF NOT EXISTS idx_awards_giver_created "
+                    "ON awards (giver_uid, created_at)"
+                )
+                db.query(
+                    "CREATE INDEX IF NOT EXISTS idx_awards_generated "
+                    "ON awards (receiver_uid, generated_at)"
+                )
+    except Exception as e:
+        logger.warning(f"Could not create awards indexes: {e}")
+
     seo_metadata = get_table("seo_metadata")
     for column, example in (
         ("uid", ""),
@@ -727,10 +805,11 @@ def init_db():
             user_activity.create_column_by_example(column, example)
     try:
         if "user_activity" in db.tables:
-            db.query(
-                "CREATE UNIQUE INDEX IF NOT EXISTS idx_user_activity_user_action "
-                "ON user_activity (user_uid, action)"
-            )
+            with db:
+                db.query(
+                    "CREATE UNIQUE INDEX IF NOT EXISTS idx_user_activity_user_action "
+                    "ON user_activity (user_uid, action)"
+                )
     except Exception as e:
         logger.warning(f"Could not create unique index on user_activity: {e}")
 
@@ -745,10 +824,11 @@ def init_db():
             user_activity_seen.create_column_by_example(column, example)
     try:
         if "user_activity_seen" in db.tables:
-            db.query(
-                "CREATE UNIQUE INDEX IF NOT EXISTS idx_user_activity_seen_unique "
-                "ON user_activity_seen (user_uid, action, target)"
-            )
+            with db:
+                db.query(
+                    "CREATE UNIQUE INDEX IF NOT EXISTS idx_user_activity_seen_unique "
+                    "ON user_activity_seen (user_uid, action, target)"
+                )
     except Exception as e:
         logger.warning(f"Could not create unique index on user_activity_seen: {e}")
 
@@ -1155,7 +1235,11 @@ def init_db():
             "customization_enabled": "1",
             "customization_js_enabled": "1",
             "audit_log_retention_days": "90",
+            "statistics_tracking_enabled": "1",
             "docs_search_mode": "agent",
+            "outbound_proxy_url": "",
+            "devii_lessons_max_per_owner": "500",
+            "devii_lessons_max_age_days": "90",
         }
         for key, value in operational_defaults.items():
             existing = db["site_settings"].find_one(key=key)
@@ -1164,6 +1248,72 @@ def init_db():
                     {"uid": f"default_{key}", "key": key, "value": value}
                 )
 
+    with db:
+        db.query(
+            "CREATE TABLE IF NOT EXISTS visit_stats_hourly ("
+            "bucket_start TEXT NOT NULL, "
+            "page_group TEXT NOT NULL, "
+            "referrer_group TEXT NOT NULL, "
+            "views INTEGER NOT NULL DEFAULT 0, "
+            "member_views INTEGER NOT NULL DEFAULT 0, "
+            "guest_views INTEGER NOT NULL DEFAULT 0)"
+        )
+        db.query(
+            "CREATE TABLE IF NOT EXISTS visit_unique_slots ("
+            "bucket_start TEXT NOT NULL, "
+            "visitor_hash TEXT NOT NULL, "
+            "page_group TEXT NOT NULL, "
+            "user_uid TEXT)"
+        )
+    _index(db, "visit_stats_hourly", "idx_visit_hourly_bucket", ["bucket_start"])
+    _index(
+        db,
+        "visit_stats_hourly",
+        "idx_visit_hourly_page_time",
+        ["page_group", "bucket_start"],
+    )
+    _index(
+        db,
+        "visit_stats_hourly",
+        "idx_visit_hourly_ref_time",
+        ["referrer_group", "bucket_start"],
+    )
+    _index(
+        db,
+        "visit_stats_hourly",
+        "idx_visit_hourly_unique_row",
+        ["bucket_start", "page_group", "referrer_group"],
+        unique=True,
+    )
+    _index(db, "visit_unique_slots", "idx_visit_unique_bucket", ["bucket_start"])
+    _index(
+        db,
+        "visit_unique_slots",
+        "idx_visit_unique_hash",
+        ["bucket_start", "visitor_hash"],
+    )
+    _index(
+        db,
+        "visit_unique_slots",
+        "idx_visit_unique_row",
+        ["bucket_start", "visitor_hash", "page_group"],
+        unique=True,
+    )
+    _index(db, "issue_tickets", "idx_issue_tickets_created", ["created_at"])
+    _index(db, "devii_turns", "idx_devii_turns_started", ["started_at"])
+    _index(db, "instance_events", "idx_instance_events_created", ["created_at"])
+    _index(db, "game_steals", "idx_game_steals_stolen_at", ["stolen_at"])
+    _index(db, "messages", "idx_messages_created_at", ["created_at"])
+    _index(db, "notifications", "idx_notifications_created", ["created_at"])
+    _index(db, "follows", "idx_follows_created_at", ["created_at"])
+    _index(db, "reactions", "idx_reactions_created_at", ["created_at"])
+    _index(db, "votes", "idx_votes_created_at", ["created_at"])
+    _index(db, "bookmarks", "idx_bookmarks_created_at", ["created_at"])
+    _index(db, "badges", "idx_badges_created_at", ["created_at"])
+    _index(db, "audit_log", "idx_audit_result_created", ["result", "created_at"])
+    _index(db, "jobs", "idx_jobs_created_at", ["created_at"])
+    _index(db, "attachments", "idx_attachments_created_at", ["created_at"])
+
     _backfill_gamification()
     backfill_api_keys()
     migrate_ai_gateway_settings()
@@ -1210,6 +1360,15 @@ def migrate_ai_gateway_settings() -> None:
     if get_setting("bot_model", "") == "deepseek-chat":
         set_setting("bot_model", "molodetz")
         logger.info("Migrated bot_model to molodetz")
+    from devplacepy.services.openai_gateway.routing import (
+        migrate_retired_image_gateway,
+        seed_default_deepseek_routes,
+        seed_default_image_routes,
+    )
+
+    seed_default_deepseek_routes()
+    seed_default_image_routes()
+    migrate_retired_image_gateway()
 
 
 def backfill_api_keys() -> int:
@@ -1234,12 +1393,22 @@ def backfill_api_keys() -> int:
         users.create_column_by_example("ai_modifier_sync", 1)
     if not users.has_column("ai_modifier_prompt"):
         users.create_column_by_example("ai_modifier_prompt", DEFAULT_MODIFIER_PROMPT)
+    if not users.has_column("interactions_enabled"):
+        users.create_column_by_example("interactions_enabled", -1)
     if not users.has_column("timezone"):
         users.create_column_by_example("timezone", "")
     if not users.has_column("avatar_seed"):
         users.create_column_by_example("avatar_seed", "")
     if not users.has_column("last_seen"):
         users.create_column_by_example("last_seen", "")
+    if not users.has_column("award_count"):
+        users.create_column_by_example("award_count", 0)
+    if not users.has_column("last_award_at"):
+        users.create_column_by_example("last_award_at", "")
+    if not users.has_column("last_award_slug"):
+        users.create_column_by_example("last_award_slug", "")
+    if not users.has_column("last_award_uid"):
+        users.create_column_by_example("last_award_uid", "")
     with db:
         db.query(
             "UPDATE users SET ai_modifier_enabled = 1 WHERE ai_modifier_enabled IS NULL"
@@ -1250,6 +1419,10 @@ def backfill_api_keys() -> int:
             "WHERE ai_modifier_prompt IS NULL OR ai_modifier_prompt = ''",
             prompt=DEFAULT_MODIFIER_PROMPT,
         )
+        db.query(
+            "UPDATE users SET interactions_enabled = -1 "
+            "WHERE interactions_enabled IS NULL"
+        )
     import uuid_utils
 
     updated = 0

devplacepy/database/soft_delete.py

@@ -41,6 +41,7 @@ SOFT_DELETE_TABLES = [
     "email_accounts",
     "user_relations",
     "seo_metadata",
+    "awards",
 ]
 
 
@@ -93,11 +94,12 @@ def soft_delete_in(table_name, column, uids, deleted_by, *, stamp=None, **extra)
     for index, (key, value) in enumerate(extra.items()):
         params[f"x{index}"] = value
         extra_sql += f" AND {key} = :x{index}"
-    db.query(
-        f"UPDATE {table_name} SET deleted_at = :dat, deleted_by = :dby "
-        f"WHERE {column} IN ({placeholders}) AND deleted_at IS NULL{extra_sql}",
-        **params,
-    )
+    with db:
+        db.query(
+            f"UPDATE {table_name} SET deleted_at = :dat, deleted_by = :dby "
+            f"WHERE {column} IN ({placeholders}) AND deleted_at IS NULL{extra_sql}",
+            **params,
+        )
     return len(uids)
 
 
@@ -160,11 +162,12 @@ def restore_event(stamp):
                     s=stamp,
                 ).__next__()["n"]
             )
-            db.query(
-                f"UPDATE {table_name} SET deleted_at = NULL, deleted_by = NULL "
-                f"WHERE deleted_at = :s",
-                s=stamp,
-            )
+            with db:
+                db.query(
+                    f"UPDATE {table_name} SET deleted_at = NULL, deleted_by = NULL "
+                    f"WHERE deleted_at = :s",
+                    s=stamp,
+                )
     return restored
 
 
@@ -181,7 +184,8 @@ def purge_event(stamp):
             )
             if rows:
                 purged.append((table_name, rows))
-                db.query(
-                    f"DELETE FROM {table_name} WHERE deleted_at = :s", s=stamp
-                )
+                with db:
+                    db.query(
+                        f"DELETE FROM {table_name} WHERE deleted_at = :s", s=stamp
+                    )
     return purged

devplacepy/database/usage.py

@@ -126,3 +126,14 @@ def add_seo_usage(totals: dict) -> None:
 
 def get_seo_usage() -> dict:
     return _get_usage("seo_usage", SEO_USAGE_KEY)
+
+
+AWARD_USAGE_KEY = "award"
+
+
+def add_award_usage(totals: dict) -> None:
+    _add_usage("award_usage", AWARD_USAGE_KEY, totals)
+
+
+def get_award_usage() -> dict:
+    return _get_usage("award_usage", AWARD_USAGE_KEY)

devplacepy/db_client.py

@@ -0,0 +1,29 @@
+# retoor <retoor@molodetz.nl>
+
+import os
+
+
+def _activate() -> None:
+    if os.environ.get("DEVPLACE_DB_SERVICE") == "1":
+        return
+    if os.environ.get("DEVPLACE_REMOTE_DB") == "1":
+        from devplacepy.database.remote import activate
+
+        activate()
+
+
+_activate()
+
+import devplacepy.database as _database
+
+
+def _remote_table(table) -> bool:
+    return type(table).__name__ == "RemoteTable"
+
+
+def __getattr__(name: str):
+    return getattr(_database, name)
+
+
+def __dir__():
+    return sorted(name for name in dir(_database) if not name.startswith("_"))

devplacepy/docs_api/groups/admin.py

@@ -63,6 +63,22 @@ four ways to sign requests.
                 ],
                 sample_response={"ok": True, "redirect": "/admin/media"},
             ),
+            endpoint(
+                id="admin-revoke-award",
+                method="POST",
+                path="/admin/awards/{uid}/revoke",
+                title="Revoke award",
+                summary=(
+                    "Soft-delete a published award and its linked attachments, then recompute "
+                    "receiver stats. Restorable from admin trash."
+                ),
+                auth="admin",
+                destructive=True,
+                params=[
+                    field("uid", "path", "string", True, "", "Award uid to revoke."),
+                ],
+                sample_response={"ok": True, "redirect": "/profile/receiver?tab=awards"},
+            ),
             endpoint(
                 id="admin-media-purge",
                 method="POST",
@@ -171,6 +187,84 @@ four ways to sign requests.
                     "This is the endpoint the Devii assistant calls as `site_analytics`; see [Devii internals](/docs/devii-internals.html).",
                 ],
             ),
+            endpoint(
+                id="admin-statistics",
+                method="GET",
+                path="/admin/statistics/data",
+                title="Platform statistics",
+                summary=(
+                    "Tabbed platform statistics with KPI cards, period-over-period deltas, "
+                    "time-series data for charts, and breakdown tables. Covers visitors, members, "
+                    "content, engagement, social, AI, Devii, services, containers, game, awards, "
+                    "moderation, tools, and storage."
+                ),
+                auth="admin",
+                interactive=True,
+                params=[
+                    field(
+                        "tab",
+                        "query",
+                        "string",
+                        False,
+                        "overview",
+                        "Tab key (overview, visitors, members, content, ...).",
+                    ),
+                    field(
+                        "hours",
+                        "query",
+                        "int",
+                        False,
+                        "168",
+                        "Lookback window in hours (24, 168, 720, 2160, or 0 for all time).",
+                    ),
+                    field(
+                        "compare",
+                        "query",
+                        "int",
+                        False,
+                        "1",
+                        "Include previous-period comparison (1 or 0).",
+                    ),
+                    field(
+                        "top_n",
+                        "query",
+                        "int",
+                        False,
+                        "10",
+                        "Rows in breakdown tables (1-50).",
+                    ),
+                ],
+                notes=[
+                    "The HTML dashboard lives at `/admin/statistics`. Visitor metrics require the statistics tracking middleware (hourly aggregation, 90-day retention).",
+                ],
+            ),
+            endpoint(
+                id="admin-statistics-page",
+                method="GET",
+                path="/admin/statistics",
+                title="Statistics dashboard",
+                summary="Admin HTML dashboard for platform statistics with charts and tabs.",
+                auth="admin",
+                interactive=False,
+                params=[
+                    field(
+                        "tab",
+                        "query",
+                        "string",
+                        False,
+                        "overview",
+                        "Initial tab to render.",
+                    ),
+                    field(
+                        "hours",
+                        "query",
+                        "int",
+                        False,
+                        "168",
+                        "Initial time window in hours.",
+                    ),
+                ],
+            ),
             endpoint(
                 id="admin-ai-usage",
                 method="GET",

devplacepy/docs_api/groups/gateway.py

@@ -5,7 +5,6 @@ from .._shared import endpoint, field
 GROUP = {
         "slug": "gateway",
         "title": "OpenAI Gateway",
-        "admin": True,
         "intro": """
 # OpenAI Gateway
 
@@ -32,6 +31,31 @@ The gateway additionally serves **text embeddings** at `/openai/v1/embeddings`.
 generic model `molodetz~embed`, which the gateway maps to the configured embedding model (OpenRouter's
 Qwen3 8B embedding model by default). Usage and cost are tracked per call exactly like chat and vision.
 
+The gateway also serves **image generation** at `/openai/v1/images/generations`. Clients request the
+generic model `molodetz-img-small`, which the gateway maps to the configured image model (OpenRouter's
+Flux 1.1 Pro by default). Cost is tracked per call with a flat per-image price when the upstream
+returns no native cost.
+
+## Quick start
+
+Copy the command below and paste it into a terminal. If you are signed in the `{{ api_key }}`
+and `{{ app_reference }}` placeholders are already filled in with your own values; otherwise
+replace them with the API key from your [profile](/profile) page and any application identifier.
+
+```bash
+curl -X POST "{{ base }}/openai/v1/chat/completions" \
+  -H "Authorization: Bearer {{ api_key }}" \
+  -H "X-App-Reference: {{ app_reference }}" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "molodetz",
+    "messages": [{"role": "user", "content": "Hello, how are you?"}]
+  }'
+```
+
+The response carries `X-Gateway-*` headers with token counts and dollar cost for the call.
+For streaming, add `"stream": true` to the JSON body.
+
 ## Model routing and providers
 
 On top of the single default upstream above, an administrator can register additional named
@@ -60,7 +84,7 @@ and dollar cost directly from the response with no extra request:
 | Header | Meaning |
 |--------|---------|
 | `X-Gateway-Model` | Upstream model actually used for the call |
-| `X-Gateway-Backend` | Backend that served it: `chat`, `embed`, or passthrough |
+| `X-Gateway-Backend` | Backend that served it: `chat`, `embed`, `image`, or passthrough |
 | `X-Gateway-Prompt-Tokens` | Input (prompt) tokens |
 | `X-Gateway-Completion-Tokens` | Output (completion) tokens |
 | `X-Gateway-Total-Tokens` | Total tokens (prompt + completion) |
@@ -83,7 +107,18 @@ and dollar cost directly from the response with no extra request:
 Dollar costs use the upstream's native `cost` field when it returns one
 (`X-Gateway-Cost-Native: 1`); otherwise they are computed from the per-million prices of the matched
 model route, falling back to the prices configured on the `openai` service when no route matches. The
-one denied path that makes no upstream call (embeddings disabled) returns no usage headers.
+denied paths that make no upstream call (embeddings or image generation disabled) return no usage headers.
+
+## Request header `X-App-Reference`
+
+Clients **SHOULD** send an `X-App-Reference` header to identify themselves for cost attribution.
+The value is a free-form slug (max 30 characters, letters, digits, `_`, `.`, `-`). When missing or
+invalid, the gateway defaults to `default`. The value is recorded in every usage ledger row and can
+be queried alongside owner-kind and owner-id to attribute spending per application.
+
+```
+X-App-Reference: devplace-devii-v-1-0-0
+```
 
 Administrators enable and configure this gateway under [Background Services](/docs/services.html)
 (the `openai` service).
@@ -99,7 +134,7 @@ for signing DevPlace's own requests.
                 path="/openai/v1/chat/completions",
                 title="Chat completions",
                 summary="OpenAI-compatible chat completion. Supports streaming.",
-                auth="user",
+                auth="public",
                 encoding="json",
                 params=[
                     field(
@@ -108,7 +143,7 @@ for signing DevPlace's own requests.
                         "string",
                         False,
                         "gpt-4o-mini",
-                        "Model id. When it matches a configured model route the gateway forwards to that route's provider and upstream model; otherwise it uses the default upstream model.",
+                        "Model id. When it matches a configured model route the gateway forwards to that route's provider and upstream model; otherwise it falls back to the configured default upstream model.",
                     ),
                     field(
                         "messages",
@@ -139,7 +174,7 @@ for signing DevPlace's own requests.
                 path="/openai/v1/embeddings",
                 title="Embeddings",
                 summary="OpenAI-compatible text embeddings. Request model molodetz~embed.",
-                auth="user",
+                auth="public",
                 encoding="json",
                 params=[
                     field(
@@ -170,6 +205,55 @@ for signing DevPlace's own requests.
                 notes=[
                     "Returns `503` when the gateway service is not running or embeddings are disabled.",
                     "Every response carries the `X-Gateway-*` token and dollar-cost headers (see Per-call cost and usage headers above).",
+                    "If `model` matches a configured embed model route it is forwarded to that route's provider and upstream model; otherwise it falls back to the configured default embedding model.",
+                ],
+            ),
+            endpoint(
+                id="gateway-images",
+                method="POST",
+                path="/openai/v1/images/generations",
+                title="Image generation",
+                summary="OpenAI-compatible image generation. Request model molodetz-img-small.",
+                auth="public",
+                encoding="json",
+                params=[
+                    field(
+                        "model",
+                        "json",
+                        "string",
+                        False,
+                        "molodetz-img-small",
+                        "Image model id; the gateway maps molodetz-img-small to the configured model, or to a matching image model route's provider and target model.",
+                    ),
+                    field(
+                        "prompt",
+                        "json",
+                        "string",
+                        True,
+                        '"a decorative developer award emblem"',
+                        "Text prompt describing the image to generate.",
+                    ),
+                    field(
+                        "size",
+                        "json",
+                        "string",
+                        False,
+                        "512x512",
+                        "Output dimensions (provider-dependent).",
+                    ),
+                    field(
+                        "response_format",
+                        "json",
+                        "string",
+                        False,
+                        "b64_json",
+                        "Return format: url or b64_json.",
+                    ),
+                ],
+                notes=[
+                    "Returns `503` when the gateway service is not running or image generation is disabled.",
+                    "Every response carries the `X-Gateway-*` token and dollar-cost headers (see Per-call cost and usage headers above).",
+                    "If `model` matches a configured image model route it is forwarded to that route's provider and upstream model; otherwise it falls back to the configured default image model.",
                 ],
             ),
             endpoint(
@@ -178,7 +262,7 @@ for signing DevPlace's own requests.
                 path="/openai/v1/{path}",
                 title="Passthrough",
                 summary="Any other /v1 path is forwarded to the upstream as-is.",
-                auth="user",
+                auth="public",
                 interactive=False,
                 params=[
                     field(

devplacepy/docs_api/groups/profiles.py

@@ -32,7 +32,7 @@ four ways to sign requests.
                         False,
                         "posts",
                         "Profile tab.",
-                        ["posts", "activity", "followers", "following", "media"],
+                        ["posts", "activity", "followers", "following", "media", "awards"],
                     ),
                 ],
             ),
@@ -60,7 +60,7 @@ four ways to sign requests.
                         False,
                         "posts",
                         "Profile tab.",
-                        ["posts", "activity", "followers", "following", "media"],
+                        ["posts", "activity", "followers", "following", "media", "awards"],
                     ),
                 ],
             ),
@@ -136,7 +136,7 @@ four ways to sign requests.
                         "textarea",
                         False,
                         "Leave literary as is, only do punctuation and casing",
-                        "Correction instruction, up to 2000 characters.",
+                        "Correction instruction, up to 20000 characters.",
                     ),
                 ],
                 sample_response={
@@ -150,6 +150,53 @@ four ways to sign requests.
                     },
                 },
             ),
+            endpoint(
+                id="profile-interactions",
+                method="POST",
+                path="/profile/{username}/interactions",
+                title="Configure Devii interactive widgets",
+                summary="Enable or disable CA-IWP interactive prompts (ui_prompt) for this account, or reset to the administrator default. Guests always use the site default. Admins may target any user.",
+                auth="user",
+                encoding="form",
+                destructive=True,
+                params=[
+                    field(
+                        "username",
+                        "path",
+                        "string",
+                        True,
+                        "bob_test",
+                        "Profile owner. Must be yourself unless you are an admin.",
+                    ),
+                    field(
+                        "enabled",
+                        "form",
+                        "boolean",
+                        False,
+                        "true",
+                        "true to enable interactive widgets, false to disable. Ignored when reset is true.",
+                    ),
+                    field(
+                        "reset",
+                        "form",
+                        "boolean",
+                        False,
+                        "false",
+                        "true to clear the user override and inherit the administrator default.",
+                    ),
+                ],
+                sample_response={
+                    "ok": True,
+                    "redirect": "/profile/bob_test",
+                    "data": {
+                        "url": "/profile/bob_test",
+                        "enabled": True,
+                        "source": "user",
+                        "default": True,
+                        "override": True,
+                    },
+                },
+            ),
             endpoint(
                 id="profile-ai-modifier",
                 method="POST",
@@ -190,7 +237,7 @@ four ways to sign requests.
                         "textarea",
                         False,
                         "Execute what is behind `@ai` (the prompt) and replace that part including `@ai`",
-                        "Modifier instruction, up to 2000 characters.",
+                        "Modifier instruction, up to 20000 characters.",
                     ),
                 ],
                 sample_response={
@@ -254,6 +301,40 @@ four ways to sign requests.
                 ],
                 sample_response={"api_key": "NEW_UUID"},
             ),
+            endpoint(
+                id="profile-give-award",
+                method="POST",
+                path="/profile/{username}/award",
+                title="Give a member an award",
+                summary="Create a pending award on another member's profile and enqueue image generation.",
+                auth="user",
+                encoding="json",
+                params=[
+                    field(
+                        "username",
+                        "path",
+                        "string",
+                        True,
+                        "{{ username }}",
+                        "Receiver username.",
+                    ),
+                    field(
+                        "description",
+                        "body",
+                        "string",
+                        True,
+                        "Great work on the release!",
+                        "Award message (1-125 characters).",
+                    ),
+                ],
+                sample_response={
+                    "ok": True,
+                    "data": {
+                        "award_uid": "AWARD_UID",
+                        "award_slug": "abc123-great-work",
+                    },
+                },
+            ),
             endpoint(
                 id="profile-regenerate-avatar",
                 method="POST",
@@ -650,6 +731,37 @@ four ways to sign requests.
                 auth="public",
                 interactive=True,
             ),
+            endpoint(
+                id="award-image",
+                method="GET",
+                path="/awards/{slug_or_uid}/{size}",
+                title="Award image redirect",
+                summary="Redirect to the stored PNG attachment for a published award.",
+                auth="public",
+                params=[
+                    field(
+                        "slug_or_uid",
+                        "path",
+                        "string",
+                        True,
+                        "abc123-great-work",
+                        "Award slug or bare uid.",
+                    ),
+                    field(
+                        "size",
+                        "path",
+                        "enum",
+                        True,
+                        "256",
+                        "Image size.",
+                        ["512", "256", "64"],
+                    ),
+                ],
+                notes=[
+                    "> Pending or revoked awards return 404.",
+                    "> Response includes long-lived cache headers.",
+                ],
+            ),
             endpoint(
                 id="avatar",
                 method="GET",

devplacepy/docs_api/render.py

@@ -43,5 +43,6 @@ def render_group(slug, base, username, api_key):
         "{{ base }}": base,
         "{{ username }}": username or "YOUR_USERNAME",
         "{{ api_key }}": api_key or "YOUR_API_KEY",
+        "{{ app_reference }}": f"user-{username}-app-v-1-0-0" if username else "user-app-v-13.37.0",
     }
     return _substitute(group, replacements)

devplacepy/docs_devrant.py

@@ -237,7 +237,7 @@ DEVRANT_GROUPS = {
             encoding="form",
             params=[
                 field("rant_id", "path", type="int", required=True, example="1", description="Rant id."),
-                field("comment", "body", type="textarea", required=True, example="Great rant!", description="Comment text, 1-1000 chars."),
+                field("comment", "body", type="textarea", required=True, example="Great rant!", description="Comment text, 1-125000 chars."),
             ],
             sample_response={"success": True},
         ),

devplacepy/main.py

@@ -37,8 +37,10 @@ from devplacepy.database import (
     get_user_post_count,
     get_user_stars,
     get_blocked_uids,
+    get_top_authors,
+    get_trending_topics,
 )
-from devplacepy.templating import templates
+from devplacepy.templating import templates, jinja_unread_count
 from devplacepy.cache import TTLCache
 from devplacepy.responses import respond, wants_json, json_error
 from devplacepy.schemas import LandingOut, ValidationErrorOut
@@ -56,6 +58,7 @@ from devplacepy.routers import (
     notifications,
     votes,
     avatar,
+    awards,
     follow,
     relations,
     admin,
@@ -95,6 +98,7 @@ from devplacepy.services.jobs.issue_create_service import IssueCreateService
 from devplacepy.services.jobs.planning_service import PlanningReportService
 from devplacepy.services.jobs.seo.service import SeoService
 from devplacepy.services.jobs.seo_meta_service import SeoMetaService
+from devplacepy.services.jobs.award_service import AwardService
 from devplacepy.services.backup import BackupService
 from devplacepy.services.dbapi.service import DbApiJobService
 from devplacepy.services.pubsub import PubSubService
@@ -255,6 +259,7 @@ async def lifespan(app: FastAPI):
     service_manager.register(ForkService())
     service_manager.register(SeoService())
     service_manager.register(SeoMetaService())
+    service_manager.register(AwardService())
     service_manager.register(BackupService())
     service_manager.register(DbApiJobService())
     service_manager.register(PubSubService())
@@ -281,9 +286,15 @@ async def lifespan(app: FastAPI):
             logger.info(
                 f"Worker pid {os.getpid()} declined service lock; another worker owns background services"
             )
+    from devplacepy.services.statistics.tracking import start_visit_flusher
+
+    start_visit_flusher()
     logger.info(f"DevPlace started on port {PORT}")
     yield
     logger.info("Shutting down services...")
+    from devplacepy.services.statistics.tracking import flush_visits
+
+    flush_visits()
     await service_manager.shutdown_all()
     await background.stop()
 
@@ -428,6 +439,7 @@ app.include_router(reactions.router, prefix="/reactions")
 app.include_router(bookmarks.router, prefix="/bookmarks")
 app.include_router(polls.router, prefix="/polls")
 app.include_router(avatar.router, prefix="/avatar")
+app.include_router(awards.router, prefix="/awards")
 app.include_router(follow.router, prefix="/follow")
 app.include_router(relations.router)
 app.include_router(leaderboard.router, prefix="/leaderboard")
@@ -455,7 +467,8 @@ app.include_router(game.router, prefix="/game")
 
 @app.middleware("http")
 async def refresh_db_snapshot(request: Request, call_next):
-    refresh_snapshot()
+    if not request.url.path.startswith(("/static", "/avatar")):
+        refresh_snapshot()
     return await call_next(request)
 
 
@@ -580,6 +593,15 @@ async def track_presence(request: Request, call_next):
     return await call_next(request)
 
 
+@app.middleware("http")
+async def visit_statistics(request: Request, call_next):
+    from devplacepy.services.statistics.tracking import track_visit
+
+    response = await call_next(request)
+    track_visit(request, response.status_code)
+    return response
+
+
 @app.middleware("http")
 async def response_timing(request: Request, call_next):
     start = time.perf_counter()
@@ -589,7 +611,7 @@ async def response_timing(request: Request, call_next):
     return response
 
 
-app.add_middleware(GZipMiddleware, minimum_size=512, compresslevel=6)
+app.add_middleware(GZipMiddleware, minimum_size=512, compresslevel=5)
 
 
 _home_cache = TTLCache(ttl=int(os.environ.get("DEVPLACE_HOME_CACHE_TTL", "60")), max_size=4)
@@ -681,6 +703,14 @@ async def landing(request: Request):
         breadcrumbs=[],
         schemas=[website_schema(base)],
     )
+    user_xp = user.get("xp", 0) or 0 if user else 0
+    user_level = user.get("level", 1) or 1 if user else 1
+    xp_progress_pct = (user_xp % 100) if user_xp else 0
+    unread_count = jinja_unread_count(user["uid"]) if user else 0
+
+    top_contributors = get_top_authors(5) if not blocked else []
+    trending_topics = get_trending_topics(6) if not blocked else []
+
     return respond(
         request,
         "landing.html",
@@ -691,8 +721,14 @@ async def landing(request: Request):
             "is_authenticated": bool(user),
             "user_post_count": get_user_post_count(user["uid"]) if user else 0,
             "user_stars": get_user_stars(user["uid"]) if user else 0,
+            "user_xp": user_xp,
+            "user_level": user_level,
+            "xp_progress_pct": xp_progress_pct,
+            "unread_count": unread_count,
             "landing_articles": landing_articles,
             "landing_posts": landing_posts,
+            "top_contributors": top_contributors,
+            "trending_topics": trending_topics,
         },
         model=LandingOut,
     )

devplacepy/models.py

@@ -4,6 +4,7 @@ import re
 
 from datetime import datetime
 from typing import Literal, Optional
+from urllib.parse import urlsplit
 from pydantic import BaseModel, Field, field_validator, model_validator
 from devplacepy.constants import TOPICS, REACTION_EMOJI
 from devplacepy.config import DEFAULT_CORRECTION_PROMPT, DEFAULT_MODIFIER_PROMPT
@@ -157,7 +158,7 @@ class PostEditForm(BaseModel):
 
 
 class CommentForm(BaseModel):
-    content: str = Field(min_length=3, max_length=1000)
+    content: str = Field(min_length=3, max_length=125000)
     target_uid: str = Field(default="", max_length=36)
     post_uid: str = Field(default="", max_length=36)
     target_type: Literal["post", "project", "news", "issue", "gist"] = "post"
@@ -172,7 +173,7 @@ class CommentForm(BaseModel):
 
 
 class CommentEditForm(BaseModel):
-    content: str = Field(min_length=3, max_length=1000)
+    content: str = Field(min_length=3, max_length=125000)
 
 
 class ProjectForm(BaseModel):
@@ -238,6 +239,10 @@ class CustomizationToggleForm(BaseModel):
     value: bool = False
 
 
+class AwardGiveForm(BaseModel):
+    description: str = Field(min_length=1, max_length=125)
+
+
 class NotificationPrefForm(BaseModel):
     notification_type: str = Field(min_length=1, max_length=40)
     channel: Literal["in_app", "push", "telegram"]
@@ -253,13 +258,18 @@ class NotificationDefaultForm(BaseModel):
 class AiCorrectionForm(BaseModel):
     enabled: bool = False
     sync: bool = False
-    prompt: str = Field(default=DEFAULT_CORRECTION_PROMPT, max_length=2000)
+    prompt: str = Field(default=DEFAULT_CORRECTION_PROMPT, max_length=20000)
 
 
 class AiModifierForm(BaseModel):
     enabled: bool = False
     sync: bool = False
-    prompt: str = Field(default=DEFAULT_MODIFIER_PROMPT, max_length=2000)
+    prompt: str = Field(default=DEFAULT_MODIFIER_PROMPT, max_length=20000)
+
+
+class InteractionsForm(BaseModel):
+    enabled: bool = True
+    reset: bool = False
 
 
 class TelegramPairForm(BaseModel):
@@ -551,8 +561,20 @@ class AdminSettingsForm(BaseModel):
     maintenance_mode: str = Field(default="", max_length=1)
     maintenance_message: str = Field(default="", max_length=300)
     docs_search_mode: str = Field(default="", max_length=20)
+    outbound_proxy_url: str = Field(default="", max_length=500)
     extra_head: str = Field(default="", max_length=50000)
 
+    @field_validator("outbound_proxy_url")
+    @classmethod
+    def validate_outbound_proxy_url(cls, value):
+        text = value.strip()
+        if not text:
+            return text
+        parsed = urlsplit(text)
+        if parsed.scheme not in ("http", "https", "socks5", "socks5h") or not parsed.hostname:
+            raise ValueError("Proxy URL must be http(s):// or socks5(h):// with a host, e.g. http://user:pass@host:port")
+        return text
+
 
 class GamePlantForm(BaseModel):
     slot: int = Field(ge=0, le=64)

devplacepy/rendering.py

@@ -45,6 +45,9 @@ def write_emoji_module(path: Path = EMOJI_JS_PATH) -> int:
 
 EMOJI_MAP = build_emoji_shortcodes()
 
+_WIDGET_RE = re.compile(r"<dp-widget>(.*?)</dp-widget>", re.DOTALL | re.IGNORECASE)
+_WIDGET_PH = "\x00WIDGET_{}\x00"
+
 _SHORTCODE_RE = re.compile(r":([A-Za-z0-9_+\-]+):")
 _YOUTUBE_RE = re.compile(
     r"(?:https?://)?(?:www\.)?"
@@ -66,6 +69,9 @@ _YOUTUBE_ALLOW = (
     "gyroscope; picture-in-picture"
 )
 
+_EMAIL_RE = re.compile(r"\b[A-Za-z0-9._%+\-]+@[A-Za-z0-9.\-]+\.[A-Za-z]{2,}\b")
+_EMAIL_KEEP_DOMAIN = "molodetz.nl"
+
 _MEDIA_SKIP_TAGS = {"a", "code", "pre"}
 _TITLE_INLINE_TAGS = {
     "b", "strong", "i", "em", "code", "del", "s", "mark", "sub", "sup", "span", "br",
@@ -102,7 +108,17 @@ _content_markdown = mistune.create_markdown(
 
 
 def _normalize_dashes(text: str) -> str:
-    return text.replace("\u2014", "-")
+    text = text.replace("\u2014", "-")
+    text = text.replace("\u2013", "-")
+    text = text.replace("&mdash;", "-")
+    text = text.replace("&ndash;", "-")
+    text = text.replace("&#8212;", "-")
+    text = text.replace("&#8211;", "-")
+    text = text.replace("&#x2014;", "-")
+    text = text.replace("&#X2014;", "-")
+    text = text.replace("&#x2013;", "-")
+    text = text.replace("&#X2013;", "-")
+    return text
 
 
 def _replace_shortcodes(text: str) -> str:
@@ -140,12 +156,26 @@ def _embed_url(url: str) -> str:
     )
 
 
+def _mask_email(match: re.Match) -> str:
+    email = match.group(0)
+    local, _, domain = email.partition("@")
+    lowered = domain.lower()
+    if lowered == _EMAIL_KEEP_DOMAIN or lowered.endswith("." + _EMAIL_KEEP_DOMAIN):
+        return email
+    reveal = max(1, len(local) - round(len(local) * 0.8))
+    return f"{local[:reveal]}{'*' * (len(local) - reveal)}@{domain}"
+
+
+def _mask_emails(text: str) -> str:
+    return _EMAIL_RE.sub(_mask_email, text)
+
+
 def _transform_text(text: str) -> str:
     out: list[str] = []
     pos = 0
     for match in _TOKEN_RE.finditer(text):
         if match.start() > pos:
-            out.append(html.escape(text[pos:match.start()]))
+            out.append(html.escape(_mask_emails(text[pos:match.start()])))
         if match.group("url"):
             out.append(_embed_url(match.group("url")))
         else:
@@ -156,7 +186,7 @@ def _transform_text(text: str) -> str:
             )
         pos = match.end()
     if pos < len(text):
-        out.append(html.escape(text[pos:]))
+        out.append(html.escape(_mask_emails(text[pos:])))
     return "".join(out)
 
 
@@ -192,7 +222,7 @@ class _MediaProcessor(HTMLParser):
 
     def handle_data(self, data: str) -> None:
         if self._skip_depth > 0:
-            self._out.append(html.escape(data))
+            self._out.append(html.escape(_mask_emails(data)))
         else:
             self._out.append(_transform_text(data))
 
@@ -218,7 +248,7 @@ class _InlineFilter(HTMLParser):
             self._out.append(f"</{tag}>")
 
     def handle_data(self, data: str) -> None:
-        self._out.append(html.escape(data))
+        self._out.append(html.escape(_mask_emails(data)))
 
     def result(self) -> str:
         return "".join(self._out).strip()
@@ -252,16 +282,42 @@ def _render_title(text: str) -> str:
     return _keep_inline(_content_markdown(text))
 
 
-def render_content(text) -> Markup:
-    if not text:
-        return Markup("")
-    return Markup(_render_content(str(text)))
+def _extract_widgets(text: str) -> tuple[str, list[str]]:
+    widgets: list[str] = []
+    def _replacer(m: re.Match) -> str:
+        widgets.append(m.group(1))
+        return _WIDGET_PH.format(len(widgets) - 1)
+    return _WIDGET_RE.sub(_replacer, text), widgets
 
 
-def render_title(text) -> Markup:
+def _reinsert_widgets(text: str, widgets: list[str]) -> str:
+    for i, widget in enumerate(widgets):
+        text = text.replace(_WIDGET_PH.format(i), widget)
+    return text
+
+
+def render_content(text, author_is_admin: bool = False) -> Markup:
     if not text:
         return Markup("")
-    return Markup(_render_title(str(text)))
+    text_str = str(text)
+    if author_is_admin and _WIDGET_RE.search(text_str):
+        modified, widgets = _extract_widgets(text_str)
+        rendered = _render_content(modified)
+        result = _reinsert_widgets(rendered, widgets)
+        return Markup(result)
+    return Markup(_render_content(text_str))
+
+
+def render_title(text, author_is_admin: bool = False) -> Markup:
+    if not text:
+        return Markup("")
+    text_str = str(text)
+    if author_is_admin and _WIDGET_RE.search(text_str):
+        modified, widgets = _extract_widgets(text_str)
+        rendered = _render_title(modified)
+        result = _reinsert_widgets(rendered, widgets)
+        return Markup(result)
+    return Markup(_render_title(text_str))
 
 
 def content_preview(text, length: int = 60) -> str:

devplacepy/routers/CLAUDE.md

@@ -14,7 +14,7 @@ Prefixes are wired in `main.py`:
 | `/comments` | comments.py |
 | `/projects` | projects/ package - `index.py` (listing/detail/create/delete plus owner-only visibility toggles `POST /{slug}/private` and `POST /{slug}/readonly`, and async `POST /{slug}/fork`), `files.py`, and the `containers/` subpackage (below). `main.py` mounts the whole `/projects` tree from this one package. See `routers/projects/CLAUDE.md` for the deep detail on this tree |
 | `/projects/{slug}/files` | projects/files.py - per-project virtual filesystem (CRUD dirs/files, upload, inline edit, plus line-range ops: `lines` read, `replace-lines`, `insert-lines`, `delete-lines`, `append` for large text files) |
-| `/profile` | profile/ package - `index.py` (page/search/followers/update/api-key; the **Posts** tab shows only the 10 most recent posts, while the sidebar `posts_count` stat always uses the real `count()` so it is never capped to the rendered 10), `customization.py`, `notifications.py`, `ai_correction.py` (`POST /{username}/ai-correction`, owner-or-admin AI content correction toggle + prompt), `ai_modifier.py` (`POST /{username}/ai-modifier`, owner-or-admin AI modifier toggle + prompt), `telegram.py` (`POST /{username}/telegram`, owner-or-admin Telegram pairing-code request/unpair), `usage.py` (`_ai_quota`, re-exported from the package) |
+| `/profile` | profile/ package - `index.py` (page/search/followers/update/api-key; the **Posts** tab shows only the 10 most recent posts, while the sidebar `posts_count` stat always uses the real `count()` so it is never capped to the rendered 10), `customization.py`, `notifications.py`, `ai_correction.py` (`POST /{username}/ai-correction`, owner-or-admin AI content correction toggle + prompt), `ai_modifier.py` (`POST /{username}/ai-modifier`, owner-or-admin AI modifier toggle + prompt), `interactions.py` (`POST /{username}/interactions`, owner-or-admin Devii interactive widgets preference; admin default via `devii_interactions_default`), `telegram.py` (`POST /{username}/telegram`, owner-or-admin Telegram pairing-code request/unpair), `usage.py` (`_ai_quota`, re-exported from the package) |
 | `/messages` | messages.py - real-time DM chat. `GET ""` (page; the conversation sidebar is one windowed SQL aggregate and an opened thread loads only the newest `CONVERSATION_MESSAGE_LIMIT` = 500 messages - keep these queries bounded), `GET /search`, `POST /send` (no-JS fallback), and `WS /messages/ws` (NOT lock-owner gated - accepts on every worker; closes `1008` for guests). The WS adds live bidirectional delivery, typing, read receipts, and in-process presence on top of the existing `messages` table; both the WS `send` and HTTP `POST /send` route through the shared `services/messaging/persist.py` `persist_message` choke point (identical audit/notification/mention) and then through `_finalize_and_broadcast`, which awaits any pending SYNC AI correction/modifier futures (DM content runs through both, so typing `@ai <instruction>` executes live) and broadcasts the FINAL corrected/modified content (the WS path passes `request=websocket` and awaits directly since HTTP middleware does not run for websockets; the frontend swaps the sender's optimistic bubble to the authoritative content). Same-worker delivery is instant via the per-worker `message_hub`; cross-worker delivery is filled by `services/messaging/relay.py` `message_relay` (per-worker ~1s DB poll on the `messages.id` watermark, deduped against the hub's delivered set). See `devplacepy/services/messaging/CLAUDE.md` |
 | `/notifications` | notifications.py |
 | `/votes` | votes.py |

devplacepy/routers/admin/__init__.py

@@ -1,6 +1,7 @@
 # retoor <retoor@molodetz.nl>
 
 from devplacepy.routers.admin import (
+    awards,
     aiquota,
     aiusage,
     auditlog,
@@ -14,13 +15,16 @@ from devplacepy.routers.admin import (
     notifications,
     services,
     settings,
+    statistics,
     trash,
     users,
 )
 from devplacepy.routers.admin.index import router
 
+router.include_router(awards.router)
 router.include_router(users.router)
 router.include_router(aiusage.router)
+router.include_router(statistics.router)
 router.include_router(aiquota.router)
 router.include_router(media.router)
 router.include_router(trash.router)

devplacepy/routers/admin/awards.py

@@ -0,0 +1,49 @@
+# retoor <retoor@molodetz.nl>
+
+import logging
+
+from fastapi import APIRouter, Request
+from devplacepy.database import get_table
+from devplacepy.database.awards import revoke_award
+from devplacepy.responses import action_result, json_error, wants_json
+from devplacepy.services.audit import record as audit
+from devplacepy.utils import not_found, require_admin, safe_next
+
+logger = logging.getLogger(__name__)
+router = APIRouter()
+
+
+def _redirect_back(request: Request, award: dict) -> str:
+    referer = request.headers.get("referer", "")
+    if referer and safe_next(referer, "") == referer:
+        return referer
+    receiver = get_table("users").find_one(uid=award.get("receiver_uid", ""))
+    if receiver:
+        return f"/profile/{receiver['username']}?tab=awards"
+    return "/admin"
+
+
+@router.post("/awards/{uid}/revoke")
+async def admin_revoke_award(request: Request, uid: str):
+    admin = require_admin(request)
+    row = revoke_award(uid, admin["uid"])
+    if not row:
+        if wants_json(request):
+            return json_error(404, "Award not found")
+        raise not_found("Award not found")
+    receiver = get_table("users").find_one(uid=row.get("receiver_uid", ""))
+    logger.info("Admin %s revoked award %s", admin["username"], uid)
+    audit.record(
+        request,
+        "award.revoke",
+        user=admin,
+        target_type="award",
+        target_uid=uid,
+        target_label=row.get("slug", uid),
+        summary=f"admin {admin['username']} revoked award {row.get('slug', uid)}",
+        links=[
+            audit.target("award", uid, row.get("slug")),
+            audit.target("user", row.get("receiver_uid"), receiver.get("username") if receiver else None),
+        ],
+    )
+    return action_result(request, _redirect_back(request, row))

devplacepy/routers/admin/gateway_configs.py

@@ -25,6 +25,8 @@ def _default_provider_summary() -> dict:
         "model": cfg.get("gateway_model", ""),
         "embed_url": cfg.get("gateway_embed_url", ""),
         "embed_model": cfg.get("gateway_embed_model", ""),
+        "image_url": cfg.get("gateway_image_url", ""),
+        "image_model": cfg.get("gateway_image_model", ""),
         "vision_url": cfg.get("gateway_vision_url", ""),
         "vision_model": cfg.get("gateway_vision_model", ""),
     }

devplacepy/routers/admin/statistics.py

@@ -0,0 +1,90 @@
+# retoor <retoor@molodetz.nl>
+
+import logging
+from fastapi import APIRouter, Request
+from fastapi.responses import HTMLResponse, JSONResponse
+from devplacepy.responses import respond
+from devplacepy.schemas.statistics import StatisticsOut
+from devplacepy.seo import base_seo_context, site_url, website_schema
+from devplacepy.services.statistics.build import build_statistics_tab
+from devplacepy.services.statistics.common import VALID_TABS
+from devplacepy.utils import require_admin
+
+logger = logging.getLogger(__name__)
+router = APIRouter()
+
+TAB_LABELS = (
+    ("overview", "Overview", "\U0001f4ca"),
+    ("visitors", "Visitors", "\U0001f441\ufe0f"),
+    ("members", "Members", "\U0001f465"),
+    ("content", "Content", "\U0001f4dd"),
+    ("engagement", "Engagement", "\U0001f525"),
+    ("social", "Social", "\U0001f91d"),
+    ("ai", "AI", "\U0001f916"),
+    ("devii", "Devii", "\u2728"),
+    ("services", "Services", "\u2699\ufe0f"),
+    ("containers", "Containers", "\U0001f4e6"),
+    ("game", "Game", "\U0001f3ae"),
+    ("awards", "Awards", "\U0001f3c6"),
+    ("moderation", "Moderation", "\U0001f6e1\ufe0f"),
+    ("tools", "Tools", "\U0001f527"),
+    ("storage", "Storage", "\U0001f4be"),
+)
+
+
+@router.get("/statistics", response_class=HTMLResponse)
+async def admin_statistics(request: Request, tab: str = "overview", hours: int = 168):
+    admin = require_admin(request)
+    active = tab if tab in VALID_TABS else "overview"
+    base = site_url(request)
+    seo_ctx = base_seo_context(
+        request,
+        title="Statistics - Admin",
+        description="Platform statistics with trends, visitors, content, engagement, and operations.",
+        robots="noindex,nofollow",
+        breadcrumbs=[
+            {"name": "Home", "url": "/feed"},
+            {"name": "Admin", "url": "/admin"},
+            {"name": "Statistics", "url": "/admin/statistics"},
+        ],
+        schemas=[website_schema(base)],
+    )
+    initial = build_statistics_tab(active, hours, compare=True, top_n=10)
+    tabs = [
+        {"key": key, "label": label, "icon": icon, "active": key == active}
+        for key, label, icon in TAB_LABELS
+    ]
+    return respond(
+        request,
+        "admin_statistics.html",
+        {
+            **seo_ctx,
+            "request": request,
+            "user": admin,
+            "admin_section": "statistics",
+            "tabs": tabs,
+            "active_tab": active,
+            "window_hours": hours,
+            "initial": initial,
+        },
+        model=StatisticsOut,
+    )
+
+
+@router.get("/statistics/data")
+async def admin_statistics_data(
+    request: Request,
+    tab: str = "overview",
+    hours: int = 168,
+    compare: int = 1,
+    top_n: int = 10,
+):
+    require_admin(request)
+    return JSONResponse(
+        build_statistics_tab(
+            tab,
+            hours,
+            compare=bool(compare),
+            top_n=top_n,
+        )
+    )

devplacepy/routers/admin/trash.py

@@ -24,13 +24,14 @@ logger = logging.getLogger(__name__)
 router = APIRouter()
 
 TRASH_TABLES = [
-    {"key": "posts", "label": "Posts", "type": "post"},
-    {"key": "comments", "label": "Comments", "type": "comment"},
-    {"key": "gists", "label": "Gists", "type": "gist"},
-    {"key": "projects", "label": "Projects", "type": "project"},
-    {"key": "news", "label": "News", "type": "news"},
-    {"key": "project_files", "label": "Project files", "type": None},
-    {"key": "attachments", "label": "Attachments", "type": None},
+    {"key": "posts", "label": "Posts", "icon": "\U0001f4dd", "type": "post"},
+    {"key": "comments", "label": "Comments", "icon": "\U0001f4ac", "type": "comment"},
+    {"key": "gists", "label": "Gists", "icon": "\U0001f4cb", "type": "gist"},
+    {"key": "projects", "label": "Projects", "icon": "\U0001f680", "type": "project"},
+    {"key": "news", "label": "News", "icon": "\U0001f4f0", "type": "news"},
+    {"key": "awards", "label": "Awards", "icon": "\U0001f3c6", "type": "award"},
+    {"key": "project_files", "label": "Project files", "icon": "\U0001f4c1", "type": None},
+    {"key": "attachments", "label": "Attachments", "icon": "\U0001f4ce", "type": None},
 ]
 _TRASH_KEYS = {entry["key"] for entry in TRASH_TABLES}
 _TRASH_TYPE = {entry["key"]: entry["type"] for entry in TRASH_TABLES}
@@ -113,6 +114,10 @@ async def admin_trash_restore(request: Request, table: str, uid: str):
     row = get_table(table).find_one(uid=uid)
     if row and row.get("deleted_at"):
         restored = restore_event(row["deleted_at"])
+        if table == "awards":
+            from devplacepy.database.awards import recompute_user_award_stats
+
+            recompute_user_award_stats(row.get("receiver_uid", ""))
         logger.info(
             f"Admin {admin['username']} restored {table} {uid} ({restored} rows)"
         )

devplacepy/routers/avatar.py

@@ -17,15 +17,14 @@ _CACHE_CONTROL = f"public, max-age={SECONDS_PER_DAY}, immutable"
 
 @router.get("/{style}/{seed}")
 async def avatar_proxy(request: Request, style: str, seed: str, size: int = 128):
-    cache_key = f"{seed}:{size}"
-    etag = '"' + hashlib.md5(cache_key.encode("utf-8")).hexdigest() + '"'
+    etag = '"' + hashlib.md5(f"{seed}:{size}".encode("utf-8")).hexdigest() + '"'
     headers = {"ETag": etag, "Cache-Control": _CACHE_CONTROL}
 
     if request.headers.get("if-none-match") == etag:
         return Response(status_code=304, headers=headers)
 
-    svg = _cache.get(cache_key)
+    svg = _cache.get(seed)
     if svg is None:
         svg = generate_avatar_svg(seed)
-        _cache.set(cache_key, svg)
+        _cache.set(seed, svg)
     return Response(content=svg, media_type="image/svg+xml", headers=headers)

devplacepy/routers/awards.py

@@ -0,0 +1,36 @@
+# retoor <retoor@molodetz.nl>
+
+from fastapi import APIRouter, Request
+from fastapi.responses import RedirectResponse
+from devplacepy.attachments import _row_to_attachment
+from devplacepy.database import get_table, resolve_by_slug
+from devplacepy.utils import not_found
+
+router = APIRouter()
+
+_VALID_SIZES = {"512", "256", "64"}
+_CACHE_CONTROL = "public, max-age=86400, immutable"
+
+
+@router.get("/{slug_or_uid}/{size}")
+async def award_image(request: Request, slug_or_uid: str, size: str):
+    if size not in _VALID_SIZES:
+        raise not_found("Award image not found")
+    award = resolve_by_slug(get_table("awards"), slug_or_uid)
+    if not award or not award.get("generated_at"):
+        raise not_found("Award image not found")
+    attachment_uid = award.get(f"attachment_uid_{size}") or ""
+    if not attachment_uid:
+        raise not_found("Award image not found")
+    row = get_table("attachments").find_one(uid=attachment_uid, deleted_at=None)
+    attachment = _row_to_attachment(row) if row else None
+    if not attachment:
+        raise not_found("Award image not found")
+    url = attachment.get("url") or ""
+    if not url:
+        raise not_found("Award image not found")
+    headers = {
+        "Cache-Control": _CACHE_CONTROL,
+        "ETag": f'"{attachment_uid}"',
+    }
+    return RedirectResponse(url=url, status_code=302, headers=headers)

devplacepy/routers/devii.py

@@ -185,7 +185,10 @@ async def clippy_proxy(request: Request):
         return JSONResponse({"error": "Devii is unavailable"}, status_code=503)
     cfg = svc.effective_config()
     body = await request.body()
-    headers = {"Content-Type": "application/json"}
+    headers = {
+        "Content-Type": "application/json",
+        "X-App-Reference": "devplace-devii-v-1-0-0",
+    }
     if cfg.get("devii_ai_key"):
         headers["Authorization"] = f"Bearer {cfg['devii_ai_key']}"
     async with stealth.stealth_async_client(timeout=45.0) as client:
@@ -260,6 +263,8 @@ async def devii_ws(websocket: WebSocket):
                 if command == "reset":
                     await session.reset()
                     continue
+                if await session.try_answer_interaction(text):
+                    continue
                 if svc.quota_exceeded(owner_kind, owner_id, owner_is_admin):
                     limit = svc.daily_limit_for(owner_kind, owner_is_admin)
                     audit.record_system(
@@ -302,6 +307,11 @@ async def devii_ws(websocket: WebSocket):
                 )
             elif kind in ("avatar_result", "client_result"):
                 session.resolve_query(str(data.get("id", "")), data.get("result"))
+            elif kind == "interaction_result":
+                session.resolve_interaction(
+                    str(data.get("id", data.get("interaction_id", ""))),
+                    data.get("result") or data,
+                )
     except WebSocketDisconnect:
         pass
     except Exception:  # noqa: BLE001 - never let the socket loop crash the worker

devplacepy/routers/docs/pages.py

@@ -123,6 +123,12 @@ DOCS_PAGES = [
         "kind": "prose",
         "section": SECTION_GENERAL,
     },
+    {
+        "slug": "awards",
+        "title": "Profile awards",
+        "kind": "prose",
+        "section": SECTION_GENERAL,
+    },
     {
         "slug": "ai-correction",
         "title": "AI content correction",

devplacepy/routers/gists.py

@@ -57,6 +57,7 @@ LANGUAGES = [
     ("yaml", "YAML"),
     ("json", "JSON"),
     ("markdown", "Markdown"),
+    ("markdown_rendered", "Markdown Rendered"),
     ("swift", "Swift"),
     ("php", "PHP"),
     ("ruby", "Ruby"),

devplacepy/routers/messages.py

@@ -231,7 +231,7 @@ async def send_message(request: Request, data: Annotated[MessageForm, Depends(js
 async def broadcast_message(
     sender: dict, message: dict, client_id: Optional[str] = None
 ) -> None:
-    frame = message_frame(message, sender.get("username", ""), client_id)
+    frame = message_frame(message, sender.get("username", ""), client_id, sender_role=sender.get("role"))
     message_hub.mark_delivered(message["uid"])
     targets = [message["sender_uid"], message["receiver_uid"]]
     await message_hub.send_to_users(targets, frame)

devplacepy/routers/profile/__init__.py

@@ -4,17 +4,21 @@ from devplacepy.routers.profile import (
     ai_correction,
     ai_modifier,
     avatar,
+    award,
     customization,
+    interactions,
     notifications,
     telegram,
 )
 from devplacepy.routers.profile.index import router
 from devplacepy.routers.profile.usage import _ai_quota
 
+router.include_router(award.router)
 router.include_router(customization.router)
 router.include_router(notifications.router)
 router.include_router(ai_correction.router)
 router.include_router(ai_modifier.router)
+router.include_router(interactions.router)
 router.include_router(avatar.router)
 router.include_router(telegram.router)
 

devplacepy/routers/profile/award.py

@@ -0,0 +1,106 @@
+# retoor <retoor@molodetz.nl>
+
+import logging
+from datetime import datetime, timezone
+from typing import Annotated
+
+from fastapi import APIRouter, Request
+from devplacepy.database import get_blocked_uids, get_table
+from devplacepy.database.awards import can_give_award, has_giver_cooldown, has_receiver_cooldown
+from devplacepy.dependencies import json_or_form
+from devplacepy.models import AwardGiveForm
+from devplacepy.responses import action_result, json_error, wants_json
+from devplacepy.services.audit import record as audit
+from devplacepy.services.jobs import queue
+from devplacepy.utils import generate_uid, make_combined_slug, require_user
+
+logger = logging.getLogger(__name__)
+router = APIRouter()
+
+
+@router.post("/{username}/award")
+async def give_award(
+    request: Request,
+    username: str,
+    data: Annotated[AwardGiveForm, json_or_form(AwardGiveForm)],
+):
+    giver = require_user(request)
+    target = get_table("users").find_one(username=username)
+    redirect = f"/profile/{username}"
+
+    def deny(message: str, status: int = 400):
+        if wants_json(request):
+            return json_error(status, message)
+        return action_result(request, redirect, status_code=302)
+
+    if not target:
+        return deny("User not found", 404)
+    if target["uid"] == giver["uid"]:
+        return deny("You cannot give yourself an award")
+    blocked = get_blocked_uids(giver["uid"])
+    if target["uid"] in blocked:
+        return deny("You cannot give an award to a blocked user")
+    reverse_blocked = get_blocked_uids(target["uid"])
+    if giver["uid"] in reverse_blocked:
+        return deny("You cannot give an award to this user")
+    if has_giver_cooldown(giver["uid"]):
+        return deny("You can give another award later")
+    if has_receiver_cooldown(target["uid"]):
+        return deny("This user received an award recently")
+    if not (giver.get("api_key") or "").strip():
+        return deny("Your account has no API key for award generation")
+
+    description = data.description.strip()
+    uid = generate_uid()
+    slug = make_combined_slug(description, uid)
+    now = datetime.now(timezone.utc).isoformat()
+    get_table("awards").insert(
+        {
+            "uid": uid,
+            "slug": slug,
+            "description": description,
+            "giver_uid": giver["uid"],
+            "receiver_uid": target["uid"],
+            "attachment_uid_512": "",
+            "attachment_uid_256": "",
+            "attachment_uid_64": "",
+            "generated_at": None,
+            "created_at": now,
+            "job_uid": "",
+            "deleted_at": None,
+            "deleted_by": None,
+        }
+    )
+    job_uid = queue.enqueue(
+        "award",
+        {
+            "award_uid": uid,
+            "giver_uid": giver["uid"],
+            "receiver_uid": target["uid"],
+            "description": description,
+            "api_key": giver.get("api_key", ""),
+        },
+        "user",
+        giver["uid"],
+    )
+    get_table("awards").update({"uid": uid, "job_uid": job_uid}, ["uid"])
+    logger.info("%s gave award %s to %s", giver["username"], uid, username)
+    audit.record(
+        request,
+        "award.give",
+        user=giver,
+        target_type="user",
+        target_uid=target["uid"],
+        target_label=username,
+        summary=f"{giver['username']} gave award to {username}",
+        links=[
+            audit.target("user", target["uid"], username),
+            audit.target("award", uid, slug),
+            audit.job(job_uid),
+        ],
+    )
+    return action_result(
+        request,
+        redirect,
+        data={"ok": True, "award_uid": uid, "award_slug": slug},
+    )

devplacepy/routers/profile/index.py

@@ -12,6 +12,7 @@ from devplacepy.database import (
     get_notification_prefs,
     get_user_stars,
     get_user_rank,
+    get_user_post_count,
     get_comment_counts_by_post_uids,
     get_reactions_by_targets,
     get_user_bookmarks,
@@ -28,6 +29,11 @@ from devplacepy.database import (
     mark_notifications_read_by_target,
     resolve_object_url,
 )
+from devplacepy.database.awards import (
+    can_give_award,
+    get_prominent_award,
+    get_user_awards,
+)
 from devplacepy.content import can_view_project, enrich_items
 from devplacepy.utils import (
     get_current_user,
@@ -147,6 +153,17 @@ async def profile_page(
     if tab == "media":
         media, media_pagination = get_user_media(profile_user["uid"], page)
 
+    awards, awards_pagination = [], None
+    if tab == "awards":
+        awards, awards_pagination = get_user_awards(profile_user["uid"], page)
+    prominent_award = get_prominent_award(profile_user)
+    awards_count = int(profile_user.get("award_count") or 0)
+    can_give = bool(
+        current_user
+        and current_user["uid"] != profile_user["uid"]
+        and can_give_award(current_user["uid"], profile_user["uid"])
+    )
+
     posts = []
     if tab == "posts":
         posts_table = get_table("posts")
@@ -195,9 +212,7 @@ async def profile_page(
         )
         for g in gists_raw:
             gists.append({"gist": g, "time_ago": time_ago(g["created_at"])})
-    posts_count = get_table("posts").count(
-        user_uid=profile_user["uid"], deleted_at=None
-    )
+    posts_count = get_user_post_count(profile_user["uid"])
 
     activities = []
     if tab == "activity":
@@ -286,6 +301,18 @@ async def profile_page(
         if is_owner
         else None
     )
+    from devplacepy.services.devii.interaction import prefs as interaction_prefs
+
+    interactions_snap = (
+        interaction_prefs.snapshot("user", profile_user["uid"], profile_user)
+        if is_owner
+        else {
+            "enabled": True,
+            "source": None,
+            "default": True,
+            "override": None,
+        }
+    )
     from devplacepy.services.telegram import store as telegram_store
 
     telegram_paired = (
@@ -386,6 +413,10 @@ async def profile_page(
             "ai_modifier_enabled": ai_modifier_enabled,
             "ai_modifier_sync": ai_modifier_sync,
             "ai_modifier_prompt": ai_modifier_prompt,
+            "interactions_enabled": interactions_snap["enabled"],
+            "interactions_source": interactions_snap["source"],
+            "interactions_default": interactions_snap["default"],
+            "interactions_override": interactions_snap["override"],
             "telegram_paired": telegram_paired,
             "notif_telegram_paired": notif_telegram_paired,
             "can_manage_customization": can_manage_customization,
@@ -404,6 +435,11 @@ async def profile_page(
             "follow_pagination": follow_pagination,
             "followers_count": follow_counts["followers"],
             "following_count": follow_counts["following"],
+            "awards": awards,
+            "awards_pagination": awards_pagination,
+            "awards_count": awards_count,
+            "prominent_award": prominent_award,
+            "can_give_award": can_give,
         },
         model=ProfileOut,
     )

devplacepy/routers/profile/interactions.py

@@ -0,0 +1,61 @@
+# retoor <retoor@molodetz.nl>
+
+import logging
+from typing import Annotated
+
+from fastapi import Depends, APIRouter, Request
+
+from devplacepy.models import InteractionsForm
+from devplacepy.responses import action_result
+from devplacepy.services.audit import record as audit
+from devplacepy.services.devii.interaction import prefs
+from devplacepy.routers.profile._shared import resolve_customization_target
+from devplacepy.dependencies import json_or_form
+
+logger = logging.getLogger(__name__)
+router = APIRouter()
+
+
+@router.post("/{username}/interactions")
+async def set_interactions(
+    request: Request,
+    username: str,
+    data: Annotated[InteractionsForm, Depends(json_or_form(InteractionsForm))],
+):
+    target, denied = resolve_customization_target(request, username)
+    if denied is not None:
+        return denied
+    if data.reset:
+        snap = prefs.set_user_pref(target["uid"], None)
+        summary = f"reset interactive widgets to admin default for {target['username']}"
+        new_value = -1
+    else:
+        snap = prefs.set_user_pref(target["uid"], bool(data.enabled))
+        summary = (
+            f"{'enabled' if data.enabled else 'disabled'} interactive widgets "
+            f"for {target['username']}"
+        )
+        new_value = 1 if data.enabled else 0
+    logger.info(summary)
+    audit.record(
+        request,
+        "profile.interactions",
+        target_type="user",
+        target_uid=target["uid"],
+        target_label=target["username"],
+        new_value=new_value,
+        summary=summary,
+        links=[audit.target("user", target["uid"], target["username"])],
+    )
+    url = f"/profile/{target['username']}"
+    return action_result(
+        request,
+        url,
+        data={
+            "url": url,
+            "enabled": snap["enabled"],
+            "source": snap["source"],
+            "default": snap["default"],
+            "override": snap["override"],
+        },
+    )

devplacepy/routers/xmlrpc.py

@@ -4,6 +4,7 @@ import logging
 
 import httpx
 from fastapi import APIRouter, Request
+from starlette.requests import ClientDisconnect
 from starlette.responses import PlainTextResponse, Response
 
 from devplacepy.config import XMLRPC_BIND, XMLRPC_PORT
@@ -49,7 +50,11 @@ async def info(request: Request, path: str = "") -> PlainTextResponse:
 @router.post("/")
 @router.post("/{path:path}")
 async def proxy(request: Request, path: str = "") -> Response:
-    body = await request.body()
+    try:
+        body = await request.body()
+    except ClientDisconnect:
+        logger.warning("XML-RPC client disconnected before request body was read")
+        return PlainTextResponse("Client disconnected", status_code=400)
     headers = {
         key: value
         for key, value in request.headers.items()

devplacepy/schemas/__init__.py

@@ -114,6 +114,7 @@ from devplacepy.schemas.gateway import (
     GatewayUsageOut,
     UserAiUsageOut,
 )
+from devplacepy.schemas.statistics import StatisticsOut
 from devplacepy.schemas.auth import (
     AuthPageOut,
     DeviiPageOut,

devplacepy/schemas/auth.py

@@ -40,12 +40,23 @@ class LandingPostOut(_Out):
     slug: str = ""
 
 
+class TrendingTopicOut(_Out):
+    topic: str = ""
+    count: int = 0
+
+
 class LandingOut(_Out):
     is_authenticated: bool = False
     user_post_count: int = 0
     user_stars: int = 0
+    user_xp: int = 0
+    user_level: int = 1
+    xp_progress_pct: int = 0
+    unread_count: int = 0
     landing_articles: list[LandingArticleOut] = []
     landing_posts: list[LandingPostOut] = []
+    top_contributors: list = []
+    trending_topics: list[TrendingTopicOut] = []
 
 
 class DeviiPageOut(_Out):

devplacepy/schemas/awards.py

@@ -0,0 +1,19 @@
+# retoor <retoor@molodetz.nl>
+
+from typing import Optional
+
+from devplacepy.schemas.base import _Out
+from devplacepy.schemas.content import UserOut
+
+
+class AwardOut(_Out):
+    uid: str = ""
+    slug: str = ""
+    description: str = ""
+    giver_uid: str = ""
+    receiver_uid: str = ""
+    generated_at: Optional[str] = None
+    created_at: Optional[str] = None
+    image_url: Optional[str] = None
+    thumb_url: Optional[str] = None
+    giver: Optional[UserOut] = None

devplacepy/schemas/profile.py

@@ -6,6 +6,7 @@ from typing import Any, Optional
 
 from devplacepy.schemas.base import _Out
 from devplacepy.schemas.content import BadgeOut, ProjectOut, UserOut
+from devplacepy.schemas.awards import AwardOut
 from devplacepy.schemas.listings import FeedItemOut, GistItemOut
 
 
@@ -49,6 +50,10 @@ class ProfileOut(_Out):
     ai_modifier_enabled: bool = False
     ai_modifier_sync: bool = False
     ai_modifier_prompt: Optional[str] = None
+    interactions_enabled: bool = True
+    interactions_source: Optional[str] = None
+    interactions_default: bool = True
+    interactions_override: Optional[bool] = None
     telegram_paired: bool = False
     notif_telegram_paired: bool = False
     can_manage_customization: bool = False
@@ -70,6 +75,11 @@ class ProfileOut(_Out):
     media: list[MediaItemOut] = []
     media_pagination: Optional[Any] = None
     notification_prefs: list[Any] = []
+    awards: list[AwardOut] = []
+    awards_pagination: Optional[Any] = None
+    awards_count: int = 0
+    prominent_award: Optional[AwardOut] = None
+    can_give_award: bool = False
 
 
 class TelegramPairOut(_Out):

devplacepy/schemas/statistics.py

@@ -0,0 +1,52 @@
+# retoor <retoor@molodetz.nl>
+
+from __future__ import annotations
+
+from typing import Any, Optional
+
+from pydantic import BaseModel
+
+
+class StatisticsMetricOut(BaseModel):
+    key: str
+    label: str
+    value: Any
+    format: str = "int"
+    delta: Optional[float] = None
+    direction: Optional[str] = None
+
+
+class StatisticsPointOut(BaseModel):
+    t: str
+    v: float
+
+
+class StatisticsSeriesOut(BaseModel):
+    key: str
+    label: str
+    points: list[StatisticsPointOut]
+
+
+class StatisticsTableOut(BaseModel):
+    key: str
+    title: str
+    columns: list[str]
+    rows: list[list[Any]]
+
+
+class StatisticsHighlightOut(BaseModel):
+    label: str
+    value: Any
+
+
+class StatisticsOut(BaseModel):
+    tab: str
+    window_hours: int
+    granularity: str
+    generated_at: str
+    compare: bool = True
+    cards: list[StatisticsMetricOut] = []
+    series: list[StatisticsSeriesOut] = []
+    tables: list[StatisticsTableOut] = []
+    highlights: list[StatisticsHighlightOut] = []
+    notes: dict[str, Any] = {}

devplacepy/services/CLAUDE.md

@@ -22,9 +22,9 @@ Generic, lightweight, **fire-and-forget** offload for non-critical side-effects
 
 **Opt-in, default off, server-side.** When a user turns it on (profile **AI content correction** block, owner-only, saved at `POST /profile/{username}/ai-correction`), the prose they author is rewritten by the AI gateway. The per-user `ai_correction_sync` flag (0 = background default, 1 = sync) picks the **apply mode**: background rewrites the stored fields a moment after the write (never slows the request); sync makes the HTTP response wait until the correction is applied.
 
-- **Sync must never block the event loop (the self-deadlock trap).** The correction POSTs to the in-process gateway (`INTERNAL_GATEWAY_URL` = `localhost:{PORT}`), and the hooked content helpers are synchronous on the loop thread, so a blocking inline call self-deadlocks the (single) worker against its own gateway request - on a single worker that loop is the only thing that can serve the gateway request it is waiting on, so a blocking inline call deadlocks the worker until the httpx timeout, then fail-softs (server hangs, no correction). The fix: sync runs `_run_correction` via `loop.run_in_executor` (loop stays free), stashes the future on `request.scope[PENDING_SCOPE_KEY]`, and the `await_pending_corrections` middleware in `main.py` awaits it after the handler. Never reintroduce a blocking inline correction on the loop thread.
+- **Sync must never block the event loop (the self-deadlock trap).** The correction POSTs to the in-process gateway (`INTERNAL_GATEWAY_URL` = `localhost:{PORT}`), and the hooked content helpers are synchronous on the loop thread, so a blocking inline call self-deadlocks the (single) worker against its own gateway request - on a single worker that loop is the only thing that can serve the gateway request it is waiting on, so a blocking inline call deadlocks the worker until the httpx timeout, then fail-softs (server hangs, no correction). The fix: sync runs `_run_correction` on the dedicated `AI_APPLY_EXECUTOR` thread pool (`correction.py`, 4 workers) via `loop.run_in_executor` (loop stays free, and the default executor - shared with `asyncio.to_thread` password hashing at login/signup - is never occupied by blocking AI HTTP calls), stashes the future on `request.scope[PENDING_SCOPE_KEY]`, and the `await_pending_corrections` middleware in `main.py` awaits it after the handler. Never reintroduce a blocking inline correction on the loop thread.
 - **Field registry is the single source of truth.** `CORRECTABLE_FIELDS: dict[str, tuple[str, ...]]` maps each correctable table to its prose columns: `posts` -> `(title, content)`, `projects`/`gists` -> `(title, description)`, `comments`/`messages` -> `(content,)`, `users` -> `(bio,)`. `gists.source_code`, project files, and Gitea issues are intentionally excluded - code and external systems are never corrected.
-- **Choke entrypoint.** `schedule_correction(user, table, uid, request=None)` is the only thing handlers call (every hook passes `request`). It is a no-op unless: a user dict is present, `table` is in the registry, `user["ai_correction_enabled"]` is truthy, and the user has a non-empty `api_key`. In **sync** mode (`user["ai_correction_sync"]`) it calls `_run_inline_awaited`, which - only when a `request.scope` and a running event loop exist - submits `_run_correction` to `loop.run_in_executor` (off the loop thread) and stashes the future on `request.scope[PENDING_SCOPE_KEY]` for the middleware to await; if there is no request/loop it returns False and falls back to background. In **background** mode (default) it `background.submit`s `_run_correction` (per-worker queue; also runs inline under `DEVPLACE_DISABLE_SERVICES=1`). The same `_run_correction` worker function runs in all cases.
+- **Choke entrypoint.** `schedule_correction(user, table, uid, request=None)` is the only thing handlers call (every hook passes `request`). It is a no-op unless: a user dict is present, `table` is in the registry, `user["ai_correction_enabled"]` is truthy, and the user has a non-empty `api_key`. In **sync** mode (`user["ai_correction_sync"]`) it calls `_run_inline_awaited`, which - only when a `request.scope` and a running event loop exist - submits `_run_correction` to `loop.run_in_executor(AI_APPLY_EXECUTOR, ...)` (off the loop thread, on the dedicated AI pool) and stashes the future on `request.scope[PENDING_SCOPE_KEY]` for the middleware to await; if there is no request/loop it returns False and falls back to background. In **background** mode (default) it `background.submit`s `_run_correction` (per-worker queue; also runs inline under `DEVPLACE_DISABLE_SERVICES=1`). The same `_run_correction` worker function runs in all cases.
 - **The hooks (covers UI + REST + Devii + devRant in one place):** `content.create_content_item` (posts/projects/gists), `content.edit_content_item`, `content.create_comment_record`, `content.edit_comment_record`, `services/messaging/persist.persist_message` (DMs - sender is the user), `routers/profile/index.update_profile` (bio), and the two devRant direct-update edit paths (`routers/devrant/rants.edit_rant`, `routers/devrant/comments.edit_comment`). devRant create paths route through the shared content cores, so they are already hooked. Code-only and external paths (project files, Gitea) are deliberately not hooked.
 - **The gateway call is fail-soft.** `correct_text(api_key, prompt, text)` is synchronous (runs on the background worker thread), POSTs to `INTERNAL_GATEWAY_URL` with `model=INTERNAL_MODEL` via `stealth.stealth_sync_client`, authenticated with the user's own `Bearer` api_key (per-user attribution). It returns the ORIGINAL text on any error, empty output, or suspiciously large output (`len > len(text) * MAX_GROWTH_FACTOR + 200`, rejecting hallucinated expansion). `_run_correction` reads the row back, corrects each registry field that has non-blank text, writes only changed fields via `table.update(updates, ["uid"])` without touching `updated_at` (auto-correction is not a user edit) or the slug (slugs are permanent), and `clear_user_cache(user_uid)` when `table == "users"` so the corrected bio re-caches.
 - **Per-user usage aggregation (only on success).** `correct_text` returns `(text, usage)`; `usage` is parsed from the gateway's `X-Gateway-*` response headers (`_usage_from_headers`) whenever the upstream call returned 200 (cost was incurred, even if the corrected output was rejected), else `None` on any failure. `_usage_from_headers` captures the token and cost headers PLUS the timing headers `X-Gateway-Upstream-Latency-Ms` and `X-Gateway-Total-Latency-Ms` (as `upstream_latency_ms`/`total_latency_ms`), so each call's timing is metered. `_run_correction` accumulates the per-field `usage` into one `totals` dict and, when `totals["calls"] > 0`, makes ONE call to `database.add_correction_usage(user_uid, totals)` (a single `totals` dict, not positional args) - so a 2-field content item is a single aggregated write, and a failed/empty correction records nothing. `add_correction_usage` (and `add_modifier_usage`) delegate to the shared `database._add_usage(usage_table, user_uid, totals)`: a single atomic `INSERT ... ON CONFLICT(user_uid) DO UPDATE SET col = col + excluded.col` upsert against the `correction_usage` table (per-user running SUMS: `calls`/`prompt_tokens`/`completion_tokens`/`total_tokens`/`cost_usd`/`upstream_latency_ms`/`total_latency_ms`/`updated_at`, unique index `idx_correction_usage_user`, the two latency columns REAL default 0.0, all ensured in `init_db`). It is a derived counter table (NOT in `SOFT_DELETE_TABLES`, like `gateway_usage_ledger`) and is deliberately separate from `users` so accumulating never invalidates the auth/user cache. `database.get_correction_usage(user_uid)` (via `_get_usage`) returns the stored sums PLUS computed averages: `avg_tokens` (total_tokens/calls), `avg_upstream_latency_ms`, `avg_total_latency_ms`, `avg_tokens_per_second` (completion_tokens over total upstream seconds), and `avg_cost_usd`.
@@ -57,7 +57,7 @@ This section covers only the shared machinery. The individual services built on
 
 In production (`make prod`, 2 workers) only the lock-holding worker runs services, but an admin request can hit either worker. State therefore lives in the DB, not process memory:
 - **Desired/config state** in `site_settings` (via `get_setting`/`set_setting`): `service_<name>_enabled` (`"0"`/`"1"` - also the boot flag), `service_<name>_command` (`"<verb>:<counter>"`, verbs `run`/`clear`), `service_<name>_log_size`, plus each declared config field's own key.
-- **Observed state** in the `service_state` table (one row per service): `status`, `last_run`, `next_run`, `started_at`, `heartbeat`, `logs` (JSON), `updated_at`. Written by the supervising worker, read by any worker.
+- **Observed state** in the `service_state` table (one row per service): `status`, `last_run`, `next_run`, `started_at`, `heartbeat`, `logs` (JSON), `updated_at`. Written by the supervising worker, read by any worker. The heartbeat persists every `PERSIST_SECONDS` (8s, under the 15s `STALE_SECONDS` liveness window) and caches the row id so a persist is one UPDATE, not a `find_one` + UPDATE. `collect_metrics()` runs on its own slower `METRICS_SECONDS` cadence (15s default, per-class overridable - `AuditService` uses 300s because its metric is a `COUNT(*)` over the ever-growing audit table); between refreshes the last snapshot is re-persisted, and a `force` persist (transitions, run-now) always recomputes. Keep expensive aggregates out of the 1s tick: put them in `collect_metrics` and, if still heavy, raise the subclass `METRICS_SECONDS`.
 
 `main.py` registers every service in **all** workers (so `describe_all()` works anywhere) but only the lock worker calls `service_manager.supervise()`.
 
@@ -118,7 +118,7 @@ devplace devii reset-quota --all        # Reset every quota (users and guests)
 
 `uvicorn --workers N` = N independent processes sharing only the filesystem and SQLite DB. Module-global caches/counters are per-process, so a local `clear()` is invisible to siblings. Full reference: admin docs `Production -> Multi-worker and concurrency` (`templates/docs/production-concurrency.html`). Enforce these:
 
-- **In-process caches are never authoritative across workers.** Coordinate invalidation through the `cache_state` version table in `database.py`: call `bump_cache_version(name)` at every write path and `sync_local_cache(name, cache)` before every read path. Existing names: `auth` (guards `_user_cache`; bumped by `clear_user_cache`/`clear_session_cache` on logout, ban, role/password change), `settings` (guards `_settings_cache`; bumped by `set_setting`/`clear_settings_cache`), `relations` (`_relations_cache`), `customizations` (`_customizations_cache`), `notif_prefs` (`_notification_prefs_cache`), `gateway_routing` (`_ROUTING_CACHE`), and **`admins`** (guards `_admins_cache`, which memoizes `get_admin_uids()` / `get_primary_admin_uid()`). The `_cache_version_cache` (`ttl=1`) caches the version reads themselves, so the bump is seen by other workers within ~1s on their next read (the originating worker is immediate); the version read/bump fail open (logged, never raise). When you add a per-process cache whose staleness matters, give it a name and wire both calls - do not invent a second invalidation mechanism.
+- **In-process caches are never authoritative across workers.** Coordinate invalidation through the `cache_state` version table in `database.py`: call `bump_cache_version(name)` at every write path and `sync_local_cache(name, cache)` before every read path. Existing names: `auth` (guards `_user_cache`; bumped by `clear_user_cache`/`clear_session_cache` on logout, ban, role/password change - a version bump makes EVERY worker clear its WHOLE `_user_cache`, so display-only refreshes (XP/level in `award_xp`, AI-corrected bio) call `clear_user_cache(uid, propagate=False)` for a local pop without the global bump; identity/authz changes (logout, ban, role, password, api-key/token revoke) MUST keep the default propagate=True), `settings` (guards `_settings_cache`; bumped by `set_setting`/`clear_settings_cache`), `relations` (`_relations_cache`), `customizations` (`_customizations_cache`), `notif_prefs` (`_notification_prefs_cache`), `gateway_routing` (`_ROUTING_CACHE`), and **`admins`** (guards `_admins_cache`, which memoizes `get_admin_uids()` / `get_primary_admin_uid()`). The `_cache_version_cache` (`ttl=1`) caches the version reads themselves, so the bump is seen by other workers within ~1s on their next read (the originating worker is immediate); the version read/bump fail open (logged, never raise). When you add a per-process cache whose staleness matters, give it a name and wire both calls - do not invent a second invalidation mechanism.
 - **Admin-set cache invariant (load-bearing).** `_admins_cache` makes `get_admin_uids()` / `get_primary_admin_uid()` (hit on the projects-listing visibility filter, `_owner_is_admin`, and every primary-admin gate: `/dbapi`, backup-archive download) a dict lookup instead of a per-call `SELECT`. It is NOT the authorization gate - `is_admin(user)`/`require_admin` read the role off the user object (independently invalidated via `clear_user_cache`), so a stale admin set cannot grant access. But **any code that writes `users.role` MUST call `database.invalidate_admins_cache()`** (clears local + bumps the `admins` version), exactly like the soft-delete and `with db:` rules. Current role-write sites all do: `routers/admin/users.py` (role change), `cli.py` (`role set`), and `utils._create_account` (first user -> Admin). Omitting the call leaves the admin set stale for up to the 300s TTL.
 - **Deterministic / eventual caches skip version-sync on purpose.** Two caches need no `cache_state` name because correctness does not depend on cross-worker freshness: (1) `docs_prose._render_markdown` is an `@lru_cache` on the **static** prose source string (pure function of template content, so identical on every worker; changes only on deploy), and (2) `main._home_cache` (`TTLCache ttl=60`) memoizes the guest `/` blocks - the featured-news block (identical for everyone) and the latest-posts block **only for the no-block case**; a viewer with a non-empty block set bypasses the cache and is computed fresh, so the original block-filter semantics are preserved byte-for-byte and there is zero cross-user leakage. Worst case is a soft-deleted/edited public post lingering on `/` for <=60s. Use a plain `TTLCache`/`lru_cache` (no version name) ONLY when the value is deterministic or its staleness is purely cosmetic; anything whose staleness affects correctness or permissions MUST use the version-sync pattern above.
 - **Authoritative state lives in the DB**, memory is only a short-TTL accelerator (sessions, the Devii 24h cap via `devii_usage_ledger`, cost counters).
@@ -132,7 +132,7 @@ Read-mostly aggregates that were recomputed per request or per vote sit behind s
 
 | Cache | Where | Key | TTL | Invalidation |
 |---|---|---|---|---|
-| `_authors_cache` | `database/ranking.py` | `ranked`/`rank_map` | 15s | TTL only - `update_target_stars` deliberately does NOT clear it per vote anymore (the old per-vote `clear()` forced a full UNION-JOIN ranking recompute on the next feed/leaderboard/landing request and never propagated cross-worker anyway) |
+| `_authors_cache` | `database/ranking.py` | `ranked`/`rank_map` | 60s | TTL only - `update_target_stars` deliberately does NOT clear it per vote anymore (the old per-vote `clear()` forced a full UNION-JOIN ranking recompute on the next feed/leaderboard/landing request and never propagated cross-worker anyway) |
 | `_stars_cache` | `database/ranking.py` | user uid | 15s | TTL + `clear_user_stars(owner_uid)` from `content.apply_vote`, so a user's own total updates immediately on the voting worker |
 | `_leaderboard_cache` | `services/game/store/farm.py` | `top:{limit}` | 15s | TTL only (the farm scan + Python `farm_score` sort runs at most once per 15s per worker) |
 | `_projects_cache` | `templating.py` | user uid | 10s | TTL + `clear_user_projects_cache(uid)` from the `content.py` project create/delete choke points (in-function import - `templating` imports `content` at module level). `jinja_user_projects` also filters `deleted_at=None` now (the composer dropdown previously listed soft-deleted projects) |

devplacepy/services/access_tokens.py

@@ -90,6 +90,9 @@ def revoke_token(uid: str) -> bool:
         return False
     stamp = datetime.now(timezone.utc).isoformat()
     tokens.update({"id": row["id"], "deleted_at": stamp, "deleted_by": "manual"}, ["id"])
+    from devplacepy.utils.authcache import clear_user_cache
+
+    clear_user_cache(row["user_uid"])
     return True
 
 
@@ -101,6 +104,10 @@ def revoke_all(user_uid: str) -> int:
     for row in list(tokens.find(user_uid=user_uid, deleted_at=None)):
         tokens.update({"id": row["id"], "deleted_at": stamp, "deleted_by": "manual"}, ["id"])
         count += 1
+    if count:
+        from devplacepy.utils.authcache import clear_user_cache
+
+        clear_user_cache(user_uid)
     return count
 
 

devplacepy/services/ai_modifier.py

@@ -61,19 +61,20 @@ def schedule_modification(
     prompt = (user.get("ai_modifier_prompt") or DEFAULT_MODIFIER_PROMPT).strip()
     user_uid = user.get("uid") or ""
     if user.get("ai_modifier_sync") and schedule_pending(
-        _run_modification, request, api_key, prompt, table, uid, user_uid
+        _run_modification, request, api_key, prompt, table, uid, user_uid, row
     ):
         return
     background.submit(_run_modification, api_key, prompt, table, uid, user_uid)
 
 
 def _run_modification(
-    api_key: str, prompt: str, table: str, uid: str, user_uid: str
+    api_key: str, prompt: str, table: str, uid: str, user_uid: str, row: dict | None = None
 ) -> None:
     fields = CORRECTABLE_FIELDS.get(table)
     if not fields:
         return
-    row = get_table(table).find_one(uid=uid)
+    if row is None:
+        row = get_table(table).find_one(uid=uid)
     if not row:
         return
     updates: dict = {}
@@ -97,6 +98,6 @@ def _run_modification(
         if table == "users" and user_uid:
             from devplacepy.utils import clear_user_cache
 
-            clear_user_cache(user_uid)
+            clear_user_cache(user_uid, propagate=False)
     if totals["calls"] and user_uid:
         add_modifier_usage(user_uid, totals)

devplacepy/services/audit/categories.py

@@ -4,6 +4,7 @@ CATEGORY_BY_PREFIX: dict[str, str] = {
     "auth": "auth",
     "profile": "account",
     "follow": "social",
+    "award": "social",
     "relation": "social",
     "push": "push",
     "notification": "notification",

devplacepy/services/audit/service.py

@@ -18,6 +18,7 @@ class AuditService(BaseService):
     description = "Prunes audit_log rows and their links older than the retention window."
     default_enabled = True
     min_interval = 3600
+    METRICS_SECONDS = 300
     config_fields = [
         ConfigField(
             RETENTION_KEY,

devplacepy/services/audit/store.py

@@ -101,16 +101,15 @@ def insert_event(row: dict) -> str:
         record["created_at"] = now()
     if record.get("via_agent") is None:
         record["via_agent"] = 0
-    get_table(AUDIT_TABLE).insert(record)
+    get_table(AUDIT_TABLE).insert(record, ensure=False)
     return record["uid"]
 
 
 def insert_links(audit_uid: str, links: list[dict]) -> int:
     if not links:
         return 0
-    table = get_table(LINKS_TABLE)
     stamp = now()
-    count = 0
+    records = []
     for link in links:
         record = _empty_link()
         record.update(
@@ -121,9 +120,10 @@ def insert_links(audit_uid: str, links: list[dict]) -> int:
         record["created_at"] = stamp
         if not record.get("object_uid"):
             continue
-        table.insert(record)
-        count += 1
-    return count
+        records.append(record)
+    if records:
+        get_table(LINKS_TABLE).insert_many(records, ensure=False)
+    return len(records)
 
 
 def get_event(uid: str) -> Optional[dict]:

devplacepy/services/background.py

@@ -64,6 +64,16 @@ class BackgroundQueue:
                 if inline:
                     self._inline += 1
             logger.warning("background task %s failed: %s", getattr(fn, "__name__", fn), exc)
+        finally:
+            self._release_db_lock()
+
+    def _release_db_lock(self) -> None:
+        try:
+            from devplacepy.database import refresh_snapshot
+
+            refresh_snapshot()
+        except Exception as exc:
+            logger.warning("background queue could not release db lock: %s", exc)
 
     async def start(self) -> None:
         if self.running:

devplacepy/services/base.py

@@ -123,7 +123,8 @@ class BaseService(ABC):
     details = ""
     DEFAULT_LOG_SIZE = 20
     TICK_SECONDS = 1
-    PERSIST_SECONDS = 3
+    PERSIST_SECONDS = 8
+    METRICS_SECONDS = 15
     STALE_SECONDS = 15
 
     def __init__(self, name: str, interval_seconds: int = 3600):
@@ -144,6 +145,9 @@ class BaseService(ABC):
         self._run_pending = False
         self._last_command = None
         self._last_persist = None
+        self._last_metrics = None
+        self._metrics_snapshot = {}
+        self._state_row_id = None
         self.enabled_field = ConfigField(
             self.enabled_key,
             "Enabled",
@@ -234,6 +238,14 @@ class BaseService(ABC):
             logger.warning(f"Could not collect metrics for {self.name}: {e}")
             return {}
 
+    def _current_metrics(self, now, force: bool = False) -> dict:
+        if not force and self._last_metrics is not None:
+            if (now - self._last_metrics).total_seconds() < self.METRICS_SECONDS:
+                return self._metrics_snapshot
+        self._last_metrics = now
+        self._metrics_snapshot = self._safe_metrics()
+        return self._metrics_snapshot
+
     def start_supervisor(self) -> None:
         if self._task is not None:
             return
@@ -344,16 +356,18 @@ class BaseService(ABC):
             "started_at": self._started_at.isoformat() if self._started_at else "",
             "heartbeat": now.isoformat(),
             "logs": json.dumps(list(self.log_buffer)),
-            "metrics": json.dumps(self._safe_metrics()),
+            "metrics": json.dumps(self._current_metrics(now, force=force)),
             "updated_at": now.isoformat(),
         }
         try:
             table = get_table("service_state")
-            existing = table.find_one(name=self.name)
-            if existing:
-                table.update({**record, "id": existing["id"]}, ["id"])
+            if self._state_row_id is None:
+                existing = table.find_one(name=self.name)
+                self._state_row_id = existing["id"] if existing else None
+            if self._state_row_id is not None:
+                table.update({**record, "id": self._state_row_id}, ["id"])
             else:
-                table.insert(record)
+                self._state_row_id = table.insert(record)
         except Exception as e:
             logger.warning(f"Could not persist state for {self.name}: {e}")
 

devplacepy/services/bot/engage.py

@@ -387,7 +387,7 @@ class BotEngageMixin:
             idx = random.randrange(limit)
             target = comments.nth(idx)
             try:
-                text = (await target.inner_text() or "")[:1000]
+                text = (await target.inner_text() or "")[:125000]
             except Exception:
                 continue
             if not text:

devplacepy/services/bot/social.py

@@ -433,7 +433,7 @@ class BotSocialMixin:
         mentioner = ""
         if comment_loc is not None:
             try:
-                parent_text = (await comment_loc.inner_text() or "")[:1000]
+                parent_text = (await comment_loc.inner_text() or "")[:125000]
             except Exception:
                 parent_text = ""
             try:

devplacepy/services/containers/files/bot.py

@@ -72,7 +72,7 @@ MENTION_POLL_SECONDS = int(os.environ.get("MENTION_POLL_SECONDS", "30"))
 DM_POLL_SECONDS = int(os.environ.get("DM_POLL_SECONDS", "10"))
 BOT_USERNAME = os.environ.get("BOT_USERNAME", "")
 
-COMMENT_CHAR_LIMIT = 1000
+COMMENT_CHAR_LIMIT = 125000
 MESSAGE_CHAR_LIMIT = 2000
 PART_SUFFIX_RESERVE = 12
 PART_DELIVERY_DELAY = 0.5

devplacepy/services/containers/files/bot.py.bak

@@ -72,7 +72,7 @@ MENTION_POLL_SECONDS = int(os.environ.get("MENTION_POLL_SECONDS", "30"))
 DM_POLL_SECONDS = int(os.environ.get("DM_POLL_SECONDS", "10"))
 BOT_USERNAME = os.environ.get("BOT_USERNAME", "")
 
-COMMENT_CHAR_LIMIT = 1000
+COMMENT_CHAR_LIMIT = 125000
 MESSAGE_CHAR_LIMIT = 2000
 PART_SUFFIX_RESERVE = 12
 PART_DELIVERY_DELAY = 0.5

devplacepy/services/correction.py

@@ -2,6 +2,8 @@
 
 import asyncio
 import logging
+import threading
+from concurrent.futures import ThreadPoolExecutor
 
 import httpx
 
@@ -30,6 +32,22 @@ CORRECTION_TIMEOUT_SECONDS = 20.0
 MAX_GROWTH_FACTOR = 3
 PENDING_SCOPE_KEY = "devplace_pending_corrections"
 
+AI_APPLY_EXECUTOR = ThreadPoolExecutor(max_workers=4, thread_name_prefix="ai-apply")
+
+_gateway_client: httpx.Client | None = None
+_gateway_client_lock = threading.Lock()
+
+
+def _client() -> httpx.Client:
+    global _gateway_client
+    if _gateway_client is None:
+        with _gateway_client_lock:
+            if _gateway_client is None:
+                _gateway_client = stealth.stealth_sync_client(
+                    timeout=CORRECTION_TIMEOUT_SECONDS
+                )
+    return _gateway_client
+
 
 def _usage_from_headers(response_headers) -> dict:
     parsed = parse_usage_headers(response_headers)
@@ -72,21 +90,25 @@ def gateway_complete(
         ],
         "temperature": 0.1,
     }
-    headers = {"Content-Type": "application/json"}
+    headers = {
+        "Content-Type": "application/json",
+        "X-App-Reference": "devplace-correction-v-1-0-0",
+    }
     if api_key:
         headers["Authorization"] = f"Bearer {api_key}"
     try:
-        with stealth.stealth_sync_client(timeout=timeout) as client:
-            response = client.post(INTERNAL_GATEWAY_URL, json=payload, headers=headers)
-            response.raise_for_status()
-            usage = _usage_from_headers(response.headers)
-            content = (
-                response.json()
-                .get("choices", [{}])[0]
-                .get("message", {})
-                .get("content", "")
-                .strip()
-            )
+        response = _client().post(
+            INTERNAL_GATEWAY_URL, json=payload, headers=headers, timeout=timeout
+        )
+        response.raise_for_status()
+        usage = _usage_from_headers(response.headers)
+        content = (
+            response.json()
+            .get("choices", [{}])[0]
+            .get("message", {})
+            .get("content", "")
+            .strip()
+        )
     except (httpx.HTTPError, ValueError, KeyError, IndexError) as exc:
         logger.warning("AI gateway completion failed, keeping original: %s", exc)
         return text, None
@@ -138,7 +160,7 @@ def schedule_pending(fn, request: object, *args) -> bool:
         loop = asyncio.get_running_loop()
     except RuntimeError:
         return False
-    future = loop.run_in_executor(None, fn, *args)
+    future = loop.run_in_executor(AI_APPLY_EXECUTOR, fn, *args)
     scope.setdefault(PENDING_SCOPE_KEY, []).append(future)
     return True
 
@@ -182,6 +204,6 @@ def _run_correction(
         if table == "users" and user_uid:
             from devplacepy.utils import clear_user_cache
 
-            clear_user_cache(user_uid)
+            clear_user_cache(user_uid, propagate=False)
     if totals["calls"] and user_uid:
         add_correction_usage(user_uid, totals)

devplacepy/services/dbapi/nl2sql.py

@@ -92,7 +92,11 @@ async def _complete(messages: list[dict], api_key: str, model: str) -> str:
         "max_tokens": MAX_TOKENS,
         "temperature": 0.0,
     }
-    headers = {"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"}
+    headers = {
+        "Authorization": f"Bearer {api_key}",
+        "Content-Type": "application/json",
+        "X-App-Reference": "devplace-dbapi-v-1-0-0",
+    }
     async with stealth.stealth_async_client(timeout=GATEWAY_TIMEOUT_SECONDS) as client:
         response = await client.post(INTERNAL_GATEWAY_URL, json=payload, headers=headers)
     if response.status_code >= 400:

devplacepy/services/deepsearch/chat.py

@@ -82,7 +82,7 @@ class DeepsearchChat:
                     stored_dim,
                 )
                 return []
-        return self.store.hybrid_search(question, query_vector, top_k=CHAT_TOP_K)
+        return await self.store.hybrid_search(question, query_vector, top_k=CHAT_TOP_K)
 
     async def answer(self, question: str, history: list[dict] | None = None) -> ChatAnswer:
         chunks = await self.retrieve(question)

devplacepy/services/deepsearch/embeddings.py

@@ -9,8 +9,8 @@ import re
 import time
 from dataclasses import dataclass, field
 
-from devplacepy import stealth
 from devplacepy.config import INTERNAL_EMBED_MODEL, INTERNAL_EMBED_URL
+from devplacepy.services.deepsearch.llm import gateway_client
 
 logger = logging.getLogger(__name__)
 
@@ -112,13 +112,15 @@ async def embed_texts(
     headers = {
         "Authorization": f"Bearer {api_key}",
         "Content-Type": "application/json",
+        "X-App-Reference": "devplace-deepsearch-v-1-0-0",
     }
     payload = {"model": INTERNAL_EMBED_MODEL, "input": pending_text}
     start = time.monotonic()
     backend = "gateway"
     try:
-        async with stealth.stealth_async_client(timeout=EMBED_TIMEOUT_SECONDS) as client:
-            response = await client.post(gateway_url, json=payload, headers=headers)
+        response = await gateway_client().post(
+            gateway_url, json=payload, headers=headers, timeout=EMBED_TIMEOUT_SECONDS
+        )
         if response.status_code >= 400:
             raise RuntimeError(f"embed gateway returned {response.status_code}")
         data = response.json()

devplacepy/services/deepsearch/llm.py

@@ -5,6 +5,8 @@ from __future__ import annotations
 import logging
 import time
 
+import httpx
+
 from devplacepy import stealth
 from devplacepy.config import INTERNAL_GATEWAY_URL, INTERNAL_MODEL
 
@@ -13,6 +15,15 @@ logger = logging.getLogger(__name__)
 CHAT_TIMEOUT_SECONDS = 120.0
 DEFAULT_MAX_TOKENS = 1200
 
+_gateway_client: httpx.AsyncClient | None = None
+
+
+def gateway_client() -> httpx.AsyncClient:
+    global _gateway_client
+    if _gateway_client is None or _gateway_client.is_closed:
+        _gateway_client = stealth.stealth_async_client(timeout=CHAT_TIMEOUT_SECONDS)
+    return _gateway_client
+
 
 async def request_completion(
     messages: list[dict],
@@ -33,10 +44,12 @@ async def request_completion(
     headers = {
         "Authorization": f"Bearer {api_key}",
         "Content-Type": "application/json",
+        "X-App-Reference": "devplace-deepsearch-v-1-0-0",
     }
     start: float = time.monotonic()
-    async with stealth.stealth_async_client(timeout=timeout) as client:
-        response = await client.post(gateway_url, json=payload, headers=headers)
+    response = await gateway_client().post(
+        gateway_url, json=payload, headers=headers, timeout=timeout
+    )
     elapsed_ms: int = int((time.monotonic() - start) * 1000)
     if response.status_code >= 400:
         raise RuntimeError(f"chat gateway returned {response.status_code}")

devplacepy/services/deepsearch/store.py

@@ -2,6 +2,7 @@
 
 from __future__ import annotations
 
+import asyncio
 import logging
 import math
 import re
@@ -19,6 +20,7 @@ BM25_B = 0.75
 RRF_K = 60.0
 DEFAULT_TOP_K = 8
 CANDIDATE_MULTIPLIER = 4
+CHROMADB_TIMEOUT = 30.0
 
 
 @dataclass
@@ -45,6 +47,20 @@ class VectorStore:
         self._collection = None
         self._dims: int | None = None
 
+    async def _run_sync(self, func, *args, timeout: float = CHROMADB_TIMEOUT):
+        """Run a synchronous ChromaDB call in a thread executor with a timeout."""
+        try:
+            return await asyncio.wait_for(
+                asyncio.to_thread(func, *args), timeout=timeout
+            )
+        except asyncio.TimeoutError:
+            logger.error(
+                "deepsearch ChromaDB operation timed out after %.1fs on collection %s",
+                timeout,
+                self.collection_name,
+            )
+            raise
+
     def _ensure(self):
         if self._collection is not None:
             return self._collection
@@ -71,7 +87,7 @@ class VectorStore:
             return self._dims
         return self._dims
 
-    def add(self, chunks: list[Chunk], vectors: list[list[float]]) -> None:
+    def _add_sync(self, chunks: list[Chunk], vectors: list[list[float]]) -> None:
         if not chunks:
             return
         keep_chunks: list[Chunk] = []
@@ -111,9 +127,17 @@ class VectorStore:
             ],
         )
 
-    def all_chunks(self) -> list[Chunk]:
+    async def add(self, chunks: list[Chunk], vectors: list[list[float]]) -> None:
+        if not chunks:
+            return
+        await self._run_sync(self._add_sync, chunks, vectors)
+
+    def _all_chunks_sync(self, limit: int = 0) -> list[Chunk]:
         collection = self._ensure()
-        data = collection.get(include=["documents", "metadatas"])
+        kwargs: dict = {"include": ["documents", "metadatas"]}
+        if limit > 0:
+            kwargs["limit"] = limit
+        data = collection.get(**kwargs)
         chunks: list[Chunk] = []
         ids = data.get("ids") or []
         documents = data.get("documents") or []
@@ -134,13 +158,17 @@ class VectorStore:
             )
         return chunks
 
-    def count(self) -> int:
+    async def all_chunks(self, limit: int = 1000) -> list[Chunk]:
+        return await self._run_sync(self._all_chunks_sync, limit)
+
+    async def count(self) -> int:
         try:
-            return self._ensure().count()
+            collection = self._ensure()
+            return await self._run_sync(collection.count)
         except Exception:
             return 0
 
-    def vector_search(
+    def _vector_search_sync(
         self, query_vector: list[float], top_k: int, where: dict | None = None
     ) -> list[Chunk]:
         collection = self._ensure()
@@ -173,6 +201,13 @@ class VectorStore:
             )
         return chunks
 
+    async def vector_search(
+        self, query_vector: list[float], top_k: int, where: dict | None = None
+    ) -> list[Chunk]:
+        return await self._run_sync(
+            self._vector_search_sync, query_vector, top_k, where
+        )
+
     def keyword_scores(self, query: str, chunks: list[Chunk]) -> dict[str, float]:
         terms = _tokenize(query)
         if not terms or not chunks:
@@ -205,14 +240,14 @@ class VectorStore:
             scores[chunk.uid] = score
         return scores
 
-    def hybrid_search(
+    async def hybrid_search(
         self,
         query: str,
         query_vector: list[float],
         top_k: int = DEFAULT_TOP_K,
         where: dict | None = None,
     ) -> list[Chunk]:
-        candidates = self.vector_search(
+        candidates = await self.vector_search(
             query_vector, top_k * CANDIDATE_MULTIPLIER, where
         )
         if not candidates:
@@ -234,10 +269,11 @@ class VectorStore:
         candidates.sort(key=lambda chunk: chunk.score, reverse=True)
         return candidates[:top_k]
 
-    def coverage_analytics(self) -> dict:
-        chunks = self.all_chunks()
+    async def coverage_analytics(self, sample_limit: int = 5000) -> dict:
+        chunks = await self.all_chunks(limit=sample_limit)
         if not chunks:
-            return {"chunks": 0, "domains": 0, "sources": 0, "avg_chunk_chars": 0}
+            total = await self.count()
+            return {"chunks": total, "domains": 0, "sources": 0, "avg_chunk_chars": 0}
         domains = {
             urlparse(chunk.metadata.get("url", "")).netloc for chunk in chunks
         }

devplacepy/services/devii/CLAUDE.md

@@ -35,6 +35,12 @@ Conversation history persists to `devii_conversations` (rehydrated on reconnect,
 
 **Per-owner self-learning memory is privacy-critical.** The `LessonStore` (reflect/recall) is **owner-scoped, never shared**: `LessonStore(db, owner_kind, owner_id)` filters every read/write by owner. The hub builds one per session over the same `owned_db` as the task store - the main `db` (table `devii_lessons`) for signed-in users (persistent, isolated, survives restarts) and a fresh `memory_db()` for guests (ephemeral, scoped to that web session). `forget_lessons` (agentic tool -> `LessonStore.clear()`/`delete()`) lets the user purge them; the system prompt forbids storing credentials/secrets in lessons. **Regression to avoid: do NOT share one `LessonStore` across sessions** - that leaked one user's reflected lessons (including credentials) into every other user's recall.
 
+**Lesson retention and deduplication.** Every `add()` deduplicates against existing active lessons via Jaccard similarity (threshold 0.70): a near-duplicate bumps the original's `hits` counter and refreshes `created_at` instead of inserting a new row. A per-owner cap (`devii_lessons_max_per_owner`, default 500, configurable on `/admin/services` and `site_settings`) is enforced on every insert: when exceeded, the oldest lessons are soft-deleted (`deleted_by="retention"`). Age-based pruning runs on `DeviiService.run_once()` (every 60s on the lock-owner worker): lessons older than `devii_lessons_max_age_days` (default 90, configurable) are soft-deleted across all owners. Both settings persist in `site_settings` and are seeded in `init_db`. The `devii_lessons` table is in `SOFT_DELETE_TABLES` for admin Trash restore/purge.
+
+**Quality signals.** Each lesson has a `rating` column (integer, default 0). The `lesson_rate(uid, value)` agentic tool (value 1 for useful, -1 for unhelpful) lets the agent self-rate lessons. In `_rebuild()`, lessons with `rating <= -3` are excluded from the BM25 index and therefore never returned by `recall()`. The `lesson_count()` tool reports active lesson count.
+
+**CLI:** `devplace devii lessons count` reports active/soft-deleted totals; `devplace devii lessons prune --all-owners|--username USER` soft-deletes old lessons; `devplace devii lessons clear --force` hard-deletes all rows. Rate-limiting guard: `run_once` swallows all exceptions so a bad schema never stops housekeeping.
+
 ## Reminders and scheduled tasks (persistent across reboot)
 
 `create_task` queues a self-contained prompt that a fresh agent runs later (`services/devii/tasks/`): `kind=once` (`delay_seconds` for relative, `run_at` UTC for absolute), `interval` (`every_seconds`), or `cron`. The per-session `Scheduler` ticks every 1s and executes due rows through the session's executor, so the result is broadcast to any connected tab and buffered (`type:"task"` frame) when the terminal is closed.
@@ -142,9 +148,57 @@ Devii can partially configure its OWN system message: every system prompt ends w
 
 - **Store** (`behavior/store.py`). `BehaviorStore(db, owner_kind, owner_id)` over `devii_behavior` (one upserted row per owner keyed on `owner_kind`/`owner_id`; `text()` reads, `set()` upserts; index `idx_devii_behavior_owner`). Persistent for users, `memory_db()` for guests (built in `hub.get_or_create` from the shared `owned_db`, like the other owner stores).
 - **Controller** (`behavior/controller.py`). `BehaviorController(store)`, `dispatch("update_behavior", args)` -> `store.set(behavior)`. Built in `DeviiSession` and passed to `Dispatcher(behavior=...)`; the dispatcher routes `handler="behavior"` to it and degrades gracefully ("not available in this context") when unwired (e.g. the standalone CLI, same as `virtual_tools`/`avatar`).
-- **Injection and refresh** (`session.py`). `_compose_system_prompt()` = base prompt (`_system_prompt_for(is_admin)`) + `\n\n` + `BEHAVIOR_HEADER` (+ `\n` + body when non-empty; just the header when empty). Used to seed the `Agent` and the scheduler executor worker, and `_refresh_system_prompt()` rewrites `agent._messages[0]["content"]` **at the top of every `_run_turn`** (next to `_refresh_tools`), so a mid-conversation `update_behavior` takes effect on the following turn and the live system message is never overridden by the stale base.
+- **Injection and refresh** (`session.py`). `_compose_system_prompt()` = base prompt (`_system_prompt_for(is_admin)`) + CA-IWP fragment + live `CHANNEL` block + `\n\n` + `BEHAVIOR_HEADER` (+ `\n` + body when non-empty; just the header when empty). Used to seed the `Agent` and the scheduler executor worker, and `_refresh_system_prompt()` rewrites `agent._messages[0]["content"]` **at the top of every `_run_turn`** (next to `_refresh_tools`), so a mid-conversation `update_behavior` takes effect on the following turn and the live system message is never overridden by the stale base.
 - Registered via `BEHAVIOR_ACTIONS` (`registry.py`); the system-prompt **SELF-CONFIGURED BEHAVIOR (TRUTH RULES)** section in `agent.py` steers when/how to call it.
 
+## Channel-Aware Interactive Widget Protocol (CA-IWP / Speak With Buttons)
+
+Devii can ask the user for decisions through a gated tool surface instead of inventing HTML or channel-specific prose. Spec lives as the CA-IWP document; implementation is Devii-only under `services/devii/interaction/`.
+
+### Layers
+
+| Piece | Role |
+|-------|------|
+| `interaction/capabilities.py` | Maps session channel (`main`/`docs` -> `site-chat`, `telegram`, `cli`) to capability flags + limits; builds the compact `CHANNEL` / `CAPS` / `LIMITS` / `TOOLS` / `INTERACTION` fragment injected every turn. |
+| `interaction/schema.py` | Pydantic validation for `ui_prompt` args (widget catalog: confirm, choice, choice_multi, text, number, date, select, `//`, group). Untrusted model input is sanitized and capped. |
+| `interaction/broker.py` | Validates, assigns `interaction_id`, single-flight open interactions, routes to site / telegram / plain adapters, returns structured `{status, values, meta}`. |
+| `interaction/controller.py` | Dispatches `ui_prompt` / `ui_cancel` / `ui_notify` (`handler="interaction"`). |
+| `interaction/actions.py` | Catalog entries registered in `registry.py` as `INTERACTION_ACTIONS`. |
+| `interaction/markdown.py` + `parse.py` | Dual-surface markdown projection and plain/CLI reply parsers. |
+
+### Tool gating and preferences (admin default + user override)
+
+`DeviiSession._builtin_tools()` filters UI tools through `channel_context().tools`. Docs channel stays search-only (no UI tools). Open interaction (single-flight) exposes only `ui_cancel` plus preference tools.
+
+**Admin default:** Devii service ConfigField `devii_interactions_default` (bool, default on) on `/admin/services`. Guests always use this default.
+
+**User override:** column `users.interactions_enabled` (`-1` = inherit default, `0` = off, `1` = on). New signups insert `-1`. Resolution: `interaction/prefs.py` `effective_for(owner_kind, owner_id)`.
+
+**Surfaces (same fan-out as AI correction):**
+- Devii tools `interactions_get` / `interactions_set` (`requires_auth=True`; set accepts `enabled` or `reset=true` to inherit again). Always offered to signed-in users even when widgets are off, so they can re-enable.
+- HTTP `POST /profile/{username}/interactions` (owner-or-admin, form `InteractionsForm`), profile card + `InteractionsPref.js`, `docs_api` endpoint, audit `profile.interactions`, profile JSON keys on `ProfileOut`.
+- When effective is off, `ui_prompt` / `ui_notify` are absent from the tool list and the controller refuses them; the model must use degraded markdown menus.
+
+### Site-chat path
+
+`ui_prompt` blocks like client tools: session `_interaction_wait` emits `{type:"interaction", id, args}` on the WebSocket; `devii-terminal.js` mounts an `<ai-interaction>` tree via `createElement` (never model HTML), lazy-loads widget modules through `AiAutoload` (`static/js/autoload/AiAutoload.js`, allowlisted tag -> module map only), and replies with `{type:"interaction_result"}`. Router resolves via `session.resolve_interaction`. Custom elements live under `static/js/components/Ai*.js` with per-component CSS under `static/css/components/ai-*.css`. Light DOM only; `Application` boots the shell (`AiInteraction`, `AiStatus`, `AiActions`, `AiHelp`, `AiOption`) and starts the autoloader.
+
+### Telegram path
+
+`TelegramConnection` handles `type:"interaction"`: sends dual-surface plain text plus inline keyboard for confirm / single choice (`callback_data` short tokens). `TelegramBridge` registers a pending future **before** the chat lock so the next message or `callback_query` can resolve mid-turn. Worker `allowed_updates` includes `callback_query`; service routes `type:"callback"` to the bridge.
+
+### Plain / CLI path
+
+Broker uses `present_plain` or emits a plain menu and waits; `parse_plain_reply` accepts y/n, indices, value tokens, cancel.
+
+### System prompt
+
+Every non-docs turn appends `CA_IWP_SYSTEM_FRAGMENT` plus the live channel fragment from `_compose_system_prompt()`. Model rules: prefer `ui_prompt` when gated on; never invent HTML/CE tags; branch on `status` (submitted / cancelled / timeout / superseded / error).
+
+### Tests
+
+Unit coverage under `tests/unit/services/devii/interaction/` (capabilities, schema, markdown, parse, broker, controller) plus session tool-gating assertions and telegram callback emission. Mirror this layout for any extension.
+
 ## Related Devii tool wrappers (customization, container)
 
 Two more Devii-side controllers live under `services/devii/` but their full mechanism is documented in their owning subsystem's file, not here:

devplacepy/services/devii/actions/ai_correction_actions.py

@@ -61,7 +61,7 @@ AI_CORRECTION_ACTIONS: tuple[Action, ...] = (
             ),
             arg(
                 "prompt",
-                "The correction instruction (max 2000 chars). Omit to keep the current one.",
+                "The correction instruction (max 20000 chars). Omit to keep the current one.",
             ),
         ),
     ),

devplacepy/services/devii/actions/ai_modifier_actions.py

@@ -64,7 +64,7 @@ AI_MODIFIER_ACTIONS: tuple[Action, ...] = (
             ),
             arg(
                 "prompt",
-                "The modifier instruction (max 2000 chars). Omit to keep the current one.",
+                "The modifier instruction (max 20000 chars). Omit to keep the current one.",
             ),
         ),
     ),

devplacepy/services/devii/actions/catalog/admin.py

@@ -31,6 +31,25 @@ ADMIN_ACTIONS: tuple[Action, ...] = (
         params=(query("top_n", "How many top authors to include (1-50)."),),
         requires_admin=True,
     ),
+    Action(
+        name="admin_statistics",
+        method="GET",
+        path="/admin/statistics/data",
+        summary="Platform statistics tab data with trends (admin only)",
+        description=(
+            "Returns JSON for one statistics tab: KPI cards with period-over-period deltas, "
+            "time-series points for line charts, breakdown tables, and highlight metrics. "
+            "Tabs: overview, visitors, members, content, engagement, social, ai, devii, "
+            "services, containers, game, awards, moderation, tools, storage."
+        ),
+        params=(
+            query("tab", "Tab key (default overview)."),
+            query("hours", "Lookback window in hours (24, 168, 720, 2160, or 0 for all time)."),
+            query("compare", "Include previous-period comparison (1 or 0, default 1)."),
+            query("top_n", "Rows in breakdown tables (default 10)."),
+        ),
+        requires_admin=True,
+    ),
     Action(
         name="ai_usage",
         method="GET",
@@ -305,6 +324,18 @@ ADMIN_ACTIONS: tuple[Action, ...] = (
         params=(path("uid", "Attachment uid to purge."), confirm()),
         requires_admin=True,
     ),
+    Action(
+        name="admin_revoke_award",
+        method="POST",
+        path="/admin/awards/{uid}/revoke",
+        summary="Revoke a published award (admin only)",
+        description=(
+            "Soft-deletes the award and linked attachments, then recomputes receiver stats. "
+            "Confirmation is required in the UI; Devii should confirm before calling."
+        ),
+        params=(path("uid", "Award uid to revoke."), confirm()),
+        requires_admin=True,
+    ),
     Action(
         name="admin_reset_guest_ai_quota",
         method="POST",

devplacepy/services/devii/actions/catalog/comments.py

@@ -28,7 +28,7 @@ COMMENTS_ACTIONS: tuple[Action, ...] = (
         summary="Edit the body of one of your own comments",
         params=(
             path("comment_uid", "Uid of the comment."),
-            body("content", "New comment body, 3-1000 characters.", required=True),
+            body("content", "New comment body, 3-125000 characters.", required=True),
         ),
     ),
     Action(

devplacepy/services/devii/actions/catalog/gateway.py

@@ -27,7 +27,7 @@ GATEWAY_ACTIONS: tuple[Action, ...] = (
         summary="Create or update a gateway provider (admin only)",
         description=(
             "Adds or updates a named upstream provider. base_url is the OpenAI-compatible "
-            "chat-completions endpoint; the embeddings endpoint is derived from it."
+            "chat-completions endpoint; the embeddings and images endpoints are derived from it."
         ),
         handler="http",
         requires_admin=True,
@@ -60,7 +60,8 @@ GATEWAY_ACTIONS: tuple[Action, ...] = (
         summary="List OpenAI gateway model routes (admin only)",
         description=(
             "Returns JSON: every source-model route with its provider, target model, kind "
-            "(chat/embed), optional vision model, context window, and per-model pricing economy."
+            "(chat/embed/image), optional vision model, context window, and per-model pricing economy "
+            "(including any tiered/off-peak pricing configured on it)."
         ),
         handler="http",
         requires_admin=True,
@@ -73,9 +74,14 @@ GATEWAY_ACTIONS: tuple[Action, ...] = (
         summary="Create or update a gateway model route (admin only)",
         description=(
             "Maps a requested source_model onto a provider + target_model, each with its own "
-            "pricing. kind is 'chat' or 'embed'. A vision_model adds image-to-text augmentation "
-            "for chat routes. Prices are USD per 1,000,000 tokens; chat uses cache-hit/cache-miss/"
-            "output, embeddings use input, vision uses input/output."
+            "pricing. kind is 'chat', 'embed', or 'image'. A vision_model adds image-to-text augmentation "
+            "for chat routes. Prices are USD per 1,000,000 tokens for chat/embed/vision; image routes "
+            "use price_input_per_m as a flat USD per generated image. Optionally, a route can also "
+            "charge a different (tier-2) rate once the request's input tokens exceed "
+            "context_tier_threshold_tokens, and/or apply a percentage discount during a fixed "
+            "UTC off-peak window - leave the tier2/off-peak fields unset to keep the flat rates "
+            "above at all times. off_peak_start_minute and off_peak_end_minute must be set "
+            "together (both or neither) or the call is rejected."
         ),
         handler="http",
         requires_admin=True,
@@ -83,14 +89,22 @@ GATEWAY_ACTIONS: tuple[Action, ...] = (
             body("source_model", "Model name clients request.", required=True),
             body("provider", "Provider name, or blank for the default upstream."),
             body("target_model", "Model name sent upstream.", required=True),
-            body("kind", "Route kind: 'chat' or 'embed'."),
+            body("kind", "Route kind: 'chat', 'embed', or 'image'."),
             body("vision_provider", "Provider for image description, or blank for the route provider."),
             body("vision_model", "Vision model name (blank disables the merge)."),
             Param(name="context_window", location="body", description="Max context tokens (0 = unknown).", required=False, type="integer"),
             Param(name="price_cache_hit_per_m", location="body", description="USD per 1M cache-hit input tokens.", required=False, type="number"),
             Param(name="price_cache_miss_per_m", location="body", description="USD per 1M cache-miss input tokens.", required=False, type="number"),
             Param(name="price_output_per_m", location="body", description="USD per 1M output tokens.", required=False, type="number"),
-            Param(name="price_input_per_m", location="body", description="USD per 1M input tokens (embed/vision).", required=False, type="number"),
+            Param(name="price_input_per_m", location="body", description="USD per 1M input tokens (embed/vision), or flat USD per image for image routes.", required=False, type="number"),
+            Param(name="context_tier_threshold_tokens", location="body", description="Input tokens above which tier-2 rates apply (0 disables tiering).", required=False, type="integer"),
+            Param(name="price_cache_hit_per_m_tier2", location="body", description="Tier-2 USD per 1M cache-hit input tokens (unset = keep tier-1 rate above threshold).", required=False, type="number"),
+            Param(name="price_cache_miss_per_m_tier2", location="body", description="Tier-2 USD per 1M cache-miss input tokens (unset = keep tier-1 rate above threshold).", required=False, type="number"),
+            Param(name="price_output_per_m_tier2", location="body", description="Tier-2 USD per 1M output tokens (unset = keep tier-1 rate above threshold).", required=False, type="number"),
+            Param(name="price_input_per_m_tier2", location="body", description="Tier-2 USD per 1M input tokens, embed/vision (unset = keep tier-1 rate above threshold).", required=False, type="number"),
+            Param(name="off_peak_start_minute", location="body", description="Off-peak window start, UTC minutes since midnight (0-1439). Must be set together with off_peak_end_minute.", required=False, type="integer"),
+            Param(name="off_peak_end_minute", location="body", description="Off-peak window end, UTC minutes since midnight (0-1439). A value less than the start wraps past midnight.", required=False, type="integer"),
+            Param(name="off_peak_discount_pct", location="body", description="Percentage discount (0-100) applied to the active tier's rates during the off-peak window.", required=False, type="number"),
             Param(name="is_active", location="body", description="Whether the route is active ('1' or '0').", required=False, type="boolean"),
         ),
     ),

devplacepy/services/devii/actions/catalog/profile.py

@@ -61,6 +61,21 @@ PROFILE_ACTIONS: tuple[Action, ...] = (
             ),
         ),
     ),
+    Action(
+        name="give_award",
+        method="POST",
+        path="/profile/{username}/award",
+        summary="Give another member an award on their profile",
+        description=(
+            "Creates a pending award row and enqueues image generation billed to the giver's "
+            "API key. The receiver is notified when generation completes. Cannot award yourself; "
+            "blocked pairs are rejected; cooldowns apply."
+        ),
+        params=(
+            path("username", "Receiver username."),
+            body("description", "Short award message (1-125 characters).", required=True),
+        ),
+    ),
     Action(
         name="regenerate_avatar",
         method="POST",

devplacepy/services/devii/actions/dispatcher.py

@@ -116,6 +116,7 @@ _DEVII_MECHANIC_EVENTS = {
     "email_mark": "email.message.flag",
     "email_set_flags": "email.message.flag",
     "telegram_send": "telegram.send",
+    "interactions_set": "profile.interactions",
 }
 
 _DEVII_CONTAINER_EVENTS = {
@@ -268,6 +269,7 @@ class Dispatcher:
         owner_id: str = "",
         virtual_tools: Any = None,
         behavior: Any = None,
+        interaction: Any = None,
     ) -> None:
         self._actions = catalog.by_name()
         self._client = client
@@ -308,6 +310,7 @@ class Dispatcher:
         self._telegram = TelegramSendController(owner_kind, owner_id)
         self._virtual_tools = virtual_tools
         self._behavior = behavior
+        self._interaction = interaction
         self._read_files: set[tuple[str, str]] = set()
 
     @staticmethod
@@ -500,6 +503,15 @@ class Dispatcher:
                 )
             return await self._behavior.dispatch(action.name, arguments)
 
+        if action.handler == "interaction":
+            if self._interaction is None:
+                return error_result(
+                    ToolInputError(
+                        "Interactive prompts are not available in this context."
+                    )
+                )
+            return await self._interaction.dispatch(action.name, arguments)
+
         if action.handler == "virtual_tool":
             if self._virtual_tools is None:
                 return error_result(

devplacepy/services/devii/actions/notification_actions.py

@@ -18,7 +18,7 @@ def arg(
 
 
 TYPES = (
-    "One of: comment, reply, mention, vote, follow, message, badge, level, issue."
+    "One of: comment, reply, mention, vote, follow, message, badge, level, issue, award."
 )
 
 NOTIFICATION_ACTIONS: tuple[Action, ...] = (

devplacepy/services/devii/actions/spec.py

@@ -3,6 +3,7 @@
 from __future__ import annotations
 
 from dataclasses import dataclass, field
+from functools import lru_cache
 from typing import Any, Literal
 
 ParamLocation = Literal["path", "query", "body", "file"]
@@ -49,6 +50,7 @@ class Action:
         "virtual_tool",
         "email",
         "telegram",
+        "interaction",
     ] = "http"
     freeform_body: bool = False
     ajax: bool = False
@@ -117,16 +119,27 @@ class Catalog:
     def tool_schemas(self) -> list[dict[str, Any]]:
         return [action.tool_schema() for action in self.actions]
 
+    @lru_cache(maxsize=16)
+    def _schemas_for(
+        self,
+        authenticated: bool,
+        is_admin: bool,
+        is_primary_admin: bool,
+    ) -> tuple[dict[str, Any], ...]:
+        return tuple(
+            action.tool_schema()
+            for action in self.actions
+            if (authenticated or not action.requires_auth)
+            and (is_admin or not action.requires_admin)
+            and (is_primary_admin or not action.requires_primary_admin)
+        )
+
     def tool_schemas_for(
         self,
         authenticated: bool,
         is_admin: bool = False,
         is_primary_admin: bool = False,
     ) -> list[dict[str, Any]]:
-        return [
-            action.tool_schema()
-            for action in self.actions
-            if (authenticated or not action.requires_auth)
-            and (is_admin or not action.requires_admin)
-            and (is_primary_admin or not action.requires_primary_admin)
-        ]
+        return list(
+            self._schemas_for(bool(authenticated), bool(is_admin), bool(is_primary_admin))
+        )

devplacepy/services/devii/agentic/actions.py

@@ -150,6 +150,34 @@ AGENTIC_ACTIONS: tuple[Action, ...] = (
             ),
         ),
     ),
+    Action(
+        name="lesson_rate",
+        method="LOCAL",
+        path="",
+        summary="Vote on a lesson's quality to help the system keep the best ones and drop bad ones",
+        description=(
+            "Rate a lesson recorded by reflect(). A positive vote (1) marks it as useful; "
+            "a negative vote (-1) marks it as unhelpful. Lessons with a cumulative rating of "
+            "-3 or lower are excluded from recall() results. Use this when you notice a lesson "
+            "was particularly helpful or misleading."
+        ),
+        handler="agentic",
+        requires_auth=False,
+        params=(
+            arg("uid", "The uid of the lesson to rate, from a recall() result.", required=True),
+            arg("value", "1 for useful, -1 for unhelpful.", required=True, kind="integer"),
+        ),
+    ),
+    Action(
+        name="lesson_count",
+        method="LOCAL",
+        path="",
+        summary="Report how many learned lessons are stored for this session or account",
+        description="Returns the number of active lessons with an optional per-tag breakdown.",
+        handler="agentic",
+        requires_auth=False,
+        params=(),
+    ),
     Action(
         name="eval",
         method="LOCAL",

devplacepy/services/devii/agentic/compaction.py

@@ -6,13 +6,17 @@ import json
 import logging
 from typing import Any
 
+from ..text import normalize_newlines
+
 logger = logging.getLogger("devii.agentic.compaction")
 
 SUMMARY_INPUT_CAP = 600_000
 SUMMARY_PROMPT = (
     "Summarize the following assistant conversation segment as a concise factual log of "
     "actions taken, tools called, entities created or changed, conclusions reached, and "
-    "outstanding work. Keep identifiers, slugs, uids, and decisions verbatim. Maximum 800 words.\n\n"
+    "outstanding work. Keep identifiers, slugs, uids, and decisions verbatim. "
+    "Write plain markdown with real line breaks (never the two-character sequence \\n). "
+    "Use headings and bullet lists where helpful. Maximum 800 words.\n\n"
     "---\n\n"
 )
 
@@ -32,6 +36,37 @@ def find_compaction_split(messages: list[dict[str, Any]], keep_tail: int) -> int
     return 1
 
 
+def _segment_plain(messages: list[dict[str, Any]]) -> str:
+    parts: list[str] = []
+    for message in messages:
+        role = str(message.get("role") or "unknown")
+        content = message.get("content")
+        if isinstance(content, str) and content.strip():
+            parts.append(f"{role}:\n{normalize_newlines(content)}")
+            continue
+        if isinstance(content, list):
+            chunks: list[str] = []
+            for part in content:
+                if isinstance(part, dict) and part.get("type") == "text":
+                    chunks.append(str(part.get("text") or ""))
+                elif isinstance(part, str):
+                    chunks.append(part)
+            text = normalize_newlines("\n".join(c for c in chunks if c))
+            if text.strip():
+                parts.append(f"{role}:\n{text}")
+            continue
+        tool_calls = message.get("tool_calls")
+        if tool_calls:
+            names = []
+            for call in tool_calls:
+                fn = (call or {}).get("function") or {}
+                name = fn.get("name") or "tool"
+                names.append(str(name))
+            if names:
+                parts.append(f"{role}: called {', '.join(names)}")
+    return "\n\n".join(parts)
+
+
 async def compact_messages(
     llm: Any, messages: list[dict[str, Any]], keep_tail: int
 ) -> list[dict[str, Any]]:
@@ -46,16 +81,24 @@ async def compact_messages(
     if not middle:
         return messages
 
-    segment = json.dumps(middle, default=str)[:SUMMARY_INPUT_CAP]
+    segment = _segment_plain(middle)[:SUMMARY_INPUT_CAP]
+    if not segment.strip():
+        segment = json.dumps(middle, default=str)[:SUMMARY_INPUT_CAP]
     try:
         summary = await llm.summarize(SUMMARY_PROMPT + segment)
     except Exception:  # noqa: BLE001 - compaction must never break the loop
         logger.exception("Compaction summary failed; keeping full context")
         return messages
 
+    summary = normalize_newlines(summary or "").strip()
+    if not summary:
+        return messages
     logger.info("Compacted %d messages into a summary", len(middle))
     return [
         system_message,
-        {"role": "assistant", "content": f"[compacted earlier turns]\n\n{summary}"},
+        {
+            "role": "assistant",
+            "content": f"[compacted earlier turns]\n\n{summary}",
+        },
         *tail,
     ]

devplacepy/services/devii/agentic/controller.py

@@ -63,6 +63,8 @@ class AgenticController:
             "reflect": self._reflect,
             "recall": self._recall,
             "forget_lessons": self._forget,
+            "lesson_rate": self._lesson_rate,
+            "lesson_count": self._lesson_count,
             "verify": self._verify,
             "delegate": self._delegate,
             "eval": self._eval,
@@ -162,6 +164,31 @@ class AgenticController:
             ensure_ascii=False,
         )
 
+    async def _lesson_rate(self, arguments: dict[str, Any]) -> str:
+        uid = str(arguments.get("uid", "")).strip()
+        if not uid:
+            raise ToolInputError("lesson_rate requires a lesson uid.")
+        raw_value = arguments.get("value")
+        if raw_value is None:
+            raise ToolInputError("lesson_rate requires a value (1 or -1).")
+        try:
+            value = int(raw_value)
+        except (ValueError, TypeError):
+            raise ToolInputError("lesson_rate value must be an integer (1 or -1).") from None
+        if value not in (1, -1):
+            raise ToolInputError("lesson_rate value must be 1 (useful) or -1 (unhelpful).")
+        ok = self._lessons.rate(uid, value)
+        if not ok:
+            raise ToolInputError(f"No lesson found with uid '{uid}'.")
+        return json.dumps({"status": "success", "lesson_uid": uid, "rated": value}, ensure_ascii=False)
+
+    async def _lesson_count(self, arguments: dict[str, Any]) -> str:
+        count = self._lessons.count()
+        return json.dumps(
+            {"status": "success", "active_lessons": count},
+            ensure_ascii=False,
+        )
+
     async def _verify(self, arguments: dict[str, Any]) -> str:
         summary = str(arguments.get("summary", "")).strip()
         if not summary:

devplacepy/services/devii/agentic/lessons.py

@@ -7,6 +7,7 @@ import logging
 import math
 import re
 import uuid
+from datetime import timedelta
 from typing import Any
 
 from ..tasks.schedule import now_utc, to_iso
@@ -18,6 +19,11 @@ TOKEN_RE = re.compile(r"[A-Za-z_][A-Za-z0-9_]*|\d+")
 BM25_K1 = 1.5
 BM25_B = 0.75
 
+DEDUP_JACCARD_THRESHOLD = 0.70
+DEFAULT_MAX_PER_OWNER = 500
+DEFAULT_MAX_AGE_DAYS = 90
+LOW_QUALITY_THRESHOLD = -3
+
 
 def tokenize(text: str) -> list[str]:
     tokens = TOKEN_RE.findall((text or "").lower())
@@ -27,6 +33,33 @@ def tokenize(text: str) -> list[str]:
     return list(dict.fromkeys(tokens + extra))
 
 
+def _jaccard(tokens_a: set[str], tokens_b: set[str]) -> float:
+    if not tokens_a and not tokens_b:
+        return 0.0
+    if not tokens_a or not tokens_b:
+        return 0.0
+    return len(tokens_a & tokens_b) / len(tokens_a | tokens_b)
+
+
+def _read_retention_settings(db: Any) -> tuple[int, int]:
+    max_per = DEFAULT_MAX_PER_OWNER
+    max_age = DEFAULT_MAX_AGE_DAYS
+    if "site_settings" not in db.tables:
+        return max_per, max_age
+    for row in db["site_settings"].find(key={"in": ["devii_lessons_max_per_owner", "devii_lessons_max_age_days"]}):
+        if row["key"] == "devii_lessons_max_per_owner":
+            try:
+                max_per = int(row["value"])
+            except (ValueError, TypeError):
+                pass
+        elif row["key"] == "devii_lessons_max_age_days":
+            try:
+                max_age = int(row["value"])
+            except (ValueError, TypeError):
+                pass
+    return max_per, max_age
+
+
 class LessonStore:
     def __init__(self, db: Any, owner_kind: str, owner_id: str) -> None:
         self._db = db
@@ -39,9 +72,10 @@ class LessonStore:
         self._idf: dict[str, float] = {}
         self._avgdl = 0.0
         self._n = 0
+        self._ensure_columns()
         self._ensure_indexes()
 
-    def _ensure_indexes(self) -> None:
+    def _ensure_columns(self) -> None:
         if TABLE not in self._db.tables:
             return
         table = self._db[TABLE]
@@ -49,7 +83,15 @@ class LessonStore:
             table.create_column_by_example("deleted_at", "")
         if not table.has_column("deleted_by"):
             table.create_column_by_example("deleted_by", "")
+        if not table.has_column("rating"):
+            table.create_column_by_example("rating", 0)
+
+    def _ensure_indexes(self) -> None:
+        if TABLE not in self._db.tables:
+            return
+        table = self._db[TABLE]
         table.create_index(["owner_kind", "owner_id"])
+        table.create_index(["owner_kind", "owner_id", "created_at"])
 
     @property
     def _table(self) -> Any:
@@ -64,17 +106,99 @@ class LessonStore:
             return 0
         return self._table.count(deleted_at=None, **self._scope)
 
+    def _row_text(self, row: dict[str, Any]) -> str:
+        return " ".join(
+            str(row.get(field) or "")
+            for field in ("observation", "conclusion", "next_action", "tags")
+        )
+
+    def _find_similar(self, text: str, threshold: float = DEDUP_JACCARD_THRESHOLD) -> dict[str, Any] | None:
+        query_tokens = set(tokenize(text))
+        if not query_tokens:
+            return None
+        all_rows = self.all()
+        best_row: dict[str, Any] | None = None
+        best_score = 0.0
+        for row in all_rows:
+            row_tokens = set(tokenize(self._row_text(row)))
+            score = _jaccard(query_tokens, row_tokens)
+            if score > best_score and score >= threshold:
+                best_score = score
+                best_row = row
+        return best_row
+
+    def _enforce_cap(self, max_per_owner: int) -> int:
+        soft_deleted = 0
+        while True:
+            current = self.count()
+            if current <= max_per_owner:
+                break
+            excess = current - max_per_owner
+            rows = list(
+                self._table.find(
+                    deleted_at=None,
+                    order_by=["created_at"],
+                    _limit=excess,
+                    **self._scope,
+                )
+            )
+            if not rows:
+                break
+            now = to_iso(now_utc())
+            for row in rows:
+                self._table.update(
+                    {
+                        "id": row["id"],
+                        "deleted_at": now,
+                        "deleted_by": "retention",
+                    },
+                    ["id"],
+                )
+                soft_deleted += 1
+            self._dirty = True
+        if soft_deleted:
+            logger.info(
+                "Retention cap pruned %d lesson(s) for owner=%s/%s",
+                soft_deleted,
+                self._owner_kind,
+                self._owner_id,
+            )
+        return soft_deleted
+
     def add(
         self, observation: str, conclusion: str, next_action: str, tags: str = ""
     ) -> dict[str, Any]:
+        text = " ".join([observation, conclusion, next_action, tags])
+        similar = self._find_similar(text)
+        if similar and similar.get("id") is not None:
+            hits = (similar.get("hits") or 0) + 1
+            self._table.update(
+                {
+                    "id": similar["id"],
+                    "hits": hits,
+                    "created_at": to_iso(now_utc()),
+                },
+                ["id"],
+            )
+            self._dirty = True
+            logger.info(
+                "Lesson deduplicated owner=%s/%s hits=%d",
+                self._owner_kind,
+                self._owner_id,
+                hits,
+            )
+            return {**similar, "hits": hits, "deduplicated": True}
+
+        uid = uuid.uuid4().hex
         record = {
-            "uid": uuid.uuid4().hex,
+            "uid": uid,
             "observation": observation,
             "conclusion": conclusion,
             "next_action": next_action,
             "tags": tags,
             "created_at": to_iso(now_utc()),
             "hits": 0,
+            "rating": 0,
             "deleted_at": None,
             "deleted_by": None,
             **self._scope,
@@ -84,6 +208,8 @@ class LessonStore:
         logger.info(
             "Lesson stored owner=%s/%s tags=%s", self._owner_kind, self._owner_id, tags
         )
+        max_per, _ = _read_retention_settings(self._db)
+        self._enforce_cap(max_per)
         return record
 
     def all(self) -> list[dict[str, Any]]:
@@ -118,17 +244,108 @@ class LessonStore:
         )
         return n
 
+    def rate(self, uid: str, value: int) -> bool:
+        if TABLE not in self._db.tables:
+            return False
+        row = self._table.find_one(uid=uid, deleted_at=None, **self._scope)
+        if not row:
+            return False
+        current = row.get("rating") or 0
+        self._table.update(
+            {"id": row["id"], "rating": current + value},
+            ["id"],
+        )
+        self._dirty = True
+        logger.info(
+            "Lesson %s rated %+d (now %d) owner=%s/%s",
+            uid,
+            value,
+            current + value,
+            self._owner_kind,
+            self._owner_id,
+        )
+        return True
+
+    def prune(self, max_age_days: int | None = None) -> int:
+        if TABLE not in self._db.tables:
+            return 0
+        if max_age_days is None:
+            _, max_age_days = _read_retention_settings(self._db)
+        cutoff = now_utc() - timedelta(days=max_age_days)
+        cutoff_iso = to_iso(cutoff)
+        rows = list(
+            self._table.find(
+                deleted_at=None,
+                created_at={"<": cutoff_iso},
+                **self._scope,
+            )
+        )
+        if not rows:
+            return 0
+        now = to_iso(now_utc())
+        soft_deleted = 0
+        for row in rows:
+            self._table.update(
+                {
+                    "id": row["id"],
+                    "deleted_at": now,
+                    "deleted_by": "retention",
+                },
+                ["id"],
+            )
+            soft_deleted += 1
+        if soft_deleted:
+            self._dirty = True
+            logger.info(
+                "Pruned %d old lesson(s) for owner=%s/%s",
+                soft_deleted,
+                self._owner_kind,
+                self._owner_id,
+            )
+        return soft_deleted
+
+    def prune_all_owners(self, max_age_days: int | None = None) -> int:
+        if TABLE not in self._db.tables:
+            return 0
+        if max_age_days is None:
+            _, max_age_days = _read_retention_settings(self._db)
+        cutoff = now_utc() - timedelta(days=max_age_days)
+        cutoff_iso = to_iso(cutoff)
+        rows = list(
+            self._table.find(
+                deleted_at=None,
+                created_at={"<": cutoff_iso},
+            )
+        )
+        if not rows:
+            return 0
+        now = to_iso(now_utc())
+        soft_deleted = 0
+        for row in rows:
+            self._table.update(
+                {
+                    "id": row["id"],
+                    "deleted_at": now,
+                    "deleted_by": "retention",
+                },
+                ["id"],
+            )
+            soft_deleted += 1
+        if soft_deleted:
+            logger.info("Pruned %d old lesson(s) across all owners", soft_deleted)
+        return soft_deleted
+
     def _rebuild(self) -> None:
         rows = self.all()
         df: collections.Counter = collections.Counter()
         docs: list[dict[str, Any]] = []
         tf_list: list[collections.Counter] = []
-        dl: list[int] = []
+        dl_list: list[int] = []
         for row in rows:
-            text = " ".join(
-                str(row.get(field) or "")
-                for field in ("observation", "conclusion", "next_action", "tags")
-            )
+            rating = row.get("rating") or 0
+            if rating <= LOW_QUALITY_THRESHOLD:
+                continue
+            text = self._row_text(row)
             tokens = tokenize(text)
             if not tokens:
                 continue
@@ -137,12 +354,12 @@ class LessonStore:
                 df[term] += 1
             docs.append(row)
             tf_list.append(tf)
-            dl.append(len(tokens))
+            dl_list.append(len(tokens))
         self._docs = docs
         self._tf = tf_list
-        self._dl = dl
+        self._dl = dl_list
         self._n = len(docs)
-        self._avgdl = sum(dl) / max(self._n, 1)
+        self._avgdl = sum(dl_list) / max(self._n, 1)
         self._idf = {
             term: math.log((self._n - freq + 0.5) / (freq + 0.5) + 1)
             for term, freq in df.items()

devplacepy/services/devii/ai_correction/controller.py

@@ -70,7 +70,7 @@ class AiCorrectionController:
         if prompt_raw is None:
             prompt = user.get("ai_correction_prompt") or DEFAULT_CORRECTION_PROMPT
         else:
-            prompt = str(prompt_raw).strip()[:2000] or DEFAULT_CORRECTION_PROMPT
+            prompt = str(prompt_raw).strip()[:20000] or DEFAULT_CORRECTION_PROMPT
         get_table("users").update(
             {
                 "uid": self._owner_id,

devplacepy/services/devii/ai_modifier/controller.py

@@ -70,7 +70,7 @@ class AiModifierController:
         if prompt_raw is None:
             prompt = user.get("ai_modifier_prompt") or DEFAULT_MODIFIER_PROMPT
         else:
-            prompt = str(prompt_raw).strip()[:2000] or DEFAULT_MODIFIER_PROMPT
+            prompt = str(prompt_raw).strip()[:20000] or DEFAULT_MODIFIER_PROMPT
         get_table("users").update(
             {
                 "uid": self._owner_id,

devplacepy/services/devii/config.py

@@ -178,6 +178,7 @@ FIELD_PRICE_CACHE_HIT = "devii_price_cache_hit"
 FIELD_PRICE_CACHE_MISS = "devii_price_cache_miss"
 FIELD_PRICE_OUTPUT = "devii_price_output"
 FIELD_ALLOW_EVAL = "devii_allow_eval"
+FIELD_INTERACTIONS_DEFAULT = "devii_interactions_default"
 FIELD_BROWSER_CONTROL = "devii_browser_control"
 FIELD_TIMEOUT = "devii_timeout"
 FIELD_FETCH_TIMEOUT = "devii_fetch_timeout"

devplacepy/services/devii/fetch/controller.py

@@ -21,32 +21,6 @@ from ..text import html_to_text
 
 logger = logging.getLogger("devii.fetch")
 
-CHROME_VERSION = "131"
-CHROME_VERSION_FULL = "131.0.0.0"
-STEALTH_HEADERS = {
-    "User-Agent": (
-        "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
-        f"(KHTML, like Gecko) Chrome/{CHROME_VERSION_FULL} Safari/537.36"
-    ),
-    "Accept": (
-        "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,"
-        "image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"
-    ),
-    "Accept-Language": "en-US,en;q=0.9",
-    "sec-ch-ua": (
-        f'"Google Chrome";v="{CHROME_VERSION}", "Chromium";v="{CHROME_VERSION}", '
-        '"Not_A Brand";v="24"'
-    ),
-    "sec-ch-ua-mobile": "?0",
-    "sec-ch-ua-platform": '"Linux"',
-    "Sec-Fetch-Dest": "document",
-    "Sec-Fetch-Mode": "navigate",
-    "Sec-Fetch-Site": "none",
-    "Sec-Fetch-User": "?1",
-    "Upgrade-Insecure-Requests": "1",
-    "Cache-Control": "max-age=0",
-    "DNT": "1",
-}
 TITLE = re.compile(r"<title[^>]*>(.*?)</title>", re.IGNORECASE | re.DOTALL)
 MIN_FETCH_CHARS = 1000
 ALLOWED_METHODS = ("GET", "POST", "PUT", "PATCH", "DELETE", "HEAD", "OPTIONS")
@@ -230,9 +204,7 @@ class FetchController:
         raise_5xx: bool = False,
     ) -> tuple[str, str, str, int, dict[str, str]]:
         limit = self._settings.fetch_max_bytes
-        merged_headers = dict(STEALTH_HEADERS)
-        if headers:
-            merged_headers.update({str(k): str(v) for k, v in headers.items()})
+        merged_headers = {str(k): str(v) for k, v in headers.items()} if headers else {}
         request_kwargs: dict[str, Any] = {}
         if json_body is not None:
             request_kwargs["json"] = json_body
@@ -240,43 +212,48 @@ class FetchController:
             request_kwargs["data"] = form
         elif content is not None:
             request_kwargs["content"] = content
+
+        async def _once(client: httpx.AsyncClient, target: str) -> tuple[str, str, str, int, dict[str, str]]:
+            async with client.stream(method, target, **request_kwargs) as response:
+                if raise_5xx and response.status_code >= 500:
+                    raise UpstreamError(
+                        f"Server returned {response.status_code}.",
+                        status=response.status_code,
+                        url=target,
+                    )
+                chunks: list[bytes] = []
+                total = 0
+                async for chunk in response.aiter_bytes():
+                    chunks.append(chunk)
+                    total += len(chunk)
+                    if total >= limit:
+                        break
+                raw = b"".join(chunks)[:limit]
+                encoding = response.encoding or "utf-8"
+                try:
+                    body = raw.decode(encoding, errors="replace")
+                except LookupError:
+                    body = raw.decode("utf-8", errors="replace")
+                content_type = response.headers.get("content-type", "").lower()
+                response_headers = {key: value for key, value in response.headers.items()}
+                return body, str(response.url), content_type, response.status_code, response_headers
+
         try:
             async with stealth.stealth_async_client(
                 headers=merged_headers,
                 follow_redirects=True,
                 timeout=self._settings.fetch_timeout_seconds,
             ) as client:
-                async with client.stream(method, url, **request_kwargs) as response:
-                    if raise_5xx and response.status_code >= 500:
-                        raise UpstreamError(
-                            f"Server returned {response.status_code}.",
-                            status=response.status_code,
-                            url=url,
-                        )
-                    chunks: list[bytes] = []
-                    total = 0
-                    async for chunk in response.aiter_bytes():
-                        chunks.append(chunk)
-                        total += len(chunk)
-                        if total >= limit:
-                            break
-                    raw = b"".join(chunks)[:limit]
-                    encoding = response.encoding or "utf-8"
-                    try:
-                        body = raw.decode(encoding, errors="replace")
-                    except LookupError:
-                        body = raw.decode("utf-8", errors="replace")
-                    content_type = response.headers.get("content-type", "").lower()
-                    response_headers = {
-                        key: value for key, value in response.headers.items()
-                    }
-                    return (
-                        body,
-                        str(response.url),
-                        content_type,
-                        response.status_code,
-                        response_headers,
-                    )
+                result = await _once(client, url)
+                if method == "GET" and "html" in result[2]:
+                    gate_url = stealth.detect_consent_gate(result[0])
+                    if gate_url:
+                        try:
+                            await client.get(gate_url)
+                            result = await _once(client, url)
+                        except httpx.HTTPError as exc:
+                            logger.debug("consent gate follow-up failed for %s: %s", url, exc)
+                return result
         except httpx.TimeoutException as exc:
             raise NetworkError(f"Request timed out fetching {url}", url=url) from exc
         except httpx.HTTPError as exc:

devplacepy/services/devii/interaction/__init__.py

@@ -0,0 +1,37 @@
+# retoor <retoor@molodetz.nl>
+
+from __future__ import annotations
+
+from .capabilities import (
+    CHANNEL_SITE_CHAT,
+    CHANNEL_TELEGRAM,
+    CHANNEL_CLI,
+    CHANNEL_API,
+    CHANNEL_UNKNOWN,
+    ChannelContext,
+    channel_id_for_session,
+    context_for,
+    fragment_for,
+    interactions_enabled,
+)
+from .controller import InteractionController
+from . import prefs
+from .schema import InteractionRequest, InteractionResult, validate_prompt_args
+
+__all__ = [
+    "CHANNEL_SITE_CHAT",
+    "CHANNEL_TELEGRAM",
+    "CHANNEL_CLI",
+    "CHANNEL_API",
+    "CHANNEL_UNKNOWN",
+    "ChannelContext",
+    "channel_id_for_session",
+    "context_for",
+    "fragment_for",
+    "interactions_enabled",
+    "InteractionController",
+    "InteractionRequest",
+    "InteractionResult",
+    "validate_prompt_args",
+    "prefs",
+]

devplacepy/services/devii/interaction/actions.py

@@ -0,0 +1,130 @@
+# retoor <retoor@molodetz.nl>
+
+from __future__ import annotations
+
+from ..actions.spec import Action, Param
+
+
+def arg(
+    name: str, description: str, required: bool = False, kind: str = "string"
+) -> Param:
+    return Param(
+        name=name,
+        location="body",
+        description=description,
+        required=required,
+        type=kind,
+    )
+
+
+UI = (
+    "Presents a channel-aware interactive prompt (CA-IWP). The host renders the best "
+    "affordance for the current channel (site custom elements, Telegram inline keys, or "
+    "plain numbered menus). Blocks until the user submits, cancels, or the timeout fires. "
+    "Never invent HTML or custom-element tags in your prose - call this tool instead."
+)
+
+INTERACTION_ACTIONS: tuple[Action, ...] = (
+    Action(
+        name="ui_prompt",
+        method="LOCAL",
+        path="",
+        summary="Ask the user an interactive question with widgets (confirm, choice, text, form)",
+        description=(
+            UI
+            + " Prefer this for decisions and short forms. Returns structured "
+            "{interaction_id, status, values, meta}. Branch on status: submitted, cancelled, "
+            "timeout, superseded, or error. Keep labels short and value tokens stable."
+        ),
+        handler="interaction",
+        requires_auth=False,
+        params=(
+            arg("title", "Short question title (≤120 chars).", required=True),
+            arg("description", "Optional help prose (≤500 chars)."),
+            arg(
+                "widgets",
+                "Ordered list of widget objects. Types: confirm, choice, choice_multi, "
+                "text, number, date, select, // (help), group. Each input needs name+label; "
+                "choice types need options[{value,label}].",
+                required=True,
+                kind="array",
+            ),
+            arg("submit_label", "Primary action label (default Confirm)."),
+            arg("cancel_label", "Cancel action label (default Cancel)."),
+            arg("cancelable", "Whether the user may cancel (default true).", kind="boolean"),
+            arg(
+                "timeout_sec",
+                "Optional timeout in seconds (0 = none, max 86400).",
+                kind="integer",
+            ),
+            arg("id", "Optional stable interaction id (a-z, digits, hyphens)."),
+        ),
+    ),
+    Action(
+        name="ui_cancel",
+        method="LOCAL",
+        path="",
+        summary="Cancel the open interactive prompt",
+        description="Closes the current open interaction (or the one named by id) with status cancelled.",
+        handler="interaction",
+        requires_auth=False,
+        params=(
+            arg("id", "Optional interaction id to cancel; defaults to the open one."),
+            arg("reason", "Optional cancel reason."),
+        ),
+    ),
+    Action(
+        name="ui_notify",
+        method="LOCAL",
+        path="",
+        summary="Show a non-blocking ephemeral status on channels that support it",
+        description=(
+            "Sends a short status line to the user without waiting. Only effective when the "
+            "channel advertises ephemeral_status (site-chat). Elsewhere it is skipped."
+        ),
+        handler="interaction",
+        requires_auth=False,
+        params=(arg("text", "Status text to show (≤200 chars).", required=True),),
+    ),
+    Action(
+        name="interactions_get",
+        method="LOCAL",
+        path="",
+        summary="Show whether interactive widgets (ui_prompt) are enabled for the user",
+        description=(
+            "Returns the effective interactive-widgets preference: enabled (bool), source "
+            "('user' override or 'default'), the admin default, and the user's override if set. "
+            "The site default is set by administrators on the Devii service; users may override "
+            "it on their profile or with interactions_set."
+        ),
+        handler="interaction",
+        requires_auth=True,
+        read_only=True,
+    ),
+    Action(
+        name="interactions_set",
+        method="LOCAL",
+        path="",
+        summary="Enable, disable, or reset interactive widgets for the user",
+        description=(
+            "Sets whether this account accepts interactive ui_prompt widgets. Pass enabled=true "
+            "or false to override the admin default; pass reset=true (or omit enabled and set "
+            "reset) to clear the override and inherit the admin default again. Guests cannot "
+            "set a preference."
+        ),
+        handler="interaction",
+        requires_auth=True,
+        params=(
+            arg(
+                "enabled",
+                "true to enable interactive widgets, false to disable them. Omit when reset=true.",
+                kind="boolean",
+            ),
+            arg(
+                "reset",
+                "true to clear the user override and inherit the administrator default again.",
+                kind="boolean",
+            ),
+        ),
+    ),
+)

devplacepy/services/devii/interaction/broker.py

@@ -0,0 +1,513 @@
+# retoor <retoor@molodetz.nl>
+
+from __future__ import annotations
+
+import asyncio
+import logging
+import time
+from typing import Any, Awaitable, Callable, Optional
+
+import uuid_utils
+
+from .capabilities import (
+    CHANNEL_CLI,
+    CHANNEL_SITE_CHAT,
+    CHANNEL_TELEGRAM,
+    ChannelContext,
+    channel_id_for_session,
+    context_for,
+)
+from .markdown import project_markdown, project_plain_menu
+from .parse import parse_plain_reply
+from .schema import (
+    InteractionRequest,
+    InteractionResult,
+    count_input_fields,
+    validate_prompt_args,
+)
+
+logger = logging.getLogger("devii.interaction")
+
+EmitFn = Callable[[dict[str, Any]], Awaitable[None]]
+WaitFn = Callable[[str, dict[str, Any], float], Awaitable[Any]]
+TelegramPresentFn = Callable[[dict[str, Any]], Awaitable[Any]]
+PlainPresentFn = Callable[[str], Awaitable[str]]
+
+
+class InteractionBroker:
+    def __init__(
+        self,
+        session_channel: str = "main",
+        *,
+        owner_kind: str = "guest",
+        owner_id: str = "",
+        emit: Optional[EmitFn] = None,
+        wait_site: Optional[WaitFn] = None,
+        present_telegram: Optional[TelegramPresentFn] = None,
+        present_plain: Optional[PlainPresentFn] = None,
+    ) -> None:
+        self._session_channel = session_channel
+        self._channel_id = channel_id_for_session(session_channel)
+        self._owner_kind = owner_kind
+        self._owner_id = owner_id
+        self._emit = emit
+        self._wait_site = wait_site
+        self._present_telegram = present_telegram
+        self._present_plain = present_plain
+        self._open: dict[str, dict[str, Any]] = {}
+        self._lock = asyncio.Lock()
+        self._single_flight = True
+
+    @property
+    def channel_id(self) -> str:
+        return self._channel_id
+
+    def set_channel_id(self, channel_id: str) -> None:
+        self._channel_id = channel_id
+
+    def open_id(self) -> str | None:
+        if not self._open:
+            return None
+        return next(iter(self._open))
+
+    def open_request(self) -> InteractionRequest | None:
+        open_id = self.open_id()
+        if not open_id:
+            return None
+        payload = self._open.get(open_id) or {}
+        request = payload.get("request_model")
+        if isinstance(request, InteractionRequest):
+            return request
+        raw = payload.get("request")
+        if isinstance(raw, dict):
+            try:
+                return InteractionRequest.model_validate(raw)
+            except Exception:
+                return None
+        return None
+
+    def answer_text(self, text: str) -> dict[str, Any] | None:
+        open_id = self.open_id()
+        if not open_id:
+            return None
+        request = self.open_request()
+        if request is None:
+            return {
+                "handled": True,
+                "status": "error",
+                "error": "Open interaction has no parseable schema.",
+                "interaction_id": open_id,
+            }
+        status, values, error = parse_plain_reply(request, text)
+        if status == "error":
+            return {
+                "handled": True,
+                "status": "error",
+                "error": error or "Could not parse reply.",
+                "interaction_id": open_id,
+            }
+        result = {
+            "status": status,
+            "interaction_id": open_id,
+            "values": values if status == "submitted" else {},
+            "meta": {
+                "adapter": self._channel_id,
+                "degraded": True,
+                "via": "text",
+            },
+            "error": error,
+        }
+        return {
+            "handled": True,
+            "status": status,
+            "interaction_id": open_id,
+            "result": result,
+        }
+
+    def channel_context(self) -> ChannelContext:
+        owner_kind = getattr(self, "_owner_kind", "guest")
+        owner_id = getattr(self, "_owner_id", "")
+        return context_for(
+            self._channel_id,
+            open_interaction_id=self.open_id(),
+            owner_kind=owner_kind,
+            owner_id=owner_id,
+        )
+
+    def bind(
+        self,
+        *,
+        emit: Optional[EmitFn] = None,
+        wait_site: Optional[WaitFn] = None,
+        present_telegram: Optional[TelegramPresentFn] = None,
+        present_plain: Optional[PlainPresentFn] = None,
+    ) -> None:
+        if emit is not None:
+            self._emit = emit
+        if wait_site is not None:
+            self._wait_site = wait_site
+        if present_telegram is not None:
+            self._present_telegram = present_telegram
+        if present_plain is not None:
+            self._present_plain = present_plain
+
+    async def prompt(self, arguments: dict[str, Any]) -> InteractionResult:
+        request = validate_prompt_args(arguments)
+        async with self._lock:
+            if self._single_flight and self._open:
+                for open_id in list(self._open):
+                    await self._finish(
+                        open_id,
+                        status="superseded",
+                        values={},
+                        error=None,
+                    )
+            interaction_id = request.id or f"ix-{uuid_utils.uuid7().hex[:8]}"
+            if not interaction_id.startswith("ix-") and not request.id:
+                interaction_id = f"ix-{interaction_id}"
+            started = time.monotonic()
+            payload = {
+                "interaction_id": interaction_id,
+                "request": request.model_dump(),
+                "request_model": request,
+                "markdown": project_markdown(request, interaction_id),
+                "plain": project_plain_menu(request),
+                "channel_id": self._channel_id,
+                "started": started,
+                "future": asyncio.get_event_loop().create_future(),
+            }
+            self._open[interaction_id] = payload
+            logger.info(
+                "interaction open id=%s channel=%s widgets=%d",
+                interaction_id,
+                self._channel_id,
+                count_input_fields(request.widgets),
+            )
+
+        try:
+            result = await self._present(interaction_id, request, payload)
+        except Exception as exc:
+            logger.exception("interaction failed id=%s", interaction_id)
+            result = InteractionResult(
+                interaction_id=interaction_id,
+                status="error",
+                channel_id=self._channel_id,
+                values={},
+                meta={"adapter": self._channel_id, "degraded": True},
+                error=str(exc),
+            )
+            await self._resolve_local(interaction_id, result)
+            return result
+
+        latency_ms = int((time.monotonic() - started) * 1000)
+        if "latency_ms" not in result.meta:
+            result.meta["latency_ms"] = latency_ms
+        if "adapter" not in result.meta:
+            result.meta["adapter"] = self._channel_id
+        self._open.pop(interaction_id, None)
+        logger.info(
+            "interaction closed id=%s status=%s latency_ms=%s",
+            interaction_id,
+            result.status,
+            result.meta.get("latency_ms"),
+        )
+        return result
+
+    async def cancel(
+        self, interaction_id: str = "", reason: str = "cancelled"
+    ) -> InteractionResult:
+        target = interaction_id or self.open_id()
+        if not target or target not in self._open:
+            return InteractionResult(
+                interaction_id=target or "",
+                status="error",
+                channel_id=self._channel_id,
+                error="No open interaction to cancel.",
+            )
+        status = reason if reason in (
+            "cancelled",
+            "timeout",
+            "superseded",
+            "error",
+            "channel_changed",
+        ) else "cancelled"
+        result = InteractionResult(
+            interaction_id=target,
+            status=status,  # type: ignore[arg-type]
+            channel_id=self._channel_id,
+            values={},
+            meta={"adapter": self._channel_id, "reason": reason},
+        )
+        await self._resolve_local(target, result)
+        return result
+
+    def resolve(
+        self,
+        interaction_id: str,
+        *,
+        status: str,
+        values: dict[str, Any] | None = None,
+        error: str | None = None,
+        meta: dict[str, Any] | None = None,
+    ) -> bool:
+        payload = self._open.get(interaction_id)
+        if payload is None:
+            return False
+        result = InteractionResult(
+            interaction_id=interaction_id,
+            status=status,  # type: ignore[arg-type]
+            channel_id=self._channel_id,
+            values=values or {},
+            meta=meta or {"adapter": self._channel_id},
+            error=error,
+        )
+        future = payload.get("future")
+        if future is not None and not future.done():
+            future.set_result(result)
+        return True
+
+    async def _present(
+        self,
+        interaction_id: str,
+        request: InteractionRequest,
+        payload: dict[str, Any],
+    ) -> InteractionResult:
+        timeout = float(request.timeout_sec or 0)
+        if self._wait_site is not None and self._channel_id in (
+            CHANNEL_SITE_CHAT,
+            CHANNEL_TELEGRAM,
+        ):
+            return await self._present_site(interaction_id, request, payload, timeout)
+        if self._channel_id == CHANNEL_TELEGRAM:
+            return await self._present_telegram_path(
+                interaction_id, request, payload, timeout
+            )
+        return await self._present_plain_path(
+            interaction_id, request, payload, timeout
+        )
+
+    async def _present_site(
+        self,
+        interaction_id: str,
+        request: InteractionRequest,
+        payload: dict[str, Any],
+        timeout: float,
+    ) -> InteractionResult:
+        if self._wait_site is None:
+            return await self._present_plain_path(
+                interaction_id, request, payload, timeout, degraded=True
+            )
+        frame = {
+            "interaction_id": interaction_id,
+            "channel": self._channel_id,
+            "title": request.title,
+            "description": request.description,
+            "widgets": [w.model_dump() for w in request.widgets],
+            "submit_label": request.submit_label,
+            "cancel_label": request.cancel_label,
+            "cancelable": request.cancelable,
+            "timeout_sec": request.timeout_sec,
+            "markdown": payload["markdown"],
+            "plain": payload["plain"],
+            "request": request.model_dump(),
+        }
+        try:
+            raw = await self._wait_site(interaction_id, frame, timeout)
+        except asyncio.TimeoutError:
+            return InteractionResult(
+                interaction_id=interaction_id,
+                status="timeout",
+                channel_id=self._channel_id,
+                meta={
+                    "adapter": self._channel_id,
+                    "degraded": self._channel_id != CHANNEL_SITE_CHAT,
+                },
+            )
+        return self._normalize_site_result(
+            interaction_id,
+            raw,
+            adapter=self._channel_id,
+            degraded=self._channel_id != CHANNEL_SITE_CHAT,
+        )
+
+    async def _present_telegram_path(
+        self,
+        interaction_id: str,
+        request: InteractionRequest,
+        payload: dict[str, Any],
+        timeout: float,
+    ) -> InteractionResult:
+        if self._present_telegram is not None:
+            try:
+                raw = await self._present_telegram(
+                    {
+                        "interaction_id": interaction_id,
+                        "request": request.model_dump(),
+                        "markdown": payload["markdown"],
+                        "plain": payload["plain"],
+                        "timeout_sec": request.timeout_sec,
+                    }
+                )
+                return self._normalize_site_result(
+                    interaction_id, raw, adapter=CHANNEL_TELEGRAM, degraded=True
+                )
+            except asyncio.TimeoutError:
+                return InteractionResult(
+                    interaction_id=interaction_id,
+                    status="timeout",
+                    channel_id=self._channel_id,
+                    meta={"adapter": CHANNEL_TELEGRAM, "degraded": True},
+                )
+            except Exception as exc:
+                logger.exception("telegram adapter failed, falling back to plain")
+                payload["plain_error"] = str(exc)
+        return await self._present_plain_path(
+            interaction_id, request, payload, timeout, degraded=True
+        )
+
+    async def _present_plain_path(
+        self,
+        interaction_id: str,
+        request: InteractionRequest,
+        payload: dict[str, Any],
+        timeout: float,
+        *,
+        degraded: bool = False,
+    ) -> InteractionResult:
+        menu = payload["plain"]
+        if self._present_plain is not None:
+            try:
+                if timeout > 0:
+                    reply = await asyncio.wait_for(
+                        self._present_plain(menu), timeout=timeout
+                    )
+                else:
+                    reply = await self._present_plain(menu)
+            except asyncio.TimeoutError:
+                return InteractionResult(
+                    interaction_id=interaction_id,
+                    status="timeout",
+                    channel_id=self._channel_id,
+                    meta={
+                        "adapter": self._channel_id or CHANNEL_CLI,
+                        "degraded": True,
+                    },
+                )
+            status, values, error = parse_plain_reply(request, reply)
+            return InteractionResult(
+                interaction_id=interaction_id,
+                status=status,  # type: ignore[arg-type]
+                channel_id=self._channel_id,
+                values=values,
+                meta={
+                    "adapter": self._channel_id or CHANNEL_CLI,
+                    "degraded": True,
+                },
+                error=error,
+            )
+        future: asyncio.Future = payload["future"]
+        if self._emit is not None:
+            await self._emit(
+                {
+                    "type": "interaction",
+                    "id": interaction_id,
+                    "mode": "plain",
+                    "text": menu,
+                    "markdown": payload["markdown"],
+                }
+            )
+        try:
+            if timeout > 0:
+                result = await asyncio.wait_for(future, timeout=timeout)
+            else:
+                result = await future
+            if isinstance(result, InteractionResult):
+                result.meta.setdefault("degraded", degraded or True)
+                return result
+            return self._normalize_site_result(
+                interaction_id, result, adapter=self._channel_id, degraded=True
+            )
+        except asyncio.TimeoutError:
+            return InteractionResult(
+                interaction_id=interaction_id,
+                status="timeout",
+                channel_id=self._channel_id,
+                meta={"adapter": self._channel_id, "degraded": True},
+            )
+
+    def _normalize_site_result(
+        self,
+        interaction_id: str,
+        raw: Any,
+        *,
+        adapter: str = CHANNEL_SITE_CHAT,
+        degraded: bool = False,
+    ) -> InteractionResult:
+        if isinstance(raw, InteractionResult):
+            return raw
+        if not isinstance(raw, dict):
+            return InteractionResult(
+                interaction_id=interaction_id,
+                status="error",
+                channel_id=self._channel_id,
+                error="Invalid interaction result payload.",
+                meta={"adapter": adapter, "degraded": degraded},
+            )
+        if raw.get("error") and not raw.get("status"):
+            return InteractionResult(
+                interaction_id=interaction_id,
+                status="error",
+                channel_id=self._channel_id,
+                error=str(raw.get("error")),
+                meta={"adapter": adapter, "degraded": degraded},
+            )
+        status = str(raw.get("status") or "submitted")
+        if status not in (
+            "submitted",
+            "cancelled",
+            "timeout",
+            "superseded",
+            "error",
+            "channel_changed",
+        ):
+            status = "error"
+        values = raw.get("values") if isinstance(raw.get("values"), dict) else {}
+        meta = raw.get("meta") if isinstance(raw.get("meta"), dict) else {}
+        meta.setdefault("adapter", adapter)
+        meta.setdefault("degraded", degraded)
+        return InteractionResult(
+            interaction_id=str(raw.get("interaction_id") or interaction_id),
+            status=status,  # type: ignore[arg-type]
+            channel_id=self._channel_id,
+            values=values,
+            meta=meta,
+            error=str(raw["error"]) if raw.get("error") else None,
+        )
+
+    async def _finish(
+        self,
+        interaction_id: str,
+        *,
+        status: str,
+        values: dict[str, Any],
+        error: str | None,
+    ) -> None:
+        result = InteractionResult(
+            interaction_id=interaction_id,
+            status=status,  # type: ignore[arg-type]
+            channel_id=self._channel_id,
+            values=values,
+            meta={"adapter": self._channel_id},
+            error=error,
+        )
+        await self._resolve_local(interaction_id, result)
+
+    async def _resolve_local(
+        self, interaction_id: str, result: InteractionResult
+    ) -> None:
+        payload = self._open.pop(interaction_id, None)
+        if payload is None:
+            return
+        future = payload.get("future")
+        if future is not None and not future.done():
+            future.set_result(result)

devplacepy/services/devii/interaction/capabilities.py

@@ -0,0 +1,217 @@
+# retoor <retoor@molodetz.nl>
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Any
+
+CHANNEL_SITE_CHAT = "site-chat"
+CHANNEL_TELEGRAM = "telegram"
+CHANNEL_CLI = "cli"
+CHANNEL_API = "api"
+CHANNEL_UNKNOWN = "unknown"
+
+UI_TOOL_NAMES = frozenset(
+    {
+        "ui_prompt",
+        "ui_cancel",
+        "ui_notify",
+        "interactions_get",
+        "interactions_set",
+    }
+)
+
+DEFAULT_LIMITS = {
+    "max_options": 32,
+    "max_fields": 24,
+    "max_label_chars": 200,
+    "max_help_chars": 500,
+    "callback_data_bytes": 0,
+}
+
+_SITE_CAPS = {
+    "rich_widgets": True,
+    "markdown": True,
+    "confirm": True,
+    "choice_single": True,
+    "choice_multi": True,
+    "text_input": True,
+    "number_input": True,
+    "date_input": True,
+    "file_input": False,
+    "inline_buttons": True,
+    "polls": False,
+    "web_app": False,
+    "streaming_partial": True,
+    "ephemeral_status": True,
+}
+
+_TELEGRAM_CAPS = {
+    "rich_widgets": False,
+    "markdown": True,
+    "confirm": True,
+    "choice_single": True,
+    "choice_multi": True,
+    "text_input": True,
+    "number_input": True,
+    "date_input": True,
+    "file_input": False,
+    "inline_buttons": True,
+    "polls": True,
+    "web_app": False,
+    "streaming_partial": False,
+    "ephemeral_status": False,
+}
+
+_PLAIN_CAPS = {
+    "rich_widgets": False,
+    "markdown": True,
+    "confirm": True,
+    "choice_single": True,
+    "choice_multi": True,
+    "text_input": True,
+    "number_input": True,
+    "date_input": True,
+    "file_input": False,
+    "inline_buttons": False,
+    "polls": False,
+    "web_app": False,
+    "streaming_partial": False,
+    "ephemeral_status": False,
+}
+
+_EMPTY_CAPS = {key: False for key in _SITE_CAPS}
+
+_MATRICES: dict[str, dict[str, bool]] = {
+    CHANNEL_SITE_CHAT: _SITE_CAPS,
+    CHANNEL_TELEGRAM: _TELEGRAM_CAPS,
+    CHANNEL_CLI: _PLAIN_CAPS,
+    CHANNEL_API: _PLAIN_CAPS,
+    CHANNEL_UNKNOWN: _EMPTY_CAPS,
+}
+
+_SESSION_CHANNEL_MAP = {
+    "main": CHANNEL_SITE_CHAT,
+    "docs": CHANNEL_SITE_CHAT,
+    "telegram": CHANNEL_TELEGRAM,
+    "cli": CHANNEL_CLI,
+    "api": CHANNEL_API,
+}
+
+
+@dataclass
+class ChannelContext:
+    channel_id: str
+    capabilities: dict[str, bool] = field(default_factory=dict)
+    limits: dict[str, int] = field(default_factory=dict)
+    tools: list[str] = field(default_factory=list)
+    open_interaction_id: str | None = None
+
+    def to_dict(self) -> dict[str, Any]:
+        return {
+            "channel_id": self.channel_id,
+            "capabilities": dict(self.capabilities),
+            "limits": dict(self.limits),
+            "tools": list(self.tools),
+            "open_interaction_id": self.open_interaction_id,
+        }
+
+
+def channel_id_for_session(session_channel: str) -> str:
+    return _SESSION_CHANNEL_MAP.get(str(session_channel or "").strip().lower(), CHANNEL_UNKNOWN)
+
+
+def capabilities_for(channel_id: str) -> dict[str, bool]:
+    return dict(_MATRICES.get(channel_id, _EMPTY_CAPS))
+
+
+def limits_for(channel_id: str) -> dict[str, int]:
+    limits = dict(DEFAULT_LIMITS)
+    if channel_id == CHANNEL_TELEGRAM:
+        limits["callback_data_bytes"] = 64
+        limits["max_options"] = 16
+    return limits
+
+
+def interactions_enabled(
+    owner_kind: str = "guest", owner_id: str = ""
+) -> bool:
+    from .prefs import effective_for
+
+    return effective_for(owner_kind, owner_id)
+
+
+def context_for(
+    channel_id: str,
+    *,
+    open_interaction_id: str | None = None,
+    tools_allowed: bool = True,
+    owner_kind: str = "guest",
+    owner_id: str = "",
+) -> ChannelContext:
+    caps = capabilities_for(channel_id)
+    limits = limits_for(channel_id)
+    tools: list[str] = []
+    pref_tools = ["interactions_get", "interactions_set"]
+    if owner_kind == "user":
+        tools.extend(pref_tools)
+    if (
+        tools_allowed
+        and interactions_enabled(owner_kind, owner_id)
+        and channel_id in (CHANNEL_SITE_CHAT, CHANNEL_TELEGRAM, CHANNEL_CLI, CHANNEL_API)
+    ):
+        if open_interaction_id:
+            tools = [t for t in tools if t in pref_tools] + ["ui_cancel"]
+        else:
+            tools = list(dict.fromkeys(tools + ["ui_prompt", "ui_cancel"]))
+            if caps.get("ephemeral_status"):
+                tools.append("ui_notify")
+    return ChannelContext(
+        channel_id=channel_id,
+        capabilities=caps,
+        limits=limits,
+        tools=tools,
+        open_interaction_id=open_interaction_id,
+    )
+
+
+def fragment_for(ctx: ChannelContext) -> str:
+    active_caps = [name for name, on in ctx.capabilities.items() if on]
+    caps_text = ",".join(active_caps) if active_caps else "none"
+    tools_text = ",".join(ctx.tools) if ctx.tools else "none"
+    interaction = (
+        f"open:{ctx.open_interaction_id}" if ctx.open_interaction_id else "none"
+    )
+    return (
+        f"CHANNEL: {ctx.channel_id}\n"
+        f"CAPS: {caps_text}\n"
+        f"LIMITS: options≤{ctx.limits.get('max_options', 32)} "
+        f"fields≤{ctx.limits.get('max_fields', 24)}\n"
+        f"TOOLS: {tools_text}\n"
+        f"INTERACTION: {interaction}"
+    )
+
+
+CA_IWP_SYSTEM_FRAGMENT = (
+    "# Interactive asks (CA-IWP)\n"
+    "You are channel-aware. Current channel and capabilities are injected each turn as "
+    "CHANNEL / CAPS / LIMITS / TOOLS / INTERACTION.\n"
+    "HARD RULES when TOOLS includes ui_prompt (this is the interactive mode):\n"
+    "1. You MUST call ui_prompt for every decision, confirm, choice, short form, yes/no, "
+    "pick-one, multi-select, text/number/date ask, or demo of interactive widgets. Do not "
+    "simulate buttons in markdown when ui_prompt is available.\n"
+    "2. Do NOT fall back to numbered lists, fake [Yes]/[No] prose, or 'I would show a widget' "
+    "while ui_prompt is in TOOLS. Only use degraded markdown menus when ui_prompt is absent "
+    "from TOOLS.\n"
+    "3. Never invent HTML, JavaScript, or custom-element tags in your visible reply. "
+    "The host renders UI from tool args.\n"
+    "4. Never claim buttons/widgets exist on a channel that lacks them; match wording to CHANNEL.\n"
+    "5. One open interaction at a time. Wait for the ui_prompt tool result before continuing.\n"
+    "6. The user may answer with the on-screen controls OR by typing a clear answer "
+    "(yes/no, option label/number, free text field value, DD/MM/YYYY, or cancel). "
+    "If they type a new request instead of an answer, treat the interaction as cancelled "
+    "and continue with their new request.\n"
+    "7. On status=cancelled or timeout, do not pretend the user agreed. Branch explicitly.\n"
+    "8. Keep tool arguments compact: short labels, stable value tokens, no essays inside options.\n"
+    "9. Do not use ui_prompt for long-form research answers or pure information delivery."
+)

devplacepy/services/devii/interaction/controller.py

@@ -0,0 +1,131 @@
+# retoor <retoor@molodetz.nl>
+
+from __future__ import annotations
+
+import json
+import logging
+from typing import Any
+
+from devplacepy.services.devii.errors import ToolInputError
+
+from .broker import InteractionBroker
+from .capabilities import interactions_enabled
+from . import prefs
+from .schema import sanitize_text
+
+logger = logging.getLogger("devii.interaction")
+
+
+class InteractionController:
+    def __init__(
+        self,
+        broker: InteractionBroker,
+        owner_kind: str = "guest",
+        owner_id: str = "",
+    ) -> None:
+        self._broker = broker
+        self._owner_kind = owner_kind
+        self._owner_id = owner_id
+
+    @property
+    def broker(self) -> InteractionBroker:
+        return self._broker
+
+    async def dispatch(self, name: str, arguments: dict[str, Any]) -> str:
+        if name == "interactions_get":
+            return self._pref_get()
+        if name == "interactions_set":
+            return self._pref_set(arguments or {})
+        if name in ("ui_prompt", "ui_notify") and not interactions_enabled(
+            self._owner_kind, self._owner_id
+        ):
+            raise ToolInputError(
+                "Interactive widgets are disabled for this account. "
+                "Enable them with interactions_set or on the profile, or use a short numbered menu."
+            )
+        if name == "ui_prompt":
+            return await self._prompt(arguments)
+        if name == "ui_cancel":
+            return await self._cancel(arguments)
+        if name == "ui_notify":
+            return await self._notify(arguments)
+        raise ToolInputError(f"Unknown interaction tool: {name}")
+
+    def _require_user(self) -> None:
+        if self._owner_kind != "user" or not self._owner_id:
+            raise ToolInputError(
+                "Interactive-widget preferences are only available for signed-in users."
+            )
+
+    def _pref_get(self) -> str:
+        self._require_user()
+        data = prefs.snapshot(self._owner_kind, self._owner_id)
+        return json.dumps({"status": "success", **data}, ensure_ascii=False)
+
+    def _pref_set(self, arguments: dict[str, Any]) -> str:
+        self._require_user()
+        reset = arguments.get("reset")
+        if isinstance(reset, bool):
+            do_reset = reset
+        else:
+            do_reset = str(reset or "").strip().lower() in ("true", "1", "yes", "on")
+        if do_reset:
+            data = prefs.set_user_pref(self._owner_id, None)
+            return json.dumps({"status": "success", **data}, ensure_ascii=False)
+        if "enabled" not in arguments or arguments.get("enabled") is None:
+            raise ToolInputError(
+                "Pass enabled=true/false to override, or reset=true to inherit the admin default."
+            )
+        raw = arguments.get("enabled")
+        if isinstance(raw, bool):
+            enabled = raw
+        else:
+            enabled = str(raw).strip().lower() in ("true", "1", "yes", "on")
+        data = prefs.set_user_pref(self._owner_id, enabled)
+        return json.dumps({"status": "success", **data}, ensure_ascii=False)
+
+    async def _prompt(self, arguments: dict[str, Any]) -> str:
+        ctx = self._broker.channel_context()
+        if "ui_prompt" not in ctx.tools and self._broker.open_id():
+            raise ToolInputError(
+                "An interaction is already open. Wait for it, or call ui_cancel first."
+            )
+        if "ui_prompt" not in ctx.tools:
+            raise ToolInputError(
+                "ui_prompt is not available. Enable interactive widgets with interactions_set "
+                "or use a short numbered markdown menu."
+            )
+        result = await self._broker.prompt(arguments or {})
+        return json.dumps(result.to_dict(), ensure_ascii=False)
+
+    async def _cancel(self, arguments: dict[str, Any]) -> str:
+        interaction_id = sanitize_text(arguments.get("id", ""), 64)
+        reason = sanitize_text(arguments.get("reason", "cancelled"), 40) or "cancelled"
+        result = await self._broker.cancel(interaction_id, reason=reason)
+        return json.dumps(result.to_dict(), ensure_ascii=False)
+
+    async def _notify(self, arguments: dict[str, Any]) -> str:
+        text = sanitize_text(arguments.get("text", ""), 200)
+        if not text:
+            raise ToolInputError("'text' is required for ui_notify.")
+        ctx = self._broker.channel_context()
+        if not ctx.capabilities.get("ephemeral_status"):
+            return json.dumps(
+                {
+                    "status": "skipped",
+                    "channel_id": ctx.channel_id,
+                    "message": "ephemeral status is not supported on this channel",
+                },
+                ensure_ascii=False,
+            )
+        emit = getattr(self._broker, "_emit", None)
+        if emit is not None:
+            await emit({"type": "status", "text": text})
+        return json.dumps(
+            {
+                "status": "ok",
+                "channel_id": ctx.channel_id,
+                "text": text,
+            },
+            ensure_ascii=False,
+        )

devplacepy/services/devii/interaction/markdown.py

@@ -0,0 +1,106 @@
+# retoor <retoor@molodetz.nl>
+
+from __future__ import annotations
+
+from .schema import InteractionRequest, WidgetModel, flatten_widgets
+
+
+def project_markdown(request: InteractionRequest, interaction_id: str) -> str:
+    lines: list[str] = [f"### {request.title}", ""]
+    if request.description:
+        lines.append(request.description)
+        lines.append("")
+    for widget in request.widgets:
+        lines.extend(_widget_lines(widget))
+    actions = f"[{request.submit_label}]"
+    if request.cancelable:
+        actions += f" [{request.cancel_label}]"
+    lines.append("")
+    lines.append(actions)
+    lines.append(f"<!-- ai-interaction:{interaction_id} -->")
+    return "\n".join(lines).strip() + "\n"
+
+
+def project_plain_menu(request: InteractionRequest) -> str:
+    lines: list[str] = [request.title]
+    if request.description:
+        lines.append(request.description)
+    lines.append("")
+    widgets = [
+        w
+        for w in flatten_widgets(request.widgets)
+        if w.type != "//"
+    ]
+    if len(widgets) == 1 and widgets[0].type == "confirm":
+        lines.append(f"Reply: yes | no | cancel")
+        return "\n".join(lines).strip() + "\n"
+    if len(widgets) == 1 and widgets[0].type in ("choice", "select"):
+        options = widgets[0].options[:12]
+        lines.append(f"{widgets[0].label}")
+        for index, option in enumerate(options, start=1):
+            lines.append(f"{index}. {option.label} (`{option.value}`)")
+        lines.append("")
+        lines.append(f"Reply with a number (1-{len(options)}) or `cancel`.")
+        return "\n".join(lines).strip() + "\n"
+    for widget in widgets:
+        if widget.type in ("choice", "choice_multi", "select"):
+            lines.append(f"{widget.label}:")
+            for index, option in enumerate(widget.options[:12], start=1):
+                lines.append(f"  {index}. {option.label} (`{option.value}`)")
+        elif widget.type == "confirm":
+            lines.append(f"{widget.label}: yes | no")
+        else:
+            lines.append(f"{widget.label}: (free text)")
+    lines.append("")
+    lines.append("Reply with values, or `cancel`.")
+    return "\n".join(lines).strip() + "\n"
+
+
+def _widget_lines(widget: WidgetModel, indent: str = "") -> list[str]:
+    if widget.type == "//":
+        return [f"{indent}{widget.text or widget.label}", ""]
+    if widget.type == "group":
+        lines = [f"{indent}**{widget.label}**", ""]
+        for child in widget.widgets:
+            lines.extend(_widget_lines(child, indent + "  "))
+        return lines
+    req = "required" if widget.required else "optional"
+    lines = [f"{indent}- [ ] **{widget.name}** ({req}) — {widget.label}"]
+    if widget.help:
+        lines.append(f"{indent}  _{widget.help}_")
+    if widget.type == "confirm":
+        default = widget.default
+        yes = "•" if default is True else " "
+        no = "•" if default is False else " "
+        lines.append(f"{indent}  - ({yes}) `true` — Yes")
+        lines.append(f"{indent}  - ({no}) `false` — No")
+    elif widget.type in ("choice", "select"):
+        default = str(widget.default) if widget.default is not None else ""
+        for option in widget.options:
+            mark = "•" if option.value == default else " "
+            desc = f" — {option.description}" if option.description else ""
+            lines.append(
+                f"{indent}  - ({mark}) `{option.value}` — {option.label}{desc}"
+            )
+    elif widget.type == "choice_multi":
+        defaults = set(widget.default or []) if isinstance(widget.default, list) else set()
+        for option in widget.options:
+            mark = "x" if option.value in defaults else " "
+            desc = f" — {option.description}" if option.description else ""
+            lines.append(
+                f"{indent}  - [{mark}] `{option.value}` — {option.label}{desc}"
+            )
+    elif widget.type == "text":
+        extra = f", max {widget.max_length}" if widget.max_length else ""
+        lines.append(f"{indent}  - _free text{extra}_")
+    elif widget.type == "number":
+        parts = []
+        if widget.min is not None:
+            parts.append(f"min {widget.min}")
+        if widget.max is not None:
+            parts.append(f"max {widget.max}")
+        hint = (", " + ", ".join(parts)) if parts else ""
+        lines.append(f"{indent}  - _number{hint}_")
+    elif widget.type == "date":
+        lines.append(f"{indent}  - _date DD/MM/YYYY_")
+    return lines

devplacepy/services/devii/interaction/parse.py

@@ -0,0 +1,187 @@
+# retoor <retoor@molodetz.nl>
+
+from __future__ import annotations
+
+import re
+from datetime import datetime
+from typing import Any
+
+from .schema import InteractionRequest, WidgetModel, flatten_widgets
+
+YES = frozenset({"y", "yes", "true", "1", "ok", "confirm", "ja", "oui", "j"})
+NO = frozenset({"n", "no", "false", "0", "nee", "non"})
+CANCEL = frozenset({"cancel", "c", "abort", "quit", "annuleren", "stop"})
+DATE_FORMATS = (
+    "%d/%m/%Y",
+    "%d-%m-%Y",
+    "%d.%m.%Y",
+    "%d/%m/%y",
+    "%d-%m-%y",
+    "%Y-%m-%d",
+)
+
+
+def parse_plain_reply(
+    request: InteractionRequest, text: str
+) -> tuple[str, dict[str, Any], str | None]:
+    raw = (text or "").strip()
+    if not raw:
+        return "error", {}, "Empty reply."
+    lower = raw.lower()
+    if lower in CANCEL:
+        return "cancelled", {}, None
+
+    widgets = [w for w in flatten_widgets(request.widgets) if w.type != "//"]
+    if not widgets:
+        return "submitted", {}, None
+
+    if len(widgets) == 1:
+        return _parse_single(widgets[0], raw, strict=True)
+
+    if not _looks_like_multi_answer(raw, widgets):
+        return "error", {}, "Not a structured answer for the open form."
+
+    values: dict[str, Any] = {}
+    chunks = [part.strip() for part in re.split(r"[;\n]+", raw) if part.strip()]
+    if len(chunks) == 1 and "," in chunks[0] and len(widgets) > 1:
+        chunks = [part.strip() for part in chunks[0].split(",") if part.strip()]
+    if len(chunks) != len(widgets):
+        return (
+            "error",
+            {},
+            f"Expected {len(widgets)} values (one per field), got {len(chunks)}.",
+        )
+    for widget, chunk in zip(widgets, chunks):
+        status, partial, err = _parse_single(widget, chunk, strict=True)
+        if status != "submitted":
+            return status, {}, err
+        values.update(partial)
+    return "submitted", values, None
+
+
+def _looks_like_multi_answer(raw: str, widgets: list[WidgetModel]) -> bool:
+    if ";" in raw or "\n" in raw:
+        return True
+    if "," in raw and len(widgets) > 1:
+        return True
+    return False
+
+
+def _parse_single(
+    widget: WidgetModel, raw: str, *, strict: bool = False
+) -> tuple[str, dict[str, Any], str | None]:
+    text = raw.strip()
+    lower = text.lower()
+    if lower in CANCEL:
+        return "cancelled", {}, None
+    name = widget.name
+    if widget.type == "confirm":
+        if lower in YES:
+            return "submitted", {name: True}, None
+        if lower in NO:
+            return "submitted", {name: False}, None
+        return "error", {}, "Reply yes/no (or ja/nee)."
+    if widget.type in ("choice", "select"):
+        value = _match_option(widget, text)
+        if value is None:
+            return "error", {}, f"Unknown option for {name}."
+        return "submitted", {name: value}, None
+    if widget.type == "choice_multi":
+        parts = [p.strip() for p in re.split(r"[,\s]+", text) if p.strip()]
+        values: list[str] = []
+        for part in parts:
+            matched = _match_option(widget, part)
+            if matched is None:
+                return "error", {}, f"Unknown option {part!r} for {name}."
+            if matched not in values:
+                values.append(matched)
+        if widget.required and not values:
+            return "error", {}, f"{name} requires at least one option."
+        return "submitted", {name: values}, None
+    if widget.type == "number":
+        if strict and not re.fullmatch(r"[+-]?\d+(?:[.,]\d+)?", text):
+            return "error", {}, f"{name} must be a number."
+        normalized = text.replace(",", ".")
+        try:
+            number = float(normalized) if "." in normalized else int(normalized)
+        except ValueError:
+            return "error", {}, f"{name} must be a number."
+        if widget.min is not None and number < widget.min:
+            return "error", {}, f"{name} must be ≥ {widget.min}."
+        if widget.max is not None and number > widget.max:
+            return "error", {}, f"{name} must be ≤ {widget.max}."
+        return "submitted", {name: number}, None
+    if widget.type == "date":
+        if strict and not _looks_like_date(text):
+            return "error", {}, f"{name} must be a date as DD/MM/YYYY."
+        european = _parse_date_european(text)
+        if european is None:
+            return "error", {}, f"{name} must be a date as DD/MM/YYYY."
+        return "submitted", {name: european}, None
+    if widget.type == "text":
+        if strict and _looks_like_command(text):
+            return "error", {}, f"{name} does not look like a field answer."
+        if widget.max_length and len(text) > widget.max_length:
+            return "error", {}, f"{name} exceeds max length {widget.max_length}."
+        if widget.required and not text:
+            return "error", {}, f"{name} is required."
+        return "submitted", {name: text}, None
+    return "error", {}, f"Unsupported widget type {widget.type}."
+
+
+def _looks_like_date(text: str) -> bool:
+    return bool(
+        re.fullmatch(r"\d{1,2}[/.-]\d{1,2}[/.-]\d{2,4}", text.strip())
+        or re.fullmatch(r"\d{4}-\d{2}-\d{2}", text.strip())
+    )
+
+
+def _looks_like_command(text: str) -> bool:
+    value = text.strip().lower()
+    if not value:
+        return False
+    if value.startswith("/"):
+        return True
+    starters = (
+        "show me ",
+        "laat ",
+        "geef ",
+        "toon ",
+        "please ",
+        "can you ",
+        "could you ",
+        "i want ",
+        "ik wil ",
+        "volgende",
+        "next ",
+    )
+    return any(value.startswith(prefix) for prefix in starters)
+
+
+def _parse_date_european(text: str) -> str | None:
+    value = text.strip()
+    for fmt in DATE_FORMATS:
+        try:
+            return datetime.strptime(value, fmt).strftime("%d/%m/%Y")
+        except ValueError:
+            continue
+    return None
+
+
+def _match_option(widget: WidgetModel, text: str) -> str | None:
+    raw = text.strip()
+    if raw.isdigit():
+        index = int(raw)
+        if 1 <= index <= len(widget.options):
+            option = widget.options[index - 1]
+            if not option.disabled:
+                return option.value
+    lower = raw.lower()
+    for option in widget.options:
+        if option.disabled:
+            continue
+        if option.value == raw or option.value.lower() == lower:
+            return option.value
+        if option.label.lower() == lower:
+            return option.value
+    return None

devplacepy/services/devii/interaction/prefs.py

@@ -0,0 +1,119 @@
+# retoor <retoor@molodetz.nl>
+
+from __future__ import annotations
+
+from typing import Any
+
+INHERIT = -1
+FIELD_DEFAULT = "devii_interactions_default"
+FIELD_LEGACY = "devii_interactions_enabled"
+
+
+def _truthy(raw: Any, default: bool = True) -> bool:
+    if raw is None:
+        return default
+    if isinstance(raw, bool):
+        return raw
+    if isinstance(raw, (int, float)):
+        return raw != 0
+    text = str(raw).strip().lower()
+    if text in ("",):
+        return default
+    return text not in ("0", "false", "off", "no")
+
+
+def admin_default() -> bool:
+    try:
+        from devplacepy.database import get_setting
+
+        raw = get_setting(FIELD_DEFAULT, "")
+        if raw == "" or raw is None:
+            raw = get_setting(FIELD_LEGACY, "1")
+        return _truthy(raw, True)
+    except Exception:
+        return True
+
+
+def _coerce_user_pref(raw: Any) -> int | None:
+    if raw is None:
+        return None
+    try:
+        value = int(raw)
+    except (TypeError, ValueError):
+        return None
+    if value == INHERIT:
+        return None
+    if value in (0, 1):
+        return value
+    return None
+
+
+def user_pref_raw(user: dict[str, Any] | None) -> int | None:
+    if not user:
+        return None
+    return _coerce_user_pref(user.get("interactions_enabled"))
+
+
+def effective_for_user(user: dict[str, Any] | None) -> bool:
+    pref = user_pref_raw(user)
+    if pref is None:
+        return admin_default()
+    return bool(pref)
+
+
+def effective_for(owner_kind: str, owner_id: str = "") -> bool:
+    if owner_kind != "user" or not owner_id:
+        return admin_default()
+    try:
+        from devplacepy.database import get_table
+
+        user = get_table("users").find_one(uid=owner_id)
+    except Exception:
+        return admin_default()
+    return effective_for_user(user)
+
+
+def snapshot(owner_kind: str, owner_id: str = "", user: dict[str, Any] | None = None) -> dict[str, Any]:
+    default = admin_default()
+    if owner_kind != "user":
+        return {
+            "enabled": default,
+            "source": "default",
+            "default": default,
+            "override": None,
+        }
+    if user is None and owner_id:
+        try:
+            from devplacepy.database import get_table
+
+            user = get_table("users").find_one(uid=owner_id)
+        except Exception:
+            user = None
+    pref = user_pref_raw(user)
+    if pref is None:
+        return {
+            "enabled": default,
+            "source": "default",
+            "default": default,
+            "override": None,
+        }
+    return {
+        "enabled": bool(pref),
+        "source": "user",
+        "default": default,
+        "override": bool(pref),
+    }
+
+
+def set_user_pref(user_uid: str, enabled: bool | None) -> dict[str, Any]:
+    from devplacepy.database import get_table
+    from devplacepy.utils import clear_user_cache
+
+    value = INHERIT if enabled is None else (1 if enabled else 0)
+    get_table("users").update(
+        {"uid": user_uid, "interactions_enabled": value},
+        ["uid"],
+    )
+    clear_user_cache(user_uid)
+    user = get_table("users").find_one(uid=user_uid) or {"uid": user_uid}
+    return snapshot("user", user_uid, user)