` while running, server-rendered report via `render_content` when done), `GET /tools/isslop/{uid}/report.md`, `GET /tools/isslop/{uid}/badge.svg` (embeddable SVG authenticity badge). Analyses/reports/badges are permanent public capability URLs (`IsslopService.cleanup` never deletes them; only the job row is swept). Reachable by direct URL only; the former **Tools** header dropdown in `base.html` was removed |
+| `/projects/{slug}/containers` | projects/containers/ subpackage - admin per-project container manager (`instances.py` for creation/lifecycle/exec/logs/metrics/sync plus the exec websocket, `schedules.py` for cron/interval/once schedules, shared helpers in `_shared.py`). Every instance runs the shared `ppy` image. Reachable by direct URL and from the admin index; the former project detail page **Containers** button was removed |
| `/admin/containers` | admin/containers.py - admin **Containers** manager: `/admin/containers` lists every instance across all projects with inline actions (start/stop/restart/terminal/edit/delete) plus a create modal (project search-select, run-as user search-select, boot language + source editor, restart policy, start-on-boot, env/ports/limits/ingress); `/admin/containers/{uid}` is the per-instance detail page (lifecycle, live logs/metrics, PTY terminal, schedules, ingress, sync, status history) and `/admin/containers/{uid}/edit` edits run-as user, boot language/script/command, restart policy, start-on-boot, and limits. `POST /admin/containers/create`, `/{uid}/edit`, `/{uid}/{start,stop,restart,pause,resume}`, `/{uid}/sync`, `/{uid}/delete` call `api.*` directly under `require_admin` (no docker/exec backend duplicated); `GET /admin/containers/projects/search` and `/users/search` back the create/edit search-selects. The lifecycle and detail views stay layered over the per-project `/projects/{slug}/containers/instances/{uid}/...` endpoints (the instance carries its `project_uid`) |
| `/p/{slug}` | proxy.py - public ingress reverse proxy (HTTP + WebSocket) to a running container instance's published host port, opt-in per instance via `ingress_slug` |
| `/xmlrpc` | xmlrpc.py - reverse-proxies XML-RPC calls to the forking XML-RPC bridge (`services/xmlrpc/`, supervised by `XmlrpcService` on loopback `config.XMLRPC_PORT`), which generates one XML-RPC method per documented REST endpoint from `docs_api.API_GROUPS` (`posts.create`, `feed.list`, ...). One struct of named params per call; auth via in-band `api_key`, `X-API-KEY`/`Bearer` header, or `http://user:pass@host/xmlrpc` Basic. Full introspection + `system.multicall`; REST errors become XML-RPC faults. Exempt from the rate limiter (enforced on the forwarded internal hop). See `devplacepy/services/xmlrpc/CLAUDE.md` |
@@ -49,7 +50,7 @@ Prefixes are wired in `main.py`:
| `/quizzes` | quizzes/ package - **Quizzes**: `index.py` (hub, create, import, detail, export, edit, delete, publish, per-quiz leaderboard, cross-quiz scoreboard), `questions.py` (builder page + question CRUD + reorder), `attempts.py` (start/resume, play, answer, finish, results). The hub `GET /quizzes` is the approved three-column `.feed-layout` reused verbatim from `/feed`: filters + search left (`?filter=all|todo|done|mine|drafts`, `?search=`, `?page=`), the quiz list centre with the **New quiz** and **Create quiz with Devii** actions, and the cross-quiz **scoreboard** rail right (score per user, best attempt per quiz, 15s display cache). Publishing is terminal - every write on a published quiz is a 400 and there is no unpublish route. See `devplacepy/services/quiz/CLAUDE.md` |
| `/battles` | battles.py - **Opinion Wars**, week-long two-faction battles attached to posts: `GET ""` (listing, quizzes-style filters active/ended/mine + search), `GET /{uid}` (JSON-only full state, runs lazy resolution), `GET /{uid}/events?after=` (durable event trail replay), `POST /{uid}/join` (join or switch faction), `POST /{uid}/fight` (spend 25 Code Farm coins, deal level-weighted damage, 24h cooldown). The battle card renders on the parent post (no battle detail HTML page). See `devplacepy/services/opinionwar/CLAUDE.md` |
| `/game` | game/ package - the **Code Farm** idle game (`index.py` + `farm.py`): `GET /game` (page), `GET /game/state`, `GET /game/leaderboard?board=`, `POST /game/{plant,harvest,buy-plot,upgrade,fertilize,daily,grant,perk,prestige,legacy,mastery,quests/claim}`, `POST /game/{defense/upgrade,defense/downgrade,infrastructure/buy,cosmetics/buy,cosmetics/equip}`, plus social `GET /game/farm/{username}` and `POST /game/farm/{username}/{water,steal}`. See `devplacepy/services/game/CLAUDE.md` |
-| (none) | push.py - push + PWA: `GET /push.json` (VAPID public key + the providers accepting registrations; `apns` includes `environment` when active), `POST /push.json` (register with any active provider; a body without `provider` is a `webpush` body; an APNs body is `token` plus optional `client_id` for device-stable upsert; created/revived rows are probed and the JSON may include `delivered`/`error`; an unknown, disabled or unconfigured provider is a 400), `GET /service-worker.js`, `GET /manifest.json`. Provider protocol and delivery live in `devplacepy/push/` - see `devplacepy/push/CLAUDE.md` |
+| (none) | push.py - push + PWA: `GET /push.json` (VAPID public key + the providers accepting registrations; `apns` includes `environment` when active), `POST /push.json` (register with any active provider; a body without `provider` is a `webpush` body; an APNs body is `token` plus optional `client_id` for device-stable upsert; created/revived rows are probed and the JSON may include `delivered`/`error`; an unknown, disabled or unconfigured provider is a 400), `DELETE /push.json` (unregister exactly the one registration named by `endpoint`/`token`/`client_id`, same identity priority as registration; idempotent, always 200 `{unregistered}`; wired into `PushManager.js`'s logout-link interceptor so a webpush subscription is dropped before the browser navigates to `/auth/logout`), `GET /service-worker.js`, `GET /manifest.json`. Provider protocol and delivery live in `devplacepy/push/` - see `devplacepy/push/CLAUDE.md` |
| (none) | docs.py (docs/ package) - the documentation site (prose pages + API reference). See `routers/docs/CLAUDE.md` for the deep detail on this tree (`DOCS_PAGES`, audience tiers, prose rendering pipeline) |
| `/pubsub` | pubsub.py - **database-free** publish/subscribe bus. `WS /pubsub/ws` (subscribe/unsubscribe/publish frames, `foo.*` wildcards), `POST /pubsub/publish` + `GET /pubsub/topics` (admin/internal). Lock-owner-gated WS (`4013` retry) so subscribers converge on one worker; topic authz in `services/pubsub/policy.py` (`user.{uid}.*` private, `public.*` shared, admin/internal anywhere, guests opt-in). In-process `services.pubsub.publish(topic, data)` for backends; frontend `app.pubsub`. See `devplacepy/services/pubsub/CLAUDE.md` |
| (none) | seo.py - `/robots.txt`, `/sitemap.xml` |
@@ -92,6 +93,10 @@ The `comments` table uses `(target_type, target_uid)` so the same `_comment_sect
Every post card on the feed has an inline comment form (`.feed-comment-form`) beneath the post actions. It posts to `/comments/create` like the detail page comment form. Tests must scope Post button clicks to `#create-post-modal button.btn-primary:has-text('Post')` to avoid matching the inline comment's Post button.
+### Comment permalinks
+
+Every comment's action bar (`_comment.html`) includes a "Copy link" button built on the existing `id="comment-{uid}"` anchor (already used by notifications, the profile Activity tab, and `NotificationManager.js`'s scroll-to-highlight) - no new route or URL scheme. It is a plain `data-share="#comment-{{ item.comment['uid'] }}"` button: `DomUtils.initShareButtons` resolves the relative hash against `window.location.href` at click time (so it always copies the exact page the viewer is on, canonical or not), copies it via `navigator.clipboard.writeText`, and flashes "Copied!" on the button (`Toast.flash`) - the same mechanism the gist detail page's Share button already uses.
+
### Comment editing
A comment's owner (only the owner, never an admin) sees an inline "Edit" button (`data-action='edit'`) in `_comment.html`. `CommentManager.toggleEditForm` swaps the `.comment-text` for a textarea seeded from its `data-raw` attribute (the raw markdown, since `contentRenderer.applyTo` overwrites `textContent` on first render), posts via `Http.send` to `POST /comments/edit/{comment_uid}`, then re-renders the new body in place with `contentRenderer.applyTo`. The route (`content.edit_comment_record`) is `is_owner`-only, writes `content` + `updated_at`, records the `comment.edit` audit event, and branches on `wants_json`: JSON clients get `CommentEditOut{uid, content, url, updated_at}`, the no-JS form falls back to a redirect to the comment anchor. Edits are NOT soft-delete related (the body is overwritten in place). Devii tool: `edit_comment` (owner-only, no confirm). Scope test Edit clicks to `.comment-action-btn:has-text('Edit')`.
@@ -147,8 +152,9 @@ A dedicated `/gists` page for sharing code snippets. Uses `gists` table (auto-cr
- Source code rendered in `` block on detail page
- Syntax highlighting handled by existing `highlight.js` loaded globally in `base.html`
-- Copy button uses `navigator.clipboard.writeText()`
+- Copy button uses `navigator.clipboard.writeText()` via the shared `data-copy` handler (`DomUtils.initClipboardCopy`)
- Cards in listing show language badge, title, truncated description, author, star count
+- **Raw/rendered toggle for `language == "markdown"`:** unlike `"markdown_rendered"` (always rendered) and every other language (always raw), the plain `"markdown"` gist dual-renders server-side in `gist_detail.html` - the existing raw `` block plus a `render_content(gist['source_code'], ...)` block, the second one starting `hidden`. A `View rendered`/`View raw` button next to Copy uses the generic `data-view-toggle`/`data-view-toggle-alt` (+ the two `-label`/`-label-alt` pairs) attribute pair (`DomUtils.initViewToggles`, see `static/js/CLAUDE.md`) to swap the `hidden` class between the two blocks and its own label - no fetch, no new endpoint.
### Sitemap
@@ -317,6 +323,11 @@ All three columns are populated by `routers/posts.py` `post_page_context()`, the
- `POST /bookmarks/{target_type}/{target_uid}` toggles a `bookmarks` row; `GET /bookmarks/saved` renders the personal list (`saved.html`). Target types: `post`, `gist`, `project`, `news`.
- `_bookmark_button.html` takes `_type`, `_uid`, `_bookmarked`; `BookmarkManager.js` swaps the label/`bookmarked` class from the JSON `{saved}`. Batch state via `get_user_bookmarks(user_uid, target_type, uids)`.
+### Personal notes
+- A `notes` row is a private, per-user text annotation on a target (`post`, `gist`, `project`, `news` - the four detail pages, unlike bookmarks' listing-card coverage, since a note is read/written on the full content view, not skimmed from a card). `POST /notes/{target_type}/{target_uid}` (`NoteForm{content}`, max 4000 chars) creates or replaces the caller's own note on that target (revives a soft-deleted row rather than duplicating it, exactly like bookmarks); `POST /notes/{target_type}/{target_uid}/delete` soft-deletes it; `GET /notes/saved` renders the personal notes list (`notes.html`), mirroring `saved.html` but including each note's body.
+- There is no read/edit access for anyone but the author - the route only ever looks up `user_uid=user["uid"]`, so there is no "someone else's note" to view or moderate. `database/moderation.py` lists `notes` in `UNREPORTABLE_TABLES` ("private to the owner") for that reason.
+- `_note_button.html` takes `_type`, `_uid`, `_note` (the current content or `None`) and renders a button that opens a small inline textarea editor (not a modal - the body is short and the surrounding action bar has no room for a full dialog); `NoteManager.js` (`app.notes`) wires open/cancel/save/delete and swaps the button label/`has-note` class from the JSON `{content}` / `{deleted}` response, extending the shared `OptimisticAction` base like the other engagement controllers. Batch/single state via `get_user_notes(user_uid, target_type, uids)` (`database/engagement.py`), wired into `content.load_detail`/`detail_context` (post/gist/project) and `routers/news.py`'s detail route as `note_content` on the page context and the matching `*DetailOut` schema.
+
### Polls
- A poll rides on a post (one `polls` row keyed by `post_uid`, options in `poll_options`, one-per-user votes in `poll_votes`). Created in `posts.py:create_poll` when `poll_question` plus >= 2 non-empty `poll_options` are submitted (capped at 6). Both `create_post` and `edit_post` accept the poll fields; `edit_post` only attaches a poll when the post has **none** yet (it never replaces an existing poll). The builders live in the create-post modal (`feed.html`) and the edit-post modal (`post.html`, rendered only when the post has no poll) using `data-poll-toggle` / `data-poll-add-option`.
- `poll_options` accepts either repeated form fields (the web builders, which preserve commas inside an option label) **or** a single newline- or comma-separated string (the API/Devii path). `models.py:normalize_poll_options` splits a lone delimited element - applied as a `mode="before"` validator on `PostForm`/`PostEditForm` - so the documented "one per line or comma separated" agent format actually produces a multi-option poll instead of a single dropped option.
diff --git a/devplacepy/routers/admin/backups.py b/devplacepy/routers/admin/backups.py
index d9d2edb..32accc7 100644
--- a/devplacepy/routers/admin/backups.py
+++ b/devplacepy/routers/admin/backups.py
@@ -1,5 +1,6 @@
# retoor
+import asyncio
import logging
from pathlib import Path
from typing import Annotated
@@ -61,11 +62,12 @@ def _targets() -> list[dict]:
for key, meta in store.BACKUP_TARGETS.items()
]
-def _dashboard(can_download: bool) -> dict:
+async def _dashboard(can_download: bool) -> dict:
backups = [_backup_payload(row, can_download) for row in store.list_backups()]
schedules = store.list_schedules()
+ storage = await asyncio.to_thread(store.compute_storage_stats)
return {
- "storage": store.compute_storage_stats(),
+ "storage": storage,
"backups": backups,
"schedules": schedules,
"targets": _targets(),
@@ -77,7 +79,7 @@ def _dashboard(can_download: bool) -> dict:
@router.get("/backups", response_class=HTMLResponse)
async def admin_backups(request: Request):
admin = require_admin(request)
- data = _dashboard(is_primary_admin(admin))
+ data = await _dashboard(is_primary_admin(admin))
base = site_url(request)
seo_ctx = base_seo_context(
request,
@@ -107,7 +109,7 @@ async def admin_backups(request: Request):
@router.get("/backups/data")
async def admin_backups_data(request: Request):
admin = require_admin(request)
- data = _dashboard(is_primary_admin(admin))
+ data = await _dashboard(is_primary_admin(admin))
return JSONResponse(BackupDashboardOut.model_validate(data).model_dump(mode="json"))
@router.post("/backups/run")
diff --git a/devplacepy/routers/devii.py b/devplacepy/routers/devii.py
index 146ca22..7fd7281 100644
--- a/devplacepy/routers/devii.py
+++ b/devplacepy/routers/devii.py
@@ -286,6 +286,7 @@ async def devii_ws(websocket: WebSocket):
}
)
continue
+ svc.maybe_warn_quota_threshold(owner_kind, owner_id, owner_is_admin)
session.spawn_turn(text)
elif kind == "reset":
await session.reset()
diff --git a/devplacepy/routers/messages.py b/devplacepy/routers/messages.py
index 452ec42..9c615a1 100644
--- a/devplacepy/routers/messages.py
+++ b/devplacepy/routers/messages.py
@@ -39,6 +39,7 @@ from devplacepy.services.messaging import (
persist_message,
redeem_ticket,
stamp_content_revision,
+ touch_active_conversation,
)
logger = logging.getLogger(__name__)
@@ -276,6 +277,7 @@ async def send_message(request: Request, data: Annotated[MessageForm, Depends(js
data.attachment_uids,
request=request,
origin="web",
+ client_id=data.client_id,
)
if message is None:
return action_result(request, "/messages")
@@ -457,6 +459,7 @@ async def messages_ws(websocket: WebSocket):
attachment_uids,
request=websocket,
origin="websocket",
+ client_id=client_id,
)
except ContentRefused as exc:
await websocket.send_json(
@@ -483,6 +486,10 @@ async def messages_ws(websocket: WebSocket):
await message_hub.send_to_user(
receiver_uid, {"type": "typing", "from_uid": user_uid}
)
+ elif kind == "active":
+ with_uid = str(data.get("with_uid", "")).strip()
+ if with_uid:
+ touch_active_conversation(user_uid, with_uid)
elif kind == "read":
with_uid = str(data.get("with_uid", "")).strip()
if with_uid:
diff --git a/devplacepy/routers/news.py b/devplacepy/routers/news.py
index 3b58d24..92f4cca 100644
--- a/devplacepy/routers/news.py
+++ b/devplacepy/routers/news.py
@@ -13,6 +13,7 @@ from devplacepy.database import (
get_news_images_by_uids,
get_recent_comments_by_target_uids,
get_user_bookmarks,
+ get_user_notes,
paginate,
resolve_object_url,
mark_notifications_read_by_target,
@@ -125,6 +126,11 @@ async def news_detail_page(request: Request, news_slug: str):
bookmarked = bool(user) and article["uid"] in get_user_bookmarks(
user["uid"], "news", [article["uid"]]
)
+ note_content = (
+ get_user_notes(user["uid"], "news", [article["uid"]]).get(article["uid"])
+ if user
+ else None
+ )
base = site_url(request)
page_url = f"{base}/news/{canonical_slug}"
@@ -157,6 +163,7 @@ async def news_detail_page(request: Request, news_slug: str):
"time_ago": time_ago(article["synced_at"]),
"comments": comments,
"bookmarked": bookmarked,
+ "note_content": note_content,
"maturity": get_maturity("news", article["uid"])["level"],
},
model=NewsDetailOut,
diff --git a/devplacepy/routers/notes.py b/devplacepy/routers/notes.py
new file mode 100644
index 0000000..aad62a0
--- /dev/null
+++ b/devplacepy/routers/notes.py
@@ -0,0 +1,188 @@
+# retoor
+import logging
+from datetime import datetime, timezone
+from typing import Annotated
+from fastapi import APIRouter, Form, Request
+from fastapi.responses import RedirectResponse, JSONResponse, HTMLResponse
+from devplacepy.database import get_table, db, paginate, resolve_object_url, _now_iso
+from devplacepy.models import NoteForm
+from devplacepy.utils import generate_uid, require_user, time_ago, redirect_back
+from devplacepy.seo import base_seo_context
+from devplacepy.responses import respond
+from devplacepy.schemas import NotesOut
+from devplacepy.services.audit import record as audit
+
+logger = logging.getLogger(__name__)
+router = APIRouter()
+
+NOTABLE: set[str] = {"post", "gist", "project", "news"}
+
+TABLE_BY_TYPE: dict[str, str] = {
+ "post": "posts",
+ "gist": "gists",
+ "project": "projects",
+ "news": "news",
+}
+
+LABEL_BY_TYPE: dict[str, str] = {
+ "post": "Post",
+ "gist": "Gist",
+ "project": "Project",
+ "news": "Article",
+}
+
+
+@router.get("/saved", response_class=HTMLResponse)
+async def notes_page(request: Request, before: str = None):
+ user = require_user(request)
+ notes = get_table("notes")
+ rows, next_cursor = paginate(notes, before=before, user_uid=user["uid"])
+
+ uids_by_type: dict[str, list] = {}
+ for row in rows:
+ uids_by_type.setdefault(row["target_type"], []).append(row["target_uid"])
+
+ resolved: dict[tuple, dict] = {}
+ for target_type, uids in uids_by_type.items():
+ table_name = TABLE_BY_TYPE.get(target_type)
+ if not table_name or table_name not in db.tables:
+ continue
+ table = get_table(table_name)
+ clauses = [table.table.columns.uid.in_(uids)]
+ if table.has_column("deleted_at"):
+ clauses.append(table.table.columns.deleted_at.is_(None))
+ for obj in table.find(*clauses):
+ resolved[(target_type, obj["uid"])] = obj
+
+ items = []
+ for row in rows:
+ obj = resolved.get((row["target_type"], row["target_uid"]))
+ if not obj:
+ continue
+ title = obj.get("title") or (obj.get("content", "") or "")[:80] or "Untitled"
+ items.append(
+ {
+ "target_type": row["target_type"],
+ "type_label": LABEL_BY_TYPE.get(
+ row["target_type"], row["target_type"].title()
+ ),
+ "title": title,
+ "url": resolve_object_url(row["target_type"], row["target_uid"]),
+ "content": row["content"],
+ "time_ago": time_ago(row.get("updated_at") or row["created_at"]),
+ "updated_at": row.get("updated_at"),
+ }
+ )
+
+ seo_ctx = base_seo_context(
+ request,
+ title="Notes",
+ description="Your personal notes on DevPlace.",
+ robots="noindex,nofollow",
+ )
+ return respond(
+ request,
+ "notes.html",
+ {
+ **seo_ctx,
+ "request": request,
+ "user": user,
+ "items": items,
+ "next_cursor": next_cursor,
+ },
+ model=NotesOut,
+ )
+
+
+@router.post("/{target_type}/{target_uid}")
+async def set_note(
+ request: Request,
+ target_type: str,
+ target_uid: str,
+ data: Annotated[NoteForm, Form()],
+):
+ user = require_user(request)
+ if target_type not in NOTABLE:
+ return JSONResponse({"error": "Invalid target"}, status_code=400)
+
+ notes = get_table("notes")
+ existing = notes.find_one(
+ user_uid=user["uid"], target_uid=target_uid, target_type=target_type
+ )
+ now = datetime.now(timezone.utc).isoformat()
+ content = data.content.strip()
+ if existing:
+ uid = existing["uid"]
+ notes.update(
+ {
+ "id": existing["id"],
+ "content": content,
+ "updated_at": now,
+ "deleted_at": None,
+ "deleted_by": None,
+ },
+ ["id"],
+ )
+ else:
+ uid = generate_uid()
+ notes.insert(
+ {
+ "uid": uid,
+ "user_uid": user["uid"],
+ "target_uid": target_uid,
+ "target_type": target_type,
+ "content": content,
+ "created_at": now,
+ "updated_at": now,
+ "deleted_at": None,
+ "deleted_by": None,
+ }
+ )
+
+ audit.record(
+ request,
+ "note.set",
+ user=user,
+ target_type=target_type,
+ target_uid=target_uid,
+ summary=f"{user['username']} saved a note on {target_type} {target_uid}",
+ links=[audit.target(target_type, target_uid)],
+ )
+
+ if request.headers.get("x-requested-with") == "fetch":
+ return JSONResponse({"uid": uid, "content": content})
+ return RedirectResponse(url=redirect_back(request), status_code=302)
+
+
+@router.post("/{target_type}/{target_uid}/delete")
+async def delete_note(request: Request, target_type: str, target_uid: str):
+ user = require_user(request)
+ if target_type not in NOTABLE:
+ return JSONResponse({"error": "Invalid target"}, status_code=400)
+
+ notes = get_table("notes")
+ existing = notes.find_one(
+ user_uid=user["uid"], target_uid=target_uid, target_type=target_type
+ )
+ if existing and not existing.get("deleted_at"):
+ notes.update(
+ {
+ "id": existing["id"],
+ "deleted_at": _now_iso(),
+ "deleted_by": user["uid"],
+ },
+ ["id"],
+ )
+ audit.record(
+ request,
+ "note.delete",
+ user=user,
+ target_type=target_type,
+ target_uid=target_uid,
+ summary=f"{user['username']} deleted a note on {target_type} {target_uid}",
+ links=[audit.target(target_type, target_uid)],
+ )
+
+ if request.headers.get("x-requested-with") == "fetch":
+ return JSONResponse({"deleted": True})
+ return RedirectResponse(url=redirect_back(request), status_code=302)
diff --git a/devplacepy/routers/projects/CLAUDE.md b/devplacepy/routers/projects/CLAUDE.md
index 137e42f..d9805b9 100644
--- a/devplacepy/routers/projects/CLAUDE.md
+++ b/devplacepy/routers/projects/CLAUDE.md
@@ -11,7 +11,7 @@ Each project card links to `/projects/{project_uid}` showing full project detail
**Project overview page.** The detail page is a dedicated project showcase: one encompassing dark card (`.project-shell`, the site `--bg-card` surface with clipped corners) wraps the hero, the section tab bar and the two-column body, and every inner panel (tab bar, sidebar cards, devlog post cards, empty state, comments section) sits one elevation lighter on `--bg-secondary`. The hero's cover banner is the attachment referenced by `projects.cover_attachment_uid`, falling back to the first image attachment (brand-gradient band when neither exists); the title block, type/platform chips and author row render OVERLAID on the banner behind a bottom scrim (dark text-shadow for readability) beside the optional `projects.logo_attachment_uid` tile, with an owner-set **Visit Website** CTA (`projects.website_url`). Cover and logo ride the ONE existing upload pipeline: `dp-upload` widgets (`name="cover_attachment_uid"`/`"logo_attachment_uid"`, `max-files="1"`) in the create/edit modals upload to `/uploads/upload`, the route validates each uid via `database.get_user_attachment` (must exist, belong to the actor, be an image - `_hero_attachment_uid`) and links it to the project through `attachments.link_attachments`; an empty value on edit keeps the current image (no removal control). `website_url`/`repo_url` are normalized by `models.normalize_website_url` (scheme-less input gets `https://`, non-http(s) rejected) and render with `rel="noopener nofollow"`. Below the hero an anchor **section tab bar** (`.project-tabs`, underline style, Overview `.active`) links `#about` / `#devlog` / `#screenshots` (only when gallery images exist) / `#comments` / the Files page - server-rendered anchors, no JS tab state. The main column holds **About** (description + non-image attachments), the **Devlog** (every post whose `project_uid` points at the project via `_post_card.html` - the template loads `feed.css` for the card styles alongside `post.css`, the same rule as `news.html`) with `devlog_count` (`content.count_project_devlog`) and an owner **Post update** button (`.project-devlog-post-btn`) opening the shared composer preset to `topic=devlog` + this project (the form lives ONCE in `templates/_post_composer_form.html`, locals `_composer_topic`/`_composer_project`, included by `feed.html` and `project_detail.html` - never fork a second copy), a **Screenshots** gallery (image attachments minus the cover/logo, thumbnails, `data-lightbox`, capped at 12 rendered), and the comment thread; the sidebar holds Links (website/repository/files/fork source), the Stats card (5 `.project-stat` entries + a last-update line) and the Author card. Owners add gallery images via the More-menu **Add screenshots** modal: `_attachment_form.html` uploads, then `POST /projects/{slug}/screenshots` (`ProjectScreenshotsForm`, owner-only, audit `project.screenshots.add`) links the uids through the same `link_attachments` choke point; Devii action `project_add_screenshots`, docs id `projects-screenshots`. `comment_count`/`devlog_count` ride `ProjectDetailOut`; the new project fields ride `ProjectOut`; the page og:image prefers the cover attachment. **Locator discipline:** the page has several `Files` anchors (action row, tab bar, sidebar) and, for owners, a second hidden `textarea[name='content']`/Post button inside the composer modal - tests MUST scope (`.project-detail-actions a:has-text('Files')`, `.comment-form textarea[name='content']`).
-**Action row overflow.** The detail page has more actions than fit one line, so `project_detail.html` keeps the engagement actions inline (Files, Share, star vote, bookmark, reactions) and collapses the rest behind a single **More** button (`.project-actions-more`) that opens the shared `app.contextMenu`. The secondary actions (Workspace, Containers, Download zip, Fork, the owner Edit/Private/Read-only/Delete controls) live as real elements inside a hidden `.project-actions-overflow` container, each tagged `data-menu-action` plus `data-menu-icon`/`data-menu-label`. `static/js/ProjectActionsMenu.js` builds the menu from those elements and each item's `onSelect` simply `.click()`s the real element, so all existing wiring is reused unchanged - `app.zipDownloader` (`data-zip-download`), `app.projectForker` (`data-fork-project`), `data-share`, the `data-modal` Edit trigger, and the delegated `data-confirm`/`data-confirm-danger` dialog on the owner forms. The open handler must `stopPropagation()` because `app.contextMenu`'s document-level close listener would otherwise dismiss it on the same click (every other caller opens it from a right-click `attach`, not a left-click). Reuse this pattern - a `More` trigger over `[data-menu-action]` real elements - for any future action row that overflows; do not duplicate controller logic into menu callbacks.
+**Action row overflow.** The detail page has more actions than fit one line, so `project_detail.html` keeps the engagement actions inline (Files, Share, star vote, bookmark, reactions) and collapses the rest behind a single **More** button (`.project-actions-more`) that opens the shared `app.contextMenu`. The secondary actions (Download zip, Fork, the owner Edit/Private/Read-only/Delete controls) live as real elements inside a hidden `.project-actions-overflow` container, each tagged `data-menu-action` plus `data-menu-icon`/`data-menu-label`. `static/js/ProjectActionsMenu.js` builds the menu from those elements and each item's `onSelect` simply `.click()`s the real element, so all existing wiring is reused unchanged - `app.zipDownloader` (`data-zip-download`), `app.projectForker` (`data-fork-project`), `data-share`, the `data-modal` Edit trigger, and the delegated `data-confirm`/`data-confirm-danger` dialog on the owner forms. The open handler must `stopPropagation()` because `app.contextMenu`'s document-level close listener would otherwise dismiss it on the same click (every other caller opens it from a right-click `attach`, not a left-click). Reuse this pattern - a `More` trigger over `[data-menu-action]` real elements - for any future action row that overflows; do not duplicate controller logic into menu callbacks.
**Owner editing.** Mirrors post editing exactly: an owner-only **Edit** menu item (`data-modal="edit-project-modal"`) opens the `modal()` macro's `edit-project-modal`, a plain `POST` form to `/projects/edit/{slug}` (route `edit_project` in `routers/projects/index.py`, body `ProjectEditForm`, owner-gated through the shared `content.edit_content_item` which returns 403 JSON / redirect for non-owners and stamps `updated_at`). The modal is the create modal pre-filled from the `project` row (title, description, type/status radios pre-checked, dates via `format_date()` back to DD/MM/YYYY). `is_private`/`read_only` are NOT edited here - they stay on their dedicated toggles. The platforms tag widget reuses the create modal's `platforms-input`/`platforms`/`platforms-tags` ids; `ProfileEditor.initPlatformTags` now **seeds existing tags** from the hidden `#platforms` value on load, so both the empty create form and the pre-filled edit form work from the same code. Devii tool `edit_project`; documented in `docs_api.py` (`projects-edit`).
diff --git a/devplacepy/routers/push.py b/devplacepy/routers/push.py
index e8b1b51..a64bbf6 100644
--- a/devplacepy/routers/push.py
+++ b/devplacepy/routers/push.py
@@ -91,6 +91,49 @@ async def push_register(request: Request) -> JSONResponse:
return JSONResponse(payload)
+@router.delete("/push.json")
+async def push_unregister(request: Request) -> JSONResponse:
+ user = require_user_api(request)
+ try:
+ body = await request.json()
+ except ValueError:
+ return JSONResponse({"error": "Invalid JSON"}, status_code=400)
+
+ if not isinstance(body, dict):
+ return JSONResponse({"error": "Invalid request"}, status_code=400)
+
+ provider = providers.get(body.get("provider"))
+ if provider is None:
+ return JSONResponse({"error": "Unknown provider"}, status_code=400)
+
+ identity = {key: body.get(key) for key in ("client_id", "token", "endpoint")}
+ if not any(isinstance(value, str) and value.strip() for value in identity.values()):
+ return JSONResponse({"error": "Invalid request"}, status_code=400)
+
+ removed = push.unregister(user["uid"], provider.name, identity)
+
+ if removed:
+ endpoint = identity.get("endpoint")
+ audit.record(
+ request,
+ "push.unsubscribe",
+ user=user,
+ target_type="user",
+ target_uid=user["uid"],
+ target_label=user.get("username"),
+ metadata={
+ "provider": provider.name,
+ "endpoint_host": urlparse(endpoint).hostname
+ if isinstance(endpoint, str) and endpoint
+ else None,
+ "has_client_id": bool(identity.get("client_id")),
+ },
+ summary=f"{user.get('username')} unregistered a push subscription",
+ links=[audit.target("user", user["uid"], user.get("username"))],
+ )
+ return JSONResponse({"unregistered": removed})
+
+
@router.get("/service-worker.js")
async def service_worker() -> FileResponse:
return FileResponse(
diff --git a/devplacepy/routers/seo.py b/devplacepy/routers/seo.py
index 2ced4b9..47b0085 100644
--- a/devplacepy/routers/seo.py
+++ b/devplacepy/routers/seo.py
@@ -15,8 +15,8 @@ async def robots_txt(request: Request):
return PlainTextResponse(
f"""User-agent: *
Disallow: /auth/
-Disallow: /messages/
-Disallow: /notifications/
+Disallow: /messages
+Disallow: /notifications
Disallow: /votes/
Disallow: /avatar/
Disallow: /follow/
@@ -24,6 +24,7 @@ Disallow: /admin/
Disallow: /uploads/
Disallow: /reports/mine
Disallow: /profile/*/delete
+Disallow: /game
Disallow: /*?tab=
Disallow: /*?sort=
Allow: /static/
diff --git a/devplacepy/routers/tools/deepsearch.py b/devplacepy/routers/tools/deepsearch.py
index 08523e7..5ec6d04 100644
--- a/devplacepy/routers/tools/deepsearch.py
+++ b/devplacepy/routers/tools/deepsearch.py
@@ -11,7 +11,7 @@ from devplacepy import database
from devplacepy.config import DEEPSEARCH_DIR
from devplacepy.models import DeepsearchChatForm, DeepsearchRunForm
from devplacepy.responses import respond
-from devplacepy.schemas import DeepsearchJobOut, DeepsearchSessionOut
+from devplacepy.schemas import DeepsearchHistoryOut, DeepsearchJobOut, DeepsearchSessionOut
from devplacepy.seo import base_seo_context, site_url, web_application_schema, website_schema
from devplacepy.services.deepsearch.chat import DeepsearchChat
from devplacepy.services.deepsearch.export import to_json, to_markdown, to_pdf
@@ -197,6 +197,52 @@ def _enqueue(uid: str, payload: dict, owner_kind: str, owner_id: str, query: str
}
)
+def _history_item(row: dict) -> dict:
+ uid = row.get("uid", "")
+ status = row.get("status", "")
+ job = queue.get_job(uid)
+ return {
+ "uid": uid,
+ "query": row.get("query"),
+ "status": status,
+ "score": row.get("score"),
+ "confidence": row.get("confidence"),
+ "source_diversity": row.get("source_diversity"),
+ "page_count": int(row.get("page_count") or 0),
+ "chunk_count": int(row.get("chunk_count") or 0),
+ "summary": row.get("summary") or None,
+ "reopen_url": f"/tools/deepsearch/{uid}/session",
+ "chat_available": status == "done" and job is not None,
+ "available": job is not None,
+ "created_at": row.get("created_at"),
+ "completed_at": row.get("completed_at") or None,
+ }
+
+@router.get("/history")
+async def deepsearch_history(request: Request, limit: int = 20):
+ owner_kind, owner_id = owner_for(request)
+ user = get_current_user(request)
+ rows = database.list_deepsearch_sessions(owner_kind, owner_id, min(max(1, limit), 100))
+ sessions = [_history_item(row) for row in rows]
+ seo_ctx = base_seo_context(
+ request,
+ title="DeepSearch History",
+ description="Past DeepSearch research runs, with links back to each report and its grounded chat.",
+ robots="noindex,nofollow",
+ breadcrumbs=[
+ {"name": "Home", "url": "/feed"},
+ {"name": "Tools", "url": "/tools"},
+ {"name": "DeepSearch", "url": "/tools/deepsearch"},
+ {"name": "History", "url": "/tools/deepsearch/history"},
+ ],
+ )
+ return respond(
+ request,
+ "tools/deepsearch_history.html",
+ {**seo_ctx, "request": request, "user": user, "sessions": sessions},
+ model=DeepsearchHistoryOut,
+ )
+
@router.get("/{uid}")
async def deepsearch_status(request: Request, uid: str):
job = queue.get_job(uid)
diff --git a/devplacepy/schemas/__init__.py b/devplacepy/schemas/__init__.py
index 00967fc..00a5a09 100644
--- a/devplacepy/schemas/__init__.py
+++ b/devplacepy/schemas/__init__.py
@@ -43,6 +43,8 @@ from devplacepy.schemas.listings import (
NewsDetailOut,
NewsListItemOut,
NewsListOut,
+ NoteItemOut,
+ NotesOut,
NotificationGroupOut,
NotificationItemOut,
NotificationsOut,
@@ -88,6 +90,8 @@ from devplacepy.schemas.containers import (
)
from devplacepy.schemas.jobs import (
DbQueryJobOut,
+ DeepsearchHistoryItemOut,
+ DeepsearchHistoryOut,
DeepsearchJobOut,
DeepsearchSessionOut,
ForkJobOut,
diff --git a/devplacepy/schemas/jobs.py b/devplacepy/schemas/jobs.py
index 2aacb58..23007fb 100644
--- a/devplacepy/schemas/jobs.py
+++ b/devplacepy/schemas/jobs.py
@@ -143,6 +143,27 @@ class DeepsearchSessionOut(_Out):
completed_at: Optional[str] = None
+class DeepsearchHistoryItemOut(_Out):
+ uid: str = ""
+ query: Optional[str] = None
+ status: str = ""
+ score: Optional[int] = None
+ confidence: Optional[float] = None
+ source_diversity: Optional[float] = None
+ page_count: int = 0
+ chunk_count: int = 0
+ summary: Optional[str] = None
+ reopen_url: Optional[str] = None
+ chat_available: bool = False
+ available: bool = False
+ created_at: Optional[str] = None
+ completed_at: Optional[str] = None
+
+
+class DeepsearchHistoryOut(_Out):
+ sessions: list = []
+
+
class DbQueryJobOut(_Out):
uid: str = ""
kind: str = ""
diff --git a/devplacepy/schemas/listings.py b/devplacepy/schemas/listings.py
index 80d7aa5..dfd7b06 100644
--- a/devplacepy/schemas/listings.py
+++ b/devplacepy/schemas/listings.py
@@ -152,6 +152,7 @@ class PostDetailOut(_Out):
attachments: list[AttachmentOut] = []
reactions: ReactionsOut = ReactionsOut()
bookmarked: bool = False
+ note_content: Optional[str] = None
poll: Optional[PollOut] = None
war: Optional[WarOut] = None
comment_count: Optional[int] = None
@@ -187,6 +188,7 @@ class ProjectDetailOut(_Out):
attachments: list[AttachmentOut] = []
reactions: ReactionsOut = ReactionsOut()
bookmarked: bool = False
+ note_content: Optional[str] = None
platforms: Optional[Any] = None
is_private: bool = False
read_only: bool = False
@@ -227,6 +229,7 @@ class GistDetailOut(_Out):
attachments: list[AttachmentOut] = []
reactions: ReactionsOut = ReactionsOut()
bookmarked: bool = False
+ note_content: Optional[str] = None
class NewsListOut(_Out):
@@ -243,6 +246,7 @@ class NewsDetailOut(_Out):
time_ago: Optional[str] = None
comments: list[CommentItemOut] = []
bookmarked: bool = False
+ note_content: Optional[str] = None
class MessagesOut(_Out):
@@ -275,3 +279,19 @@ class SavedOut(_Out):
items: list[SavedItemOut] = []
next_cursor: Optional[str] = None
+
+class NoteItemOut(_Out):
+ target_type: Optional[str] = None
+ target_uid: Optional[str] = None
+ type_label: Optional[str] = None
+ title: Optional[str] = None
+ url: Optional[str] = None
+ content: Optional[str] = None
+ time_ago: Optional[str] = None
+ updated_at: Optional[str] = None
+
+
+class NotesOut(_Out):
+ items: list[NoteItemOut] = []
+ next_cursor: Optional[str] = None
+
diff --git a/devplacepy/services/CLAUDE.md b/devplacepy/services/CLAUDE.md
index d968539..3d52c4f 100644
--- a/devplacepy/services/CLAUDE.md
+++ b/devplacepy/services/CLAUDE.md
@@ -27,9 +27,10 @@ Generic, lightweight, **fire-and-forget** offload for non-critical side-effects
- **Choke entrypoint.** `schedule_correction(user, table, uid, request=None)` is the only thing handlers call (every hook passes `request`). It is a no-op unless: a user dict is present, `table` is in the registry, `user["ai_correction_enabled"]` is truthy, and the user has a non-empty `api_key`. In **sync** mode (`user["ai_correction_sync"]`) it calls `_run_inline_awaited`, which - only when a `request.scope` and a running event loop exist - submits `_run_correction` to `loop.run_in_executor(AI_APPLY_EXECUTOR, ...)` (off the loop thread, on the dedicated AI pool) and stashes the future on `request.scope[PENDING_SCOPE_KEY]` for the middleware to await; if there is no request/loop it returns False and falls back to background. In **background** mode (default) it `background.submit`s `_run_correction` (per-worker queue; also runs inline under `DEVPLACE_DISABLE_SERVICES=1`). The same `_run_correction` worker function runs in all cases.
- **The hooks (covers UI + REST + Devii + devRant in one place):** `content.create_content_item` (posts/projects/gists), `content.edit_content_item`, `content.create_comment_record`, `content.edit_comment_record`, `services/messaging/persist.persist_message` (DMs - sender is the user), `routers/profile/index.update_profile` (bio), and the two devRant direct-update edit paths (`routers/devrant/rants.edit_rant`, `routers/devrant/comments.edit_comment`). devRant create paths route through the shared content cores, so they are already hooked. Code-only and external paths (project files, Gitea) are deliberately not hooked.
- **The gateway call is fail-soft.** `correct_text(api_key, prompt, text)` is synchronous (runs on the background worker thread), POSTs to `INTERNAL_GATEWAY_URL` with `model=correction_model()` (`get_setting("correction_model", "") or INTERNAL_MODEL` - admin-configurable at `/admin/settings`, blank falls back to the gateway default `molodetz`; the gateway URL itself is never configurable per feature, always `INTERNAL_GATEWAY_URL`) via `stealth.stealth_sync_client`, authenticated with the user's own `Bearer` api_key (per-user attribution). It returns the ORIGINAL text on any error, empty output, or suspiciously large output (`len > len(text) * MAX_GROWTH_FACTOR + 200`, rejecting hallucinated expansion). `_run_correction` reads the row back, corrects each registry field that has non-blank text, writes only changed fields via `table.update(updates, ["uid"])` without touching `updated_at` (auto-correction is not a user edit) or the slug (slugs are permanent), and `clear_user_cache(user_uid)` when `table == "users"` so the corrected bio re-caches.
+- **Structural markdown-preservation guard (issue #84).** The growth-factor check alone never caught a correction that kept a similar length while stripping/rewrapping markdown structure (a flattened list, a dropped code fence, a removed header, a stripped link) - correction is prompt-only ("preserve... markdown") with zero enforcement otherwise. `gateway_complete(..., check_structure=False)` gained an opt-in structural check, wired on ONLY from `correct_text` (`check_structure=True`), that runs right after the growth-factor check: `rendering.markdown_structure_signature(text)` parses the text through a dedicated mistune AST pipeline (`_structure_markdown = mistune.create_markdown(renderer=None, plugins=["strikethrough", "table"])`, matching `_content_markdown`'s plugin set) and counts four structural signals recursively over the token tree - `code_fences` (`block_code`), `list_items` (`list_item`, ordered+unordered together), `headers` (`heading`, all levels together), `links` (`link`). `correction.structure_diverges(original, corrected)` compares the two signatures per key with `STRUCTURE_DROP_TOLERANCE = 1`: a count that drops from >0 to exactly 0 always trips it (total loss of a structural element type is the corruption signature - a single header/link/fence is as real as five), any other decrease bigger than the 1-item tolerance also trips it (catches a 5-item list collapsed to 1, not just to 0), and any increase or a decrease of at most 1 passes (tolerates a single word fixed inside a list item, or two adjacent items reflowed into one, without ever changing the actual count of an unrelated structural family). `gateway_complete` logs and returns the ORIGINAL text on divergence, exactly like the growth-factor rejection - no partial write, no exception. `ai_modifier.py`'s `modify_text` intentionally does NOT set `check_structure` - its whole job is to genuinely restructure content per an explicit `@ai` instruction (e.g. "@ai turn this into a numbered list"), so a structural-divergence check there would reject the very thing the user asked for; distinguishing "restructuring the user asked for" from "restructuring elsewhere in the same field the instruction didn't target" is not reliably automatable from a signature diff alone, so the guard is scoped to `correct_text` only, which never intentionally restructures.
- **Per-user usage aggregation (only on success).** `correct_text` returns `(text, usage)`; `usage` is parsed from the gateway's `X-Gateway-*` response headers (`_usage_from_headers`) whenever the upstream call returned 200 (cost was incurred, even if the corrected output was rejected), else `None` on any failure. `_usage_from_headers` captures the token and cost headers PLUS the timing headers `X-Gateway-Upstream-Latency-Ms` and `X-Gateway-Total-Latency-Ms` (as `upstream_latency_ms`/`total_latency_ms`), so each call's timing is metered. `_run_correction` accumulates the per-field `usage` into one `totals` dict and, when `totals["calls"] > 0`, makes ONE call to `database.add_correction_usage(user_uid, totals)` (a single `totals` dict, not positional args) - so a 2-field content item is a single aggregated write, and a failed/empty correction records nothing. `add_correction_usage` (and `add_modifier_usage`) delegate to the shared `database._add_usage(usage_table, user_uid, totals)`: a single atomic `INSERT ... ON CONFLICT(user_uid) DO UPDATE SET col = col + excluded.col` upsert against the `correction_usage` table (per-user running SUMS: `calls`/`prompt_tokens`/`completion_tokens`/`total_tokens`/`cost_usd`/`upstream_latency_ms`/`total_latency_ms`/`updated_at`, unique index `idx_correction_usage_user`, the two latency columns REAL default 0.0, all ensured in `init_db`). It is a derived counter table (NOT in `SOFT_DELETE_TABLES`, like `gateway_usage_ledger`) and is deliberately separate from `users` so accumulating never invalidates the auth/user cache. `database.get_correction_usage(user_uid)` (via `_get_usage`) returns the stored sums PLUS computed averages: `avg_tokens` (total_tokens/calls), `avg_upstream_latency_ms`, `avg_total_latency_ms`, `avg_tokens_per_second` (completion_tokens over total upstream seconds), and `avg_cost_usd`.
- **Profile display:** `routers/profile/usage._correction_usage(uid, include_cost)` shapes it via the shared `_usage_view(data, include_cost)` (mirrors `_ai_quota`); `profile/index.py` builds it only for `is_owner or viewer_is_admin` and passes `include_cost=viewer_is_admin`, exposed as the `correction_usage` dict on the context and `ProfileOut`. The view surfaces the sums plus averages - `avg_tokens` (avg tokens/call), `avg_latency_ms` (avg upstream latency), `avg_total_latency_ms`, `avg_tokens_per_second` (avg speed), and `total_time_s` (total upstream seconds) - rendered as extra tiles on the card. **Financial gating:** tokens/call-count and the performance tiles show to the owner and admins; the dollar `cost_usd` and `avg_cost_usd` keys are present ONLY when `viewer_is_admin`, in BOTH the HTML card (`templates/profile.html`, `.correction-usage-*`) and the `respond(..., model=ProfileOut)` JSON (same rule as `_ai_quota`'s `spent_usd` - hiding it in the template alone would leak it to a member fetching their own profile as JSON). The card renders only when `correction_usage.calls` > 0.
-- **Import-cycle discipline.** `correction.py` imports only `stealth`, `config`, `database.get_table`/`add_correction_usage`, and `services.background.background` at module top; `clear_user_cache` is imported lazily inside `_run_correction`. Never import `content` or `utils` at module top.
+- **Import-cycle discipline.** `correction.py` imports only `stealth`, `config`, `database.get_table`/`add_correction_usage`, `rendering.markdown_structure_signature` (structural guard above - `rendering.py` has zero internal imports, so this adds no cycle risk), and `services.background.background` at module top; `clear_user_cache` is imported lazily inside `_run_correction`. Never import `content` or `utils` at module top.
- **Settings live on `users`:** three columns `ai_correction_enabled` (0/1), `ai_correction_sync` (0/1, default 0 = background), and `ai_correction_prompt` (text, default `config.DEFAULT_CORRECTION_PROMPT`), ensured in `database.backfill_api_keys()` (the user column-ensure block run by `init_db`) and seeded born-live in `utils._create_account`. The edit route is the owner-or-admin leaf `POST /profile/{username}/ai-correction` (`routers/profile/ai_correction.py`, `AiCorrectionForm{enabled, sync, prompt}`, audit key `profile.ai_correction`). The owner-only values are exposed on the profile page context and `ProfileOut` (`ai_correction_enabled`/`ai_correction_sync`/`ai_correction_prompt`, gated by `is_owner`), the UI block lives in `profile.html` (owner-only: enable checkbox, **Apply mode** select, prompt textarea) wired by `static/js/AiCorrection.js` (`app.aiCorrection`), and Devii drives it via the owner-scoped `ai_correction_get`/`ai_correction_set` tools (`services/devii/ai_correction/`, `handler="ai_correction"`, `requires_auth=True`, not confirm-gated - it is a reversible per-user toggle; `ai_correction_set` accepts `enabled`, optional `sync`, optional `prompt`).
## AI modifier (`services/ai_modifier.py`, `services/ai_context.py`)
diff --git a/devplacepy/services/ai_modifier.py b/devplacepy/services/ai_modifier.py
index c7328ba..dc78856 100644
--- a/devplacepy/services/ai_modifier.py
+++ b/devplacepy/services/ai_modifier.py
@@ -43,7 +43,13 @@ def modify_text(
+ context
)
return gateway_complete(
- api_key, system, text, MODIFIER_TIMEOUT_SECONDS, None, model=modifier_model()
+ api_key,
+ system,
+ text,
+ MODIFIER_TIMEOUT_SECONDS,
+ None,
+ model=modifier_model(),
+ bypass_preamble=True,
)
diff --git a/devplacepy/services/backup/CLAUDE.md b/devplacepy/services/backup/CLAUDE.md
index 588f32f..34b1eeb 100644
--- a/devplacepy/services/backup/CLAUDE.md
+++ b/devplacepy/services/backup/CLAUDE.md
@@ -12,14 +12,14 @@ Admin-only, enterprise-grade backups built on the **same async-job pattern as zi
## Storage and data model
- **Storage:** archives go under `config.BACKUPS_DIR` (`data/backups/`, in `DATA_PATHS`) sharded with `attachments._directory_for` on the **random uuid tail** (same load-bearing reason as zips/blobs), named `{target}-{YYYYMMDD-HHMMSS}-{tail}.tar.gz`. Staging is `config.BACKUP_STAGING_DIR` (`data/backup_staging/`), removed in `process` `finally`.
-- **Data model** (`store.py`, ensured in `database.init_db` via `backup_store.ensure_tables()`): `backups` (NOT soft-deletable - an archive is a reclaimable operational artifact, hard-deleted like zips) and `backup_schedules` (in `SOFT_DELETE_TABLES`, born-live `deleted_at:None`). `store` holds all CRUD plus `compute_storage_stats()` (du of every major data area + `shutil.disk_usage`, run in `asyncio.to_thread` from the route, 30s in-process TTL cache so the walk never blocks).
+- **Data model** (`store.py`, ensured in `database.init_db` via `backup_store.ensure_tables()`): `backups` (NOT soft-deletable - an archive is a reclaimable operational artifact, hard-deleted like zips) and `backup_schedules` (in `SOFT_DELETE_TABLES`, born-live `deleted_at:None`). `store` holds all CRUD plus two storage helpers that must never be confused: `disk_usage()` is an O(1) `shutil.disk_usage` of `DATA_DIR` (15s TTL) and is the only thing the service tick may call; `compute_storage_stats()` does a **single** `os.walk` of `DATA_DIR` (overlapping path buckets, 600s TTL, in-process lock so callers cannot stampede), and is run in `asyncio.to_thread` from the admin route and the live-view relay. **Never call `compute_storage_stats()` on the event loop.** A 30s cache with a walk that itself takes >=30s never hits, pins one core in `pathlib.rglob` / `is_symlink`, stops `accept()`, and the reverse proxy returns 502. That was a production outage.
- **Permanent artifact:** `cleanup(job)` only removes leftover staging, NEVER the archive. Job retention prunes the `jobs` row; the archive and `backups` row persist until an admin deletes it, a schedule rotates it out (`keep_last`), or `devplace backups clear`. Deleting a backup is a HARD delete (unlink file + delete row) - correct because backups are GC artifacts, the documented exception to the soft-delete rule.
## Remote offload
`devplacepy/services/backup/offload.py` ships completed archives to a Hetzner Storage Box over WebDAV via `rclone` (`config.RCLONE_BIN`/`config.RCLONE_CONFIG_FILE`, remote name `config.BACKUP_OFFLOAD_REMOTE`, default `storagebox:devplacepy-backups`) - deliberately **not** the `/backup` davfs2 mount, whose FUSE metadata cache lives on the root filesystem and breaks exactly when disk fills (the original outage cause). `BackupService._run_offload_cycle` (throttled to `backup_offload_interval_seconds`, default 300s, via `ConfigField`s in the `Offload` group) runs each cycle after `_fire_due_schedules`:
-1. `upload_pending` - every `done` backup with `remote_uploaded_at` unset and a live `local_path` is `rclone copyto`'d to `//`, then verified by exact byte-size match (`rclone size --json`) against `size_bytes` recorded at finalize time. Only on a verified match does `store.mark_remote_uploaded` set `remote_path`/`remote_uploaded_at`. A failed or unverified upload is silently retried next cycle - `remote_uploaded_at` is the only source of truth for "is this backup actually safe off-box."
+1. `upload_pending` - every `done` backup with `remote_uploaded_at` unset and a live `local_path` is `rclone copyto`'d to `//`, then verified by exact byte-size match (`rclone size --json`) against `size_bytes` recorded at finalize time. Only on a verified match does `store.mark_remote_uploaded` set `remote_path`/`remote_uploaded_at`. A failed or unverified upload is retried next cycle - `remote_uploaded_at` is the only source of truth for "is this backup actually safe off-box." **Auth failures short-circuit the rest of the pending list** (HTTP 401 / "didn't find section"): one stale `rclone.conf` password must not retry every local archive in the same cycle. The davfs2 mount at `/backup` is a credential oracle for the same WebDAV host; if rclone 401s while that mount still works, the obscured `pass` in `rclone.conf` is stale and must be re-obscured from `/etc/davfs2/secrets`. Do not "fix" 401 by writing through the davfs2 mount.
2. `enforce_local_retention` (`backup_offload_keep_local`, default 1) - per target, keeps the newest N **offloaded** local copies and unlinks the rest (`store.mark_local_purged`: clears `local_path`, sets `local_purged_at`, row and `remote_path` persist). A backup with no confirmed remote copy is never touched, no matter how old.
3. `enforce_remote_retention` (`backup_offload_keep_remote`, default 30) - per target, `rclone lsjson` the remote dir and `deletefile` anything beyond the newest N, sorted by filename (safe because the `{target}-YYYYMMDD-HHMMSS-*` name is lexicographically chronological, same property `schedule.to_iso` relies on).
@@ -29,6 +29,10 @@ Admin-only, enterprise-grade backups built on the **same async-job pattern as zi
**Operational prerequisite (production, not automatic):** the `rclone` binary is installed in the shipped Docker image, but a working WebDAV remote still needs to exist at `config.RCLONE_CONFIG_FILE` (default `$HOME/.config/rclone/rclone.conf` inside the app container, overridable via `DEVPLACE_RCLONE_CONFIG`) with a remote named to match `config.BACKUP_OFFLOAD_REMOTE`'s prefix (default `storagebox`) pointing at the Hetzner Storage Box's WebDAV endpoint and credentials - `rclone config` (interactive) or a hand-written `rclone.conf` generates it. Until that file exists, every `upload_pending` attempt fails fast (`rclone` errors "didn't find section") and is logged and retried next cycle; local retention and rotation both stay disabled the whole time (see above), so backups simply accumulate locally with no data loss, they just never leave the box. **In Docker, `HOME=/app` (the bind-mounted repo root, `docker-compose.yml`), so the default config path resolves to `/.config/rclone/rclone.conf` on the host - `.gitignore` excludes `/.config/` precisely because this file holds live remote-storage credentials; never force-add it.**
+## Disk usage warning
+
+`disk_usage()` is the only source of the used/free percentage. `BackupService._check_disk_usage()` and `collect_metrics()` call it (never `compute_storage_stats()`). `disk_warn_percent_field` (`backup_disk_warn_percent`, default 90, group "Alerts") is compared with simple hysteresis - `self.log(...)` fires once when usage reaches the threshold ("Disk usage critical: ...") and once more when it drops back below ("Disk usage back under threshold: ..."), never repeating every tick while the state is unchanged. The warning is visible in this service's own Logs tab (`/admin/services/backup`), which also gets a live "Disk usage" stat card via `collect_metrics()` (`super().collect_metrics()` from `JobService` plus the disk percentage/free space). The admin dashboard's per-directory file counts still come from `compute_storage_stats()`, off-thread. This is intentionally minimal - a log line and a stat card via the existing `BaseService` mechanisms, no new table, route, or notification channel.
+
## Schedules
`backup_schedules` carry `kind` (`interval`|`cron`), `every_seconds`/`cron`, `enabled`, `keep_last`, `next_run_at`, run bookkeeping. `_fire_due_schedules` (lock-owner only, so each fires once) compares `next_run_at <= to_iso(now_utc())` and enqueues a `backup` job + a `backups` record, then advances `next_run_at` via `schedule.next_run`. **Timestamp format is load-bearing:** schedule `next_run_at` uses the devii `schedule.to_iso` format (`%Y-%m-%dT%H:%M:%S`, no tz/micros) on BOTH sides of the comparison so lexicographic compare equals chronological - do not mix it with `datetime.isoformat()`.
diff --git a/devplacepy/services/backup/offload.py b/devplacepy/services/backup/offload.py
index dffa824..2a8892c 100644
--- a/devplacepy/services/backup/offload.py
+++ b/devplacepy/services/backup/offload.py
@@ -30,6 +30,16 @@ def _remote_dir(target: str) -> str:
return f"{config.BACKUP_OFFLOAD_REMOTE}/{target}"
+def _is_auth_error(err: str) -> bool:
+ text = err.lower()
+ return (
+ "401" in text
+ or "unauthorized" in text
+ or "didn't find section" in text
+ or "did not find section" in text
+ )
+
+
async def upload_pending(log=lambda message: None) -> int:
uploaded = 0
for row in store.list_pending_offload():
@@ -38,6 +48,12 @@ async def upload_pending(log=lambda message: None) -> int:
code, _, err = await _run_rclone("copyto", str(local_path), remote_path)
if code != 0:
log(f"Offload failed for {row['filename']}: {err.strip()[:300]}")
+ if _is_auth_error(err):
+ log(
+ "Offload halted: remote storage rejected credentials; "
+ "remaining uploads skipped until the next cycle"
+ )
+ break
continue
size_code, size_out, size_err = await _run_rclone("size", remote_path, "--json")
if size_code != 0:
diff --git a/devplacepy/services/backup/service.py b/devplacepy/services/backup/service.py
index ec63519..9e7c8b6 100644
--- a/devplacepy/services/backup/service.py
+++ b/devplacepy/services/backup/service.py
@@ -27,6 +27,7 @@ WORKER_MODULE = "devplacepy.services.jobs.backup_worker"
DEFAULT_OFFLOAD_INTERVAL_SECONDS = 300
DEFAULT_OFFLOAD_KEEP_LOCAL = 1
DEFAULT_OFFLOAD_KEEP_REMOTE = 30
+DEFAULT_DISK_WARN_PERCENT = 90
class BackupService(JobService):
@@ -41,6 +42,21 @@ class BackupService(JobService):
def __init__(self):
super().__init__(name="backup", interval_seconds=15)
self._last_offload_at = 0.0
+ self._disk_warned = False
+ self.disk_warn_percent_field = ConfigField(
+ "backup_disk_warn_percent",
+ "Disk usage warning threshold (%)",
+ type="int",
+ default=DEFAULT_DISK_WARN_PERCENT,
+ minimum=50,
+ maximum=99,
+ help=(
+ "Log a warning (visible in this service's Logs tab) once the data "
+ "volume's used disk percentage reaches this threshold, and again "
+ "when it drops back below it."
+ ),
+ group="Alerts",
+ )
self.offload_enabled_field = ConfigField(
"backup_offload_enabled",
"Offload to remote storage",
@@ -82,6 +98,7 @@ class BackupService(JobService):
group="Offload",
)
self.config_fields += [
+ self.disk_warn_percent_field,
self.offload_enabled_field,
self.offload_interval_field,
self.offload_keep_local_field,
@@ -95,6 +112,36 @@ class BackupService(JobService):
except Exception as exc:
self.log(f"Schedule pass failed: {exc}")
await self._run_offload_cycle()
+ try:
+ self._check_disk_usage()
+ except Exception as exc:
+ self.log(f"Disk usage check failed: {exc}")
+
+ def _check_disk_usage(self) -> None:
+ threshold = int(self.disk_warn_percent_field.read())
+ used_percent = store.disk_usage()["used_percent"]
+ if used_percent >= threshold:
+ if not self._disk_warned:
+ self._disk_warned = True
+ self.log(
+ f"Disk usage critical: {used_percent}% used on the data "
+ f"volume (warning threshold {threshold}%)"
+ )
+ elif self._disk_warned:
+ self._disk_warned = False
+ self.log(f"Disk usage back under threshold: {used_percent}% used")
+
+ def collect_metrics(self) -> dict:
+ metrics = super().collect_metrics()
+ disk = store.disk_usage()
+ metrics["stats"] = [
+ *metrics.get("stats", []),
+ {
+ "label": "Disk usage",
+ "value": f"{disk['used_percent']}% ({disk['free_human']} free)",
+ },
+ ]
+ return metrics
async def _run_offload_cycle(self) -> None:
if not self.offload_enabled_field.read():
diff --git a/devplacepy/services/backup/store.py b/devplacepy/services/backup/store.py
index 27c2ddd..1bcee78 100644
--- a/devplacepy/services/backup/store.py
+++ b/devplacepy/services/backup/store.py
@@ -1,6 +1,9 @@
# retoor
+import os
import shutil
+import stat
+import threading
import time
from datetime import datetime, timezone
from pathlib import Path
@@ -33,9 +36,12 @@ STATUS_RUNNING = "running"
STATUS_DONE = "done"
STATUS_FAILED = "failed"
-STORAGE_CACHE_TTL_SECONDS = 30
+STORAGE_CACHE_TTL_SECONDS = 600
+DISK_CACHE_TTL_SECONDS = 15
_storage_cache: dict = {"at": 0.0, "data": None}
+_disk_cache: dict = {"at": 0.0, "data": None}
+_storage_lock = threading.Lock()
def now_iso() -> str:
@@ -395,20 +401,57 @@ def delete_schedule(uid: str, deleted_by: str) -> bool:
return True
+def clear_storage_stats_cache() -> None:
+ _storage_cache["data"] = None
+ _storage_cache["at"] = 0.0
+ _disk_cache["data"] = None
+ _disk_cache["at"] = 0.0
+
+
+def disk_usage() -> dict:
+ now = time.monotonic()
+ cached = _disk_cache["data"]
+ if cached is not None and (now - _disk_cache["at"]) < DISK_CACHE_TTL_SECONDS:
+ return cached
+ target = config.DATA_DIR
+ probe = target if target.exists() else target.parent
+ usage = shutil.disk_usage(str(probe if probe.exists() else Path("/")))
+ data = {
+ "total_bytes": usage.total,
+ "used_bytes": usage.used,
+ "free_bytes": usage.free,
+ "total_human": human_bytes(usage.total),
+ "used_human": human_bytes(usage.used),
+ "free_human": human_bytes(usage.free),
+ "used_percent": round(usage.used / usage.total * 100, 1) if usage.total else 0.0,
+ }
+ _disk_cache["data"] = data
+ _disk_cache["at"] = now
+ return data
+
+
def _path_size(path: Path) -> tuple[int, int]:
- if not path.exists():
+ try:
+ info = os.lstat(path)
+ except OSError:
+ return 0, 0
+ if stat.S_ISLNK(info.st_mode):
+ return 0, 0
+ if stat.S_ISREG(info.st_mode):
+ return info.st_size, 1
+ if not stat.S_ISDIR(info.st_mode):
return 0, 0
- if path.is_file():
- return path.stat().st_size, 1
total = 0
files = 0
- for entry in path.rglob("*"):
- try:
- if entry.is_file() and not entry.is_symlink():
- total += entry.stat().st_size
+ for root, _dirs, names in os.walk(path, followlinks=False):
+ for name in names:
+ try:
+ info = os.lstat(os.path.join(root, name))
+ except OSError:
+ continue
+ if stat.S_ISREG(info.st_mode):
+ total += info.st_size
files += 1
- except OSError:
- continue
return total, files
@@ -429,17 +472,57 @@ def _storage_paths() -> list[tuple[str, str, Path]]:
]
-def compute_storage_stats() -> dict:
- now = time.monotonic()
- if (
- _storage_cache["data"] is not None
- and (now - _storage_cache["at"]) < STORAGE_CACHE_TTL_SECONDS
- ):
- return _storage_cache["data"]
+def _inventory() -> tuple[list[dict], int, int]:
+ declared = _storage_paths()
+ sizes = {key: [0, 0] for key, _label, _path in declared}
+ dir_prefixes: list[tuple[str, str]] = []
+ file_exact: dict[str, str] = {}
+ data_root = os.path.realpath(config.DATA_DIR) if config.DATA_DIR.exists() else ""
+
+ for key, _label, path in declared:
+ try:
+ info = os.lstat(path)
+ except OSError:
+ continue
+ if stat.S_ISREG(info.st_mode):
+ file_exact[os.path.realpath(path)] = key
+ sizes[key] = [info.st_size, 1]
+ continue
+ if stat.S_ISDIR(info.st_mode):
+ real = os.path.realpath(path)
+ dir_prefixes.append((key, real.rstrip(os.sep) + os.sep))
+
+ data_size = 0
+ data_files = 0
+ if data_root and os.path.isdir(data_root):
+ for root, _dirs, names in os.walk(data_root, followlinks=False):
+ for name in names:
+ full = os.path.join(root, name)
+ try:
+ info = os.lstat(full)
+ except OSError:
+ continue
+ if not stat.S_ISREG(info.st_mode):
+ continue
+ size = info.st_size
+ data_size += size
+ data_files += 1
+ matched = file_exact.get(full)
+ if matched is not None:
+ sizes[matched] = [size, 1]
+ for key, prefix in dir_prefixes:
+ if full.startswith(prefix):
+ sizes[key][0] += size
+ sizes[key][1] += 1
+ else:
+ for key, _label, path in declared:
+ sizes[key] = list(_path_size(path))
+ data_size += sizes[key][0]
+ data_files += sizes[key][1]
paths = []
- for key, label, path in _storage_paths():
- size, files = _path_size(path)
+ for key, label, path in declared:
+ size, files = sizes[key]
paths.append(
{
"key": key,
@@ -451,41 +534,46 @@ def compute_storage_stats() -> dict:
"exists": path.exists(),
}
)
+ return paths, data_size, data_files
- data_size, data_files = _path_size(config.DATA_DIR)
- backups_size, backups_files = _path_size(config.BACKUPS_DIR)
- backup_count = len(
- [b for b in list_backups(limit=100000) if b.get("status") == STATUS_DONE]
- )
- usage = shutil.disk_usage(str(config.DATA_DIR))
- data = {
- "paths": paths,
- "data_dir": {
- "path": str(config.DATA_DIR),
- "size_bytes": data_size,
- "size_human": human_bytes(data_size),
- "file_count": data_files,
- },
- "backups_total": {
- "count": backup_count,
- "size_bytes": backups_size,
- "size_human": human_bytes(backups_size),
- "file_count": backups_files,
- },
- "disk": {
- "total_bytes": usage.total,
- "used_bytes": usage.used,
- "free_bytes": usage.free,
- "total_human": human_bytes(usage.total),
- "used_human": human_bytes(usage.used),
- "free_human": human_bytes(usage.free),
- "used_percent": round(usage.used / usage.total * 100, 1)
- if usage.total
- else 0.0,
- },
- "generated_at": now_iso(),
- }
- _storage_cache["data"] = data
- _storage_cache["at"] = now
- return data
+def compute_storage_stats() -> dict:
+ now = time.monotonic()
+ cached = _storage_cache["data"]
+ if cached is not None and (now - _storage_cache["at"]) < STORAGE_CACHE_TTL_SECONDS:
+ return cached
+ with _storage_lock:
+ now = time.monotonic()
+ cached = _storage_cache["data"]
+ if (
+ cached is not None
+ and (now - _storage_cache["at"]) < STORAGE_CACHE_TTL_SECONDS
+ ):
+ return cached
+ paths, data_size, data_files = _inventory()
+ backups_entry = next((row for row in paths if row["key"] == "backups"), None)
+ backups_size = backups_entry["size_bytes"] if backups_entry else 0
+ backups_files = backups_entry["file_count"] if backups_entry else 0
+ backup_count = len(
+ [b for b in list_backups(limit=100000) if b.get("status") == STATUS_DONE]
+ )
+ data = {
+ "paths": paths,
+ "data_dir": {
+ "path": str(config.DATA_DIR),
+ "size_bytes": data_size,
+ "size_human": human_bytes(data_size),
+ "file_count": data_files,
+ },
+ "backups_total": {
+ "count": backup_count,
+ "size_bytes": backups_size,
+ "size_human": human_bytes(backups_size),
+ "file_count": backups_files,
+ },
+ "disk": disk_usage(),
+ "generated_at": now_iso(),
+ }
+ _storage_cache["data"] = data
+ _storage_cache["at"] = time.monotonic()
+ return data
diff --git a/devplacepy/services/bot/CLAUDE.md b/devplacepy/services/bot/CLAUDE.md
index e233790..82004ad 100644
--- a/devplacepy/services/bot/CLAUDE.md
+++ b/devplacepy/services/bot/CLAUDE.md
@@ -15,7 +15,7 @@ This file documents the Playwright-driven AI persona fleet. Claude Code auto-loa
| `state.py` | `BotState` dataclass + JSON persistence (incl. the per-bot `identity` card) |
| `llm.py` | `LLMClient` (parameterized: key/url/model/costs); content generation + quality checks + the `decide`/`generate_identity` decision engine |
| `news_fetcher.py` | `NewsFetcher` - TTL-cached article source |
-| `browser.py` | `BotBrowser` - Playwright wrapper (human-like typing/clicking/scrolling) |
+| `browser.py` | `BotBrowser` - Playwright wrapper (human-like typing/clicking/scrolling). `fill` types character-by-character for text fields; date/time inputs (`type=date` and the other `NATIVE_FILL_TYPES`) use Playwright's native `locator.fill(value)` because Chromium date widgets ignore keystrokes and a typed `YYYY-MM-DD` never lands, which made every bot signup POST 400. |
| `registry.py` | `ArticleRegistry` - flock-based cross-bot article dedupe |
| `bot.py` | `DevPlaceBot` - the orchestrator (sessions, action cycle, `run_forever`) |
| `service.py` | `BotsService(BaseService)` - fleet manager + metrics |
diff --git a/devplacepy/services/bot/browser.py b/devplacepy/services/bot/browser.py
index 9ff9122..4509471 100644
--- a/devplacepy/services/bot/browser.py
+++ b/devplacepy/services/bot/browser.py
@@ -22,6 +22,13 @@ DEFAULT_USER_AGENT = (
"(KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36"
)
DEFAULT_VIEWPORT = {"width": 1280, "height": 900}
+NATIVE_FILL_TYPES = frozenset(
+ {"date", "datetime-local", "month", "time", "week"}
+)
+
+
+def uses_native_fill(input_type: str) -> bool:
+ return (input_type or "").lower() in NATIVE_FILL_TYPES
class BotBrowser:
@@ -157,11 +164,15 @@ class BotBrowser:
return False
await el.click(timeout=3000)
await self._idle(0.05, 0.15)
- await el.fill("")
- for ch in val:
- await self._page.keyboard.type(ch, delay=random.randint(20, 60))
- if ch == " ":
- await asyncio.sleep(random.uniform(0.02, 0.08))
+ input_type = (await el.get_attribute("type") or "").lower()
+ if uses_native_fill(input_type):
+ await el.fill(val)
+ else:
+ await el.fill("")
+ for ch in val:
+ await self._page.keyboard.type(ch, delay=random.randint(20, 60))
+ if ch == " ":
+ await asyncio.sleep(random.uniform(0.02, 0.08))
await self.capture("field input")
return True
except Exception as e:
diff --git a/devplacepy/services/containers/CLAUDE.md b/devplacepy/services/containers/CLAUDE.md
index eae177c..d779678 100644
--- a/devplacepy/services/containers/CLAUDE.md
+++ b/devplacepy/services/containers/CLAUDE.md
@@ -58,7 +58,7 @@ Every exec passes `-w /app` explicitly (`DockerCliBackend.exec` and the PTY exec
## HTTP routing surface
-- `/projects/{slug}/containers` - the `routers/projects/containers/` subpackage: `instances.py` (creation/lifecycle/exec/logs/metrics/sync plus the exec websocket), `schedules.py` (cron/interval/once schedules), shared helpers in `_shared.py`. Every instance runs the shared `ppy` image. Discoverable from the project detail page's admin-only **Containers** button (gated by `content.can_view_project_containers` via the `viewer_can_containers` context flag) and from the admin index.
+- `/projects/{slug}/containers` - the `routers/projects/containers/` subpackage: `instances.py` (creation/lifecycle/exec/logs/metrics/sync plus the exec websocket), `schedules.py` (cron/interval/once schedules), shared helpers in `_shared.py`. Every instance runs the shared `ppy` image. Reachable by direct URL and from the admin index; the project detail page's admin-only **Containers** button was removed (the `viewer_can_containers` context flag and `content.can_view_project_containers` gate remain).
- `/admin/containers` - `routers/admin/containers.py`: lists every instance across all projects with inline actions (start/stop/restart/terminal/edit/delete) plus a create modal (project search-select, run-as user search-select, boot language + source editor, restart policy, start-on-boot, env/ports/limits/ingress); `/admin/containers/{uid}` is the per-instance detail page (lifecycle, live logs/metrics, PTY terminal, schedules, ingress, sync, status history) and `/admin/containers/{uid}/edit` edits run-as user, boot language/script/command, restart policy, start-on-boot, and limits.
**Key reuse rule:** the admin Containers section adds NO lifecycle/logs/exec endpoints of its own - the instance carries its `project_uid`, so the admin detail route resolves the project and its frontend targets the existing `/projects/{slug}/containers/instances/{uid}/...` routes. Add any new instance operation to `routers/projects/containers/instances.py` only; the admin detail page picks it up for free.
@@ -67,7 +67,7 @@ Every exec passes `-w /app` explicitly (`DockerCliBackend.exec` and the PTY exec
Both are discoverable (an earlier version of the per-project page had zero links to it - fixed).
-1. **Per-project manager**: `templates/containers.html` + `static/js/ContainerManager.js` handles instance creation through an app modal form (the `_macros.html` `modal()` macro + `ModalManager` `.visible` toggle); its instance list links out to the shared detail page. Reached from the project detail page's Containers button, and passes breadcrumbs so content clears the fixed nav.
+1. **Per-project manager**: `templates/containers.html` + `static/js/ContainerManager.js` handles instance creation through an app modal form (the `_macros.html` `modal()` macro + `ModalManager` `.visible` toggle); its instance list links out to the shared detail page. Reachable by direct URL only now (the project detail page's Containers button was removed); it passes breadcrumbs so content clears the fixed nav.
2. **Admin Containers section**: `routers/admin/containers.py` (mounted `/admin/containers`, sidebar link in `admin_base.html`, `admin_section="containers"`). `GET /admin/containers` lists every instance via `store.all_instances()` (decorated with project title/slug from one `projects` lookup) in an `.admin-table`. `GET /admin/containers/data` is the poll JSON. `GET /admin/containers/{uid}` renders `templates/containers_instance.html` + `static/js/ContainerInstance.js` - a dedicated detail page (lifecycle, poll logs/metrics, schedules add/delete, ingress, sync, interactive exec over a PTY WebSocket gated on the lock owner).
All container CSS (`static/css/containers.css`) uses app design tokens (`--bg-card`, `--text-primary`, `--success`/`--danger`/`--warning`, `--radius`) and the shared `.card` recipe.
@@ -728,19 +728,19 @@ view is the only trigger, there is no `Forward a Port` command in the palette in
required field makes an anonymous request 422 instead of 401 and `tests/api/auth/matrix.py` fails.
Give the field a default and validate it inside the handler after `require_user`.
-**The editor opens through one shared partial.** The project detail page renders an inline **Editor**
-button in `.project-detail-actions` via `templates/_editor_open.html` (`target="_blank"`, plus the
-`data-editor-*` attributes `EditorLauncher` reads) straight to the code-server proxy
-`/projects/{slug}/containers/instances/{uid}/code/`, built by `_editor_url` in
-`routers/projects/index.py` and carried as `workspace_editor_url` on the context and
+**The editor opens through one shared partial.** `templates/_editor_open.html` renders the **Open
+editor** link (`target="_blank"`, plus the `data-editor-*` attributes `EditorLauncher` reads) straight
+to the code-server proxy `/projects/{slug}/containers/instances/{uid}/code/`, built by `_editor_url`
+in `routers/projects/index.py` and carried as `workspace_editor_url` on the context and
`ProjectDetailOut`. It is emitted ONLY when the viewer passes `can_open_workspace` AND
`provision.editor_ready(instance)` holds: the workspace exists, is not suspended, is
`store.ST_RUNNING`, AND the editor port answers a TCP connect (`api.editor_reachable`, the same
`tunnel_target` the proxy dials) - the states the `editor_proxy` route itself refuses (403 suspended,
409 not running, 502 no port) plus the boot window in which the container is up but code-server is
-not yet listening, so the button can never open a dead editor. When there is no ready workspace the
-button is absent and the Workspace menu item below is the way in (create/start it there). The
-workspace page's own **Open editor** link opens in a new tab too; keep both in step.
+not yet listening, so the link can never open a dead editor. The link is rendered by the workspace
+page's **ready** phase; the project detail page's inline **Editor** button and its overflow
+**Workspace** item were removed, so the workspace page is the only surface and is reachable by direct
+URL now.
**The workspace page renders a PHASE, and the phase is computed once, server-side.**
`provision.phase(instance, ready)` is a pure function of `suspended_at`, `desired_state`, `status`
@@ -772,16 +772,14 @@ the same `{workspace, editor_url}` shape the page JSON carries. Only the start/s
(`data-workspace-action`) go through the manager; tunnels, editor preferences and delete keep
their native page-reloading submit, which the e2e tests assert with `wait_for_url`.
-**Member entry point** is the project detail page's overflow menu (`project_detail.html`), gated by
-the `viewer_can_workspace` context flag (`can_open_workspace(project, user)`, set in
-`routers/projects/index.py` and declared on `ProjectDetailOut`) - exactly the pattern the admin-only
-**Containers** item uses with `viewer_can_containers`. `can_open_workspace` folds in the
-`workspace_enabled` master switch, so the item disappears for everyone while the feature is off and
-the route's own `_guard` stays the authority. **A workspace surface with no context flag is
-unreachable**: the whole feature shipped once with routes, Devii tools and docs but no link into
-`/projects/{slug}/workspace`, so it was reachable only by typing the URL - and then 404'd anyway
-because `workspace_enabled` defaults to `"0"`. Any new workspace surface needs both the flag on the
-page that links to it and the setting turned on.
+**Member entry point** was the project detail page's overflow menu (`project_detail.html`), gated by
+the `viewer_can_workspace` context flag (`can_open_workspace(project, user)`, still set in
+`routers/projects/index.py` and declared on `ProjectDetailOut`). That overflow **Workspace** item
+(and the admin-only **Containers** item, gated the same way by `viewer_can_containers`) was removed,
+so `/projects/{slug}/workspace` is reached by direct URL until the feature is fully retired.
+`can_open_workspace` folds in the `workspace_enabled` master switch, and the route's own `_guard`
+stays the authority. Re-adding any workspace link to the project page must gate it on the context
+flag again, with the setting turned on.
**Admin console** is `/admin/workspaces` (`routers/admin/workspaces.py`, `admin_workspaces.html`):
list, start/stop, suspend/unsuspend with a required reason, raise/resolve/dismiss flags, per-user
diff --git a/devplacepy/services/correction.py b/devplacepy/services/correction.py
index 5ee0031..81b9d90 100644
--- a/devplacepy/services/correction.py
+++ b/devplacepy/services/correction.py
@@ -13,7 +13,13 @@ from devplacepy.config import (
INTERNAL_GATEWAY_URL,
INTERNAL_MODEL,
)
-from devplacepy.database import add_correction_usage, get_setting, get_table
+from devplacepy.database import (
+ add_correction_usage,
+ get_setting,
+ get_table,
+ internal_gateway_key,
+)
+from devplacepy.rendering import markdown_structure_signature
from devplacepy.services.background import background
from devplacepy.services.openai_gateway.usage import parse_usage_headers
@@ -31,6 +37,7 @@ CORRECTABLE_FIELDS: dict[str, tuple[str, ...]] = {
CORRECTION_TIMEOUT_SECONDS = 20.0
MAX_GROWTH_FACTOR = 3
+STRUCTURE_DROP_TOLERANCE = 1
PENDING_SCOPE_KEY = "devplace_pending_corrections"
AI_APPLY_EXECUTOR = ThreadPoolExecutor(max_workers=4, thread_name_prefix="ai-apply")
@@ -73,6 +80,18 @@ def _usage_from_headers(response_headers) -> dict:
}
+def structure_diverges(original: str, corrected: str) -> bool:
+ baseline = markdown_structure_signature(original)
+ candidate = markdown_structure_signature(corrected)
+ for key, before in baseline.items():
+ after = candidate.get(key, 0)
+ if before > 0 and after == 0:
+ return True
+ if after < before - STRUCTURE_DROP_TOLERANCE:
+ return True
+ return False
+
+
def gateway_complete(
api_key: str,
system: str,
@@ -80,6 +99,8 @@ def gateway_complete(
timeout: float,
max_growth_factor: int | None = None,
model: str = INTERNAL_MODEL,
+ bypass_preamble: bool = False,
+ check_structure: bool = False,
) -> tuple[str, dict | None]:
text = text or ""
if not text.strip():
@@ -92,12 +113,18 @@ def gateway_complete(
],
"temperature": 0.1,
}
+ if bypass_preamble:
+ payload["bypass_preamble"] = True
headers = {
"Content-Type": "application/json",
"X-App-Reference": "devplace-correction-v-1-0-0",
}
if api_key:
headers["Authorization"] = f"Bearer {api_key}"
+ if bypass_preamble:
+ internal_key = internal_gateway_key()
+ if internal_key:
+ headers["X-Gateway-Internal-Key"] = internal_key
try:
response = _client().post(
INTERNAL_GATEWAY_URL, json=payload, headers=headers, timeout=timeout
@@ -119,6 +146,9 @@ def gateway_complete(
if max_growth_factor and len(content) > len(text) * max_growth_factor + 200:
logger.warning("AI gateway output too large, keeping original")
return text, usage
+ if check_structure and structure_diverges(text, content):
+ logger.warning("AI gateway output changed markdown structure, keeping original")
+ return text, usage
return content, usage
@@ -141,6 +171,8 @@ def correct_text(api_key: str, prompt: str, text: str) -> tuple[str, dict | None
CORRECTION_TIMEOUT_SECONDS,
MAX_GROWTH_FACTOR,
model=correction_model(),
+ bypass_preamble=True,
+ check_structure=True,
)
diff --git a/devplacepy/services/devii/CLAUDE.md b/devplacepy/services/devii/CLAUDE.md
index f7629eb..137802c 100644
--- a/devplacepy/services/devii/CLAUDE.md
+++ b/devplacepy/services/devii/CLAUDE.md
@@ -96,6 +96,8 @@ A task created as a reminder carries `notify=1`: when it finishes, `session._del
**Quotas are resettable** (clearing the owner's `devii_usage_ledger` rows): per-user via `POST /admin/users/{uid}/reset-ai-quota` (button on the admin Users page), globally via `POST /admin/ai-quota/reset-guests` and `/reset-all` (buttons on `/admin/ai-usage`), and from the CLI via `devplace devii reset-quota | --guests | --all`.
+**Proactive 80% quota warning.** `DeviiService.maybe_warn_quota_threshold(owner_kind, owner_id, is_admin)` is called right after the interactive quota gate passes (`routers/devii.py`'s `/devii/ws` handler and `services/telegram/bridge.py`'s `_run_turn`, the two turn-spawning chokepoints), so a signed-in user is warned before they hit the 100% block instead of only discovering it once blocked. It is a no-op for guests (no notifications inbox to deliver to), for administrators, and for an unlimited (`0`) cap; otherwise it fires the `ai_quota_warning` notification (via the single `create_notification` funnel, `NOTIFICATION_TYPES`, toggleable like any other type) once the owner's rolling `spent_24h` reaches 80% of their `daily_limit_for`. Dedup is a plain lookback query on the `notifications` table (`type="ai_quota_warning" AND created_at >= now-24h`) rather than a new table/column - a fresh warning can fire again only once the prior one ages out of the rolling 24h window. The message states only proximity, never a dollar figure, matching the "financial data is admin-only" rule above.
+
**Financial data is admin-only:** any monetary figure (USD cost, pricing, spend, limit) is restricted to administrators; members and guests see only the **percentage** of their 24h quota used. The owner's admin status is resolved server-side (`is_admin(user)`) and threaded WS -> `hub.get_or_create(is_admin=)` -> `DeviiSession(is_admin=)` -> `Dispatcher(is_admin=)`. The `Action` dataclass has a `requires_admin` flag (`cost_stats` is admin-only USD; the member-safe `usage_quota` tool returns **only** `{used_pct, turns_today, limit_reached}` with no money and is available to everyone). Gating is double: `Catalog.tool_schemas_for(authenticated, is_admin)` never hands an admin-only tool's schema to a non-admin, and the dispatcher independently raises `AuthRequiredError` for any `requires_admin` action a non-admin attempts. `usage_quota`'s data comes from `DeviiSession._quota_snapshot` (ledger `spent_24h`/`turns_24h` over `settings.daily_limit_usd`); for non-admin owners the system prompt also appends a hard rule forbidding any cost disclosure (defense in depth). `GET /devii/usage` mirrors this: it always returns `used_pct`/`turns_today` and adds `spent_24h`/`limit` only for admins. The standalone `devii` CLI runs with `is_admin=True` (the local operator owns the process). The catalog's cost/analytics HTTP tools `ai_usage` (GET `/admin/ai-usage/data`, USD breakdown) and `site_analytics` (GET `/admin/analytics`) are also `requires_admin=True`, so a non-admin session is never offered them and the dispatcher blocks them even if named - the platform 403 is no longer the only guard. The profile page is correctly split too (`_ai_quota(include_cost=viewer_is_admin)` emits dollars only for admins, in both its HTML and JSON forms, so a member fetching their own profile with `Accept: application/json` never sees dollars either).
## Aggregate analytics (no pagination)
diff --git a/devplacepy/services/devii/actions/catalog/engagement.py b/devplacepy/services/devii/actions/catalog/engagement.py
index ea1528e..d6572a5 100644
--- a/devplacepy/services/devii/actions/catalog/engagement.py
+++ b/devplacepy/services/devii/actions/catalog/engagement.py
@@ -22,7 +22,7 @@ ENGAGEMENT_ACTIONS: tuple[Action, ...] = (
),
body(
"value",
- "Vote value: 1 to upvote, -1 to downvote (re-send to remove).",
+ "Vote value: 1 to upvote, -1 to downvote (re-send to remove), 0 to explicitly retract your vote.",
required=True,
),
),
@@ -65,6 +65,45 @@ ENGAGEMENT_ACTIONS: tuple[Action, ...] = (
),
),
),
+ Action(
+ name="list_notes",
+ method="GET",
+ path="/notes/saved",
+ summary="List your personal notes",
+ description="Notes are private annotations only you can ever see.",
+ params=(query("before", "Pagination cursor."),),
+ ),
+ Action(
+ name="set_note",
+ method="POST",
+ path="/notes/{target_type}/{target_uid}",
+ summary="Add or update a private note on a target",
+ description="Overwrites any existing note on the target. Only you can ever see this note.",
+ ajax=True,
+ params=(
+ path("target_type", TARGET_TYPE),
+ path(
+ "target_uid",
+ "Uid of the target, copied from a listing response; do not invent it.",
+ ),
+ body("content", "Note text, up to 4000 characters.", required=True),
+ ),
+ ),
+ Action(
+ name="delete_note",
+ method="POST",
+ path="/notes/{target_type}/{target_uid}/delete",
+ summary="Delete your private note from a target",
+ description="Returns {deleted: true}.",
+ ajax=True,
+ params=(
+ path("target_type", TARGET_TYPE),
+ path(
+ "target_uid",
+ "Uid of the target, copied from a listing response; do not invent it.",
+ ),
+ ),
+ ),
Action(
name="vote_poll",
method="POST",
diff --git a/devplacepy/services/devii/actions/catalog/tools.py b/devplacepy/services/devii/actions/catalog/tools.py
index 2109149..953d5d7 100644
--- a/devplacepy/services/devii/actions/catalog/tools.py
+++ b/devplacepy/services/devii/actions/catalog/tools.py
@@ -3,7 +3,7 @@
from __future__ import annotations
from ..spec import Action
-from ._shared import body, path
+from ._shared import body, path, query
TOOLS_ACTIONS: tuple[Action, ...] = (
@@ -108,6 +108,19 @@ TOOLS_ACTIONS: tuple[Action, ...] = (
params=(path("uid", "DeepSearch job uid returned by deepsearch."),),
requires_auth=False,
),
+ Action(
+ name="deepsearch_history",
+ method="GET",
+ path="/tools/deepsearch/history",
+ summary="List the user's past DeepSearch research runs",
+ description=(
+ "Returns the signed-in user's (or guest's) DeepSearch history, newest first, each "
+ "with its query, status, score and a reopen_url. A completed session can be reopened "
+ "with deepsearch_session and its grounded chat continued exactly as when it finished."
+ ),
+ params=(query("limit", "Maximum sessions to return (1-100, default 20)."),),
+ requires_auth=False,
+ ),
Action(
name="deepsearch_pause",
method="POST",
diff --git a/devplacepy/services/devii/registry.py b/devplacepy/services/devii/registry.py
index e943f45..ccdd372 100644
--- a/devplacepy/services/devii/registry.py
+++ b/devplacepy/services/devii/registry.py
@@ -60,7 +60,7 @@ GROUP_LABELS: dict[str, str] = {
"profile": "Profile",
"messages": "Direct Messages",
"notifications": "Notifications (HTTP)",
- "engagement": "Reactions, Bookmarks, Polls, Follow",
+ "engagement": "Reactions, Bookmarks, Notes, Polls, Follow",
"social": "Leaderboard and Social",
"issues": "Issue Tracker",
"gists": "Gists",
diff --git a/devplacepy/services/devii/service.py b/devplacepy/services/devii/service.py
index dfd24f6..541bbe2 100644
--- a/devplacepy/services/devii/service.py
+++ b/devplacepy/services/devii/service.py
@@ -21,6 +21,9 @@ logger = logging.getLogger("devii.service")
INSTANCE_ORIGIN_DEFAULT = f"http://127.0.0.1:{PORT}"
+QUOTA_WARNING_RATIO = 0.8
+QUOTA_WARNING_NOTIFICATION_TYPE = "ai_quota_warning"
+
class DeviiService(BaseService):
default_enabled = False
@@ -421,6 +424,38 @@ class DeviiService(BaseService):
limit = self.daily_limit_for(owner_kind, is_admin)
return limit > 0 and self.spent_24h(owner_kind, owner_id) >= limit
+ def maybe_warn_quota_threshold(
+ self, owner_kind: str, owner_id: str, is_admin: bool = False
+ ) -> None:
+ if owner_kind != "user" or is_admin or not owner_id:
+ return
+ limit = self.daily_limit_for(owner_kind, is_admin)
+ if limit <= 0:
+ return
+ spent = self.spent_24h(owner_kind, owner_id)
+ if spent < limit * QUOTA_WARNING_RATIO:
+ return
+ from datetime import datetime, timedelta, timezone
+
+ from devplacepy.database import get_table
+ from devplacepy.utils import create_notification
+
+ cutoff = (datetime.now(timezone.utc) - timedelta(hours=24)).isoformat()
+ already_warned = get_table("notifications").find_one(
+ user_uid=owner_id,
+ type=QUOTA_WARNING_NOTIFICATION_TYPE,
+ created_at={">=": cutoff},
+ )
+ if already_warned:
+ return
+ create_notification(
+ owner_id,
+ QUOTA_WARNING_NOTIFICATION_TYPE,
+ "You are approaching your daily AI usage limit.",
+ owner_id,
+ "/devii",
+ )
+
def reset_quota(self, owner_kind: str, owner_id: str) -> int:
return self.hub().ledger.reset(owner_kind, owner_id)
diff --git a/devplacepy/services/jobs/CLAUDE.md b/devplacepy/services/jobs/CLAUDE.md
index a948a74..4e0a9dd 100644
--- a/devplacepy/services/jobs/CLAUDE.md
+++ b/devplacepy/services/jobs/CLAUDE.md
@@ -49,9 +49,9 @@ Forking copies a source project into a brand-new project owned by the forking us
## SEO Diagnostics tool - SeoService (kind `seo`, `services/jobs/seo/`, `routers/tools/`)
-The public **Tools -> SEO Diagnostics** auditor crawls a URL or sitemap with a headless browser and runs a broad battery of SEO checks, on the **same async-job pattern as zip/fork** plus a live websocket. It is **public (guests included)**; abuse is bounded by the per-IP POST rate limit, a per-owner one-active-job cap, a page cap, and the shared SSRF guard.
+The public **SEO Diagnostics** auditor crawls a URL or sitemap with a headless browser and runs a broad battery of SEO checks, on the **same async-job pattern as zip/fork** plus a live websocket. It is **public (guests included)**; abuse is bounded by the per-IP POST rate limit, a per-owner one-active-job cap, a page cap, and the shared SSRF guard.
-- **Surface:** a collapsible **Tools** dropdown in `base.html` (desktop center nav + a mobile section, visible to everyone) toggled by `MobileNav.initToolsDropdown`. `GET /tools` lists tools; `GET /tools/seo` is the auditor page (`static/js/SeoDiagnostics.js` -> `app.seoDiagnostics`, instantiated page-side in the template, not in `Application.js`).
+- **Surface:** reachable by direct URL (`GET /tools` lists tools; `GET /tools/seo` is the auditor page, `static/js/SeoDiagnostics.js` -> `app.seoDiagnostics`, instantiated page-side in the template, not in `Application.js`). The former topnav **Tools** dropdown was removed, so there is currently no navigation entry point.
- **Enqueue:** `POST /tools/seo/run` (`routers/tools/seo.py`, body `SeoRunForm{url, mode: url|sitemap, max_pages 1-50}`). Owner is `("user", uid)` or `("guest", X-Real-IP)`. It rejects with `429` if the owner already has a pending/running `seo` job, then enqueues `{url, mode, max_pages, allow_private:False}` and returns `{uid, status_url, ws_url}`.
- **`process`** writes the payload to `config.SEO_REPORTS_DIR/{uid}/payload.json`, launches `python -m devplacepy.services.jobs.seo.worker ` via `create_subprocess_exec` (high `limit=` so big lines never overflow the StreamReader), reads **NDJSON frames from stdout** line by line (stage/target/progress/page/site_checks/report_ready), forwards each into the in-process **`ProgressHub`** (`services/jobs/seo/progress.py`, uid -> set of `asyncio.Queue`), and on completion loads `output_dir/report.json` as the job result. `cleanup()` clears the hub buffer and removes the report dir.
- **Worker** (`worker.py`, subprocess): `crawler.crawl_target` resolves the target (single URL, or sitemap `` URLs capped at `max_pages`) and fetches `robots.txt`/`sitemap.xml`/`llms.txt` with `httpx`. The audited target host is guarded once with `net_guard.guard_public_url`; candidate URLs **sharing that host are pre-approved** (no redundant per-URL `getaddrinfo` - a transient DNS failure or a self-hosted server resolving its own domain must not blank the whole crawl), and only cross-host sitemap entries are re-guarded. **In sitemap mode the crawler never falls back to auditing the sitemap document itself**: if no page URLs survive it raises a clear error (a stray `or [target]` fallback previously rendered the sitemap XML as one 56k-node "page" with no title/H1). For each page it launches one Playwright navigation: a single `page.evaluate(EXTRACT_SCRIPT)` returns the whole DOM contract (title/metas/canonical/headings/images/links/jsonld/og/twitter/semantic/mixed-content/word-count), an injected `PerformanceObserver` (`add_init_script(INIT_SCRIPT)`) captures LCP/CLS, navigation timing gives TTFB/FCP/transfer/protocol, a mobile-viewport pass measures overflow/tap-targets, a screenshot is saved, and a raw `httpx` GET supplies the SSR HTML for the rendered-vs-server parity check.
@@ -77,7 +77,7 @@ The public **Tools -> SEO Diagnostics** auditor crawls a URL or sitemap with a h
## DeepSearch tool - DeepsearchService (kind `deepsearch`, `services/jobs/deepsearch/`, `services/deepsearch/`, `routers/tools/deepsearch.py`)
-The public **Tools -> DeepSearch** researcher is a multi-agent deep web researcher built on the **same async-job + ProgressHub + 4013-WS pattern as the SEO tool**, plus a per-session vector store and a grounded RAG chat. It is **public (guests included)**; abuse is bounded by the per-IP POST rate limit, a per-owner one-active-job cap, a page cap (1-30), depth cap (1-4), and the shared SSRF guard. Reuse the SEO tool as the template for any new Tools async job.
+The public **DeepSearch** researcher is a multi-agent deep web researcher built on the **same async-job + ProgressHub + 4013-WS pattern as the SEO tool**, plus a per-session vector store and a grounded RAG chat. It is **public (guests included)**; abuse is bounded by the per-IP POST rate limit, a per-owner one-active-job cap, a page cap (1-30), depth cap (1-4), and the shared SSRF guard. Reuse the SEO tool as the template for any new Tools async job.
- **Owner helper is shared:** `routers/tools/_shared.py` `owner_for(request)` returns `("user", uid)` or `("guest", X-Real-IP)`; both `seo.py` and `deepsearch.py` import it (do not re-inline the owner derivation).
- **Enqueue:** `POST /tools/deepsearch/run` (body `DeepsearchRunForm{query, depth 1-4, max_pages 1-30}`). It rejects with `429` if the owner already has a pending/running `deepsearch` job. It resolves the **logged-in user's `users.api_key`** (guests use `database.internal_gateway_key()`) into the job payload for per-user embedding/LLM spend attribution, generates the uid up front, writes a `deepsearch_sessions` row (`create_deepsearch_session`), enqueues the job carrying `{query, depth, max_pages, api_key, collection}`, and returns `{uid, status_url, ws_url}`. The enqueue uses a local `_enqueue` (not `queue.enqueue`) so the session uid and the job uid match.
@@ -92,6 +92,7 @@ The public **Tools -> DeepSearch** researcher is a multi-agent deep web research
- **Vector store (`services/deepsearch/store.py`):** `VectorStore` wraps `chromadb.PersistentClient(path=config.DEEPSEARCH_CHROMA_DIR)`, one collection per session (`ds_`). `Chunk` is the dataclass. `hybrid_search` blends cosine vector similarity with a BM25 keyword score (weights `HYBRID_VECTOR_WEIGHT`/`HYBRID_KEYWORD_WEIGHT`) over the candidate set, with optional metadata `where` filters. `embeddings.py` `embed_texts` calls the gateway embeddings endpoint and **falls back to a deterministic local hashing vector** on any failure (so the tool degrades, never breaks).
- **RAG chat (`services/deepsearch/chat.py` + `WS /tools/deepsearch/{uid}/chat`):** a dedicated lightweight loop (NOT the Devii hub), served **only by the service-lock owner** (closes `4013` for fast retry). Answers are grounded ONLY in the session collection via `hybrid_search`, cited inline, rendered client-side via `dp-content`. Turns persist to `deepsearch_messages` and audit `deepsearch.chat`. Frontend component `` (`static/js/components/AppDeepsearchChat.js`) clones `AppDocsChat`'s framing but uses its own WebSocket to the chat path.
- **Status/report/export routes:** `GET /tools/deepsearch/{uid}` (`DeepsearchJobOut`), `GET /tools/deepsearch/{uid}/session` (`respond(..., DeepsearchSessionOut)`, HTML or JSON), `GET /tools/deepsearch/{uid}/export.{md,json,pdf}` (`services/deepsearch/export.py`; PDF via weasyprint). All are **capability URLs** scoped by the unguessable uuid7. **Viewer-flag discipline:** the session schema/context use `viewer_is_admin`/`viewer_owns` (never `is_admin`/`owns`) so a `respond()` context key never shadows a Jinja global (the same class of issue as the issues `/{number}` route). `tests/api/tools/deepsearch/session.py` guards the HTML render.
+- **History and reopen (`GET /tools/deepsearch/history`).** `deepsearch_sessions` already persists every run's query/status/score/confidence/summary/timestamps independently of the `jobs` table row (unlike the disposable collection + report dir), so the only genuinely missing piece was a listing surface, not new storage. `database.list_deepsearch_sessions(owner_kind, owner_id, limit)` (index `idx_deepsearch_sessions_owner_created` on `(owner_kind, owner_id, created_at)`) backs the owner-scoped, newest-first history via the same `routers/tools/_shared.py` `owner_for(request)` every other DeepSearch route uses - no separate guest-cookie identity, matching the run/status/control routes' IP-based guest scoping. Each item carries a `reopen_url` (`/tools/deepsearch/{uid}/session`), `chat_available` (session status is `done` AND the job row still exists), and `available` (the job row has not yet been swept by the `deepsearch` job kind's 7-day retention - past that point the collection and report are gone even though the `deepsearch_sessions` row survives, so the history item shows "Expired" instead of a dead link). **Reopening a completed session's chat needed no backend change**: the `WS /tools/deepsearch/{uid}/chat` guard already accepts `job.status == queue.DONE OR session.status == "done"`, so a session reached fresh from the history list (not from the just-finished live-progress flow) renders `chat_ws_url` on `GET .../session` exactly the same way. Schema `DeepsearchHistoryOut`/`DeepsearchHistoryItemOut`; Devii tool `deepsearch_history` (public, mirrors `isslop_list`); docs `tools-deepsearch-history`.
- **Completion race (load-bearing read-path fix).** The worker writes `report.json` to disk and `service.process` publishes the `done` frame **from inside `process()`**, but the `JobService` framework only commits `jobs.result`/`status=DONE` afterwards, in `_reap()` -> `_finish_done()` on a later tick. The frontend navigates to the session page the instant it receives `done`, so a read that keyed only off `jobs.status == DONE` returned an EMPTY report (`None` score, 0 sources) until a manual refresh. Fix: `_report_for(uid, job)` returns `job.result.report` when the job is `DONE` and non-empty, else falls back to the on-disk `report.json` (`_report_from_disk`, `DEEPSEARCH_DIR/{uid}/report.json`) - which exists before the `done` frame is ever sent - and returns `{}` only for a `FAILED` job or a genuinely still-running job with no report on disk. `_session_context` derives `done`/`status` from `bool(report)` (not raw job status), and the chat WS gate accepts `session.status == "done"` (set inside `process()` before the publish) as ready. `_export_report` reuses the same fallback. Regression: `tests/api/tools/deepsearch/session.py::test_session_reads_disk_report_before_result_commit`. **Any new read of a job result that a client reaches immediately after a `done`/`session_url` frame must use this same on-disk fallback, never bare `jobs.status`.**
- **Clickable inline citations (`services/deepsearch/citations.py`).** The report/findings carry `[n]` markers (and the model sometimes emits `[3][9][1-2]`); the `link_citations(html, source_count)` template global (registered in `templating.py`) rewrites each `[n]` and each `[a-b]` range into `[n]` anchors that jump to the numbered `` in the Sources list (source numbering is page order, matching the `[n]` the summarizer was given). It splits out ``/``/`` regions first so markers inside links/code are left alone, expands ranges to individual links, and drops out-of-range numbers (no broken anchors). The session template nests it over the server render: `{{ link_citations(render_content(summary), sources|length) }}` and `{{ link_citations(finding.detail|e, sources|length) }}`, plus a per-finding `.ds-finding-cites` chip row from `finding.citations`. `.ds-cite`/`.ds-sources li:target` styling lives in `deepsearch.css`. The report prompt asks for one number per bracket (never a range) so output is consistent, but the linkifier handles ranges regardless. Regression: `tests/unit/services/deepsearch/citations.py`.
- **Tables (`deepsearch_sessions`, `deepsearch_messages` soft-deletable + in `SOFT_DELETE_TABLES`; `deepsearch_url_cache` GC-only):** columns are ensured in `init_db()` (every queried column) with indexes. Every insert writes `deleted_at:None/deleted_by:None`; every read filters `deleted_at IS NULL`.
@@ -105,7 +106,7 @@ The public **Tools -> DeepSearch** researcher is a multi-agent deep web research
## AI Usage Analyzer tool - IsslopService (kind `isslop`, `services/jobs/isslop/`, `routers/tools/isslop.py`)
-The public **Tools -> AI Usage Analyzer** classifies a git repository or website as AI slop, sophisticated AI-assisted work or genuine human work. It is built on the standard async-job pattern (a `JobService` running a subprocess worker), but its live channel is **pub/sub, not a dedicated WS route**: every worker event is published to `public.isslop.{uid}` AND persisted to `isslop_events`, and the frontend pairs the pub/sub subscription with an incremental `GET /tools/isslop/{uid}/events?after=SEQ` poll, so guests (who cannot subscribe to `public.*` unless `pubsub_allow_guests` is on) and reconnecting tabs replay from the durable trail. **Never rely on pub/sub alone for this tool: the DB event trail is the source of truth, pub/sub is the fast path.**
+The public **AI Usage Analyzer** classifies a git repository or website as AI slop, sophisticated AI-assisted work or genuine human work. It is built on the standard async-job pattern (a `JobService` running a subprocess worker), but its live channel is **pub/sub, not a dedicated WS route**: every worker event is published to `public.isslop.{uid}` AND persisted to `isslop_events`, and the frontend pairs the pub/sub subscription with an incremental `GET /tools/isslop/{uid}/events?after=SEQ` poll, so guests (who cannot subscribe to `public.*` unless `pubsub_allow_guests` is on) and reconnecting tabs replay from the durable trail. **Never rely on pub/sub alone for this tool: the DB event trail is the source of truth, pub/sub is the fast path.**
- **Engine layout:** `services/jobs/isslop/` holds `acquisition/` (git probe via `git ls-remote`, depth-1 clone with size preflight + live 3 GB kill guard, stealth Playwright website crawler with HTTP fallback, path-traversal-safe workspace helpers), `analysis/` (exclusion rules, stylometric metrics, language detection, per-repo baselines, `signals/` with one detector family per file, two-axis scoring), `agent/` (gateway LLM client, per-file classifier, vision reviewer, report writer with deterministic fallback), plus `pipeline.py` (the event-yielding run), `worker.py` (subprocess entry), `events.py` (frame protocol), `persistence.py` (`EventPersister` writes events/file results/image results/report and stamps the analysis row), `store.py` (all DB access), `badge.py` (SVG), `service.py` (`IsslopService`), `config.py` (all constants + `WorkerSettings`).
- **Worker contract:** `IsslopService.process` writes the worker payload (url + admin toggles + gateway endpoint/model/key) to `config.ISSLOP_RUNS_DIR/{uid}/payload.json`, resolves the workspace under `config.ISSLOP_WORKSPACES_DIR` (`workspace_for` rejects any path escaping the root), launches `python -m devplacepy.services.jobs.isslop.worker `, and relays each NDJSON stdout line through `EventPersister.apply` (SQLite) then `pubsub.publish`. The workspace and run dir are removed in a `finally`; the pipeline also deletes the workspace itself as its final act, so **no acquired source survives an analysis** - only the report and its evidence rows.
diff --git a/devplacepy/services/jobs/seo_meta_service.py b/devplacepy/services/jobs/seo_meta_service.py
index 358e72e..0674227 100644
--- a/devplacepy/services/jobs/seo_meta_service.py
+++ b/devplacepy/services/jobs/seo_meta_service.py
@@ -173,6 +173,7 @@ class SeoMetaService(JobService):
source_text,
GENERATION_TIMEOUT_SECONDS,
model=model,
+ bypass_preamble=True,
)
if usage:
for key in totals:
diff --git a/devplacepy/services/live_view_relay.py b/devplacepy/services/live_view_relay.py
index d1e302d..f4d2602 100644
--- a/devplacepy/services/live_view_relay.py
+++ b/devplacepy/services/live_view_relay.py
@@ -113,7 +113,7 @@ async def _ai_usage(match: re.Match) -> dict:
async def _backups(_match: re.Match) -> dict:
from devplacepy.routers.admin.backups import _dashboard
- return _dashboard(can_download=False)
+ return await _dashboard(can_download=False)
async def _workspace_detail(match: re.Match) -> Optional[dict]:
diff --git a/devplacepy/services/messaging/CLAUDE.md b/devplacepy/services/messaging/CLAUDE.md
index c165f7a..7dff88d 100644
--- a/devplacepy/services/messaging/CLAUDE.md
+++ b/devplacepy/services/messaging/CLAUDE.md
@@ -45,7 +45,13 @@ Because the WS accepts on every worker, a message persisted on worker A must sti
## DRY persist choke point
-`services/messaging/persist.py` `persist_message(sender, receiver_uid, content, attachment_uids, *, request=None, origin)` is the ONE function that inserts the row, links attachments, fires `create_notification` + `clear_messages_cache` + `create_mention_notifications`, logs, and writes the `message.send` audit event. Both the WS `send` handler and the HTTP `POST /messages/send` handler call it, so audit/notification/mention behavior is byte-identical on both paths (the Devii `send_message` action is `handler="http"` -> `POST /messages/send`, so it also flows through here and broadcasts live). Content is capped at 2000 chars server-side; `content` itself is allowed to be empty (`MessageForm.content` is `min_length=0`) as long as at least one attachment is present - `persist_message` is the single source of truth for that rule (`if not content and not attachment_uids: return None`), so the HTTP form model deliberately does not duplicate it. Whenever `request` is not `None` it audits via `audit.record(request, ...)` - this covers BOTH the HTTP `Request` and the WS path, since the WS handler passes `request=websocket` and a `WebSocket` object is just as non-`None` as a `Request` (`audit.record` never reads HTTP-specific attributes off it beyond what's already supplied explicitly via `user=sender`). `audit.record_system(..., actor_kind="user", origin=origin)` is the fallback used ONLY when `persist_message` is called with no request/websocket context at all (e.g. a future internal/system-originated send) - same `message.send` key/category either way, no new event invented; typing/read are ephemeral and NOT audited.
+`services/messaging/persist.py` `persist_message(sender, receiver_uid, content, attachment_uids, *, request=None, origin, client_id=None)` is the ONE function that inserts the row, links attachments, fires `create_notification` + `clear_messages_cache` + `create_mention_notifications`, logs, and writes the `message.send` audit event. Both the WS `send` handler and the HTTP `POST /messages/send` handler call it, so audit/notification/mention behavior is byte-identical on both paths (the Devii `send_message` action is `handler="http"` -> `POST /messages/send`, so it also flows through here and broadcasts live). Content is capped at 2000 chars server-side; `content` itself is allowed to be empty (`MessageForm.content` is `min_length=0`) as long as at least one attachment is present - `persist_message` is the single source of truth for that rule (`if not content and not attachment_uids: return None`), so the HTTP form model deliberately does not duplicate it. Whenever `request` is not `None` it audits via `audit.record(request, ...)` - this covers BOTH the HTTP `Request` and the WS path, since the WS handler passes `request=websocket` and a `WebSocket` object is just as non-`None` as a `Request` (`audit.record` never reads HTTP-specific attributes off it beyond what's already supplied explicitly via `user=sender`). `audit.record_system(..., actor_kind="user", origin=origin)` is the fallback used ONLY when `persist_message` is called with no request/websocket context at all (e.g. a future internal/system-originated send) - same `message.send` key/category either way, no new event invented; typing/read are ephemeral and NOT audited.
+
+### Send dedupe (double-submit safety)
+
+`persist_message` dedupes on `(sender_uid, client_id)` **before** it ever inserts a row: `_find_recent_duplicate(sender_uid, client_id)` looks up the most recent `messages` row with the same `sender_uid` and the same client-supplied `client_id` whose `created_at` falls inside `DEDUPE_WINDOW_SECONDS` (30s). A hit short-circuits `persist_message` and returns that ALREADY-persisted row verbatim (no error, no second insert, no second AI correction/notification/audit) - both the WS `send` handler and `POST /messages/send` now pass their `client_id` straight through, so a network-level retry that resends the identical payload (same `client_id`) collapses onto the original row instead of producing a second, independently-AI-corrected message. `client_id` is attacker-controlled but the dedupe key is always scoped by the server-resolved `sender_uid`, so a client can only dedupe its own sends. The column is nullable (`messages.client_id`, ensured in `init_db()`, indexed by `idx_messages_dedupe (sender_uid, client_id)`); a request with no `client_id` (any future non-`dp-chat` caller) skips the check entirely and always inserts. This is defense-in-depth alongside the client-side send-lock below - the lock stops a human double-click from ever firing two requests, the server dedupe stops a retried *identical* request (same `client_id`) from ever becoming two rows.
+
+**Client-side send-lock (`AppChat.js`).** `_initComposer`'s submit handler and the Enter-to-send path both funnel through one `form submit` listener, which now bails out while `this._sendLocked` is true. `_sendViaSocket` sets the lock (and disables the send button via the existing `_refreshSendButton()`, reusing the `.is-sending` spinner state) for every genuine new send (never for a `.failed`-bubble retry, tracked separately so a retry never blocks a fresh message) and only releases it once that specific `client_id` clears out of `_pendingSends` - on the echoed reconcile (`_reconcilePending` -> `_clearPendingSend`) or the 20s send timeout (`_failPendingSend`). Multiple sends never race the lock: `_lockingSends` is a `Set` of in-flight new-send `client_id`s, and the composer re-enables only when it is empty.
## AI correction and AI modifier apply to direct messages, with LIVE delivery of the final content
@@ -53,7 +59,16 @@ Because the WS accepts on every worker, a message persisted on worker A must sti
## WS protocol frames
-Client -> server: `{type:"send", receiver_uid, content, attachment_uids?, client_id}`, `{type:"typing", receiver_uid}` (throttled client-side), `{type:"read", with_uid}`, `{type:"sync", since, with_uid?}` (replay created-or-revised rows after `since`), `{type:"ping"}` (keepalive, ignored). Server -> client: `{type:"ready", user_uid}` (sent first on connect), `{type:"message", uid, sender_uid, sender_username, sender_role, receiver_uid, content, created_at, time_ago, client_id, attachments, ai_processed, ai_pending}` (broadcast to BOTH sender and receiver sockets so multi-tab and the sender's own optimistic bubble reconcile via the echoed `client_id`; a first frame may set `ai_pending` while a sync job runs, and a later frame for the same `uid` sets `ai_processed` with the rewritten body), `{type:"typing", from_uid}` (to the receiver only), `{type:"read", by_uid}` (read-receipt to the other user), `{type:"error", client_id, text}` (sender only, on a dropped send, e.g. the recipient blocked the sender, or a screened body). There is no `presence` frame on this socket - see below.
+Client -> server: `{type:"send", receiver_uid, content, attachment_uids?, client_id}`, `{type:"typing", receiver_uid}` (throttled client-side), `{type:"read", with_uid}`, `{type:"active", with_uid}` (marks the sender as currently viewing that conversation - see "Active-conversation notification suppression" below), `{type:"sync", since, with_uid?}` (replay created-or-revised rows after `since`), `{type:"ping"}` (keepalive, ignored). Server -> client: `{type:"ready", user_uid}` (sent first on connect), `{type:"message", uid, sender_uid, sender_username, sender_role, receiver_uid, content, created_at, time_ago, client_id, attachments, ai_processed, ai_pending}` (broadcast to BOTH sender and receiver sockets so multi-tab and the sender's own optimistic bubble reconcile via the echoed `client_id`; a first frame may set `ai_pending` while a sync job runs, and a later frame for the same `uid` sets `ai_processed` with the rewritten body), `{type:"typing", from_uid}` (to the receiver only), `{type:"read", by_uid}` (read-receipt to the other user), `{type:"error", client_id, text}` (sender only, on a dropped send, e.g. the recipient blocked the sender, or a screened body). There is no `presence` frame on this socket - see below. `active` has no server -> client echo; it only writes the DB-backed marker consumed server-side by `persist_message`.
+
+## Active-conversation notification suppression
+
+A `message` notification (and its live toast) is suppressed when the receiver is demonstrably looking at that exact conversation right now, so an open DM thread never also produces a redundant bell/toast for the same message. The message itself still delivers live over the WS/relay path exactly as before - only the `create_notification("message", ...)` call in `persist_message` is skipped.
+
+- **Marker: two columns on `users`**, `active_conversation_uid` (the uid of the conversation partner currently being viewed) and `active_conversation_at` (UTC ISO timestamp of the last time it was refreshed), ensured in `database.backfill_api_keys()` like every other non-signup `users` column. This follows the `services/presence.py` `last_seen` pattern deliberately - a WS connection is per-worker (`message_hub` has no cross-worker visibility), but the sender and receiver of a DM can be on different uvicorn workers, so the "is the receiver looking at this" check has to be readable from ANY worker. A DB column is the only cross-worker-correct medium here, exactly as presence is.
+- **Write path**: `services/messaging/active_conversation.py` `touch_active_conversation(viewer_uid, with_uid)`, called from the WS handler's new `{type:"active", with_uid}` frame. Throttled per-worker exactly like `presence.touch` (`WRITE_THROTTLE_SECONDS = 5`, an in-memory `dict[f"{viewer_uid}:{with_uid}" -> monotonic]` so a burst of focus/visibility events costs at most one write per 5s per conversation) before the `UPDATE users SET active_conversation_uid=..., active_conversation_at=...` write.
+- **Client trigger (`AppChat.js`)**: `_sendActiveMarker()` sends the frame whenever `_threadIsActive()` is true (desktop: any open thread; mobile: only when the thread pane, not the list pane, is showing). It fires wherever the existing `_markRead()` already fires (initial WS connect, opening/switching a conversation) plus two new listeners bound in `_bindActiveConversation()`: `window` `focus` and `document` `visibilitychange` (`visibilityState === "visible"`) - so returning to a backgrounded/blurred tab that still has the conversation open re-asserts the marker.
+- **Read path / freshness window**: `is_actively_viewing(viewer_uid, other_uid)` (same module) reads the receiver's user row, checks `active_conversation_uid == other_uid`, and requires `active_conversation_at` to be less than `FRESH_SECONDS = 20` old. `persist_message` calls it as `is_actively_viewing(receiver_uid, sender_uid)` right before the `create_notification` call and skips only that call on a hit; `clear_messages_cache(receiver_uid)` still runs unconditionally, and the WS/relay delivery of the message frame is completely unaffected (it is broadcast from `routers/messages.py`, not from `persist_message`). The freshness window (20s) is intentionally wider than the write throttle (5s) so it never lapses between two consecutive throttled writes, and the marker is a soft "recently confirmed as active" signal, not a hard connection state - if it goes stale (tab closed, thread switched away without a fresh trigger) the very next message just resumes notifying normally, which is the safe failure direction.
## Presence is NOT part of the messaging WS
@@ -65,7 +80,9 @@ Live/echoed message bubbles are NEVER injected as raw HTML. `AppChat._buildBubbl
## Frontend
-The messaging frontend is the single self-booting custom element `` (`static/js/components/AppChat.js`), which replaced the old page-controller trio `MessagesLayout.js`/`MessagesSocket.js`/`MessageSearch.js` (all deleted) and `messages.css` (superseded by `static/css/chat.css`) - see `devplacepy/static/js/CLAUDE.md` for the component roster entry. `MobileNav.js` does **not** own the mobile pane (that duplicate was removed; `dp-chat` is the only pane controller). `templates/messages.html` renders `` wrapping the SAME server-rendered `.messages-list`/`.messages-main`/`.messages-thread`/`.message-bubble` markup as before (no-JS/crawler fallback), which the component *adopts* on `connectedCallback` rather than discarding. It owns a `ChatSocket` (`static/js/chat/ChatSocket.js`, exponential backoff, `4013` fast-path, 25s ping), sends over WS with an optimistic pending bubble keyed by `client_id`, reconciles on the echoed `message` frame, applies later `ai_processed` frames in place (compare `dp-content[data-source]`, never rendered `textContent`), injects the Report control on every live incoming bubble, catches up with `{type:"sync"}` on reconnect, and switches conversations without dropping the socket (`GET /messages?with_uid=` JSON + `history.pushState`). Older history loads via `GET /messages?with_uid=&before=` when the thread is scrolled to the top. Mobile: CSS hides the inactive pane from `with-uid` / `.show-list` before JS runs (no stacked FOUC); the composer does not auto-focus on coarse pointers; keyboard inset uses `visualViewport.offsetTop + height`; Enter-to-send is desktop-only. The send button is disabled only while `dp-upload` is busy, never while a send is in flight. Presence in `mode="page"` stays on the page-global `PresenceManager`. An opt-in `ai-indicator="true"` attribute shows "Adjusting..." while `ai_pending` and "Adjusted by AI" when the revision lands. Styling is in `static/css/chat.css` using `variables.css` tokens only, responsive down to 360px.
+The messaging frontend is the single self-booting custom element `` (`static/js/components/AppChat.js`), which replaced the old page-controller trio `MessagesLayout.js`/`MessagesSocket.js`/`MessageSearch.js` (all deleted) and `messages.css` (superseded by `static/css/chat.css`) - see `devplacepy/static/js/CLAUDE.md` for the component roster entry. `MobileNav.js` does **not** own the mobile pane (that duplicate was removed; `dp-chat` is the only pane controller). `templates/messages.html` renders `` wrapping the SAME server-rendered `.messages-list`/`.messages-main`/`.messages-thread`/`.message-bubble` markup as before (no-JS/crawler fallback), which the component *adopts* on `connectedCallback` rather than discarding. It owns a `ChatSocket` (`static/js/chat/ChatSocket.js`, exponential backoff, `4013` fast-path, 25s ping), sends over WS with an optimistic pending bubble keyed by `client_id`, reconciles on the echoed `message` frame, applies later `ai_processed` frames in place (compare `dp-content[data-source]`, never rendered `textContent`), injects the Report control on every live incoming bubble, catches up with `{type:"sync"}` on reconnect, and switches conversations without dropping the socket (`GET /messages?with_uid=` JSON + `history.pushState`). Older history loads via `GET /messages?with_uid=&before=` when the thread is scrolled to the top. Mobile: CSS hides the inactive pane from `with-uid` / `.show-list` before JS runs (no stacked FOUC); the composer does not auto-focus on coarse pointers; keyboard inset uses `visualViewport.offsetTop + height`; Enter-to-send is desktop-only. **The send button is disabled while `dp-upload` is busy AND while a genuine new send is in flight** (the send-lock, see "Send dedupe" above) - a `.failed`-bubble retry never engages it, so retrying an old message never blocks typing/sending a new one. Presence in `mode="page"` stays on the page-global `PresenceManager`. An opt-in `ai-indicator="true"` attribute shows "Adjusting..." while `ai_pending` and "Adjusted by AI" when the revision lands. Styling is in `static/css/chat.css` using `variables.css` tokens only, responsive down to 360px.
+
+**Auto-scroll never fights a manual scroll-up.** `_startScrollWatcher`/`_stabilizeScroll` force `scrollTop = scrollHeight` on every thread mutation (new bubble, attachment/image finishing loading) ONLY while `_userAtBottom` is true, re-checked every `requestAnimationFrame` up to `STABILIZE_MAX_FRAMES`. The one thing that used to break this: while a multi-frame stabilize loop was in flight (`_stabilizePending`), the real `scroll` event listener (`_bindAutoScroll`) ignored EVERY scroll event outright, including a genuine user wheel/touch scroll-up that happened to land inside that window - so a rapid burst of incoming messages (each restarting the stabilize loop while attachments were still loading) could silently overwrite a user's attempt to scroll up to read history. The fix: every programmatic scroll (`_scrollThreadToEnd`, `_stabilizeScroll`) records the exact `scrollTop` it just set in `_lastForcedScrollTop`; the `scroll` listener only skips its own echo (`_stabilizePending && scrollTop === _lastForcedScrollTop`) and always processes a scroll that lands anywhere else, so a real user scroll-up is detected and `_userAtBottom` flips to `false` immediately, even mid-stabilization. Never gate scroll-intent detection on `_stabilizePending` alone again.
## Attachments stream live
diff --git a/devplacepy/services/messaging/__init__.py b/devplacepy/services/messaging/__init__.py
index e3efd8f..bd71ccc 100644
--- a/devplacepy/services/messaging/__init__.py
+++ b/devplacepy/services/messaging/__init__.py
@@ -1,5 +1,9 @@
# retoor
+from devplacepy.services.messaging.active_conversation import (
+ is_actively_viewing,
+ touch_active_conversation,
+)
from devplacepy.services.messaging.hub import message_hub
from devplacepy.services.messaging.persist import (
message_frame,
@@ -11,6 +15,7 @@ from devplacepy.services.messaging.relay import message_relay
from devplacepy.services.messaging.tickets import issue_ticket, redeem_ticket
__all__ = [
+ "is_actively_viewing",
"issue_ticket",
"message_frame",
"message_hub",
@@ -19,4 +24,5 @@ __all__ = [
"push_content_revision",
"redeem_ticket",
"stamp_content_revision",
+ "touch_active_conversation",
]
diff --git a/devplacepy/services/messaging/active_conversation.py b/devplacepy/services/messaging/active_conversation.py
new file mode 100644
index 0000000..addde15
--- /dev/null
+++ b/devplacepy/services/messaging/active_conversation.py
@@ -0,0 +1,52 @@
+# retoor
+
+import time
+from datetime import datetime, timezone
+from typing import Optional
+
+from devplacepy.database import get_table
+
+WRITE_THROTTLE_SECONDS = 5
+FRESH_SECONDS = 20
+
+_last_write: dict[str, float] = {}
+
+
+def touch_active_conversation(viewer_uid: str, with_uid: str) -> None:
+ if not viewer_uid or not with_uid:
+ return
+ now = time.monotonic()
+ key = f"{viewer_uid}:{with_uid}"
+ if now - _last_write.get(key, 0.0) < WRITE_THROTTLE_SECONDS:
+ return
+ _last_write[key] = now
+ get_table("users").update(
+ {
+ "uid": viewer_uid,
+ "active_conversation_uid": with_uid,
+ "active_conversation_at": datetime.now(timezone.utc).isoformat(),
+ },
+ ["uid"],
+ )
+
+
+def _seconds_since(stamped: Optional[str]) -> Optional[float]:
+ if not stamped:
+ return None
+ try:
+ seen = datetime.fromisoformat(stamped)
+ except (TypeError, ValueError):
+ return None
+ if seen.tzinfo is None:
+ seen = seen.replace(tzinfo=timezone.utc)
+ return (datetime.now(timezone.utc) - seen).total_seconds()
+
+
+def is_actively_viewing(viewer_uid: str, other_uid: str) -> bool:
+ if not viewer_uid or not other_uid:
+ return False
+ row = get_table("users").find_one(uid=viewer_uid)
+ if not row or row.get("active_conversation_uid") != other_uid:
+ return False
+ elapsed = _seconds_since(row.get("active_conversation_at"))
+ return elapsed is not None and elapsed < FRESH_SECONDS
diff --git a/devplacepy/services/messaging/persist.py b/devplacepy/services/messaging/persist.py
index ed9b5d6..129515b 100644
--- a/devplacepy/services/messaging/persist.py
+++ b/devplacepy/services/messaging/persist.py
@@ -2,7 +2,7 @@
import asyncio
import logging
-from datetime import datetime, timezone
+from datetime import datetime, timedelta, timezone
from typing import Any, Optional
from devplacepy.attachments import get_attachments, link_attachments
@@ -18,6 +18,7 @@ from devplacepy.utils import (
from devplacepy.services.audit import record as audit
from devplacepy.services.correction import schedule_correction
from devplacepy.services.ai_modifier import schedule_modification
+from devplacepy.services.messaging.active_conversation import is_actively_viewing
from devplacepy.services.moderation.screening import (
record as record_screening,
refuse_if_blocked,
@@ -27,6 +28,7 @@ from devplacepy.services.moderation.screening import (
logger = logging.getLogger("messaging.persist")
MAX_CONTENT_LENGTH = 2000
+DEDUPE_WINDOW_SECONDS = 30
def _slim_attachment(attachment: dict[str, Any]) -> dict[str, Any]:
@@ -64,6 +66,18 @@ def message_frame(
}
+def _find_recent_duplicate(sender_uid: str, client_id: str) -> Optional[dict[str, Any]]:
+ cutoff = (
+ datetime.now(timezone.utc) - timedelta(seconds=DEDUPE_WINDOW_SECONDS)
+ ).isoformat()
+ row = get_table("messages").find_one(
+ sender_uid=sender_uid,
+ client_id=client_id,
+ created_at={">=": cutoff},
+ )
+ return dict(row) if row else None
+
+
def persist_message(
sender: dict[str, Any],
receiver_uid: str,
@@ -72,23 +86,30 @@ def persist_message(
*,
request: Any = None,
origin: str = "web",
+ client_id: Optional[str] = None,
) -> Optional[dict[str, Any]]:
content = (content or "").strip()[:MAX_CONTENT_LENGTH]
attachment_uids = attachment_uids or []
if not content and not attachment_uids:
return None
+ sender_uid = sender["uid"]
+ client_id = (client_id or "").strip()[:64] or None
+ if client_id:
+ duplicate = _find_recent_duplicate(sender_uid, client_id)
+ if duplicate is not None:
+ return duplicate
+
receiver = get_table("users").find_one(uid=receiver_uid)
if not receiver:
return None
- if sender["uid"] in get_blocked_uids(receiver_uid):
+ if sender_uid in get_blocked_uids(receiver_uid):
return None
screening = screen_fields("messages", {"content": content})
refuse_if_blocked(screening)
- sender_uid = sender["uid"]
sender_username = sender.get("username", "")
messages_table = get_table("messages")
msg_uid = generate_uid()
@@ -102,6 +123,7 @@ def persist_message(
"read": False,
"created_at": created_at,
"updated_at": None,
+ "client_id": client_id,
}
)
@@ -117,13 +139,14 @@ def persist_message(
schedule_modification(sender, "messages", msg_uid, request)
if sender_uid != receiver_uid:
- create_notification(
- receiver_uid,
- "message",
- f"{sender_username} sent you a message",
- sender_uid,
- f"/messages?with_uid={sender_uid}",
- )
+ if not is_actively_viewing(receiver_uid, sender_uid):
+ create_notification(
+ receiver_uid,
+ "message",
+ f"{sender_username} sent you a message",
+ sender_uid,
+ f"/messages?with_uid={sender_uid}",
+ )
clear_messages_cache(receiver_uid)
create_mention_notifications(
@@ -182,6 +205,7 @@ def persist_message(
"read": False,
"created_at": created_at,
"updated_at": None,
+ "client_id": client_id,
}
diff --git a/devplacepy/services/openai_gateway/CLAUDE.md b/devplacepy/services/openai_gateway/CLAUDE.md
index 20313c2..e4b5ead 100644
--- a/devplacepy/services/openai_gateway/CLAUDE.md
+++ b/devplacepy/services/openai_gateway/CLAUDE.md
@@ -42,6 +42,16 @@ Callers may send an optional `X-App-Reference` header to tag gateway calls by ap
**Per-user attribution:** the gateway also accepts a real user's `api_key` (Bearer / X-API-KEY / session), gated by `gateway_allow_users` (default **on**); `resolve_owner()` records `(owner_kind, owner_id)` per call, so usage is attributed and limitable per user. With `gateway_allow_users` off, only internal/access/admin keys work and per-user attribution never happens. Devii operating a signed-in user authenticates its LLM calls with that user's own `api_key` (set as the session's `ai_key` in `build_settings(..., owner_kind="user")`), so a user's full gateway spend (Devii and direct API calls) rolls up under their uid - surfaced admin-only on the profile page via `build_user_usage()` / `GET /admin/users/{uid}/ai-usage`. Guests keep the internal key.
+### Failed-auth throttle (`services/openai_gateway/auth_throttle.py`)
+
+A dedicated per-IP throttle protects `/openai/v1/*` against unauthenticated probing WITHOUT IP-allowlisting the endpoint (which would break legitimate external API consumers using `gateway_access_key`/`gateway_allow_users`). It is scoped ONLY to failed authentication attempts, never to general traffic - the existing global rate limiter already exempts `/openai` entirely (`main.py`), and this throttle does not change that exemption.
+
+- **Config (`gateway_auth_throttle_enabled` default on, `gateway_auth_throttle_max_failures` default 10, `gateway_auth_throttle_window_seconds` default 60, all group "Access").** An admin can retune or disable the throttle with no code change, same as every other gateway setting.
+- **Counter.** `auth_throttle.py` is a small in-process sliding-window store (`dict[ip] -> list[float]` timestamps), the same shape as the global rate limiter's `_rate_limit_store` in `main.py`, but a separate, dedicated store local to the gateway rather than reusing that one (the two throttles key on different events - all mutating traffic vs. failed gateway auth - and `/openai` is deliberately exempt from the general limiter). `record_failure(ip, window)` appends a timestamp and returns the running in-window count; `is_throttled(ip, threshold, window)` checks without recording. Per-worker, like every other in-process cache in this codebase - not divided by worker count, so the effective aggregate threshold across `N` workers is up to `N x max_failures`; this is an accepted tradeoff (it still closes the probe, just at a slightly higher aggregate bound) rather than importing `main.WEB_WORKERS` into a leaf service module.
+- **`GatewayService.authorize()` records a failure on every deny path** (a cheap static-key/internal-key match still short-circuits first and is never counted), so a wrong or garbage key counts exactly like presenting nothing. **The fast 429 short-circuit fires only when the request carries no credential at all** (no `X-API-KEY`, no `Authorization` header of any scheme, no `session` cookie) AND the IP is already over threshold - this is the "skip the rest of `authorize()`" defense against wasted CPU on a blind flood. A request presenting ANY credential - even one that turns out invalid - always runs the full check, so it can never be blocked by another caller's failures sharing its IP (a NAT/proxy caller with its own valid key is never punished for a neighbor's failed probes); only a request presenting nothing is fast-denied once its IP has tripped.
+- **Audit.** Tripping the throttle (the exact request whose failure brings the window count to the configured threshold) records `ai.gateway_auth_throttle.tripped` (category `ai`, `result="denied"`, metadata `ip`/`failed_attempts`/`threshold`/`window_seconds`) via `audit.record(request, ...)`. Every subsequent fast-denied request in the same trip is silent (no audit spam under an actual flood) - the trip itself is the visible signal for admins.
+- **Internal traffic cannot trip this by construction.** Every internal consumer (Devii guests, AI correction/modifier, quiz grading, SEO metadata generation, news, bots) authenticates with `database.internal_gateway_key()`, which is the exact same value `authorize()` compares against as `gateway_internal_key` - that comparison is the FIRST check in `authorize()`, before the throttle is even consulted, so internal self-dial calls always succeed there and never reach the failure-recording branch.
+
## Config fields
All config is `config_fields` (upstream url/model/key, force-model, the Prompt group's `gateway_system_preamble` and `gateway_thinking` (default off), timeout, instances, vision url/model/key/cache/toggle, the Embeddings group (`gateway_embed_enabled`/`_url`/`_model`/`_key`), the auth toggles + static key + internal key, plus the Pricing/Reliability/Tracking groups); live metrics (requests/errors/in-flight/vision-calls/embed-calls/latency plus 24h cost/tokens/success rollups) via `collect_metrics`.
@@ -57,6 +67,15 @@ Two behaviors:
Embeddings/passthrough are untouched by this composition step.
+### Per-call operator preamble bypass (`bypass_preamble`, internal-only)
+
+A structured/deterministic internal caller (one that expects a strict, machine-parsed output shape - grading JSON, a corrected text field, generated SEO metadata) can ask the gateway to skip `gateway_system_preamble` for that one call, since an operator preamble written for conversational tone can conflict with a task that demands exact output discipline. The date line and the client's own system content are unaffected either way.
+
+- **Signal.** The chat request body carries `"bypass_preamble": true`. `GatewayRuntime.handle_chat` pops it off `body` before it is ever copied into the upstream `payload` (so it never leaks to the upstream provider) and only honors it when the caller also proved internal credentials for that same request (see below); otherwise it is silently discarded, exactly like an ordinary, unsupported field.
+- **Internal-only gate.** `GatewayService.internal_bypass_allowed(request, cfg)` requires the request to carry the `X-Gateway-Internal-Key` header equal to the configured `gateway_internal_key` setting. This is a SEPARATE header from `Authorization`/`X-API-KEY` on purpose: a caller that authenticates with a real user's own `api_key` (correction, the `@ai` modifier, quiz free-text grading - all per-user-attributed on purpose, for correct spend attribution) still proves it is DevPlace's own server code, not the user's browser or a malicious client holding that same api_key, by additionally presenting the internal key on this header. A caller that authenticates AS the internal key directly (SEO metadata generation, via `database.internal_gateway_key()`) can present the same header too - one mechanism covers both shapes of internal caller. `GatewayService.handle()` computes this boolean once per request and passes it into `handle_chat` as `bypass_allowed`; a request with no valid internal credentials gets `bypass_allowed=False` and the body field is ignored even if present - it can never come from an external API consumer or a public-facing client call.
+- **Callers that set it.** The four structured/deterministic internal callers all route through the single shared `services/correction.py::gateway_complete(..., bypass_preamble=True)` helper, which sets both the body field and the `X-Gateway-Internal-Key` header (via `database.internal_gateway_key()`) in one place: `services/correction.py::correct_text` (AI content correction), `services/ai_modifier.py::modify_text` (the `@ai` inline modifier), `services/quiz/grading.py::grade_free_text` (quiz free-text grading), and `services/jobs/seo_meta_service.py`'s generation call (SEO metadata). **Genuinely conversational internal callers are deliberately left unchanged** and keep receiving the operator preamble: Devii chat sessions and the docs chat do not call `gateway_complete` at all, so they are unaffected by construction.
+- **Other internal self-dial callers found but NOT updated** (each builds its own request instead of using `gateway_complete`, and each is a candidate for the same treatment in a follow-up): `services/gitea/enhance.py::enhance_ticket` and `services/gitea/planning.py::generate_plan` (both structured, single-shot ticket text generation), `services/dbapi/nl2sql.py::design_query` (NL-to-SQL, strictly structured), `services/deepsearch/llm.py`/`services/jobs/deepsearch/enhance.py` (planner/synthesis calls), and `services/jobs/isslop/agent/llm.py` (per-file classification). None of these were in scope for this change.
+
## Thinking default (fast path)
`thinking.py` (`apply_thinking`) runs in `handle_chat` after `stream_options` is stripped, and in the vision describe call. DeepSeek V4 **enables thinking by default**; leaving the payload alone is the slow path. The gateway therefore always writes an explicit thinking field in the upstream dialect:
@@ -127,7 +146,7 @@ The human-facing **Quota rules** tab of `/admin/gateway` is a third, independent
## Embeddings
-`POST /openai/v1/embeddings` exposes an OpenAI-compatible text-embeddings model. Clients send the generic model `molodetz~embed` (`config.INTERNAL_EMBED_MODEL`), which `handle_embeddings` remaps to `gateway_embed_model` exactly like chat remaps `molodetz` -> `gateway_model` (also remapped when `gateway_force_model` is on or the model is empty). It defaults to OpenRouter's `qwen/qwen3-embedding-8b` at `https://openrouter.ai/api/v1/embeddings` (`config.EMBED_*_DEFAULT`, $0.01 per 1M input tokens). `handle_embeddings` mirrors `handle_chat` but is simpler: no vision augmentation and no streaming - build the payload, forward via `_send`, and record one ledger row through the same `finalize(...)` closure. The config fields are the **Embeddings** group (`gateway_embed_enabled` default on, `gateway_embed_url`, `gateway_embed_model`, `gateway_embed_key`) plus the Pricing-group `gateway_embed_price_input_per_m`. `effective_config()` falls the embed key back to `gateway_vision_key` then `OPENROUTER_API_KEY` (NOT `gateway_api_key`: that is the DeepSeek chat upstream key, whereas embeddings target OpenRouter like vision does). Usage is recorded with **`backend="embed"`**; `usage.compute_cost` adds an `embed` branch (input-only, completion always 0, native OpenRouter `cost` still preferred) and `Pricing` gained `embed_input_per_m`. `analytics.py` groups by `backend` generically, so embed rows roll up automatically; `caching_savings` counts only **non-native** chat rows (native-priced rows did not use the configured cache-hit/miss rates, so folding them in would report a fictional saving). When `gateway_embed_enabled` is off the endpoint returns 503 with no ledger row.
+`POST /openai/v1/embeddings` exposes an OpenAI-compatible text-embeddings model. Clients send the generic model `molodetz~embed` (`config.INTERNAL_EMBED_MODEL`), which `handle_embeddings` remaps to `gateway_embed_model` exactly like chat remaps `molodetz` -> `gateway_model` (also remapped when `gateway_force_model` is on or the model is empty). It defaults to OpenRouter's `qwen/qwen3-embedding-8b` at `https://openrouter.ai/api/v1/embeddings` (`config.EMBED_*_DEFAULT`, $0.01 per 1M input tokens). `handle_embeddings` mirrors `handle_chat` but is simpler: no vision augmentation and no streaming - build the payload, forward via `_send`, and record one ledger row through the same `finalize(...)` closure. The config fields are the **Embeddings** group (`gateway_embed_enabled` default on, `gateway_embed_url`, `gateway_embed_model`, `gateway_embed_key`) plus the Pricing-group `gateway_embed_price_input_per_m`. `effective_config()` falls the embed key back to `gateway_vision_key` then `OPENROUTER_API_KEY` (NOT `gateway_api_key`: that is the DeepSeek chat upstream key, whereas embeddings target OpenRouter like vision does). Usage is recorded with **`backend="embed"`**; `usage.compute_cost` adds an `embed` branch (input-only, completion always 0, native OpenRouter `cost` still preferred) and `Pricing` gained `embed_input_per_m`. `analytics.py` groups by `backend` generically, so embed rows roll up automatically; `caching_savings` counts only **non-native** chat rows (native-priced rows did not use the configured cache-hit/miss rates, so folding them in would report a fictional saving). When `gateway_embed_enabled` is off the endpoint returns 503 with no ledger row. **When the resolved `gateway_embed_key` is still blank after the `effective_config()`/route-overlay merge** (all of `gateway_embed_key`, `gateway_vision_key`, and `OPENROUTER_API_KEY` are unset, or a per-route provider was configured with no `api_key` of its own), `handle_embeddings` returns the same `503` shape (`{"error": {"message": "Embeddings are not configured", "type": "embeddings_not_configured"}}`) with an `ai.gateway.call` audit row (`result="denied"`, `summary="no embeddings key configured"`) instead of forwarding an unauthenticated request upstream - mirrors the `gateway_embed_enabled` disabled-check immediately above it, no ledger row either. `routing.embed_overlay` only overlays `gateway_embed_key` when the matched route's provider actually has a non-blank `api_key`, so a misconfigured route can never downgrade an already-good fallback key to empty.
## Single point of truth for AI
diff --git a/devplacepy/services/openai_gateway/auth_throttle.py b/devplacepy/services/openai_gateway/auth_throttle.py
new file mode 100644
index 0000000..b2383cb
--- /dev/null
+++ b/devplacepy/services/openai_gateway/auth_throttle.py
@@ -0,0 +1,56 @@
+# retoor
+
+import time
+from collections import defaultdict
+
+_failures: dict[str, list[float]] = defaultdict(list)
+_last_sweep = 0.0
+SWEEP_INTERVAL_SECONDS = 60.0
+
+
+def _window_slice(ip: str, window_seconds: int, now: float) -> list[float]:
+ window_start = now - window_seconds
+ timestamps = [t for t in _failures.get(ip, ()) if t > window_start]
+ if timestamps:
+ _failures[ip] = timestamps
+ elif ip in _failures:
+ del _failures[ip]
+ return timestamps
+
+
+def _sweep(window_start: float) -> None:
+ stale = [
+ ip
+ for ip, timestamps in _failures.items()
+ if not timestamps or timestamps[-1] <= window_start
+ ]
+ for ip in stale:
+ del _failures[ip]
+
+
+def is_throttled(ip: str, threshold: int, window_seconds: int) -> bool:
+ now = time.time()
+ return len(_window_slice(ip, window_seconds, now)) >= threshold
+
+
+def record_failure(ip: str, window_seconds: int) -> int:
+ global _last_sweep
+ now = time.time()
+ window_start = now - window_seconds
+ if now - _last_sweep >= SWEEP_INTERVAL_SECONDS:
+ _sweep(window_start)
+ _last_sweep = now
+ timestamps = _window_slice(ip, window_seconds, now)
+ timestamps.append(now)
+ _failures[ip] = timestamps
+ return len(timestamps)
+
+
+def reset(ip: str) -> None:
+ _failures.pop(ip, None)
+
+
+def clear() -> None:
+ _failures.clear()
+ global _last_sweep
+ _last_sweep = 0.0
diff --git a/devplacepy/services/openai_gateway/config.py b/devplacepy/services/openai_gateway/config.py
index 6b2f804..bea32e4 100644
--- a/devplacepy/services/openai_gateway/config.py
+++ b/devplacepy/services/openai_gateway/config.py
@@ -6,6 +6,9 @@ TIMEOUT_DEFAULT = 300
TIMEOUT_MIN = 300
INSTANCES_DEFAULT = 4
+AUTH_THROTTLE_MAX_FAILURES_DEFAULT = 10
+AUTH_THROTTLE_WINDOW_SECONDS_DEFAULT = 60
+
SYSTEM_PREAMBLE_DEFAULT = ""
THINKING_DEFAULT = False
diff --git a/devplacepy/services/openai_gateway/gateway.py b/devplacepy/services/openai_gateway/gateway.py
index c0e7b90..fbd21cc 100644
--- a/devplacepy/services/openai_gateway/gateway.py
+++ b/devplacepy/services/openai_gateway/gateway.py
@@ -350,9 +350,17 @@ class GatewayRuntime:
return resp, None, timing
async def handle_chat(
- self, body: dict, cfg: dict, owner: tuple, user_agent: str, app_reference: str, log=None
+ self,
+ body: dict,
+ cfg: dict,
+ owner: tuple,
+ user_agent: str,
+ app_reference: str,
+ log=None,
+ bypass_allowed: bool = False,
):
log = log or (lambda message: None)
+ bypass_preamble = bool(body.pop("bypass_preamble", False)) and bypass_allowed
overlay = chat_overlay(body.get("model"), cfg)
base_cfg = cfg
if overlay:
@@ -391,7 +399,8 @@ class GatewayRuntime:
self.vision_calls += augmenter.calls
vision_cost = augmenter.cost_usd
- messages = apply_system_directives(messages, cfg.get("gateway_system_preamble", ""))
+ preamble = "" if bypass_preamble else cfg.get("gateway_system_preamble", "")
+ messages = apply_system_directives(messages, preamble)
requested = body.get("model")
allow_client_model = bool(cfg.get("gateway_allow_client_model"))
@@ -842,6 +851,35 @@ class GatewayRuntime:
}
},
)
+ if not cfg["gateway_embed_key"]:
+ from devplacepy.services.audit import record as audit
+ from devplacepy.services.openai_gateway.usage import audit_actor_for
+
+ actor_kind, actor_uid, actor_role = audit_actor_for(owner[0], owner[1])
+ audit.record_system(
+ "ai.gateway.call",
+ actor_kind=actor_kind,
+ actor_uid=actor_uid,
+ actor_role=actor_role,
+ origin="api",
+ result="denied",
+ summary="no embeddings key configured",
+ metadata={
+ "backend": "embed",
+ "endpoint": "embeddings",
+ "owner_kind": owner[0],
+ "owner_id": owner[1],
+ },
+ )
+ return JSONResponse(
+ status_code=503,
+ content={
+ "error": {
+ "message": "Embeddings are not configured",
+ "type": "embeddings_not_configured",
+ }
+ },
+ )
client, sem = self._ensure(cfg)
params = extract_params(body)
handle_start = time.monotonic()
@@ -866,13 +904,8 @@ class GatewayRuntime:
"Content-Type": "application/json",
**_attribution_headers(),
**_extra_provider_headers(cfg),
+ "Authorization": f"Bearer {cfg['gateway_embed_key']}",
}
- if cfg["gateway_embed_key"]:
- headers["Authorization"] = f"Bearer {cfg['gateway_embed_key']}"
- else:
- log(
- "No upstream embeddings API key configured (gateway_embed_key / gateway_vision_key / OPENROUTER_API_KEY); upstream will likely reject the request"
- )
resp, exc, timing = await self._send(
client,
diff --git a/devplacepy/services/openai_gateway/service.py b/devplacepy/services/openai_gateway/service.py
index 054f298..7028e47 100644
--- a/devplacepy/services/openai_gateway/service.py
+++ b/devplacepy/services/openai_gateway/service.py
@@ -11,16 +11,19 @@ from fastapi.responses import JSONResponse
from devplacepy.database import get_int_setting
from devplacepy.services.base import BaseService, ConfigField
-from devplacepy.services.openai_gateway import config, quota
+from devplacepy.services.openai_gateway import auth_throttle, config, quota
from devplacepy.services.openai_gateway.analytics import summary_metrics
from devplacepy.services.openai_gateway.gateway import GatewayRuntime
from devplacepy.services.openai_gateway.routing import model_store
from devplacepy.utils import get_current_user
+from devplacepy.utils.auth import _request_has_auth
+from devplacepy.utils.guards import client_ip
logger = logging.getLogger(__name__)
APP_REFERENCE_PATTERN = re.compile(r"^[a-zA-Z0-9_.-]{1,30}$")
DEFAULT_APP_REFERENCE = "default"
+INTERNAL_KEY_HEADER = "X-Gateway-Internal-Key"
USER_CONTENT_OWNER_KINDS = ("user", "admin")
@@ -324,6 +327,36 @@ class GatewayService(BaseService):
"with this key. Clear it and restart to rotate.",
group="Access",
),
+ ConfigField(
+ "gateway_auth_throttle_enabled",
+ "Failed-auth throttle",
+ type="bool",
+ default=True,
+ help="Track failed authentication attempts per IP and block further "
+ "unauthenticated attempts from an IP once it crosses the failure "
+ "threshold within the window. Never blocks a request that presents "
+ "valid credentials, regardless of what its IP has done.",
+ group="Access",
+ ),
+ ConfigField(
+ "gateway_auth_throttle_max_failures",
+ "Failed-auth threshold",
+ type="int",
+ default=config.AUTH_THROTTLE_MAX_FAILURES_DEFAULT,
+ minimum=1,
+ help="Failed authentication attempts allowed from one IP within the "
+ "window before further unauthenticated attempts are blocked with 429.",
+ group="Access",
+ ),
+ ConfigField(
+ "gateway_auth_throttle_window_seconds",
+ "Failed-auth window (seconds)",
+ type="int",
+ default=config.AUTH_THROTTLE_WINDOW_SECONDS_DEFAULT,
+ minimum=1,
+ help="Sliding window the failed-auth threshold is counted over.",
+ group="Access",
+ ),
ConfigField(
quota.FIELD_DEFAULT_USER,
"Default per-user daily cap ($)",
@@ -537,6 +570,24 @@ class GatewayService(BaseService):
)
return cfg
+ def _audit_throttle_tripped(
+ self, request: Request, ip: str, count: int, threshold: int, window: int
+ ) -> None:
+ from devplacepy.services.audit import record as audit
+
+ audit.record(
+ request,
+ "ai.gateway_auth_throttle.tripped",
+ result="denied",
+ summary=f"AI gateway auth throttle tripped for {ip} after {count} failed attempts",
+ metadata={
+ "ip": ip,
+ "failed_attempts": count,
+ "threshold": threshold,
+ "window_seconds": window,
+ },
+ )
+
def authorize(self, request: Request) -> bool:
cfg = self.get_config()
if not cfg["gateway_require_auth"]:
@@ -548,12 +599,35 @@ class GatewayService(BaseService):
return True
if presented and internal_key and presented == internal_key:
return True
+ throttle_enabled = cfg.get("gateway_auth_throttle_enabled", True)
+ threshold = max(1, int(cfg.get("gateway_auth_throttle_max_failures", 10)))
+ window = max(1, int(cfg.get("gateway_auth_throttle_window_seconds", 60)))
+ has_credential = (
+ bool(presented)
+ or bool(request.cookies.get("session"))
+ or _request_has_auth(request)
+ )
+ ip = client_ip(request, default="unknown")
+ if (
+ throttle_enabled
+ and not has_credential
+ and auth_throttle.is_throttled(ip, threshold, window)
+ ):
+ raise HTTPException(
+ status_code=429,
+ detail="Too many failed authentication attempts. Try again later.",
+ headers={"Retry-After": str(window)},
+ )
user = get_current_user(request)
if user:
if user.get("role") == "Admin" and cfg["gateway_allow_admins"]:
return True
if cfg["gateway_allow_users"]:
return True
+ if throttle_enabled:
+ count = auth_throttle.record_failure(ip, window)
+ if count == threshold:
+ self._audit_throttle_tripped(request, ip, count, threshold, window)
return False
def resolve_owner(self, request: Request) -> tuple:
@@ -577,6 +651,13 @@ class GatewayService(BaseService):
return (kind, user.get("uid") or "unknown")
return ("anonymous", "anonymous")
+ def internal_bypass_allowed(self, request: Request, cfg: dict) -> bool:
+ internal_key = cfg.get("gateway_internal_key")
+ if not internal_key:
+ return False
+ presented = request.headers.get(INTERNAL_KEY_HEADER, "").strip()
+ return bool(presented) and presented == internal_key
+
def user_content_owner(self, owner: tuple) -> str:
if owner[0] in USER_CONTENT_OWNER_KINDS and owner[1]:
return owner[1]
@@ -722,7 +803,10 @@ class GatewayService(BaseService):
if not isinstance(body, dict):
self.log("Rejected chat request: JSON body was not an object")
raise HTTPException(status_code=400, detail="Invalid JSON body")
- return await runtime.handle_chat(body, cfg, owner, user_agent, app_reference, self.log)
+ bypass_allowed = self.internal_bypass_allowed(request, cfg)
+ return await runtime.handle_chat(
+ body, cfg, owner, user_agent, app_reference, self.log, bypass_allowed
+ )
if subpath == "embeddings" and request.method == "POST":
try:
body = await request.json()
diff --git a/devplacepy/services/quiz/grading.py b/devplacepy/services/quiz/grading.py
index 35271cc..cffd2f9 100644
--- a/devplacepy/services/quiz/grading.py
+++ b/devplacepy/services/quiz/grading.py
@@ -62,6 +62,7 @@ def grade_free_text(api_key: str, question: dict, answer_text: str) -> scoring.G
build_prompt(question, answer_text),
QUIZ_GRADING_TIMEOUT_SECONDS,
model=grading_model(),
+ bypass_preamble=True,
)
except Exception as exc:
logger.warning("Quiz AI grading failed: %s", exc)
diff --git a/devplacepy/services/telegram/bridge.py b/devplacepy/services/telegram/bridge.py
index 6cd34a1..eb979d8 100644
--- a/devplacepy/services/telegram/bridge.py
+++ b/devplacepy/services/telegram/bridge.py
@@ -404,6 +404,7 @@ class TelegramBridge:
chat_id, "Your daily AI quota is reached (100%). Please try again later."
)
return
+ devii.maybe_warn_quota_threshold("user", owner_id, owner_is_admin)
session = devii.hub().get_or_create(
"user",
owner_id,
diff --git a/devplacepy/static/css/CLAUDE.md b/devplacepy/static/css/CLAUDE.md
index 989c328..dac4d9f 100644
--- a/devplacepy/static/css/CLAUDE.md
+++ b/devplacepy/static/css/CLAUDE.md
@@ -40,3 +40,4 @@ One global rule at the end of `base.css` collapses every animation/transition to
- Every fluid grid/flex column sets `min-width: 0`, and every fluid grid track that holds content is written `minmax(0, 1fr)`, never a bare `1fr`. A `1fr` track's automatic minimum is the item's min-content size, so one unwrappable line inside a rendered code block (`.rendered-content pre`, `white-space: pre`) widens the track and blows the whole page open sideways. This is why a detail page (a `max-width` block, definite width, the `pre` scrolls inside it) survives content that destroys a listing grid. The rule covers a column that is fixed-width at desktop but becomes the single fluid column at a breakpoint (`.profile-sidebar`), and it is regression-tested by `assert_no_horizontal_overflow` in `tests/conftest.py`.
- `!important` is allowed only for: the `.hidden`/`[hidden]` display utilities, the global reduced-motion rule, and the devii-avatar third-party-beating override. Anything else is a specificity problem to be fixed structurally.
- Page-specific CSS lives in its own `static/css/*.css` loaded via `{% block extra_head %}`, never an inline `