feat: add backup CLI commands, AI correction/modifier services, and timezone-aware date display
DevPlace CI / test (push) Failing after 25m18s
DevPlace CI / test (push) Failing after 25m18s
- Add `devplace backups` CLI subcommands (list, run, prune, clear) with job enqueueing and orphan cleanup - Introduce `BACKUPS_DIR` and `BACKUP_STAGING_DIR` config paths for backup storage - Implement `schedule_correction` and `schedule_modification` calls in content creation, comment creation, and comment editing flows - Add `DEFAULT_CORRECTION_PROMPT` and `DEFAULT_MODIFIER_PROMPT` config constants for AI content processing - Document timezone-aware date display using `local_dt`/`dt_ago` Jinja globals with client-side `Intl` localization - Update README with AI content correction/modifier support in direct messages via `@ai` inline instructions - Add `track_action(user["uid"], "vote")` call on upvote in `apply_vote`
This commit is contained in:
@@ -43,6 +43,10 @@ devplace seo prune # delete expired SEO audit reports + job rows
|
||||
devplace seo clear # delete every SEO audit report + job row
|
||||
devplace deepsearch prune # delete expired DeepSearch sessions + job rows + collections
|
||||
devplace deepsearch clear # delete every DeepSearch session + job row + collection
|
||||
devplace backups list # list recorded backups
|
||||
devplace backups run <database|uploads|keys|full> # enqueue a backup (processed by the running server)
|
||||
devplace backups prune # remove backup records whose archive file is missing
|
||||
devplace backups clear # delete every backup archive + record
|
||||
devplace containers list # list container instances
|
||||
devplace containers reconcile # run one reconcile pass (desired vs docker ps)
|
||||
devplace containers prune # reap orphan containers + dangling images
|
||||
@@ -91,13 +95,13 @@ Routers in `devplacepy/routers/` are organised as a **directory tree that mirror
|
||||
| `/comments` | comments.py |
|
||||
| `/projects` | projects/ package - `index.py` (listing/detail/create/delete plus owner-only visibility toggles `POST /{slug}/private` and `POST /{slug}/readonly`, and async `POST /{slug}/fork`), `files.py`, and the `containers/` subpackage (below). `main.py` mounts the whole `/projects` tree from this one package |
|
||||
| `/projects/{slug}/files` | projects/files.py - per-project virtual filesystem (CRUD dirs/files, upload, inline edit, plus line-range ops: `lines` read, `replace-lines`, `insert-lines`, `delete-lines`, `append` for large text files) |
|
||||
| `/profile` | profile/ package - `index.py` (page/search/followers/update/api-key), `customization.py`, `notifications.py`, `usage.py` (`_ai_quota`, re-exported from the package) |
|
||||
| `/messages` | messages.py - real-time DM chat. `GET ""` (page), `GET /search`, `POST /send` (no-JS fallback), and `WS /messages/ws` (NOT lock-owner gated - accepts on every worker; closes `1008` for guests). The WS adds live bidirectional delivery, typing, read receipts, and in-process presence on top of the existing `messages` table; both the WS `send` and HTTP `POST /send` route through the shared `services/messaging/persist.py` `persist_message` choke point (identical audit/notification/mention). Same-worker delivery is instant via the per-worker `message_hub`; cross-worker delivery is filled by `services/messaging/relay.py` `message_relay` (per-worker ~1s DB poll on the `messages.id` watermark, deduped against the hub's delivered set). See `AGENTS.md` -> "Real-time messaging" |
|
||||
| `/profile` | profile/ package - `index.py` (page/search/followers/update/api-key; the **Posts** tab shows only the 10 most recent posts, while the sidebar `posts_count` stat always uses the real `count()` so it is never capped to the rendered 10), `customization.py`, `notifications.py`, `ai_correction.py` (`POST /{username}/ai-correction`, owner-or-admin AI content correction toggle + prompt), `ai_modifier.py` (`POST /{username}/ai-modifier`, owner-or-admin AI modifier toggle + prompt), `usage.py` (`_ai_quota`, re-exported from the package) |
|
||||
| `/messages` | messages.py - real-time DM chat. `GET ""` (page), `GET /search`, `POST /send` (no-JS fallback), and `WS /messages/ws` (NOT lock-owner gated - accepts on every worker; closes `1008` for guests). The WS adds live bidirectional delivery, typing, read receipts, and in-process presence on top of the existing `messages` table; both the WS `send` and HTTP `POST /send` route through the shared `services/messaging/persist.py` `persist_message` choke point (identical audit/notification/mention) and then through `_finalize_and_broadcast`, which awaits any pending SYNC AI correction/modifier futures (DM content runs through both, so typing `@ai <instruction>` executes live) and broadcasts the FINAL corrected/modified content (the WS path passes `request=websocket` and awaits directly since HTTP middleware does not run for websockets; the frontend swaps the sender's optimistic bubble to the authoritative content). Same-worker delivery is instant via the per-worker `message_hub`; cross-worker delivery is filled by `services/messaging/relay.py` `message_relay` (per-worker ~1s DB poll on the `messages.id` watermark, deduped against the hub's delivered set). See `AGENTS.md` -> "Real-time messaging" |
|
||||
| `/notifications` | notifications.py |
|
||||
| `/votes` | votes.py |
|
||||
| `/avatar` | avatar.py |
|
||||
| `/follow` | follow.py |
|
||||
| `/admin` | admin/ package - one leaf per sub-resource (`index`, `users`, `aiusage`, `aiquota`, `media`, `trash`, `settings`, `notifications`, `news`, `auditlog`) plus the folded-in `services.py` and `containers.py` (mounted with `/services` and `/containers` sub-prefixes). `main.py` mounts the whole `/admin` tree from this one package |
|
||||
| `/admin` | admin/ package - one leaf per sub-resource (`index`, `users`, `aiusage`, `aiquota`, `media`, `trash`, `settings`, `notifications`, `news`, `auditlog`, `backups`) plus the folded-in `services.py` and `containers.py` (mounted with `/services` and `/containers` sub-prefixes). `main.py` mounts the whole `/admin` tree from this one package. The `backups` leaf is the admin **Backups** dashboard (`BackupService`, kind `backup`): storage usage, backup archives, and interval/cron backup schedules (CRUD + rotation). See AGENTS.md -> "Backup service" |
|
||||
| `/admin/services` | admin/services.py |
|
||||
| `/issues` | issues/ package - issue tracker backed by Gitea (no local issue store): `index.py` (list `?state=`/`?page=`, detail `/{number}` with comments), `create.py` (async AI-enhanced filing `/create` enqueues a `issue_create` job, status at `/jobs/{uid}`), `comment.py` (synchronous, pushes to Gitea + notifies admins), `status.py` (admin open/closed) |
|
||||
| `/gists` | gists.py |
|
||||
@@ -150,12 +154,18 @@ Batch helpers (`get_users_by_uids`, `get_comment_counts_by_post_uids`, `get_vote
|
||||
`dataset.find()` does NOT accept raw SQL. Use kwargs for equality, dict syntax (`{">=": value}`) for comparisons, or `table.table.columns.<col>.in_([...])` for IN clauses. `db.query("... :p", p=val)` is the raw-SQL escape hatch (named params as kwargs, not a dict). `table.update({"uid": x, ...}, ["uid"])` requires the key columns as a list. Always `if "table" in db.tables:` before raw SQL, since `dataset` only auto-creates tables when something is first inserted.
|
||||
|
||||
### Background task queue (`devplacepy/services/background.py`)
|
||||
The `background` singleton is a generic, lightweight, fire-and-forget offload for non-critical side-effects: `background.submit(fn, *args, **kwargs)` enqueues a synchronous callable onto one in-process `asyncio.Queue` drained by a single consumer, so the work leaves the request path. It is **per-worker** (started in `main.py` `startup()` for every worker outside the service-lock branch, drained in `shutdown()`), unlike `BaseService`/`JobService` which run only in the lock owner - audit records are produced in every worker and a DB-backed queue would only add writes. When the consumer is not running (tests with `DEVPLACE_DISABLE_SERVICES=1`, unit tests, full queue), `submit` runs the callable **inline**, so audit/notification/XP writes stay deterministic for the suite while production defers them. Durability is best-effort in-memory (graceful shutdown drains; a hard crash drops unflushed items), which fits audit/notifications already never raising. Deferral happens at the **canonical choke points** so callers need no change: audit-log persistence (`services/audit/record.py` `_write`), **every XP award** (`utils.award_rewards` -> `_apply_rewards`, covering posts/comments/projects/gists/follow/votes), and **every notification** (`utils.create_notification` -> `_deliver_notification`, the single funnel). Because the reward and notification funnels self-defer, handlers call `award_rewards`/`create_notification` directly - never wrap them in `background.submit` (that double-queues). Handler pattern for a NEW side-effect: do the user-visible write inline, then `background.submit(...)` the rest. See `AGENTS.md` -> "Background task queue".
|
||||
The `background` singleton is a generic, lightweight, fire-and-forget offload for non-critical side-effects: `background.submit(fn, *args, **kwargs)` enqueues a synchronous callable onto one in-process `asyncio.Queue` drained by a single consumer, so the work leaves the request path. It is **per-worker** (started in `main.py` `startup()` for every worker outside the service-lock branch, drained in `shutdown()`), unlike `BaseService`/`JobService` which run only in the lock owner - audit records are produced in every worker and a DB-backed queue would only add writes. When the consumer is not running (tests with `DEVPLACE_DISABLE_SERVICES=1`, unit tests, full queue), `submit` runs the callable **inline**, so audit/notification/XP writes stay deterministic for the suite while production defers them. Durability is best-effort in-memory (graceful shutdown drains; a hard crash drops unflushed items), which fits audit/notifications already never raising. Deferral happens at the **canonical choke points** so callers need no change: audit-log persistence (`services/audit/record.py` `_write`), **every XP award** (`utils.award_rewards` -> `_apply_rewards`, covering posts/comments/projects/gists/follow/votes), and **every notification** (`utils.create_notification` -> `_deliver_notification`, the single funnel). Because the reward and notification funnels self-defer, handlers call `award_rewards`/`create_notification` directly - never wrap them in `background.submit` (that double-queues). Handler pattern for a NEW side-effect: do the user-visible write inline, then `background.submit(...)` the rest. See `AGENTS.md` -> "Background task queue". **Feature/first-use badges** ride the same self-deferring model: call `utils.track_action(user_uid, action[, target])` inline at a feature's success point (do NOT wrap it - it `background.submit`s itself), where `action` is a key in `utils.ACHIEVEMENTS` (`action -> [(threshold, badge), ...]`, threshold 1 = first use; `UNIQUE_ACTIONS` use `record_unique_activity` for "N distinct things" like `docs.read`). Source-derivable milestones (counts of existing tables) instead go in `check_milestone_badges`/`_COUNT_MILESTONES`. Badge metadata + `group` live in `BADGE_CATALOG`; the profile shows a grouped earned/locked **Achievements** showcase via `build_achievements`. See `AGENTS.md` -> "Gamification".
|
||||
|
||||
### AI content correction (`devplacepy/services/correction.py`)
|
||||
**Opt-in, default off, server-side.** When a user turns it on (profile **AI content correction** block, owner-only, saved at `POST /profile/{username}/ai-correction`), the prose they author is rewritten by the AI gateway. The per-user `ai_correction_sync` flag (0 = background default, 1 = sync) picks the **apply mode**: background rewrites the stored fields a moment after the write (never slows the request); sync makes the HTTP response wait until the correction is applied. **Sync must never block the event loop:** the correction POSTs to the in-process gateway (`localhost:{PORT}`) and the content helpers are synchronous on the loop thread, so a blocking inline call self-deadlocks the (single) worker against its own gateway request. Instead, sync runs `_run_correction` via `loop.run_in_executor` (loop stays free), stashes the future on `request.scope[PENDING_SCOPE_KEY]`, and the `await_pending_corrections` middleware in `main.py` awaits it after the handler. `CORRECTABLE_FIELDS` is the single source of truth mapping each table to its prose columns (`posts`->title/content, `projects`/`gists`->title/description, `comments`/`messages`->content, `users`->bio); `gists.source_code`, project files and Gitea issues are intentionally excluded (code and external systems). The choke entrypoint `schedule_correction(user, table, uid)` is a no-op unless correction is enabled and the user has an `api_key`, then it runs `_run_correction` off-thread via `loop.run_in_executor` with the future awaited by middleware (sync mode) or via `background.submit` (default), which reads the row back, calls the fail-soft synchronous `correct_text` (POST to `INTERNAL_GATEWAY_URL` via `stealth.stealth_sync_client`, authenticated with the user's OWN api_key for per-user attribution, returning the original on any error or suspiciously large output), and writes only changed fields without touching `updated_at` or the slug. It is hooked at `content.create_content_item`/`edit_content_item`/`create_comment_record`/`edit_comment_record`, `messaging/persist.persist_message`, `profile/index.update_profile`, and the two devRant direct-edit paths, so it applies identically across the web UI, REST/JSON API, Devii and devRant. Settings are three `users` columns (`ai_correction_enabled`/`ai_correction_sync`/`ai_correction_prompt`, ensured in `database.backfill_api_keys()`, seeded born-live in `utils._create_account`); Devii drives it via the owner-scoped `ai_correction_get`/`ai_correction_set` tools. Keep `correction.py` free of `content`/`utils` top-level imports (cycle); `clear_user_cache` is imported lazily. **Per-user usage aggregation:** `correct_text` returns `(text, usage)` parsed from the gateway `X-Gateway-*` headers on success, now including the timing headers `X-Gateway-Upstream-Latency-Ms`/`X-Gateway-Total-Latency-Ms`; `_run_correction` accumulates the fields' usage into a `totals` dict and, only when a call succeeded, makes ONE atomic `INSERT ... ON CONFLICT DO UPDATE SET col = col + excluded.col` upsert via `database.add_correction_usage(user_uid, totals)` (single dict arg) into the per-user `correction_usage` table (running SUMS `calls`/token/`cost_usd` plus the two latency columns, separate from `users` so it never busts the auth cache). `database.get_correction_usage` also returns computed AVERAGES (avg tokens/call, avg latency, avg tokens/sec, total time). The profile shows the sums plus those performance averages (`_correction_usage`, `correction_usage` on context + `ProfileOut`) to owner-or-admin; the dollar `cost_usd`/`avg_cost_usd` are admin-only in both HTML and JSON (mirrors `_ai_quota`). See `AGENTS.md` -> "AI content correction".
|
||||
|
||||
#### AI modifier (`devplacepy/services/ai_modifier.py`)
|
||||
A sibling of AI content correction that reuses its plumbing (`CORRECTABLE_FIELDS`, `PENDING_SCOPE_KEY`, `gateway_complete`, the sync/background apply-mode mechanics, the per-user usage upsert) and differs only in two ways: it is **enabled by default** and **synchronous by default**, and it runs ONLY where an authored prose field contains an inline `@ai <instruction>` directive (`has_ai_directive`, regex `@ai\s+\S`). When present, the configured prompt (default `config.DEFAULT_MODIFIER_PROMPT` = "Execute what is behind `@ai` (the prompt) and replace that part including `@ai`") tells the model to execute the instruction and replace the marked part including the `@ai` marker; triggerless writes cost nothing. **It is context-aware:** `_run_modification` builds a grounding **context block** once per row via `services/ai_context.py` `build_context(table, uid, row, user_uid)` and passes it to `modify_text`, which appends it to the system message. The block (fail-soft, length-capped) carries the date, the author's stats (username/role/level/stars/posts/rank/followers/member-since/bio), and table-specific location/thread context (a comment's target post + parent reply; a post's topic/project; a project/gist's title/description, and a gist's language + read-only `source_code`; a DM's recipient + recent conversation), so `@ai answer the question above` / `@ai write my bio from my stats` / `@ai reply to this` work. `schedule_modification(user, table, uid, request)` is hooked alongside `schedule_correction` at the same content/comment/messaging/profile entrypoints. Settings are three `users` columns (`ai_modifier_enabled`/`ai_modifier_sync`/`ai_modifier_prompt`, seeded born-live in `utils._create_account`), edited at the owner-or-admin leaf `POST /profile/{username}/ai-modifier` (`routers/profile/ai_modifier.py`, `AiModifierForm`, audit `profile.ai_modifier`), surfaced owner-only on the profile (`ai_modifier_*` + `modifier_usage` on context + `ProfileOut`, dollar cost admin-only), wired by `static/js/AiModifier.js` (`app.aiModifier`), and driven via the Devii `ai_modifier_get`/`ai_modifier_set` tools (`services/devii/ai_modifier/`, `handler="ai_modifier"`). Per-user totals accumulate via `database.add_modifier_usage(user_uid, totals)` into the `modifier_usage` table (same SUMS-plus-latency columns and computed performance averages as correction, shown on the profile with dollar averages admin-only). See `AGENTS.md` -> "AI modifier".
|
||||
|
||||
### Background services (`devplacepy/services/`)
|
||||
`BaseService` provides the async run loop, a `deque(maxlen=20)` log buffer, and graceful cancellation. `ServiceManager` is a singleton that registers, starts, and stops services. `NewsService` fetches articles from `news_api_url`, grades each via `news_ai_url`, and inserts ALL of them into `news` with `status="published"` or `"draft"` based on the configurable `news_grade_threshold` - nothing is silently skipped. The `/admin/services` page polls live status and the log tail.
|
||||
|
||||
`GatewayService` (`/openai/v1/*`) is the **single point of truth for AI**. Every other AI consumer (news, bots, Devii guests) defaults its endpoint to `config.INTERNAL_GATEWAY_URL` (`http://localhost:10500/openai/v1/chat/completions`, override via `DEVPLACE_INTERNAL_BASE_URL`), sends the generic model `molodetz`, and authenticates with the auto-generated `gateway_internal_key` (via `database.internal_gateway_key()`) when its own key is unset. **Per-user attribution:** the gateway also accepts a real user's `api_key` (Bearer / X-API-KEY / session), gated by `gateway_allow_users` (default on); `resolve_owner()` records `(owner_kind, owner_id)` per call, so usage is attributed and limitable per user. Devii operating a signed-in user authenticates its LLM calls with that user's own `api_key` (set as the session's `ai_key` in `build_settings(..., owner_kind="user")`), so a user's full gateway spend (Devii and direct API calls) rolls up under their uid - surfaced admin-only on the profile page via `build_user_usage()` / `GET /admin/users/{uid}/ai-usage`. Guests keep the internal key. **System-message composition (`services/openai_gateway/system_message.py`, `handle_chat` only):** before assembling the upstream chat payload (after vision augmentation, so it reaches every chat consumer) the gateway composes ONE system message in the fixed order **operator preamble -> date line -> client system content**. (1) The admin-editable `gateway_system_preamble` config field (Prompt group, multiline text, default empty) is prepended ahead of the client's system content on every chat call; if the client sent no system message it becomes a new leading one (a blank preamble is a no-op). (2) The **current date in EU `DD/MM/YYYY` (date only, never time** - omitting time keeps the message byte-stable per day so upstream prompt caching stays effective) is injected only when the composed text has NO date in any common format, detected by the `contains_date` regex set (ISO `YYYY-MM-DD`, `D/M/YYYY`, `DD-MM-YYYY`, `DD.MM.YYYY`, `MM/DD/YYYY`, and textual months like `14 June 2026`/`June 14, 2026`/`14 Jun 2026`). Embeddings/passthrough are untouched. Real provider keys/URLs/models live only in the gateway; `database.migrate_ai_gateway_settings()` (run in `init_db()`) generates the internal key, migrates `DEEPSEEK_API_KEY`/`OPENROUTER_API_KEY` env into the editable, non-secret `gateway_api_key`/`gateway_vision_key` settings, and rewrites old external AI URLs to the gateway. The gateway is `default_enabled=True`. The gateway also serves OpenAI-compatible text **embeddings** at `POST /openai/v1/embeddings`: clients send the client model `molodetz~embed`, which `handle_embeddings` remaps to the configured embedding model (default `qwen/qwen3-embedding-8b` on OpenRouter, $0.01/1M input tokens), tracked per call as `backend="embed"` in `gateway_usage_ledger`. The gateway records every upstream call to `gateway_usage_ledger` (cost from the upstream native `cost` field when present, else computed from per-1M pricing) and samples `gateway_concurrency_samples`; `services/openai_gateway/analytics.py` rolls these into per-hour/24h token, cost, latency, error, and behavior metrics shown on `/admin/ai-usage` (`AiUsageMonitor.js`, schema `GatewayUsageOut`, Devii action `ai_usage`). It also retries transient upstream failures and trips a circuit breaker (`services/openai_gateway/reliability.py`). TTFT/inter-token latency are not tracked because the upstream is called non-streaming.
|
||||
`GatewayService` (`/openai/v1/*`) is the **single point of truth for AI**. Every other AI consumer (news, bots, Devii guests) defaults its endpoint to `config.INTERNAL_GATEWAY_URL` (`http://localhost:10500/openai/v1/chat/completions`, override via `DEVPLACE_INTERNAL_BASE_URL`), sends the generic model `molodetz`, and authenticates with the auto-generated `gateway_internal_key` (via `database.internal_gateway_key()`) when its own key is unset. **Per-user attribution:** the gateway also accepts a real user's `api_key` (Bearer / X-API-KEY / session), gated by `gateway_allow_users` (default on); `resolve_owner()` records `(owner_kind, owner_id)` per call, so usage is attributed and limitable per user. Devii operating a signed-in user authenticates its LLM calls with that user's own `api_key` (set as the session's `ai_key` in `build_settings(..., owner_kind="user")`), so a user's full gateway spend (Devii and direct API calls) rolls up under their uid - surfaced admin-only on the profile page via `build_user_usage()` / `GET /admin/users/{uid}/ai-usage`. Guests keep the internal key. **System-message composition (`services/openai_gateway/system_message.py`, `handle_chat` only):** before assembling the upstream chat payload (after vision augmentation, so it reaches every chat consumer) the gateway composes ONE system message in the fixed order **operator preamble -> date line -> client system content**. (1) The admin-editable `gateway_system_preamble` config field (Prompt group, multiline text, default empty) is prepended ahead of the client's system content on every chat call; if the client sent no system message it becomes a new leading one (a blank preamble is a no-op). (2) The **current date in EU `DD/MM/YYYY` (date only, never time** - omitting time keeps the message byte-stable per day so upstream prompt caching stays effective) is injected only when the composed text has NO date in any common format, detected by the `contains_date` regex set (ISO `YYYY-MM-DD`, `D/M/YYYY`, `DD-MM-YYYY`, `DD.MM.YYYY`, `MM/DD/YYYY`, and textual months like `14 June 2026`/`June 14, 2026`/`14 Jun 2026`). Embeddings/passthrough are untouched. Real provider keys/URLs/models live only in the gateway; `database.migrate_ai_gateway_settings()` (run in `init_db()`) generates the internal key, migrates `DEEPSEEK_API_KEY`/`OPENROUTER_API_KEY` env into the editable, non-secret `gateway_api_key`/`gateway_vision_key` settings, and rewrites old external AI URLs to the gateway. The gateway is `default_enabled=True`. The gateway also serves OpenAI-compatible text **embeddings** at `POST /openai/v1/embeddings`: clients send the client model `molodetz~embed`, which `handle_embeddings` remaps to the configured embedding model (default `qwen/qwen3-embedding-8b` on OpenRouter, $0.01/1M input tokens), tracked per call as `backend="embed"` in `gateway_usage_ledger`. The gateway records every upstream call to `gateway_usage_ledger` (cost from the upstream native `cost` field when present, else computed from per-1M pricing) and samples `gateway_concurrency_samples`; `services/openai_gateway/analytics.py` rolls these into per-hour/24h token, cost, latency, error, and behavior metrics shown on `/admin/ai-usage` (`AiUsageMonitor.js`, schema `GatewayUsageOut`, Devii action `ai_usage`). It also retries transient upstream failures and trips a circuit breaker (`services/openai_gateway/reliability.py`). TTFT/inter-token latency are not tracked because the upstream is called non-streaming. **Every gateway response also carries per-call `X-Gateway-*` headers** (model, full token breakdown, `Cost-USD`/input/output dollars, native-cost flag, tokens/sec, latency, context window/utilization): `GatewayUsageLedger.record` returns the computed ledger row and each handler's `finalize` maps it through `usage.usage_response_headers` onto the `Response`, so any caller can read its own spend (the AI-correction worker uses this).
|
||||
|
||||
To add a service: extend `BaseService`, override `async def run_once(self) -> None`, register in `main.py`'s startup with `service_manager.register(YourService())`. It auto-appears on the services page.
|
||||
|
||||
@@ -188,7 +198,10 @@ Devii's project filesystem tools include surgical **line-range editing** (`proje
|
||||
Users and guests inject their own **CSS** (look) and **JS** (behaviour), scoped to a **page type** or **globally**, configured **only through Devii** (no HTTP routes - persistence is owner-scoped from the trusted Devii session, like `LessonStore`). It is **per-owner only**: the code runs solely in that owner's own browser sessions (self-XSS, like a userscript), never in anyone else's view - never key injection off URL-supplied data. **Page type** = the matched route template (`request.scope["route"].path`, e.g. `/posts/{slug}`) via `page_type_for(request)`; one value covers all instances of a type (all posts), never one post; also rendered as `data-page-type` on `<main>` and surfaced to Devii via `DeviiClient._context().pageType`. **Owner** = `owner_for(request)`: user uid, else the `DEVII_GUEST_COOKIE` (`constants.py`, shared with `routers/devii.py`), else none. **Storage** (`database.py`): table `user_customizations` (per `owner_kind`/`owner_id`/`scope`/`lang`) with helpers `get_custom_overrides` (merges global-then-page-type, cached under the `"customizations"` cache-version name via `sync_local_cache`/`bump_cache_version`), `set_custom_override`, `delete_custom_override`. **Injection**: template globals `custom_css_tag`/`custom_js_tag` (registered in `templating.py`) emit a `<style id="user-custom-css">` after `extra_head` (CSS overrides the design system; `</`->`<\/` neutralizes breakout) and a JSON-island `<script>` after `Application.js` (`<`/`>` escaped, run via `new Function(JSON.parse(...))` so `app` exists). Both fail closed to empty Markup - a customization issue must never break page render. Kill-switches: `customization_enabled`, `customization_js_enabled`. **Devii** (`services/devii/customization/`, `handler="customization"`): `customize_list`/`get`/`set_css`/`set_js`/`reset`, all `requires_auth=False`; `CustomizationController(owner_kind, owner_id)` is built in `Dispatcher.__init__` (owner threaded from `DeviiSession`); set/reset are in `CONFIRM_REQUIRED` so `confirmation_error` forces Devii to ask **page-type vs global** before `confirm=true`. Devii previews live with `run_js` (ids `devii-preview-css`/`devii-preview-js`) and `reload_page` after saving.
|
||||
|
||||
### Notification preferences (`database.py`)
|
||||
Every notification type (`NOTIFICATION_TYPES`) is independently toggleable per user on two channels (in-app, push). Resolution `notification_enabled(user_uid, type, channel)` = live `notification_preferences` override -> admin global default (`notif_default_{type}_{channel}` setting, default `1`) -> on. Enforcement is the single funnel `utils.create_notification` (in-app insert gated by the `in_app` channel, `_schedule_push` by `push`); the `messages.py` DM path was routed through `create_notification` so it is gated too. **`create_notification` defers its work to the background task queue** (`background.submit(_deliver_notification, ...)`), so the preference reads + in-app insert + push schedule + audit all run off the request path (inline in tests). See `### Background task queue`. UI: profile **Notifications** tab (`/profile/{username}?tab=notifications`, owner-or-admin, `POST /profile/{username}/notifications` + `/reset`) and admin global defaults at `/admin/notifications`. Devii tools `notification_list`/`notification_set`/`notification_reset` (`handler="notification"`, owner-scoped, `set`/`reset` in `CONFIRM_REQUIRED`). Storage is the soft-deletable `notification_preferences` table, cached under the `"notif_prefs"` cache-version name. Reuse this canonical-list + resolver + override-table + single-funnel pattern for future per-user per-channel preferences.
|
||||
Every notification type (`NOTIFICATION_TYPES`) is independently toggleable per user on two channels (in-app, push). Resolution `notification_enabled(user_uid, type, channel)` = live `notification_preferences` override -> admin global default (`notif_default_{type}_{channel}` setting, default `1`) -> on. Enforcement is the single funnel `utils.create_notification` (in-app insert gated by the `in_app` channel, `_schedule_push` by `push`); the `messages.py` DM path was routed through `create_notification` so it is gated too. **`create_notification` defers its work to the background task queue** (`background.submit(_deliver_notification, ...)`), so the preference reads + in-app insert + push schedule + audit all run off the request path (inline in tests). See `### Background task queue`. UI: profile **Notifications** tab (`/profile/{username}?tab=notifications`, owner-or-admin, `POST /profile/{username}/notifications` + `/reset`) and admin global defaults at `/admin/notifications`. Devii tools `notification_list`/`notification_set`/`notification_reset` (`handler="notification"`, owner-scoped, `set`/`reset` in `CONFIRM_REQUIRED`). Storage is the soft-deletable `notification_preferences` table, cached under the `"notif_prefs"` cache-version name. Reuse this canonical-list + resolver + override-table + single-funnel pattern for future per-user per-channel preferences. **Live toasts ride the `in_app` channel:** `services/notification_relay.py` `NotificationRelayService` (lock-owner `BaseService`, like `MessageRelay`) polls `notifications.id > watermark` and publishes each new row to the pub/sub topic `user.{uid}.notifications`; because it runs on the lock owner where every `/pubsub/ws` subscriber converges, the in-process `services.pubsub.publish` reaches the recipient, and because the row exists only when `in_app` is enabled the toast fires exactly for enabled notifications (no new channel). Frontend `static/js/LiveNotifications.js` (`app.liveNotifications`) reads `<body data-user-uid>`, subscribes via `app.pubsub`, and raises a click-through `app.toast`. See AGENTS.md -> "Notification preferences". **The same relay tick also publishes fresh unread counts** `{notifications, messages}` to `user.{uid}.counts` (one publish point covers new notifications AND new DMs, since a DM inserts a `message`-type notification row); `static/js/CounterManager.js` (`app.counters`, constructed with `this.pubsub`) subscribes for instant badge bumps and keeps a 60s `/notifications/counts` poll as a decrement/missed-frame fallback.
|
||||
|
||||
### Live view relay (`devplacepy/services/live_view_relay.py`)
|
||||
`LiveViewRelayService` is a second lock-owner pub/sub bridge (default-enabled, 1s interval, registered in `main.py`) that **replaces per-client HTTP polling on the admin live views with server push, computing a snapshot only for topics that currently have subscribers**. Each tick it reads `pubsub.topics()`, matches concrete subscribed topics against the `VIEWS` registry `(regex, async compute, min_interval)`, throttles per topic, and publishes the **same payload shape the matching HTTP endpoint already returns** (so frontend render code is unchanged). Topics + cadence: `container.list` (4s), `project.{slug}.containers` (3s), `container.{uid}.detail` (4s), `container.{uid}.logs` (3s), `fleet.bots` (2s), `admin.services` (5s), `admin.services.{name}` (5s), `admin.ai-usage.{hours}` (15s) - all admin-only by pub/sub policy. The frontend monitors (`ContainerInstance`/`ContainerList`/`ContainerManager`/`BotMonitor`/`ServiceMonitor`/`AiUsageMonitor`) subscribe via `window.app.pubsub` and keep a lengthened (15-20s) HTTP poll as initial-load + fallback. To add an admin live view: add one `VIEWS` row returning the endpoint payload and subscribe on the frontend. Keep durable/ordered/stateful/raw-I/O channels (messages, Devii, SEO/DeepSearch progress, container PTY) OFF pub/sub. See `pubreport.md` and AGENTS.md -> "Live view relay".
|
||||
|
||||
### Devii user-defined tools (`services/devii/virtual_tools/`)
|
||||
Users invent new Devii tools in natural language ("when I say woeii, do Y"); each is stored per-owner and dynamically added to Devii's LLM tool list, and when called its handler **re-prompts Devii itself** (a self-eval sub-agent) with the stored prompt plus the user's single free-form `input`. CRUD is **Devii-only** (`tool_create`/`tool_list`/`tool_get`/`tool_update`/`tool_delete`, `handler="virtual_tool"`). **Dynamic tool list:** the session tool list is normally frozen, but `DeviiSession._refresh_tools()` runs at the top of every `_run_turn` and rewrites `self.tools` **in place** (`self.tools[:] = CATALOG.tool_schemas_for(...) + virtual_store.tool_schemas()`); since `Agent`/`AgenticController`/`react_loop` share that list object, new/edited/deleted tools take effect next turn with no rebuild - never reassign `self.tools`, always mutate in place. **Self-eval engine:** `agentic/controller.py` `_spawn` (shared by `_delegate`, `run_subagent`, and the new `eval` tool) runs `react_loop` under a contextvar **depth guard** (`agentic/state.py`, `MAX_EVAL_DEPTH=2`) so self-calls can't loop. **Store** `VirtualToolStore(db, owner_kind, owner_id)` over `devii_virtual_tools` (persistent for users, `memory_db()` for guests, built in `hub.get_or_create`). **Dispatch:** the dispatcher's unknown-name branch (`if action is None`) resolves a virtual tool via `self._virtual_tools.has/run` before erroring, so virtual tools live only in the store, not the static catalog. Name validation rejects built-in collisions (`set(CATALOG.by_name())`).
|
||||
@@ -210,7 +223,7 @@ A second REST surface mounted at `/api` that mimics the public devRant API on De
|
||||
- **Form validation is Pydantic-native.** Routers declare a typed body param `data: Annotated[SomeForm, Form()]` (models live in `models.py`); FastAPI validates it and the global `RequestValidationError` handler in `main.py` re-renders auth pages with messages (400) or redirects other routes. Read raw `await request.form()` only when also reading an uploaded file alongside a model - adding a `File()` param would embed the model under its parameter name.
|
||||
- **`RedirectResponse(url=..., status_code=302)`** - always pass `status_code` as an integer literal, never `status.HTTP_302_FOUND`.
|
||||
- **Ownership checks:** compare `resource["user_uid"] == user["uid"]` before edit/delete (use `content.is_owner`). **Deletes additionally allow any admin:** every content delete guards `is_owner(...) or is_admin(user)` on the endpoint (posts/gists/projects in `content.delete_content_item`, `comments.delete_comment`, `project_files.project_file_delete`, `media.delete_media`, `uploads.delete_attachment_route`; news is admin-only), so an administrator (including via Devii, which only calls the API) may delete any member's content while members are limited to their own. Edits stay owner-only. Deletes are **soft** (`deleted_at`/`deleted_by`) and cascade comments → votes → engagement → resource under one shared stamp.
|
||||
- **All dates shown to users are DD/MM/YYYY** (European). Use `format_date()` from `utils.py` (template global), or `time_ago()` which auto-switches to DD/MM/YYYY past 30 days.
|
||||
- **All dates shown to users are DD/MM/YYYY** (European) and rendered **in the viewer's own timezone, client-side**. Timestamps are stored/emitted as UTC ISO; the display is localized in the browser. Use the `local_dt(iso, mode)` / `dt_ago(iso)` Jinja globals (`templating.py`) for any user-facing **instant** - they emit `<time datetime="UTC-ISO" data-dt data-dt-mode="date|datetime|ago">server-fallback</time>`, and `static/js/LocalTime.js` (`app.localTime`) reformats every `[data-dt]` to the browser's local timezone (`Intl`), with a `MutationObserver` so dynamically-inserted dates (feed, messages, comments) localize automatically and a 60s refresh for relative times. The server fallback (the `<time>` text content, via `format_date`/`time_ago`) is what no-JS clients see. **`format_date()`/`time_ago()` stay as plain-text helpers** for JSON responses, no-JS fallbacks, and the few non-timestamp or attribute-context dates (e.g. project `release_date`/`demo_date`, which are user-entered DD/MM/YYYY strings, and `value=`/`title=` attribute uses) - do NOT wrap those in `local_dt`. Pattern for converting a precomputed `time_ago` string display: `{{ dt_ago(x.created_at) if x.created_at else x.time_ago }}` (falls back to the original string if the raw ISO is absent). New user-facing timestamp displays should use `local_dt`/`dt_ago`.
|
||||
- **Slug + UUID lookup:** resources with slugs (posts, gists, news, projects) accept either the slug or the bare UUID - see `resolve_by_slug()` in `database.py`. Slugs are generated via `make_combined_slug(title, uid)` which prefixes the last 12 hex chars of the UUID (the uuid7 **random tail**, NOT its leading bytes - the first 48 bits are a millisecond timestamp, so a head prefix is identical for all rows written in the same ~65s window and collides with the same title, making `resolve_by_slug` ambiguous; same reasoning as the blob-shard tail rule).
|
||||
- **Username:** 3-32 chars, letters/numbers/hyphens/underscores. **Password:** min 6 chars.
|
||||
- **Roles are stored capitalized:** the `users.role` value is exactly `"Admin"` or `"Member"` (the first registered user becomes `"Admin"`; `auth.py`). Always test admin-ness through the `is_admin(user)` global (`utils.py`, also a Jinja global), which compares `user.get("role") == "Admin"` case-sensitively - never hand-roll a lowercase compare. The **only** lowercase surface is the CLI (`devplace role set <username> <member|admin>` accepts lowercase but writes `role.capitalize()`; `role get` prints `.lower()` for display). A lowercase role in the DB silently fails every admin check.
|
||||
|
||||
Reference in New Issue
Block a user