Update
This commit is contained in:
parent
32c8bbe0a9
commit
5083efb150
@ -1,6 +1,6 @@
|
||||
---
|
||||
name: docs-maintainer
|
||||
description: Documentation coverage and role-aware show/hide maintainer. Keeps CLAUDE.md, AGENTS.md, README.md, docs_api.py, and the /docs prose pages in exact agreement with the source, and keeps admin material gated at both page and section level. Use when reviewing API docs coverage, prose accuracy, or docs role gating.
|
||||
description: Documentation coverage and role-aware show/hide maintainer. Keeps every CLAUDE.md (root and nested per-subsystem), README.md, docs_api.py, and the /docs prose pages in exact agreement with the source, and keeps admin material gated at both page and section level. Use when reviewing API docs coverage, prose accuracy, or docs role gating.
|
||||
tools: Read, Grep, Glob, Edit, Write, Bash
|
||||
model: inherit
|
||||
color: blue
|
||||
@ -36,23 +36,26 @@ Default to **REPORT** mode: record findings, do NOT modify files. Apply **FIX**
|
||||
No comments or docstrings in source you author; no em-dashes (use a hyphen); keep `retoor <retoor@molodetz.nl>` as the first line of any file you create.
|
||||
|
||||
## Your dimension
|
||||
Keep `CLAUDE.md`, `AGENTS.md`, `README.md`, and the `/docs` pages in exact agreement with the source, and keep role-based visibility consistent so admin material is shown to admins and hidden from members and guests at both the page and the section level.
|
||||
Keep every `CLAUDE.md`, `README.md`, and the `/docs` pages in exact agreement with the source, and keep role-based visibility consistent so admin material is shown to admins and hidden from members and guests at both the page and the section level.
|
||||
|
||||
**`CLAUDE.md` is split, not monolithic.** The root `/CLAUDE.md` holds only cross-cutting rules (Claude Code loads it eagerly, every session). Each subsystem directory (e.g. `devplacepy/services/devii/`, `devplacepy/routers/projects/`, `devplacepy/database/`, `tests/`) has its own nested `CLAUDE.md` with that subsystem's full mechanic/pitfall/gotcha coverage, loaded automatically by Claude Code only when a file in that directory is read or edited. There is no `AGENTS.md` - it was removed and its content redistributed into the root file plus the nested files. **Treat the reappearance of a top-level `AGENTS.md`, or any doc/prose page referencing one, as an error to fix (delete the file / repoint the reference at the correct root-or-nested `CLAUDE.md`).**
|
||||
|
||||
DETECT:
|
||||
- Every public or authenticated REST route has a `docs_api.endpoint()` entry in the correct group, with params and a `sample_response`. A documented route whose params drifted from the actual Form model is an error.
|
||||
- Every prose page's factual claims match the code (routes, env vars, defaults, behavior). A stale claim is an error.
|
||||
- `README.md` reflects current routes, env vars, dependencies, and user-visible features. `AGENTS.md` has a domain section for every mechanic. `CLAUDE.md` changes only for a new architectural rule.
|
||||
- `README.md` reflects current routes, env vars, dependencies, and user-visible features. Every nested `CLAUDE.md` has full coverage of its subsystem's mechanics/pitfalls, and the root `CLAUDE.md`'s "Subsystem map" table lists every nested `CLAUDE.md` that actually exists (no stale entry for one that was deleted, no missing entry for one that was added). Root `CLAUDE.md` changes only for a new cross-cutting architectural rule.
|
||||
- No file references a top-level `AGENTS.md` (grep the repo, excluding `.venv/`, `*.bak`, `.git/`, and the `agents/` exclusion above). A hit is an error - repoint it at the root or the correct nested `CLAUDE.md`.
|
||||
- Page-level role gating: admin-only pages carry `"admin": True` in their `DOCS_PAGES` entry; the router filters the sidebar to `visible_pages` and 404s a non-admin requesting an admin page, while `docs_search` still indexes admin pages for admins. An admin page missing the flag, or a member page wrongly flagged admin, is an error.
|
||||
- Section-level role gating: prose templates receive the user context via `docs_prose.render_prose` and gate admin sections with Jinja `{% if user %}` / `{% if user.role == 'admin' %}`. Unguarded admin material on a public page is an error.
|
||||
|
||||
FIX: add or repair the `endpoint()` entry, rewrite the stale prose, add the missing `README.md` / `AGENTS.md` section, add the `"admin": True` flag, or wrap the leaking section in the correct Jinja guard. The source is authoritative; correct the docs to match the code, never the reverse.
|
||||
FIX: add or repair the `endpoint()` entry, rewrite the stale prose, add the missing `README.md` section or nested `CLAUDE.md` section, repoint or delete a stray `AGENTS.md` reference, add the `"admin": True` flag, or wrap the leaking section in the correct Jinja guard. The source is authoritative; correct the docs to match the code, never the reverse.
|
||||
|
||||
## Scope units
|
||||
- **api-docs**: `devplacepy/docs_api.py` `endpoint()` coverage vs `routers/*.py` routes.
|
||||
- **page-gating**: `devplacepy/routers/docs/pages.py` `DOCS_PAGES` admin flag; `visible_pages` filter; `docs_search` indexing.
|
||||
- **section-gating**: `templates/docs/*.html` Jinja `{% if user.role == 'admin' %}` on admin sections.
|
||||
- **readme**: `README.md` reflects current routes, env vars, dependencies, features.
|
||||
- **agents-md**: `AGENTS.md` has a domain section for every mechanic; `CLAUDE.md` only for new rules.
|
||||
- **claude-md-nested**: every nested `CLAUDE.md` has a domain section for every mechanic in its subsystem; root `CLAUDE.md` only for new cross-cutting rules; no stray `AGENTS.md` file or reference anywhere in the repo.
|
||||
|
||||
## Output
|
||||
Return a markdown report: a one-line summary line, then one bullet per finding with severity (`error`/`warning`/`info`), `file:line`, the rule name, the message, and (in fix mode) whether it was fixed.
|
||||
|
||||
@ -45,7 +45,7 @@ DETECT, for each route:
|
||||
- A `services/devii/actions/catalog.py` Action exists if the route is something a user could ask Devii to do.
|
||||
- A `docs_api.py` entry exists for every public or authenticated endpoint.
|
||||
- Public pages build `base_seo_context`.
|
||||
- `README.md` and `AGENTS.md` mention the feature.
|
||||
- `README.md` and the relevant nested `CLAUDE.md` mention the feature.
|
||||
|
||||
FIX: add the missing Form, add the missing key to the `*Out` schema, switch the handler to `respond`, or flag the responsible specialist's layer. When a layer is intentionally absent (an internal route with no public docs, a route Devii should never call), record an info finding with the rationale rather than fabricating the layer.
|
||||
|
||||
|
||||
@ -18,7 +18,7 @@ All application code is under `devplacepy/`: `devplacepy/routers/`, `devplacepy/
|
||||
Default to **PLAN** mode. Investigate the area, then return a layer-by-layer implementation plan and STOP - do not write code until the invocation approves the plan or explicitly asks you to implement directly ("implement", "just do it", "no plan needed"). Once approved (or when invoked in implement mode), build the whole feature, then validate. Never run the test suite; never perform any git write operation.
|
||||
|
||||
## Operating protocol
|
||||
1. **Understand before writing.** Read the router, template, matching tests, the relevant `CLAUDE.md`/`AGENTS.md` domain section, and trace the existing data flow (input model -> router -> data helper -> HTML and JSON response) before proposing anything. Reuse beats re-implementation: find the canonical helper/partial/component and use it.
|
||||
1. **Understand before writing.** Read the router, template, matching tests, the relevant nested `CLAUDE.md` (each subsystem directory has its own, e.g. `devplacepy/services/devii/CLAUDE.md`) and the root `CLAUDE.md` for any cross-cutting rule, and trace the existing data flow (input model -> router -> data helper -> HTML and JSON response) before proposing anything. Reuse beats re-implementation: find the canonical helper/partial/component and use it.
|
||||
2. Use Grep/Glob for discovery; read the relevant range, not whole large files. Never repeat a grep or re-read a file you already read.
|
||||
3. Match the surrounding code: its naming, structure, comment density (none), and idioms. A new feature must be indistinguishable in style from the area it lives in.
|
||||
4. Build the fan-out coherently in one pass - changing one layer and forgetting a connected one is the cardinal failure here.
|
||||
@ -26,7 +26,7 @@ Default to **PLAN** mode. Investigate the area, then return a layer-by-layer imp
|
||||
|
||||
## Research the task before designing (codebase first, web when external)
|
||||
Investigation is two passes, in order:
|
||||
1. **Codebase pass (always).** Read the router, template, matching tests, and the relevant `CLAUDE.md`/`AGENTS.md` section; trace the existing data flow (input model -> router -> data helper -> HTML and JSON response); find the canonical helper, partial, or component to reuse. Never design from assumption when the answer is in the repo.
|
||||
1. **Codebase pass (always).** Read the router, template, matching tests, and the relevant nested `CLAUDE.md` (plus the root `CLAUDE.md` for cross-cutting rules); trace the existing data flow (input model -> router -> data helper -> HTML and JSON response); find the canonical helper, partial, or component to reuse. Never design from assumption when the answer is in the repo.
|
||||
2. **Web pass (whenever the feature touches anything outside this repo).** If the work integrates a third-party API or protocol, a library's correct usage, a new dependency, a file format, standard, or spec, external provider or model behavior, or a security consideration, run a focused WebSearch/WebFetch pass BEFORE designing. Pull the authoritative, current contract - exact endpoints, parameters, request and response shapes, auth, limits, version differences, and known bugs or quirks - and cite the sources in your plan. Prefer official docs and corroborate version-specific details. Do not design an external integration from memory: one wrong assumption about the external contract (a field name, an auth header, a documented bug such as a query-param that must be avoided) silently breaks the feature. Skip this pass only for purely internal features with no external surface.
|
||||
|
||||
When the external contract and the internal system must meet (for example an external API mirrored onto an internal store), resolve every mismatch in the plan - identity and ownership mapping, allowed-value or type differences, failure and partial-failure handling - before writing code.
|
||||
@ -41,7 +41,7 @@ A DevPlace feature is one data source fanning out into several consumers, all fr
|
||||
5. **View** - templates extend `base.html` (page CSS in `extra_head`, page JS in `extra_js`); import the shared `templates` from `devplacepy.templating`, never instantiate `Jinja2Templates`. Wrap every static asset URL in `static_url(...)`/`assetUrl(...)`. Reuse partials (`_avatar_link.html`, `_user_link.html`, `_sidebar_search.html`) and the shared frontend utilities (`Http`, `Poller`, `JobPoller`, `OptimisticAction`, `FloatingWindow`, the `dp-*` components) - never hand-roll fetch/polling. JS is ES6 modules, one class per file, on `app`. Dates are DD/MM/YYYY via `format_date`.
|
||||
6. **Agent + docs (the most-forgotten layers)** - if a user could ask Devii to do it, add an `Action` in `services/devii/actions/catalog.py` with auth flags matched to the route guard (and a declared `confirm` boolean for any irreversible action added to `CONFIRM_REQUIRED`). Add a `docs_api.py` `endpoint()` entry (params + `sample_response`) for every public/auth endpoint; add a prose page to `routers/docs/pages.py` `DOCS_PAGES` when warranted. State-changing actions need an audit event (`events.md` key, `category_for`, recorder call at the mutation point).
|
||||
7. **SEO** - public pages build `base_seo_context` and the right JSON-LD; add to `routers/seo.py` sitemap when indexable.
|
||||
8. **Docs of record** - update `README.md` (product-facing) and `AGENTS.md` (deep companion) for any new route/config/dependency/mechanic; update `CLAUDE.md` only when a NEW architectural rule or convention is introduced.
|
||||
8. **Docs of record** - update `README.md` (product-facing) and the relevant nested `CLAUDE.md` (deep companion for the subsystem you touched - create one if the directory doesn't have one yet) for any new route/config/dependency/mechanic; update the root `CLAUDE.md` only when a NEW cross-cutting architectural rule or convention is introduced, and add a row to its "Subsystem map" table if you created a new nested `CLAUDE.md`.
|
||||
9. **Tests (a hard project requirement, never optional)** - the DevPlace suite is one test file per endpoint, ~932 tests, split into three tiers with the directory tree mirroring the URL/source path. Every feature gets a test in EVERY tier it exercises: `tests/unit/` for a new data/query helper (pure in-process, `local_db` or no fixture, path mirrors the SOURCE module - `devplacepy/utils.py` -> `tests/unit/utils.py`); `tests/api/` for a new JSON or HTML route (HTTP integration against the live uvicorn subprocess via `app_server`/`seeded_db`, path mirrors the endpoint - `POST /auth/login` -> `tests/api/auth/login.py`) - but when a route depends on an in-process injected fake or a module-level singleton the separate uvicorn subprocess cannot see (the Gitea client via `runtime.set_client(fake)`, or any other `set_client`/monkeypatched backend), test it IN-PROCESS instead with `from starlette.testclient import TestClient; TestClient(m.app)`, the fake set in the test process, and auth via a `create_session(uid)` `session` cookie, asserting JSON with `Accept: application/json` (the `tests/api/issues/` files are the canonical example); `tests/e2e/` for a new interactive UI flow (Playwright `page`/`alice`/`bob`, path mirrors the endpoint - `GET /admin/ai-usage` -> `tests/e2e/admin/aiusage.py`). A route or feature with no test in any tier is incomplete. Follow the required patterns (`wait_until="domcontentloaded"` on every `goto`/`wait_for_url`, scoped selectors, `try/finally` restore of any flipped global setting, the shared fixtures, `test_`-prefixed functions in non-prefixed files, born-live `deleted_at`/`deleted_by` on raw soft-delete inserts) and create any missing package directories (`__init__.py`). WRITE them; validate each by a clean import only; NEVER run them.
|
||||
|
||||
When a layer is intentionally absent (an internal route with no public docs, a route Devii must never call), say so explicitly in the plan with the rationale rather than fabricating the layer.
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
---
|
||||
name: style-maintainer
|
||||
description: Coding-rule compliance. Enforces the explicit CLAUDE.md and AGENTS.md coding rules across all source - forbidden naming (context-aware), no comments/docstrings, em-dash (context-aware), full typing, pathlib over os, dataclasses over fixed-key dicts, no version pinning, file headers, no magic numbers. Use for style/convention review. Most surface name/em-dash hits are false positives - run the decision algorithm.
|
||||
description: Coding-rule compliance. Enforces the explicit CLAUDE.md (root and nested per-subsystem) coding rules across all source - forbidden naming (context-aware), no comments/docstrings, em-dash (context-aware), full typing, pathlib over os, dataclasses over fixed-key dicts, no version pinning, file headers, no magic numbers. Use for style/convention review. Most surface name/em-dash hits are false positives - run the decision algorithm.
|
||||
tools: Read, Grep, Glob, Edit, Write, Bash
|
||||
model: inherit
|
||||
color: orange
|
||||
@ -36,7 +36,7 @@ Default to **REPORT** mode: record findings, do NOT modify files. Apply **FIX**
|
||||
No comments or docstrings in source you author; no em-dashes (use a hyphen); keep `retoor <retoor@molodetz.nl>` as the first line of any file you create.
|
||||
|
||||
## Your dimension
|
||||
Enforce the explicit CLAUDE.md and AGENTS.md coding rules across all source.
|
||||
Enforce the explicit CLAUDE.md (root plus every nested per-subsystem `CLAUDE.md`) coding rules across all source.
|
||||
|
||||
### Forbidden naming prefixes and suffixes (CONTEXT-AWARE)
|
||||
The banned tokens are `_new`, `_old`, `_current`, `_prev`, `_next` (outside iteration), `_temp`, `_tmp`, `_v1`/`_v2`/`_v3`, `better_`, `best_`, `simple_`, `my_`, `the_`, `_data`, `_info`, and the rest of the forbidden list. This rule targets LAZY, RENAMEABLE VARIABLE AND HELPER names you own. It is NOT a blind substring sweep, and most surface hits on `_data`/`_info`/`_item`/`_val` are FALSE POSITIVES. Run this decision algorithm for EVERY candidate before recording it, and skip it the moment any test fails:
|
||||
|
||||
@ -1,5 +1,5 @@
|
||||
---
|
||||
description: Explain a DevPlace subsystem, route, or file - read the relevant AGENTS.md section and the code, then summarize architecture, data flow, invariants, and entry points. Read-only.
|
||||
description: Explain a DevPlace subsystem, route, or file - read the relevant nested CLAUDE.md and the code, then summarize architecture, data flow, invariants, and entry points. Read-only.
|
||||
argument-hint: <area, route, or file>
|
||||
allowed-tools: Read, Grep, Glob, Bash(git log:*)
|
||||
---
|
||||
@ -8,13 +8,13 @@ Orient me on: **$ARGUMENTS**
|
||||
Investigate before explaining; confirm every claim against the source.
|
||||
|
||||
1. Locate the code: the router under `devplacepy/routers/`, the template under `devplacepy/templates/`, data helpers in `devplacepy/database.py`, schemas in `devplacepy/schemas.py`, and any service under `devplacepy/services/`.
|
||||
2. Read the matching domain section in `AGENTS.md` (the long-form companion) and the relevant part of `CLAUDE.md`.
|
||||
2. Read the matching nested `CLAUDE.md` for the subsystem (e.g. `devplacepy/services/devii/CLAUDE.md`), plus the relevant cross-cutting part of the root `CLAUDE.md`.
|
||||
3. Trace the data flow: input model (`models.py`) -> router handler + guard -> data helper -> response (HTML via `respond` + template, JSON via the `*Out` schema), plus the Devii action (`catalog.py`) and API docs (`docs_api.py`) where present.
|
||||
|
||||
Then give a tight explanation:
|
||||
- What it does and where it lives, with `file:line` references.
|
||||
- The request pipeline and data flow.
|
||||
- Key invariants and gotchas (pull these from AGENTS.md).
|
||||
- Key invariants and gotchas (pull these from the nested CLAUDE.md).
|
||||
- The fan-out: which of the nine feature layers exist for it.
|
||||
|
||||
Do not modify anything.
|
||||
|
||||
@ -8,7 +8,7 @@ You are orchestrating the DevPlace maintenance fleet. Each dimension is a projec
|
||||
## Dimension to subagent map
|
||||
| Dimension | Subagent | Enforces |
|
||||
|-----------|----------|----------|
|
||||
| style | `style-maintainer` | CLAUDE.md/AGENTS.md coding rules (context-aware names, em-dash, typing, pathlib, headers) |
|
||||
| style | `style-maintainer` | CLAUDE.md (root/nested) coding rules (context-aware names, em-dash, typing, pathlib, headers) |
|
||||
| dry | `dry-maintainer` | duplication and reuse of canonical shared utilities |
|
||||
| security | `security-maintainer` | auth guards, project visibility, read-only guards, input validation, XSS |
|
||||
| audit | `audit-maintainer` | audit-log coverage and event catalogue |
|
||||
|
||||
@ -12,5 +12,5 @@ Mirror an existing service - read `devplacepy/services/base.py` (BaseService) an
|
||||
3. Register it in `main.py` startup: `service_manager.register(YourService())`, under the same `DEVPLACE_DISABLE_SERVICES` guard as the others. It then auto-appears on `/admin/services`.
|
||||
4. If it calls an LLM, default its endpoint to `config.INTERNAL_GATEWAY_URL` and authenticate with the internal gateway key, like the other AI consumers.
|
||||
5. Emit audit events via `record_system` for any state change it makes.
|
||||
6. Document it in `AGENTS.md` (Background services section) and in `README.md` if user-visible.
|
||||
6. Document it in `devplacepy/services/CLAUDE.md` (Background services base machinery section, or the service's own nested `CLAUDE.md` if it has one) and in `README.md` if user-visible.
|
||||
7. Validate with `hawk` on the touched files and `python -c "from devplacepy.main import app"`.
|
||||
|
||||
@ -35,7 +35,7 @@ const FANOUT = [
|
||||
'6. services/devii/actions/catalog.py - an Action(name, method, path, summary, params, requires_auth, requires_admin) if a user could ask Devii to do it; a confirm param plus membership in CONFIRM_REQUIRED if destructive.',
|
||||
'7. docs_api.py - an endpoint() entry in the right group with params and sample_response for every public or authenticated route.',
|
||||
'8. seo.py - base_seo_context(request, ...) merged into the context for public pages; a sitemap entry in routers/seo.py if indexable.',
|
||||
'9. README.md (product) + AGENTS.md (mechanics) + CLAUDE.md (only for a genuinely new architectural rule).',
|
||||
'9. README.md (product) + the relevant nested CLAUDE.md (mechanics) + the root CLAUDE.md (only for a genuinely new architectural rule).',
|
||||
].join('\n')
|
||||
|
||||
const TESTS = [
|
||||
@ -175,7 +175,7 @@ const LIVE_SCHEMA = {
|
||||
log(`Feature: ${ask}`)
|
||||
|
||||
const map = await agent(
|
||||
`Map the area of the DevPlace codebase relevant to this feature request, so it can be implemented. Read the closest existing feature end to end (its router, template, tests, and AGENTS.md section) as the pattern to follow. Do not write anything.\n\nFeature request: ${ask}\n\n${FANOUT}\n\nReturn: a summary of how this should be built, the concrete files to touch or create, the most similar existing feature to mirror, and any constraints.`,
|
||||
`Map the area of the DevPlace codebase relevant to this feature request, so it can be implemented. Read the closest existing feature end to end (its router, template, tests, and the matching nested CLAUDE.md) as the pattern to follow. Do not write anything.\n\nFeature request: ${ask}\n\n${FANOUT}\n\nReturn: a summary of how this should be built, the concrete files to touch or create, the most similar existing feature to mirror, and any constraints.`,
|
||||
{ agentType: 'Explore', label: 'understand', phase: 'Understand', schema: MAP_SCHEMA }
|
||||
)
|
||||
|
||||
|
||||
@ -32,7 +32,7 @@ const CHECKLIST = [
|
||||
'6. docs_api.py - endpoint() entries for the enqueue, status, and download routes.',
|
||||
'7. static/js - wire JobPoller.run(statusUrl, {onDone, onFailed, onTimeout}) on the triggering element.',
|
||||
'8. CLI (optional) - a prune/clear subcommand if artifacts accumulate.',
|
||||
'9. README.md + AGENTS.md - document the new job kind.',
|
||||
'9. README.md + devplacepy/services/jobs/CLAUDE.md - document the new job kind.',
|
||||
].join('\n')
|
||||
|
||||
const TESTS = [
|
||||
|
||||
@ -968,7 +968,7 @@ Changes are promoted through automated DTAP streets: Development (`make dev`), T
|
||||
2. Validate each touched language (Python compiles/imports, JS parses, CSS and HTML balance)
|
||||
3. `make test` - run all tests (fail-fast)
|
||||
4. Add tests in the matching tier and endpoint file (`tests/{unit,api,e2e}/<endpoint>.py`) for new functionality
|
||||
5. Update `AGENTS.md` and `README.md` if new conventions were introduced
|
||||
5. Update the relevant nested `CLAUDE.md` and `README.md` if new conventions were introduced
|
||||
|
||||
## License
|
||||
|
||||
|
||||
200
devplacepy/database/CLAUDE.md
Normal file
200
devplacepy/database/CLAUDE.md
Normal file
@ -0,0 +1,200 @@
|
||||
This file documents devplacepy/database/ - the dataset/SQLite data layer, indexing rules, and the project-wide soft-delete model. Claude Code loads it automatically whenever a file under this directory is read or edited.
|
||||
|
||||
## Database engine and dataset library
|
||||
|
||||
SQLite via `dataset` with these pragmas on every connection:
|
||||
|
||||
```python
|
||||
PRAGMA journal_mode=WAL; -- concurrent readers + writers
|
||||
PRAGMA synchronous=NORMAL; -- safe with WAL mode
|
||||
PRAGMA busy_timeout=30000; -- wait 30s instead of failing on lock
|
||||
PRAGMA cache_size=-8000; -- 8MB page cache
|
||||
PRAGMA temp_store=MEMORY; -- temp tables in memory
|
||||
```
|
||||
|
||||
Configured via `dataset.connect(engine_kwargs={"connect_args": {"timeout": 30, "check_same_thread": False}}, on_connect_statements=[...])`. In addition to the pragmas above, the connection is tuned with a 30s busy timeout, an 8MB page cache, and a 256MB mmap.
|
||||
|
||||
`init_db()` is idempotent - every index is created via `CREATE INDEX IF NOT EXISTS` (through the `_index()` helper, wrapped in try/except so it is safe to run on every startup regardless of table state), and seed defaults for `site_settings` only insert when the key doesn't already exist.
|
||||
|
||||
**`init_db()` also ensures the full column set of any table that code filters on** (`news`, `news_sync`, `attachments` use `get_table(name)` + `create_column_by_example` before their indexes). This is mandatory, not optional: `dataset` gives a lazily-created table ONLY the columns of its first insert, so a partial insert from a CLI tool/test/maintenance script would otherwise create the table with a reduced schema, and the long-running server caches that stale schema - making later queries on the missing column 500 with `no such column`. When adding a new filtered/indexed column, add it to the matching `init_db()` ensure-block:
|
||||
|
||||
```python
|
||||
news = get_table("news")
|
||||
for column, example in (("uid", ""), ("status", ""), ("synced_at", ""), ("external_id", ""), ...):
|
||||
if not news.has_column(column):
|
||||
news.create_column_by_example(column, example)
|
||||
_index(db, "news", "idx_news_status", ["status"]) # now safe - column exists
|
||||
```
|
||||
|
||||
Do this with `get_table(name)` (NOT `if name in db.tables`): `get_table` + `create_column_by_example` creates the table if it is absent, so the schema is guaranteed regardless of which process or which insert touches the table first. Symptom to watch for: a page renders fine in isolation but shows empty data or 500s only after a CLI/maintenance test runs earlier in the suite (it created the table partially).
|
||||
|
||||
The `_index(...)` helper supports `where=` (partial) and `unique=` indexes; every table with a `uid` column gets a UNIQUE `idx_<table>_uid`, soft-delete tables get a PARTIAL `idx_<table>_trash` (`WHERE deleted_at IS NOT NULL`) and NEVER a bare `deleted_at` index (it mis-steers the planner on live reads), and "live newest-first" listings need a composite/live-partial index that carries the sort column (see "Indexing conventions" below). `init_db()` finishes with `ANALYZE`/`PRAGMA optimize`. Verify any index change with `EXPLAIN QUERY PLAN`.
|
||||
|
||||
**SQLite is synchronous by design and will never be made async (hard rule).** The `dataset`/SQLite layer is synchronous and is called directly inside the async route handlers. This is intentional and more than fast enough for this platform: the database is a local file tuned with WAL, `synchronous=NORMAL`, a 30s busy timeout, an 8MB page cache, and a 256MB mmap, so queries are sub-millisecond and never meaningfully stall the event loop. Do NOT convert the database layer to async, do NOT wrap DB calls in a threadpool/`run_in_executor`/`to_thread`, and do NOT raise the synchronous DB access as a performance, blocking-I/O, or "requests aren't fully async" concern. This is a settled, sophisticated design choice and is not open for revisiting.
|
||||
|
||||
Runtime config lives in `site_settings`, read via `get_setting(key, default)` / `get_int_setting(key, default)` (60s TTL cache, invalidated cross-worker via the `cache_state` version table - `get_setting` calls `sync_local_cache("settings", ...)`, writes call `bump_cache_version("settings")`; the `_user_cache` in `utils.py` uses the same primitive under the `auth` name). Consumers always pass the production default to `get_setting`, so behavior is correct even before the row exists. Numeric operational values are floored at the call site so an invalid `0` can't lock out writes or stall a service. Booleans are stored as `"0"`/`"1"` and rendered as `<select>` (not checkboxes) because the settings save handler skips empty form values - an unchecked checkbox could never be turned off. See "Site settings" and "Operational settings" below for the full key registry.
|
||||
|
||||
Batch helpers (`get_users_by_uids`, `get_comment_counts_by_post_uids`, `get_vote_counts`, `load_comments`, `get_attachments_by_type`) exist specifically to avoid N+1 queries - use them in feed/listing routes instead of per-row lookups.
|
||||
|
||||
## Dataset rules (hard-learned)
|
||||
|
||||
**`find()` does NOT accept raw SQL strings.** It takes keyword arguments for equality filters, dict comparison operators, or SQLAlchemy column expressions.
|
||||
|
||||
```python
|
||||
# WRONG - causes 500 Internal Server Error:
|
||||
table.find("created_at >= :start", {"start": today})
|
||||
table.find(text("created_at >= :start"), start=today)
|
||||
|
||||
# CORRECT - dict comparison syntax:
|
||||
table.find(created_at={">=": today})
|
||||
|
||||
# CORRECT - keyword equality:
|
||||
table.find(country="France")
|
||||
|
||||
# CORRECT - SQLAlchemy column expression for IN clause:
|
||||
table.find(table.table.columns.user_uid.in_(["uid1", "uid2"]))
|
||||
|
||||
# CORRECT - multiple equality filters combined:
|
||||
table.find(topic="devlog", user_uid=some_uid)
|
||||
```
|
||||
|
||||
**`update()` requires a key column list as second argument.** The first dict contains all fields including the key column.
|
||||
|
||||
```python
|
||||
table.update({"uid": user_uid, "bio": "new bio"}, ["uid"])
|
||||
```
|
||||
|
||||
**`db.query()` accepts raw SQL with named params as keyword arguments:**
|
||||
|
||||
```python
|
||||
db.query("SELECT * FROM posts WHERE topic = :t", t="devlog")
|
||||
# NOT: db.query("...", {"t": "devlog"})
|
||||
```
|
||||
|
||||
**`db.query()` WRITES DO NOT AUTO-COMMIT - wrap any `db.query` INSERT/UPDATE/DELETE in `with db:` (load-bearing, caused a production deadlock).** The dataset table API (`table.insert`/`update`/`delete`) calls `db._auto_commit()` internally, but `db.query()` does NOT. SQLAlchemy 2.x autobegins a transaction on first `execute`, so a raw `db.query` write leaves an open transaction holding the SQLite write lock until that thread's connection next commits. On the request/loop thread this is masked (the next table op's `_auto_commit` flushes it), but on a **background-queue or `run_in_executor` worker thread** the thread goes idle still holding the lock, and EVERY subsequent write app-wide blocks for the 30s busy-timeout then fails `database is locked` - a full deadlock. Always commit raw writes:
|
||||
|
||||
```python
|
||||
with db: # commits + releases the write lock on exit
|
||||
db.query("INSERT INTO t (...) VALUES (:a) ON CONFLICT(...) DO UPDATE SET ...", a=1)
|
||||
```
|
||||
|
||||
Atomic counters (e.g. `add_correction_usage`) must use raw `ON CONFLICT DO UPDATE SET col = col + excluded.col` (the table API cannot increment), so they MUST use the `with db:` wrapper. Prefer the table API whenever an atomic SQL increment is not required.
|
||||
|
||||
**Always check `tables` list before raw SQL queries:**
|
||||
|
||||
```python
|
||||
if "comments" not in db.tables:
|
||||
return {} # table doesn't exist yet
|
||||
```
|
||||
|
||||
**Batch queries eliminate N+1 problems.** Use `get_users_by_uids()`, `get_comment_counts_by_post_uids()`, and `get_vote_counts()` from `database.py` instead of per-row lookups in loops.
|
||||
|
||||
**`init_db()` MUST create every queried column for any table that code filters on, even if the table is created lazily.** dataset creates a table on its FIRST insert and gives it ONLY the columns in that insert. If any code path can insert a *partial* row before the full schema exists (a CLI tool, a test fixture, a maintenance script), the table is born with a reduced schema and every later query against a missing column throws `sqlite3.OperationalError: no such column: X` (a 500), or - for an indexed column - logs a `Could not create index ... no such column` warning at startup. This is worsened by **cross-process metadata staleness**: the long-running uvicorn server reflects a table's columns once and caches them, so a column another process adds afterward is invisible to the server until it reconnects. The defence is to make `init_db()` ensure the complete column set up front, exactly like the existing `news`, `news_sync`, and `attachments` blocks (see the code example under "Database engine and dataset library" above). When you add a NEW column that any query filters/indexes, add it to the `init_db()` ensure-block too - never rely on the first insert to define it.
|
||||
|
||||
## Indexing conventions (the soft-delete planner trap)
|
||||
|
||||
`init_db()` owns every index. The `_index(db, table, name, columns, *, where=None, unique=False)` helper builds the DDL; it supports **partial** indexes (`where=`) and **unique** indexes, and wraps each `CREATE`/`DROP` in `with db:` (DDL via `db.query` does not auto-commit - see "Dataset rules" above). Three load-bearing rules learned from an `EXPLAIN QUERY PLAN` audit against production data:
|
||||
|
||||
- **Every table with a `uid` column gets `idx_<table>_uid` (UNIQUE).** `dataset` makes its own `id` autoincrement PK and does NOT key `uid`, so `find_one(uid=...)`, `resolve_by_slug`, `soft_delete`, and `table.update({...}, ["uid"])` full-SCAN without it. `init_db()` loops `for table in db.tables: _uid_index(db, table)` (falls back to a non-unique index if a UNIQUE build ever fails on legacy duplicate data). New tables are covered automatically.
|
||||
|
||||
- **NEVER index the bare `deleted_at` column - use a PARTIAL trash index `WHERE deleted_at IS NOT NULL`.** `ensure_soft_delete_columns` creates `idx_<table>_trash ON (deleted_at) WHERE deleted_at IS NOT NULL` (and drops any legacy full `idx_<table>_deleted`). A full `deleted_at` index is a planner hazard: the column is one giant `NULL` bucket plus many unique delete-timestamps, so `sqlite_stat1` mis-estimates `deleted_at IS NULL` as returning ~2 rows and the planner picks that index for live reads, then `USE TEMP B-TREE FOR ORDER BY` to sort the whole live set (the global feed was doing exactly this, with 82% of posts soft-deleted). The partial index serves the admin Trash view (`deleted_at IS NOT NULL`) cheaply and stops poisoning live `IS NULL` queries.
|
||||
|
||||
- **For "live, newest-first" listings add a composite or live-partial index that includes the sort column.** A `WHERE deleted_at IS NULL ORDER BY created_at` query needs the ordering in the index or it filesorts. Posts use a partial `idx_posts_live_created ON (created_at) WHERE deleted_at IS NULL` (feed) plus `idx_posts_user_created (user_uid, created_at)` (profile); comments use `idx_comments_target_created (target_type, target_uid, created_at)`; votes use `idx_votes_user_target (user_uid, target_uid)` (the per-user "my_vote" check on every card); notifications/gists/projects use `(user_uid, created_at)`. All were verified to drop the `USE TEMP B-TREE FOR ORDER BY` step.
|
||||
|
||||
- **Index the non-`uid` lookup keys too, not just the sort/owner columns.** A demand-vs-supply audit added the last missing single-key lookups: the `resolve_by_slug` hot path filters `slug` on content detail pages, so posts/gists/news/projects each get `idx_<table>_slug (slug)`; `get_setting`/`set_setting` filter `key`, so `idx_site_settings_key (key)`; the container store's `find_one(slug=)`/`find_one(name=)` fallbacks get `idx_instances_slug`/`idx_instances_name`. The DM thread load `find(sender_uid=, receiver_uid=)` gets the covering composites `idx_messages_conversation (sender_uid, receiver_uid)` + `idx_messages_conversation_rev (receiver_uid, sender_uid)` (the read-flag `UPDATE` uses the reverse); the badge-has check gets `idx_badges_user_name (user_uid, badge_name)`; the admin user list `ORDER BY -created_at` gets `idx_users_created_at (created_at)` (the existing `(role, created_at)` cannot serve a full-table created_at sort). All are non-unique so `_index` always creates them even if legacy duplicate data exists. Column sets already resolved to ~1 row by an existing prefix index (votes `+target_type`, game_quests `+kind`, poll_options `position`) are intentionally left uncovered - a trailing column there only adds write cost.
|
||||
|
||||
`init_db()` ends with `_refresh_query_planner_stats()`: a one-time `ANALYZE` when `sqlite_stat1` is absent, then `PRAGMA optimize` every boot (both inside `with db:`). Validate any index change with `EXPLAIN QUERY PLAN` against `data/devplace.db` (or a copy) - a correct plan reads `SEARCH ... USING INDEX <name>` with no temp b-tree.
|
||||
|
||||
## Project-wide soft delete (hard rule)
|
||||
|
||||
Attachments (see "Profile media gallery and soft-deleted attachments" below) are one instance of a platform-wide model: **every removal is a soft delete; only garbage collection is a hard delete.** Every removable row carries two columns, `deleted_at` (ISO timestamp) and `deleted_by` (actor uid, or `system`). A live row has `deleted_at = NULL`, and every list/count read filters `deleted_at IS NULL`.
|
||||
|
||||
- **Tables (`database.SOFT_DELETE_TABLES`):** posts, comments, gists, projects, news, news_images, project_files, attachments, votes, reactions, bookmarks, follows, poll_votes, polls, poll_options, sessions, instances, instance_schedules, devii_conversations, devii_tasks, devii_lessons, devii_virtual_tools, user_customizations, project_forks. `init_db` loops `ensure_soft_delete_columns(t)` for each (adds both columns + `idx_<t>_deleted` / the partial trash index described above). The Devii task/lesson/virtual-tool stores run on their own DB handles, so their `_ensure_indexes` adds the columns there too.
|
||||
- **Every INSERT into a soft-deletable table MUST write `deleted_at: None, deleted_by: None`.** `dataset.find(deleted_at=None)` on a table that lacks the column matches NOTHING (a false predicate), silently hiding all rows - the born-live insert is what creates the column. Add the pair to any new insert.
|
||||
- **Central helpers (`database/`):** `soft_delete(table, deleted_by, *, stamp=None, **criteria)` (equality), `soft_delete_in(table, column, uids, deleted_by, *, stamp=None, **extra)` (IN-clause cascade), `restore(table, **criteria)`, `purge(table, **criteria)` (real delete), `list_deleted(table, page)` / `count_deleted(table)` (trash listings), and the event helpers `restore_event(stamp)` / `purge_event(stamp)` that act across ALL tables sharing one `deleted_at` stamp.
|
||||
- **Two generic chokepoints are conditionally filtered:** `resolve_by_slug(table, slug, include_deleted=False)` (detail-page lookups; restore passes `include_deleted=True`) and `paginate(table, ...)` (auto-appends `deleted_at IS NULL` when the table has the column and the caller did not pass `deleted_at`). `seo._collect` does the same for the sitemap. Read filters were threaded through every batch helper, analytics/activity/leaderboard UNION, feed/profile/listing route, and store; never re-introduce an unfiltered read of a soft-deletable table.
|
||||
- **Any new read** (find/count/query) of a soft-deletable table MUST filter `deleted_at IS NULL`. Use the central helpers/chokepoints instead of inline deletes.
|
||||
- **Toggles revive, they do not duplicate.** votes/reactions/bookmarks/follows/poll_votes look up the physical row regardless of `deleted_at`: toggle-off stamps `deleted_at`; re-toggle clears it on the same row. Counts/state reads filter `deleted_at IS NULL`.
|
||||
- **Cascades share one stamp.** `content.delete_content_item` soft-deletes the item plus its comments, votes, engagement, project files, fork relations, and attachments with one shared `stamp` and `deleted_by = actor`. That timestamp identifies the whole event, so `restore_event`/`purge_event` reverse or finalize it atomically.
|
||||
- **Delete authorization is owner-OR-admin, enforced on the endpoint** (`is_owner(...) or is_admin(user)`): posts/gists/projects (`content.delete_content_item`, also rejects a missing item), `comments.delete_comment`, `project_files.project_file_delete`, `media.delete_media`, `uploads.delete_attachment_route`; news (`admin_news_delete`) is admin-only. Because the check is on the endpoint, one rule covers the human UI and **Devii** at once - Devii only ever calls the platform API, authenticated as the signed-in user, so an admin's Devii may soft-delete any member's content and a member's is refused with no agent-side logic. The matching `delete_*` Devii catalog tools stay `requires_auth` (not `requires_admin`) so a member can still delete their own, and every one is in the dispatcher's confirmation gate (`CONFIRM_REQUIRED`) so a delete only runs on a repeat call with `confirm=true`. Standalone `comment` and uploaded-`attachment` deletes are soft like the rest (`soft_delete` cascade / `soft_delete_attachment`); the only attachment hard delete is the admin `/admin/media/{uid}/purge` and the CLI prune. Any NEW content delete path must reuse this guard, soft-delete, and (for the Devii tool) be added to `dispatcher.CONFIRM_REQUIRED`.
|
||||
- **What stays HARD (GC / the empty-trash stage):** the async-job sweep + CLI prune/clear, the container metrics ring trim, gateway and Devii usage-ledger retention prunes and quota resets, the expired-session cleanup branch in `utils._user_from_session`, fork-rollback of a half-created project, the news-sync image replacement, and the admin **Purge** action. Logout is a soft delete (auditable via `deleted_by`); only expiry GC is hard.
|
||||
- **Admin Trash surface:** `/admin/trash` (sidebar **Trash**, `routers/admin/` package, `admin_trash.html`, `AdminTrashOut`) lists soft-deleted rows per table with restore/purge per row. Restore calls `restore_event(row.deleted_at)`; Purge calls `purge_event(...)` and unlinks attachment files / project-file blobs. The attachment-specific `/admin/media` view is unchanged. Admin-only docs: `docs/soft-delete.html` (`admin: True`, Administration section).
|
||||
|
||||
## Profile media gallery and soft-deleted attachments
|
||||
|
||||
The profile **Media** tab (`/profile/{username}?tab=media`, public) is a paginated grid of every attachment a user uploaded, newest first, across all `target_type`s. Attachments support a **soft delete** so a user can remove one upload without affecting its parent object.
|
||||
|
||||
- **One `deleted_at` column** on `attachments` (ISO string, mirrors the `push.py` precedent), ensured idempotently in `init_db` via `create_column_by_example("deleted_at", "")` plus the `idx_attachments_user_created` index. `store_attachment` writes `deleted_at: None`.
|
||||
- **Soft delete preserves the relation and the file.** `attachments.soft_delete_attachment(uid)` only stamps `deleted_at`; it never touches `target_type`/`target_uid` and never unlinks the file. `restore_attachment(uid)` clears `deleted_at`, so the item reappears on its parent object and in the gallery with zero extra bookkeeping.
|
||||
- **Three read paths filter `deleted_at IS NULL`** so a soft-deleted item vanishes everywhere (the gallery AND its parent post/project/etc.): `get_attachments`, `get_attachments_batch` (both in `attachments.py`), and `database.get_user_media`. **The hard-delete cascades (`delete_attachments_for`, `delete_target_attachments`) stay UNFILTERED** so permanently deleting a parent object still removes ALL its attachment files, including soft-deleted ones - never add the filter there.
|
||||
- **Queries:** `database.get_user_media(user_uid, page)` (linked, non-deleted, newest first; each item gets a `target_url` via `resolve_object_url`) and `database.get_deleted_media(page)` (the admin trash, joined to uploader username).
|
||||
- **Authorization:** `POST /media/{uid}/delete` (`routers/media.py`) is owner-or-admin (`attachment["user_uid"] == user["uid"] or is_admin(user)`); `POST /media/{uid}/restore` and `POST /admin/media/{uid}/purge` (the only hard delete, via `delete_attachment`) are admin-only (`routers/admin/` package, sidebar **Media** -> `/admin/media`). The tab itself is public.
|
||||
- **Frontend:** `_media_gallery.html` reuses the `_attachment_display.html` type branches and the `dp-lightbox` contract (`data-lightbox`/`data-full`). The delete button carries `data-media-delete` + `data-confirm`; `ModalManager.initConfirmations` shows the confirm and `MediaGallery.js` (`app.mediaGallery`) does the optimistic `Http.send` delete, fades the tile, and toasts. A `<noscript>` form is the no-JS fallback. Grid styling is `static/css/media.css`.
|
||||
- **Devii:** `list_media` (public) and `delete_media` (auth, in `CONFIRM_REQUIRED`) in the catalog.
|
||||
- **Docs visibility (deliberate):** members and guests must never be told this is a *soft* delete. The public prose page `docs/media-gallery.html` (General) and the member-facing `media-delete` API endpoint (Profiles group) describe deletion as a plain "remove" - no soft-delete, restore, trash, or purge language. All moderation mechanics live on the admin-only `docs/media-moderation` prose page (`admin: True`) and in the admin API group (`media-restore`, `admin-media`, `admin-media-purge`, all `auth="admin"`), which `docs_search` excludes from member results and `routers/docs/` package 404s for non-admins. Because `docs_search._strip` keeps the text *inside* `{% if %}` blocks, admin content must live on a separate `admin: True` page, never inline-gated on a public page (a public page may only carry an admin-gated *link*). The member `MediaItemOut` schema omits `deleted_at`; the admin-only `AdminMediaItemOut` adds it.
|
||||
|
||||
## Role-based visibility (generic + DRY)
|
||||
|
||||
- **One source of truth for role/visibility checks**, registered as Jinja globals in `templating.py` - never hand-roll `user.get('role') == 'Admin'` or `user['uid'] == x['user_uid']` in a template again:
|
||||
- `is_admin(user)` (also `utils.is_admin`, reused by `require_admin` and `docs.py`) - admin-only UI.
|
||||
- `owns(item, user)` (= `content.is_owner`) - per-item ownership (e.g. each comment). Page-level detail templates keep using the `is_owner` **bool** passed in their context (post/gist/project/profile); do not call `is_owner(...)` as a function - that name is a context bool and shadows globals.
|
||||
- `is_self(user, uid)` - "is this me" (profile follow vs edit, leaderboard highlight).
|
||||
- `guest_disabled(user)` -> emits ` disabled aria-disabled="true" title="Log in to participate"` for guests (empty for members); `login_hint(user)` -> a small login link. Both return `Markup`.
|
||||
- **Role values are stored capitalized:** `users.role` is exactly `"Admin"` or `"Member"` (first registered user is `"Admin"`, `auth.py`); `is_admin` compares `== "Admin"` case-sensitively. The CLI is the only lowercase surface (`devplace role set ... <member|admin>` writes `role.capitalize()`; `role get` prints `.lower()`). A lowercase role in the DB silently defeats every admin check - never write a raw lowercase role.
|
||||
- **The shadow rule generalizes beyond `is_owner` to ANY Jinja global** (`is_admin`, `avatar_url`, `format_date`, `is_self`, `owns`, `guest_disabled`): `respond(req, tmpl, ctx, model=XOut)` hands the **same** `ctx` to the Pydantic model and the template, and a context key shadows the same-named global across the whole `base.html` chain. A bool named `is_admin` in the context makes `base.html`'s `{% if is_admin(user) %}` raise `TypeError: 'bool' object is not callable` - a 500 that only fires for the branch invoking the global (logged-in users, not guests, which is why guest-only smoke tests miss it). Name viewer/permission flags distinctly (`viewer_is_admin`) in both schema and context. Real issue fixed on `/issues/{number}`; regression-guarded by `tests/api/issues/create.py::test_issue_detail_renders_for_{member,admin}` (they render the page as an authenticated Member/Admin and assert 200 + the admin-only control).
|
||||
- **Policy enforced everywhere:** guests see all non-admin content read-only with action controls **shown but disabled** (`guest_disabled` on vote/star/react/poll/bookmark/follow/comment submit; create FABs become `/auth/login` links via `.feed-fab.login-required`); members get full member actions; **role badges render only to admin viewers** (`{% if is_admin(user) %}` around every `*.role` label). Backend stays the real gate (`require_user`/`require_admin`).
|
||||
- Docs admin gating is unchanged behaviourally but now uses `is_admin` (`docs_base.html` `DEVPLACE_DOCS.isAdmin`, `docs/index.html`, `docs.py`).
|
||||
- **Tests:** the role-gating e2e tests across `tests/e2e/` (guest/member/admin via `page`/`bob`/`alice`) are the UI enforcement; `tests/api/auth/matrix.py` is the backend companion. Guest action controls are asserted **disabled** (not absent) - don't reintroduce `count() == 0` assertions for them.
|
||||
|
||||
## Database tables
|
||||
|
||||
| Table | Purpose |
|
||||
|-------|---------|
|
||||
| `news` | All synced articles with `status` (published/draft), `grade`, `slug`, `show_on_landing` |
|
||||
| `news_images` | Images extracted from article URLs |
|
||||
| `news_sync` | Sync state per article `guid` - tracks grading history |
|
||||
|
||||
The platform-wide soft-delete table set (`database.SOFT_DELETE_TABLES`) is listed in full under "Project-wide soft delete (hard rule)" above.
|
||||
|
||||
## Site settings
|
||||
|
||||
Site settings are seeded on startup (`site_settings` table):
|
||||
|
||||
| Key | Default | Purpose |
|
||||
|-----|---------|---------|
|
||||
| `site_name` / `site_description` / `site_tagline` | DevPlace branding | General site metadata |
|
||||
| `news_grade_threshold` | `"7"` | Minimum AI grade for auto-publish |
|
||||
| `news_api_url` | `"https://news.app.molodetz.nl/api"` | News source API |
|
||||
| `news_ai_url` | `"https://openai.app.molodetz.nl/v1/chat/completions"` | AI grading endpoint |
|
||||
| `news_ai_model` | `"molodetz"` | AI model identifier |
|
||||
| `max_upload_size_mb` / `allowed_file_types` / `max_attachments_per_resource` | `"10"` / `""` / `"10"` | Upload limits |
|
||||
| `rate_limit_per_minute` | `"60"` | Mutating requests per IP per window (`main.py` middleware); a `429` carries a `Retry-After: <window>` header |
|
||||
| `rate_limit_window_seconds` | `"60"` | Rolling window for the rate limit (also the `Retry-After` value) |
|
||||
| `news_service_interval` | `"3600"` | Seconds between news fetch cycles (`NewsService.run_once` re-reads each cycle) |
|
||||
| `session_max_age_days` | `"7"` | Standard session cookie + DB session lifetime |
|
||||
| `session_remember_days` | `"30"` | Remember-me session lifetime |
|
||||
| `registration_open` | `"1"` | When `"0"`, signup GET shows a closed notice and POST is rejected (`auth.py`) |
|
||||
| `maintenance_mode` | `"0"` | When `"1"`, non-admins get a 503 (`main.py` maintenance middleware) |
|
||||
| `maintenance_message` | scheduled-maintenance text | Body shown on the maintenance 503 page |
|
||||
| `customization_enabled` | `"1"` | When `"0"`, `custom_css_tag`/`custom_js_tag` inject nothing (feature off) |
|
||||
| `customization_js_enabled` | `"1"` | When `"0"`, custom CSS still serves but custom JS is suppressed |
|
||||
| `extra_head` | `""` | Raw HTML emitted verbatim into every page `<head>` by `templating.extra_head_tag()`; site-wide trusted-admin input, not sanitized |
|
||||
|
||||
Besides the site/news/upload keys above, the **Operational** group is admin-editable at `/admin/settings`: `site_url` (public origin for absolute links incl. container ingress; resolved by `seo.public_base_url()` = setting -> `DEVPLACE_SITE_URL` env -> request origin), `rate_limit_per_minute`, `rate_limit_window_seconds`, `news_service_interval`, `session_max_age_days`, `session_remember_days`, `registration_open`, `maintenance_mode`, `maintenance_message`, `docs_search_mode` (`agent`|`bm25`, default `agent` - picks the `/docs/search.html` surface: the in-page Devii chat or the classic BM25 list; a viewer over their daily AI limit, or a guest when Devii is disabled, auto-falls-back to BM25 via `routers/docs/views.py` `_agent_search_state`). The **Custom Code** key `extra_head` is the sole key in the settings handler's `CLEARABLE_SETTINGS` set, so saving an empty textarea removes it (the default loop skips empty values).
|
||||
|
||||
The seed block in `database.py` is guarded by `if "site_settings" in tables:` - on a brand-new DB the table does not exist yet (dataset creates tables lazily on first insert), so none of these rows are written until the table exists. Correct runtime behavior therefore relies on every consumer passing the production default to `get_setting`/`get_int_setting`, not on the seed.
|
||||
|
||||
## Operational settings
|
||||
|
||||
Operational settings - read sites and rules:
|
||||
|
||||
| Setting(s) | Read at | Notes |
|
||||
|-----------|---------|-------|
|
||||
| `rate_limit_*` | `rate_limit_middleware` in `main.py` | `max(1, get_int_setting(...))` so `0` can't block all writes |
|
||||
| `maintenance_mode` / `maintenance_message` | `maintenance_middleware` in `main.py` | Allows `/static`, `/avatar`, `/auth`, `/admin` and admins; everyone else gets `error.html` at 503 |
|
||||
| `news_service_interval` | `BaseService` reconciling loop via `current_interval()` | `max(60, ...)`; edited on the Services tab (not `/admin/settings`); a change applies on the next cycle |
|
||||
| `service_<name>_enabled` / `service_<name>_command` / `service_<name>_log_size` | `BaseService` reconciling loop | Generic per-service controls written by the Services tab; the loop reconciles within ~1s |
|
||||
| `session_max_age_days` / `session_remember_days` | `auth.py` signup + login | Multiplied by `SECONDS_PER_DAY`; passed to `create_session(uid, max_age)` so the cookie and the DB session row expire together |
|
||||
| `registration_open` | `auth.py` `signup_page` (GET) and `signup` (POST) | POST returns before any DB write when closed |
|
||||
|
||||
**Booleans are `<select>`, never checkboxes.** The settings save handler (`admin.py`) skips empty form values so empty fields don't clobber existing rows. An unchecked checkbox submits nothing, so it could never be turned off - `registration_open` and `maintenance_mode` use `<option value="1">`/`<option value="0">` so a value is always submitted.
|
||||
335
devplacepy/routers/CLAUDE.md
Normal file
335
devplacepy/routers/CLAUDE.md
Normal file
@ -0,0 +1,335 @@
|
||||
This file documents devplacepy/routers/ - route organization, the full URL prefix map, and small single-file feature behaviors. Claude Code loads it automatically whenever a file under this directory is read or edited.
|
||||
|
||||
## Routing layout / prefix table
|
||||
|
||||
Routers in `devplacepy/routers/` are organised as a **directory tree that mirrors the endpoint (URL) path**, exactly like `tests/`. A domain with a single resource stays one flat file (`feed.py`, `posts.py`, `gists.py`, ...); a domain with several sub-resources is a **package directory** split **one file per sub-resource** (a distinct noun under the domain), never one file per individual endpoint. Each leaf module declares its own `router = APIRouter()` keeping the exact path strings, and the package `__init__.py` aggregates them with `router.include_router(...)`. Because a package exposes `.router` like a module, `main.py` mounts each domain at its prefix unchanged.
|
||||
|
||||
Prefixes are wired in `main.py`:
|
||||
|
||||
| Prefix | Router |
|
||||
|--------|--------|
|
||||
| `/auth` | auth/ package - one leaf per flow (`signup`, `login`, `logout`, `forgotpassword`, `resetpassword`) |
|
||||
| `/feed` | feed.py |
|
||||
| `/posts` | posts.py |
|
||||
| `/comments` | comments.py |
|
||||
| `/projects` | projects/ package - `index.py` (listing/detail/create/delete plus owner-only visibility toggles `POST /{slug}/private` and `POST /{slug}/readonly`, and async `POST /{slug}/fork`), `files.py`, and the `containers/` subpackage (below). `main.py` mounts the whole `/projects` tree from this one package. See `routers/projects/CLAUDE.md` for the deep detail on this tree |
|
||||
| `/projects/{slug}/files` | projects/files.py - per-project virtual filesystem (CRUD dirs/files, upload, inline edit, plus line-range ops: `lines` read, `replace-lines`, `insert-lines`, `delete-lines`, `append` for large text files) |
|
||||
| `/profile` | profile/ package - `index.py` (page/search/followers/update/api-key; the **Posts** tab shows only the 10 most recent posts, while the sidebar `posts_count` stat always uses the real `count()` so it is never capped to the rendered 10), `customization.py`, `notifications.py`, `ai_correction.py` (`POST /{username}/ai-correction`, owner-or-admin AI content correction toggle + prompt), `ai_modifier.py` (`POST /{username}/ai-modifier`, owner-or-admin AI modifier toggle + prompt), `telegram.py` (`POST /{username}/telegram`, owner-or-admin Telegram pairing-code request/unpair), `usage.py` (`_ai_quota`, re-exported from the package) |
|
||||
| `/messages` | messages.py - real-time DM chat. `GET ""` (page; the conversation sidebar is one windowed SQL aggregate and an opened thread loads only the newest `CONVERSATION_MESSAGE_LIMIT` = 500 messages - keep these queries bounded), `GET /search`, `POST /send` (no-JS fallback), and `WS /messages/ws` (NOT lock-owner gated - accepts on every worker; closes `1008` for guests). The WS adds live bidirectional delivery, typing, read receipts, and in-process presence on top of the existing `messages` table; both the WS `send` and HTTP `POST /send` route through the shared `services/messaging/persist.py` `persist_message` choke point (identical audit/notification/mention) and then through `_finalize_and_broadcast`, which awaits any pending SYNC AI correction/modifier futures (DM content runs through both, so typing `@ai <instruction>` executes live) and broadcasts the FINAL corrected/modified content (the WS path passes `request=websocket` and awaits directly since HTTP middleware does not run for websockets; the frontend swaps the sender's optimistic bubble to the authoritative content). Same-worker delivery is instant via the per-worker `message_hub`; cross-worker delivery is filled by `services/messaging/relay.py` `message_relay` (per-worker ~1s DB poll on the `messages.id` watermark, deduped against the hub's delivered set). See `devplacepy/services/messaging/CLAUDE.md` |
|
||||
| `/notifications` | notifications.py |
|
||||
| `/votes` | votes.py |
|
||||
| `/reactions` | reactions.py - emoji reaction toggle on any target: `POST /reactions/{target_type}/{target_uid}` (optimistic `ReactionBar`) |
|
||||
| `/bookmarks` | bookmarks.py - bookmark/favorite toggle: `GET /bookmarks/saved` (the viewer's saved list) and `POST /bookmarks/{target_type}/{target_uid}` (toggle a bookmark) |
|
||||
| `/polls` | polls.py - poll voting: `POST /polls/{poll_uid}/vote` |
|
||||
| `/avatar` | avatar.py |
|
||||
| `/follow` | follow.py |
|
||||
| (none) | relations.py - per-user block/mute relations: `POST /block/{username}`, `/block/unblock/{username}`, `/mute/{username}`, `/mute/unmute/{username}` (soft-deletable `user_relations` rows) |
|
||||
| `/leaderboard` | leaderboard.py - `GET /leaderboard` XP/stars leaderboard page |
|
||||
| `/admin` | admin/ package - one leaf per sub-resource (`index`, `users`, `aiusage`, `aiquota`, `media`, `trash`, `settings`, `notifications`, `news`, `auditlog`, `backups`) plus the folded-in `services.py` and `containers.py` (mounted with `/services` and `/containers` sub-prefixes). `main.py` mounts the whole `/admin` tree from this one package. The `backups` leaf is the admin **Backups** dashboard (`BackupService`, kind `backup`): storage usage, backup archives, and interval/cron backup schedules (CRUD + rotation). **Archive download is restricted to the primary administrator** (the earliest-created Admin, resolved by `database.get_primary_admin_uid` / `utils.is_primary_admin`): `GET /admin/backups/{uid}/download` 403s every other admin, the `download_url` field is withheld from them at every endpoint (`can_download = is_primary_admin(admin)`, surfaced as `BackupDashboardOut.can_download_backups`), and `BackupMonitor.js` renders their Download control as a disabled button tooltipped `Not available`. See `devplacepy/services/backup/CLAUDE.md` |
|
||||
| `/admin/services` | admin/services.py |
|
||||
| `/issues` | issues/ package - issue tracker backed by Gitea (no local issue store): `index.py` (list `?state=`/`?page=`, detail `/{number}` with comments), `create.py` (async AI-enhanced filing `/create` enqueues a `issue_create` job, status at `/jobs/{uid}`), `comment.py` (synchronous, pushes to Gitea + notifies admins), `status.py` (admin open/closed), `attachments.py` (file attachments on open issues + comments, mirrored to Gitea native assets; add/list/delete with owner-or-admin + open-state guards) |
|
||||
| `/gists` | gists.py |
|
||||
| `/news` | news.py |
|
||||
| `/uploads` | uploads.py |
|
||||
| `/media` | media.py - profile media gallery item soft delete/restore: `POST /media/{uid}/delete` and `POST /media/{uid}/restore` (owner or admin) |
|
||||
| `/openai` | openai_gateway.py |
|
||||
| `/devii` | devii.py - WebSocket terminal (`/devii/ws`), page, `/devii/usage`, `/devii/session` |
|
||||
| `/zips` | zips.py - generic zip job status (`/zips/{uid}`) and archive download (`/zips/{uid}/download`); enqueued from `/projects/{slug}/zip` and `/projects/{slug}/files/zip` |
|
||||
| `/forks` | forks.py - fork job status (`/forks/{uid}`); enqueued from `/projects/{slug}/fork`. When done its `project_url` points at the new forked project |
|
||||
| `/tools` | tools/ package - public developer tools surface. `index.py` (`/tools` landing) plus `seo.py` (**SEO Diagnostics**): `GET /tools/seo` page, `POST /tools/seo/run` (enqueue `seo` job, per-owner one-active-job cap), `GET /tools/seo/{uid}` (`SeoJobOut`), `GET /tools/seo/{uid}/report` (HTML+JSON `SeoReportOut`), `WS /tools/seo/{uid}/ws` (live progress, lock-owner gated, close `4013` retry), `GET /tools/seo/{uid}/screenshot/{n}`. Also `deepsearch.py` (**DeepSearch**): `GET /tools/deepsearch` page, `POST /tools/deepsearch/run` (enqueue `deepsearch` job, per-owner one-active-job cap; resolves the user `api_key` into the payload, guests use the internal key), `GET /tools/deepsearch/{uid}` (`DeepsearchJobOut`), `GET /tools/deepsearch/{uid}/session` (HTML+JSON `DeepsearchSessionOut`), `WS /tools/deepsearch/{uid}/ws` (live progress, lock-owner gated, `4013` retry), `WS /tools/deepsearch/{uid}/chat` (grounded RAG chat over the session collection), `POST /tools/deepsearch/{uid}/{pause|resume|cancel}`, `GET /tools/deepsearch/{uid}/export.{md,json,pdf}`. The shared owner helper is `routers/tools/_shared.py` `owner_for`. Also `isslop.py` (**AI Usage Analyzer**): `GET /tools/isslop` page, `POST /tools/isslop/run` (enqueue `isslop` job, per-owner one-active-job cap; owner = user uid or the shared `DEVII_GUEST_COOKIE` guest identity, minted when absent), `GET /tools/isslop/list` (owner history; a signed-in request first claims any guest-cookie analyses via `store.claim_guest_analyses` - a move, never a copy), `GET /tools/isslop/{uid}` (`IsslopAnalysisOut`), `GET /tools/isslop/{uid}/events` (persisted ordered event trail, `?after=SEQ` incremental poll; live frames also publish on pub/sub `public.isslop.{uid}` - the DB trail is the source of truth, pub/sub the fast path), `GET /tools/isslop/{uid}/report` (HTML+JSON `IsslopReportOut`; live `<dp-isslop-run>` while running, server-rendered report via `render_content` when done), `GET /tools/isslop/{uid}/report.md`, `GET /tools/isslop/{uid}/badge.svg` (embeddable SVG authenticity badge). Analyses/reports/badges are permanent public capability URLs (`IsslopService.cleanup` never deletes them; only the job row is swept). Surfaced by a collapsible **Tools** header dropdown (`base.html`, visible to all) |
|
||||
| `/projects/{slug}/containers` | projects/containers/ subpackage - admin per-project container manager (`instances.py` for creation/lifecycle/exec/logs/metrics/sync plus the exec websocket, `schedules.py` for cron/interval/once schedules, shared helpers in `_shared.py`). Every instance runs the shared `ppy` image. Discoverable from the project detail page (admin-only **Containers** button) and from the admin index |
|
||||
| `/admin/containers` | admin/containers.py - admin **Containers** manager: `/admin/containers` lists every instance across all projects with inline actions (start/stop/restart/terminal/edit/delete) plus a create modal (project search-select, run-as user search-select, boot language + source editor, restart policy, start-on-boot, env/ports/limits/ingress); `/admin/containers/{uid}` is the per-instance detail page (lifecycle, live logs/metrics, PTY terminal, schedules, ingress, sync, status history) and `/admin/containers/{uid}/edit` edits run-as user, boot language/script/command, restart policy, start-on-boot, and limits. `POST /admin/containers/create`, `/{uid}/edit`, `/{uid}/{start,stop,restart,pause,resume}`, `/{uid}/sync`, `/{uid}/delete` call `api.*` directly under `require_admin` (no docker/exec backend duplicated); `GET /admin/containers/projects/search` and `/users/search` back the create/edit search-selects. The lifecycle and detail views stay layered over the per-project `/projects/{slug}/containers/instances/{uid}/...` endpoints (the instance carries its `project_uid`) |
|
||||
| `/p/{slug}` | proxy.py - public ingress reverse proxy (HTTP + WebSocket) to a running container instance's published host port, opt-in per instance via `ingress_slug` |
|
||||
| `/xmlrpc` | xmlrpc.py - reverse-proxies XML-RPC calls to the forking XML-RPC bridge (`services/xmlrpc/`, supervised by `XmlrpcService` on loopback `config.XMLRPC_PORT`), which generates one XML-RPC method per documented REST endpoint from `docs_api.API_GROUPS` (`posts.create`, `feed.list`, ...). One struct of named params per call; auth via in-band `api_key`, `X-API-KEY`/`Bearer` header, or `http://user:pass@host/xmlrpc` Basic. Full introspection + `system.multicall`; REST errors become XML-RPC faults. Exempt from the rate limiter (enforced on the forwarded internal hop). See `devplacepy/services/xmlrpc/CLAUDE.md` |
|
||||
| `/api` | devrant/ package - devRant-compatible REST protocol (`auth.py`, `rants.py`, `comments.py`, `notifs.py`). Translates devRant requests onto DevPlace data: rants<->posts, comments/votes onto the native engagement layer, devRant integer ids onto each table's auto-increment `id`. Token auth via `devrant_tokens`. See `routers/devrant/CLAUDE.md` for the deep detail on this tree |
|
||||
| `/dbapi` | dbapi/ package - **primary-administrator-only, strictly READ-ONLY** generic database API over `dataset` (never inserts/updates/deletes/restores; the write surface was removed because it bypassed every per-route admin safeguard). `tables.py` (`GET /tables`, `GET /{table}/schema`), `crud.py` (read only: `GET /{table}` + `GET /{table}/{key}/{value}`, `?include_deleted`), `query.py` (`POST /query` hard SELECT-only validated read-only run; `POST /query/async` + `GET /query/{uid}` + `WS /query/{uid}/ws` via `DbApiJobService` kind `dbquery`), `nl.py` (`POST /nl` natural-language->SQL via the AI gateway, returns validated SELECT, `execute=true` runs it read-only). **Auth = the PRIMARY administrator only** (the earliest-created Admin, `utils.is_primary_admin` / `database.get_primary_admin_uid` - the same identity that gates backup downloads) by session/api_key (`services/dbapi/policy.py`); every other administrator is refused like a member, and there is **no internal-key path** - the gateway `internal_gateway_key()` is NOT accepted (no service-to-service access). SQL validated by `services/dbapi/validate.py` (sqlglot classify + suspicious-flag + read-only `EXPLAIN` dry-run). Devii tools `db_*` are read-only (list/get/query/nl; no write tools) and flagged `requires_primary_admin=True`, so they are added to the LLM tool list ONLY for the primary administrator (every other session never sees them and Devii is unaware they exist). `services/pubsub/policy.py` resolves its own admin/internal actor and does NOT reuse `dbapi.policy.caller_for`, so the primary-admin restriction does not leak onto the pub/sub bus. See `devplacepy/services/dbapi/CLAUDE.md` |
|
||||
| `/game` | game/ package - the **Code Farm** idle game (`index.py` + `farm.py`): `GET /game` (page), `GET /game/state`, `GET /game/leaderboard`, `POST /game/{plant,harvest,buy-plot,upgrade,fertilize,daily,perk,prestige,legacy,quests/claim}`, plus social `GET /game/farm/{username}` and `POST /game/farm/{username}/{water,steal}` |
|
||||
| (none) | push.py - web push + PWA: `GET`/`POST /push.json` (VAPID public key / subscription register), `GET /service-worker.js`, `GET /manifest.json` |
|
||||
| (none) | docs.py (docs/ package) - the documentation site (prose pages + API reference). See `routers/docs/CLAUDE.md` for the deep detail on this tree (`DOCS_PAGES`, audience tiers, prose rendering pipeline) |
|
||||
| `/pubsub` | pubsub.py - **database-free** publish/subscribe bus. `WS /pubsub/ws` (subscribe/unsubscribe/publish frames, `foo.*` wildcards), `POST /pubsub/publish` + `GET /pubsub/topics` (admin/internal). Lock-owner-gated WS (`4013` retry) so subscribers converge on one worker; topic authz in `services/pubsub/policy.py` (`user.{uid}.*` private, `public.*` shared, admin/internal anywhere, guests opt-in). In-process `services.pubsub.publish(topic, data)` for backends; frontend `app.pubsub`. See `devplacepy/services/pubsub/CLAUDE.md` |
|
||||
| (none) | seo.py - `/robots.txt`, `/sitemap.xml` |
|
||||
|
||||
## Route aggregation rules
|
||||
|
||||
**Aggregation rules:** a leaf that owns the domain's collection-root (`""`) route must be the package's base router (FastAPI rejects an empty path included under an empty prefix), so the package `__init__` imports that leaf's `router` and includes the rest onto it (see `routers/issues`, `routers/admin`, `routers/projects`); leaves carved out of a former monolith keep their relative path strings and are included with no sub-prefix, while a router folded in from a deeper mount keeps its own paths and is included with a sub-prefix - `admin/__init__` includes `services.router` with `prefix="/services"` and `containers.router` with `prefix="/containers"`. Module-private helpers shared across a package's leaves live in its `_shared.py`. The `/projects` tree (project CRUD + `files.py` + `containers/`) and the `/admin` tree (every admin sub-resource plus the folded-in `services` and `containers`) are each mounted from a single package.
|
||||
|
||||
## HTML/JSON content negotiation
|
||||
|
||||
Every page/redirect endpoint also returns JSON when the client asks. Core in `devplacepy/responses.py`: `wants_json(request)` (true for `Accept: application/json` or `Content-Type: application/json`; browser `text/html` -> HTML, so existing behaviour is unchanged). **`X-Requested-With: fetch` is deliberately NOT a trigger** - the frontend sends it on form/engagement fetches expecting the old redirect, and the four legacy engagement endpoints (votes/reactions/bookmarks/polls) handle that header themselves. Two helpers replace the direct returns:
|
||||
|
||||
- **Page GETs:** `return respond(request, "x.html", context, model=XOut)` - HTML renders the template; JSON does `XOut.model_validate(context).model_dump()`. One context, two renderings.
|
||||
- **Action POSTs:** `return action_result(request, url, data=<resource|None>)` - HTML 302 redirects; JSON returns `{ok, redirect, data}`. (Set cookies on the returned response after calling it, as `auth.py` login/signup do.)
|
||||
|
||||
Response models live in `devplacepy/schemas.py` (Pydantic v2, `extra="ignore"`, all-Optional so they validate the existing context dicts directly). **Always project users through `UserOut`** (and `AdminUserOut`) - the raw user rows contain `email`/`api_key`/`password_hash`, and the models drop them; never serialize a raw user row. List item shapes vary (feed/gists/news/admin-news are wrapped `{post|gist|article: ...}`; projects are flat rows with `author_name`/`my_vote`) - match the context exactly. Errors negotiate centrally: `main.py` 404/500/validation handlers and the rate-limit/maintenance middleware, plus `utils.require_user`/`require_admin` (401/403 for JSON, 303 redirect for browsers). The four legacy AJAX endpoints (votes/reactions/bookmarks/polls) keep their original flat JSON shapes and are left untouched. Documented in `docs_api.py`'s Conventions group.
|
||||
|
||||
## FastAPI patterns
|
||||
|
||||
- **All routes are async.** Form data is validated via a typed Pydantic body param: `data: Annotated[SomeForm, Form()]` (models in `models.py`). Read raw `await request.form()` only when also handling an uploaded file (a separate `File()` param would embed the model under its parameter name).
|
||||
- **Return `RedirectResponse(url=..., status_code=302)`** for redirects - always pass `status_code` as an integer literal, never `status.HTTP_302_FOUND`.
|
||||
- **Return `templates.TemplateResponse("name.html", {...})`** from `devplacepy.templating` to render.
|
||||
- **Never create your own `Jinja2Templates` instance.** Import the shared one: `from devplacepy.templating import templates`.
|
||||
- **Register new routers in `main.py`:** `app.include_router(router_instance, prefix="/{path}")`
|
||||
- **`require_user(request)`** redirects guests (303) to login, but raises **401** when credentials *were* supplied yet invalid (so API clients get a clear error). Only post/comment/vote/etc. routes use it - the feed is public. `require_admin` and `require_user_api` (401-only) build on it. None of these changed signatures, so all auth schemes work through existing call sites.
|
||||
- **For a missing detail resource, `raise not_found("X not found")`** (`utils.py`) - it returns an `HTTPException(404)` that the global handler renders as `error.html`. Do not return a bare `HTMLResponse(..., status_code=404)`.
|
||||
- **Detail pages reuse `load_detail(table, target_type, slug, user)`** (`content.py`) for item+author+comments+attachments+`star_count`+`my_vote`; list pages reuse `enrich_items(items, key, authors, extra_maps, user=...)`. Prefer these over manual per-row loading (posts/projects/gists detail and feed/gists/profile lists already do).
|
||||
- **`get_current_user(request)` resolves ALL auth schemes** (`utils.py`), in order: `session` cookie -> `X-API-KEY` header -> `Authorization: Bearer <api_key>` -> `Authorization: Basic base64(username-or-email:password)`. It memoizes the result on `request.state._auth_user` (resolved once per request - it is called by both the maintenance middleware and the route) and caches users in `_user_cache` (by session token, `"k:"+api_key`, or a hash of the Basic header). Because every router already routes through this one function, API-key/Bearer/Basic auth work on every page/action with no per-route code. Use it for pages viewable by guests too (feed, news detail, projects).
|
||||
- **Post deletion must cascade:** delete comments and votes first, then the post. Always check ownership: `post["user_uid"] == user["uid"]`.
|
||||
- **Message deduplication needed** when `sender_uid == receiver_uid` (messaging yourself): `seen = set()` of message UIDs before appending to result list.
|
||||
|
||||
## Polymorphic comments and votes
|
||||
|
||||
The `comments` table uses `(target_type, target_uid)` so the same `_comment_section.html` component works for `post`, `project`, `gist`, and `news`. Votes follow the same shape via `/votes/{target_type}/{uid}`. `resolve_target_redirect()` in `comments.py` maps target_type back to the correct detail URL.
|
||||
|
||||
## Small feature notes: comment editing, post editing, inline comments
|
||||
|
||||
### Inline comment on feed cards
|
||||
|
||||
Every post card on the feed has an inline comment form (`.feed-comment-form`) beneath the post actions. It posts to `/comments/create` like the detail page comment form. Tests must scope Post button clicks to `#create-post-modal button.btn-primary:has-text('Post')` to avoid matching the inline comment's Post button.
|
||||
|
||||
### Comment editing
|
||||
|
||||
A comment's owner (only the owner, never an admin) sees an inline "Edit" button (`data-action='edit'`) in `_comment.html`. `CommentManager.toggleEditForm` swaps the `.comment-text` for a textarea seeded from its `data-raw` attribute (the raw markdown, since `contentRenderer.applyTo` overwrites `textContent` on first render), posts via `Http.send` to `POST /comments/edit/{comment_uid}`, then re-renders the new body in place with `contentRenderer.applyTo`. The route (`content.edit_comment_record`) is `is_owner`-only, writes `content` + `updated_at`, records the `comment.edit` audit event, and branches on `wants_json`: JSON clients get `CommentEditOut{uid, content, url, updated_at}`, the no-JS form falls back to a redirect to the comment anchor. Edits are NOT soft-delete related (the body is overwritten in place). Devii tool: `edit_comment` (owner-only, no confirm). Scope test Edit clicks to `.comment-action-btn:has-text('Edit')`.
|
||||
|
||||
### Post editing
|
||||
|
||||
Post owners see an "Edit" button on the post detail page that opens `#edit-post-modal`. The edit form allows changing title, content, and topic. The POST route is `/posts/edit/{post_uid}` with ownership check. The edit modal's textarea has `id="edit-content"` - tests must scope to `.comment-form textarea[name='content']` for comment operations.
|
||||
|
||||
## Gists
|
||||
|
||||
A dedicated `/gists` page for sharing code snippets. Uses `gists` table (auto-created by `dataset`).
|
||||
|
||||
### Database columns
|
||||
|
||||
| Column | Type | Notes |
|
||||
|--------|------|-------|
|
||||
| `uid` | text | UUID |
|
||||
| `user_uid` | text | FK -> users.uid |
|
||||
| `title` | text | Required, max 200 |
|
||||
| `description` | text | Optional, max 5000, markdown (rendered by ContentRenderer) |
|
||||
| `source_code` | text | Required, max 50000 |
|
||||
| `language` | text | One of 27 supported languages |
|
||||
| `slug` | text | `make_combined_slug(title, uid)` |
|
||||
| `stars` | int | Net vote count (via `/votes/gist/{uid}`) |
|
||||
| `created_at` | text | ISO datetime |
|
||||
|
||||
### Routes
|
||||
|
||||
| Method | Path | Handler | Auth |
|
||||
|--------|------|---------|------|
|
||||
| GET | `/gists` | `gists_page` | No |
|
||||
| GET | `/gists/{slug}` | `gist_detail` | No |
|
||||
| POST | `/gists/create` | `create_gist` | Yes |
|
||||
| POST | `/gists/delete/{slug}` | `delete_gist` | Yes (owner) |
|
||||
|
||||
### Polymorphic reuse
|
||||
|
||||
- **Comments**: Uses `_comment_section.html` with `target_type="gist"` - same component as posts/projects
|
||||
- **Voting**: Uses existing `/votes/gist/{uid}` route - updates `gists.stars`
|
||||
- **Content rendering**: Description rendered via `ContentRenderer.js` (`.rendered-content[data-render]`)
|
||||
- **Profile tab**: "Gists" tab between Projects and Activity on profile pages
|
||||
|
||||
### CodeMirror editor
|
||||
|
||||
- CodeMirror 5 loaded from CDN in `gists.html` via `{% block extra_js %}`
|
||||
- 22 language modes pre-loaded (Python, JS, TS, HTML, CSS, C, C++, Java, Go, Rust, SQL, Bash, YAML, Markdown, Swift, PHP, Ruby, Kotlin, Haskell, Lua, Perl, R, Dart, Scala)
|
||||
- `GistEditor.js` initializes CodeMirror on `#gist-source-editor` textarea
|
||||
- Language selector dropdown dynamically switches CodeMirror mode
|
||||
- `Ctrl+S` shortcut saves and submits the form
|
||||
- On form submit, `editor.save()` syncs CodeMirror content back to the hidden textarea
|
||||
|
||||
### Display
|
||||
|
||||
- Source code rendered in `<pre><code class="language-xxx">` block on detail page
|
||||
- Syntax highlighting handled by existing `highlight.js` loaded globally in `base.html`
|
||||
- Copy button uses `navigator.clipboard.writeText()`
|
||||
- Cards in listing show language badge, title, truncated description, author, star count
|
||||
|
||||
### Sitemap
|
||||
|
||||
- Latest 500 gists included in sitemap, `changefreq="weekly"`, `priority="0.6"`
|
||||
|
||||
## Feed and listing features
|
||||
|
||||
### Politics category
|
||||
|
||||
The `politics` topic is available as a feed filter sidebar item (icon `🏛`) and post topic. It defines the CSS variable `--topic-politics: #00bcd4` in `variables.css` and the `.badge-politics` class in `base.css`. It is validated like every topic via the canonical `TOPICS` list in `constants.py` (consumed by `models.py` `valid_topic`, `templating.py` `TOPICS` global, `routers/posts.py`, `docs_api`). Unlike the former `signals` topic, `politics` is also part of the bot fleet's rotation - it is in `services/bot/config.py` `CATEGORIES`/`FEED_TOPICS`, carries a modest per-persona weight in `PERSONA_CATEGORY_WEIGHTS`, and has a writing instruction in `services/bot/llm.py` `category_extras`.
|
||||
|
||||
### Date format
|
||||
|
||||
All dates displayed to users use European DD/MM/YYYY format. Implemented via:
|
||||
|
||||
- **`format_date(dt_str, include_time=False)`** in `utils.py` - converts ISO datetime -> `DD/MM/YYYY` or `DD/MM/YYYY HH:MM`
|
||||
- Registered as template global in `templating.py`: `{{ format_date(dt) }}`
|
||||
- **`time_ago()`** returns `DD/MM/YYYY` for items older than 30 days (instead of `"Xmo ago"`)
|
||||
- Services page has a JS `formatDate()` function for live polling updates
|
||||
|
||||
### Admin pagination
|
||||
|
||||
Both `/admin/users` and `/admin/news` use offset-based pagination via a reusable component:
|
||||
|
||||
- **`templates/_pagination.html`** - numbered page links with ellipsis, Previous/Next buttons, total count
|
||||
- Routes accept `?page=N` query param, clamped to valid range
|
||||
- `per_page = 25`, pagination metadata computed server-side and passed as `pagination` dict
|
||||
- Only renders when `total_pages > 1`
|
||||
- CSS in `admin.css` (`.pagination`, `.pagination-btn`, `.pagination-page`, `.pagination-ellipsis`)
|
||||
|
||||
### News detail and comments
|
||||
|
||||
News articles have an internal detail page at `/news/{slug}` with full comment support:
|
||||
|
||||
- **Route:** `GET /news/{news_slug}` in `routers/news.py` - resolves by slug first, then UUID
|
||||
- **Template:** `templates/news_detail.html` - shows image, source, grade, description, content, external link
|
||||
- **Comments:** Uses `_comment_section.html` with `target_type="news"` - same component as posts/projects
|
||||
- **`resolve_target_redirect()`** in `comments.py` handles `"news"` -> `/news/{slug}`
|
||||
- Listing links in `news.html` point to internal detail page; "Read on Source" still goes to external URL
|
||||
|
||||
### Home page (`GET /`)
|
||||
|
||||
The home route (`main.py` `landing()`) never redirects - it renders `templates/landing.html` for everyone, branching on `user`:
|
||||
|
||||
- **Guests** get the marketing hero (`Join DevPlace Free` CTA + features grid).
|
||||
- **Signed-in users** get a personalized hero (`.landing-hero-user`): avatar, "Welcome back, {username}", a `Go to your feed` CTA, a Posts/Stars/Level stat strip (`user_post_count` + the user dict's `stars`/`level`), and quick links. Styles live in `.landing-hero-user`/`.landing-welcome`/`.landing-stats`/`.landing-quicklinks` in `static/css/landing.css`.
|
||||
- Both states share the Latest Posts + Developer News + "Build With Us" sections. The **Build With Us** section is static HTML/CSS (`.landing-help-*` in `landing.css`): four cards linking to `/docs/index.html` (Documentation), `/swagger` + `/openapi.json` (API Reference), `/issues` (Contribute & Report), and Devii. The Devii card's `Launch Devii` button is a plain `<button data-devii-open>` that opens the globally mounted `DeviiTerminal` (`app.devii`) in place - no extra JS, route, schema, or Devii action; a secondary link points to `/devii/` for the full terminal page.
|
||||
- Context adds `user`, `is_authenticated`, `user_post_count`; `LandingOut` carries `is_authenticated`/`user_post_count` for the JSON form. `GET /` is documented in `docs_api.py` (id `home`, mapped to `LandingOut`).
|
||||
|
||||
### Author diversity (interleave, never drop)
|
||||
|
||||
Both the home page Latest Posts section and the main `/feed` *interleave* authors so one prolific account cannot fill a contiguous run, **without dropping any post** (the old per-author cap is gone). Helpers in `database.py`:
|
||||
|
||||
- `interleave_by_author(rows, uid_key="user_uid")` - pure reordering of a row list. Greedy and order-preserving: it walks the list and at each step emits the earliest row whose author differs from the last emitted one (deferring a same-author run to the next available author, recursively); when only same-author rows remain it emits them in order. The result is a permutation of the input - never a subset - so each author's own posts keep their relative (date) order and no two consecutive rows share an author unless the whole list is one author. The home route fetches the 6 newest posts and interleaves them.
|
||||
- `paginate_diverse(table, *clauses, ..., uid_key="user_uid", **filters)` - a drop-in replacement for `paginate` (same `(rows, next_cursor)` contract) that calls `paginate` then `interleave_by_author` on the page. Because the interleave is a pure permutation of the page, the cursor (oldest `created_at` in the page) is identical to `paginate`'s, so infinite scroll stays correct with no gaps/overlaps. `feed.py` `get_feed_posts` uses it for every tab.
|
||||
|
||||
When building any new "recent items" list, reuse these instead of raw `find(order_by=...)` / `paginate`.
|
||||
|
||||
### Listing search (feed, gists, projects)
|
||||
|
||||
The three public listings - `/feed`, `/gists`, `/projects` - share one free-text search box at the top of the left filter panel. Two reuse points keep it DRY and consistent:
|
||||
|
||||
- **Data layer:** `database.text_search_clause(table, search, fields=("title", "description"), author_field=None)` returns a single SQLAlchemy `or_(col.ilike("%term%") ...)` clause over the given columns (skipping any not present on the table), or `None` when `search` is blank or the table does not exist yet. Append it as a positional clause to `paginate` / `paginate_diverse` / `table.count(...)` **only when not None** (no search leaves the queries unchanged). Field mapping: projects/gists use `("title", "description")`, posts use `("title", "content")`. **Author-username matching:** posts/gists/projects store `user_uid`, not the author's username text, so a username query never matched a text column - the fix is the `author_field` argument. When set (all three listings pass `author_field="user_uid"`), `text_search_clause` resolves usernames matching the search term to uids via `database.get_uids_by_username_match(search)` (a capped `users.username LIKE` over the existing `idx_users_username` index) and OR-includes `columns[author_field].in_(uids)` in the same clause - no SQL JOIN, staying within the `dataset` query pattern. So each listing now matches its text fields **plus** the author username. `projects.py`, `gists.py`, `feed.py`, and the devRant `services/devrant/feed.py` all call it; never re-inline an `ilike` search clause and never add a JOIN - resolve username->uids and reuse `author_field`.
|
||||
- **View layer:** `templates/_sidebar_search.html` is the single search-box partial (the `Filter` heading + the GET form). Include it at the very top of `<aside class="sidebar-card">` with three locals: `_action` (the listing URL), `_placeholder`, and `_hidden` (a dict of `name -> value` rendered as hidden inputs so the active category/tab filter survives a search submit - e.g. `{"tab": current_tab, "topic": current_topic}` for the feed). It reads the `search` context var for the current value. Any future listing with a left filter panel reuses this partial plus `text_search_clause`; do not hand-roll another search form.
|
||||
|
||||
The `search` value is threaded back into each listing's context and exposed on `FeedOut.search` / `GistsOut.search` / `ProjectsOut.search`, documented as a `search` query param on the `feed-list` / `gists-list` / `projects-list` endpoints in `docs_api.py`, and offered to Devii as the `search` query param on `view_feed` / `list_gists` / `list_projects`.
|
||||
|
||||
### Recent comments on listing/feed cards (consistent across post, gist, project, news)
|
||||
|
||||
Every public listing card that shows the last few comments must render them with the **same visual hierarchy as the detail page** (depth indentation, vote rail, reply nesting). One pipeline drives all four surfaces - never inline a flat-only render:
|
||||
|
||||
- **Data layer:** `database.get_recent_comments_by_target_uids(target_type, target_uids, limit=3, user=None)` is the single batch helper. It runs one `ROW_NUMBER() OVER (PARTITION BY target_uid ORDER BY created_at DESC, id DESC)` window over `comments` filtered by `target_type=:tt` + `deleted_at IS NULL`, builds each item via `_build_comment_items` (identical dict shape to the detail page: `comment`/`author`/`time_ago`/`votes`/`my_vote`/`children`/`attachments`/`reactions`), then **nests the limited set by `parent_uid` into `children`** so replies render indented. Returns `{target_uid: [top-level items]}`. `get_recent_comments_by_post_uids` is now a thin wrapper delegating with `target_type="post"`, so `feed.py` is unchanged and there is no behavior drift.
|
||||
- **Routers:** `feed.py` (post), `gists.py` (gist), `projects/index.py` (project), `news.py` (news) each batch-fetch with the helper and attach `recent_comments` per item. Gists and news are wrapper dicts (`item["recent_comments"]`); projects are **flat dicts** (`project["recent_comments"]`).
|
||||
- **Schemas:** `FeedItemOut`, `GistItemOut`, `NewsListItemOut`, and the flat `ProjectListItemOut` all carry `recent_comments: list[CommentItemOut] = []`, so the `respond(model=...)` JSON exposes the same nested tree and never silently drops the key.
|
||||
- **View:** each card renders the block with the shared `_comment.html` `render_comment(c, 0)` macro inside a `.post-card-comments` wrapper (copied from `_post_card.html`). The macro recurses on `item.children` for indentation. **CSS requirement:** the template must load `post.css` (the `.comment`/`.comment-depth`/`.comment-replies` hierarchy) and `feed.css` (the `.post-card-comments` wrapper). `feed.html`/`gists.html`/`projects.html` already loaded `feed.css`; `news.html` loads neither by default, so it loads both.
|
||||
|
||||
### Landing page news
|
||||
|
||||
Articles can be toggled to appear on the landing page via `/admin/news/{uid}/landing`:
|
||||
|
||||
- **`show_on_landing`** field on `news` table
|
||||
- Landing route (`main.py` `GET /`) fetches up to 6 articles with `show_on_landing=1`
|
||||
- Rendered as a 3-column card grid with image, source, title, date (responsive -> 1 column on mobile)
|
||||
- Toggleable individually from the admin news table
|
||||
|
||||
### Public feed
|
||||
|
||||
The feed page (`GET /feed`) is accessible without authentication:
|
||||
|
||||
- Uses `get_current_user(request)` instead of `require_user()` - returns `None` for guests
|
||||
- Guests see posts but not the FAB, create modal, inline comment forms, or following tab
|
||||
- All POST routes (create, comment, vote) remain guarded by `require_user()`
|
||||
- Topnav shows Login/Sign Up for unauthenticated visitors; Messages, Admin, notifications for authenticated
|
||||
|
||||
## SEO implementation
|
||||
|
||||
All SEO features are implemented across the following locations:
|
||||
|
||||
### Core SEO utilities
|
||||
- `devplacepy/seo.py` - JSON-LD schema generators (WebSite, BreadcrumbList, DiscussionForumPosting, ProfilePage, SoftwareApplication), meta description truncation, schema combiner, sitemap XML generator
|
||||
- `routers/seo.py` - robots.txt and sitemap.xml routes
|
||||
|
||||
### SEO template context
|
||||
- Every router passes `page_title`, `meta_description`, `meta_robots`, `canonical_url`, `og_title`, `og_description`, `og_image`, `og_type`, `breadcrumbs`, `page_schema` via `base_seo_context()`
|
||||
- Auth pages: `noindex,nofollow`
|
||||
- Messages/Notifications: `noindex,nofollow`
|
||||
- Profiles with < 2 posts: `noindex,follow`
|
||||
- All other pages: `index,follow`
|
||||
|
||||
### Template layer
|
||||
- `templates/base.html` - dynamic `<title>`, `<meta description>`, `<link canonical>`, `<meta robots>`, Open Graph, Twitter Cards, JSON-LD injection, breadcrumb nav, CDN `dns-prefetch`/`preconnect`
|
||||
- `static/css/base.css` - `.breadcrumb` (aria-label breadcrumb nav), `.sr-only` (accessible hidden headings)
|
||||
|
||||
### Heading hierarchy
|
||||
- `feed.html` - `<h1 class="sr-only">Feed</h1>`
|
||||
- `profile.html` - username rendered as `<h1 class="profile-name">`
|
||||
- `messages.html` - `<h1 class="sr-only">Messages</h1>`
|
||||
- `projects.html` - `<h1>Projects</h1>`
|
||||
- `post.html` - post title as `<h1>`, "Related Discussions" as `<h3>`
|
||||
|
||||
### Post slugs
|
||||
- Slug generated on post creation via `slugify()` and stored in `posts.slug` column
|
||||
- Posts can be looked up by slug or UUID
|
||||
- Minimum content validation: post body >= 10 chars, comment >= 3 chars
|
||||
|
||||
### Related posts
|
||||
- `templates/post.html` - "Related Discussions" widget at bottom of post page (queried by matching topic)
|
||||
|
||||
### Performance
|
||||
- `loading="lazy"` on all avatar images
|
||||
- `dns-prefetch` + `preconnect` for CDN resources in `<head>`
|
||||
- Security headers middleware: `X-Robots-Tag`, `X-Content-Type-Options`
|
||||
|
||||
### Default OG image
|
||||
- `static/og-default.svg` - 1200x630 SVG with DevPlace branding
|
||||
- Used as fallback `og:image` on all pages
|
||||
|
||||
### SEO tests
|
||||
- SEO tests are split by surface: `tests/api/robotstxt.py`, `tests/api/sitemapxml.py`, and `tests/unit/seo.py` plus the per-page e2e checks - covering robots.txt, sitemap.xml, page titles, noindex, canonical URLs, OG tags, Twitter cards, structured data, security headers
|
||||
|
||||
## Engagement: reactions, bookmarks, polls, contribution heatmap
|
||||
|
||||
### Emoji reactions
|
||||
- Curated palette only: `REACTION_EMOJI` in `constants.py` (registered as a template global). `ReactionForm` rejects anything outside it; free-text emoji are not allowed.
|
||||
- Endpoint `POST /reactions/{target_type}/{target_uid}` (`routers/reactions.py`) toggles one `(user, target, emoji)` row in the `reactions` table. Target types: `post`, `comment`, `gist`, `project`. AJAX (`x-requested-with: fetch`) returns `{counts, mine}`.
|
||||
- Reactions are **non-ranking** - they never touch `stars` or XP and intentionally send **no notifications** (votes already notify; reactions would be notification spam).
|
||||
- Batch reads via `get_reactions_by_targets(target_type, uids, user)` in `database.py` (used by feed, profile, comment loader, `load_detail`) - never per-row. The `_reaction_bar.html` partial takes `_type`, `_uid`, `_reactions` ({counts, mine}) and renders the full palette as toggle chips; `ReactionBar.js` uses document-level click delegation. All four engagement controllers (`ReactionBar`, `VoteManager`, `BookmarkManager`, `PollManager`) extend the shared `OptimisticAction` base (the `Http.sendForm -> render -> error` core); each keeps only its own event wiring and `_render`.
|
||||
|
||||
### Bookmarks
|
||||
- `POST /bookmarks/{target_type}/{target_uid}` toggles a `bookmarks` row; `GET /bookmarks/saved` renders the personal list (`saved.html`). Target types: `post`, `gist`, `project`, `news`.
|
||||
- `_bookmark_button.html` takes `_type`, `_uid`, `_bookmarked`; `BookmarkManager.js` swaps the label/`bookmarked` class from the JSON `{saved}`. Batch state via `get_user_bookmarks(user_uid, target_type, uids)`.
|
||||
|
||||
### Polls
|
||||
- A poll rides on a post (one `polls` row keyed by `post_uid`, options in `poll_options`, one-per-user votes in `poll_votes`). Created in `posts.py:create_poll` when `poll_question` plus >= 2 non-empty `poll_options` are submitted (capped at 6). Both `create_post` and `edit_post` accept the poll fields; `edit_post` only attaches a poll when the post has **none** yet (it never replaces an existing poll). The builders live in the create-post modal (`feed.html`) and the edit-post modal (`post.html`, rendered only when the post has no poll) using `data-poll-toggle` / `data-poll-add-option`.
|
||||
- `poll_options` accepts either repeated form fields (the web builders, which preserve commas inside an option label) **or** a single newline- or comma-separated string (the API/Devii path). `models.py:normalize_poll_options` splits a lone delimited element - applied as a `mode="before"` validator on `PostForm`/`PostEditForm` - so the documented "one per line or comma separated" agent format actually produces a multi-option poll instead of a single dropped option.
|
||||
- `POST /polls/{poll_uid}/vote` toggles the voter's choice and returns the full poll dict: voting on a new option switches, voting the **same** option again **retracts** (one vote per user, like reactions). Results (bars + `%`) are **always shown** to everyone; the chosen option gets `.chosen` + a checkmark. Guests see read-only bars with a "Log in to vote" hint (options `disabled`). Batch reads via `get_polls_by_post_uids(post_uids, user)` / `get_poll_for_post(post_uid, user)`.
|
||||
- Composer poll builder (`feed.html`, `PollManager.js`): poll inputs are `disabled` until "Add poll" is opened (so a never-opened or "Remove poll"-collapsed builder submits **nothing**); a capture-phase `submit` listener blocks an incomplete poll (question + >= 2 options) with an inline error instead of silently dropping it - capture phase + `stopImmediatePropagation()` is required so it pre-empts `FormManager`'s bubble-phase submit handlers.
|
||||
|
||||
### Cascade
|
||||
- `soft_delete_engagement(target_type, uids, deleted_by)` in `database.py` soft-deletes reactions, bookmarks and (for posts) poll rows. It is called from `content.py:delete_content_item` (for the item and its comments) and `comments.py:delete_comment` (which now soft-deletes the comment plus its attachments, votes, and engagement under one `stamp`). Add it to any new delete path. The unfiltered hard `delete_engagement` remains for GC only.
|
||||
|
||||
### Contribution heatmap and streaks
|
||||
- Derived, **no new table**: `get_activity_calendar(user_uid)` aggregates `date(created_at)` across `posts`/`comments`/`gists`/`projects` (cached 120s); `get_activity_heatmap` returns 53 Monday-aligned weeks of `{date, count, level}` (level 0-4); `get_streaks` returns `{current, longest}`. Rendered server-side in `profile.html` (`.heatmap-grid`, styles in `profile.css`). The `On Fire` badge is awarded in `check_milestone_badges` when the current streak >= 7.
|
||||
|
||||
### Follow graph listings
|
||||
- `profile.html` has Followers and Following tabs (`?tab=followers` / `?tab=following`, `?page=N`). `profile_page` computes `get_follow_counts(uid)` for the tab labels and, only for those two tabs, `_follow_people(uid, tab, current_user, page)`. The same two listings are exposed as JSON at `GET /profile/{username}/followers` and `GET /profile/{username}/following` (public, 25 per page) returning `{username, mode, count, page, total_pages, <mode>: [{uid, username, bio, is_following}]}`.
|
||||
- `database.py` helpers: `get_follow_counts(uid)` (`{followers, following}`), `get_follow_list(uid, mode, page)` (paginated people + `build_pagination`, ordered newest-first), and `get_following_among(follower_uid, target_uids)` (single IN-clause set used to set `is_following` per row, avoiding N+1). `mode` is `"followers"` (people who follow `uid`) or `"following"` (people `uid` follows).
|
||||
- Devii catalog tools `list_followers` / `list_following` (`requires_auth=False`) map to the JSON endpoints; documented in `docs_api.py` under the `profiles` group.
|
||||
|
||||
### Block and mute (`routers/relations.py`)
|
||||
A logged-in user can **block** or **mute** another user; both are one-directional and reversible. **Block** hides every piece of the blocked user's content from the blocker - posts, comments (any category), feed, listings, issue list, detail pages, and DMs - everywhere EXCEPT the blocked user's own profile page (kept fully visible so the blocker can review and unblock), and it also suppresses any notification that user would generate. **Mute** is the lighter option: it only suppresses the muted user's notifications while their content stays visible. The blocked/muted user is unaffected and is not told.
|
||||
|
||||
- **One table** `user_relations(uid, user_uid, target_uid, kind, created_at, deleted_at, deleted_by)` in `SOFT_DELETE_TABLES`; `user_uid` is the actor/owner, `kind` is `"block"` or `"mute"`. Indexes `idx_user_relations_owner_kind` (`user_uid, kind`) and `idx_user_relations_target_kind` (`target_uid, kind`, backs the DM-send reverse check), ensured in `init_db`.
|
||||
- **One cached accessor** `database.get_user_relations(viewer_uid) -> {"block": frozenset, "mute": frozenset}` (single query, both kinds), cache-invalidated under the `"relations"` cache-version name; anonymous viewer returns empty frozensets with **zero queries**. Derived helpers `get_blocked_uids`, `get_muted_uids`, `get_silenced_uids` (block | mute) and `invalidate_user_relations(uid)` (local pop + `bump_cache_version("relations")`). This is read on every content listing, so it must stay cache-hot - never query `user_relations` directly in a read path.
|
||||
- **Filtering choke points (DRY, do not re-implement inline):** `paginate`/`paginate_diverse` accept a `viewer_uid` kwarg that appends `user_uid NOT IN (blocked)` only when the table has `user_uid` and the block set is non-empty (wired in `feed`/`gists`/`projects` listings; safe no-op for bookmarks/news); the three comment loaders drop blocked authors via `database._drop_blocked(raw, user)` before building trees; `content.load_detail` returns `None` (404) for a blocked author; the landing query and the issues list filter by the block set; messages filter the conversation list/thread and `persist_message` rejects a DM when the recipient blocked the sender (returns `None`, already handled).
|
||||
- **Notification suppression is one line** at the single funnel `utils._deliver_notification`: `if related_uid and related_uid in get_silenced_uids(user_uid): return`. This covers follows, mentions, DMs, votes, reactions at once (block silences too, hence `get_silenced_uids`). A non-user `related_uid` is harmless (uuids never collide).
|
||||
- **Routes** mirror `follow.py`: `POST /block/{username}`, `POST /block/unblock/{username}`, `POST /mute/{username}`, `POST /mute/unmute/{username}` (born-live insert / soft-delete revive, `invalidate_user_relations`, best-effort audit `relation.block|unblock|mute|unmute`, `action_result`). Mounted with no prefix.
|
||||
- **Fan-out:** profile route passes `is_blocked`/`is_muted` (named to avoid the `is_self` Jinja-global collision rule) on the `respond` context + `ProfileOut`; `profile.html` renders Block/Unblock + Mute/Unmute POST forms next to Follow using the existing `data-confirm` (+`data-confirm-danger`) pattern - **no new JS**. Devii actions `block_user`/`unblock_user`/`mute_user`/`unmute_user`; docs in `docs_api.py`; audit prefix `relation` -> category `social`.
|
||||
|
||||
### CSS
|
||||
- Reactions, bookmarks, polls and the saved page live in `static/css/engagement.css`, loaded globally in `base.html` (the controls appear across feed, detail pages and comments). Heatmap styles are in `profile.css`. Follow-list rows (`.follow-row`, `.follow-user`, `.follow-pagination`) are in `profile.css`.
|
||||
|
||||
## Profile activity tab (clickable comments)
|
||||
|
||||
The profile **Activity** tab (`/profile/{username}?tab=activity`, public) interleaves the user's 10 most recent posts and 10 most recent comments, newest first. Each activity item is a plain dict built inline in `routers/profile/index.py` (no dedicated DB helper) and carries a `url` so the whole card is clickable, exactly like the feed's post cards:
|
||||
|
||||
- **Post items** get `"url": resolve_object_url("post", p["uid"])` -> `/posts/{slug}`.
|
||||
- **Comment items** get `"url": f"{resolve_object_url(target_type, target_uid)}#comment-{c['uid']}"`, the SEO-friendly parent detail URL plus the `#comment-{uid}` anchor. This is **byte-identical to a comment notification's `target_url`** (`content.create_comment_record`), so clicking an activity comment reproduces the notification click exactly: it lands on the parent post and `NotificationManager.js` scrolls to / highlights the `id="comment-{uid}"` element (`_comment.html`). Reuse `resolve_object_url` (`database.py`) - never rebuild this URL by hand, and do not call `resolve_object_url("comment", uid)` here (it re-fetches the row the loop already has).
|
||||
- **Frontend:** the card is wrapped in the shared full-card overlay link (`card-link-host` + `_card_link.html`), the same pattern `_post_card.html` uses; inner content links (mentions, URLs rendered by `render_content`) stay clickable via the existing `card-link-host a:not(.card-link)` z-index rule (`base.css`). No new CSS.
|
||||
- **API:** `ProfileOut.activities` is `list[Any]`, so the added `url` (and, on comments, the existing `target_type` + `uid` = parent target uid) flow into the JSON response with no schema change - this is the parent-post reference exposed on each comment item.
|
||||
23
devplacepy/routers/devrant/CLAUDE.md
Normal file
23
devplacepy/routers/devrant/CLAUDE.md
Normal file
@ -0,0 +1,23 @@
|
||||
This file documents the devRant compatibility API mounted at `/api`. Claude Code auto-loads it whenever a file in `routers/devrant/` is read or edited.
|
||||
|
||||
## Routing overview
|
||||
|
||||
- `/api` - `devrant/` package: `devRant`-compatible REST protocol (`auth.py`, `rants.py`, `comments.py`, `notifs.py`). Translates devRant requests onto DevPlace data: rants<->posts, comments/votes onto the native engagement layer, devRant integer ids onto each table's auto-increment `id`. Token auth via `devrant_tokens`.
|
||||
|
||||
## devRant compatibility API (`routers/devrant/`, `services/devrant/`)
|
||||
|
||||
A second REST protocol mounted at `/api` that reproduces the public devRant API shape on DevPlace data, so legacy devRant clients (the `./devranta` reference client is the spec; real captured responses are in `./devranta/api_test_results.json`) run unchanged. It is a pure **translation layer** over the existing domain - it adds no new content model.
|
||||
|
||||
**Layout.** `routers/devrant/` is the thin endpoint package: `__init__.py` builds the aggregate router with `dependencies=[Depends(ensure_enabled)]` (gated by the `devrant_api_enabled` setting, default on) and includes `auth.py` (login/register/profile/edit/avatar), `rants.py` (feed, CRUD, vote, favorite, comment, search), `comments.py` (read/edit/delete/vote), `notifs.py` (feed/clear); `_shared.py` holds the `dr_ok`/`dr_error` envelope helpers + the enable gate. `services/devrant/` is the logic: `params.merge_params` (merges query string + body, accepting BOTH form-encoded and JSON since devRant clients use both, GET sends auth as query params), `tokens` (issue/validate/revoke the `devrant_tokens` triple), `ids` (`as_int`, `to_unix`, `post_by_id`/`comment_by_id`/`user_by_id`), `serializers` (`serialize_rant`/`serialize_comment`, `encode_tags`/`decode_tags`), `feed` (`list_rants`/`search_rants`/`load_rant_detail`, offset pagination via `find(_limit, _offset)`), `profile` (`build_profile`), `notifications` (`build_notif_feed`/`clear_notifications`), `avatar` (`avatar_payload` + `render_png` via `cairosvg`).
|
||||
|
||||
**ID mapping (load-bearing).** devRant integer ids ARE the auto-increment `id` PK every `dataset` table already has: `rant_id`=`posts.id`, `comment_id`=`comments.id`, `user_id`=`users.id`, `token_id`=`devrant_tokens.id`. No translation table exists - `post_by_id` is `find_one(id=...)`. Serialization converts ISO `created_at` to unix via `ids.to_unix`.
|
||||
|
||||
**Auth.** `POST /api/users/auth-token` accepts username OR email, verifies with passlib, and inserts a `devrant_tokens` row (in `SOFT_DELETE_TABLES`; born-live; `key`=`secrets.token_hex`, `expire_time` from `session_max_age_days`). Every later call re-validates `(token_id, token_key, user_id)` with `tokens.resolve_user(params)`. Read endpoints take an OPTIONAL viewer (`resolve_user` may return None); write endpoints return `_shared.unauthorized()` (401) when it does.
|
||||
|
||||
**Writes reuse the audited native cores - never duplicate.** Implementing this drove four DRY extractions in `content.py` (`apply_vote`, `create_comment_record`, `delete_comment_record`, `set_bookmark`) and one in `utils.py` (`register_account`); the native `routers/votes.py`, `routers/comments.py`, and `auth/signup.py` were refactored onto the SAME functions. So a devRant rant/comment/vote awards XP, fires notifications, writes the audit row, and soft-deletes exactly like the UI path. Rant create calls `content.create_content_item` directly; rant delete calls `content.delete_content_item` (full cascade) and returns the devRant envelope.
|
||||
|
||||
**Field mappings.** Rant `text` = `title\n\n content` (inbound rants have no title, `topic` forced to `"rant"`). devRant `tags` round-trip verbatim through a new `posts.tags` JSON column (`init_db` ensures it; `decode_tags` falls back to `[topic]`). `profile_skills` is derived from `bio` (`skills_from_bio`, no native skills field). `favorite`/`unfavorite` map to bookmarks via `set_bookmark`. Avatars are real PNGs from the multiavatar engine at `GET /api/avatars/u/{username}.png`; `user_avatar.i` points at that path with a deterministic `b` background colour. The feed envelope includes the auxiliary devRant keys (`settings`, `set`, `wrw`, `dpp`, `num_notifs`, `unread`, `news`) with safe values so clients parse cleanly.
|
||||
|
||||
**Envelope.** `dr_ok(**f)` -> `{success:true, **f}`; `dr_error(msg, status, **extra)` -> `{success:false, error:msg, ...}`. Logical failures stay `200` except bad login -> `400` (matches the captured devRant behaviour). Reference client + captured responses live in `./devranta/`. Host routing (legacy clients hard-code devrant.com) is an infra/DNS concern, out of app scope.
|
||||
|
||||
**Cross-cutting note.** AI content correction (`devplacepy/services/correction.py`) hooks the two devRant direct-edit paths alongside the native content/comment/messaging entrypoints, so it applies identically across the web UI, REST/JSON API, Devii, and devRant.
|
||||
71
devplacepy/routers/docs/CLAUDE.md
Normal file
71
devplacepy/routers/docs/CLAUDE.md
Normal file
@ -0,0 +1,71 @@
|
||||
This file documents the documentation site (`/docs`) - prose pages, API reference generation, search, and the interactive tester. Claude Code auto-loads it whenever a file in this directory is read or edited.
|
||||
|
||||
## Routing overview
|
||||
|
||||
- `(none)` - `docs.py` (`docs/` package), the documentation site (prose pages + API reference). See `DOCS_PAGES` below.
|
||||
|
||||
## Documentation site (`/docs`)
|
||||
|
||||
`routers/docs/ package` serves the docs. `DOCS_PAGES` (`routers/docs/pages.py`, re-exported from the `routers/docs` package; handlers in `routers/docs/views.py`) keeps curated prose pages (`kind: "prose"`, each with its own template under `templates/docs/<slug>.html`) and spreads `api_doc_pages()` from `devplacepy/docs_api.py` for every API reference page. Prose pages grouped by `section`: `General` (`index`, `getting-started`, `devii`, `dashboard`, `media-gallery`, `notification-settings`), admin-only `Devii internals` (`devii-*`), admin-only `Services` (`services-*`, documenting the `BaseService` framework and every background service including the live data at `GET /admin/services/data`), and admin-only `Production` (`production-*`, documenting the deployment). A page is admin-gated by `"admin": True`; search/export pick it up automatically by slug and respect the admin flag - no extra wiring.
|
||||
|
||||
## Audience tiers and navigation
|
||||
|
||||
`DOCS_PAGES` entries take optional `admin: True` (hidden + 404 for non-admins, but still indexed and surfaced only to admins by `docs_search`) and `section: "..."` (a nested sidebar group rendered by `docs_base.html`). The sidebar groups `section`s under four ordered **audience tiers** (`AUDIENCES` in `routers/docs/pages.py`): `Start here` (General), `Build with the API` (API, Components, Styles), `Contribute and internals` (Architecture, Services, Devii internals, Bots internals, Testing, Claude Code), and `Operate` (Administration, Production). `nav_groups(visible_pages)` builds the `[(audience, [(section, [pages])])]` tree from the flat visible-page list (so a section's pages collect under one heading regardless of `DOCS_PAGES` order or the API/Administration interleave from `api_doc_pages()`); `views.py` passes it as `nav`, and `docs_base.html` renders an audience super-header (`.sidebar-tier`) above each section subheading (`.sidebar-subheading`). `DOCS_PAGES` stays the canonical list for search/export/routing - the tiering is sidebar-only.
|
||||
|
||||
The public `getting-started` page (`SECTION_GENERAL`) is the new-contributor on-ramp (install/run, the four-faces workflow, validation); gate its deep-internals links with `{% if is_admin(user) %}` so guests get no 404s. Keep one canonical home per concept: the `auth` API group intro in `docs_api.py` defers method detail to the `authentication` prose page rather than re-listing the four methods. The member-facing `devii` prose page is functional; admins also get a `Devii internals` section of `devii-*` technical subpages.
|
||||
|
||||
## Single source of truth
|
||||
|
||||
`docs_api.py` `API_GROUPS` defines each reference page and its endpoints (method, path, auth, params, notes, sample_response). Add an endpoint there and the page, sidebar link, code examples, and interactive runner appear automatically.
|
||||
|
||||
## Downloads
|
||||
|
||||
`devplacepy/docs_export.py`: `/docs/download.md` returns the entire docs as one Markdown file; `/docs/download.html` returns a single **self-contained** HTML (vendored `marked` + `highlight.js` + theme inlined, the Markdown embedded base64 and rendered on open - works offline). Both are admin-filtered like the rest of the docs and reuse the same group/prose sources. Routes are declared before `/docs/{slug}.html` so `download.html` isn't caught by the slug pattern.
|
||||
|
||||
## Docs search (`/docs/search.html`)
|
||||
|
||||
A backend-rendered BM25 search over all docs pages, in `devplacepy/docs_search.py`. The corpus = prose page text (template stripped of Jinja/HTML) + each API group's intro/endpoints + the live services group; the inverted index (postings + idf) is built **once** lazily (`get_index()`, cached module-global) so a query is just postings lookups (~120us). `docs.py` special-cases `slug == "search"` (before the page registry) and renders `docs/_search.html` inside `docs_base.html` (`kind == 'search'`). Admin-only pages are filtered from results for non-admins, exactly like the sidebar. Snippets are HTML-escaped then wrapped in `<mark>` (XSS-safe). `_strip` removes `<script>`/`<style>` blocks and runs `_demarkdown` (drops headings, list/quote markers, table pipes, code fences, link/image syntax, and backtick/asterisk/tilde emphasis, but keeps `_` so identifiers like `owner_kind` stay searchable), so both the index and the snippets read as clean prose rather than raw markdown. The former `search` API group (user/recipient lookups) was renamed to slug **`lookups`** ("Search & Lookups") to free the `search` slug - keep that in mind if adding endpoints there.
|
||||
|
||||
## Token substitution
|
||||
|
||||
Intros/notes/examples may use `{{ base }}`, `{{ username }}`, `{{ api_key }}` - these are substituted server-side by `render_group()` (plain string replace, not Jinja), because the registry is data, not template source.
|
||||
|
||||
## Rendering pipeline (`kind` branch, prose rendering)
|
||||
|
||||
`docs_base.html` renders api pages via `_api_page.html` -> `_endpoint.html` per endpoint. Prose pages are rendered **server-side**: `docs.py` calls `docs_prose.render_prose(slug, ctx)` (`devplacepy/docs_prose.py`, mistune GFM with tables/strikethrough/url + `hard_wrap` to mirror the client `marked` config), which renders the template, converts the single `<div class="docs-content" data-render>` markdown block to HTML (after `html.unescape`, matching the client's `textContent` read), and **strips `data-render`**. `docs_base.html` outputs the result via `{{ prose_html|safe }}`. This eliminates the client markdown-to-HTML flicker and improves first paint. Anything outside that block (component live-demo blocks + their `<script type="module">`, the `devii.html` hero/CTA) is passed through verbatim. `hljs` highlighting and `CodeCopy` still run client-side over the now-server-rendered `<pre><code>` (text is already present, only colors/copy button appear after JS).
|
||||
|
||||
Every rendered `h2`/`h3` automatically gets a slugified `id` plus a hover permalink (`docs_prose._anchor_headings`, `heading_slug`), so any prose page can deep-link its sections; a page wanting a contents index places a `.docs-toc` nav (styled in `docs.css`) OUTSIDE the `data-render` block linking to those slugs (see `isslop-checks`). Authored example markup inside that block is therefore still HTML-escaped (`<dp-...>`); content outside the block passes through untouched. User-generated content elsewhere still uses the client `data-render` pipeline.
|
||||
|
||||
The public `Components` section (`component-*` prose pages) documents the custom web components with a **live, interactive example** on each page. The public `Styles` section (`styles`, `styles-colors`, `styles-layout`, `styles-responsiveness`, `styles-consistency`) is the design-system reference: the colour tokens and their meaning, the approved page layouts, the responsive breakpoint ladder, and the HARD structural rules every page must follow (taken from the feed/posts page as the canonical implementation). It uses the same live-demo convention (real demo markup OUTSIDE the `data-render` block, example markup inside it entity-escaped).
|
||||
|
||||
**`data-render` destroys inner HTML** (it renders `textContent`). Only group-intro / prose markdown lives inside a `data-render` block; every endpoint card, `[data-api-tester]` mount, and component live-demo lives OUTSIDE it. A prose page's `<div class="docs-content" data-render>` is rendered client-side via marked + DOMPurify on `textContent`, so any example markup shown as code inside it MUST be HTML-escaped (`<dp-dialog>`) or the browser parses it as a real element before render; the live demo itself goes in a separate block OUTSIDE the `data-render` div, where a `<script type="module">` (which imports its own component module, since it executes before `Application.js`) wires it up.
|
||||
|
||||
**`data-config` must be single-quoted:** `_endpoint.html` emits `data-config='{{ endpoint|tojson }}'`. `tojson` escapes `'` to `'`, so single quotes are safe; double quotes would break.
|
||||
|
||||
## Interactive API tester
|
||||
|
||||
The reusable widget is `static/js/ApiTester.js` (one instance per `[data-api-tester]`, wired by `ApiDocs.js` loaded in the docs `extra_js` block). It builds the param form, a **response-format picker**, live cURL/JS/Python tabs, a Send button that runs the real call, and a two-tab response area. It reads `window.DEVPLACE_DOCS` (`base`, `loggedIn`, `username`, `apiKey`, `isAdmin`) injected in `docs_base.html`. **Code blocks** are decorated by the shared `static/js/CodeBlock.js` (highlight via `hljs` + a line-number gutter + an always-visible Copy button); the widget calls `CodeBlock.refresh(pre)` on every tab switch (re-highlights - it clears `data-highlighted` first, the fix for stale tabs), the response panes use `CodeBlock.enhance(pre, {lineNumbers:false})`, and `CodeCopy.js` runs `CodeBlock.enhance` over prose `.docs-content pre`. Styling lives in `docs.css` (`.code-pre`/`.code-gutter`/`.code-has-copy`, `.format-picker`/`.format-option`, `.response-tabs`/`.response-pane`).
|
||||
|
||||
## Response-format negotiation
|
||||
|
||||
Every endpoint dict carries a `negotiation` field, set by `_classify()` in `docs_api.py` (not hand-written): `"negotiable"` (page GETs + action POSTs - toggle JSON/HTML via the `Accept` header), `"ajax"` (votes/reactions/bookmarks/polls - JSON via `X-Requested-With: fetch`, HTML shows the redirect), `"json"` (always JSON), `"none"` (avatar/proxy/redirect - no body). The picker **defaults to JSON** everywhere; `ApiTester.headerPairs()` adds `Accept: application/json|text/html` accordingly and only sends `X-Requested-With` for ajax endpoints in JSON mode, so the live request **and** the generated snippets stay in sync. The **Expected** tab (always visible, default) renders the endpoint's `sample_response`; the **Live response** tab fills in after Send.
|
||||
|
||||
## Runnable scope
|
||||
|
||||
Page GETs are `interactive: true` (they get a Send button, gated by `ctx.loggedIn`/`ctx.isAdmin` for user/admin endpoints). Mutations use `destructive: true` (confirm dialog). Only `avatar`, `gateway-passthrough`, `notifications-open`, `push-register`, `profile-regenerate-key`, and `profile-regenerate-avatar` stay `interactive: false` (image/proxy/redirect/side-effecting) - they still show the Expected tab. Admin endpoints (`auth: "admin"`) only run for admins.
|
||||
|
||||
## Enum params
|
||||
|
||||
Pass `options`, never hardcode allowed values in prose. A `field(... type="enum", options=[...])` auto-renders an "Allowed: a, b, c" line under the control in the live tester (`ApiTester.js` `buildParams`) and appends `Allowed: ...` to the Description column in the Markdown/HTML export (`docs_export.py` `_params_table`). Source enum lists from `devplacepy/constants.py` (`TOPICS`, `REACTION_EMOJI`) or the model `Literal`s (`PROJECT_TYPES`, `VOTE_TARGETS`) so docs stay in sync. Do NOT spell the values into the description - it would duplicate and drift.
|
||||
|
||||
## Minimal role documentation and validation
|
||||
|
||||
The `endpoint()` factory derives `min_role` from `auth` (`public` -> Public, `user` -> Member, `admin` -> Admin); it is rendered as the "Minimal role:" badge in `_endpoint.html` and as `*Minimal role:*` in the Markdown/HTML export. The `auth` value MUST reflect the real enforcement: `tests/api/auth/matrix.py` drives every documented endpoint (incl. the dynamic services group) as anonymous, member, and admin and asserts the enforcement matches the doc - public allows anonymous, `user` rejects anonymous (401/login-redirect), `admin` rejects a non-admin member (403 /feed-redirect). It forces explicit roles to survive the `is_first`-becomes-Admin rule. If you add/relax a route's auth, update its `auth` in `docs_api.py` or this test fails. (Example caught by it: `/notifications/counts` uses `get_current_user` and returns zeros to guests, so it is documented `public`, not `user`.)
|
||||
|
||||
## Admin-only pages
|
||||
|
||||
A group with `"admin": True` (currently `services`, `admin`) is hidden from the sidebar (`docs.py` filters `visible_pages`) and returns 404 for non-admins (`page.get("admin") and not is_admin -> not_found`, same message as unknown slugs). `index.html` wraps the operator links in `{% if user and user.get('role') == 'Admin' %}`.
|
||||
|
||||
## Dynamic Background Services page
|
||||
|
||||
The `services` group is a placeholder (`"dynamic": True`, empty endpoints). `docs.py` branches on `dynamic` and calls `build_services_group(service_manager.describe_all(), base)` to generate it live from the registered services and their `ConfigField` specs (one `POST /admin/services/{name}/config` card per service, `*_enabled` fields excluded). Add a service to `main.py` startup and it documents itself - keep `docs_api.py` import-pure (no `describe_all()` at import time).
|
||||
44
devplacepy/routers/projects/CLAUDE.md
Normal file
44
devplacepy/routers/projects/CLAUDE.md
Normal file
@ -0,0 +1,44 @@
|
||||
This file documents the project detail page, the per-project virtual filesystem, and the visibility/read-only UI-level behavior for code under `routers/projects/`. Claude Code auto-loads it whenever a file in this directory is read or edited.
|
||||
|
||||
## Routing overview
|
||||
|
||||
- `/projects` - `projects/` package: `index.py` (listing/detail/create/delete plus owner-only visibility toggles `POST /{slug}/private` and `POST /{slug}/readonly`, and async `POST /{slug}/fork`), `files.py`, and the `containers/` subpackage. `main.py` mounts the whole `/projects` tree from this one package.
|
||||
- `/projects/{slug}/files` - `projects/files.py`, the per-project virtual filesystem (CRUD dirs/files, upload, inline edit, plus line-range ops: `lines` read, `replace-lines`, `insert-lines`, `delete-lines`, `append` for large text files).
|
||||
|
||||
## Project Detail Page
|
||||
|
||||
Each project card links to `/projects/{project_uid}` showing full project details, author info, platforms, star count, delete-for-owner, and (for the owner) Private/Read-only toggle buttons plus badges (see **Project visibility and read-only** below). The route is `GET /projects/{project_uid}` in `routers/projects/index.py` and 404s when the viewer cannot see a private project. The sitemap generator links to this URL (not the old `?user_uid=` query param). The detail page also links to the project filesystem at `/projects/{slug}/files`.
|
||||
|
||||
**Action row overflow.** The detail page has more actions than fit one line, so `project_detail.html` keeps the engagement actions inline (Files, Share, star vote, bookmark, reactions) and collapses the rest behind a single **More** button (`.project-actions-more`) that opens the shared `app.contextMenu`. The secondary actions (Containers, Download zip, Fork, the owner Edit/Private/Read-only/Delete controls) live as real elements inside a hidden `.project-actions-overflow` container, each tagged `data-menu-action` plus `data-menu-icon`/`data-menu-label`. `static/js/ProjectActionsMenu.js` builds the menu from those elements and each item's `onSelect` simply `.click()`s the real element, so all existing wiring is reused unchanged - `app.zipDownloader` (`data-zip-download`), `app.projectForker` (`data-fork-project`), `data-share`, the `data-modal` Edit trigger, and the delegated `data-confirm`/`data-confirm-danger` dialog on the owner forms. The open handler must `stopPropagation()` because `app.contextMenu`'s document-level close listener would otherwise dismiss it on the same click (every other caller opens it from a right-click `attach`, not a left-click). Reuse this pattern - a `More` trigger over `[data-menu-action]` real elements - for any future action row that overflows; do not duplicate controller logic into menu callbacks.
|
||||
|
||||
**Owner editing.** Mirrors post editing exactly: an owner-only **Edit** menu item (`data-modal="edit-project-modal"`) opens the `modal()` macro's `edit-project-modal`, a plain `POST` form to `/projects/edit/{slug}` (route `edit_project` in `routers/projects/index.py`, body `ProjectEditForm`, owner-gated through the shared `content.edit_content_item` which returns 403 JSON / redirect for non-owners and stamps `updated_at`). The modal is the create modal pre-filled from the `project` row (title, description, type/status radios pre-checked, dates via `format_date()` back to DD/MM/YYYY). `is_private`/`read_only` are NOT edited here - they stay on their dedicated toggles. The platforms tag widget reuses the create modal's `platforms-input`/`platforms`/`platforms-tags` ids; `ProfileEditor.initPlatformTags` now **seeds existing tags** from the hidden `#platforms` value on load, so both the empty create form and the pre-filled edit form work from the same code. Devii tool `edit_project`; documented in `docs_api.py` (`projects-edit`).
|
||||
|
||||
## Project Filesystem
|
||||
|
||||
Each project carries a virtual filesystem so it can hold a whole software project. Logic lives in `devplacepy/project_files.py` (mirrors `attachments.py`); routes in `routers/projects/files.py` (registered at prefix `/projects`, after `projects.router`). It is **public read, owner write** - reads use `get_current_user`, mutations use `require_user` + `is_owner(project, user)`. Two project flags layer on top (see **Project visibility and read-only** below): a private project's reads are restricted to owner/admin, and a read-only project refuses every mutation at the data layer.
|
||||
|
||||
- **Model.** One `project_files` table, each row a node keyed by a normalized POSIX `path` unique per project: `uid, project_uid, user_uid, path, name, parent_path, type` (`file`/`dir`), `content` (text in DB), `is_binary`, `stored_name`, `directory`, `mime_type`, `size`, timestamps. Indexes `(project_uid, path)` and `(project_uid, parent_path)` in `init_db`.
|
||||
- **Text vs binary.** Text files store `content` in the DB (editable inline). Binary uploads write bytes to `config.PROJECT_FILES_DIR/<shard>/<stored_name>` (= `data/uploads/project_files/...`, reusing `attachments._directory_for`/`_detect_mime`) and are served via the existing `/static/uploads` mount. The `<shard>` is the canonical two-level `xx/yy` tree taken from the **random tail** of the uuid7 (`{tail[-2:]}/{tail[-4:-2]}`), never its time-ordered head - see the root CLAUDE.md "Blob sharding" note for why the tail is mandatory. `store_upload` decodes UTF-8 texty files into editable DB content; everything else is binary.
|
||||
- **Path is the security control.** `normalize_path` rejects `..`, null bytes, control chars, and empty/over-long paths; a leading `/` is normalized to relative. The logical `path` is never used as a real FS path (blobs are uid-hashed), so traversal is structurally impossible. `write`/`upload`/`mkdir` create parent dirs recursively (`ensure_dirs`). Caps: `MAX_TEXT_CHARS` 400000, `MAX_FILES_PER_PROJECT` 5000.
|
||||
- **Routes (all POST mutations, JSON via `respond`/`action_result`; file path travels in query/body `path`, not a path param):** `GET .../files` (page or `ProjectFilesOut`), `GET .../files/raw?path=`, `POST .../files/{write,upload,mkdir,move,delete}`. `move` rewrites a dir's descendants by path prefix; `delete` is recursive and unlinks blobs.
|
||||
- **Line-range editing (large text files).** `GET .../files/lines?path=&start=&end=` returns `{path, start, end, total_lines, lines, content}`; `POST .../files/{replace-lines,insert-lines,delete-lines,append}` mutate a text file surgically without resending the whole thing. Helpers in `project_files.py` (`read_lines`, `replace_lines`, `insert_lines`, `delete_lines`, `append_lines`) work on a `(lines, trailing_newline)` model via `_split_lines`/`_join_lines`, are 1-indexed inclusive, enforce `MAX_TEXT_CHARS`, and reject binary/dir/missing paths. These are the preferred way to edit existing files; `write` replaces the whole file. They never create parents (the file must exist).
|
||||
- **Cascade.** `content.py` `delete_content_item` calls `delete_all_project_files(uid)` when `target_type == "project"`.
|
||||
- **Frontend.** `templates/project_files.html` (IDE layout: tree + CodeMirror editor / media preview, same CodeMirror vendor as gists), `static/js/ProjectFiles.js` (tree from the flat list, open/save/new/upload/rename/delete over JSON), `static/css/project_files.css`. The CodeMirror mode map is shared via `static/js/codemirrorModes.js` (used by `GistEditor.js` too).
|
||||
- **Devii + API.** `http` catalog actions cover the full filesystem: `project_list_files`/`read_file`/`read_lines`/`write_file`/`replace_lines`/`insert_lines`/`delete_lines`/`append_file`/`upload_file`/`make_dir`/`move_file`/`delete_file`, plus the `project-files` `docs_api.py` group. Because `PlatformClient` sends `Accept: application/json`, the same routes serve the agent and the API.
|
||||
- **Overwrite-protection guard (agent only).** `Dispatcher._run_http` blocks `project_write_file` for a `(slug, path)` only when the file **already exists** and the agent has not read it this session - it must call `project_read_file` first. Existence is probed live via `_file_exists` (`GET /projects/{slug}/files/raw`, 200 = exists, 404 = new), so creating a brand-new file is never blocked. Reads of `project_read_file`/`project_read_lines` (and a successful write) record the path in the per-session `Dispatcher._read_files` set; the key is `normalize_path`d so read and write paths match. This enforces "look before you overwrite" and steers the agent to the surgical line tools for existing files; it does not affect the human UI or the public HTTP API (the guard lives in the Devii dispatcher, not the route). The system prompt's `WRITING AND EDITING FILES` rule mirrors this.
|
||||
|
||||
## Project visibility and read-only
|
||||
|
||||
Two owner-controlled flags on the `projects` row, both integer `0`/`1` (default `0`, set on the create insert so the columns always exist): `is_private` hides the project from other viewers, `read_only` freezes its filesystem against every mutation. **The full security predicates** (`content.can_view_project`, `owns_instance`, `can_view_project_containers`, `can_view_instance`, `can_manage_instance`, the primary-administrator container isolation rules) are documented once in the root CLAUDE.md's "Project visibility (`is_private`) and read-only" section - this file covers only the project-page/filesystem UI-level behavior that sits on top of those predicates.
|
||||
|
||||
**Toggle routes** (owner-only, `routers/projects/index.py`): `POST /projects/{slug}/private` and `POST /projects/{slug}/readonly`, each taking `ProjectFlagForm{value: bool}` and routed through `_set_project_flag`. The create form carries an `is_private` checkbox (`ProjectForm.is_private`); `project_detail.html` shows Private/Read-only badges and owner toggle buttons (**both** the private and read-only buttons carry `data-confirm` so `ModalManager.initConfirmations` gates them through `app.dialog`), and `project_files.html` hides the editing toolbar + shows a banner when read-only. Schemas: `is_private`/`read_only` on `ProjectOut` and `ProjectDetailOut`, `read_only`/`is_private` on `ProjectFilesOut`.
|
||||
|
||||
**Devii.** Actions `project_set_private` and `project_set_readonly` (catalog). **Both** are gated by an explicit-confirmation requirement: `dispatcher.confirmation_error(name, arguments)` (driven by the `CONFIRM_REQUIRED` set, which lists both action names) is checked in `Dispatcher.dispatch` BEFORE any HTTP call and returns a `ToolInputError` telling the agent to obtain user confirmation unless `confirm` is truthy (the message for `project_set_private` adapts to the `value` argument: making private vs. making public). Each action declares `confirm` as a required body param, and the `PROJECT PRIVACY AND READ-ONLY` system-prompt rule tells the agent to ask first and only then pass `confirm=true`. This mirrors the overwrite-protection guard pattern above: a Devii-only gate that the human UI and HTTP API do not see.
|
||||
|
||||
### Deletion confirmation gate
|
||||
|
||||
`confirmation_error` also blocks every Devii deletion until `confirm` is truthy. Unconditional entries in `CONFIRM_REQUIRED`: every content delete - `delete_post`, `delete_comment`, `delete_gist`, `delete_project`, `project_delete_file`, `delete_media`, `delete_attachment`, `admin_delete_news` - plus `project_set_readonly` and the customization mutations; a generic fallback message covers any name in the set that lacks a bespoke message. **Load-bearing invariant: every gated tool MUST declare a `confirm` body param** (the `confirm()` helper in `catalog.py`, or `arg("confirm", ..., kind="boolean")` for container actions). Tool schemas are `additionalProperties: false`, so the model can only send arguments that are declared properties - a gated tool with NO `confirm` param can never be confirmed, so `confirmation_error` refuses every call and the agent loops forever asking for a confirmation it has no way to express. This was a real production issue: an admin asked Devii to delete a user's 27 posts, confirmed repeatedly, and all 27 `delete_post` calls were rejected because `delete_post` (and `delete_project`/`delete_media`/`project_delete_file`/`container_*`) never exposed `confirm`. The param is harmlessly forwarded to the HTTP endpoint (routes do not declare it, FastAPI drops undeclared form fields) and ignored by the LOCAL container/customization controllers. Conditional entries (gated by inspecting arguments, not membership): `container_instance_action` only when `action="delete"` (start/stop/restart/pause/resume/sync are unaffected), and `container_exec` only when its `command` matches `dispatcher.DESTRUCTIVE_COMMAND` - a token-aware regex for `rm`/`rmdir`/`unlink`/`shred`/`truncate`/`dd`/`mkfs*`/`wipefs` (with or without a leading `sudo`), `find ... -delete`, redirection over a file (`> /...`), and SQL `drop table|database` / `delete from`. Benign commands (`pip install`, `ls`, `grep -rm`) are not matched. The error message echoes the exact path/command so the agent shows the user what will be removed; after a clear yes it retries with `confirm=true`. The `DELETING IS ALWAYS CONFIRMED` system-prompt section in `agent.py` mirrors the rule. This exists because an unconfirmed `container_exec rm -f` once deleted a live project database mid-"test the site"; the gate is the data-loss backstop. The regex is a heuristic (it cannot catch `python -c "os.remove(...)"` and similar), so it is defense-in-depth, not a sandbox - the system-prompt rule is the primary control.
|
||||
|
||||
## Async fork
|
||||
|
||||
`POST /projects/{slug}/fork` enqueues an async job that copies the whole virtual filesystem into a new project owned by the forking user. See `devplacepy/services/jobs/CLAUDE.md` for `ForkService` internals.
|
||||
190
devplacepy/services/CLAUDE.md
Normal file
190
devplacepy/services/CLAUDE.md
Normal file
@ -0,0 +1,190 @@
|
||||
This file documents the service files that live directly in devplacepy/services/ (background task queue, AI correction/modifier, presence, live view relay, and the BaseService/ServiceManager base machinery). Claude Code loads it automatically whenever a file directly under devplacepy/services/ (or any of its subdirectories) is read or edited - domain-specific services (Devii, containers, gateway, jobs, audit, telegram, email, gitea, news, bot, messaging, xmlrpc, dbapi, pubsub, game) have their own more specific nested CLAUDE.md files.
|
||||
|
||||
## Background task queue (`services/background.py`)
|
||||
|
||||
Generic, lightweight, **fire-and-forget** offload for non-critical side-effects so they leave the request path. The singleton `background` (`from devplacepy.services.background import background`) wraps one in-process `asyncio.Queue` drained by a single consumer task. The whole API is one call: `background.submit(fn, *args, **kwargs)` enqueues a **synchronous** callable and returns immediately (`put_nowait`). The consumer runs each callable inside its own `try/except`, so a failing task is logged and never kills the loop; FIFO order is preserved.
|
||||
|
||||
- **Per-worker, not lock-gated (the reason it is NOT a `JobService`/`BaseService`).** Producers run in every uvicorn worker, and process memory is not shared, so the drain must run wherever requests are handled. `main.py` `startup()` calls `await background.start()` for every worker (inside the `if not DEVPLACE_DISABLE_SERVICES` guard, **outside** the `acquire_service_lock()` branch); `shutdown()` calls `await background.stop()`. A `BaseService` supervisor only runs in the lock owner, which would strand records produced by the other worker. The `JobService` queue is also wrong here: it is DB-backed, so deferring a tiny audit insert would *add* writes instead of removing them.
|
||||
- **Inline fallback (load-bearing).** When the consumer is not running - tests with `DEVPLACE_DISABLE_SERVICES=1`, unit tests, request-less bootstrap, or a full queue - `submit` runs `fn` **inline and synchronously**. This keeps audit/notification/XP writes deterministic and immediately visible to the test suite (which asserts audit rows right after an action) while production defers them. No test changes are needed.
|
||||
- **Best-effort durability (intentional).** In-memory only; `stop()` drains whatever remains synchronously so a **graceful** shutdown loses nothing, but a hard crash drops unflushed items. This matches audit/notifications already being best-effort (the recorder never raises). Do not put response-critical or money-touching work on it.
|
||||
- **Capture plain data, never the `Request`.** Its lifecycle ends with the response - build the row/payload synchronously on the request thread and submit only the resulting dict/scalars.
|
||||
- **Consumers today** (deferred at their canonical choke points, so callers need no change):
|
||||
- **Audit** - `services/audit/record.py` `_write` builds the row + links synchronously, generates the `uid`/`created_at` eagerly so `record()` still returns the real uid, then `background.submit(_persist, row, links)` does the two `store` inserts off-thread.
|
||||
- **XP/rewards** - `utils.award_rewards` defers its body via `background.submit(_apply_rewards, ...)`, so every XP award (posts/comments/projects/gists/follow/votes) leaves the request path.
|
||||
- **Notifications** - `utils.create_notification` (the single notification funnel) defers via `background.submit(_deliver_notification, ...)`, so every in-app insert + push schedule + audit for a notification (vote/follow/comment/mention/message/badge/level) runs on the consumer.
|
||||
- **Mention fan-out** - `utils.create_mention_notifications` defers its whole body (`_deliver_mention_notifications`): the `extract_mentions` regex + the `@`-username lookup query + the per-user loop all run off-thread (it is called on every post/gist/project/comment/message create).
|
||||
- **Issue-comment admin fan-out** - `routers/issues/comment.py` defers the `_notify_admins` loop (admin lookup + per-admin notify) via `background.submit`, after the synchronous Gitea call.
|
||||
- Because the reward and notification funnels self-defer, request handlers just **call `award_rewards`/`create_notification` directly** - do NOT wrap them in `background.submit` (that double-queues). The pattern for any NEW side-effect: do the user-visible write inline, then call the self-deferring funnel (or `background.submit(...)` a one-off).
|
||||
- **Feature/first-use badges** ride the same self-deferring model: call `utils.track_action(user_uid, action[, target])` inline at a feature's success point (do NOT wrap it - it `background.submit`s itself), where `action` is a key in `utils.ACHIEVEMENTS` (`action -> [(threshold, badge), ...]`, threshold 1 = first use; `UNIQUE_ACTIONS` use `record_unique_activity` for "N distinct things" like `docs.read`). Source-derivable milestones (counts of existing tables) instead go in `check_milestone_badges`/`_COUNT_MILESTONES`. Badge metadata + `group` live in `BADGE_CATALOG`; the profile shows a grouped earned/locked **Achievements** showcase via `build_achievements`.
|
||||
- **Deliberately NOT deferred (would break correctness):** cache invalidations (`clear_user_cache`/`clear_unread_cache`/`clear_messages_cache`/`bump_cache_version`) must run before the response so the next read is fresh (and they are microsecond version bumps); the vote/reaction count aggregation feeds the AJAX response body; and synchronous **external** calls whose result the response needs or must surface on failure (the Gitea comment/status calls; file/thumbnail writes whose returned URL must already exist) belong in an async **JobService** (durable + retryable), not this fire-and-forget queue.
|
||||
|
||||
## AI content correction (`services/correction.py`)
|
||||
|
||||
**Opt-in, default off, server-side.** When a user turns it on (profile **AI content correction** block, owner-only, saved at `POST /profile/{username}/ai-correction`), the prose they author is rewritten by the AI gateway. The per-user `ai_correction_sync` flag (0 = background default, 1 = sync) picks the **apply mode**: background rewrites the stored fields a moment after the write (never slows the request); sync makes the HTTP response wait until the correction is applied.
|
||||
|
||||
- **Sync must never block the event loop (the self-deadlock trap).** The correction POSTs to the in-process gateway (`INTERNAL_GATEWAY_URL` = `localhost:{PORT}`), and the hooked content helpers are synchronous on the loop thread, so a blocking inline call self-deadlocks the (single) worker against its own gateway request - on a single worker that loop is the only thing that can serve the gateway request it is waiting on, so a blocking inline call deadlocks the worker until the httpx timeout, then fail-softs (server hangs, no correction). The fix: sync runs `_run_correction` via `loop.run_in_executor` (loop stays free), stashes the future on `request.scope[PENDING_SCOPE_KEY]`, and the `await_pending_corrections` middleware in `main.py` awaits it after the handler. Never reintroduce a blocking inline correction on the loop thread.
|
||||
- **Field registry is the single source of truth.** `CORRECTABLE_FIELDS: dict[str, tuple[str, ...]]` maps each correctable table to its prose columns: `posts` -> `(title, content)`, `projects`/`gists` -> `(title, description)`, `comments`/`messages` -> `(content,)`, `users` -> `(bio,)`. `gists.source_code`, project files, and Gitea issues are intentionally excluded - code and external systems are never corrected.
|
||||
- **Choke entrypoint.** `schedule_correction(user, table, uid, request=None)` is the only thing handlers call (every hook passes `request`). It is a no-op unless: a user dict is present, `table` is in the registry, `user["ai_correction_enabled"]` is truthy, and the user has a non-empty `api_key`. In **sync** mode (`user["ai_correction_sync"]`) it calls `_run_inline_awaited`, which - only when a `request.scope` and a running event loop exist - submits `_run_correction` to `loop.run_in_executor` (off the loop thread) and stashes the future on `request.scope[PENDING_SCOPE_KEY]` for the middleware to await; if there is no request/loop it returns False and falls back to background. In **background** mode (default) it `background.submit`s `_run_correction` (per-worker queue; also runs inline under `DEVPLACE_DISABLE_SERVICES=1`). The same `_run_correction` worker function runs in all cases.
|
||||
- **The hooks (covers UI + REST + Devii + devRant in one place):** `content.create_content_item` (posts/projects/gists), `content.edit_content_item`, `content.create_comment_record`, `content.edit_comment_record`, `services/messaging/persist.persist_message` (DMs - sender is the user), `routers/profile/index.update_profile` (bio), and the two devRant direct-update edit paths (`routers/devrant/rants.edit_rant`, `routers/devrant/comments.edit_comment`). devRant create paths route through the shared content cores, so they are already hooked. Code-only and external paths (project files, Gitea) are deliberately not hooked.
|
||||
- **The gateway call is fail-soft.** `correct_text(api_key, prompt, text)` is synchronous (runs on the background worker thread), POSTs to `INTERNAL_GATEWAY_URL` with `model=INTERNAL_MODEL` via `stealth.stealth_sync_client`, authenticated with the user's own `Bearer` api_key (per-user attribution). It returns the ORIGINAL text on any error, empty output, or suspiciously large output (`len > len(text) * MAX_GROWTH_FACTOR + 200`, rejecting hallucinated expansion). `_run_correction` reads the row back, corrects each registry field that has non-blank text, writes only changed fields via `table.update(updates, ["uid"])` without touching `updated_at` (auto-correction is not a user edit) or the slug (slugs are permanent), and `clear_user_cache(user_uid)` when `table == "users"` so the corrected bio re-caches.
|
||||
- **Per-user usage aggregation (only on success).** `correct_text` returns `(text, usage)`; `usage` is parsed from the gateway's `X-Gateway-*` response headers (`_usage_from_headers`) whenever the upstream call returned 200 (cost was incurred, even if the corrected output was rejected), else `None` on any failure. `_usage_from_headers` captures the token and cost headers PLUS the timing headers `X-Gateway-Upstream-Latency-Ms` and `X-Gateway-Total-Latency-Ms` (as `upstream_latency_ms`/`total_latency_ms`), so each call's timing is metered. `_run_correction` accumulates the per-field `usage` into one `totals` dict and, when `totals["calls"] > 0`, makes ONE call to `database.add_correction_usage(user_uid, totals)` (a single `totals` dict, not positional args) - so a 2-field content item is a single aggregated write, and a failed/empty correction records nothing. `add_correction_usage` (and `add_modifier_usage`) delegate to the shared `database._add_usage(usage_table, user_uid, totals)`: a single atomic `INSERT ... ON CONFLICT(user_uid) DO UPDATE SET col = col + excluded.col` upsert against the `correction_usage` table (per-user running SUMS: `calls`/`prompt_tokens`/`completion_tokens`/`total_tokens`/`cost_usd`/`upstream_latency_ms`/`total_latency_ms`/`updated_at`, unique index `idx_correction_usage_user`, the two latency columns REAL default 0.0, all ensured in `init_db`). It is a derived counter table (NOT in `SOFT_DELETE_TABLES`, like `gateway_usage_ledger`) and is deliberately separate from `users` so accumulating never invalidates the auth/user cache. `database.get_correction_usage(user_uid)` (via `_get_usage`) returns the stored sums PLUS computed averages: `avg_tokens` (total_tokens/calls), `avg_upstream_latency_ms`, `avg_total_latency_ms`, `avg_tokens_per_second` (completion_tokens over total upstream seconds), and `avg_cost_usd`.
|
||||
- **Profile display:** `routers/profile/usage._correction_usage(uid, include_cost)` shapes it via the shared `_usage_view(data, include_cost)` (mirrors `_ai_quota`); `profile/index.py` builds it only for `is_owner or viewer_is_admin` and passes `include_cost=viewer_is_admin`, exposed as the `correction_usage` dict on the context and `ProfileOut`. The view surfaces the sums plus averages - `avg_tokens` (avg tokens/call), `avg_latency_ms` (avg upstream latency), `avg_total_latency_ms`, `avg_tokens_per_second` (avg speed), and `total_time_s` (total upstream seconds) - rendered as extra tiles on the card. **Financial gating:** tokens/call-count and the performance tiles show to the owner and admins; the dollar `cost_usd` and `avg_cost_usd` keys are present ONLY when `viewer_is_admin`, in BOTH the HTML card (`templates/profile.html`, `.correction-usage-*`) and the `respond(..., model=ProfileOut)` JSON (same rule as `_ai_quota`'s `spent_usd` - hiding it in the template alone would leak it to a member fetching their own profile as JSON). The card renders only when `correction_usage.calls` > 0.
|
||||
- **Import-cycle discipline.** `correction.py` imports only `stealth`, `config`, `database.get_table`/`add_correction_usage`, and `services.background.background` at module top; `clear_user_cache` is imported lazily inside `_run_correction`. Never import `content` or `utils` at module top.
|
||||
- **Settings live on `users`:** three columns `ai_correction_enabled` (0/1), `ai_correction_sync` (0/1, default 0 = background), and `ai_correction_prompt` (text, default `config.DEFAULT_CORRECTION_PROMPT`), ensured in `database.backfill_api_keys()` (the user column-ensure block run by `init_db`) and seeded born-live in `utils._create_account`. The edit route is the owner-or-admin leaf `POST /profile/{username}/ai-correction` (`routers/profile/ai_correction.py`, `AiCorrectionForm{enabled, sync, prompt}`, audit key `profile.ai_correction`). The owner-only values are exposed on the profile page context and `ProfileOut` (`ai_correction_enabled`/`ai_correction_sync`/`ai_correction_prompt`, gated by `is_owner`), the UI block lives in `profile.html` (owner-only: enable checkbox, **Apply mode** select, prompt textarea) wired by `static/js/AiCorrection.js` (`app.aiCorrection`), and Devii drives it via the owner-scoped `ai_correction_get`/`ai_correction_set` tools (`services/devii/ai_correction/`, `handler="ai_correction"`, `requires_auth=True`, not confirm-gated - it is a reversible per-user toggle; `ai_correction_set` accepts `enabled`, optional `sync`, optional `prompt`).
|
||||
|
||||
## AI modifier (`services/ai_modifier.py`, `services/ai_context.py`)
|
||||
|
||||
**A sibling of AI content correction that runs only on an explicit inline directive.** The engine reuses the correction plumbing wholesale (`CORRECTABLE_FIELDS`, `PENDING_SCOPE_KEY`, `gateway_complete`, the sync/background apply-mode mechanics, the per-user usage upsert) and differs only in the trigger and the apply-mode/enabled defaults: it is **enabled by default** and **synchronous by default**, and it runs ONLY where an authored prose field contains an inline `@ai <instruction>` directive.
|
||||
|
||||
- **The `@ai` gate is the whole difference.** `has_ai_directive(text)` matches the regex `@ai\s+\S` (case-insensitive). `schedule_modification(user, table, uid, request=None)` is a no-op unless a user is present, `table` is in `CORRECTABLE_FIELDS`, `user["ai_modifier_enabled"]` is truthy, the user has an `api_key`, AND at least one of the table's registry fields actually contains an `@ai` directive. `_run_modification` re-checks the gate per field, so untriggered fields are never sent to the gateway and never metered. Triggerless writes cost nothing. The configured prompt (default `config.DEFAULT_MODIFIER_PROMPT` = "Execute what is behind `@ai` (the prompt) and replace that part including `@ai`") tells the model to execute the instruction and replace the marked part including the `@ai` marker.
|
||||
- **Total reuse of the correction layer.** `CORRECTABLE_FIELDS`, `PENDING_SCOPE_KEY`, `gateway_complete`, the sync/background apply-mode mechanics (`_run_inline_awaited` -> `loop.run_in_executor` + `request.scope[PENDING_SCOPE_KEY]` awaited by the `await_pending_corrections` middleware in `main.py`, the same self-deadlock-avoiding path), and the per-user usage upsert pattern are all imported from / mirror `services/correction.py`. `modify_text(api_key, prompt, text, context="")` composes a modifier system message and calls the shared `gateway_complete`. The same hooks fire it: `profile/index.update_profile` calls both `schedule_correction` and `schedule_modification`, and the content/comment/messaging cores invoke it alongside correction, so it covers the web UI, REST, devRant, and Devii in one place. Code and source files are never modified (same `CORRECTABLE_FIELDS` registry, `gists.source_code`/project files/Gitea excluded). `schedule_modification` is hooked alongside `schedule_correction` at the same content/comment/messaging/profile entrypoints.
|
||||
- **Context-aware (modifier only, not correction).** Unlike correction, the modifier gives the model a grounding **context block** so an `@ai` instruction can reason about who is asking and what it is attached to. `services/ai_context.py` `build_context(table, uid, row, user_uid) -> str` assembles it; `_run_modification` builds it **lazily once per row** (only after a field is confirmed to contain `@ai`, so triggerless writes do no extra queries) and passes it to every field's `modify_text`, which appends it to the system message under a `# Context (use it to inform the result; never echo this block)` header. The block (fail-soft, length-capped, each part wrapped in try/except so a failed lookup never blocks the modification) has three parts:
|
||||
1. **date** - `Today is DD/MM/YYYY on the DevPlace developer network.`
|
||||
2. **author/stats** - the author's username, role, level, stars, post count (`get_user_post_count`), leaderboard rank (`get_user_rank`), follower count (`get_follow_counts`), member-since date, and bio (capped `MAX_BIO`).
|
||||
3. **location/thread** - table-specific: a comment gets the target post/project/gist/news title + excerpt and the parent comment it replies to; a post gets its topic and attached project; a project/gist gets its sibling title/description and, for gists, the language + `source_code` (truncated, marked "reference only" - this is read context, the modifier still never WRITES `source_code`); a direct message gets the recipient and the last `MAX_THREAD_MESSAGES` messages of the conversation (each truncated).
|
||||
All excerpts are length-capped (`MAX_*` constants) to bound token cost; the added context tokens are billed and metered like any other input via the usage stats. So `@ai answer the question above` in a comment, `@ai write my bio from my stats`, or `@ai reply to this` in a DM all work. The message-thread query binds a param named `:current` (NOT `:self`, which collides with `db.query`'s bound-method argument).
|
||||
- **Defaults differ:** enabled **ON** by default and apply mode defaults to **synchronous** (`ai_modifier_sync` column defaults to 1), the inverse of correction.
|
||||
- **Per-user usage (only on success).** Same plumbing as correction: `_usage_from_headers` captures the token, cost, and timing headers (`X-Gateway-Upstream-Latency-Ms`/`X-Gateway-Total-Latency-Ms`); `_run_modification` accumulates per-field gateway usage into a `totals` dict and, when `totals["calls"] > 0`, makes ONE `database.add_modifier_usage(user_uid, totals)` call (a single `totals` dict) that delegates to the shared `database._add_usage` atomic `INSERT ... ON CONFLICT DO UPDATE SET col = col + excluded.col` upsert into the `modifier_usage` table (running SUMS `calls`/token/`cost_usd`/`upstream_latency_ms`/`total_latency_ms` totals, the two latency columns REAL default 0.0, separate from `users` so it never busts the auth cache, NOT in `SOFT_DELETE_TABLES`, ensured in `init_db`). `database.get_modifier_usage(user_uid)` (via `_get_usage`) returns the sums plus the same computed averages as correction (`avg_tokens`, `avg_upstream_latency_ms`, `avg_total_latency_ms`, `avg_tokens_per_second`, `avg_cost_usd`).
|
||||
- **Settings live on `users`:** three columns `ai_modifier_enabled` (0/1, default 1), `ai_modifier_sync` (0/1, default 1 = sync), and `ai_modifier_prompt` (text, default `config.DEFAULT_MODIFIER_PROMPT`), ensured in `database.backfill_api_keys()` (the user column-ensure block run by `init_db`) and seeded born-live in `utils._create_account`. **Because the feature is on by default, `backfill_api_keys()` ALSO runs a `with db:` `UPDATE users SET ... WHERE ... IS NULL` so EXISTING accounts inherit the on/sync defaults** (a freshly `create_column_by_example`-added column is NULL on existing rows, which would read as disabled - the backfill is what makes "on by default" true for everyone, not just new accounts). The `with db:` wrapper is load-bearing: a raw `db.query` write that does not commit holds the SQLite write lock. The edit route is the owner-or-admin leaf `POST /profile/{username}/ai-modifier` (`routers/profile/ai_modifier.py`, `AiModifierForm{enabled, sync, prompt}`, audit key `profile.ai_modifier`). The owner-only values are exposed on the profile page context and `ProfileOut` (`ai_modifier_enabled`/`ai_modifier_sync`/`ai_modifier_prompt`, gated by `is_owner`); `modifier_usage` is built by `routers/profile/usage._modifier_usage(uid, include_cost=viewer_is_admin)` (via the shared `_usage_view`) for `is_owner or viewer_is_admin`, surfacing the sums plus the performance averages (avg tokens/call, avg latency, avg tokens/sec, total time) like `correction_usage`; the dollar `cost_usd` and `avg_cost_usd` are present only when `viewer_is_admin`, in both HTML and JSON. The UI block lives in `profile.html` (owner-only: enable checkbox, **Apply mode** select defaulting to sync, prompt textarea, plus the usage card reusing the `.correction-usage-*` classes with the extra performance tiles) wired by `static/js/AiModifier.js` (`app.aiModifier`). Devii drives it via the owner-scoped `ai_modifier_get`/`ai_modifier_set` tools (`services/devii/ai_modifier/`, `handler="ai_modifier"`, `requires_auth=True`, not confirm-gated; `ai_modifier_set` accepts `enabled`, optional `sync`, optional `prompt`).
|
||||
|
||||
## Background services base machinery (`services/base.py`, `services/manager.py`)
|
||||
|
||||
`devplacepy/services/` provides a generic framework for running background async services alongside the FastAPI server. `BaseService` provides the async run loop, a `deque(maxlen=20)` log buffer, and graceful cancellation; `ServiceManager` is a singleton that registers, starts, and stops services. Services are fully managed from the **Services admin tab** (`/admin/services`): start/stop, enable-on-boot, run-now, edit parameters, clear logs, adjustable log buffer size, with live status. All of this is generic - a new service gets it for free by declaring its config and implementing `run_once`.
|
||||
|
||||
This section covers only the shared machinery. The individual services built on top of it live in their own subdirectories with their own nested CLAUDE.md: `NewsService` (`services/news/`, see `devplacepy/services/news/CLAUDE.md`), `GatewayService` and provider/model routing (`services/openai_gateway/`, see `devplacepy/services/openai_gateway/CLAUDE.md`), `DeviiService` (`services/devii/`, see `devplacepy/services/devii/CLAUDE.md`), and the bot fleet service (`services/bot/`, see `devplacepy/services/bot/CLAUDE.md`).
|
||||
|
||||
### DB-backed state (correct across workers)
|
||||
|
||||
In production (`make prod`, 2 workers) only the lock-holding worker runs services, but an admin request can hit either worker. State therefore lives in the DB, not process memory:
|
||||
- **Desired/config state** in `site_settings` (via `get_setting`/`set_setting`): `service_<name>_enabled` (`"0"`/`"1"` - also the boot flag), `service_<name>_command` (`"<verb>:<counter>"`, verbs `run`/`clear`), `service_<name>_log_size`, plus each declared config field's own key.
|
||||
- **Observed state** in the `service_state` table (one row per service): `status`, `last_run`, `next_run`, `started_at`, `heartbeat`, `logs` (JSON), `updated_at`. Written by the supervising worker, read by any worker.
|
||||
|
||||
`main.py` registers every service in **all** workers (so `describe_all()` works anywhere) but only the lock worker calls `service_manager.supervise()`.
|
||||
|
||||
### `ConfigField` (`services/base.py`)
|
||||
|
||||
Declarative parameter spec. `type` in `int`/`str`/`url`/`text`/`password`/`bool`/`select`. `coerce(raw)` validates and types (raises `ValueError`, never silent); `read()` is the lenient runtime reader (falls back to `default`); `spec()` renders the field for the template/JSON (secrets masked, never sent). Optional `minimum`/`maximum`/`options`/`help`/`secret`. `group="..."` lets the detail page render fields as sections (built-in fields are grouped "General"/"Advanced").
|
||||
|
||||
### `BaseService` (`services/base.py`)
|
||||
|
||||
Abstract class for all services:
|
||||
- **`name`**, **`interval_key`** (default `service_<name>_interval`; a subclass may point it at an existing key), **`min_interval`**.
|
||||
- **`config_fields`** - class-level list of `ConfigField`. The framework prepends built-in `enabled` and interval fields and appends `log_size`; `all_fields()` returns the full set.
|
||||
- **`get_config()`** returns a typed dict from settings; **`is_enabled()`**, **`current_interval()`** (floored to `min_interval`).
|
||||
- **`log(message)`** - writes to `log_buffer` and standard `logging`.
|
||||
- **`run_once()`** - abstract; override with actual work. Read parameters via `self.get_config()`.
|
||||
- **Reconciling loop** (`_run_loop`/`_tick`, ~1s tick): honors `enabled`, runs `run_once()` when due or on a `run` command, handles `clear`, rebuilds the log deque if `log_size` changed, and persists observed state (throttled, force on transitions). Start/stop/run/clear from any worker are just DB writes the loop reacts to within ~1s - services are never hard-restarted.
|
||||
- **Lifecycle hooks** `async on_enable()` / `async on_disable()` - called on transition to enabled / disabled (and `on_disable` on shutdown). Override for long-running resources that outlive one `run_once` (e.g. the bot fleet); `run_once` then becomes a periodic reconcile/metrics tick.
|
||||
- **`collect_metrics() -> dict`** - generic live metrics. Return `{"stats": [{"label", "value"}], "table": {"columns": [...], "rows": [[...]]}}`. Persisted to `service_state.metrics` each tick and rendered live in the Services tab (stat cards + table) by `ServiceMonitor.js`. Default returns `{}`.
|
||||
- **`default_enabled`** class flag (default `True`) - set `False` for opt-in services so they never auto-start on boot (the bots service uses this).
|
||||
- **`title` / `description`** class attrs - shown in the admin UI; every service should set a `description` of what it does.
|
||||
|
||||
**Admin UI** (`routers/admin/services.py`): `/admin/services` is a slim **index** (`services.html`, an `admin-table` of name/description/status/uptime/last-run/next-run/interval) where each row links to `/admin/services/{name}`. The **detail** page (`service_detail.html`) holds everything for one service - header with controls (start/stop/run/clear), and `[data-tabs]` tabs **Overview** (meta + metrics), **Configuration** (the config form rendered as a `<fieldset>` per `field_groups` entry), **Logs**. `ServiceMonitor.js` is dual-mode (polls `/admin/services/data` for the index, `/admin/services/{name}/data` for the detail, via the shared `Poller`; its config-save POST goes through `Http.sendForm`) and shares one `update(root, svc)` over `data-*` hooks; tabs use the generic reusable `static/js/Tabs.js` (`[data-tabs]`/`[data-tab]`/`[data-tab-pane]`, wired in `Application.js`). Adding a service still needs zero UI code.
|
||||
|
||||
- **`describe()`** - builds the dict the UI consumes: `title`, `description`, derived display status (`stopped` when disabled, `running` with fresh heartbeat, `stalled` when enabled but heartbeat older than `STALE_SECONDS`), meta, logs, metrics, flat `fields`, and `field_groups` (ordered `{name, fields}` sections) read from `service_state` + settings.
|
||||
|
||||
### `ServiceManager` (`services/manager.py`)
|
||||
|
||||
Singleton that manages all registered services:
|
||||
- `register(service)` - add a service
|
||||
- `describe_all()` -> `list[dict]` (per-service `describe()`)
|
||||
- `set_enabled(name, bool)` - Start/Stop (writes the enabled flag)
|
||||
- `send_command(name, "run"|"clear")` - one-shot command channel
|
||||
- `save_config(name, form)` - strict validation via each `ConfigField`; blank secret = keep existing; returns `{ok, errors}`
|
||||
- `supervise()` - start the reconciling task per service (lock worker only)
|
||||
- `shutdown_all()` - cancel tasks on server shutdown
|
||||
|
||||
### Adding a new service
|
||||
|
||||
1. Create `devplacepy/services/your_service.py` with a class extending `BaseService`. Declare `config_fields` (and set `interval_key`/`min_interval` if reusing an existing key); read them in `run_once` via `self.get_config()`.
|
||||
2. Override `async def run_once(self) -> None`.
|
||||
3. Register in `main.py` startup event (outside the lock block, so all workers know the roster):
|
||||
```python
|
||||
from devplacepy.services.your_service import YourService
|
||||
service_manager.register(YourService())
|
||||
```
|
||||
4. The service appears automatically on `/admin/services` with start/stop, enable-on-boot, run-now, a generic config form, clear-logs, live status, and the log tail - no template, JS, or router changes needed.
|
||||
|
||||
### CLI
|
||||
|
||||
```bash
|
||||
devplace news clear # Delete all news from local database
|
||||
devplace devii reset-quota <username> # Reset one user's rolling 24h AI quota
|
||||
devplace devii reset-quota --guests # Reset every guest quota
|
||||
devplace devii reset-quota --all # Reset every quota (users and guests)
|
||||
```
|
||||
|
||||
## Multi-worker concurrency (preferred rules)
|
||||
|
||||
`uvicorn --workers N` = N independent processes sharing only the filesystem and SQLite DB. Module-global caches/counters are per-process, so a local `clear()` is invisible to siblings. Full reference: admin docs `Production -> Multi-worker and concurrency` (`templates/docs/production-concurrency.html`). Enforce these:
|
||||
|
||||
- **In-process caches are never authoritative across workers.** Coordinate invalidation through the `cache_state` version table in `database.py`: call `bump_cache_version(name)` at every write path and `sync_local_cache(name, cache)` before every read path. Existing names: `auth` (guards `_user_cache`; bumped by `clear_user_cache`/`clear_session_cache` on logout, ban, role/password change), `settings` (guards `_settings_cache`; bumped by `set_setting`/`clear_settings_cache`), `relations` (`_relations_cache`), `customizations` (`_customizations_cache`), `notif_prefs` (`_notification_prefs_cache`), `gateway_routing` (`_ROUTING_CACHE`), and **`admins`** (guards `_admins_cache`, which memoizes `get_admin_uids()` / `get_primary_admin_uid()`). The `_cache_version_cache` (`ttl=1`) caches the version reads themselves, so the bump is seen by other workers within ~1s on their next read (the originating worker is immediate); the version read/bump fail open (logged, never raise). When you add a per-process cache whose staleness matters, give it a name and wire both calls - do not invent a second invalidation mechanism.
|
||||
- **Admin-set cache invariant (load-bearing).** `_admins_cache` makes `get_admin_uids()` / `get_primary_admin_uid()` (hit on the projects-listing visibility filter, `_owner_is_admin`, and every primary-admin gate: `/dbapi`, backup-archive download) a dict lookup instead of a per-call `SELECT`. It is NOT the authorization gate - `is_admin(user)`/`require_admin` read the role off the user object (independently invalidated via `clear_user_cache`), so a stale admin set cannot grant access. But **any code that writes `users.role` MUST call `database.invalidate_admins_cache()`** (clears local + bumps the `admins` version), exactly like the soft-delete and `with db:` rules. Current role-write sites all do: `routers/admin/users.py` (role change), `cli.py` (`role set`), and `utils._create_account` (first user -> Admin). Omitting the call leaves the admin set stale for up to the 300s TTL.
|
||||
- **Deterministic / eventual caches skip version-sync on purpose.** Two caches need no `cache_state` name because correctness does not depend on cross-worker freshness: (1) `docs_prose._render_markdown` is an `@lru_cache` on the **static** prose source string (pure function of template content, so identical on every worker; changes only on deploy), and (2) `main._home_cache` (`TTLCache ttl=60`) memoizes the guest `/` blocks - the featured-news block (identical for everyone) and the latest-posts block **only for the no-block case**; a viewer with a non-empty block set bypasses the cache and is computed fresh, so the original block-filter semantics are preserved byte-for-byte and there is zero cross-user leakage. Worst case is a soft-deleted/edited public post lingering on `/` for <=60s. Use a plain `TTLCache`/`lru_cache` (no version name) ONLY when the value is deterministic or its staleness is purely cosmetic; anything whose staleness affects correctness or permissions MUST use the version-sync pattern above.
|
||||
- **Authoritative state lives in the DB**, memory is only a short-TTL accelerator (sessions, the Devii 24h cap via `devii_usage_ledger`, cost counters).
|
||||
- **Global rates/quotas are worker-count-aware or DB-backed.** The rate limiter enforces `ceil(limit / DEVPLACE_WEB_WORKERS)` per process so the aggregate matches the configured value with no per-request write. `DEVPLACE_WEB_WORKERS` MUST equal the real `--workers` (set in `make prod`=2 and the Dockerfile=2; default 1 for dev). Update it whenever you change worker count.
|
||||
- **First-boot side effects are flock-serialized and idempotent.** `init_db()` runs under a blocking `flock` on `devplace-init.lock` (the seed-if-missing inserts would otherwise race to duplicate rows); `ensure_certificates()` early-returns when keys exist and otherwise generates under `.vapid.lock`. Any new run-once-at-startup work follows the same pattern (flock + exists-check / `INSERT OR IGNORE`).
|
||||
- **Run-once background work stays behind the services lock**, never per-worker: gate it on `service_manager.owns_lock()` (held via `devplace-services.lock`), as news/bots/Devii hub do.
|
||||
|
||||
## Performance caches (request-path)
|
||||
|
||||
Read-mostly aggregates that were recomputed per request or per vote sit behind short per-worker `TTLCache`s (`devplacepy/cache.py`). These are DISPLAY caches: staleness up to the TTL is accepted by design, so none of them uses the cross-worker `cache_state` versioning (that is reserved for correctness-critical caches like auth/settings/admins). The registry:
|
||||
|
||||
| Cache | Where | Key | TTL | Invalidation |
|
||||
|---|---|---|---|---|
|
||||
| `_authors_cache` | `database/ranking.py` | `ranked`/`rank_map` | 15s | TTL only - `update_target_stars` deliberately does NOT clear it per vote anymore (the old per-vote `clear()` forced a full UNION-JOIN ranking recompute on the next feed/leaderboard/landing request and never propagated cross-worker anyway) |
|
||||
| `_stars_cache` | `database/ranking.py` | user uid | 15s | TTL + `clear_user_stars(owner_uid)` from `content.apply_vote`, so a user's own total updates immediately on the voting worker |
|
||||
| `_leaderboard_cache` | `services/game/store/farm.py` | `top:{limit}` | 15s | TTL only (the farm scan + Python `farm_score` sort runs at most once per 15s per worker) |
|
||||
| `_projects_cache` | `templating.py` | user uid | 10s | TTL + `clear_user_projects_cache(uid)` from the `content.py` project create/delete choke points (in-function import - `templating` imports `content` at module level). `jinja_user_projects` also filters `deleted_at=None` now (the composer dropdown previously listed soft-deleted projects) |
|
||||
|
||||
Rules: a new hot read-path aggregate follows this exact pattern (module-level `TTLCache`, 10-15s, `clear_*` helper only when a same-worker write must be visible immediately); never reintroduce a whole-cache `clear()` on a per-event write path; profile `projects`/`gists` loads are tab-gated in `routers/profile/index.py` (`tab == X or wants_json(request)` - the JSON `ProfileOut` keeps serializing both, HTML tabs that do not render them get `[]`).
|
||||
|
||||
## Live view relay (`services/live_view_relay.py`)
|
||||
|
||||
`LiveViewRelayService` is a second lock-owner pub/sub bridge (default-enabled, 1s interval, registered in `main.py` after `NotificationRelayService`) that replaces the per-client HTTP polling on the admin live views with server push, **computing a snapshot only for topics that currently have subscribers**. Each tick it reads `pubsub.topics()` (the hub's live subscription set, which converges on the lock owner), matches each concrete subscribed topic against the `VIEWS` registry (a list of `(compiled_regex, compute_callable, min_interval_seconds)`), throttles per topic via a `time.monotonic()` map, computes the payload, and publishes it on the same topic. The compute callables (async, lazy-importing to avoid cycles, defensive try/except -> skip) reproduce the **exact payload shape the matching HTTP endpoint already returns**, so the frontend render code is unchanged - only the data-arrival path is added. Registry:
|
||||
|
||||
| Topic | Cadence | Payload (same as endpoint) |
|
||||
|-------|---------|----------------------------|
|
||||
| `container.list` | 4s | `{instances}` (admin all-instances, `_decorate`) |
|
||||
| `project.{slug}.containers` | 3s | `{instances}` (per-project, slug resolved via `resolve_by_slug`) |
|
||||
| `container.{uid}.detail` | 4s | `{instance, events, schedules, stats, runtime}` |
|
||||
| `container.{uid}.logs` | 3s | `{logs}` (backend log tail 400) |
|
||||
| `fleet.bots` | 2s | bot frames payload (`routers/admin/bots._frames_payload`) |
|
||||
| `admin.services` | 5s | `{services}` (`service_manager.describe_all()`) |
|
||||
| `admin.services.{name}` | 5s | `{service}` |
|
||||
| `admin.ai-usage.{hours}` | 15s | `build_analytics(hours)` (hours parsed from the topic) |
|
||||
| `admin.backups` | 8s | `routers/admin/backups._dashboard(can_download=False)` (storage, backups, schedules, metrics) |
|
||||
|
||||
All these topics are admin-only by pub/sub policy (non-`public`, non-`user.{uid}` -> `privileged` required), matching the admin-only pages. Frontend monitors (`ContainerInstance`, `ContainerList`, `ContainerManager`, `BotMonitor`, `ServiceMonitor`, `AiUsageMonitor`, `BackupMonitor`) each `window.app.pubsub.subscribe(topic, render)` in their init and keep a **lengthened HTTP poll (15-30s) as initial-load + fallback** - the relay drives liveness at the cadence above. `AiUsageMonitor` re-subscribes (unsubscribe old, subscribe new) when the window-hours selector changes, since hours is in the topic.
|
||||
|
||||
**Container topics never broadcast private-project instances**: `container.list` publishes only public-project rows with `partial: true` (`ContainerList.merge` updates by uid, never removes, so private rows from the authoritative HTTP poll survive), and `project.{slug}.containers` / `container.{uid}.detail` / `container.{uid}.logs` skip private-project targets entirely (owners fall back to their HTTP polls).
|
||||
|
||||
**`admin.backups` is the one view with a per-viewer field:** the backup `download_url` is restricted to the primary administrator at every endpoint, so the relay publishes the snapshot with `can_download=False` (the bus broadcasts one payload to every admin subscriber, and is therefore treated as another endpoint that must withhold it). `BackupMonitor` derives `canDownload` from its authoritative per-viewer HTTP poll only (never from a pushed frame) and builds the `/admin/backups/{uid}/download` URL client-side; the download route itself stays primary-admin-gated.
|
||||
|
||||
To add a new admin live view: add one `(regex, compute, interval)` row to `VIEWS` whose compute returns the endpoint payload, and have the frontend subscribe to the topic while keeping a fallback poll. Reuse this subscriber-gated snapshot pattern for any future "several admins watch a periodically-recomputed server snapshot" surface; keep durable/ordered/stateful/raw-I/O channels (messages, Devii, SEO/DeepSearch progress, container PTY) OFF pub/sub. See `pubreport.md` for the full migration analysis.
|
||||
|
||||
## Notification relay (`services/notification_relay.py`)
|
||||
|
||||
**Live toasts ride the `in_app` channel** (no new channel). `NotificationRelayService` is a lock-owner `BaseService` (registered in `main.py`, default-enabled, 1s interval) modeled on `MessageRelay`: it primes a watermark to `MAX(notifications.id)` on first tick, then each tick selects `notifications WHERE id > watermark` and publishes `{uid, type, message, target_url}` to the per-recipient pub/sub topic `user.{user_uid}.notifications`. It runs **only on the service lock owner**, which is exactly where every pub/sub WS subscriber converges (non-owner `/pubsub/ws` closes `4013`), so the in-process `services.pubsub.publish` reaches the recipient's browser. Because the `notifications` row exists only when the `in_app` channel is enabled, a live toast fires **exactly** for the notifications a user has enabled - the relay needs no preference lookup of its own. The watermark always advances (boot-priming + every-tick) so a newly-connected subscriber never gets a backlog flood.
|
||||
|
||||
**Frontend:** `base.html` exposes the viewer uid as `<body data-user-uid>`, `static/js/LiveNotifications.js` (`app.liveNotifications`, constructed with `this.pubsub`/`this.toast`) subscribes to that topic and calls `app.toast.show(message, {type:"info", ms, url})`. **The toast click reproduces a bell-item click exactly:** it targets `GET /notifications/open/{uid}` (which marks the row read and redirects to its `target_url`), not the bare `target_url` - the relay publishes `uid` for this. `AppToast` gained a generic click action via `_action(options)`: `options.onClick` (a function) or `options.url` (navigate); either adds the `.dp-toast-link` pointer cursor, so any caller can attach a click action. DMs toast too (type `message` is not excluded). Reuse this relay-on-the-lock-owner pattern for any future "live mirror of a DB-persisted, per-user event."
|
||||
|
||||
**Unread-count badges ride the same relay.** The same `NotificationRelayService` tick, after publishing the toast rows, also publishes the recipient's fresh unread counts `{notifications, messages}` to `user.{uid}.counts` (computed with a direct DB query on the lock owner, never the per-worker `_unread_cache`, which could be stale there). Because a new DM also inserts a `message`-type notification row through `persist_message -> create_notification`, this one publish point covers both new notifications and new messages, so the header badge bumps instantly. `static/js/CounterManager.js` (`app.counters`, constructed with `this.pubsub`) subscribes to `user.{uid}.counts` and applies the pushed counts; its HTTP poll of `/notifications/counts` stays as a 60s reconciliation fallback (covers decrements on read and any missed frame). Read events still clear the per-worker cache as before.
|
||||
|
||||
## Online presence (`services/presence.py`, `services/presence_relay.py`)
|
||||
|
||||
Online status is a single **`users.last_seen`** UTC-ISO column (ensured in `database.backfill_api_keys`), never a new table and never a per-request insert. The design is dictated by the multi-worker architecture: a profile of user X renders on **any** worker, so "is X online" needs cross-worker state, and pub/sub is in-process only (a non-owner worker cannot publish to the lock owner), leaving SQLite as the sole shared medium. So presence is a heavily-throttled in-place UPDATE.
|
||||
|
||||
**Write path (all workers):** `main.py`'s `track_presence` HTTP middleware resolves the cached current user on every non-`/static`, non-`/avatar` request and calls `presence.touch(uid)`. `touch` keeps a per-worker in-memory `_last_write: dict[uid -> monotonic]` and writes `users.last_seen` (via `database.set_last_seen`) only when the last write for that uid is older than `config.PRESENCE_WRITE_SECONDS` (= `PRESENCE_TIMEOUT_SECONDS // 2`). So continuous browsing is a dict lookup; a write happens at most ~once per half-window per active user per worker, and the row is updated in place (zero growth). It deliberately does **not** call `clear_user_cache` (that would defeat the 300s auth cache; the stale cached self-row is irrelevant since presence of *other* users is always read from a fresh row).
|
||||
|
||||
**Read path (any worker):** `presence.is_online(user_row)` = `now - last_seen < PRESENCE_TIMEOUT_SECONDS` (env `DEVPLACE_PRESENCE_TIMEOUT_SECONDS`, default 60). Profile (`routers/profile/index.py` -> `profile_online`) and messages (`routers/messages.py` seed) read `last_seen` off the user row they already loaded - no extra query. Exposed as the Jinja global `is_online(user)` (`templating.py`), on `UserOut.last_seen` and `ProfileOut.profile_online`. This is the **only** cross-worker-correct approach here because pub/sub is in-process.
|
||||
|
||||
**Live path (lock owner only), change-only + hysteresis:** `PresenceRelayService` (`services/presence_relay.py`, `BaseService`, default-enabled, 2s tick, registered in `main.py`) is a sibling of `NotificationRelayService`/`LiveViewRelayService`. Each tick it recomputes ONE global online set `self._online` (dot-subscribed uids batch-read via `get_users_by_uids` + roster candidates) and drives BOTH the per-user dots and the feed roster from that one set, so they can never disagree. The set uses **hysteresis** via `presence.stays_online(elapsed, was_online)`: a user becomes online at `PRESENCE_TIMEOUT_SECONDS` but only drops after `+ PRESENCE_ONLINE_MARGIN_SECONDS` (env `DEVPLACE_PRESENCE_ONLINE_MARGIN_SECONDS`, default 20) - **quick to go online, slow (grace margin) to go offline** - which kills boundary flicker for a user hovering near the timeout. It publishes `{online, last_seen}` to `public.presence.{uid}` **only when a topic's `online` bool changed OR the topic is newly subscribed (first-seen)** - never on a fixed interval, so a steady page emits nothing after the initial frame (`self._published[topic] -> bool`, pruned to active topics). `public.presence.{uid}` is subscribable by any logged-in user (`pubsub/policy.py` allows `public.*`); guests fall back to the server-rendered initial state. The one-directional grace also means dots and roster stay consistent across viewers (the shared `self._online` is the single authority). To keep it lightweight the relay reads all due users in **one batched `get_users_by_uids`** per tick.
|
||||
|
||||
**Frontend:** `static/js/PresenceManager.js` (`app.presence`, constructed with `this.pubsub` in `Application.js`, mirroring `LocalTime`/`CounterManager`) scans `[data-presence-uid]` elements, subscribes each uid to `public.presence.{uid}` (deduped per uid, so a repeated author is one subscription), and treats each frame's `online` flag as **authoritative** (`entry.online`), toggling the `online` class + (for `data-presence-label` elements) the "online / last seen X / offline" text. Because the relay is change-only and authoritative, a live-subscribed dot is **not** expired by the client clock (no false-offline flicker for an active user whose `last_seen` the client cannot see advancing); the 20s `last_seen` staleness timer only applies to entries that never received a frame (guests / degraded). The window is read from `<body data-presence-timeout>`. Both the profile `.profile-presence` dot and the messages `#messages-presence` span carry `data-presence-uid`/`data-presence-last-seen`.
|
||||
|
||||
**Online-now roster (feed):** the same relay maintains ONE shared topic `public.presence.roster`, republished **only when the SET of online uids changes** (a `frozenset` compare, so pure reordering never republishes). `services/presence.py` `online_users(limit)` (strict, feed initial render) and `online_candidates(limit)` (grace window, relay hysteresis) both go through `database.get_online_users(cutoff_iso, limit)`, which reads users with `last_seen >= cutoff` via the `idx_users_last_seen` index (the one place presence is queried by `last_seen`; `config.PRESENCE_ONLINE_LIMIT`, env `DEVPLACE_PRESENCE_ONLINE_LIMIT`, default 30). **The list is ordered ALPHABETICALLY by username** (`get_online_users` `order_by=["username"]` + case-insensitive `presence.sort_by_username`), NOT by recency, so avatars keep a stable position and do not needlessly reshuffle as people's `last_seen` ticks. `routers/feed.py` puts `online_users` on the context (`FeedOut.online_users`) and `feed.html` renders the initial **Online now** panel as a `.sidebar-section` at the bottom of the left feed sidebar (`aside.sidebar-card`); `static/js/OnlineUsers.js` (`app.onlineUsers`) subscribes to `public.presence.roster` and re-renders the avatar list + count live. Roster avatars use a plain green `.presence-dot` with NO `data-presence-uid` (list membership IS the presence, so no per-user subscription - the relay drops a user from the roster when they go offline).
|
||||
|
||||
**Avatar presence dot (sitewide, DRY):** a small corner dot on **every** user avatar (green online, muted grey offline) comes from ONE reusable partial `templates/_presence_dot.html` - `<span class="presence-dot" data-presence-uid data-presence-last-seen>` guarded on `_user.get('uid')` (a partial-dict author, e.g. the issues includes, renders no dot). It carries **no** `data-presence-label`, so `PresenceManager` colours it with zero extra JS. It is included by the shared avatar partial `templates/_avatar_link.html` (its `.user-avatar-link` anchor is the positioning host, covering ~19 sites) and by the handful of raw-`<img class="avatar-img">` sites wrapped in a positioned `<span class="avatar-badge">` (the two `base.html` nav avatars, the `profile.html` hero + followers list, the `messages.html` conversation list). CSS in `static/css/base.css` (`.user-avatar-link`/`.avatar-badge` `position:relative;display:inline-flex`, `.presence-dot` sized `30%` of the avatar clamped 8-14px with a `--bg-card` ring, `.online` -> `--success`), so it is proportional and responsive at every avatar size with no per-size class. `database/follows.py` `get_follow_list` now carries `last_seen` in its trimmed dict so the followers/following dots resolve (all other author dicts are full `get_users_by_uids` rows). `dp-avatar` (`AppAvatar.js`) is docs-demo only (no real user avatars) and is intentionally out of scope. Reuse `_presence_dot.html` + the `.avatar-badge` wrapper for any new avatar surface - never hand-roll a presence dot.
|
||||
|
||||
**Messaging refactor:** the old presence was per-worker and WS-connect-based (`message_hub.is_online`/`last_seen`, `_announce_presence`, the WS `presence` frame) and broke with >1 worker. That display path was **removed**; `message_hub` keeps only its socket connection tracking for message delivery. The messages header presence is now the shared `PresenceManager`, so chat presence is finally cross-worker correct. **Never re-implement WS-connect presence** - reuse `presence.is_online`, the `public.presence.{uid}` topic, and `PresenceManager`.
|
||||
58
devplacepy/services/audit/CLAUDE.md
Normal file
58
devplacepy/services/audit/CLAUDE.md
Normal file
@ -0,0 +1,58 @@
|
||||
This file documents the audit log subsystem. Claude Code auto-loads it when a file under `devplacepy/services/audit/` is read or edited.
|
||||
|
||||
## Audit log (`services/audit/`)
|
||||
|
||||
**Admin-only, append-only** record of every state-changing action. The authoritative event catalogue is `events.md` (its storage model, relation vocabulary, and per-event specs are authoritative); the catalogue currently spans 223 keys across 38 domains.
|
||||
|
||||
### Package layout
|
||||
|
||||
- `store.py` - the `audit_log` + `audit_log_links` tables, `ensure_tables`, `insert_event`/`insert_links`/`get_event`/`get_links`/`sweep`.
|
||||
- `categories.py` - `category_for(event_key)` (longest-prefix map).
|
||||
- `record.py` - the recorder + link builders.
|
||||
- `query.py` - `list_events`/`filter_options`/`get_event_with_links` for the admin UI (admin list/detail).
|
||||
- `service.py` - `AuditService` (retention sweep).
|
||||
|
||||
`ensure_tables()` + the 11 indexes are wired into `database.py` `init_db()` via a **local import** (audit modules import `database`/`utils`, so the import is deferred to avoid the cycle). `AuditService` is registered in `main.py`.
|
||||
|
||||
### Two recorder entrypoints (the reuse keystone)
|
||||
|
||||
`record(request, event_key, *, user=_UNSET, actor_kind=None, target_type/uid/label, old_value, new_value, summary, metadata, result="success", origin, via_agent, links, category)` is for HTTP/WebSocket handlers - `request` may be a Starlette `Request` OR `WebSocket` (both expose `.headers`/`.url`/`.client`). It auto-derives the actor from `get_current_user(request)` (or an explicit `user=`), the request fields, and the **`X-Devii-Agent`** header (-> `origin=devii`, `via_agent=1`), and auto-appends the `actor` link.
|
||||
|
||||
`record_system(event_key, *, actor_kind, actor_uid, actor_username, actor_role, origin, via_agent, ...)` is for request-less contexts (rewards, notifications, the AI gateway ledger, the news/container services, Devii turns/tasks, job services, the CLI).
|
||||
|
||||
**Both are best-effort: wrapped in try/except, they NEVER raise into the caller** - the audited action is never blocked by a logging failure. `summary` is HTML-stripped and capped at 140 chars; `metadata` is JSON-serialised. Link builders (`audit.target`, `audit.parent`, `audit.author`, `audit.recipient`, `audit.project`, `audit.instance`, `audit.setting`, `audit.job`, `audit.poll`, `audit.option`, `audit.task`, ...) keep call sites terse.
|
||||
|
||||
### Emission convention
|
||||
|
||||
Emission is **explicit per mutation** - `audit.record(...)` / `record_system(...)` calls placed after the mutation succeeds, or on the guard branch with `result="denied"`/`"failure"`. DRY choke points:
|
||||
|
||||
- Posts/projects/gists create/edit/delete record inside `content.py` (`create_content_item`/`edit_content_item`/`delete_content_item`, the latter two derive the key from `target_type`).
|
||||
- Project file ops record via the `_audit_file`/`_edit`/`_fail` helpers in `project_files.py` (read-only guard -> `result="denied"`).
|
||||
- Services via `_audit_service`.
|
||||
- Containers via `audit_instance` (in `routers/projects/containers/_shared.py`) for the HTTP path, and in the Devii `actions/dispatcher.py` `_audit_mechanic` hook (after a successful `_run`) for the agent path - the two paths are disjoint (Devii calls `api.py` directly, never the routes), so there is no double counting. The dispatcher hook also emits the `devii.*` self-config mechanics (behavior/tools/tasks/lessons/customization) from one place.
|
||||
|
||||
The dispatcher's **authorization guard** (before `_run`) mirrors this with `_audit_denied`: a tool call refused for `requires_auth`/`requires_admin`/`requires_primary_admin` records a `security.authz.denied` event (`origin="devii"`, `via_agent=1`, `result="denied"`, `metadata.tool`/`reason`) so an agent-driven escalation attempt is visible in the admin Audit Log, not just the app log. This is what surfaces a non-admin's Devii probe of `admin_*`/`db_*` tools (the tool schemas are already withheld from non-admins by `tool_schemas_for`, so this fires only when the model invents a name it never received).
|
||||
|
||||
### Failures and denials are events
|
||||
|
||||
`auth.login.failure`, `auth.password.forgot_request` (unknown email), `security.rate_limit.block`, `security.maintenance.block`, `security.authz.denied` (emitted inside `require_user`/`require_admin` on the HTTP side, and inside the Devii dispatcher's `_audit_denied` for a refused agent tool call), self-role-change and self-disable denials, **admin-seniority denials** (a junior admin managing a senior admin), read-only write attempts, and `ai.quota.exceeded` all record with `result` set to `failure`/`denied`. Auth failures, authz denials (in `require_user`/`require_admin`), rate-limit/maintenance blocks, self-role-change/self-disable denials, and quota blocks are recorded with `result` set.
|
||||
|
||||
### Admin UI
|
||||
|
||||
`GET /admin/audit-log` (paginated, filterable list, `admin_section="audit-log"`) and `GET /admin/audit-log/{uid}` (detail + links) in the `routers/admin/` package, both `require_admin`, both negotiate via `respond(..., model=AuditLogOut|AuditEventOut)`. Templates `admin_audit_log.html` / `admin_audit_event.html` reuse `admin.css` + a small `audit.css`; `_pagination.html` gained an optional backward-compatible `pagination_query` prefix (default empty) so filters survive paging. Sidebar link sits last, before Settings.
|
||||
|
||||
### Devii access (read-only)
|
||||
|
||||
Admins query the same two routes conversationally via the admin-only Devii tools `audit_log` (GET `/admin/audit-log`) and `audit_event` (GET `/admin/audit-log/{uid}`) (`services/devii/actions/catalog.py`, `handler="http"`, `requires_admin=True`) - no new endpoint, since the `respond(...)` routes already serve JSON to Devii's `Accept: application/json` client (like `admin_list_users`). `audit_log` forwards every filter as a query param (`page`, `event_key`, `category`, `actor_role`, `actor_uid`, `origin`, `result`, `q`, `date_from`, `date_to`) and returns `AuditLogOut`, whose `options` object lists the valid values for each filter so the agent can discover them in one call. `audit_event` returns `AuditEventOut` (the row + related links). A system-prompt steer in `agent.py` (the AGGREGATES block) routes audit/history/"who did X" questions to these tools.
|
||||
|
||||
### Deferred persistence
|
||||
|
||||
`record`/`record_system` do all the cheap prep (actor resolve, sanitize, `json.dumps`, link assembly) on the request thread, then hand the two DB inserts to the **background task queue** (`services/background.py`, `background.submit(_persist, row, links)`) so the request returns without waiting on SQLite. The `uid`/`created_at` are generated eagerly at record time (so the return value and the audit timestamp reflect the action, not the flush). Under `DEVPLACE_DISABLE_SERVICES=1` (tests) the queue runs inline, so audit rows are visible immediately. See CLAUDE.md -> "Background task queue".
|
||||
|
||||
### Retention
|
||||
|
||||
`AuditService` (the `audit` service, daily) prunes rows older than `audit_log_retention_days` (default 90; `0` disables) via `store.sweep`. Configurable on the Services page like any other service.
|
||||
|
||||
### Adding an event
|
||||
|
||||
Pick/extend a key in `events.md`, add the `category_for` prefix if it is a new domain, then call `audit.record`/`record_system` at the mutation point with the right links and `result`. Never gate the audited action on the recording.
|
||||
32
devplacepy/services/backup/CLAUDE.md
Normal file
32
devplacepy/services/backup/CLAUDE.md
Normal file
@ -0,0 +1,32 @@
|
||||
This file documents the Backup service (`devplacepy/services/backup/`, `devplacepy/services/jobs/backup_worker.py`, `devplacepy/routers/admin/backups.py`). Claude Code loads it automatically whenever a file under `devplacepy/services/backup/` is read or edited.
|
||||
|
||||
## Overview
|
||||
|
||||
Admin-only, enterprise-grade backups built on the **same async-job pattern as zip/fork** (see `devplacepy/services/jobs/CLAUDE.md`), so nothing runs on the request path. `BackupService(JobService)` (kind `backup`) is registered in `main.py` and overrides `run_once` to call `super().run_once()` (reap/refill/sweep) and then `_fire_due_schedules()`.
|
||||
|
||||
## Targets and worker
|
||||
|
||||
- **Targets** (`store.BACKUP_TARGETS`): `database` (consistent SQLite snapshot of the main DB + `devii_tasks.db` + `devii_lessons.db` via the `sqlite3` online backup API, so it is consistent under WAL - never a raw file copy), `uploads` (`UPLOADS_DIR`), `keys` (`KEYS_DIR`), `full` (database snapshot + uploads + keys; regenerable/volatile dirs - staging, locks, zips, container workspaces, chroma - are intentionally excluded). `service._materialize(target, staging)` returns a list of `{root, path}` sources; DB targets are snapshotted into `staging/db_snapshot` first.
|
||||
- **Worker** (`services/jobs/backup_worker.py`, stdlib only): reads a JSON spec `{sources:[{root,path}]}` and an output path, builds a deterministic `tar.gz` (uid/gid 0, symlinks skipped) rooted at each `root/`, and returns `{bytes_in, bytes_out, file_count, dir_count, sha256}`. Invoked via `asyncio.create_subprocess_exec` like `zip_worker`.
|
||||
|
||||
## Storage and data model
|
||||
|
||||
- **Storage:** archives go under `config.BACKUPS_DIR` (`data/backups/`, in `DATA_PATHS`) sharded with `attachments._directory_for` on the **random uuid tail** (same load-bearing reason as zips/blobs), named `{target}-{YYYYMMDD-HHMMSS}-{tail}.tar.gz`. Staging is `config.BACKUP_STAGING_DIR` (`data/backup_staging/`), removed in `process` `finally`.
|
||||
- **Data model** (`store.py`, ensured in `database.init_db` via `backup_store.ensure_tables()`): `backups` (NOT soft-deletable - an archive is a reclaimable operational artifact, hard-deleted like zips) and `backup_schedules` (in `SOFT_DELETE_TABLES`, born-live `deleted_at:None`). `store` holds all CRUD plus `compute_storage_stats()` (du of every major data area + `shutil.disk_usage`, run in `asyncio.to_thread` from the route, 30s in-process TTL cache so the walk never blocks).
|
||||
- **Permanent artifact:** `cleanup(job)` only removes leftover staging, NEVER the archive. Job retention prunes the `jobs` row; the archive and `backups` row persist until an admin deletes it, a schedule rotates it out (`keep_last`), or `devplace backups clear`. Deleting a backup is a HARD delete (unlink file + delete row) - correct because backups are GC artifacts, the documented exception to the soft-delete rule.
|
||||
|
||||
## Schedules
|
||||
|
||||
`backup_schedules` carry `kind` (`interval`|`cron`), `every_seconds`/`cron`, `enabled`, `keep_last`, `next_run_at`, run bookkeeping. `_fire_due_schedules` (lock-owner only, so each fires once) compares `next_run_at <= to_iso(now_utc())` and enqueues a `backup` job + a `backups` record, then advances `next_run_at` via `schedule.next_run`. **Timestamp format is load-bearing:** schedule `next_run_at` uses the devii `schedule.to_iso` format (`%Y-%m-%dT%H:%M:%S`, no tz/micros) on BOTH sides of the comparison so lexicographic compare equals chronological - do not mix it with `datetime.isoformat()`.
|
||||
|
||||
## Routes and frontend
|
||||
|
||||
- **Routes** (`routers/admin/backups.py`, all `require_admin`, mounted via `admin/__init__`): `GET /admin/backups` (dashboard, `respond(..., model=BackupDashboardOut)`), `GET /admin/backups/data` (JSON dashboard for the monitor + Devii), `POST /admin/backups/run`, `GET /admin/backups/jobs/{uid}` (`BackupJobOut` for `JobPoller`), `POST /admin/backups/{uid}/delete`, `GET /admin/backups/{uid}/download` (`FileResponse`, path-guarded against `BACKUPS_DIR`), `GET /admin/backups/{uid}` (`BackupOut`), and schedule CRUD `schedules/create|{uid}/edit|toggle|run|delete`. **Route order:** every literal sub-path (`data`, `run`, `jobs/{uid}`, `schedules/...`) is declared BEFORE the `{uid}` catch-alls.
|
||||
- **Frontend:** `static/js/BackupMonitor.js` (`window.BackupMonitor`, started inline from `admin_backups.html` like `AiUsageMonitor`) polls `/admin/backups/data` every 8s and renders storage/backup/schedule sections; run uses `Http.send` + `JobPoller`; delete/toggle/run/schedule actions use delegated clicks + `Http.send`; the schedule modal (create/edit) reuses the shared `.modal-overlay`/`data-modal` pattern. CSS `static/css/backups.css`. Sidebar link in `admin_base.html` (`admin_section == 'backups'`).
|
||||
- **Devii** (all `requires_admin=True`, `handler="http"`): `backups_overview`, `backup_run`, `backup_status`, `backup_delete` (+confirm, in `CONFIRM_REQUIRED`), `backup_schedule_create`, `backup_schedule_delete` (+confirm). **CLI:** `devplace backups list|run <target>|prune|clear`. **Audit:** `job.backup.complete|failed` (category `backup`), `admin.backup.run|delete`, `admin.backup_schedule.create|update|toggle|delete` (category `admin`). **Docs:** admin API group endpoints + admin prose page `backups`.
|
||||
|
||||
## Download restricted to the primary administrator (load-bearing)
|
||||
|
||||
A backup archive contains the whole database, uploads, and VAPID keys, so `GET /admin/backups/{uid}/download` is restricted beyond `require_admin` to the **primary administrator** - the earliest-created user who currently holds the Admin role (the founder; `utils._create_account` auto-promotes the first registered user). The keystone is `database.get_primary_admin_uid()` (`SELECT uid FROM users WHERE role='Admin' ORDER BY created_at ASC, id ASC LIMIT 1`; users have no `deleted_at`, do NOT filter it) and `utils.is_primary_admin(user)` (admin AND `uid == get_primary_admin_uid()`, also a Jinja global). The download endpoint records `security.authz.denied` and raises `403` for any other admin. The `download_url` field is gated identically at every emission point (`_download_url`/`_backup_payload`/`_job_payload`, threaded `can_download = is_primary_admin(admin)`), so `GET /admin/backups/data`, `/jobs/{uid}`, and `/{uid}` never expose the URL to a non-primary admin; `BackupDashboardOut.can_download_backups` carries the flag. **Client:** `BackupMonitor.js` reads `data.can_download_backups`; a `done` backup renders an enabled `<a>` Download only for the primary admin, otherwise a **disabled** `<button>` with `title="Not available"`. The disabled state cannot be bypassed - the URL is never sent and the endpoint 403s regardless. If the founder is demoted/removed the crown passes to the next-oldest admin. Creating/running/deleting/scheduling backups stay open to all admins.
|
||||
|
||||
No restore into a live server (by design - overwriting the DB/uploads while running risks corruption). Restore is a manual ops procedure documented on the `backups` docs page. The live view relay's `admin.backups` topic publishes with `can_download=False` unconditionally (see `devplacepy/services/CLAUDE.md` -> Live view relay) - `BackupMonitor` derives real download capability only from its authoritative HTTP poll.
|
||||
60
devplacepy/services/bot/CLAUDE.md
Normal file
60
devplacepy/services/bot/CLAUDE.md
Normal file
@ -0,0 +1,60 @@
|
||||
# Bot fleet (`devplacepy/services/bot/`)
|
||||
|
||||
This file documents the Playwright-driven AI persona fleet. Claude Code auto-loads it when a file under `devplacepy/services/bot/` is read or edited.
|
||||
|
||||
## Overview
|
||||
|
||||
`BotsService` (`services/bot/`) is a Playwright-driven fleet of AI personas that browse and interact with a DevPlace instance (posts, comments, votes, gists, projects, issues, follows, messages). It is the former standalone `dpbot.py` refactored into a package and wired into the service framework. Opt-in (`default_enabled = False`). Every other AI consumer's endpoint defaults reference "news, bots, Devii guests" collectively as internal-gateway consumers (see `GatewayService`).
|
||||
|
||||
## Module layout (one responsibility per file)
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `config.py` | Static constants + defaults (URLs, model, costs, personas, categories, paths) |
|
||||
| `runtime.py` | `BotRuntimeConfig` dataclass - per-run settings threaded into a bot |
|
||||
| `state.py` | `BotState` dataclass + JSON persistence (incl. the per-bot `identity` card) |
|
||||
| `llm.py` | `LLMClient` (parameterized: key/url/model/costs); content generation + quality checks + the `decide`/`generate_identity` decision engine |
|
||||
| `news_fetcher.py` | `NewsFetcher` - TTL-cached article source |
|
||||
| `browser.py` | `BotBrowser` - Playwright wrapper (human-like typing/clicking/scrolling) |
|
||||
| `registry.py` | `ArticleRegistry` - flock-based cross-bot article dedupe |
|
||||
| `bot.py` | `DevPlaceBot` - the orchestrator (sessions, action cycle, `run_forever`) |
|
||||
| `service.py` | `BotsService(BaseService)` - fleet manager + metrics |
|
||||
|
||||
## Mechanics
|
||||
|
||||
- `run_once` is a **reconcile tick**: it ensures `bot_fleet_size` bot tasks are alive (relaunching dead ones), scales the fleet up/down, and is a no-op heavy work otherwise. Each bot is an `asyncio` task running `DevPlaceBot.run_forever` in the lock worker's loop. `on_disable` cancels the fleet (each bot closes its browser on `CancelledError`).
|
||||
- All "startup parameters" of the old script are `config_fields`: `bot_fleet_size` (was `--bots`), `bot_headless` (was `--headed`), `bot_max_actions` (was `--actions`), plus `bot_base_url`, `bot_api_url` (defaults to the internal gateway), `bot_news_api`, `bot_model` (defaults to `molodetz`), `bot_api_key` (secret; falls back to `internal_gateway_key()`), `bot_input_cost_per_1m`, `bot_output_cost_per_1m`.
|
||||
- **A bot uses its own account's `api_key` for gateway calls once it has an account.** `bot_api_key` (-> `cfg.api_key`, the shared `internal_gateway_key()` fallback) is only the bootstrap credential used until the bot registers/logs in. After `_ensure_auth()` succeeds in `run_forever`, `DevPlaceBot._adopt_account_api_key()` fetches the bot's own `users.api_key` from its profile JSON (`fetch('/profile/{username}', {Accept: 'application/json'})` over the authenticated browser session, where `ProfileOut.api_key` is exposed to the owner) and switches `LLMClient.api_key` to it, persisting it in `BotState.account_api_key`. The key is read once and reused on every later session (the `LLMClient` is constructed with `state.account_api_key or cfg.api_key`, and `_raw_call` reads `self.api_key` per request so a mid-session swap takes effect on the next call). This routes each bot's gateway spend through its own user attribution (`resolve_owner()` -> `(owner_kind="user", owner_id=uid)`) instead of the shared internal key. Adoption is best-effort: a fetch failure leaves the bot on the fallback key.
|
||||
- **Behavior and pacing tuning are admin `config_fields` too**, threaded through `BotRuntimeConfig` into the bot (never read module constants directly for these). The `config.py` constants (`MAX_BOTS_PER_ARTICLE`, `ARTICLE_TTL_DAYS`, `GIST_MIN_LINES`, `ACTION_PAUSE_MIN_SECONDS`/`ACTION_PAUSE_MAX_SECONDS`, `BREAK_SCALE_DEFAULT`) are only the **defaults**; the live values come from the settings: **Behavior** group - `bot_max_per_article` (-> `ArticleRegistry.max_per_article`), `bot_article_ttl_days` (-> `ArticleRegistry.ttl_days`), `bot_gist_min_lines` (-> `LLMClient.gist_min_lines`); **Pacing** group - `bot_action_pause_min_seconds`/`bot_action_pause_max_seconds` (the intra-session pause, clamped lo/hi in `DevPlaceBot.__init__` so an inverted pair is harmless) and `bot_break_scale` (a float multiplier on the between-session break, floored at 0.1 and the result floored at 5s); **Decisions** group - `bot_ai_decisions` (-> `BotRuntimeConfig.ai_decisions`, the AI-decision kill switch) and `bot_decision_temperature` (-> `BotRuntimeConfig.decision_temperature`). To add a new tuning knob: add the default to `config.py`, a `ConfigField` to `BotsService.config_fields` (its `group` creates/joins a UI section automatically), a field to `BotRuntimeConfig`, the `cfg[...]` mapping in `_launch_slot`, and consume it via the runtime config - do not reach for the module constant at the call site.
|
||||
- **Live pricing** is published via `collect_metrics()`: fleet stat cards (bots running, fleet cost, LLM calls, tokens, posts/comments/votes) plus a per-bot table. `Fleet cost` is cumulative across restarts because each bot seeds its `LLMClient` totals from its saved `BotState`. **Cost is read from the gateway's authoritative per-call headers, not recomputed locally.** `LLMClient._account_usage(resp.headers, body_usage)` parses the `X-Gateway-*` response headers via `openai_gateway.usage.parse_usage_headers` (the shared parser, also used by `services/correction.py`) and takes `X-Gateway-Cost-USD` plus the prompt/completion token counts straight from the gateway, so the figure is cache-aware and native-cost-aware (it matches what `/admin/ai-usage` attributes to each bot's account). The `bot_input_cost_per_1m`/`bot_output_cost_per_1m` settings are **fallback-only**, applied to the response-body token counts solely when the endpoint returns no gateway headers (a non-gateway OpenAI URL); their defaults (`0.14`/`0.28`) mirror the gateway's chat pricing so even the fallback is sane. (Before this, the bots flatly multiplied every prompt token by an inflated `0.27`/`1.10` with no cache awareness, which massively overstated cost and the 24h projection.) `Cost rate` and `Projected 24h cost` come from a **sliding window**, not lifetime: `_sample_cost()` appends `(now, total_cost)` each tick into `_cost_samples`, evicts samples older than `config.COST_WINDOW_SECONDS` (600s), and measures the cost delta over the retained span; the projection is shown only once the window reaches `config.COST_WARMUP_SECONDS` (120s), otherwise both cards read "warming up". Samples reset when `_started_at` changes (`_cost_anchor`). Never project from lifetime `Fleet cost`/uptime, and never anchor the rate at startup - the boot burst (all bots ramping at once) then dominates the average and massively overstates a 24h projection extrapolated 274x from a few minutes.
|
||||
- Heavy deps (`playwright`, `faker`) are **lazily imported** inside `run_once`/`_launch_slot` (install the `bots` extra). The service registers and shows in the admin tab even when they are absent; it logs and stays idle. Per-bot state lives in `~/.devplace_bots/state_slot{N}.json`; the article registry in `~/.dpbot_article_registry.json`.
|
||||
- **Direct messages stay a dialog, never a monologue.** A bot may open a conversation, but `_compose_message` first calls `_spoke_last_in_thread()` and refuses to send when the last `.message-bubble` in `.messages-thread` is `.mine` (the bot itself sent the most recent message). So a bot only replies after the other person has spoken since its last message; it never sends two messages in a row. An empty thread (no bubbles) is allowed, so a bot can still initiate contact. This guards both reply paths (`_check_messages` and `_send_message`), mirroring how a real person waits for a reply before writing again.
|
||||
|
||||
## Live screenshot monitor (`services/bot/monitor.py`, `/admin/bots`)
|
||||
|
||||
The admin **Bot Monitor** (`/admin/bots`, sidebar link in `admin_base.html`, `require_admin`, `noindex`) shows the **latest low-quality screenshot per bot** with its username, persona, and current action/status, auto-refreshing every 2s. It is the constructive counterpart to the cost metrics: cost tells you what the fleet spent, the monitor tells you what each bot is looking at right now.
|
||||
|
||||
- **Capture is a `BotBrowser` concern** because the browser owns the Playwright `page`. `BotBrowser.bind_monitor(slot, username, persona)` (called from `DevPlaceBot.run_forever` once auth completes), `BotBrowser.note(action=, status=)`, and `BotBrowser.capture(reason, force=)` are the surface. `capture` takes a JPEG via `page.screenshot(type="jpeg", quality=MONITOR_JPEG_QUALITY=35, scale="css")`, throttled to one every `MONITOR_MIN_INTERVAL_SECONDS` (1s) unless `force=True`. **Capture is best-effort and never raises into the bot loop.** It fires at the meaningful-event choke points already centralized in `BotBrowser`: `goto` (page load, forced), `scroll`, `fill` (field input), `reload` (forced), plus `DevPlaceBot._action(tag, detail)` (every post/comment/vote/react/gist - an action-labelled forced capture via `asyncio.create_task`).
|
||||
- **Storage is in-memory + one file per bot, never accumulating.** `monitor` (the `BotMonitor` singleton) holds a `slot -> BotFrame` dataclass map of tiny metadata, and writes the JPEG bytes to `BOT_DIR/monitor/slot{N}.jpg` (`config.DATA_DIR`, OUTSIDE the package, NOT under `/static`), **overwritten atomically in place** (`.tmp` then `replace`). One frame per slot, capped at the fleet size (`bot_fleet_size`, 0 to 20). `_stop_slot` calls `monitor.drop(slot)` to evict the row + unlink the file. No DB table, no soft-delete (nothing is persisted as a row), no shard tree (one file per bot is already bounded). This is what keeps it cheap at full fleet: at most 20 small JPEGs in RAM-backed files, refreshed in place.
|
||||
- **Transport is polling, not WebSocket - deliberately.** Frames live only in the worker running the Bots service (the service lock owner), exactly like container logs/metrics. Rather than a lock-owner-gated WS with the 4013 retry, the monitor reuses the **existing `service_state` cross-worker bridge**: the lock owner publishes the frame metadata in `collect_metrics()["frames"]` (persisted to the `service_state` DB row by `BaseService._persist_state`), and ANY worker reads it back via `service_manager.get_service("bots").describe()["metrics"]["frames"]`. The page polls `GET /admin/bots/data` (`Poller`, 2s, `pauseHidden`) for that metadata and renders `<img>` tags at `GET /admin/bots/{slot}/frame.jpg` (raw bytes from the shared `BOT_DIR/monitor/` file, `Cache-Control: no-store`, served by any worker since it is a file under `DATA_DIR`). The `frame_url` carries a `?t=captured_at` cache-buster so a new frame is fetched only when it changed. This is the lightest correct option for the multi-worker model: no extra socket, no handshake, no 4013 bounce, and it reuses the same DB-backed metrics path the services page already polls. JSON shape is `schemas.AdminBotsOut`/`BotFrameOut`; Devii reads the same `/admin/bots/data` via the admin `bot_monitor` action; docs in `docs_api.py` (`admin-bots-*`, the frame endpoint in `NON_BODY_ENDPOINTS`).
|
||||
- **Frontend** is `static/js/BotMonitor.js` (`app.botMonitor`, one class, `Http` + `Poller`) rendering the `static/css/bots.css` responsive grid (`bot-active`/`bot-idle` by frame age, design tokens only). Add a new capture point by calling `self.b.capture("reason")` (or `note(...)` then capture) at the new event; do not screenshot outside `BotBrowser` (it owns the page and the throttle).
|
||||
|
||||
## Realism: titles, topic spread, gist quality, inter-bot threads, thread-aware distinct opinions
|
||||
|
||||
Five content/behaviour mechanics keep the fleet from reading as machine-generated. Treat them as the realism contract; do not regress them.
|
||||
|
||||
- **Post titles are rewritten, never the raw headline.** A post body reacts to a news article, but the title is generated separately by `llm.generate_post_title(headline, persona, category)` - a short (3 to 8 word) human title in the persona's voice, not the verbatim article headline. The raw headline is still the registry dedupe key (`clean_title`), but the typed title is `type_title` (the rewritten one, `strip_md`-capped at 120). Same applies to projects: `generate_project_title`/`generate_project_desc` take the persona and run through `strip_label`. **`LLMClient.strip_label`** removes LLM label echoes (`Title:`, `Project Name:`, `Snippet name:`, and a trailing ` Concept:`/` Description:` clause) so a title can never be `Project Name: X Concept: Y`. Run every generated title through `strip_label`.
|
||||
- **Personality drives topic and category, not just voice.** Category is chosen by `config.pick_category(persona)` (a persona-weighted `random.choices` over `PERSONA_CATEGORY_WEIGHTS`), replacing the old content classifier - a grumpy_senior rants, an enthusiastic_junior shows off and asks questions. Article selection is persona-scored: `config.persona_article_score(article, persona)` counts `SEARCH_TERMS` keyword hits, and both `_pick_article` (cache path) and `ArticleRegistry.reserve_unused` (direct path) rank articles by score plus a `random()` jitter, so different personas gravitate to different news rather than all reacting to the same headline.
|
||||
- **Gists are quality-gated and non-trivial.** `generate_gist` now writes the code first (8 to 20 lines, persona-flavoured via `PERSONA_GIST_FLAVOR`, language drawn from `PERSONA_LANGUAGES`), then titles/describes it from the code. `_create_gist` runs a two-attempt loop through `llm.gist_quality_check(title, code, language)`: a `TRIVIAL_GIST_TERMS` blocklist on the title, a minimum non-empty line count (`LLMClient.gist_min_lines`, admin `bot_gist_min_lines`, default 6), and a strict LLM judge that rejects 101-tutorial snippets. Mirrors the post/comment `quality_check` loop; a reject regenerates once then skips.
|
||||
- **Every comment is thread-aware and must hold a distinct opinion.** A bot never comments blind to what others already said. Before generating, `_comment_on_post` / `_reply_to_random_comment` call `DevPlaceBot._existing_comments()`, which scrapes the last `MAX_SIBLING_COMMENTS` (8) visible `.comment-text` bodies with their `/profile/` author (own comments excluded, each capped at `SIBLING_SNIPPET_LEN`=240). The block is threaded into `llm.generate_comment(..., existing_comments=)`, whose system prompt then forces a point none of them made - a different angle, a missed caveat, or respectful disagreement with one of them by name - and forbids repeating any opinion, framing, example, or question already raised. The mechanical backstop is in `llm.quality_check(..., siblings=)`: a candidate whose `_overlap_ratio` against the joined sibling text exceeds `SIBLING_OVERLAP_THRESHOLD` (0.55) is rejected as `echoes another comment` and regenerated (same two-attempt loop as the source-restatement gate at `RESTATEMENT_OVERLAP_THRESHOLD`=0.6). This is what stops a salient post from drawing a dozen near-identical "tipping point / feedback loop / cooldown" replies. Any new comment-generation path MUST gather siblings and pass them to both `generate_comment` and `quality_check`.
|
||||
- **Usernames read like nerd handles, not real names.** Bots do NOT sign up with faker person names; they pick devRant/Hacker-News-style handles. Two layers, LLM-first with an offline fallback: (1) `llm.generate_handle_candidates(persona)` asks the model for ~8 distinct handles grounded in the persona and its `SEARCH_TERMS` interests (tech nouns, leetspeak, adjective+noun, creatures, short word+number), each run through `handles.sanitize_handle`; the pool is cached on `BotState.handle_candidates` and consumed by `_next_handle()`. (2) `handles.make_handle(interests)` is the pure, offline algorithmic generator (curated word banks + probabilistic leetspeak + number/separator/casing decoration, persona-seeded from `SEARCH_TERMS`), used as the fallback whenever the LLM pool is empty or a signup collides. Both layers emit only `[A-Za-z0-9_-]`, 3 to 20 chars (the old `first.last` styles silently failed signup validation, which rejects dots). On a third signup retry a random number is appended to bust collisions. Word banks and tuning constants live in `services/bot/handles.py`; never reintroduce faker person-name handles.
|
||||
- **Bots engage each other, and threads deepen.** `ArticleRegistry` allows up to `max_per_article` holders per article (admin `bot_max_per_article`, default 2), each with a **distinct category** (stored as `holders: [{bot, category, time}]`; old `{bot, time}` rows are normalized on load), so two bots can post different angles on the same trending news - which gives them each other's posts to discuss. A holder ages out after `ttl_days` (admin `bot_article_ttl_days`, default 7). `_engage_community()` (called once per normal/deep session after notifications) goes to `/feed?tab=recent|trending`, ranks non-own posts with a preference for `known_users` authors, opens one, comments, and replies into the comment thread. The on-post reply-to-comment probability is also raised so multi-bot threads actually form. `reserve(title, owner, category)` is idempotent per owner and enforces the same distinct-angle / max-holders rules as `reserve_unused`.
|
||||
|
||||
## AI-driven decisions and identity cards (`bot_ai_decisions`)
|
||||
|
||||
Opt-in mechanic (off by default; full design and cost model in `aibots.md`). When `bot_ai_decisions` is on, a bot replaces the procedural action cascade with one LLM decision call per page, driven by a unique AI-generated identity. **Do not regress the grounding and the kill-switch fallback.**
|
||||
|
||||
- **Two LLM entrypoints, both on `LLMClient`.** `generate_identity(archetype)` runs once per bot (seeded from the persona archetype) and returns the identity card: `name`, `backstory`, `interests`, `dislikes`, `temperament`, and four 0.0-1.0 scalars (`verbosity`, `contrarianness`, `generosity`, `curiosity`) plus `rhythm`. `_normalize_identity` clamps every scalar and defaults `interests` to the archetype's `SEARCH_TERMS`, so a malformed model reply can never produce a broken card. The card persists on `BotState.identity` (serialised to `state_slot{N}.json`); old state files load with an empty dict and regenerate on first AI session. `decide(identity, page_state, menu, history, temperature)` returns `{plan, energy, stop_after}` (strict JSON), the per-page ordered action plan.
|
||||
- **JSON is parsed raw, never `clean()`-ed.** `_call` runs the output through `clean()` (which strips `*`/`_` and collapses whitespace, corrupting JSON keys like `stop_after`); the decision engine instead uses `_raw_call` (the extracted HTTP+cost-accounting core, no `clean`) and `_parse_json` (tolerates code fences and prose wrappers by slicing the first `{`..`}`). When adding any JSON-returning LLM method, use `_raw_call`, not `_call`.
|
||||
- **The harness supplies the options; the model only picks.** `DevPlaceBot._build_menu(page_state)` builds the action menu from the same observable-page predicates the random `_cycle` uses, listing only actions that are physically possible and policy-allowed (throttles like `can_post`/`can_project`/own-post-no-mention are enforced by *omitting* the action). `decide` filters the returned plan against that menu (`_filter_plan` drops any unknown action), re-asks once on malformed/empty output, and the executor `_run_plan` falls back to a single `navigate` to feed when the plan is still empty - **never a `random` draw**. `_dispatch_action` maps each action string to the existing per-action method (`_comment_on_post`, `_vote_for_page`, `_react_to_object`, `_navigate_to`, ...); content actions still run their `quality_check`/`gist_quality_check` gates unchanged, so quality cannot degrade.
|
||||
- **`_cycle_ai` mirrors `_cycle`'s signature and return tuple** so `run_forever` swaps between them with `(self._cycle_ai if use_ai else self._cycle)(...)`. `energy` and `stop_after` replace `_session_type` mood and `_fatigue_check`: the session ends when `session_actions >= stop_after`, the intra-action pause is `_scaled_pause()` (scaled by `energy`), and the between-session break is scaled by `energy` too. Timing jitter (`_read_like_human`, `_type_like_human`, `_idle`) stays procedural - the model decides *what*, not the millisecond cadence. `use_ai` is `self._ai_decisions and bool(self.state.identity)`; identity generation and the flag are gated so a disabled fleet pays nothing and behaves exactly as before.
|
||||
200
devplacepy/services/containers/CLAUDE.md
Normal file
200
devplacepy/services/containers/CLAUDE.md
Normal file
@ -0,0 +1,200 @@
|
||||
This file documents the Container manager subsystem (devplacepy/services/containers/, plus its routers/projects/containers and routers/admin/containers.py HTTP surface, and the shared ppy Docker image). Claude Code loads it automatically whenever a file under devplacepy/services/containers/ is read or edited.
|
||||
|
||||
## Overview
|
||||
|
||||
Admin-only. Supervises container instances, all running one shared prebuilt image. Driven through the `docker` CLI via `asyncio.create_subprocess_exec` behind a pluggable **`Backend` ABC**, never a docker SDK. Reference: admin docs `Services -> Container Manager` and the `containers` API group.
|
||||
|
||||
## Backend seam
|
||||
|
||||
`backend/base.py` defines the `Backend` ABC. `DockerCliBackend` drives `docker` via `asyncio.create_subprocess_exec`, streaming run logs line-by-line (`_stream`). `FakeBackend` is the in-memory test double (its `image_exists` returns `True`). `runtime.get_backend()`/`set_backend(b)` selects the active backend - tests call `set_backend(FakeBackend())`. A future Kubernetes/remote backend implements the same ABC. The container OS workspace mount point is the single constant `backend/base.py` `WORKSPACE_MOUNT` (`/app`).
|
||||
|
||||
## Security (load-bearing)
|
||||
|
||||
Mounting the docker socket is root-equivalent on the host. Every run/exec/lifecycle/schedule operation is gated `require_admin` (HTTP) and `requires_admin=True` (Devii). `--privileged` is never passed. User input is never shell-interpolated - every call is an arg-list subprocess, never `shell=True`. Resource limits are applied via `--cpus`/`--memory`.
|
||||
|
||||
On top of the admin gate, **per-user container isolation** applies (see root CLAUDE.md "Project visibility (is_private) and read-only" convention): only the **primary administrator** (earliest-created Admin) and the instance owner (creator or project owner) may manage an instance; other admins get read-only access, and only when the instance's project is public. Enforced via `content.py` predicates `owns_instance`, `can_view_project_containers`, `can_view_instance`, `can_manage_instance` (all reusing `is_primary_admin`): the primary administrator sees and manages every container including those on private projects; any other admin can VIEW containers of others only when the instance's project is public (private-project containers of others are invisible; of private containers they see only their own), and can MANAGE (edit/lifecycle/exec/terminal/sync/delete/schedules) only instances they own - non-owners get 403 plus an audit row with `result="denied"` (WS close `1008`). This is enforced at every entrypoint: `routers/projects/containers/` (`project_for` + `manage_guard` + the exec WS), the admin Containers manager (`_decorate` filtering + per-row `can_manage`, `_viewable_instance_or_404`, `_manage_denied`, create + project-search), the Devii container tools (`ContainerController._project` + `_require_manage`), and the live view relay (broadcast topics never carry private-project instances).
|
||||
|
||||
## One shared image, no in-app builds (load-bearing)
|
||||
|
||||
Every instance runs `config.CONTAINER_IMAGE` (default `ppy:latest`, override `DEVPLACE_CONTAINER_IMAGE`), built ONCE from `ppy.Dockerfile` via `make ppy` (context `devplacepy/services/containers/files`). There are no per-project Dockerfiles, builds, `ContainerBuildService`, or build UI - all removed.
|
||||
|
||||
`service._launch` calls `api.run_spec_for(inst, config.CONTAINER_IMAGE)`. `api.create_instance(project, *, name, ...)` fails fast with `ContainerError` if `backend.image_exists(CONTAINER_IMAGE)` is `false` ("run 'make ppy'"); it no longer takes a dockerfile/build. `backend.image_exists(ref)` (`docker image inspect`) is part of the `Backend` ABC.
|
||||
|
||||
Migrating off the old per-project-build model: `devplace containers prune-builds` removes the legacy per-project images (`backend.remove_image`) and clears the `dockerfiles`/`dockerfile_versions`/`builds` tables (one-time); existing instances auto-repoint to `ppy` (their stored `build_uid` is ignored).
|
||||
|
||||
**Default idle:** `run_spec_for` runs `["sleep", "infinity"]` when an instance has no `boot_command`, so a bare instance stays up instead of exiting (the image `CMD` is the same, but the explicit command makes it independent of the image).
|
||||
|
||||
## Data model
|
||||
|
||||
`store.py`, indexed in `init_db`:
|
||||
- `instances`
|
||||
- `instance_events` (audit trail; also home of the status-history rows written by `_set_status`)
|
||||
- `instance_metrics` (ring buffer, `METRICS_RING=720`; sweep keeps it bounded)
|
||||
- `instance_schedules` (cron/interval/once)
|
||||
|
||||
## Reconciler (`ContainerService`, `service.py`)
|
||||
|
||||
A `BaseService` reconciler; the model is desired-vs-actual, NOT a task per container. Each tick:
|
||||
1. `backend.ps(label=devplace.instance)` snapshots `docker ps` (label `devplace.instance=<uid>` is the join key).
|
||||
2. Converge each instance to its `desired_state`.
|
||||
3. Apply restart policies.
|
||||
4. Reap orphan containers (labeled but no DB row -> no orphans, no lost state).
|
||||
5. Fire due `instance_schedules` (reusing `devii/tasks/schedule.py` `cron_next`/`next_run`).
|
||||
6. Sample `docker stats`.
|
||||
|
||||
Only the service lock owner reconciles; HTTP handlers only flip `desired_state` / write events. `docker run --name <slug>` collision is the double-launch guard.
|
||||
|
||||
## Workspace materialization
|
||||
|
||||
Workspace = `/app` (the `WORKSPACE_MOUNT` constant). `project_files.export_to_dir` materializes the project to `config.CONTAINER_WORKSPACES_DIR/<project>` (`DATA_DIR/container_workspaces`, default `data/`) once, bind-mounted RW; `project_files.import_from_dir` (the inverse) syncs it back on the sync action. Both run via `asyncio.to_thread`.
|
||||
|
||||
**Runtime data must live OUTSIDE the `devplacepy/` package and NOT under `/static`**: workspaces (and zips) live in `config.DATA_DIR` (configurable via `DEVPLACE_DATA_DIR`). Never put generated data under `STATIC_DIR` - it is served publicly AND watched by dev `--reload` (scoped to `--reload-dir devplacepy`). The docker daemon must be able to bind-mount `DATA_DIR` for `/app`.
|
||||
|
||||
Every exec passes `-w /app` explicitly (`DockerCliBackend.exec` and the PTY exec in `routers/projects/containers/instances.py`), so one-shot `container_exec`, the interactive shell, and tmux sessions all start in the project workspace. The agent is told this (system prompt + `container_exec` tool summary) so it never prefixes a command with `cd /app`.
|
||||
|
||||
## Shared operations
|
||||
|
||||
`api.py` holds the operations shared by `routers/projects/containers/instances.py` (and the rest of the `routers/projects/containers/` subpackage) and the Devii `ContainerController` (`services/devii/container/`, `handler="container"`, 7 instance tools). The reconciler service defaults **disabled** (needs docker); enable it on `/admin/services`.
|
||||
|
||||
## HTTP routing surface
|
||||
|
||||
- `/projects/{slug}/containers` - the `routers/projects/containers/` subpackage: `instances.py` (creation/lifecycle/exec/logs/metrics/sync plus the exec websocket), `schedules.py` (cron/interval/once schedules), shared helpers in `_shared.py`. Every instance runs the shared `ppy` image. Discoverable from the project detail page's admin-only **Containers** button (gated by `content.can_view_project_containers` via the `viewer_can_containers` context flag) and from the admin index.
|
||||
- `/admin/containers` - `routers/admin/containers.py`: lists every instance across all projects with inline actions (start/stop/restart/terminal/edit/delete) plus a create modal (project search-select, run-as user search-select, boot language + source editor, restart policy, start-on-boot, env/ports/limits/ingress); `/admin/containers/{uid}` is the per-instance detail page (lifecycle, live logs/metrics, PTY terminal, schedules, ingress, sync, status history) and `/admin/containers/{uid}/edit` edits run-as user, boot language/script/command, restart policy, start-on-boot, and limits.
|
||||
|
||||
**Key reuse rule:** the admin Containers section adds NO lifecycle/logs/exec endpoints of its own - the instance carries its `project_uid`, so the admin detail route resolves the project and its frontend targets the existing `/projects/{slug}/containers/instances/{uid}/...` routes. Add any new instance operation to `routers/projects/containers/instances.py` only; the admin detail page picks it up for free.
|
||||
|
||||
## UI - two surfaces over one API set
|
||||
|
||||
Both are discoverable (an earlier version of the per-project page had zero links to it - fixed).
|
||||
|
||||
1. **Per-project manager**: `templates/containers.html` + `static/js/ContainerManager.js` handles instance creation through an app modal form (the `_macros.html` `modal()` macro + `ModalManager` `.visible` toggle); its instance list links out to the shared detail page. Reached from the project detail page's Containers button, and passes breadcrumbs so content clears the fixed nav.
|
||||
2. **Admin Containers section**: `routers/admin/containers.py` (mounted `/admin/containers`, sidebar link in `admin_base.html`, `admin_section="containers"`). `GET /admin/containers` lists every instance via `store.all_instances()` (decorated with project title/slug from one `projects` lookup) in an `.admin-table`. `GET /admin/containers/data` is the poll JSON. `GET /admin/containers/{uid}` renders `templates/containers_instance.html` + `static/js/ContainerInstance.js` - a dedicated detail page (lifecycle, poll logs/metrics, schedules add/delete, ingress, sync, interactive exec over a PTY WebSocket gated on the lock owner).
|
||||
|
||||
All container CSS (`static/css/containers.css`) uses app design tokens (`--bg-card`, `--text-primary`, `--success`/`--danger`/`--warning`, `--radius`) and the shared `.card` recipe.
|
||||
|
||||
## Admin CRUD manager (`/admin/containers`)
|
||||
|
||||
`/admin/containers` is a full manager, not a read-only list. `routers/admin/containers.py` adds (all `require_admin`, all reuse `api.*` directly - no docker/exec backend duplicated):
|
||||
- `POST /create` (`ContainerAdminCreateForm` -> `api.create_instance`, project search-select + run-as user + boot fields + restart policy + start_on_boot + env/ports/limits/ingress).
|
||||
- `GET`/`POST /{uid}/edit` (`ContainerEditForm` -> `api.update_instance_config`, edits `run_as_uid`/`boot_language`/`boot_script`/`boot_command`/`restart_policy`/`start_on_boot`/`cpu_limit`/`mem_limit`).
|
||||
- Stacked literal lifecycle routes `POST /{uid}/{start,stop,restart,pause,resume}` (the no-wildcard rule below).
|
||||
- `POST /{uid}/sync` (bidirectional).
|
||||
- `POST /{uid}/delete` (soft, owner-or-admin via the admin guard).
|
||||
- Two JSON search endpoints `GET /{projects,users}/search` (back the create/edit search-selects via `db.query LIKE`).
|
||||
|
||||
**Route order:** `/{uid}/edit` and the literal verb routes are declared BEFORE the `/{uid}` catch-all.
|
||||
|
||||
Frontend: `static/js/ContainerList.js` (poll-render rows with inline action buttons + the create modal with two debounced search-select widgets and a boot-language toggle) and `static/js/ContainerEdit.js` (the edit page). Both route through `Http.send`/`Http.getJson` and surface errors via `app.toast`; the Terminal button reuses `app.containerTerminals.open(slug, uid, name)`. The instance detail page (`containers_instance.html`) has a config summary (boot language, run-as uid, start-on-boot) plus a **Status history** card rendering the server-side `instance_events`.
|
||||
|
||||
Devii: `container_create_instance` takes the boot/run-as/start_on_boot args; `container_configure_instance` (`controller._configure_instance` -> `api.update_instance_config`) edits the same fields; both admin-only, audited via `_DEVII_CONTAINER_EVENTS`.
|
||||
|
||||
## No wildcard verb route (load-bearing)
|
||||
|
||||
`routers/projects/containers/instances.py` registers every instance verb as a **literal** route - `start`/`stop`/`pause`/`resume`/`restart` (stacked `@router.post` decorators on one `instance_action` handler that reads the verb from `request.url.path`), plus `delete`/`exec`/`sync`/`schedules`. There is deliberately NO `POST /instances/{uid}/{action}` catch-all: a wildcard segment matches its literal siblings too (Starlette matches first-declared), so a catch-all would silently swallow `exec`/`sync`/`delete`/`schedules` as `{"error": "unknown action: exec"}` whenever it sat above them. Add any new instance verb as a literal route (and to the `instance_action` decorator stack if it just flips desired state) - never reintroduce a `{action}` wildcard.
|
||||
|
||||
## Host port allocation
|
||||
|
||||
Host ports are auto-allocated and globally unique. `parse_ports` accepts a bare container port (`8899`, host side `0` = auto) or a pinned `host:container`. `api.assign_host_ports` (called in `create_instance`) resolves every `host=0` to the lowest free port in `[HOST_PORT_MIN=20001, 65535]` that is neither in `used_host_ports()` (the union of every instance's published host ports from `ports_json`) nor currently bound on the host (`_host_port_free` test-binds `0.0.0.0:port`). A pinned host port already published by another instance is rejected up front. This guarantees no two instances ever collide on a host port - previously a silent `docker run` "port is already allocated" failure. The resolved host port is what gets stored in `ports_json` and what the ingress proxy reads.
|
||||
|
||||
## Floating terminals (interactive shell)
|
||||
|
||||
An instance's shell is NOT inline - it is a floating `<container-terminal>` (`static/js/components/ContainerTerminal.js`) over the exec WS (`/projects/{slug}/containers/instances/{uid}/exec/ws`, `pty.openpty()` + `docker exec -it`, admin + `service_manager.owns_lock()` gated). xterm.js + the fit addon are vendored under `static/vendor/xterm/` and lazy-loaded. Opened by the instance-page **Open terminal** button (`app.containerTerminals.open(slug, uid, name)`, `ContainerTerminalManager`) or by telling Devii to "attach <container>" - the admin-only `open_terminal` **client** action (`client_actions.py`, `requires_admin=True`) -> `DeviiClient._openTerminal` -> the same manager.
|
||||
|
||||
**`FloatingWindow` base.** It extends the generic `FloatingWindow` base (`static/js/components/FloatingWindow.js` + `static/css/floating-window.css`), which owns the Devii-style chrome: drag, native `resize:both`, maximize/fullscreen, geometry persistence. `app.windows` (`WindowManager`, `components/WindowManager.js`) assigns z-index on `pointerdown` so the last-clicked window is on top (below the avatar/toasts).
|
||||
|
||||
**The Devii terminal also extends `FloatingWindow`.** `devii/devii-terminal.js` reuses the base drag/geometry/preset/resize/persist plus `_window`/`_registerWindow` (z-order + context-menu) machinery, overriding only the parts that differ - `_setState` (Devii adds a `closed` state + launcher FAB), `_defaultGeometry` (bottom-right, 760x600), `_changeFont` (per-user `--devii-font-size` CSS var), `_contextItems`, and `_persistGeometry` (delegates to its richer `{state,geometry,focused}` blob under `devii-terminal-state`). Devii keeps native `resize:both` (no `.fw-resize` grip); `devii.css` is unchanged - the only base change required was a `get _dragClass()` getter (default `fw-dragging`) that Devii overrides to `devii-dragging`, so the base drag handlers toggle Devii's own class and its existing CSS drives everything.
|
||||
|
||||
**Resize protocol.** The exec WS treats a brace-leading JSON frame `{"type":"resize","cols","rows"}` as a control message: it `TIOCSWINSZ`-es the host pty (`fcntl.ioctl`) **and** signals the `docker exec` client (`proc.send_signal(SIGWINCH)`) so the new size forwards through to the container's exec tty (vim/top reflow). The SIGWINCH is required: with `start_new_session=True` the host slave is not the client's controlling terminal, so the kernel does not auto-deliver it; the pair shares its winsize, so the client reads the new size off its slave stdio and calls Docker's exec-resize API. Any non-resize frame is raw stdin (backward compatible). The fit addon drives it client-side, and the window has a dedicated bottom-right resize grip (`.fw-resize`, since xterm covers the native CSS resize corner - the base `FloatingWindow` adds a manual grip instead of relying on `resize:both`). xterm renders ANSI natively, so the old `cleanTerm` stripping is gone for the interactive shell (the one-shot exec box still strips output, since `docker exec` without `-t` is non-TTY).
|
||||
|
||||
**Persistence (tmux).** A `?session=<name>` query param (validated `^[A-Za-z0-9_-]{1,64}$`) makes the WS run the shell inside tmux (`tmux new-session -A -s <name>`, falling back to bash if tmux is absent), so the session survives WS disconnect/window close - `proc.kill` on disconnect kills the tmux *client*, the server daemon + session live on. The frontend keeps exactly **one persistent slot** at a time (`ContainerTerminalManager`, the last-opened terminal): it attaches to session `devplace`, **auto-reconnects** with capped backoff on unexpected WS close (not on code `1008`), and is remembered per-user in `localStorage` (`ct-persistent:<scope>`); `app.containerTerminals.restore()` (called once in `Application.js` after `window.app`) reopens it on the next page load. Opening another terminal demotes/calls `setPersistent(false)` on the previous one (its tmux session lingers in the container, but the frontend stops auto-reconnecting/remembering it); explicit window-close (`_onClose`) detaches + forgets (session still survives, reopen re-attaches).
|
||||
|
||||
**Right-click context menu.** Every window wires `app.contextMenu.attach(this.win, () => this._contextItems())` (right-click + long-press), opening the shared `dp-context-menu`. The base `FloatingWindow._contextItems` is Minimize/Normalize/Close; `ContainerTerminal` overrides it with Copy (xterm `getSelection`, disabled when `!term.hasSelection()`) / Paste (clipboard -> WS stdin) / Sync files / Restart / Terminate (`dp-dialog` confirm, then delete + close) / Minimize / Normalize / Project (-> `/projects/{slug}`) / Files (-> `/projects/{slug}/files`) / Close - the lifecycle items POST to the existing `instances/{uid}/{sync,restart,delete}` routes. Devii's `_contextItems` override returns Copy (live `window.getSelection()`, or the current input line when empty) / Paste (clipboard inserted into `.devii-input` at the caret, `selectionStart/End` splice, then refocus) / Minimize / Normalize / Close (it is not bound to a container or project). Copy/Paste both use the async Clipboard API with a hidden-`textarea` + `execCommand("copy")` write fallback.
|
||||
|
||||
**Minimize/Normalize.** Geometry presets (`_presetGeometry(w,h)`, smallest-usable / comfortable), exposed both as titlebar buttons (`data-win="minimize|normalize"`) and menu items on every window.
|
||||
|
||||
## Pravda image (load-bearing, workspace ownership)
|
||||
|
||||
The `ppy` image (`ppy.Dockerfile`, built by `make ppy`, context `devplacepy/services/containers/files`) is a `python:3.13-slim-bookworm` base with Playwright plus a broad set of common Python libraries preinstalled, plus CLI tools (`tmux`, `apache2-utils` for `ab`, `procps`/`htop`/`iftop`/`iotop`, `netcat-openbsd` for `nc`, `zip`/`unzip`, `fakeroot`, git/curl/wget/vim/ack).
|
||||
|
||||
The security hotpatch that used to run per build is now baked into `ppy.Dockerfile` once. The final stage:
|
||||
- `COPY`s the **sudo superclone** (`files/sudo`) over `/usr/local/bin/sudo` (+ symlink `/usr/bin/sudo`; the real `sudo` package is not installed).
|
||||
- `COPY`s the **`aptroot` fakeroot wrapper** (`files/aptroot`, symlinked over `apt`/`apt-get`/`dpkg` in `/usr/local/bin` so pravda installs system packages without root).
|
||||
- `COPY`s **`pagent`** (`files/pagent`, the stdlib AI agent; reads `DEVPLACE_OPENAI_URL`+`DEVPLACE_API_KEY`, falling back to its public endpoint + `DEEPSEEK_API_KEY`) to `/usr/bin/pagent.py`, plus `files/.vimrc` to `/home/pravda/.vimrc` (whose AI helper - `AiEditSelection` - targets the same gateway as pagent via `DEVPLACE_OPENAI_URL`/`DEVPLACE_API_KEY`, with a public fallback, never `api.openai.com`).
|
||||
- Evicts any pre-existing uid-1000 user, creates user **`pravda` at `1000:1000`**.
|
||||
- Hands pravda ownership of the toolchain AND the OS package trees (`chown -R pravda` over `/usr/local/lib`, `/usr/local/bin`, `/usr/lib/python3`, `/opt`, `/app`, `/home/pravda`, plus `/usr/lib`, `/usr/bin`, `/usr/sbin`, `/usr/share`, `/usr/include`, `/etc`, `/var/lib`, `/var/cache`, `/var/log`, `/srv` so `apt`/`dpkg` can write; `~/.local/bin` on `PATH`).
|
||||
- Ends on `USER pravda`.
|
||||
|
||||
**Why uid 1000.** `/app` is bind-mounted from the host, and under DooD a container's UID maps 1:1 to the host. A container running as **root** wrote root-owned files into the workspace; the app process (host `retoor`, uid 1000) then hit `[Errno 13] Permission denied` in `export_to_dir` on the next instance create - a permanent per-project brick. Pinning the container UID to `1000` makes every write land as `retoor`, and pravda owning the global site-packages means runtime `pip install` succeeds as pravda with no elevation.
|
||||
|
||||
**The sudo superclone (`files/sudo`, POSIX `sh`)** is sudo-CLI-compatible but **never swaps user**: it parses the full flag interface (`-u/-g/-E/-H/-n/-S/-i/-s/--`, leading `VAR=value` env assignments, `-V/-l/-v/-k/-K` short-circuits), exports `SUDO_USER`/`SUDO_UID`/`SUDO_GID`/`SUDO_COMMAND`, then `exec`s the command **as the current user (pravda)**. So even a blind `sudo apt ...` / `sudo -u root ...` runs as uid 1000 and **cannot create a root-owned file** - the residual risk is gone by construction.
|
||||
|
||||
**Rootless apt (`files/aptroot`).** Pravda installs system packages directly (`apt install <pkg>`, no sudo). `aptroot` is symlinked over `apt`/`apt-get`/`dpkg` in `/usr/local/bin` (ahead of `/usr/bin` on `PATH`) and execs the real tool under **`fakeroot`** with `APT::Sandbox::User=root`; dpkg's chown-to-root calls are virtualized while every file actually written lands owned by **pravda (uid 1000)**. Combined with pravda owning the system trees (`/usr`, `/etc`, `/var/lib`, `/var/cache`, `/var/log`, `/srv`, ...), `apt install <pkg>` works with no real euid 0 and still cannot brick the bind mount. Run `apt update` first (the image clears `/var/lib/apt/lists`); a package whose maintainer script needs genuinely privileged syscalls may still fail - bake those into `ppy.Dockerfile` (its `RUN apt-get` runs as root before `USER pravda`), then `make ppy`.
|
||||
|
||||
**Trade-off (intentional).** The only genuinely-root operation that still does NOT escalate is binding a port < 1024 - use a high port + `/p/<slug>` ingress instead. Enforcement lives entirely in the Dockerfile (no `--user` on `docker run`). The `export_to_dir` unlink-before-write fix remains as belt-and-suspenders (the app owns the workspace dir, so it may delete any stale file in it regardless of owner before rewriting it).
|
||||
|
||||
## `PRAVDA_*` runtime env injection
|
||||
|
||||
`api.run_spec_for` merges `api.pravda_env(instance)` over the instance's own `env_json` (PRAVDA keys win), so every running container gets these platform vars:
|
||||
- `DEVPLACE_BASE_URL` - the `site_url` setting via `seo.public_base_url()`.
|
||||
- `DEVPLACE_OPENAI_URL` - `{base}/openai/v1`, the OpenAI-compatible gateway base.
|
||||
- `DEVPLACE_API_KEY` - the **creating** user's (or `run_as_uid`'s) `api_key`, resolved live.
|
||||
- `DEVPLACE_USER_UID` - the project **owner**'s uid.
|
||||
- `DEVPLACE_CONTAINER_NAME` - the instance name.
|
||||
- `DEVPLACE_CONTAINER_UID` - the instance uid.
|
||||
- `DEVPLACE_INGRESS_URL` - the instance's absolute public ingress URL `{base}/p/{ingress_slug}` when published and `site_url` is set, relative `/p/{slug}` if `site_url` is unset, empty if the instance has no `ingress_slug`.
|
||||
|
||||
These let code inside a container call back into the platform and the AI gateway authenticated as the user. **Injected at run** (`docker run -e`), never baked into the image: the image is shared by every instance, but name/uid/api-key are per-instance and base-url/key must stay fresh, so they are resolved each launch and never stored as secrets in the DB.
|
||||
|
||||
Resolution needs two instance columns set at `create_instance`: `created_by` (= `actor[1]` when the actor is a user) feeds `DEVPLACE_API_KEY`, and `owner_uid` (= `project["user_uid"]`) feeds `DEVPLACE_USER_UID`. Instances created before this change have neither and degrade to an empty key/uid (base-url and container name/uid still resolve) until recreated. `DEVPLACE_BASE_URL`/`DEVPLACE_OPENAI_URL` are empty when `site_url` is unset, so set the admin `site_url` for container-to-platform calls (and ingress URLs) to work.
|
||||
|
||||
## Ingress (`/p/<slug>`)
|
||||
|
||||
`routers/proxy.py` (registered at `/p`) implements the ingress reverse proxy. An instance opts in with `ingress_slug` + `ingress_port` (must be one of its mapped container ports; slug unique, validated in `api.validate_ingress`). `_resolve(slug)` -> `store.find_instance_by_ingress` -> `api.proxy_target(instance)` returns `(gateway, host_port)` (the instance's recorded docker bridge gateway + published host port); the route reverse-proxies HTTP (httpx) and WebSocket (`websockets` client) to `http://{host}:{port}/{path}`, stripping the `/p/<slug>` prefix. The reconciler records `container_ip`/`container_gateway` from `docker inspect` each running tick.
|
||||
|
||||
**`proxy_target` dials the container's docker bridge gateway + the published host port** (`gateway:host_port`), NOT loopback and NOT the container's own bridge IP. This is deliberate: `127.0.0.1` fails where docker's `nat OUTPUT` excludes `127.0.0.0/8` from DNAT and no `docker-proxy` binds loopback for a `0.0.0.0`-published port; the container's own bridge IP can be dropped by docker's bridge-isolation rule (`DOCKER ! -i docker0 -o docker0 -j DROP`); the gateway+published-port path survives both. `DEVPLACE_CONTAINER_PROXY_HOST` overrides the host (e.g. `host.docker.internal` for a containerized app) and still uses the published host port; before a gateway is recorded the proxy falls back to `127.0.0.1`.
|
||||
|
||||
Ingress is **public** (no auth) but SSRF-safe: host/port are derived from the instance row, never from user input. The Devii container tools return an absolute `ingress_url` built from `seo.public_base_url()` (the admin `site_url` setting), so the agent fetches the public production URL, not localhost (which web tools refuse); unset `site_url` yields a relative `/p/<slug>`. nginx needs a `/p/` location with WS upgrade + long timeouts (present in `nginx/nginx.conf.template`).
|
||||
|
||||
## Production wiring
|
||||
|
||||
The container manager drives the host docker daemon, which needs heavy wiring - all carried by the opt-in `docker-compose.containers.yml` overlay (docker socket mount, `INSTALL_DOCKER_CLI=true` build arg, `group_add` the docker gid). **`make docker-build`/`make docker-up` always apply this overlay** and self-derive its inputs: `DOCKER_GID` via `stat -c '%g' /var/run/docker.sock` (the socket's owning group) and `DEVPLACE_DATA_DIR` = `$(CURDIR)/data` (the project's own dir at its real host absolute path). This makes the manager work out of the box with no `sudo`, no `/srv`, no manual `.env` edits. A plain `docker compose up -d` drops the overlay (no CLI, no socket) and silently re-breaks it - always update through the make targets.
|
||||
|
||||
**The DooD bind-mount gotcha:** `docker run -v <path>:/app` resolves `<path>` on the HOST, so `DEVPLACE_DATA_DIR` must be mounted at an identical host+container path (the make targets use `$(CURDIR)/data` on both sides; manual `docker compose` users get a `/srv/devplace-data` default). Build contexts ship via the docker API tarball, so the container temp dir is fine. Set `DEVPLACE_CONTAINER_PROXY_HOST=host.docker.internal` only when a containerized app cannot route to the recorded gateway. See README "Container Manager wiring".
|
||||
|
||||
## Bidirectional newer-wins sync (load-bearing direction rule)
|
||||
|
||||
Sync is NOT one-directional import. `project_files.sync_dir_bidirectional(project_uid, workspace, user) -> {"exported", "imported"}` is the one helper that reconciles a project's virtual FS against an instance's `workspace_dir`: per file, the side with the newer timestamp wins (project `updated_at` epoch, via `datetime.fromisoformat(...).timestamp()`, vs filesystem `st_mtime`, with a 1s skew tolerance favouring export on ties), and a file present on only one side propagates to the other. It **NEVER deletes a file** - only creates/overwrites the older side. A **read-only** project (`is_readonly`) exports only, never imports (the read-only guard direction).
|
||||
|
||||
Both `api.sync_workspace` (HTTP/Devii sync action, returns `{exported, imported}`) and `api.sync_bidirectional_sync` (reconciler, non-blocking `record_event` system actor, logs only when non-zero) call the same helper. The reconciler runs it before every `_launch` AND on a ~60s wall-clock cadence over running instances (`SYNC_EVERY_SECONDS`, gated on `time.monotonic()` independent of the 5s reconcile tick). The per-instance boot-helper files (`.devplace_boot.py`/`.devplace_boot.sh`) are in `SYNC_SKIP_NAMES` so they never round-trip into the project.
|
||||
|
||||
## Run-as user = identity + API key ONLY (load-bearing constraint)
|
||||
|
||||
An instance's `run_as_uid` column selects WHICH DevPlace user's identity and `api_key` are injected (`DEVPLACE_API_KEY`, `DEVPLACE_USER_UID`), resolved in `api.pravda_env` ahead of the `created_by`/`owner_uid` fallback chain. It does **NOT** change the container OS user, which is ALWAYS `pravda` (uid 1000) - required for the bind-mounted `/app` (DooD uid maps 1:1 to host). Validate it against an existing user via `api.validate_run_as`.
|
||||
|
||||
## Boot source precedence (load-bearing)
|
||||
|
||||
Columns `boot_language` (`none`|`python`|`bash`) + `boot_script` (multiline source) sit alongside the legacy `boot_command`. Precedence in `api.run_spec_for`: `boot_script` (by language) > `boot_command` > image CMD (`sleep infinity`). When a boot script is set, the reconciler writes it into the workspace (`api.materialize_boot_script`, `.devplace_boot.py`/`.devplace_boot.sh`, excluded from sync) before launch and runs `python|bash /app/.devplace_boot.<ext>`. `api.validate_boot` enforces the language set and a 100k char cap.
|
||||
|
||||
## start_on_boot is per-container
|
||||
|
||||
The `start_on_boot` integer flag (0/1) forces `desired_state=running` ONLY for flagged instances when `ContainerService` first runs (`_boot_pass`, one-time); all others keep their last `desired_state`.
|
||||
|
||||
## Status-change choke point
|
||||
|
||||
Every reconciler status mutation flows through `service._set_status(inst, changes, reason=)`, which diffs old vs new `status` and, on change, writes an `instance_events` `status_change` row AND an audit `container.instance.status` `record_system` event (`old_value`/`new_value`). `_reconcile`, `_launch`, and `_handle_exit` (terminal + policy_restart) all flow through it, so previously-silent transitions (`running->stopped`/`->paused`/exit) are now logged and visible on the admin detail page's **Status history** card. New code that changes an instance's status in the reconciler must use `_set_status`, never a raw `store.update_instance(... status ...)`.
|
||||
|
||||
## New columns are ensured in init_db and defaulted in store.create_instance
|
||||
|
||||
`run_as_uid`/`boot_language`/`boot_script`/`start_on_boot` are added to the `instances` ensure-block in `init_db` (filtered/queried columns must exist on the cached schema) and seeded in the `store.create_instance` base dict so pre-existing rows degrade gracefully.
|
||||
|
||||
## Testing without Docker
|
||||
|
||||
Use `FakeBackend` (its `image_exists` returns `True`) + `runtime.set_backend`, and monkeypatch `config.CONTAINER_WORKSPACES_DIR` to a tmp dir. See `tests/unit/services/containers.py` and `tests/api/containers.py` (argv, instance creation on `config.CONTAINER_IMAGE`, image-not-built guard, reconcile matrices, schedule firing, ingress validation + live HTTP proxy, HTTP admin gate).
|
||||
|
||||
## Vibe coding on-ramp (user-facing doc)
|
||||
|
||||
The container runtime is also the basis of "vibe coding": the public prose page `templates/docs/getting-started-vibing.html` (slug `getting-started-vibing`, `SECTION_GENERAL`, not admin-gated so the docs stay public while the feature is admin-only/Alpha) is the canonical user-facing guide. It teaches the Devii-driven flow project -> container -> start -> terminal (`create_project`, `container_create_instance`, `container_instance_action`, the `open_terminal` client action), documents the three baked-in agents (`dpc` = DevPlace Code at `/usr/bin/dpc`, the Claude-Code-class coding agent; `botje.py` = the copy of `services/containers/files/bot.py` at `/usr/bin/botje.py`; `pagent`), all metered through the container's own `DEVPLACE_API_KEY`, the full `PRAVDA_*` env table (see `api.pravda_env`), and ingress at `/p/<slug>` via `ingress_slug`/`ingress_port`.
|
||||
|
||||
When the runtime, the agent binaries, or the `PRAVDA_*`/ingress contract change, update this page alongside the source.
|
||||
|
||||
**The agents are gateway-only:** `dpc`/`d.py` and `botje.py`/`bot.py` use a single `molodetz` backend pointed at `DEVPLACE_OPENAI_URL` (the gateway); the former direct `api.deepseek.com` fallback backend was removed so every in-container AI call is ledgered under the run-as user and nothing bypasses `gateway_usage_ledger`. `pagent`/`.vimrc` already posted to the gateway URL (using `DEEPSEEK_API_KEY` only as a key fallback, never the DeepSeek endpoint). Rebuild the image (`make ppy`) for the change to reach running containers.
|
||||
41
devplacepy/services/dbapi/CLAUDE.md
Normal file
41
devplacepy/services/dbapi/CLAUDE.md
Normal file
@ -0,0 +1,41 @@
|
||||
# Database API service (`devplacepy/services/dbapi/`, `devplacepy/routers/dbapi/`)
|
||||
|
||||
This file documents the primary-administrator-only, read-only generic database API. Claude Code auto-loads it when a file under `devplacepy/services/dbapi/` is read or edited.
|
||||
|
||||
## Overview
|
||||
|
||||
`dbapi/` (routers package, mounted at `/dbapi`) is a **primary-administrator-only, strictly READ-ONLY** generic database API over `dataset` (never inserts/updates/deletes/restores; the write surface was removed because it bypassed every per-route admin safeguard). `tables.py` (`GET /tables`, `GET /{table}/schema`), `crud.py` (read only: `GET /{table}` + `GET /{table}/{key}/{value}`, `?include_deleted`), `query.py` (`POST /query` hard SELECT-only validated read-only run; `POST /query/async` + `GET /query/{uid}` + `WS /query/{uid}/ws` via `DbApiJobService` kind `dbquery`), `nl.py` (`POST /nl` natural-language->SQL via the AI gateway, returns validated SELECT, `execute=true` runs it read-only). **Auth = the PRIMARY administrator only** (the earliest-created Admin, `utils.is_primary_admin` / `database.get_primary_admin_uid` - the same identity that gates backup downloads) by session/api_key (`services/dbapi/policy.py`); every other administrator is refused like a member, and there is **no internal-key path** - the gateway `internal_gateway_key()` is NOT accepted (no service-to-service access). SQL validated by `services/dbapi/validate.py` (sqlglot classify + suspicious-flag + read-only `EXPLAIN` dry-run). Devii tools `db_*` are read-only (list/get/query/nl; no write tools) and flagged `requires_primary_admin=True`, so they are added to the LLM tool list ONLY for the primary administrator (every other session never sees them and Devii is unaware they exist). `services/pubsub/policy.py` resolves its own admin/internal actor and does NOT reuse `dbapi.policy.caller_for`, so the primary-admin restriction does not leak onto the pub/sub bus.
|
||||
|
||||
It exposes per-table reads, a validated raw `query()`, a natural-language-to-SQL designer backed by the internal AI gateway, and async query execution streamed over a websocket. **It can never insert, update, replace, delete, or restore data in any way** - there are no write endpoints and no write tools (this is a hard rule; removed because the generic write surface bypassed every per-route admin safeguard - role-change seniority guards, container privilege locks, the audit trail). Reuses the existing dataset helpers, the `JobService`/`ProgressHub`/lock-owner websocket pattern, the AI gateway, and the Devii action catalog - it does not re-implement any of them.
|
||||
|
||||
## Auth (single boundary, primary-admin ONLY)
|
||||
|
||||
`services/dbapi/policy.py` `caller_for(request)` returns a `Caller` ONLY for the **primary administrator** (a session OR api_key whose user passes `utils.is_primary_admin` - the earliest-created Admin, the same identity that gates backup downloads, resolved by `database.get_primary_admin_uid`), else `None`. **There is no internal-key path** - the gateway `internal_gateway_key()` is NOT accepted (removed at the user's request); there is no service-to-service access. Every other administrator is refused exactly like a member, so the `Caller.kind` is always `"admin"`. `require_caller` raises `DbApiDenied`; the router's `require_dbapi_caller` turns that into a `403` and an audited `database.access.denied`. Members, guests, non-primary admins, and internal callers never reach a handler. **Note:** `services/pubsub/policy.py` no longer reuses this `caller_for` (it would have leaked the primary-admin restriction into the pub/sub bus); it resolves its own admin/internal actor so any admin stays `privileged` on the bus.
|
||||
|
||||
## Table guard
|
||||
|
||||
`policy.assert_table(name)` enforces a name regex, existence in `db.tables`, and a deny-list (`DEFAULT_DENY = {sessions, password_resets, cache_state}` plus the `dbapi_deny_tables` setting). Used on every table-scoped path and indirectly by the validator's table extraction, so neither a crafted segment nor an NL-designed query can touch a denied table.
|
||||
|
||||
## Reads only (`services/dbapi/crud.py`, `routers/dbapi/crud.py`)
|
||||
|
||||
`GET /dbapi/{table}` (keyset pagination newest-first, `?filter.<col>=`, `?gte/lte/gt/lt.<col>=`, `?search=`, `?before=`, `?limit=`, `?include_deleted=`) and `GET /dbapi/{table}/{key}/{value}`. There are **no insert/update/delete/restore routes** - the router exposes only the two read handlers, and `services/dbapi/crud.py` keeps only read functions (`schema`, `list_rows`, `count_rows`, `get_row`). The only audit events the API emits are `database.access.denied`, `database.query`, and `database.nl.design`.
|
||||
|
||||
## `query()` is hard SELECT-only (`services/dbapi/validate.py`)
|
||||
|
||||
`classify()` parses with `sqlglot` (statement type, table list, `has_where/has_join/has_limit`, suspicious notes for missing `WHERE/JOIN/LIMIT`, multiple statements, `ATTACH/PRAGMA/VACUUM`). `validate_select()` rejects anything but a single SELECT, then `dry_run()` does `EXPLAIN` on a **separate read-only** connection (`file:...?mode=ro` + `PRAGMA query_only=ON`) for true validity. `run_select()` executes read-only. `POST /dbapi/query` returns `409` for non-SELECT (the database API is read-only; data cannot be changed through it), `400` for invalid SQL, else rows + a `suspicious` list. The database API can never mutate data through any path.
|
||||
|
||||
## NL-to-SQL (`services/dbapi/nl2sql.py`)
|
||||
|
||||
`POST /dbapi/nl` builds a system prompt from the table schema + 5 example rows + a soft-delete rule (auto `deleted_at IS NULL` unless `apply_soft_delete=false`), calls the internal gateway (`INTERNAL_GATEWAY_URL`, model `dbapi_nl_model` or `molodetz`, auth = the primary-admin caller's api_key for attribution), and **reprompts with the validator's error until the SQL validates** (max 3 attempts). Returns the validated SQL by default; `execute=true` also runs it read-only and returns rows.
|
||||
|
||||
## Async (`services/dbapi/service.py` `DbApiJobService`, kind `dbquery`)
|
||||
|
||||
`POST /dbapi/query/async` validates then `queue.enqueue`s; the service re-validates (defense in depth), streams rows in batches to its own `ProgressHub`, writes the full result to `config.DBAPI_DIR/{uid}/result.json`, and returns stats. `GET /dbapi/query/{uid}` (status), `GET /dbapi/query/{uid}/result` (rows from disk, extends retention), `WS /dbapi/query/{uid}/ws` (lock-owner gated, `4013` retry, snapshot-then-stream - the SEO pattern). Config on `/admin/services`: `dbapi_max_rows`, `dbapi_nl_model`, `dbapi_nl_system_preamble`, `dbapi_deny_tables`, plus the standard Jobs fields.
|
||||
|
||||
## Devii (primary-admin gated)
|
||||
|
||||
**Read-only** tools `db_list_tables`, `db_table_schema`, `db_list_rows`, `db_get_row`, `db_query` (SELECT-only; surface `suspicious`), and `db_design_query` (NL->SQL), all flagged `requires_primary_admin=True` (alongside `requires_admin=True`) in `actions/catalog.py`. The `Action.requires_primary_admin` flag is filtered by `Catalog.tool_schemas_for(authenticated, is_admin, is_primary_admin)` and enforced again in `Dispatcher.dispatch`, so these tools are **added to the LLM tool list only for the primary administrator** - every other administrator's (and member's/guest's) Devii never receives the schemas and is unaware the database API exists. `is_primary_admin` is threaded WS/Telegram/CLI -> `hub.get_or_create(is_primary_admin=)` -> `DeviiSession` -> `Dispatcher`, mirroring how `is_admin` flows (`routers/devii.py` `_resolve_ws_owner`, `services/telegram/bridge.py`, `services/devii/cli.py` which runs both flags `True` as the trusted local operator, and the headless scheduler in `service.py`). There are **no write tools** - the former `db_insert_row`/`db_update_row`/`db_delete_row` were removed along with the HTTP write routes, so Devii cannot change data through the database API.
|
||||
|
||||
## nginx
|
||||
|
||||
`/dbapi/query/<uid>/ws` has its own upgrade location.
|
||||
153
devplacepy/services/devii/CLAUDE.md
Normal file
153
devplacepy/services/devii/CLAUDE.md
Normal file
@ -0,0 +1,153 @@
|
||||
This file documents the Devii assistant subsystem (devplacepy/services/devii/ and all its subdirectories: virtual_tools, behavior, email, telegram, container, client, customization, tasks, rsearch, agentic, actions). Claude Code loads it automatically whenever any file under devplacepy/services/devii/ is read or edited.
|
||||
|
||||
## Overview and account scope
|
||||
|
||||
`DeviiService` is the in-platform agentic assistant (a relocated standalone agent), exposed as a WebSocket terminal at `/devii/ws` (router `routers/devii.py`, prefix `/devii`) and reachable from the **Devii** user-dropdown item (`data-devii-open`) and auto-opened on `/docs` (`data-devii-autoopen`). Opt-in (`default_enabled=False`). Frontend is `static/js/DeviiTerminal.js` (an `Application.js`-instantiated controller, `app.devii`) plus the `devii-terminal`/`devii-avatar` custom elements and ported renderers under `static/js/devii/`, the vendored md-clippy avatar under `static/vendor/md-clippy/`, and `static/css/devii.css`.
|
||||
|
||||
A signed-in user's Devii drives **their own** account: the agent's `PlatformClient` uses this instance as base URL and the user's `api_key` as Bearer auth, so admins get admin-level tools. The **LLM** calls use the same `api_key` too: the hub threads `owner_kind` into `_build_settings` -> `build_settings(cfg, base_url, api_key, owner_kind)`, which sets `Settings.ai_key = api_key` for `owner_kind == "user"` (else the configured/internal gateway key). So a user's Devii spend is attributed to them at the gateway (`user:<uid>`) and counts toward any per-user gateway limit, while guests fall back to the internal key. Guests get an unauthenticated sandbox client. Every turn is audited in `devii_turns`.
|
||||
|
||||
## Sessions, hub, and channels
|
||||
|
||||
`DeviiHub` keeps one `DeviiSession` per owner-and-**channel** `(owner_kind, owner_id, channel)` - `user`/uid or `guest`/`devii_guest` cookie, channel `main` (default, the floating terminal) or `docs` (the in-page docs chat). A session holds a `set` of WebSockets and **broadcasts** every frame to all of that owner-channel's tabs/devices. After a turn it persists `agent._messages` to `devii_conversations` (users only; rehydrated on reconnect -> survives restarts), appends a `devii_usage_ledger` row, and a `devii_turns` audit row. On `attach` it sends a `{"type":"history"}` snapshot so a new tab renders the prior conversation. Tasks persist in `devii_tasks` (owner-scoped; guests use an in-memory db via `tasks.store.memory_db()`).
|
||||
|
||||
**Channels are an independent conversation thread only.** A `channel` separates the floating terminal (`main`) from the in-page docs chat (`docs`) so each has its own conversation thread for the same owner. The WS reads `?channel=` (`routers/devii.py`, allowlisted to `{"main","docs"}`, falls back to `main`) and threads it `get_or_create(..., channel=)` -> `DeviiSession(channel=)`. **Only** `devii_conversations` is keyed `(owner_kind, owner_id, channel)` (column added + NULL->`main` backfilled + index extended in `init_db`); lessons, behavior, virtual tools, tasks, and the `devii_usage_ledger`/`devii_turns` 24h quota stay owner-scoped (deliberately shared across channels). `find(...)`/`get_or_create(...)` default `channel="main"` so existing callers (e.g. `/devii/adopt`) are unchanged. The `docs` channel is driven only by `<dp-docs-chat>` (`static/js/components/AppDocsChat.js`): a light-DOM component that calls `/devii/session` (guest cookie), opens `DeviiSocket(handlers, "docs")`, renders user/agent bubbles in docs style (`static/css/docs-chat.css`) via the devii markdown renderer, stubs any `avatar`/`client` request so the agent never hangs, and seeds its first question from the `seed` attribute (the sidebar search box, intercepted). It is mounted on `/docs/search.html` (`templates/docs/_chat.html`); the docs pages no longer carry `data-devii-autoopen`. The docs page no longer auto-opens the floating terminal.
|
||||
|
||||
## The Docii docs channel
|
||||
|
||||
The `docs` channel is branded **Docii**, a documentation-only assistant, built by specialising the same `DeviiSession` on `self.channel == "docs"` (no separate service):
|
||||
|
||||
- **Prompt.** `session.py` sets `self._system_prompt = DOCS_SYSTEM_PROMPT` (instead of `_system_prompt_for`) - it forces the agent to call `search_docs` first for every question, to recursively refine-and-search (reading the returned section `content`, then searching again with new keywords) until grounded, to answer ONLY from retrieved docs, and to do no platform actions. `_compose_system_prompt()` returns it verbatim for docs (the owner `BEHAVIOR_HEADER` is NOT appended, so a user's general Devii behavior rules can't derail it).
|
||||
- **Tools.** `_builtin_tools()` filters `CATALOG.tool_schemas_for(...)` to the `DOCS_TOOLS` allowlist (`{"search_docs"}`); `_refresh_tools()` for docs sets `self.tools[:] = _builtin_tools()` with NO virtual tools. So the docs agent has exactly one tool and cannot wander.
|
||||
- **Greeting.** `bootstrap_greeting()` returns `DOCS_GREETING` for docs.
|
||||
- **search_docs.** `services/devii/docs/controller.py` delegates to the in-process page index `docs_search.search_pages(...)` (no HTTP), returning per result `title`, `url` (`/docs/{slug}.html`), `score` (BM25), and `content` (page text truncated to `CONTENT_CHARS`), admin-filtered by the dispatcher's `is_admin`. So "visiting the search results" is repeated `search_docs` calls; the default 40 `max_tool_iterations` leaves ample room to recurse.
|
||||
- **References and inline linking.** Because results carry real `url`s and `score`s, the prompt REQUIRES the answer to (a) link inline to documented pages with markdown links to their `url` and (b) end with a `## References` list of the pages used as clickable links with their score. The devii markdown renderer makes `/docs/...` links clickable client-side.
|
||||
- **Topic gate (follow-up vs new).** `_run_turn` runs `_docs_topic_gate(text)` before the main loop (docs channel only). When there is prior history it calls `_classify_topic` (a no-tools `LLMClient.complete_text` with `TOPIC_CLASSIFIER_PROMPT`, parsed by `_parse_topic`, defaulting to `follow_up` on any failure). On `new` it resets `agent._messages` to `[system]`, clears the persisted docs conversation, and emits `{type:"topic", decision, reason}`; the frontend clears the log, re-shows the current question, and prints a reasoning note. On `follow_up` it keeps context and shows the note.
|
||||
- **Isolation from Devii (critical).** The docs channel is built with an EPHEMERAL `owned_db` (`hub.get_or_create`: `db if owner_kind=="user" and channel=="main" else memory_db()`), so its task/lesson/behavior/virtual-tool stores are private and empty - a Docii reflection never pollutes the user's Devii memory and vice-versa. The **Scheduler only starts in the `main` channel** (`ensure_scheduler_started`: `if not self._started and self.channel == "main"`), so Devii's scheduled tasks never fire inside a docs session. (Regression fixed: previously the docs session's scheduler ran owner tasks over the shared `devii_tasks` table, and the search-only Docii agent tried to fulfil a `run_js` task by spamming `search_docs` - tasks are a `main`-only feature.)
|
||||
- The `main` channel is untouched (full 90-tool catalog + behavior header + Devii greeting).
|
||||
|
||||
## Docs search mode and BM25 fallback
|
||||
|
||||
The docs search surface is admin-configurable via the `docs_search_mode` site setting (`agent` | `bm25`, default `agent`, on `/admin/settings`, seeded in `init_db` `operational_defaults`, field in `AdminSettingsForm`). `routers/docs/views.py` `docs_page` (slug `search`) resolves the effective mode: `_agent_search_state(request, user, is_admin)` returns `disabled` (Devii off, or guests-disabled for a guest), `quota` (viewer over their `daily_limit_for`/`spent_24h`), or `ok`. Agent renders **only** when `mode=="agent"` and state `ok`; otherwise it runs the classic `docs_search.search(...)` (BM25, `docs_search.py`) and renders `templates/docs/_search.html`. `docs_base.html` branches the `kind=="search"` include on `search_mode`; on a quota fallback (`quota_fallback`) the BM25 page shows a "reached your daily AI assistant limit" hint. So `docs_search.py`/`_search.html` are live again (the bm25 path), not orphaned.
|
||||
|
||||
## Conversation, lesson, and task persistence
|
||||
|
||||
Conversation history persists to `devii_conversations` (rehydrated on reconnect, surviving restarts); per-turn cost goes to `devii_usage_ledger` (the authoritative 24h-spend source - the in-memory `CostTracker` is display-only) and an audit row to `devii_turns`; tasks live in `devii_tasks` (guests use an in-memory store).
|
||||
|
||||
**Per-owner self-learning memory is privacy-critical.** The `LessonStore` (reflect/recall) is **owner-scoped, never shared**: `LessonStore(db, owner_kind, owner_id)` filters every read/write by owner. The hub builds one per session over the same `owned_db` as the task store - the main `db` (table `devii_lessons`) for signed-in users (persistent, isolated, survives restarts) and a fresh `memory_db()` for guests (ephemeral, scoped to that web session). `forget_lessons` (agentic tool -> `LessonStore.clear()`/`delete()`) lets the user purge them; the system prompt forbids storing credentials/secrets in lessons. **Regression to avoid: do NOT share one `LessonStore` across sessions** - that leaked one user's reflected lessons (including credentials) into every other user's recall.
|
||||
|
||||
## Reminders and scheduled tasks (persistent across reboot)
|
||||
|
||||
`create_task` queues a self-contained prompt that a fresh agent runs later (`services/devii/tasks/`): `kind=once` (`delay_seconds` for relative, `run_at` UTC for absolute), `interval` (`every_seconds`), or `cron`. The per-session `Scheduler` ticks every 1s and executes due rows through the session's executor, so the result is broadcast to any connected tab and buffered (`type:"task"` frame) when the terminal is closed.
|
||||
|
||||
**The scheduler is no longer tied to a live WebSocket.** It is started by `session.ensure_scheduler_started()`, called both on `attach` AND by the lock-owner `DeviiService._ensure_task_schedulers()` each `run_once` (and promptly at boot): that pass scans the shared `devii_tasks` for every user owner with an enabled `pending`/`running` task (`tasks.store.pending_owner_ids(db)`), resolves the user (api_key/username/is_admin/timezone), and `hub.get_or_create(...)`s a headless session whose scheduler then runs the task - so a queued reminder survives a server reboot and fires even if the user never reopens Devii. `hub.gc_idle()` skips a session with `has_pending_tasks()` so a task-only session is not reaped between ticks.
|
||||
|
||||
A task created as a reminder carries `notify=1`: when it finishes, `session._deliver_reminder` sends a `utils.create_notification(owner, "reminder", result, target_url="/devii")` (the `reminder` notification type), so the in-app notification + live toast reach the user regardless of the terminal.
|
||||
|
||||
**Timezone awareness.** The terminal sends a `clientinfo` frame (`Intl...timeZone` + UTC offset) on connect; `session.set_clientinfo` stores it and persists `users.timezone` (`database.set_user_timezone`). `session._compose_system_prompt()` injects a `# CURRENT TIME` block (UTC now + the user's local time/timezone, falling back to the stored `users.timezone` for a headless run) so the agent converts wall-clock requests to a correct UTC `run_at`; relative requests use `delay_seconds`. The `REMINDERS AND SCHEDULED TASKS` system-prompt rule makes the agent SCHEDULE rather than fire a delayed instruction immediately, and set `notify=true` for reminders.
|
||||
|
||||
## Cost, quota, and the financial-data-is-admin-only rule
|
||||
|
||||
**Financial cap.** The rolling-24h `$` limit is read from `devii_usage_ledger` (`UsageLedger.spent_24h`), **not** the in-memory `CostTracker` (which resets on reboot and is display-only). Checked before each turn in the router; a turn already over is blocked (the over-limit WS error states only "Daily AI quota reached (100%)", never a dollar figure). The caps are `devii_user_daily_usd`, `devii_guest_daily_usd`, and `devii_admin_daily_usd` (default `0` = unlimited, admins exempt by default), resolved via `config.effective_daily_limit`. `config.effective_daily_limit(cfg, owner_kind, is_admin)` is the single source of truth, used by both `service.daily_limit_for` and `build_settings`: guests use `devii_guest_daily_usd` (default $0.05), users `devii_user_daily_usd` ($1.00), and **administrators are exempt by default** via `devii_admin_daily_usd` (default `0.0`, where **`0` = unlimited**). All three are editable on the Devii service config (`/admin/services`). The WS gate is `if limit > 0 and spent >= limit` so a `0` cap never blocks.
|
||||
|
||||
**Every billed turn is ledgered, not just interactive ones:** `session._record_spend` writes the usage delta in the `finally` of both `_run_turn` AND the scheduler executor, and it runs even when a turn is cancelled/superseded, so scheduled-task spend and interrupted-turn spend count against the cap rather than being forgiven. Pricing is admin-configurable - `cost/tracker.py` resolves a `Pricing` object per `CostTracker` rather than import-time env constants.
|
||||
|
||||
**Quotas are resettable** (clearing the owner's `devii_usage_ledger` rows): per-user via `POST /admin/users/{uid}/reset-ai-quota` (button on the admin Users page), globally via `POST /admin/ai-quota/reset-guests` and `/reset-all` (buttons on `/admin/ai-usage`), and from the CLI via `devplace devii reset-quota <username> | --guests | --all`.
|
||||
|
||||
**Financial data is admin-only:** any monetary figure (USD cost, pricing, spend, limit) is restricted to administrators; members and guests see only the **percentage** of their 24h quota used. The owner's admin status is resolved server-side (`is_admin(user)`) and threaded WS -> `hub.get_or_create(is_admin=)` -> `DeviiSession(is_admin=)` -> `Dispatcher(is_admin=)`. The `Action` dataclass has a `requires_admin` flag (`cost_stats` is admin-only USD; the member-safe `usage_quota` tool returns **only** `{used_pct, turns_today, limit_reached}` with no money and is available to everyone). Gating is double: `Catalog.tool_schemas_for(authenticated, is_admin)` never hands an admin-only tool's schema to a non-admin, and the dispatcher independently raises `AuthRequiredError` for any `requires_admin` action a non-admin attempts. `usage_quota`'s data comes from `DeviiSession._quota_snapshot` (ledger `spent_24h`/`turns_24h` over `settings.daily_limit_usd`); for non-admin owners the system prompt also appends a hard rule forbidding any cost disclosure (defense in depth). `GET /devii/usage` mirrors this: it always returns `used_pct`/`turns_today` and adds `spent_24h`/`limit` only for admins. The standalone `devii` CLI runs with `is_admin=True` (the local operator owns the process). The catalog's cost/analytics HTTP tools `ai_usage` (GET `/admin/ai-usage/data`, USD breakdown) and `site_analytics` (GET `/admin/analytics`) are also `requires_admin=True`, so a non-admin session is never offered them and the dispatcher blocks them even if named - the platform 403 is no longer the only guard. The profile page is correctly split too (`_ai_quota(include_cost=viewer_is_admin)` emits dollars only for admins, in both its HTML and JSON forms, so a member fetching their own profile with `Accept: application/json` never sees dollars either).
|
||||
|
||||
## Aggregate analytics (no pagination)
|
||||
|
||||
Site analytics is a **documented, admin-secured HTTP endpoint** - `GET /admin/analytics` (`routers/admin/` package, returns JSON, `is_admin` check -> 403 otherwise) backed by `database.get_platform_analytics()` (single UNION query over `posts/comments/gists/projects.created_at`, TTL-cached 60s; total members, active users 24h/7d/30d, signed-in-now, signups, content totals, top authors). Devii calls it as a normal `http` catalog action `site_analytics` (GET `/admin/analytics`, `top_n` query param) - the platform enforces admin, so it behaves identically for the web and the `devii` CLI and never touches the DB directly from the agent. It is documented in the Admin API docs group (`docs_api.py`, id `admin-analytics`). The system prompt tells the model to use it for any "how many"/"how active" question instead of paging `admin_list_users` (which caused a pagination storm), and to `search_docs` before guessing a route rather than probing URLs.
|
||||
|
||||
## Context-window sizing (DeepSeek V4 Flash, 1M tokens)
|
||||
|
||||
The upstream model `deepseek-v4-flash` (what `deepseek-chat` routes to; default `gateway_model`) has a 1,048,576-token context and 384,000-token max output. All Devii size limits are **characters**, derived in `services/devii/config.py` from one source of truth: `CONTEXT_INPUT_BUDGET_TOKENS = CONTEXT_WINDOW_TOKENS - MAX_OUTPUT_TOKENS - SYSTEM_RESERVE_TOKENS` (600,576 tokens), and `DEFAULT_CONTEXT_COMPACT_THRESHOLD = CONTEXT_INPUT_BUDGET_TOKENS * CHARS_PER_TOKEN` (3 chars/token = ~1.8M chars). At the conservative 3-chars/token estimate the worst-case input at compaction plus the full max output plus the system reserve sums to exactly the 1M window, so it can never overflow. `DEFAULT_MAX_RESPONSE_CHARS` (200,000) is the master chunk knob: every non-`chunks` tool result passes through `wrap_if_large(result, max_response_chars)` in `actions/dispatcher.py`, and `read_more` slices are clamped to it, so a file up to ~200KB returns in **one** read instead of paging in 12KB slices (the old read_more storm). `OUTPUT_CAP_CHARS` (`agentic/loop.py`, 400,000) must stay **above** `max_response_chars` or it would re-truncate a full chunk envelope. `ChunkStore` caches up to `STORE_MAX_CHARS` (8MB) per entry, `STORE_MAX_ENTRIES` (16) entries; `SUMMARY_INPUT_CAP` (`agentic/compaction.py`, 600,000) bounds what the summarizer ingests when compacting. When changing the upstream model, retune from `CONTEXT_WINDOW_TOKENS`/`MAX_OUTPUT_TOKENS` - everything else derives.
|
||||
|
||||
## Multi-worker service-lock routing
|
||||
|
||||
Hubs are per-process, so `/devii/ws` is served **only** by the worker that holds the background-service lock (`service_manager.owns_lock()`, set in `main.py` startup); a non-owner worker `close(4013)` (an application code in the private 4000-4999 range, reliably delivered as the browser `CloseEvent.code`). `DeviiSocket.js` recognises 4013 as "wrong worker, not yet settled" and fast-retries in ~200ms **silently** (no "disconnected, reconnecting..." notice), so with 2 prod workers the client converges on the owner in a fraction of a second instead of bouncing every 1500ms. The visible "connected." notice is emitted on the first server frame (`onReady`), never on raw socket open, so an accepted-then-4013-closed handshake on the wrong worker is invisible. The disabled/no-service path keeps the standard `1013` (a real, user-visible disconnect). The cap stays correct regardless because it reads the DB.
|
||||
|
||||
## Client-side browser-automation tool channel
|
||||
|
||||
Beyond the avatar, Devii has a `client`/browser channel using the same request/response pattern: `services/devii/client/ClientController` is bound on `attach`, and `actions/client_actions.py` exposes `get_page_context`, `run_js`, `highlight_element`, `clear_highlights`, `show_toast`, `scroll_to_element`, `navigate_to`, `reload_page` (`handler="client"`, all `requires_auth=False`). The session's generic `_browser_request(channel, action, args)` powers both `avatar` and `client`; the router resolves both `avatar_result` and `client_result`. Frontend `static/js/devii/DeviiClient.js` executes the actions in the user's own browser and the terminal routes `type:"client"` frames to it, returning `client_result`. This powers live on-screen tutorials, page-context awareness, and navigation/refresh. `navigate_to`/`reload_page` reply *before* unloading so the turn completes, then the persistent session reconnects.
|
||||
|
||||
**Target selection (visibility-aware).** Unlike replies/traces, which `_emit` *broadcasts* to every tab, a browser request is sent to one **target** chosen by `_pick_target()`. The terminal reports each connection's `document` visibility and focus via `{"type":"visibility"}` (on connect, `focus`, `blur`, and `visibilitychange`); the session stores it in `_conn_meta` and ranks connections `(focused, visible, attach_seq)`, so the command runs on the tab the user is actually looking at, falling back to the most recently attached when none reports focus. **Regression to avoid: do NOT route to a single "last-attached primary" socket** - with the `/docs` auto-open tab or any second tab, `reload_page`/`navigate_to` ran on the wrong (hidden) tab while still reporting success, so "the reload did not happen" from the user's view. `run_js` is gated by the `devii_allow_eval` config field (default on), checked in `ClientController` before any round-trip. Overlays (`.devii-hl-box`, `.devii-hl-callout`, `.devii-toast`) attach to `document.body`, not the terminal.
|
||||
|
||||
## Shared browser session (auth adoption)
|
||||
|
||||
The terminal and the browser are one session. `/devii/ws` resolves its owner from the browser's `session` cookie (`_resolve_ws_owner`), so the terminal is whoever the browser is logged in as. Agent-initiated auth propagates: the session watches its own trace stream (`_trace`) and, on a successful `login`/`signup`, captures the real session token the `PlatformClient` minted against this instance (`session_cookie()`) and broadcasts `{"type":"auth","action":"adopt"}`; the browser navigates to single-use `GET /devii/adopt` which sets the httpOnly `session` cookie and redirects (reload). On `logout` it broadcasts `{"type":"auth","action":"logout"}` and the browser goes to `/auth/logout`. Browser->terminal: login/logout are navigations that reconnect the WS; a cross-tab change is caught by a focus check against `/devii/session` that reloads. The httpOnly cookie is only ever set/cleared by HTTP endpoints, never JS.
|
||||
|
||||
## Screen awareness
|
||||
|
||||
The `SYSTEM_PROMPT` instructs Devii, after a mutation, to `get_page_context` and `reload_page` when the page the user is viewing displays the data just changed, so settings/list/detail views the user is watching update live without a manual refresh.
|
||||
|
||||
## Project filesystem tools: line-range editing and the overwrite-protection guard
|
||||
|
||||
Devii's project filesystem tools include surgical **line-range editing** (`project_read_lines`, `project_replace_lines`, `project_insert_lines`, `project_delete_lines`, `project_append_file`) so it edits large text files without resending them, and `Dispatcher._run_http` enforces an **overwrite-protection guard**: `project_write_file` is rejected only when the target `(slug, path)` **already exists** (the guard probes `GET /projects/{slug}/files/raw`) and the agent has not read it this session (call `project_read_file` first); creating a new file needs no prior read. Reads are recorded in the per-session `Dispatcher._read_files` set. This steers the agent to update files rather than blindly overwrite them and is agent-scoped only (the human UI and public HTTP API are unaffected). The `WRITING AND EDITING FILES` system-prompt rule mirrors it.
|
||||
|
||||
## Web fetch and arbitrary HTTP (`handler="fetch"`)
|
||||
|
||||
`services/devii/fetch/FetchController` (a standalone `httpx.AsyncClient`, not `PlatformClient`) backs two tools: `fetch_url` (read a page as readable text, `requires_auth=False`) and `http_request` (`requires_auth=True`) - the general external HTTP client. `http_request` takes `method` (GET/POST/PUT/PATCH/DELETE/HEAD/OPTIONS), `headers` (object), and one body of `json` (object/array, Content-Type set automatically), `form` (url-encoded), or `body` (raw string), and returns `{http_status, headers, content_type, content}`; it does **not** raise on non-2xx so the model can read API errors. Both run through one `_stream()` (`fetch_url` via `_download` with `raise_5xx=True`; `http_request` with `raise_5xx=False`) sharing the SSRF `_guard` (refuses private/loopback unless `DEVII_FETCH_ALLOW_PRIVATE`), `fetch_max_bytes` cap, timeout, and decoding. The `object`-typed params required adding an `object` branch to `Action.tool_schema()` in `spec.py` (emits a valid open-object schema). The `SYSTEM_PROMPT` "WEB REQUESTS AND EXTERNAL APIS" rule tells the agent to call `http_request` DIRECTLY for an API and never wrap it in a self-invoking user-defined tool (a virtual tool only re-runs the agent and cannot do I/O, so wrapping the call recurses into the `MAX_EVAL_DEPTH` guard - the exact failure this fixed). To attach a remote file see `attach_url` / `store_attachment_from_url` under Attachments and media.
|
||||
|
||||
## Remote web tools (rsearch, non-platform)
|
||||
|
||||
`services/devii/rsearch/RsearchController` (`handler="rsearch"`, registered in `registry.py`, specs in `actions/rsearch_actions.py`, all `requires_auth=False`) exposes four tools that hit the EXTERNAL public service `https://rsearch.app.molodetz.nl` over a standalone `httpx.AsyncClient` (like `FetchController`, not the platform-bound `PlatformClient`): `rsearch` (web/image search, optional `content`/`deep`/`type=images`), `rsearch_answer` (AI answer grounded in fresh web search, returns answer + sources), `rsearch_chat` (direct AI chat, no search), `rsearch_describe_image` (vision describe a public image URL). These are NOT platform-specific: the `SYSTEM_PROMPT` "REMOTE WEB TOOLS" section makes the model prefer platform tools always and call `rsearch_*` only when the user explicitly asks to search the web/an outside source, never to answer questions about this instance. When adding another remote service, add a controller + handler literal in `spec.py` + a dispatcher branch rather than routing external calls through `PlatformClient` (which is hardwired to the instance base URL).
|
||||
|
||||
rsearch has its **own** read timeout (`rsearch_timeout_seconds`, field `devii_rsearch_timeout` / env `DEVII_RSEARCH_TIMEOUT`, default **300s** with a 30s connect cap via `httpx.Timeout`) - it does NOT share `fetch_timeout_seconds`, because web-grounded answers legitimately take minutes.
|
||||
|
||||
**rsearch is metered into the gateway ledger.** Because these calls never traverse the gateway upstream, `RsearchController` (constructed with `owner_kind`/`owner_id` from the dispatcher) records one `gateway_usage_ledger` row per call inside its single choke point `_request()` via `GatewayUsageLedger.record_external(...)` (backend `rsearch`, zero tokens, the flat admin-set `gateway_rsearch_cost_per_call`, success and failure both logged). The DeepSearch worker's `crawl.search_queries` emits a `{"type":"rsearch"}` NDJSON frame per search that `DeepsearchService._run_worker` ledgers in-process under the job owner. This keeps `gateway_usage_ledger` (and `/admin/ai-usage`) a complete record of platform AI spend - external AI calls included. `record_external` is the helper to reuse for any future off-gateway AI call.
|
||||
|
||||
## Email tools (IMAP/SMTP, non-platform)
|
||||
|
||||
A signed-in user's Devii can connect to their OWN external mailbox via `services/devii/email/EmailController` (`handler="email"`, specs in `actions/email_actions.py`). See `devplacepy/services/email/CLAUDE.md` for the full protocol engine, credential storage, and tool list.
|
||||
|
||||
## Telegram bot
|
||||
|
||||
Devii is reachable over Telegram via `channel="telegram"` sessions driven in-process by `services/telegram/bridge.py`. See `devplacepy/services/telegram/CLAUDE.md` for the worker/bridge architecture, pairing flow, and message-update behavior.
|
||||
|
||||
## Database API tools (`db_*`, primary-administrator only)
|
||||
|
||||
Devii exposes read-only `db_list_tables`, `db_table_schema`, `db_list_rows`, `db_get_row`, `db_query`, and `db_design_query` tools gated `requires_primary_admin=True`. See `devplacepy/services/dbapi/CLAUDE.md` for the full auth boundary, validation pipeline, and async query service - these tools are added to the LLM tool list only for the primary administrator; every other session is unaware they exist.
|
||||
|
||||
## All Devii/gateway network timeouts are admin-configurable with a five-minute (300s) floor
|
||||
|
||||
The Devii LLM-client and `PlatformClient` read timeout is `timeout_seconds` (field `devii_timeout`), the fetch/docs tool timeout is `fetch_timeout_seconds` (field `devii_fetch_timeout`), web search is `rsearch_timeout_seconds` (`devii_rsearch_timeout`); each defaults to 300s with `minimum=MIN_TIMEOUT_SECONDS` (300). The gateway upstream timeout is `gateway_timeout` (`minimum=TIMEOUT_MIN`=300), which also bounds the vision describe-image call (it shares the gateway's httpx client, no per-call override). A stored value below the floor fails `ConfigField.coerce()` and `read()` falls back to the default, so old sub-300 values upgrade automatically. Devii's LLM timeout was previously a hardcoded 45s while the gateway waited up to 180s x retries - large prompts aborted as `[model error] Could not reach the model endpoint:` (an httpx timeout, which stringifies to empty). `build_settings()` now reads these from config instead of hardcoding them.
|
||||
|
||||
## Config
|
||||
|
||||
Config is all `config_fields` (AI url/model/key, base url, plan/verify toggles, max iterations, JS-execution toggle, user/guest 24h caps, guests toggle, pricing). `effective_config()` falls back the AI key to `DEVII_AI_KEY` and the base url to the instance origin.
|
||||
|
||||
## Backend confidentiality
|
||||
|
||||
Devii must never disclose the underlying model, provider, or any upstream URL - enforced in two layers. The `SYSTEM_PROMPT` (`agent.py`) forbids it even when such values appear inside a tool result. Structurally, `text.redact_backend()` runs on every JSON HTTP tool response in `format_response()` and scrubs the values of `REDACT_FIELD_KEYS` (`gateway_upstream_url`/`gateway_model`/`gateway_vision_url`/`gateway_vision_model`) and any stat labelled in `REDACT_STAT_LABELS` (`Model`) to `[hidden]`, so the admin-services tools cannot leak the gateway's upstream even though the admin settings UI still shows the real values. Add a config key to `REDACT_FIELD_KEYS` if a new field would expose backend infrastructure to the agent.
|
||||
|
||||
## CLI
|
||||
|
||||
`pyproject.toml [project.scripts]` ships `devii = "devplacepy.services.devii.cli:main"`. No new dependency (`httpx`/`dataset` already required). The md-clippy avatar is vendored under `static/vendor/md-clippy/`; its `index.js` must not set `globalThis.app` (it would clobber the DevPlace `app`) and its AI proxy is `/devii/clippy/ai/chat`. The standalone `devii` CLI runs with `is_admin=True` (the local operator owns the process) and both `is_admin`/`is_primary_admin` `True` for the trusted local operator.
|
||||
|
||||
## Devii user-defined ("virtual") tools
|
||||
|
||||
Users invent new Devii tools in natural language ("when I say woeii, do Y"); each is stored per-owner and added to Devii's live LLM tool list, and when called its handler **re-prompts Devii itself** (a self-eval sub-agent) with the stored prompt plus the user's single free-form `input`. Full CRUD is Devii-only (`tool_create`/`tool_list`/`tool_get`/`tool_update`/`tool_delete`, `handler="virtual_tool"`).
|
||||
|
||||
- **Dynamic tool list (the crux).** The session tool list is normally frozen (`session.py` `self.tools = CATALOG.tool_schemas_for(...)`, handed by reference to `Agent`, `AgenticController.bind`, and read live by `react_loop` each `llm.complete`). `DeviiSession._refresh_tools()` runs at the top of every `_run_turn` (inside `self._lock`, before `agent.respond`) and rewrites that list **in place**: `self.tools[:] = CATALOG.tool_schemas_for(self.client.authenticated, self.is_admin) + self._virtual_tool_store.tool_schemas()`. Because every holder shares the same list object, a tool created/edited/deleted (and a mid-session login) takes effect on the **next** turn with no Agent rebuild. **Never reassign `self.tools`, always mutate in place.**
|
||||
- **Self-eval engine** (`agentic/controller.py`). `_delegate` was refactored onto `_spawn(prompt, tools, system_prompt)`, which runs `react_loop` with a fresh `messages`/`AgentState` under a **depth guard** (`agentic/state.py` `get/set/reset_eval_depth`, `MAX_EVAL_DEPTH=2`) so delegate/eval/virtual tools cannot self-call infinitely. `run_subagent(prompt)` (tools minus `delegate`) is the public engine; the `eval` agentic tool wraps it; `_delegate` keeps its richer JSON report.
|
||||
- **Store** (`services/devii/virtual_tools/store.py`). `VirtualToolStore(db, owner_kind, owner_id)` over `devii_virtual_tools` (`uid, owner_kind, owner_id, name, description, prompt, input_description, enabled, created_at, updated_at`; indexes `idx_devii_vtools_owner` / `idx_devii_vtools_name`). `tool_schemas()` builds, for each **enabled** row, an LLM schema with a single free-form `input` string. Persistent for users, `memory_db()` for guests (built in `hub.get_or_create` from the shared `owned_db`).
|
||||
- **Controller** (`virtual_tools/controller.py`). `VirtualToolController(store, evaluator, builtin_names)`, evaluator = `agentic.run_subagent`, `builtin_names = set(CATALOG.by_name())`. CRUD `tool_create/list/get/update/delete` (`handler="virtual_tool"`); `tool_create` validates name `^[a-zA-Z][a-zA-Z0-9_]{1,40}$`, rejects built-in collisions and duplicates, requires `description`+`prompt`. `has(name)` / `run(name, args)` compose `f"{prompt}\n\nUser input: {input}"` and call the evaluator.
|
||||
- **Dispatch** (`actions/dispatcher.py`). Constructed with `virtual_tools=...`; `_run` routes `handler="virtual_tool"` to CRUD; the **unknown-name branch** (`if action is None`) resolves a virtual tool via `self._virtual_tools.has(name)` -> `run(name, args)` before erroring. So virtual tools live only in the store (not the static catalog) and are resolved on demand; the static `_actions` stays built-in only.
|
||||
- Registered via `VIRTUAL_TOOL_ACTIONS` (`registry.py`); the `eval` tool via `AGENTIC_ACTIONS`. System-prompt **USER-DEFINED TOOLS (VIBE TOOLS)** section steers creation and warns against deep self-calls.
|
||||
|
||||
## Devii self-configured behavior (`services/devii/behavior/`)
|
||||
|
||||
Devii can partially configure its OWN system message: every system prompt ends with a `# TRUTH RULES AND BEHAVIOR` section (the `BEHAVIOR_HEADER` constant in `session.py`) whose body is an owner-scoped, persistent set of behavior rules. When the user tells Devii to behave differently, says they expect different behavior, or Devii upsets them, Devii calls the **`update_behavior`** tool (`handler="behavior"`, `requires_auth=False`, single `behavior` arg = the FULL new section content) to persist the change. It is **self-prompted only**: the sole way to change the section is Devii calling that tool - there is no HTTP route or UI, like `LessonStore`/customization. Because Devii already SEES the current section in its own system message, it merges (copy current rules, apply the change, send the whole result) so prior rules are not lost; this is why the tool replaces rather than appends.
|
||||
|
||||
- **Store** (`behavior/store.py`). `BehaviorStore(db, owner_kind, owner_id)` over `devii_behavior` (one upserted row per owner keyed on `owner_kind`/`owner_id`; `text()` reads, `set()` upserts; index `idx_devii_behavior_owner`). Persistent for users, `memory_db()` for guests (built in `hub.get_or_create` from the shared `owned_db`, like the other owner stores).
|
||||
- **Controller** (`behavior/controller.py`). `BehaviorController(store)`, `dispatch("update_behavior", args)` -> `store.set(behavior)`. Built in `DeviiSession` and passed to `Dispatcher(behavior=...)`; the dispatcher routes `handler="behavior"` to it and degrades gracefully ("not available in this context") when unwired (e.g. the standalone CLI, same as `virtual_tools`/`avatar`).
|
||||
- **Injection and refresh** (`session.py`). `_compose_system_prompt()` = base prompt (`_system_prompt_for(is_admin)`) + `\n\n` + `BEHAVIOR_HEADER` (+ `\n` + body when non-empty; just the header when empty). Used to seed the `Agent` and the scheduler executor worker, and `_refresh_system_prompt()` rewrites `agent._messages[0]["content"]` **at the top of every `_run_turn`** (next to `_refresh_tools`), so a mid-conversation `update_behavior` takes effect on the following turn and the live system message is never overridden by the stale base.
|
||||
- Registered via `BEHAVIOR_ACTIONS` (`registry.py`); the system-prompt **SELF-CONFIGURED BEHAVIOR (TRUTH RULES)** section in `agent.py` steers when/how to call it.
|
||||
|
||||
## Related Devii tool wrappers (customization, container)
|
||||
|
||||
Two more Devii-side controllers live under `services/devii/` but their full mechanism is documented in their owning subsystem's file, not here:
|
||||
|
||||
- **Per-user customization tools** (`services/devii/customization/`, `handler="customization"`): `customize_list`/`get`/`set_css`/`set_js`/`reset` (all `requires_auth=False`) plus `customize_set_enabled` (`requires_auth=True`, the suppression toggles). `CustomizationController(owner_kind, owner_id)` is built in `Dispatcher.__init__`; set/reset are in `CONFIRM_REQUIRED` so `confirmation_error` forces Devii to ask **page-type vs global** before `confirm=true`. Devii previews live with `run_js` (ids `devii-preview-css`/`devii-preview-js`) and `reload_page` after saving. For the storage model, injection pipeline, and per-user suppression flags, see `devplacepy/customization.py` and its own documentation.
|
||||
- **Container tools** (`services/devii/container/ContainerController`, `handler="container"`): wraps `services/containers/api.py`, the same operations shared by `routers/containers.py`. For the reconciler, backend ABC, ingress proxy, and workspace sync mechanism, see `devplacepy/services/containers/CLAUDE.md`.
|
||||
32
devplacepy/services/email/CLAUDE.md
Normal file
32
devplacepy/services/email/CLAUDE.md
Normal file
@ -0,0 +1,32 @@
|
||||
This file documents the email (IMAP/SMTP) subsystem. Claude Code auto-loads it when a file under `devplacepy/services/email/` is read or edited.
|
||||
|
||||
## Email via Devii (`services/email/` + `services/devii/email/`)
|
||||
|
||||
A signed-in user's Devii can connect to their OWN external mailbox over IMAP/SMTP (non-platform, external protocol tools).
|
||||
|
||||
### Protocol engine and adapter
|
||||
|
||||
The protocol engine `services/email/` (`EmailAccount` dataclass + `EmailClient`) is built over the Python **stdlib** `imaplib`/`smtplib`/`email` - no new dependency, since IMAP/SMTP are not HTTP the stealth-client rule does not apply. It is wrapped by the Devii adapter `services/devii/email/EmailController` (`handler="email"`, registered in `registry.py`, specs in `actions/email_actions.py`, routed in `dispatcher.py`). Every blocking protocol call runs through `asyncio.to_thread` in the controller so the single worker loop never stalls (same discipline as `correction.py`).
|
||||
|
||||
### Tools
|
||||
|
||||
- Connection CRUD: `email_account_set`/`email_account_get`/`email_accounts_list`/`email_account_delete`.
|
||||
- Message ops: `email_list_folders`, `email_list_messages`, `email_search`, `email_read_message` (read); `email_mark`/`email_set_flags`/`email_move_message` (organise); `email_delete_message`; `email_send`.
|
||||
|
||||
All are `requires_auth=True` AND additionally guarded to `owner_kind == "user"` (guests never reach email), gated by the admin `devii_email_enabled` flag (default on) / `devii_email_timeout` on the Devii service config ("Email" service-config group).
|
||||
|
||||
### Credentials
|
||||
|
||||
Credentials live in the soft-deletable per-owner `email_accounts` table (one row per `(owner_kind, owner_id, label)`; helpers `list/get/set/delete_email_account` in `database/`, sensible defaults imap 993/ssl + smtp 587/starttls applied in `set_email_account`). The password is stored **plaintext** (consistent with `api_key`/`gateway_api_key`) and is **NEVER echoed back** - `email_account_get`/`email_accounts_list` mask it as `password_set`.
|
||||
|
||||
### Configuration surface
|
||||
|
||||
Configuration is **Devii-only** (no HTTP route or profile UI), like the CSS/JS customization feature.
|
||||
|
||||
### Destructive actions and SSRF guard
|
||||
|
||||
`email_send` is a real outbound action, and the two delete tools (`email_account_delete`, `email_delete_message`) are confirmation-gated (`CONFIRM_REQUIRED`, each declaring a `confirm` param). The IMAP/SMTP host is run through `net_guard.guard_public_host_sync` before every connect (private/loopback addresses refused - SSRF/internal-scan defense, mirroring `guard_public_url`).
|
||||
|
||||
### Audit and metering
|
||||
|
||||
Audited via the existing `_audit_mechanic` choke point under the `email.*` domain (category `email`). Mutations are **not metered into the gateway ledger** (no LLM call is made).
|
||||
57
devplacepy/services/game/CLAUDE.md
Normal file
57
devplacepy/services/game/CLAUDE.md
Normal file
@ -0,0 +1,57 @@
|
||||
# Code Farm game (`devplacepy/services/game/`, `devplacepy/routers/game/`)
|
||||
|
||||
This file documents the Code Farm idle game. Claude Code auto-loads it when a file under `devplacepy/services/game/` is read or edited.
|
||||
|
||||
## Overview
|
||||
|
||||
`game/` package (routers, mounted at `/game`) is the **Code Farm** idle game (`index.py` + `farm.py`): `GET /game` (page), `GET /game/state`, `GET /game/leaderboard`, `POST /game/{plant,harvest,buy-plot,upgrade,fertilize,daily,perk,prestige,legacy,quests/claim}`, plus social `GET /game/farm/{username}` and `POST /game/farm/{username}/{water,steal}`.
|
||||
|
||||
A cooperative-and-competitive idle game (Farmville-style) mounted at `/game`, member-only to play, public to view another farm. Cooperative loop = watering a neighbour's growing build; competitive loop = stealing a neighbour's ready build.
|
||||
|
||||
## Data layer
|
||||
|
||||
**Data layer is pure + timestamp-driven (no background tick).** `services/game/economy.py` holds every constant and formula as frozen dataclasses/functions: the `CROPS` tuple (key/name/icon/cost/grow_seconds/reward_coins/reward_xp/min_level), `CI_TIERS` (speed multiplier + upgrade cost), level thresholds (`xp_threshold`/`level_for_xp`/`level_progress`), `plot_cost` (doubles per extra plot), and watering bonus. `services/game/store.py` is the only DB access (`game_farms`, `game_plots`). **A plot's state is derived from `ready_at` vs now, never stored** - growing crops finish purely by the clock, so there is no reconciler/service. `serialize_farm(farm, viewer=, owner=)` computes plot states, remaining seconds, `can_water` (viewer is not owner, build growing, viewer not already in the per-cycle `watered_by` JSON list, under `MAX_WATERS_PER_PLOT`), `can_steal`/`steal_coins` (viewer is not owner, build ready, and `now >= ready_at + STEAL_GRACE_SECONDS` so the owner gets a protection window), level progress, and the plantable crop list. `serialize_plot` takes the owner farm's `yield_level`/`prestige` (threaded from `serialize_farm`) only to compute the steal payout. Mutations (`plant`/`harvest`/`buy_plot`/`upgrade_ci`/`water`/`steal`) raise `GameError` on any invalid op (insufficient coins, locked crop, wrong state, still-protected harvest); the routers translate that to a `400` JSON error or a redirect.
|
||||
|
||||
## Tables
|
||||
|
||||
`game_farms` (one per user, `coins`/`xp`/`level`/`ci_tier`/`plot_count`/`total_harvests`, plus `prestige`/`streak`/`last_daily_at`, the four `perk_*` columns, and the endgame `stars` + five `legacy_*` columns), `game_plots` (`farm_uid`/`slot_index`/`crop_key`/`planted_at`/`ready_at`/`watered_by`), and `game_steals` (`thief_uid`/`owner_uid`/`slot_index`/`crop_key`/`coins`/`stolen_at`, the per-pair steal-cooldown ledger) are **not** soft-deletable - they are mutable game state consumed by state transitions, not user content. Columns + indexes are ensured in `database.init_db` (unique `idx_game_farms_user`, `idx_game_farms_rank`, `idx_game_plots_farm`, `idx_game_steals_pair` on `(thief_uid, owner_uid, stolen_at)`); the `_uid_index` loop adds the uid index. `ensure_farm(user_uid)` lazily creates a farm + starting plots on first access (idempotent), so there is no signup hook.
|
||||
|
||||
## Routes (`routers/game/`)
|
||||
|
||||
`index.py` is the base router - `GET /game` (own farm page), `GET /game/state` (own farm JSON), `GET /game/leaderboard`, and the action POSTs `plant`/`harvest`/`buy-plot`/`upgrade`. `farm.py` adds `GET /game/farm/{username}` (view), `POST /game/farm/{username}/water`, and `POST /game/farm/{username}/steal`. Every action returns the **full updated farm** as JSON (so the client refreshes in one round trip) or redirects for no-JS, via the shared `_respond_action` choke (the `farm.py` water/steal handlers inline the same shape). Harvest awards site XP (`award_rewards`) and the `harvest`/`water` achievements (`track_action`). All action handlers are `require_user`; reads of other farms/the leaderboard are public.
|
||||
|
||||
**Leaderboard ranking is a composite `economy.farm_score(farm)`** (one integer per row, computed in memory over the already-loaded `_farms().find()` set, so it stays fast): it sums weighted contributions from every tracked factor - `xp`, `prestige` * `SCORE_PRESTIGE` (5000, dominant since a refactor is a full completed cycle), `total_harvests` * `SCORE_HARVEST`, `coins` // `SCORE_COIN_DIVISOR`, `(ci_tier-1)` * `SCORE_CI`, `(plot_count-STARTING_PLOTS)` * `SCORE_PLOT`, the summed perk levels * `SCORE_PERK`, and `min(streak, SCORE_STREAK_CAP)` * `SCORE_STREAK` - so a player who refactored (which resets xp/level/coins/ci/plots/perks) is no longer buried below a never-refactored higher-level player. The weights are module-level constants in `economy.py` for tuning; the leaderboard entry carries `score` and `prestige` (surfaced on `GameLeaderboardEntryOut`) and `GameFarm.js` renders the score next to `Lv X`.
|
||||
|
||||
## Live + frontend
|
||||
|
||||
After any mutation the handler `await`s `_shared.notify_farm(username)` which publishes a nudge to the `public.game.farm.{username}` pub/sub topic; subscribed clients re-fetch their viewer-specific state (keeps `can_water` correct without broadcasting per-viewer payloads). `static/js/GameFarm.js` (`app.gameFarm`, auto-detects `[data-game-root]`) renders the grid/HUD/leaderboard, ticks plot countdowns client-side every second, delegates the `data-game-action` forms through `Http.send`, subscribes to the farm topic, and keeps a 20s `Poller` fallback. The server renders a full no-JS fallback grid (`templates/_game_grid.html`, shared by `game.html` and `game_farm.html` with progressive-enhancement POST forms).
|
||||
|
||||
## Fan-out
|
||||
|
||||
Devii plays via the `game_*` `http` actions in `actions/catalog.py` (state/leaderboard/view public, the rest `requires_auth`); the API reference has a **Code Farm** group in `docs_api.py`; pages are `noindex,follow` (interactive, user-specific) so they are intentionally not in the sitemap; badges live in `utils.BADGE_CATALOG` under the **Code Farm** group with `harvest`/`water`/`harvest_stolen`/`got_stolen_from` `ACHIEVEMENTS` (the steal pair awards **Cat Burglar** to the thief and **Robbed** to the victim, both threshold 1).
|
||||
|
||||
## Stealing (competitive loop, backwards compatible)
|
||||
|
||||
`store.steal(thief, owner, slot)` mirrors `water`: it requires the owner plot to be `ready` AND past the protection window `economy.effective_steal_grace(owner_defense_level)` (base `STEAL_GRACE_SECONDS` 60s, +30s per owner Branch Protection level) measured from `ready_at`, AND that the thief is **off cooldown for this victim** (`steal_cooldown_remaining(thief, owner, now) == 0`, i.e. no row in `game_steals` for the pair within `STEAL_COOLDOWN_SECONDS` = 3600 - you can raid a given neighbour only once per hour); it clears the plot exactly like a harvest (owner gets nothing), credits the **thief's** farm `economy.steal_reward_coins` (the owner's yield/prestige/legacy-multiplier realized value times `effective_steal_fraction(owner_defense_level)`, base `STEAL_FRACTION` 0.5, -5% per defense level, floored at 0.1) and **coins only** - no XP, no `total_harvests`, so the leaderboard stays earned by real farming - then inserts a `game_steals` row stamping the cooldown. The route `POST /game/farm/{username}/steal` (reusing `GameSlotForm`) then `track_action`s both sides, fires `create_notification(owner, "harvest_stolen", "Someone raided your Code Farm...", thief_uid, "/game")` (the message never names the thief; `related_uid` is internal), and `await notify_farm`s **both** the owner and the thief usernames so both farms refresh live. The new `harvest_stolen` notification type rides the existing in-app relay (live toast, no new wiring). No DB migration: grace + payout are computed from existing `ready_at`/perk/legacy columns, the `game_steals` table is created by the ensure-block (absent rows -> cooldown 0 -> first steal always allowed), and the new `GamePlotOut.can_steal`/`steal_coins`/`steal_cooldown_seconds`/`steal_reason` + `GameFarmOut.steal_cooldown_seconds` + `GameFarmViewOut.stole_coins` schema fields default safe. The per-pair cooldown is computed **once per farm** in `serialize_farm` (when the viewer is not the owner) and threaded into every `serialize_plot` as `steal_locked_until`, so `can_steal` is false and `steal_reason` is `"cooldown"`/`"protected"` accordingly. Frontend: the ready/not-owner branch of `_game_grid.html` and `GameFarm.js._plotHtml` render a `btn-danger` Steal button (`data-game-action="steal"`, `data-confirm` gated like prestige) carrying `steal_coins`, or a disabled "Raid again in <countdown>" label when on cooldown; on success `GameFarm.js._submit` toasts the payout from `data.stole_coins`.
|
||||
|
||||
## Extended mechanics (all backwards compatible)
|
||||
|
||||
Five further systems layer onto the base loop, every one defaulting gracefully for pre-existing `game_farms`/`game_plots` rows: new `game_farms` columns (`prestige`, `streak`, `last_daily_at`, `perk_yield`/`perk_growth`/`perk_discount`/`perk_xp`) are added in the `init_db` ensure-block, and `store._lvl(farm, key)` reads every one as `int(farm.get(key) or 0)` so a legacy NULL row behaves as level 0 / no streak / no perks.
|
||||
|
||||
1. **Daily bonus** (`POST /game/daily`, `store.claim_daily`): once per UTC day, consecutive days grow `streak` (reward `economy.daily_reward`, capped at `DAILY_STREAK_CAP`).
|
||||
2. **Daily quests** (`game_quests` table, `POST /game/quests/claim`): `economy.daily_quests(user_uid, day)` deterministically (sha256 of `user:day`) picks 3 of {plant, harvest, water, earn} with goals/rewards; `store.ensure_quests` lazily materializes the day's rows, `store.advance_quests(user_uid, kind, amount)` is called inside `plant`/`harvest`/`water` (wrapped in try/except so a quest write never breaks the action), `claim_quest` pays out when `progress >= goal`.
|
||||
3. **Perks** (`POST /game/perk`, `store.upgrade_perk`): four permanent upgrades (`economy.PERKS`) with escalating `perk_cost`; applied in the economy formulas - `effective_plant_cost` (discount), `grow_seconds_for(crop, ci_tier, growth_level)` via `farm_speed` (growth), `effective_reward_coins` (yield + prestige), `effective_reward_xp` (xp).
|
||||
4. **Fertilizer** (`POST /game/fertilize`, `store.fertilize`): cut a growing plot's `ready_at` by `FERTILIZE_FRACTION` for `economy.fertilize_click_cost(eff_reward, reduce_seconds, full_grow_seconds)` = `ceil(eff_reward * reduce/full_grow * FERTILIZE_TAX)` (TAX 1.05). **The cost is priced against the build's realized harvest value (`effective_reward_coins`, which already carries yield/prestige/legacy multipliers), not raw grow-seconds, so the prestige dependence cancels and fully fertilizing a crop always costs >= its harvest - fertilize is a pure time-skip and can NEVER be a profit at any prestige.** (This replaced the old grow-seconds-based `fertilize_cost`, which was an unbounded money pump at high prestige.)
|
||||
5. **Prestige/Refactor** (`POST /game/prestige`, `store.prestige`): at `PRESTIGE_MIN_LEVEL` resets coins/xp/level/ci/perks, sets `plot_count = economy.prestige_base_plots(legacy_plots_level)` (keeps/recreates plot rows up to that base, deletes the rest), increments `prestige` for a permanent `prestige_multiplier` (+25% coins each), and awards `economy.stars_for_refactor(level, prestige)` Stars (the `legacy_*`/`stars` columns are **omitted from the reset dict** so they survive every refactor, the established pattern).
|
||||
|
||||
`serialize_farm` exposes all of this (`perks`, `quests`, `streak`, `daily_available`/`daily_reward`, `prestige*`, `stars`, `legacy`, `steal_cooldown_seconds`) only to the owner; the existing `crop_payload`/`serialize_plot` now carry perk- and legacy-adjusted costs/rewards and per-plot value-based `fertilize_cost`. Frontend hosts (`[data-shop-host]`/`[data-perk-host]`/`[data-legacy-host]`/`[data-daily-host]`/`[data-quest-host]`) are server-rendered from partials (`_game_shop.html`/`_game_perks.html`/`_game_legacy.html`/`_game_daily.html`/`_game_quests.html`) and fully re-rendered by `GameFarm.js` builders; the generic `data-game-action` form delegation handles the new actions, with `data-confirm` gating the prestige reset.
|
||||
|
||||
## Endgame: Stars, Legacy upgrades, auto-harvest, golden builds (all backwards compatible)
|
||||
|
||||
The infinite progression for maxed farms. Six new default-0 `game_farms` columns (`stars`, `legacy_autoharvest`, `legacy_multiplier`, `legacy_speed`, `legacy_plots`, `legacy_defense`), all read via `_lvl`.
|
||||
|
||||
**Stars** are a meta-currency earned only on Refactor (`stars_for_refactor`), spent via `POST /game/legacy` (`store.upgrade_legacy`, `GameLegacyForm`, Devii `game_upgrade_legacy`) on `economy.LEGACY_UPGRADES` (escalating `legacy_cost` in Stars) that **survive prestige** unlike perks: `autoharvest` (CI Bot), `multiplier` (+10% coins/lvl via `legacy_multiplier`, folded into `effective_reward_coins`), `speed` (+5% base build speed/lvl via `farm_speed`/`grow_seconds_for`/`water_bonus_seconds`), `plots` (+1 base plot after refactor via `prestige_base_plots`), `defense` (steal grace/fraction via `effective_steal_grace`/`effective_steal_fraction`).
|
||||
|
||||
**Auto-harvest** is lazy and tick-free: `store._auto_harvest(farm, owner_uid, now)` runs at the top of `serialize_farm` ONLY when the viewer is the owner AND `legacy_autoharvest > 0`; it **clears each ready plot first then credits** the pre-clear crop's coins/xp/`total_harvests` in one `_update_farm`, advances quests, and re-reads the farm - clear-then-credit is idempotent (a second read sees empty plots), and it never fires on a visitor's `GET /game/farm/{username}` view or the leaderboard, honouring the no-background-service invariant (both `GET /game` and `GET /game/state` go through `state_payload` with viewer=owner, so both auto-collect).
|
||||
|
||||
**Golden builds** are deterministic and storage-free: `economy.is_golden(plot_uid, planted_at)` (sha256, ~`GOLDEN_CHANCE`) marks a planting golden for its life; `harvest`/`_auto_harvest` multiply **coins only** by `GOLDEN_MULTIPLIER` (XP unscaled to keep level pacing), and `serialize_plot.is_golden` surfaces a sparkle badge (`.game-plot-golden`). New schema fields (`GameLegacyOut`, `GameFarmOut.stars`/`legacy`, `GamePlotOut.is_golden`) all default safe; no DB migration.
|
||||
48
devplacepy/services/gitea/CLAUDE.md
Normal file
48
devplacepy/services/gitea/CLAUDE.md
Normal file
@ -0,0 +1,48 @@
|
||||
This file documents the Gitea issue-tracker integration subsystem. Claude Code auto-loads it when a file under `devplacepy/services/gitea/` is read or edited.
|
||||
|
||||
## Issue tracker (`services/gitea/`)
|
||||
|
||||
The `/issues` feature is a **full Gitea integration with no local issue store** - there is no local issue store; the listing and detail views read issues straight from Gitea (default repo `retoor/devplacepy` on `retoor.molodetz.nl`) with live status. All connection settings live in `site_settings` and are admin-editable on the `IssueTrackerService` config at `/admin/services`.
|
||||
|
||||
### Package `services/gitea/`
|
||||
|
||||
- `config.py` - the admin-editable `CONFIG_FIELDS` (Gitea base URL/owner/repo/token, AI enhance toggle/model/key) plus `gitea_config()` -> frozen `GiteaConfig` (with `api_base`/`repo_base`/`is_configured`) and `is_configured()`. The token is one shared bot account; per-user attribution is done by storing names in DevPlace, not by per-user Gitea accounts.
|
||||
- `client.py` - async `GiteaClient` over the Gitea REST API (`Authorization: token`): `create_issue`, `list_issues(state,page,limit)` returning `(issues, total)` from `X-Total-Count` and filtering out pull requests, `get_issue`, `list_comments`, `create_comment`, `set_state`. Raises `GiteaError(message, status)` on any non-2xx or transport error.
|
||||
- `fake.py` - in-memory `FakeGiteaClient` mirroring the same async interface (plus `add_external_comment` to simulate a developer reply); the test backend/double.
|
||||
- `runtime.py` - `get_client()` / `set_client()` swap (tests inject the fake). `get_client()` builds a fresh `GiteaClient` from current config unless overridden.
|
||||
- `enhance.py` - `enhance_ticket(title, description, config)` rewrites a raw report into a consistent markdown ticket via the **internal AI gateway** (`INTERNAL_GATEWAY_URL`, key defaults to the gateway internal key): a level-2-heading markdown body (`Summary` / `Steps to Reproduce` / `Expected Behaviour` / `Actual Behaviour` / `Environment`) and a tightened title. It is fail-soft to a deterministic template: any error or unparseable response falls back to a deterministic template built from the original text (`EnhancedTicket.enhanced=False`). This is the AI attachment point - the gateway, like every other AI consumer.
|
||||
- `store.py` - the **only DB state**, a thin local mapping: `issue_tickets` (gitea_number -> author_uid + original text + cached `last_status`/`last_comment_count` for update detection) and `issue_comment_authors` (gitea_comment_id -> author_uid, for display attribution and to tell DevPlace-authored comments from developer replies). Helpers: `record_ticket`, `get_ticket`, `author_uid_for_issue`, `author_map`, `record_comment_author`, `comment_author_map`, `local_comment_ids`, `tracked_tickets`, `update_ticket_cache`. Both tables are in `SOFT_DELETE_TABLES`; `init_db` ensures their full column set + indexes (so queries are safe on a fresh DB before any insert).
|
||||
- `service.py` - `IssueTrackerService(BaseService)` (`default_enabled=False`, holds `CONFIG_FIELDS`), polls tracked tickets and notifies the **reporter** on developer replies / status changes. Each tick it polls every tracked ticket: when the Gitea comment count grew, it loads comments and notifies the reporter if any of the new tail comments are **not** in `local_comment_ids` (a developer reply); when the state changed it notifies the reporter (closed/reopened). It then updates the cache. This is the single source of update notifications to the author (`issue.sync.reply` / `issue.sync.status`).
|
||||
|
||||
### Filing (async job)
|
||||
|
||||
Filing is an **async job** (`services/jobs/issue_create_service.py` `IssueCreateService`, kind `issue_create`): `process` loads the user, enhances the report, appends a `Reported by **user** via DevPlace` footer, creates the Gitea issue, records the local mapping, notifies the reporter (`Your issue report was filed as #N`), and audits `issue.create`. `cleanup` is a no-op (the issue is permanent; only the job row is swept). `POST /issues/create` enqueues and returns `{uid, status_url}`; frontend `app.issueReporter` (`static/js/IssueReporter.js`) submits the create form, polls `/issues/jobs/{uid}` via `JobPoller`, and redirects to `/issues/{number}`.
|
||||
|
||||
### Routes (`routers/issues/` package, prefix `/issues`)
|
||||
|
||||
- `GET /issues` - Gitea list, `?state=open|closed|all&page=`, admin-style `build_pagination` + `_pagination.html` with a `state=` prefix; renders an empty-state notice when unconfigured/unreachable.
|
||||
- `POST /issues/create` - enqueue (see Filing above).
|
||||
- `GET /issues/jobs/{uid}` - `IssueJobOut`.
|
||||
- `GET /issues/{number}` - `issue_detail.html`: issue body + comments rendered as markdown via `data-render`, decorated with DevPlace authors or a `dev` badge.
|
||||
- `POST /issues/{number}/comment` - synchronous: posts the comment to Gitea attributed to the user, records the author mapping, notifies **all admins**, audits `issue.comment`.
|
||||
- `POST /issues/{number}/status` - admin-only open/closed via Gitea PATCH, audits `issue.status`; non-admins get a `denied` audit + 403.
|
||||
|
||||
Comments and status changes are deliberately synchronous (one fast Gitea call, immediate redirect); only the AI-heavy create is a job. Status-change notifications to the author are NOT emitted by the route - the poller detects the diff and notifies, so the admin (or external developer) closing an issue both flow through one path.
|
||||
|
||||
### Content rendering and mentions (frontend reuse, no new modules)
|
||||
|
||||
The issue body and every comment render through the platform content pipeline like posts/comments - each is a `<div class="...rendered-content" data-render>{{ body|comment.body }}</div>` (`issue_detail.html`), so the globally-instantiated `ContentEnhancer` (`app.content`) runs `contentRenderer.applyTo` (marked -> DOMPurify -> highlight.js -> media/autolink) over Gitea-sourced markdown client-side from `element.textContent`. Gitea content is **NEVER** marked `| safe` into raw HTML; Jinja autoescapes it and the DOMPurify step inside `ContentRenderer` is the XSS control, identical to user posts. The issue authoring textareas (`#issue-description` in the create modal on `issues.html`, and the comment textarea on `issue_detail.html`) carry the `data-mention` attribute, so the same `ContentEnhancer.initMentionInputs()` -> `MentionInput` flow that wires post `@`-mention autocomplete (debounced `GET /profile/search?q=` dropdown) attaches to them with no new code. There is no local issue-edit form (issues are AI-created then commented), so "edit" reuses nothing further. `mention.css` is loaded globally in `base.html`.
|
||||
|
||||
### Attachments (mirror: local store + Gitea assets, `routers/issues/attachments.py`)
|
||||
|
||||
Issues and issue comments accept file attachments, with full add/delete CRUD allowed **only while the issue is open**. Storage is a mirror - the canonical copy is the platform attachment system (`attachments` table, keyed `target_type="issue"`/`target_uid=str(number)` and `target_type="issue_comment"`/`target_uid=str(comment_id)`), and each file is also pushed to the Gitea native asset API so it appears on the tracker. The local row carries a `gitea_asset_id` back-reference (added to the `init_db` attachments ensure-block + born-live in `store_attachment`) so a later delete removes both copies; `attachments.mirror_attachment_to_gitea(uid)` / `remove_gitea_asset(row)` branch on `target_type` to call `GiteaClient.create_issue_asset`/`create_comment_asset` / `delete_issue_asset`/`delete_comment_asset` (all best-effort - the local copy and the user action survive any Gitea failure). Upload reuses the canonical **orphan-then-link** path: `dp-upload` stores to `/uploads/upload`, the form carries `attachment_uids`, and the handler `link_attachments(uids, target_type, target_uid)` then mirrors.
|
||||
|
||||
**Role/state enforcement:** `require_user` blocks guests; add validates each uid is the caller's own unlinked orphan (admins any); delete is owner-or-admin on `attachments.user_uid`; every mutation re-checks the live Gitea `state == "open"` (closed -> 409) and audits denied branches. Attach-at-creation: `IssueForm.attachment_uids` rides into the `issue_create` job, which links + mirrors once the issue number exists; `IssueCommentForm.attachment_uids` links + mirrors inline after the comment is created (synchronous comment path).
|
||||
|
||||
Routes: `GET/POST /issues/{number}/attachments`, `DELETE /issues/{number}/attachments/{uid}`, and the `/issues/{number}/comments/{cid}/attachments[...]` comment variants. Frontend `app.issueAttachments` (`static/js/IssueAttachments.js`) submits the add form (`form[data-issue-attach]` -> POST `attachment_uids` -> reload) and handles per-attachment delete (`[data-attachment-delete]` -> confirm -> DELETE). The shared render partial is `_issue_attachments.html` (gallery + per-item delete gated by `can_modify`).
|
||||
|
||||
Note: for Gitea to accept every file type the instance `app.ini` `[attachment] ALLOWED_TYPES` must be `*/*` (infra, out of app scope); the DevPlace allow-list is the `allowed_file_types` setting, and the local copy is kept even when Gitea rejects a type. Audit events `issue.attachment.add`/`issue.attachment.delete` (category `content`). Devii tools `list_issue_attachments`, `add_issue_attachment`, `delete_issue_attachment` (+ `add_comment_attachment`/`delete_comment_attachment`); the two deletes are in `CONFIRM_REQUIRED`.
|
||||
|
||||
### Schemas, forms, and tools
|
||||
|
||||
Schemas: `IssueItemOut`/`IssueCommentOut`/`IssueDetailOut`/`IssueAttachmentsOut`/`IssuesOut`/`IssueJobOut` (`schemas.py`). Forms: `IssueForm`/`IssueCommentForm`/`IssueAttachmentForm`/`IssueStatusForm` (`models.py`). Devii tools: `list_issues`, `create_issue`, `view_issue`, `comment_issue`, `set_issue_status` (admin), plus the attachment tools above. Docs: the `issues` API group in `docs_api.py`. Footer link in `base.html` under `.site-footer`.
|
||||
127
devplacepy/services/jobs/CLAUDE.md
Normal file
127
devplacepy/services/jobs/CLAUDE.md
Normal file
@ -0,0 +1,127 @@
|
||||
This file documents the async job services subsystem (devplacepy/services/jobs/ and its subdirectories deepsearch/, isslop/, seo/) - the standard pattern for running heavy blocking work off the request path. Claude Code loads it automatically whenever a file under this directory is read or edited.
|
||||
|
||||
## Overview: the JobService pattern
|
||||
|
||||
The standard way to run heavy, blocking work off the request path and hand back a result URL, used by every job kind in this subsystem (`zip`, `planning`, `fork`, `seo`, `seo_meta`, `deepsearch`, `isslop`). Reference: admin docs `Architecture -> Async job services` and `Services -> ZipService`.
|
||||
|
||||
- **The DB row is the queue.** `services/jobs/queue.py` (`enqueue`, `get_job`, `touch_job`, `list_jobs`) is pure DB and callable from any worker's handler - enqueue is a fast, non-blocking insert, status reads work from any worker, and only the lock owner processes. All kinds share ONE `jobs` table discriminated by a `kind` column: common lifecycle timestamps, `retry_count`, `last_accessed_at`/`expires_at`, `bytes_in`/`bytes_out`/`item_count` stat columns, plus `payload`/`result` JSON columns where kind-specific fields live.
|
||||
- **`JobService(BaseService)`** runs only in the lock owner. Each `run_once`: reap finished in-flight tasks -> recover orphaned `running` rows (left by a dead owner, since there is only one processor) back to `pending` with bounded `retry_count` -> refill up to `max_concurrent` oldest `pending` jobs (uuid7 sorts FIFO) as `asyncio` tasks -> sweep rows past `expires_at` via the per-kind `cleanup()` hook. In-flight tasks span ticks (kept in an instance map); the loop returns immediately each tick, so keep the interval short (default 2s).
|
||||
- **No atomic claim is needed** (single processor by construction) and **no separate reaper exists** - retention is built into every job service, default 7 days, admin-configurable; downloading/reading a result extends `expires_at` via `touch_job`.
|
||||
- **Add a kind:** subclass `JobService`, set `kind`, implement `async process(self, job) -> dict` (return the `result` dict incl. stat keys) and `cleanup(self, job)`; register the service instance in `main.py`; add enqueue endpoints that own authz, a status route, a download/report route, a Devii tool, and docs. **A new `JobService` needs a server restart to go live.**
|
||||
- **Permanent vs expiring artifacts (load-bearing distinction - decide this per kind).** Most kinds produce a DISPOSABLE artifact that expires with the retention sweep: `ZipService`, `SeoService`, and `DeepsearchService` all delete their real output (archive / report+screenshots / vector collection+report) inside `cleanup()`. Two kinds are different: `ForkService`'s artifact is a **permanent project**, and the AI Usage Analyzer's (`isslop`) artifacts (analysis, events, file/image results, report, badge) are **permanent public capability URLs** - both make `cleanup()` a no-op on the real artifact and let the retention sweep remove only the `jobs` tracking row. Get this decision right for any new kind: if the output should outlive the job the way a fork or an isslop report does, do not wire `cleanup()` to delete it.
|
||||
|
||||
## ZipService (kind `zip`)
|
||||
|
||||
- `process` materializes a project subtree via `project_files.export_to_dir` (staging under `config.DATA_DIR/zip_staging/{uid}`), compresses it in a **subprocess** (`python -m devplacepy.services.jobs.zip_worker`, stdlib-only, prints stats JSON incl. crc32), names the output `{crc32}.{slug}.zip` under `config.DATA_DIR/zips/{uid[-2:]}/{uid[-4:-2]}/` (sharded on the random tail of the uuid7 via `attachments._directory_for`, NOT its time-ordered head), and removes staging.
|
||||
- Runtime artifacts live in `DATA_DIR` (`data/` by default), OUTSIDE the package and NOT under `/static`; the download is served by the `/zips/{uid}/download` route via `FileResponse`, not the static mount.
|
||||
- Enqueue endpoints own authz: `POST /projects/{slug}/zip`, `POST /projects/{slug}/files/zip?path=`.
|
||||
- `GET /zips/{uid}` (status `ZipJobOut`) and `GET /zips/{uid}/download` (FileResponse, extends expiry via `touch_job`) are **capability URLs** scoped only by the unguessable uuid7, not by owner - the owner is stored for attribution, not access control, matching publicly-viewable projects.
|
||||
- Frontend `app.zipDownloader` (`static/js/ZipDownloader.js`) auto-wires any `data-zip-download` element plus the files context menu: POST -> poll `/zips/{uid}` via the shared `JobPoller` -> trigger download.
|
||||
- Devii tools `zip_project`/`zip_status`; CLI `devplace zips prune|clear`.
|
||||
- Disposable: `cleanup()` deletes the archive and the retention sweep prunes both the file and the `jobs` row.
|
||||
|
||||
## Planning report generator (kind `planning`, `services/jobs/planning_service.py`)
|
||||
|
||||
`PlanningReportService` is **admin-only** and builds a complete, phased markdown implementation document from a selectable set of open Gitea tickets, intended to be handed straight to a coding agent for one-shot execution, off the request path via the same async-job pattern as zip.
|
||||
|
||||
- `process` collects open tickets via `services/gitea/planning.py` `collect_open_issues(client, limit=MAX_ISSUES)` (the paginated `list_issues(state="open", limit=50)` loop, cap 50; reused by the admin page too), narrows them to the **selected ticket numbers** carried in the job payload (`{"numbers": [int, ...]}`, preserving selection order - an empty/absent list keeps all open, so the no-arg Devii action and any legacy enqueue stay backward compatible), and builds the markdown via `generate_plan(issues, config)` (an AI-or-fallback helper mirroring `enhance.py`).
|
||||
- **AI path:** passes each ticket's FULL, verbatim description (`_body`, capped only at the `BODY_MAX=40000` per-ticket safety limit, not the old 600-char excerpt) to the internal gateway via `stealth.stealth_async_client` with a high output budget (`MAX_TOKENS=32000`, `PLAN_MAX=600000` final cap, `PLANNING_TIMEOUT_SECONDS=600` client timeout) and a `SYSTEM_PROMPT` that demands a self-contained document: a `## Execution Order` list, then `## Phase K: <name>` headings, and under each phase a `### #N <title>` subsection per ticket carrying labelled **Original ticket** (verbatim blockquote), **Goal**, **Dependencies**, **Affected areas / files**, **Implementation steps**, **Acceptance criteria**, and **Risks / open questions** blocks - preserving every detail with nothing summarised away.
|
||||
- **Because an LLM can still summarise or truncate, the AI document is never trusted to be complete on its own:** `generate_plan` always appends a deterministic `# Appendix: Source Tickets (verbatim)` section built straight from the issue dicts by `planning.py` `verbatim_tickets(issues)` (per ticket `## #N <title>`, labels, the `html_url` source link, and the full body as a blockquote), so every covered ticket's full text is guaranteed present inline AND in the appendix - the document is genuinely self-contained.
|
||||
- Fail-soft `_fallback` (when `ai_enhance` is off or any `httpx.HTTPError`/`ValueError`/`KeyError`/`IndexError` occurs) groups by primary label then ticket number, emitting the same `## Execution Order` + `## Phase K` shape with each ticket's full description reproduced verbatim (already self-contained, so no extra appendix).
|
||||
- Writes the markdown to `config.PLANNING_REPORTS_DIR/_directory_for(uid)/{crc32}.{slug}.md` (sharded on the uuid7 random tail, crc32 over the markdown bytes), and returns `{download_url, local_path, final_name, markdown, ai_used, item_count, bytes_out}`. The `markdown` string is carried in the result so the status route can hand it to the renderer without re-reading disk. `cleanup` unlinks the file (retention prunes only the artifact + job row).
|
||||
- Audit `issue.planning.request` (enqueue) and `issue.planning.generate` (success/failure) via `record`/`record_system` with an `audit.job(uid)` link.
|
||||
- Routes (all `require_admin`): `POST /issues/planning` (enqueue, `503` when Gitea unconfigured, `PlanningForm` with a comma-separated `numbers` field parsed to `list[int]` and stored in the payload, returns `{uid, status_url}`), `GET /issues/planning/{uid}` (`PlanningJobOut`: status, `markdown`, `download_url`, `issue_count`, `ai_used`), `GET /issues/planning/{uid}/download` (`FileResponse`, `text/markdown`, traversal-guarded against `PLANNING_REPORTS_DIR`, extends retention via `touch_job`). The `/issues/{number}` detail route uses the `{number:int}` path converter so `/issues/planning` is never captured by it.
|
||||
- Admin entry point: `GET /admin/issues/planning` (`routers/admin/issues.py`, `admin_issues_planning.html`) fetches the open tickets server-side via `collect_open_issues` (wrapped so a Gitea/network failure renders an empty state, never a 500) and renders a **selectable checkbox list** (all checked by default, a select-all/none master, and a live selected count) as the step in between, plus the **Generate planning** button, the `JobPoller` status panel, the `<dp-content>` render target, and a **Download** anchor; the issues listing shows an admin-only **Generate planning** link (`viewer_is_admin` in `issues_page`).
|
||||
- Frontend driver `static/js/PlanningGenerator.js` (`app.planningGenerator`): tracks the checkbox selection (master toggle, count, disables Generate at zero selected), sends the chosen numbers as the `numbers` form field, POST -> `JobPoller.run` (widened to `intervalMs:2000, maxAttempts:400` = ~800s so it outlives the longer detailed generation) -> replace the `<dp-content>` with a fresh one holding `status.markdown` so it re-renders, set the download href.
|
||||
- Devii tool `planning_report_generate` (`requires_admin=True`). The renderer's built-in copy button (the `dp-content` **Copy** button styled in `markdown.css`, identical to the per-code-block copy button) satisfies the copy-source requirement and is opt-out via the `no-copy` boolean attribute on `<dp-content>`.
|
||||
|
||||
## Project fork - ForkService (kind `fork`, `services/jobs/fork_service.py`)
|
||||
|
||||
Forking copies a source project into a brand-new project owned by the forking user, off the request path via the same async-job pattern as the zip flow.
|
||||
|
||||
- **Enqueue:** `POST /projects/{slug}/fork` (`routers/projects/index.py`) - `require_user` then `can_view_project(source, user)` (any logged-in user may fork any project they can view: public projects and their own private ones). Body is `ForkForm{title}` (the destination project name). It enqueues a `fork` job (`{source_project_uid, title, forked_by_uid}`, owner `("user", uid)`) and returns `{uid, status_url}`. No work happens on the request path.
|
||||
- **`process`** looks up the source project row and the forking user row (raises -> job `failed` if either is missing), creates the destination project with `content.create_content_item("projects", "project", user, fields, ...)` (so slug/XP/logging match a normal create; metadata - `description`, `project_type`, `platforms`, `status`, dates, `is_private` - is copied from the source, `read_only` reset to 0), copies the whole virtual FS off-thread (`export_to_dir(source, "", staging)` -> `import_from_dir(new_uid, staging, user, skip_names=set())`, an **exact** copy including binaries, staging under `config.DATA_DIR/fork_staging/{uid}`), then records the relation with `database.record_fork(source_uid, new_uid, forked_by_uid)`. Result: `{project_uid, project_url, source_project_uid, item_count}`.
|
||||
- **Rollback:** any failure after the project is created triggers `_rollback(new_uid)` (`project_files.delete_all_project_files` + `delete_fork_relations` + delete the `projects` row) before re-raising, so a failed fork leaves no orphan project.
|
||||
- **Cleanup is a no-op on the project.** Unlike zip's disposable archive, a fork's artifact is a **permanent project** that must outlive the job. `cleanup()` only removes any leftover staging dir; the retention sweep deletes the job tracking row, never the forked project. `devplace forks prune|clear` likewise deletes job rows only.
|
||||
- **Relation model:** the `project_forks` table (`uid`, `source_project_uid`, `forked_project_uid`, `forked_by_uid`, `created_at`) records direction explicitly (source -> forked), indexed on both project columns. Helpers in `database.py`: `record_fork`, `get_fork_parent(forked_uid)` (the source project row, for the "Forked from X" link on the project detail page), `count_forks(source_uid)`, and `delete_fork_relations(project_uid)` (called on project delete in `content.delete_content_item`, and in fork rollback).
|
||||
- **Frontend:** `app.projectForker` (`static/js/ProjectForker.js`) wires `data-fork-project` (a Fork button on the project detail page, visible to any logged-in user): prompt for a name via `app.dialog.prompt` -> `Http.sendForm` POST -> `JobPoller.run("/forks/{uid}", ...)` -> on `done` redirect to `project_url`. The forked project's detail page shows a "Forked from X" link (`database.get_fork_parent`).
|
||||
- **Status route** `GET /forks/{uid}` (`routers/forks.py`, `ForkJobOut`) exposes `project_uid`/`project_url`/`source_project_uid` only once `status == done`. Devii tools `fork_project`/`fork_status`; docs `projects-fork`/`forks-status`.
|
||||
|
||||
## SEO Diagnostics tool - SeoService (kind `seo`, `services/jobs/seo/`, `routers/tools/`)
|
||||
|
||||
The public **Tools -> SEO Diagnostics** auditor crawls a URL or sitemap with a headless browser and runs a broad battery of SEO checks, on the **same async-job pattern as zip/fork** plus a live websocket. It is **public (guests included)**; abuse is bounded by the per-IP POST rate limit, a per-owner one-active-job cap, a page cap, and the shared SSRF guard.
|
||||
|
||||
- **Surface:** a collapsible **Tools** dropdown in `base.html` (desktop center nav + a mobile section, visible to everyone) toggled by `MobileNav.initToolsDropdown`. `GET /tools` lists tools; `GET /tools/seo` is the auditor page (`static/js/SeoDiagnostics.js` -> `app.seoDiagnostics`, instantiated page-side in the template, not in `Application.js`).
|
||||
- **Enqueue:** `POST /tools/seo/run` (`routers/tools/seo.py`, body `SeoRunForm{url, mode: url|sitemap, max_pages 1-50}`). Owner is `("user", uid)` or `("guest", X-Real-IP)`. It rejects with `429` if the owner already has a pending/running `seo` job, then enqueues `{url, mode, max_pages, allow_private:False}` and returns `{uid, status_url, ws_url}`.
|
||||
- **`process`** writes the payload to `config.SEO_REPORTS_DIR/{uid}/payload.json`, launches `python -m devplacepy.services.jobs.seo.worker <payload_json> <output_dir>` via `create_subprocess_exec` (high `limit=` so big lines never overflow the StreamReader), reads **NDJSON frames from stdout** line by line (stage/target/progress/page/site_checks/report_ready), forwards each into the in-process **`ProgressHub`** (`services/jobs/seo/progress.py`, uid -> set of `asyncio.Queue`), and on completion loads `output_dir/report.json` as the job result. `cleanup()` clears the hub buffer and removes the report dir.
|
||||
- **Worker** (`worker.py`, subprocess): `crawler.crawl_target` resolves the target (single URL, or sitemap `<loc>` URLs capped at `max_pages`) and fetches `robots.txt`/`sitemap.xml`/`llms.txt` with `httpx`. The audited target host is guarded once with `net_guard.guard_public_url`; candidate URLs **sharing that host are pre-approved** (no redundant per-URL `getaddrinfo` - a transient DNS failure or a self-hosted server resolving its own domain must not blank the whole crawl), and only cross-host sitemap entries are re-guarded. **In sitemap mode the crawler never falls back to auditing the sitemap document itself**: if no page URLs survive it raises a clear error (a stray `or [target]` fallback previously rendered the sitemap XML as one 56k-node "page" with no title/H1). For each page it launches one Playwright navigation: a single `page.evaluate(EXTRACT_SCRIPT)` returns the whole DOM contract (title/metas/canonical/headings/images/links/jsonld/og/twitter/semantic/mixed-content/word-count), an injected `PerformanceObserver` (`add_init_script(INIT_SCRIPT)`) captures LCP/CLS, navigation timing gives TTFB/FCP/transfer/protocol, a mobile-viewport pass measures overflow/tap-targets, a screenshot is saved, and a raw `httpx` GET supplies the SSR HTML for the rendered-vs-server parity check.
|
||||
- **Check registry** (`checks/`): `base.py` defines the `Check`/`PageContext`/`SiteContext` dataclasses and the `@page_check`/`@site_check` decorators (collected into `PAGE_CHECKS`/`SITE_CHECKS`); one module per category (`crawl`, `meta`, `headings`, `links`, `structured_data`, `social`, `performance`, `mobile_a11y`, `security`, `ai_readiness`, `crosspage`). `registry.run_page_checks`/`run_site_checks` run them defensively (one failing check never aborts a page), and `compute_score` produces a severity-weighted overall score + grade and per-category subscores. **To add a check:** write a function decorated `@page_check`/`@site_check` in the right category module and import that module in `registry.py`.
|
||||
- **Live progress WS:** `WS /tools/seo/{uid}/ws` is served **only by the service-lock owner** (closes `4013` for a fast retry on a non-owner worker, like `/devii/ws`); it replays `hub.snapshot(uid)` then streams `hub.subscribe(uid)`. The frontend `SeoProgressSocket` mirrors the `DeviiSocket` 4013/reconnect pattern. **The `done` frame carries the full report inline** (and the router's late-join terminal branch reads it from `job.result`); the client renders from `frame.report` directly. This is load-bearing: `service.process` publishes `done` from inside `process()`, BEFORE the JobService base persists the result on the next reap tick (~2s later), so a client that fetched `/tools/seo/{uid}/report` on the `done` signal would race the DB write and get an empty (all-zero) report. **Do not "simplify" this back into a fetch-on-done.** After publishing `done`, `process()` calls `hub.clear(uid)` to bound the in-memory buffer (late reconnects fall back to the DB-backed terminal branch).
|
||||
- **Status/report routes:** `GET /tools/seo/{uid}` (`SeoJobOut`), `GET /tools/seo/{uid}/report` (`respond(..., SeoReportOut)`, HTML or JSON), `GET /tools/seo/{uid}/screenshot/{n}` (FileResponse from `SEO_REPORTS_DIR`, path-guarded). All are **capability URLs** scoped by the unguessable uuid7. The full report is written to `config.SEO_REPORTS_DIR/{uid}/report.json` (NOT inlined on a stdout line - avoids the StreamReader limit). Devii tools `seo_diagnostics`/`seo_status`/`seo_report` (public); docs `tools-seo-*`; CLI `devplace seo prune|clear`; audit `seo.run.request|complete|failed` (category `tools`).
|
||||
- **SSRF guard is shared:** `devplacepy/net_guard.py` (`guard_public_url`, `is_blocked_address`, `effective_address`) was extracted from the Devii fetch controller, which now imports it; the crawler reuses it. **`playwright` is a core dependency** (Chromium installed by `make install` / the Docker image).
|
||||
- Disposable: `cleanup()` removes the report dir; retention prunes both the artifacts and the `jobs` row.
|
||||
- **Production nginx needs a dedicated WS location.** In `nginx/nginx.conf.template` the catch-all `location /` sets `Connection ""` (no upgrade) and a 60s timeout, so any websocket route that falls through to it fails the handshake (browser: `WebSocket connection failed`, no close code). The progress socket has its own `location ~ ^/tools/seo/[^/]+/ws$` block forwarding `Upgrade`/`Connection` with a 1h timeout, mirroring `/devii/ws`. **Any new websocket path must add its own upgrade `location` above `location /`** (see docs `Production -> nginx`).
|
||||
|
||||
## SEO metadata service - SeoMetaService (kind `seo_meta`, `services/jobs/seo_meta_service.py`, `services/seo_meta.py`, `seo_meta_text.py`)
|
||||
|
||||
`SeoMetaService` generates a clean, SEO-optimized title/description/keywords for every published content item (types `post`, `project`, `gist`, `news`, `issue`) off the request path and **meters its own AI spend**. It is a distinct concern from the public **Tools -> SEO Diagnostics** auditor (kind `seo`) - do NOT conflate the two kinds; they share only the `jobs` queue table. It is the **constructive counterpart** to the diagnostics tool: diagnostics audits a URL, this one populates the on-page metadata. It is a `JobService` like `ForkService`: the artifact (the `seo_metadata` row) is **permanent**, so `cleanup()` is a no-op and the retention sweep removes only the job tracking row.
|
||||
|
||||
- **Tables.** `seo_metadata` is polymorphic and soft-deletable (in `SOFT_DELETE_TABLES`, born-live `deleted_at`/`deleted_by`): `uid, target_type, target_uid, seo_title, seo_description, seo_keywords, status (ready|pending|failed), source (ai|plain), generated_at, created_at, updated_at`, keyed UNIQUE on `(target_type, target_uid)`. `init_db()` ensures every column, the UNIQUE `idx_seo_metadata_target` and the live `idx_seo_metadata_status (status, deleted_at)` index. `seo_usage` is a single-row config-like usage table (NOT soft-delete) mirroring `news_usage`, keyed `SEO_USAGE_KEY="seo_meta"`. Helpers in `database.py`: `get_seo_metadata`/`get_seo_metadata_batch`/`has_fresh_seo_metadata`/`upsert_seo_metadata`/`mark_seo_metadata_stale` (every read filters `deleted_at IS NULL`; `get_seo_metadata` returns only `status="ready"` live rows) and `add_seo_usage`/`get_seo_usage`.
|
||||
- **Choke helper.** `services/seo_meta.py` `schedule_seo_meta(target_type, uid, regenerate=False)` and `schedule_seo_meta_for_table(table, uid, ...)` are import-cycle-free (only `database` + `queue`). They no-op for unknown types, missing uid, or (without `regenerate`) when a fresh `ready` row exists (`database.has_fresh_seo_metadata`); `regenerate=True` marks the row stale first (`database.mark_seo_metadata_stale`); both skip a target with an existing pending/running `seo_meta` job; otherwise `queue.enqueue("seo_meta", {target_type, target_uid}, "system", "seo_meta")`. Hooked at `content.create_content_item` (create, no-op guard) and `content.edit_content_item` (regenerate), the news publish sites in `services/news.py` (`status=="published"` only; existing-row update path uses `regenerate=True`), and `IssueCreateService` after the Gitea ticket is recorded. Because the work is async via the queue (NOT `run_in_executor`), the helper only enqueues.
|
||||
- **`process`** loads the target row (posts/projects/gists/news via `get_table`, issues via `gitea.store.get_ticket`), builds grounding via `services/ai_context.build_context` (fail-soft for news/empty `user_uid`), and calls the gateway **off-thread** (`asyncio.to_thread(correction.gateway_complete, internal_gateway_key(), system, source_text, timeout)` - the synchronous gateway call posts to the in-process gateway on localhost, so it MUST run via `to_thread` or it self-deadlocks the single worker - the same lesson as `correction.py`'s sync mode). It demands strict JSON `{seo_title, seo_description, seo_keywords}`, parses fail-soft, re-clamps every field server-side via `seo_meta_text.clamp_generated`, and falls back to `seo_meta_text.plain_seo_defaults` (status `failed`, source `plain`) on any failure - **the fields are never empty**. Usage accumulates via `correction.new_usage_totals` and flushes once with `database.add_seo_usage(totals)` when `calls>0` (the single-row `seo_usage` table, mirroring `news_usage`). It emits an audit `seo.meta.generate`/`seo.meta.failed` (`record_system`, category `tools`) and `upsert_seo_metadata(...)`.
|
||||
- **Backfill.** `run_once` calls `super().run_once()` then a bounded backfill sweep (gated by `seo_meta_backfill_enabled`, `seo_meta_backfill_batch` per type per tick) over published content lacking a fresh `ready` row, so pre-existing items get metadata with no one-shot migration.
|
||||
- **Admin surface.** `collect_metrics()` merges the `JobService` job-pipeline stats with `usage_metric_cards(get_seo_usage())`, so the **SEO Metadata** card on `/admin/services` shows both the live task pipeline and the AI cost/averages; the existing `admin.services.{name}` pub/sub topic + `live_view_relay` row pushes it live with NO new VIEWS row. A standard-paginated task list reads `queue.list_jobs(kind="seo_meta")` with `database.build_pagination` + `_pagination.html` when a dedicated page is desired.
|
||||
- **Clamps (single source of truth, `seo_meta_text.py`):** `seo_title` hard cap 60 (word-boundary, single hyphen, keyword front-loaded); `seo_description` hard cap 160 (word-boundary via `seo.truncate`, key message in the first 120 chars); `seo_keywords` 5-8 distinct lowercase comma-joined terms (the `<meta keywords>` tag is dead for Google but the feature mandates it - emit a SHORT honest list, never stuffed). `plain_text_from_markdown` reuses `rendering._render_content` + `utils.strip_html` so markdown (and em-dash) never leaks.
|
||||
- **SEO consumption fix (`seo.py` `base_seo_context`).** The meta description is now markdown-stripped (`plain_markdown`/`plain_text_from_markdown`, fixing the prior raw-markdown leak), `meta_keywords` is emitted (a safe plain string, NOT a Jinja-global name), and a `seo_target=(target_type, target_uid)` param makes it consume the ready `seo_metadata` row when present, else `plain_seo_defaults`. `base_seo_context`'s new `keywords`/`seo_target` params are OPTIONAL with safe defaults so existing callers are unaffected; the five detail routers (posts/projects/gists/news/issues) pass `seo_target`. `og_title`/`twitter:title` use the bare `seo_title` (drops the redundant " - DevPlace" suffix in social cards); `base.html` adds `<meta name="keywords">`, `og:image:width/height/alt` and `twitter:image:alt`. The per-type JSON-LD (`discussion_forum_posting`, `software_application_schema`, `news_article_schema`, `software_source_code_schema`) route their text/description through `plain_markdown`.
|
||||
- **Fan-out.** Schema `SeoMetaOut`; read route `GET /tools/seo-meta/{target_type}/{target_uid}` (`routers/tools/index.py`, public, JSON) returning the ready row or a plain default with status `pending`; Devii action `seo_meta_status` (public, read-only) + docs `tools-seo-meta-status`; CLI `devplace seo-meta prune|clear` (job rows only; the metadata persists); events `seo.meta.generate|failed`. Registered in `main.py` alongside `SeoService`. **A new `JobService` needs a server restart to go live.**
|
||||
|
||||
## DeepSearch tool - DeepsearchService (kind `deepsearch`, `services/jobs/deepsearch/`, `services/deepsearch/`, `routers/tools/deepsearch.py`)
|
||||
|
||||
The public **Tools -> DeepSearch** researcher is a multi-agent deep web researcher built on the **same async-job + ProgressHub + 4013-WS pattern as the SEO tool**, plus a per-session vector store and a grounded RAG chat. It is **public (guests included)**; abuse is bounded by the per-IP POST rate limit, a per-owner one-active-job cap, a page cap (1-30), depth cap (1-4), and the shared SSRF guard. Reuse the SEO tool as the template for any new Tools async job.
|
||||
|
||||
- **Owner helper is shared:** `routers/tools/_shared.py` `owner_for(request)` returns `("user", uid)` or `("guest", X-Real-IP)`; both `seo.py` and `deepsearch.py` import it (do not re-inline the owner derivation).
|
||||
- **Enqueue:** `POST /tools/deepsearch/run` (body `DeepsearchRunForm{query, depth 1-4, max_pages 1-30}`). It rejects with `429` if the owner already has a pending/running `deepsearch` job. It resolves the **logged-in user's `users.api_key`** (guests use `database.internal_gateway_key()`) into the job payload for per-user embedding/LLM spend attribution, generates the uid up front, writes a `deepsearch_sessions` row (`create_deepsearch_session`), enqueues the job carrying `{query, depth, max_pages, api_key, collection}`, and returns `{uid, status_url, ws_url}`. The enqueue uses a local `_enqueue` (not `queue.enqueue`) so the session uid and the job uid match.
|
||||
- **`process`** writes `control.json` (state `running`) + `payload.json` (augmented with the cross-session `cached_hashes`) under `config.DEEPSEARCH_DIR/{uid}`, launches `python -m devplacepy.services.jobs.deepsearch.worker <payload_json> <output_dir>` via `create_subprocess_exec` (high `limit=`), pumps **NDJSON stdout frames** into the in-process `ProgressHub` (`progress.py`), and on completion loads `output_dir/report.json`, persists the URL cache (`upsert_deepsearch_url_cache`), and updates the session row. `cleanup()` **drops the ChromaDB collection** (`VectorStore.drop`) and removes the session dir. Disposable: the collection + report dir are both deleted, unlike Fork/isslop.
|
||||
- **Worker pipeline** (`worker.py`, stdlib + httpx + playwright, importable subprocess): `enhance.plan_queries` (gateway -> JSON sub-queries, deterministic fallback) -> `crawl.search_queries` (rsearch via standalone httpx, never `PlatformClient`; per-query result buckets are **round-robin interleaved** so every planned angle contributes pages, never just the first query) -> `crawl.crawl` (batches of `CRAWL_CONCURRENCY` concurrent httpx fetches then playwright render fallback, `guard_public_url` on the URL and every redirect, content-hash + URL-hash dedup; **`depth` follows in-page links**: after each level the links of every crawled page are scored by query-token overlap via `extract.relevant_links` and the top `LINKS_PER_PAGE` unseen ones form the next level, `depth=1` disables following) -> `chunking.chunk_text` -> `embeddings.embed_texts` (gateway, **local hashing fallback** when unavailable) -> `store.VectorStore.add` (Chroma) -> `orchestrate.orchestrate` (retrieval-grounded agents, see below). The worker writes `report.json` and `url_cache.json` and emits a `report_ready` frame carrying `synthesis`.
|
||||
- **Search-provided content is a first-class source (`crawl.py`, the second junk-report fix).** `search_queries` calls rsearch with `content=true`, so each candidate carries the search engine's own readable `content`/`description` extract. This matters because the top sources for many questions are **bot-hostile** (X/Twitter, YouTube, Reddit, Facebook, Instagram, LinkedIn, TikTok - `HOSTILE_DOMAINS`): a headless fetch of those hits a login/consent wall ("Before you continue to YouTube", "Sign in to X") and yields near-zero text, which is why a 12-source run used to collapse to ~14 chunks. Now `crawl._resolve_candidate` **skips the fetch entirely for a hostile domain and uses the rsearch snippet** (`_snippet_page`, `source="search"`, `SNIPPET_MIN_CHARS` floor), and for every other domain it fetches normally but keeps the rsearch snippet as a **floor** (uses whichever of crawl-text vs snippet is longer), so a walled or thin page still contributes its real content instead of being dropped. This alone took a query from "cannot be answered" to a correct cited answer (14 -> 61 chunks, diversity 0.333 -> 0.75). **Never revert `content=true` and never send a headless render at a `HOSTILE_DOMAINS` host.**
|
||||
- **Content extraction (`extract.py`, stdlib only):** `extract_html(raw, base_url)` is a readability-grade `HTMLParser` extractor used by both the httpx and playwright fetch paths (the old naive regex tag-stripper produced nav/cookie-banner boilerplate as "content" - the historic root cause of junk reports). It skips `script/style/nav/header/footer/aside/form` and ARIA `role=navigation|banner|contentinfo|...` regions, prefers `<article>`/`<main>` when they carry at least `MIN_CONTENT_TOTAL` chars, drops link-dense blocks (`MAX_LINK_DENSITY`, menus) and sub-`MIN_BLOCK_CHARS` fragments, unescapes entities, and emits real paragraphs joined by blank lines - which also makes `chunking.chunk_text`'s paragraph split actually fire (the flattened text used to be sliced mid-sentence). It also returns the page's `(url, anchor_text)` links (absolute, deduped, nav links excluded) for depth crawling; `relevant_links(links, query, limit)` scores them by query-token overlap and filters non-document extensions.
|
||||
- **No "gaps"/critic agent (removed - do not reintroduce).** DeepSearch used to run a fourth "critic" agent that produced an "Open gaps" list. It was removed end-to-end (orchestrate, worker report, `DeepsearchSessionOut`, router context, session template, markdown/HTML export, `DeepsearchTool.js` agent labels, CSS) because it routinely emitted misleading, self-contradictory gaps on correctly-cited reports (the historic cause was that the critic was fed a truncated, retrieval-ordered source slice that dropped cited source numbers out of its window, so it fabricated "citation [n] is not in the report / no evidence provided"). **Do NOT reintroduce a `gaps` field or a critic agent.** The `_numbered_source_digest(pages)` helper survives and is used by the **linker** - a compact per-source `[n] title (url)\nexcerpt` block for EVERY page where the header line is always emitted even when the excerpt is trimmed, so all source numbers `1..N` are guaranteed present (never pass a raw `context[:N]` slice to an agent that reasons about source numbers). Regression: `tests/unit/services/jobs/deepsearch/orchestrate.py::test_linker_receives_full_source_list` / `::test_numbered_source_digest_keeps_every_source_number_under_cap`.
|
||||
- **Orchestration (`orchestrate.py`) is retrieval-grounded and markdown-first.** The worker indexes BEFORE analysis and passes the `VectorStore` + planned queries to `orchestrate`, which embeds the question and each sub-query and pulls `hybrid_search` top chunks (round-robin merged, up to `CONTEXT_CHUNKS_MAX`), building the context from the RETRIEVED passages grouped per source - the source numbers `[n]` align with the report's `sources` list (page order), so inline citations, finding citations, and the rendered numbered source list agree. Page-head excerpts are only the fallback when retrieval is empty. Synthesis is **two-step to avoid the markdown-inside-JSON trap**: the summarizer writes a plain markdown report (`REPORT_MAX_TOKENS`, retried once if empty), then a separate `extractor` agent returns the findings JSON (retried once, tolerant `_parse_json` handles code fences and trailing garbage); the linker (confidence) failure is caught and never discards the report. The three agents in the pipeline today are summarizer, extractor, and linker - there is no fourth "critic" stage. Only a failed/empty report falls back to `_heuristic`, which stamps `synthesis="heuristic"` and emits a `status:"failed"` agent frame - the degradation is VISIBLE: `report.synthesis` flows through `DeepsearchSessionOut.synthesis`, the session template renders a "Degraded report" banner (`.ds-degraded`), and the markdown export carries the same note. A successful run stamps `synthesis="agents"`. **Never re-inline synthesis into a single JSON blob and never let a synthesis failure ship silently.**
|
||||
- **PDF ingestion (`crawl.py` `fetch_page` + `pdf.py`):** a crawled candidate is treated as a PDF when its `content-type` is `application/pdf`/`application/x-pdf`, its URL path ends in `.pdf`, or its first bytes match the `%PDF-` magic (`pdf.is_pdf`). `fetch_page` streams the body and caps it at `MAX_PDF_BYTES` (15 MB); for a PDF it calls `pdf.extract_pdf_text`, which writes the raw binary to a `tempfile` temp location (cleaned up via `Path.unlink(missing_ok=True)` in `finally`), parses it with `pypdf` (`PdfReader`, capped at `MAX_PDF_PAGES` = 50, title pulled from metadata), and normalizes whitespace. The resulting `CrawledPage` carries `source="pdf"`; PDFs skip the Playwright fallback. Everything downstream is source-agnostic (chunking/embedding/orchestration read `page.text`/`page.source` unchanged), so no other module changes. New unpinned dep `pypdf` (pure-python, no system deps).
|
||||
- **Pause/resume/cancel:** `POST /tools/deepsearch/{uid}/{pause|resume|cancel}` (owner-gated) rewrite `control.json`; the worker's `should_stop` callback polls it between source fetches (paused = sleep-loop, cancelled = stop). State lives in a file, not the job row, so the running-in-a-subprocess worker can read it without a DB round-trip.
|
||||
- **Vector store (`services/deepsearch/store.py`):** `VectorStore` wraps `chromadb.PersistentClient(path=config.DEEPSEARCH_CHROMA_DIR)`, one collection per session (`ds_<uid>`). `Chunk` is the dataclass. `hybrid_search` blends cosine vector similarity with a BM25 keyword score (weights `HYBRID_VECTOR_WEIGHT`/`HYBRID_KEYWORD_WEIGHT`) over the candidate set, with optional metadata `where` filters. `embeddings.py` `embed_texts` calls the gateway embeddings endpoint and **falls back to a deterministic local hashing vector** on any failure (so the tool degrades, never breaks).
|
||||
- **RAG chat (`services/deepsearch/chat.py` + `WS /tools/deepsearch/{uid}/chat`):** a dedicated lightweight loop (NOT the Devii hub), served **only by the service-lock owner** (closes `4013` for fast retry). Answers are grounded ONLY in the session collection via `hybrid_search`, cited inline, rendered client-side via `dp-content`. Turns persist to `deepsearch_messages` and audit `deepsearch.chat`. Frontend component `<dp-deepsearch-chat>` (`static/js/components/AppDeepsearchChat.js`) clones `AppDocsChat`'s framing but uses its own WebSocket to the chat path.
|
||||
- **Status/report/export routes:** `GET /tools/deepsearch/{uid}` (`DeepsearchJobOut`), `GET /tools/deepsearch/{uid}/session` (`respond(..., DeepsearchSessionOut)`, HTML or JSON), `GET /tools/deepsearch/{uid}/export.{md,json,pdf}` (`services/deepsearch/export.py`; PDF via weasyprint). All are **capability URLs** scoped by the unguessable uuid7. **Viewer-flag discipline:** the session schema/context use `viewer_is_admin`/`viewer_owns` (never `is_admin`/`owns`) so a `respond()` context key never shadows a Jinja global (the same class of issue as the issues `/{number}` route). `tests/api/tools/deepsearch/session.py` guards the HTML render.
|
||||
- **Completion race (load-bearing read-path fix).** The worker writes `report.json` to disk and `service.process` publishes the `done` frame **from inside `process()`**, but the `JobService` framework only commits `jobs.result`/`status=DONE` afterwards, in `_reap()` -> `_finish_done()` on a later tick. The frontend navigates to the session page the instant it receives `done`, so a read that keyed only off `jobs.status == DONE` returned an EMPTY report (`None` score, 0 sources) until a manual refresh. Fix: `_report_for(uid, job)` returns `job.result.report` when the job is `DONE` and non-empty, else falls back to the on-disk `report.json` (`_report_from_disk`, `DEEPSEARCH_DIR/{uid}/report.json`) - which exists before the `done` frame is ever sent - and returns `{}` only for a `FAILED` job or a genuinely still-running job with no report on disk. `_session_context` derives `done`/`status` from `bool(report)` (not raw job status), and the chat WS gate accepts `session.status == "done"` (set inside `process()` before the publish) as ready. `_export_report` reuses the same fallback. Regression: `tests/api/tools/deepsearch/session.py::test_session_reads_disk_report_before_result_commit`. **Any new read of a job result that a client reaches immediately after a `done`/`session_url` frame must use this same on-disk fallback, never bare `jobs.status`.**
|
||||
- **Clickable inline citations (`services/deepsearch/citations.py`).** The report/findings carry `[n]` markers (and the model sometimes emits `[3][9][1-2]`); the `link_citations(html, source_count)` template global (registered in `templating.py`) rewrites each `[n]` and each `[a-b]` range into `<a class="ds-cite" href="#ds-source-n">[n]</a>` anchors that jump to the numbered `<li id="ds-source-n">` in the Sources list (source numbering is page order, matching the `[n]` the summarizer was given). It splits out `<a>`/`<code>`/`<pre>` regions first so markers inside links/code are left alone, expands ranges to individual links, and drops out-of-range numbers (no broken anchors). The session template nests it over the server render: `{{ link_citations(render_content(summary), sources|length) }}` and `{{ link_citations(finding.detail|e, sources|length) }}`, plus a per-finding `.ds-finding-cites` chip row from `finding.citations`. `.ds-cite`/`.ds-sources li:target` styling lives in `deepsearch.css`. The report prompt asks for one number per bracket (never a range) so output is consistent, but the linkifier handles ranges regardless. Regression: `tests/unit/services/deepsearch/citations.py`.
|
||||
- **Tables (`deepsearch_sessions`, `deepsearch_messages` soft-deletable + in `SOFT_DELETE_TABLES`; `deepsearch_url_cache` GC-only):** columns are ensured in `init_db()` (every queried column) with indexes. Every insert writes `deleted_at:None/deleted_by:None`; every read filters `deleted_at IS NULL`.
|
||||
- **Frontend** (do not hand-roll): `DeepsearchTool.js` (`app.deepsearchTool`) drives the form via `Http.send`, watches `DeepsearchProgressSocket` (cloned from `SeoProgressSocket`, 4013 retry), and wires pause/resume/cancel. `static/css/deepsearch.css` uses the design tokens and is mobile-responsive.
|
||||
- **Progress frame protocol (append-only, the worker<->JS contract):** `phases.py` is the single source of truth for phase identity, shared by `worker.py` (emit) and `DeepsearchTool.js` (render). `PHASE_ORDER = [planning, searching, crawling, indexing, analysis, synthesis]`; `worker._stage(stage, message, phase)` emits BOTH the legacy `stage` frame (byte-identical to before) AND a parallel `phase` frame `{phase, index, total, label}` so the timeline strip advances. The first emitted frame carries `version:1`. Every other frame type and its keys: `substep` (planning angles, `phase`+`message`; also emitted by analysis grounding), `queries`, `candidates`, `rsearch`, `progress` (`done`/`total`/`url`/`depth`), `page_loaded` (now `source`/`render`/`depth`/`elapsed_ms`/`done`/`total`), `page_cached`/`page_skipped`/`page_duplicate` (now `reason`/`elapsed_ms`), `embed_batch` (`batch`/`total_batches`/`backend`/`done`/`total`, emitted before AND after each batch), `embed_done` (`backend`/`chunk_count`), `agent` (`agent` one of summarizer|extractor|linker, `stage`/`status` start|done|failed, with `elapsed_ms`/`tokens_in`/`tokens_out` on done), `report_ready` (now also `synthesis`), `done` (`session_url`), `failed` (`message`). **The contract is append-only: never rename or drop a frame type**; `service._run_worker` pumps every stdout line into the `ProgressHub` untouched, so new frame types reach the WS with no handler change. `tests/api/tools/deepsearch/index.py` is the append-only regression guard.
|
||||
- **RAG-audit hardening (engine correctness, no route/schema change):** (1) **Embedding-dimension consistency** - gateway embeddings carry provider-native dims while `local_embed` is fixed 256-dim; the worker `_index_chunks` now decides the backend ONCE per job (the first gateway failure or non-gateway result forces local for ALL remaining batches), and `VectorStore.add` drops any vector whose length differs from the collection's established dim, so one collection never mixes dims (cosine search across mixed dims is corrupt). `embeddings.EmbedResult.dims` and `VectorStore.dims` (lazily probed from the collection) expose the dimension; `chat.retrieve` re-embeds the query locally and skips retrieval if it still cannot match the stored dim. (2) **Citation grounding** - `orchestrate` drops any finding with no citations, and the summarizer prompt forbids uncited claims and treats the QUESTION as data not an instruction (prompt-injection reduction via `_sanitize_question`); `chat._strip_unmatched_markers` removes any inline `[n]` marker that does not map to an emitted citation. (3) **Confidence calibration** - when only a single domain was crawled, `orchestrate` caps confidence by the source-diversity-derived ceiling (overconfidence guard). (4) `EmbeddingCache` is bounded at `EMBED_CACHE_MAX` to cap in-memory growth.
|
||||
- Devii tools `deepsearch`/`deepsearch_status`/`deepsearch_session` (public); docs `tools-deepsearch`; CLI `devplace deepsearch prune|clear`; audit `deepsearch.run.request|complete|failed` + `deepsearch.chat` (category `tools`). **New dependencies:** `chromadb`, `weasyprint`, `pypdf` (all unpinned). New runtime dirs `config.DEEPSEARCH_DIR`/`DEEPSEARCH_CHROMA_DIR` are registered in `DATA_PATHS`. **Add a dedicated nginx WS `location` for `/tools/deepsearch/{uid}/ws` and `/chat`** above `location /` for production, like the SEO and Devii sockets.
|
||||
|
||||
## AI Usage Analyzer tool - IsslopService (kind `isslop`, `services/jobs/isslop/`, `routers/tools/isslop.py`)
|
||||
|
||||
The public **Tools -> AI Usage Analyzer** classifies a git repository or website as AI slop, sophisticated AI-assisted work or genuine human work. It is built on the standard async-job pattern (a `JobService` running a subprocess worker), but its live channel is **pub/sub, not a dedicated WS route**: every worker event is published to `public.isslop.{uid}` AND persisted to `isslop_events`, and the frontend pairs the pub/sub subscription with an incremental `GET /tools/isslop/{uid}/events?after=SEQ` poll, so guests (who cannot subscribe to `public.*` unless `pubsub_allow_guests` is on) and reconnecting tabs replay from the durable trail. **Never rely on pub/sub alone for this tool: the DB event trail is the source of truth, pub/sub is the fast path.**
|
||||
|
||||
- **Engine layout:** `services/jobs/isslop/` holds `acquisition/` (git probe via `git ls-remote`, depth-1 clone with size preflight + live 3 GB kill guard, stealth Playwright website crawler with HTTP fallback, path-traversal-safe workspace helpers), `analysis/` (exclusion rules, stylometric metrics, language detection, per-repo baselines, `signals/` with one detector family per file, two-axis scoring), `agent/` (gateway LLM client, per-file classifier, vision reviewer, report writer with deterministic fallback), plus `pipeline.py` (the event-yielding run), `worker.py` (subprocess entry), `events.py` (frame protocol), `persistence.py` (`EventPersister` writes events/file results/image results/report and stamps the analysis row), `store.py` (all DB access), `badge.py` (SVG), `service.py` (`IsslopService`), `config.py` (all constants + `WorkerSettings`).
|
||||
- **Worker contract:** `IsslopService.process` writes the worker payload (url + admin toggles + gateway endpoint/model/key) to `config.ISSLOP_RUNS_DIR/{uid}/payload.json`, resolves the workspace under `config.ISSLOP_WORKSPACES_DIR` (`workspace_for` rejects any path escaping the root), launches `python -m devplacepy.services.jobs.isslop.worker <payload> <workspace>`, and relays each NDJSON stdout line through `EventPersister.apply` (SQLite) then `pubsub.publish`. The workspace and run dir are removed in a `finally`; the pipeline also deletes the workspace itself as its final act, so **no acquired source survives an analysis** - only the report and its evidence rows.
|
||||
- **AI through the gateway only:** `agent/llm.py` talks solely to `config.INTERNAL_GATEWAY_URL` with model `molodetz` and `database.internal_gateway_key()` (vision uses the same model - the gateway handles image parts). `review_available`/`vision_available` gate the AI and image stages; on any gateway failure the static engine remains authoritative and the report falls back to the deterministic composer. All HTTP (gateway, git size preflight, website fallback crawl) goes through `stealth_async_client`.
|
||||
- **Artifacts are permanent, the job row is not.** Like `ForkService`, `cleanup()` never touches `isslop_analyses`/`isslop_events`/`isslop_file_results`/`isslop_image_results`/`isslop_reports` - the report and badge are public capability URLs meant to outlive the run; the retention sweep removes only the `jobs` tracking row. `devplace isslop clear` is the only bulk hard-delete (plus per-analysis `store.purge_analysis`).
|
||||
- **Ownership and guest history sync:** the owner is `("user", uid)` or `("guest", DEVII_GUEST_COOKIE)` - NOT the tools `_shared.owner_for` IP fallback, because history must survive IP changes and be claimable. The page/list/run handlers mint the guest cookie when absent (same cookie as Devii/customization, one guest identity platform-wide). `_sync_guest_history` runs on page and list requests: when a signed-in user still carries a guest cookie, `store.claim_guest_analyses` re-owns those rows via UPDATE (a move, never a copy - no duplicate data). One active analysis per owner (`429` otherwise, audited `denied`).
|
||||
- **Tables:** `isslop_analyses` is soft-deletable (in `SOFT_DELETE_TABLES`, born-live inserts, reads filter `deleted_at IS NULL`, indexed on `(owner_kind, owner_id, created_at)`/`status`/`content_hash`); the evidence tables (`isslop_events` keyed `(analysis_uid, seq)`, `isslop_file_results`, `isslop_image_results`, `isslop_reports` UNIQUE on `analysis_uid`) are GC-only evidence purged with their analysis. All ensured in `init_db()`.
|
||||
- **Routes** (all under `/tools/isslop`, capability URLs): `GET ""` page, `POST /run` (`IsslopRunForm`, http/git/ssh URL pattern), `GET /list` (owner history), `GET /{uid}` (`IsslopAnalysisOut`), `GET /{uid}/events` (ordered replay), `GET /{uid}/report` (`respond(..., IsslopReportOut)` - HTML shows the live `<dp-isslop-run>` while running and the server-rendered report when completed; the markdown body goes through `render_content`), `GET /{uid}/report.md`, `GET /{uid}/badge.svg` (self-contained SVG, hardcoded colors by design - it must render on external sites). Badge/report URLs are absolute via `seo.site_url`.
|
||||
- **Frontend:** two site-wide web components (`static/js/components/AppIsslop.js` `<dp-isslop>` = submit form + history list; `AppIsslopRun.js` `<dp-isslop-run>` = live progress feed), registered in `components/index.js`, light DOM, reusing `Http`/`Poller` and `app.pubsub`. Page CSS `static/css/isslop.css` (design tokens). On `done` the run component reloads the page so the report is the server-rendered (SEO/`render_content`) version, never a client re-render.
|
||||
- **Verdict blending is per-file, never a global mean (load-bearing).** The AI review pass runs its per-file gateway calls concurrently (`pipeline.AI_REVIEW_CONCURRENCY`, semaphore-bounded like the image pass) and adjusts ONLY the files it actually reviewed: `pipeline.apply_ai_verdicts` blends each sampled file's static origin/quality with its own verdict (0.6/0.4), then the WHOLE repo is re-aggregated with the normal SLOC/criticality weights, and `scoring.ai_fraction` maps per-file origin scores through a smooth 35-65 ramp (never the old hard 45/55 buckets). **Never reintroduce a repo-level mean of the 12 sampled verdicts** - it hands a tiny sample a fixed 40% of the verdict, so the LLM's clustered hedging values (30/40/50) drown thousands of files of static evidence and unrelated projects converge on identical percentages (the real-world twin-84%-human defect). **Single source of truth for the final verdict:** the SCORE event, the DONE payload and `generate_report` all consume the SAME final `RepoScores` (image influence applied via `scoring.adjust_for_images`, which recomputes slop/grade/human together) - the summary grade and the report body grade can therefore never disagree; `tests/unit/services/jobs/isslop/pipeline.py` guards both invariants.
|
||||
- **Image evidence thumbnails + retry-safe evidence.** Workspaces die with the run, so the vision stage persists an aspect-preserving WebP thumbnail per reviewed image (`vision.make_thumbnail`, sha1-of-relative-path name) into `config.ISSLOP_MEDIA_DIR/{uid}` (the dir comes to the worker via the payload `media_dir`); the `thumb` name rides the `image` event, is stored on `isslop_image_results`, and is served by `GET /tools/isslop/{uid}/media/{name}` (strict `^[a-f0-9]{16}\.webp$` name pattern + `is_relative_to` root check - never loosen either). The report page renders the thumbnails with `data-lightbox` (the shared `app.lightbox` opens them full-size) and the live feed shows a tiny inline preview per image event. **`store.reset_evidence(uid)` runs at the top of every `IsslopService.process`** - a retried job (orphan recovery) previously re-inserted its events/file/image rows, duplicating every image and file in the report; any new evidence table MUST be added to `reset_evidence` AND `purge_analysis`.
|
||||
- **Template provenance (the "ships defaults" detector).** `analysis/templates.py` `detect_template(workspace)` scores starter-template evidence repo-wide (it reads files the inventory excludes, like `package.json`): known template slugs/authors in the manifest (+ `ct3aMetadata`), README template marketing (weights capped so a wordy README cannot alone confirm), and the kitchen-sink scaffold constellation (count of standard scaffold artifacts past 3 freebies). The saturating score feeds `scoring.adjust_for_template` - a no-op below 35, a 0.7x floor on ai_percent/origin when confident, 0.85x when >= 70 - applied LAST in the pipeline scoring stage, after the AI and image blends. Near-certain evidence (>= 70) also FORCES category `ai-slop` (defaults shipped as-is are slop by the canonical definition - clean scaffold code never earns an untouched template `sophisticated-ai`, whose meaning is 'the presenter decided'); the confident band caps a `human-*` category at `uncertain`. Calibration truth set (guarded by tests): the six stock boilerplates (ixartz x2, create-t3-app, vercel ai-chatbot x2, fullstack-nextjs-app-template) grade C-D with markers listed, while devplacepy itself scores 0.0 with zero markers - tune weights against BOTH sides, never only the slop set. The evidence rides the `signal` inventory event, the SCORE payload (`template_score`/`template_markers`) and the report's Template Provenance section; admin toggle `isslop_template_detection`.
|
||||
- **Rendered-DOM signal family (`analysis/domsignals/`).** A homepage-only, live-browser companion to the text-based `signals/webtells.py` checks: `acquisition/browser.py`'s `StealthBrowser.capture()` loads the page and returns the actual rendered DOM (computed styles, a class-name census, meta tags, headings/landmarks, console warnings, network response hosts/headers, a screenshot) via `DOM_EXTRACT_SCRIPT` in `acquisition/domcapture.py`. `domsignals/base.py` defines its own `@dom_check`/`@dom_site_check` decorator registry, mirroring the SEO job's `checks/` `@page_check`/`@site_check` mechanics, but emitting isslop's own `Signal` type (not a new one). Eight category modules (`builders`, `color`, `typography`, `layout`, `copy`, `metaseo`, `accessibility`, `buildsignals`) contribute 49 distinct signal codes; `domsignals/aggregate.py` `aggregate_dom_evidence` runs every registered check over the captured page(s) and saturates the weighted total into a `DomEvidence` (score/bucket/signals/detected_builder/builder_confidence). This brings the engine's total to twenty-one detector families (13 text-based in `signals/`, 8 rendered-DOM in `domsignals/`) and 126 signal codes (77 text-based, 49 rendered-DOM) - update these counts again the next time a family is added or removed, never leave them stale.
|
||||
- **Homepage-only by design (`config.DOM_ANALYSIS_MAX_PAGES = 1`).** Capturing a full DOM, console, network trail and screenshot on every crawled page would multiply the browser cost of a run, so the pass runs ONLY against the first page at crawl depth 0 (`acquisition/website.py` gates `depth == 0 and index < DOM_ANALYSIS_MAX_PAGES`). Bump the constant later if the tool needs multi-page DOM coverage - `DomSiteContext`/`dom_site_check` (e.g. `detect_duplicate_meta_description`) already support more than one page. Git-repository sources never populate `dom_snapshots` (`dom_sink` is threaded only through `crawl_website`, never `clone_repository`), and a run where the browser could not be launched simply yields an empty page list, so `aggregate_dom_evidence([])` is a clean no-op (`DomEvidence(score=0.0, bucket="none")`).
|
||||
- **Scoring order is load-bearing: images -> DOM -> template, never reorder.** `pipeline.py` applies `scoring.adjust_for_images`, then `adjust_for_dom_signals`, then `adjust_for_template`, in that exact sequence, and `adjust_for_template` MUST stay last. `adjust_for_dom_signals` blends `DomEvidence.score` into `ai_percent` at a small, deliberately cautious weight (`config.DOM_AI_WEIGHT = 0.12`, since this whole signal family is new and uncalibrated) UNLESS a confident builder match (`builder_confidence >= DOM_BUILDER_CONFIDENT_THRESHOLD`) forces the category to `ai-slop` via the same `_force_slop_category` helper the template detector uses. Because `adjust_for_template` runs after it, a template match can still floor/force the category further; nothing may run after `adjust_for_template`, since a later step would silently undo a forced `ai-slop` category.
|
||||
- **DOM evidence never attaches to a per-file `FileScore`/`FileContext` (do not bolt it on).** `DomEvidence` is repo/page-level evidence blended once into the final `RepoScores`, exactly like template provenance and the image mean - never distributed across individual files. A rendered page has no SLOC and no line numbers to weight against, and its evidence (computed styles, a screenshot, console/network output) is not textual, so it cannot be scored, sampled or displayed through the SLOC-weighted per-file model the rest of the engine uses. Any new DOM check emits page-level `Signal`s into `DomEvidence.signals`, never into a `FileScore.signals` list.
|
||||
- **`isslop_dom_results` follows the same evidence-table obligation as every other isslop table.** It mirrors `isslop_image_results` (`store.py` `TABLE_DOM_RESULTS`) and is already wired into both `store.reset_evidence(uid)` and `store.purge_analysis(uid)`; any future evidence table added under `domsignals/` must be added to both the same way.
|
||||
- **DEP_UNRESOLVED is alias-aware (do not regress).** The JS/TS unresolved-import detector (`signals/hallucination.py`) flags imports that match no package.json dependency (all four sections), Node builtin or local path - i.e. phantom/hallucinated dependencies. It MUST skip everything that is not an npm specifier: relative/absolute/URL imports, `node:` and any `scheme:` specifier, the non-package prefixes `@/`, `~`, `#`, `$` (tsconfig/subpath/Svelte aliases - none are valid npm names), and every prefix parsed from `tsconfig.json`/`jsconfig.json` `compilerOptions.paths` (`engine._javascript_alias_prefixes`, carried on `RepoContext.javascript_alias_prefixes`). tsconfig is JSONC and alias keys contain `/*`, so comments are stripped with the string-aware scanner `_strip_jsonc_comments` - NEVER a comment regex (a regex eats the `"@/*"` alias strings themselves; this was a real defect). Before the alias awareness the detector flagged nearly every file of a standard Next.js app; after, only genuine phantom deps remain. `tests/unit/services/jobs/isslop/hallucination.py` guards it.
|
||||
- **Clickable source references (annotated source viewer).** The static stage persists the FULL source of every signal-bearing file (cap `SOURCE_CAP_FILES`=60 files / 200KB each) into the same per-analysis media dir (`_persist_source`, `s<sha1>.txt`); the name rides the `file` event and the `isslop_file_results.source` column. `GET /tools/isslop/{uid}/source?path=...&line=N` renders `isslop_source.html`: server-rendered line table (Jinja autoescape covers XSS) with line-number anchors `#LN`, signal lines highlighted with inline annotation chips, a focused line, and a findings nav in the sidebar; the strict `^s[a-f0-9]{16}\.txt$` + `is_relative_to` checks mirror the media route. Everything referencing a file links there: the file-results table path, each signal chip (`&line=N#LN`), and the report prose - `_linkify_sources` rewrites backticked paths in the report markdown into links BEFORE `render_content`, and the reporter system prompt requires the model to backtick every path it mentions. `reset_evidence`/`purge_analysis` already sweep the media dir, so sources share the thumbnail lifecycle.
|
||||
- **No absolute paths ever leave the engine:** every user-facing path is workspace-relative (`relative_to(workspace)`); keep it that way in new detectors/events.
|
||||
- **Docs heading anchors + `.docs-toc` (platform-wide):** `docs_prose._render_markdown` post-processes every prose page, stamping a slugified `id` on each `h2`/`h3` (`heading_slug`, GFM-style, deduplicated with `-N` suffixes) and appending a hover-visible `.docs-heading-anchor` permalink; `scroll-margin-top` keeps targets below the topnav. The `isslop-checks` page uses this for its clickable Contents grid: a `.docs-toc` nav placed OUTSIDE the `data-render` block (raw HTML passes through untouched) whose `href="#slug"` values are computed with the SAME `heading_slug` function at generation time - reuse `.docs-toc`/`.docs-toc-item`/`.docs-toc-count` (styled in `docs.css`) for any other long docs page, and never hand-write a slug that `heading_slug` would not produce. `tests/unit/docs_prose.py` guards the slugging and injection.
|
||||
- Devii tools `isslop`/`isslop_status`/`isslop_report`/`isslop_list` (member-only, `requires_auth=True` per policy - the HTTP surface stays public); docs `tools-isslop` + `isslop-checks` (the full plain-language check catalog); CLI `devplace isslop analyze|prune|clear`; audit `isslop.run.request|complete|failed` (category `tools`); achievement key `isslop` ("Slop Hunter"). New unpinned dep `playwright-stealth`; runtime dirs `config.ISSLOP_DIR`/`ISSLOP_WORKSPACES_DIR`/`ISSLOP_RUNS_DIR` in `DATA_PATHS`.
|
||||
49
devplacepy/services/messaging/CLAUDE.md
Normal file
49
devplacepy/services/messaging/CLAUDE.md
Normal file
@ -0,0 +1,49 @@
|
||||
# Messaging (`devplacepy/services/messaging/`)
|
||||
|
||||
This file documents the real-time direct-message chat subsystem. Claude Code auto-loads it when a file under `devplacepy/services/messaging/` is read or edited.
|
||||
|
||||
## Overview
|
||||
|
||||
The `/messages` feature is a live WebSocket chat layered over the SAME `messages` table and audit/notification cores as before; there is no parallel chat store. The WS only adds live delivery, presence, typing, and read receipts on top of the existing persistence.
|
||||
|
||||
`messages.py` (the router, mounted at `/messages`) exposes: `GET ""` (page; the conversation sidebar is one windowed SQL aggregate and an opened thread loads only the newest `CONVERSATION_MESSAGE_LIMIT` = 500 messages - keep these queries bounded), `GET /search`, `POST /send` (no-JS fallback), and `WS /messages/ws` (NOT lock-owner gated - accepts on every worker; closes `1008` for guests).
|
||||
|
||||
## Bounded page queries (load-bearing performance rule)
|
||||
|
||||
`get_conversations` is ONE window-function query (`ROW_NUMBER() OVER (PARTITION BY other_uid ORDER BY created_at DESC, id DESC)` over `sender_uid = :me OR receiver_uid = :me`, the `get_recent_comments_by_target_uids` precedent) that returns only each partner's latest row - never load the user's full message history into Python to build the sidebar. `get_conversation_messages` loads at most `CONVERSATION_MESSAGE_LIMIT` (500) most-recent rows (`ORDER BY created_at DESC, id DESC LIMIT`, reversed to ascending in Python); older history stays in the DB and is simply not rendered. Both are covered by `idx_messages_conversation`/`_rev` (verified `EXPLAIN QUERY PLAN`, MULTI-INDEX OR, no table scan). `messages` is NOT in `SOFT_DELETE_TABLES`, so neither query filters `deleted_at`. Any new message listing must stay bounded and indexed the same way.
|
||||
|
||||
## WS endpoint `/messages/ws`
|
||||
|
||||
Lives on the existing messages router (no new router/prefix; already mounted at `/messages`). It is **NOT lock-owner gated** (deliberately - unlike `/devii/ws`): `await websocket.accept()`, resolve the owner via session cookie / api-key (`_resolve_ws_user` reuses `utils._user_from_session`/`_user_from_api_key`), and accept on EVERY worker. Gating on `service_manager.owns_lock()` was the original design and was wrong here: when no worker holds the service lock (services disabled, or the lock not yet acquired) every socket got closed `4013` and the client fast-retried every 200ms forever (a connection storm with no stream, and the send form fell back to a full-page POST reload). Chat must not depend on the background-service lock. Messaging requires an authenticated user - guests are closed with `1008`. The receive loop is wrapped in `try/except WebSocketDisconnect` + a broad `except` (never crash the worker) and ALWAYS unregisters the socket in `finally`.
|
||||
|
||||
## Connection registry
|
||||
|
||||
The `ConnectionManager` singleton `message_hub` (`services/messaging/hub.py`), **one per worker**: `user_uid -> set[WebSocket]`, plus an in-process `last_seen` map and a bounded `delivered` LRU set (`mark_delivered`/`was_delivered`, cap 4000) used to dedupe direct vs relayed delivery. `register`/`unregister` return whether the user just came online / went offline (for presence announces). `send_to_user`/`send_to_users` fan a frame out to all of a user's sockets (multi-tab) and swallow dead-socket errors. Presence and last-seen are in-process per worker (no DB column).
|
||||
|
||||
## Cross-worker delivery (the relay)
|
||||
|
||||
Because the WS accepts on every worker, a message persisted on worker A must still reach a recipient whose socket lives on worker B. `services/messaging/relay.py` `message_relay` (a per-worker singleton asyncio loop, started on the first socket connect, self-stops when `message_hub.has_connections()` is false) polls `SELECT * FROM messages WHERE id > :watermark` (uuid7-backed autoincrement `id`) every ~1s and pushes any row whose sender/receiver has a LOCAL socket and that was not already `was_delivered` to those local sockets, advancing the watermark to the max row seen. Same-worker sends are delivered INSTANTLY by `broadcast_message` (which calls `message_hub.mark_delivered` so the relay skips them); the relay only fills the cross-worker gap, so same-worker latency is zero and cross-worker latency is bounded by the poll interval. The relay primes its watermark to the current `MAX(id)` on first start so it never replays history. **Trade-off:** typing/read/presence frames are in-process per worker only - across the two `make prod` workers those ephemeral signals reach only same-worker peers (message delivery is always correct). On a single dev worker everything is instant and complete.
|
||||
|
||||
## DRY persist choke point
|
||||
|
||||
`services/messaging/persist.py` `persist_message(sender, receiver_uid, content, attachment_uids, *, request=None, origin)` is the ONE function that inserts the row, links attachments, fires `create_notification` + `clear_messages_cache` + `create_mention_notifications`, logs, and writes the `message.send` audit event. Both the WS `send` handler and the HTTP `POST /messages/send` handler call it, so audit/notification/mention behavior is byte-identical on both paths (the Devii `send_message` action is `handler="http"` -> `POST /messages/send`, so it also flows through here and broadcasts live). Content is capped at 2000 chars server-side. When `request` is a `Request` it audits via `audit.record`; on the WS path it now passes `request=websocket` (auditing via `audit.record_system` with `actor_kind="user"` + `origin="websocket"`, same `message.send` key/category - no new event invented; presence/typing/read are ephemeral and NOT audited).
|
||||
|
||||
## AI correction and AI modifier apply to direct messages, with LIVE delivery of the final content
|
||||
|
||||
`"messages": ("content",)` is in the `CORRECTABLE_FIELDS` registry and `persist_message` invokes `schedule_correction`/`schedule_modification` (sender = the user), so typing `@ai <instruction>` in a DM runs the AI modifier (default on + sync) and an enabled correction rewrites the content. Both send paths funnel through `routers/messages._finalize_and_broadcast(sender, message, request, client_id=None)`: it awaits any pending SYNC correction/modification futures stashed on `request.scope[PENDING_SCOPE_KEY]` (the same `PENDING_SCOPE_KEY`/`loop.run_in_executor` mechanism the HTTP `await_pending_corrections` middleware uses), RE-READS the message row, and broadcasts the FINAL (corrected/modified) content via `broadcast_message`. The HTTP `POST /send` and the WS `/ws` send path both call it; the WS path passes `request=websocket` to `persist_message` so the modifier stashes its executor future on the websocket scope and the handler awaits it directly (HTTP middleware does not run for websockets). A message with no `@ai` directive and correction off has nothing pending, so the broadcast is immediate with zero added latency. Frontend `static/js/MessagesLayout.js` reconciles the sender's own echo by `client_id` and replaces the optimistic bubble's `.rendered-content` with the server's authoritative `frame.content`, re-rendered - so the sender sees their `@ai` directive resolved in-place (and corrections become visible to the sender too). See "AI content correction" and "AI modifier".
|
||||
|
||||
## WS protocol frames
|
||||
|
||||
Client -> server: `{type:"send", receiver_uid, content, attachment_uids?, client_id}`, `{type:"typing", receiver_uid}` (throttled client-side), `{type:"read", with_uid}`, `{type:"presence", with_uid}` (one-shot presence query). Server -> client: `{type:"ready", user_uid}` (sent first on connect), `{type:"message", uid, sender_uid, sender_username, receiver_uid, content, created_at, time_ago, client_id}` (broadcast to BOTH sender and receiver sockets so multi-tab and the sender's own optimistic bubble reconcile via the echoed `client_id`), `{type:"typing", from_uid}` (to the receiver only), `{type:"read", by_uid}` (read-receipt to the other user), `{type:"presence", user_uid, online, last_seen}` (announced to everyone in an open conversation with the user on connect/disconnect, and as a direct reply to a `presence` query). The WS `read` path reuses `mark_conversation_read` (the same `UPDATE messages SET read=1` SQL the page uses) and `clear_messages_cache`.
|
||||
|
||||
## Renderer integration (XSS control)
|
||||
|
||||
Live/echoed message bubbles are NEVER injected as raw HTML. `MessagesLayout.buildBubble` creates a `<div class="rendered-content" data-render>`, sets its `textContent` to the message body, then runs `contentRenderer.applyTo(el)` + marks `img:not(.avatar-img)` with `data-lightbox` + `contentRenderer.highlightAll()` - exactly the `ContentEnhancer.initContentRenderer` sequence (emoji shortcode -> marked -> DOMPurify.sanitize -> highlight.js -> image/YouTube/autolink). This gives Discord-style emoji, image and YouTube embeds, autolinking, and sanitization for free.
|
||||
|
||||
## Frontend
|
||||
|
||||
`MessagesSocket.js` (mirrors `DeviiSocket.js`: reconnect with backoff, `4013` silent fast-retry, `onReady` fires on the first server frame not raw open) points at `/messages/ws`. `MessagesLayout.js` (instantiated on `app` in `Application.js`) opens the socket on the messages page, sends over WS with an optimistic pending bubble keyed by `client_id`, reconciles on the echoed `message` frame, appends incoming messages live through the renderer, auto-scrolls, throttles + shows the typing indicator, flips the read-receipt double-check, renders the presence dot / last-seen in the header, and live-updates the conversation-list preview + unread dot + ordering. If the socket is not open the send `<form method=POST>` submits normally (no-JS fallback). The template carries `data-self-uid`/`data-other-uid`/`data-other-online`/`data-other-last-seen` on `.messages-layout`. Styling (presence dot, typing dots, read-receipt, unread dot, pending opacity) is in `messages.css` using design tokens only, responsive down to 360px.
|
||||
|
||||
## Attachments stream live
|
||||
|
||||
The single `message_frame` builder (`services/messaging/persist.py`, used by BOTH `broadcast_message` and the relay) fetches `get_attachments("message", uid)` and includes a slimmed list (`uid`/`url`/`thumbnail_url`/`is_image`/`is_video`/`original_filename`/`file_size`/`mime_type`) on every frame. `MessagesLayout.renderAttachments` mirrors `_attachment_display.html` in the DOM (image -> `img.gallery-thumb` marked `data-lightbox`+`data-full`, video -> `<video>`, else a download link) - no `innerHTML`, so it stays XSS-safe. The sender's optimistic bubble shows an "Uploading attachment..." placeholder until its own echo arrives and swaps in the real gallery. **Attachment-only messages (empty caption) are allowed**: the content input dropped `required`, `sendViaSocket` sends when content OR attachments are present, and `persist_message` accepts empty content when `attachment_uids` is non-empty (`if not content and not attachment_uids: return None`). `dp-upload` (mode `attachment`) uploads each file to `/uploads/upload` and writes the uids into its hidden `attachment_uids` input; `collectAttachments()` reads that before the send, then `dp-upload.clear()` resets it. The chat uploader sets `hide-chips` so only the `(N)` count badge shows (no filename chips) - a generic `dp-upload` attribute. The send button shows a spinner and is disabled while busy: `MessagesLayout.refreshSendButton()` ORs `_uploading` (driven by the new `dp-upload:busy` event) with `_pendingSends` (a set of in-flight send `client_id`s, cleared on the echo or an 8s safety timeout), toggling `.is-sending` + `disabled` on `.messages-send-btn`.
|
||||
46
devplacepy/services/news/CLAUDE.md
Normal file
46
devplacepy/services/news/CLAUDE.md
Normal file
@ -0,0 +1,46 @@
|
||||
# News service (`devplacepy/services/news/`)
|
||||
|
||||
This file documents the automated developer-news import pipeline. Claude Code auto-loads it when a file under `devplacepy/services/news/` is read or edited.
|
||||
|
||||
## Overview
|
||||
|
||||
`BaseService` provides the async run loop, a `deque(maxlen=20)` log buffer, and graceful cancellation. `ServiceManager` is a singleton that registers, starts, and stops services. `NewsService` (`services/news/service.py`, `default_enabled=True`, registered unconditionally in `main.py`) is a fully automatic, zero-maintenance import pipeline: it fetches articles from `news_api_url`, **cleans** each (HTML strip + `clean_news_text`/`JUNK_PATTERNS` Reddit-boilerplate removal), **fetches and perceptually compares the images** (SSRF-guarded fetch + Pillow decode + `imagehash` phash off-thread; placeholder = too small / undecodable / a phash shared across 2+ different articles), **grades deterministically** (AI grade on cleaned text via `news_ai_url` + a reliability gate + a unique-image bonus and thin-content penalty -> effective `grade`, raw in `ai_grade`), **reformats every valid article into clean Markdown** (`_format_article` + the editable `news_format_prompt`; paragraphs, `## ` headings, lists and code, preserving every fact; fail-soft to the cleaned original, toggle `news_format_enabled`), and inserts ALL of them into `news` with `status="published"`/`"draft"` on `news_grade_threshold` - nothing is silently skipped. After the loop it **auto-rotates Featured + Landing** (`_apply_landing_selection`, top scored unique-image articles in the recent window) while honouring per-row `featured_locked`/`landing_locked` set when an admin manually toggles. **AI usage is metered like the correction/modifier consumers**: each gateway call's `X-Gateway-*` response headers are parsed (`parse_usage_headers`) and the run's totals accumulated into the durable single-row `news_usage` table (`database.add_news_usage`/`get_news_usage`, same SUMs-plus-computed-averages shape as `correction_usage`); `NewsService.collect_metrics()` surfaces calls/tokens/cost and the per-call averages as `stats` on the admin-only `/admin/services` page. The full ruleset is on `NewsService.description` and surfaces on `/admin/services`, which also polls live status and the log tail.
|
||||
|
||||
## Per-run flow
|
||||
|
||||
The per-run flow per article is: clean text -> fetch and perceptually compare images -> AI-grade the cleaned text -> reliability gate -> compute an effective score -> AI-reformat the body into Markdown (valid articles only) -> publish/Feature decision -> store -> then a single post-loop Landing rotation.
|
||||
|
||||
- Fetches `GET {news_api_url}` -> `{"articles": [...]}`; already-synced `external_id`s (from `news_sync`) are skipped.
|
||||
- Every article is graded via AI on the CLEANED text: `POST {news_ai_url}` with model `{news_ai_model}`, temperature 0. Both default to the internal gateway (`INTERNAL_GATEWAY_URL` / `molodetz`); the key falls back to `internal_gateway_key()` when `news_ai_key`/`NEWS_AI_KEY` is unset.
|
||||
- ALL articles are inserted into `news` regardless of grade (never silently skipped). Articles re-synced each run (upsert by `external_id`); grade, status, image, and images updated each cycle. Slugs via `make_combined_slug(title, uid)`.
|
||||
- All parameters (`news_api_url`, `news_ai_url`, `news_ai_model`, `news_grade_threshold`, `news_ai_key`, `news_format_enabled`, `news_format_prompt`, interval) are declared as `config_fields` and edited on the Services tab. The full grading ruleset is carried on `NewsService.description` (the `GRADING_RULES_DESCRIPTION` module string) and renders on `/admin/services`.
|
||||
|
||||
## AI reformatting (`_format_article`, `FORMAT_PROMPT_SPEC`)
|
||||
|
||||
After grading, every VALID article (one that passed the reliability gate and got a grade) has its body reformatted by the AI into clean Markdown - short paragraphs, `## ` section headings, bullet/numbered lists, and inline/fenced code - turning the source wall of text into a readable article. It reuses the grading endpoint/model/key (`news_ai_url`/`news_ai_model`/`_get_ai_key`) at `temperature 0.3`, `FORMAT_MAX_TOKENS=6000`, with the cleaned `description`+`content` (capped at `FORMAT_INPUT_MAX_CHARS=14000`) appended to the editable `news_format_prompt`. The prompt forbids inventing/removing facts and only restructures. The result is fence-stripped (`_strip_md_fence`), validated to be at least `MIN_BODY_CHARS`, capped at `FORMAT_OUTPUT_MAX_CHARS=30000`, and stored as the `news.content` (rendered server-side by `render_content`, the markdown engine, on `news_detail.html`). It is fail-soft: on any error, an empty/too-short result, or `news_format_enabled` off, the cleaned original content is stored unchanged. Toggle with the `news_format_enabled` bool config field.
|
||||
|
||||
## AI usage metering and stats (the shared `usage.py` helpers, `news_usage`, `collect_metrics`)
|
||||
|
||||
Service-level AI spend is metered exactly like the per-user `correction_usage`/`modifier_usage` consumers, and the accumulation/formatting plumbing is **shared, not copied per service**, in `services/openai_gateway/usage.py` beside `parse_usage_headers`: `USAGE_FIELDS` (the seven summed columns), `new_usage_totals()` (a zeroed totals dict), `accumulate_usage(totals, response)` (parses the `X-Gateway-*` headers via `parse_usage_headers` - defensive `getattr(response, "headers", None)`, so fake responses without headers are a no-op - and adds each field into the totals; `totals=None` is a no-op so the metering is opt-in per call site), and `usage_metric_cards(usage)` (the one stat-card list builder). For NewsService the grading and formatting gateway calls are its only AI spend: on every successful gateway response it calls `accumulate_usage(usage_totals, resp)` into a per-run dict from `new_usage_totals()`, and after the run `database.add_news_usage(usage_totals)` does ONE `INSERT ... ON CONFLICT DO UPDATE SET col = col + excluded.col` upsert into the durable single-row `news_usage` table (the same `_add_usage` helper as the correction/modifier tables, keyed on the constant `NEWS_USAGE_KEY="news"` in the `user_uid` column, ensured + unique-indexed in `init_db`). `database.get_news_usage()` (the same `_get_usage`) returns the running SUMs plus computed AVERAGES (avg tokens/call, avg cost/call, avg upstream latency, avg tokens/sec). `NewsService.collect_metrics()` is `{"stats": usage_metric_cards(get_news_usage())}`, which `BaseService._persist_state` serializes (every ~3s tick, regardless of the service's enabled state) and `ServiceMonitor.renderMetrics` shows on the admin-only `/admin/services` page - consistent with `BotsService` (live fleet cost) and the financial-data-is-admin-only rule (the services page is `require_admin`).
|
||||
|
||||
**The issue tracker is the second service consumer and reuses all of this 1:1:** its two AI call sites (`services/gitea/enhance.py` `enhance_ticket` for ticket filing and `services/gitea/planning.py` `generate_plan` for the planning document) take an optional `totals` dict and call `accumulate_usage(totals, response)` after `raise_for_status`; the owning job services (`IssueCreateService`, `PlanningReportService`) build a `new_usage_totals()`, pass it in, and `database.add_issue_usage(totals)` (constant `ISSUE_USAGE_KEY="issues"`, table `issue_usage`, ensured beside `news_usage`) when `totals["calls"]`. `IssueTrackerService.collect_metrics()` is `{"stats": usage_metric_cards(get_issue_usage())}`, so the aggregate ticket-AI spend shows on the Issue Tracker service page even though the poller defaults to disabled (the supervisor still ticks and persists metrics). To meter a NEW service AI call: thread a `new_usage_totals()` dict to the call site, `accumulate_usage` the response, persist with a matching `add_<x>_usage`/single-row `<x>_usage` table, and surface it via `collect_metrics` -> `usage_metric_cards`.
|
||||
|
||||
## Content cleaning (`clean_news_text`, `JUNK_PATTERNS`)
|
||||
|
||||
After `strip_html`, removes Reddit boilerplate (`submitted by /u/<user>`, `submitted by ... to /r/<sub>`, standalone `[link]`, `[comments]`, `[N comments]`, trailing `submitted by ... [link] [comments]`) and collapses whitespace. Applied to title (light), description, and content BEFORE grading and BEFORE storage.
|
||||
|
||||
## Image fetch + perceptual placeholder/uniqueness detection
|
||||
|
||||
Per article up to `IMG_PER_ARTICLE=5` candidate image URLs (from `_get_article_images`) are fetched through `net_guard.guarded_async_client` (SSRF-safe, re-guards redirects), capped at `IMG_FETCH_MAX_BYTES=5_000_000` and a short timeout; each is decoded with Pillow and phashed (`imagehash`) via `loop.run_in_executor` so the CPU work never blocks the loop, recording width/height. Reject as placeholder on decode/fetch failure or width/height `< MIN_IMAGE_DIMENSION=100`. Across the WHOLE batch, phashes within Hamming distance `PHASH_DISTANCE=5` (compared via `imagehash.hex_to_hash`) that span 2+ DIFFERENT articles are a shared logo/placeholder and are flagged placeholder for all of them. `has_unique_image` = at least one non-placeholder candidate; its URL becomes the article's `image_url`. Each `news_images` row stores `phash`/`width`/`height`/`is_placeholder`; the `news` row stores `image_url` and `has_unique_image`.
|
||||
|
||||
## Deterministic grading (constants at module top)
|
||||
|
||||
`MIN_TITLE_CHARS=12`, `MIN_BODY_CHARS=200`, `MAX_TITLE_CAPS_RATIO=0.6`, `MIN_IMAGE_DIMENSION=100`, `PHASH_DISTANCE=5`, `IMG_PER_ARTICLE=5`, `IMG_FETCH_MAX_BYTES=5_000_000`, `UNIQUE_IMAGE_BONUS=2`, `THIN_CONTENT_PENALTY=2`, `FEATURE_MIN_SCORE=8`, `LANDING_MIN_SCORE=9`, `LANDING_MAX=6`, `LANDING_RECENCY_DAYS=4`. Per article: (1) the raw AI grade (1-10) is stored in `ai_grade`; `None` is invalid. (2) The reliability gate (`reliability_reason`) forces draft (never Featured/Landing) when the url is empty/invalid, the cleaned title `< MIN_TITLE_CHARS`, the combined cleaned body `< MIN_BODY_CHARS`, or the title uppercase-letter ratio `> MAX_TITLE_CAPS_RATIO`; the reason is recorded in `news_sync.status` as `rejected_quality:<reason>` and in the audit metadata. (3) `effective_score = clamp(ai_grade + UNIQUE_IMAGE_BONUS if has_unique_image - THIN_CONTENT_PENALTY if body marginal-but-not-gated, 1..10)` (marginal = body `< MIN_BODY_CHARS * 2`); stored in `grade` (so `/news` ordering reflects final quality). (4) `status = "published"` when valid and `effective_score >= news_grade_threshold` else `draft`. (5) `featured = 1` when published AND `has_unique_image` AND `effective_score >= FEATURE_MIN_SCORE` (respecting the lock).
|
||||
|
||||
## Auto-rotating Featured + Landing with admin lock
|
||||
|
||||
After the per-article loop, `_apply_landing_selection` queries recent (`<= LANDING_RECENCY_DAYS`) rows that are published + featured + `has_unique_image` + NOT `landing_locked`, sorts by `(-grade, -synced_at)`, sets `show_on_landing=1` for the top `LANDING_MAX` with `grade >= LANDING_MIN_SCORE` and `show_on_landing=0` for the rest of THAT service-managed set (self-rotating). When an admin manually toggles Featured or Landing (`routers/admin/news.py`), the matching `featured_locked`/`landing_locked` is set to 1 so the service stops auto-managing that row; updates of a locked Featured row never overwrite `featured`. New `news` columns: `ai_grade`, `featured`, `has_unique_image`, `image_url`, `featured_locked`, `landing_locked`, `author`, `article_published` (all added to the `init_db` ensure-block - the stale-schema rule). Audit events `news.service.ingest`/`news.service.publish`/`news.service.draft`/`news.service.reject`/`news.service.landing` (`record_system`, category `news`). New dep: `imagehash` (`Pillow` already present).
|
||||
|
||||
## Editorial labels are admin-only on the public surfaces
|
||||
|
||||
The `★ Featured` badge and the `Grade {{ grade }}` chip are internal editorial signals, so on every PUBLIC news surface - the `/news` listing (`news.html`), the article page (`news_detail.html`), and the landing/home Developer News cards (`landing.html`) - both are wrapped in `{% if is_admin(user) %}`. Members and guests never see them; administrators still see them everywhere (including the public pages) and the admin **Manage News** area (`admin_news.html`) is unchanged, where the grade and featured state are the editorial controls. The article's source name and timestamp stay visible to everyone. Gating only the rendered labels does not touch the stored `grade`/`featured` columns, the grading/landing logic, or ordering (`/news` is still ordered by effective `grade`).
|
||||
90
devplacepy/services/openai_gateway/CLAUDE.md
Normal file
90
devplacepy/services/openai_gateway/CLAUDE.md
Normal file
@ -0,0 +1,90 @@
|
||||
This file documents the AI gateway subsystem (devplacepy/services/openai_gateway/) - the single point of truth for AI calls, usage metering, and provider/model routing. Claude Code loads it automatically whenever a file under this directory is read or edited.
|
||||
|
||||
## Overview
|
||||
|
||||
`GatewayService` (`services/openai_gateway/`) is an OpenAI-compatible LLM gateway (the ported `openai5.py`), mounted at `/openai/v1/*` (`routers/openai_gateway.py`, prefix `/openai`). It is a **service that serves an HTTP endpoint** rather than a loop.
|
||||
|
||||
- The router is thin: it calls `service_manager.get_service("openai").handle(request, subpath)`. `POST /v1/chat/completions` runs the full gateway (vision augment -> model override -> forward -> optional SSE re-emit via `_fake_stream`); `POST /v1/embeddings` forwards to the configured embeddings upstream (`handle_embeddings`, no vision/streaming); `/v1/{path:path}` is a transparent passthrough to the upstream base. Disabled -> 503, unauthorized -> 401, upstream connection failure -> 502.
|
||||
- `main.py` registers it and exempts `/openai` from the rate-limit middleware and the maintenance gate. No new dependency (`httpx` already required).
|
||||
- `GatewayService` is `default_enabled=True` so internal callers work out of the box.
|
||||
|
||||
## Per-call cost/token response headers
|
||||
|
||||
Every gateway response (chat, embeddings, passthrough; success and error) carries `X-Gateway-*` headers describing that single call: `Model`, `Backend`, `Prompt-Tokens`, `Completion-Tokens`, `Total-Tokens`, `Cache-Hit-Tokens`, `Cache-Miss-Tokens`, `Reasoning-Tokens`, `Cost-USD`/`Input-Cost-USD`/`Output-Cost-USD` (dollars), `Cost-Native` (1 if the upstream returned a native cost), `Tokens-Per-Second`, `Upstream-Latency-Ms`, and `Context-Window`/`Context-Utilization` when the model's window is known, plus the timing headers `X-Gateway-Upstream-Latency-Ms`/`X-Gateway-Total-Latency-Ms`. `GatewayUsageLedger.record(...)` RETURNS the computed ledger row (or `None` on failure); each handler's `finalize` closure maps it through `usage.usage_response_headers(row)` and attaches it to the returned `Response` (`resp_headers`). The single denied path with no upstream call (embeddings disabled) carries no headers. This is what lets any caller read its own spend - the AI correction worker reads `X-Gateway-Cost-USD`/token headers off its own correction call to accumulate per-user totals (see "AI content correction" in the root `CLAUDE.md`).
|
||||
|
||||
## Per-worker runtime
|
||||
|
||||
**Serves in every worker.** Config/enabled come from `site_settings` (via the service's `get_config()`/`is_enabled()`), so any uvicorn worker answers - not just the supervisor worker. Per-worker runtime (`GatewayRuntime`: `httpx.AsyncClient` pool + `asyncio.Semaphore` sized to `gateway_instances`, plus counters and a `VisionCache`) is created lazily on first request and rebuilt when `instances`/`timeout`/cache size change. `gateway_instances` is the **scaling knob** (concurrency per worker); process scaling is uvicorn workers.
|
||||
|
||||
## Auth
|
||||
|
||||
`authorize()` reuses `get_current_user` (cookie/X-API-KEY/Bearer/Basic). Allowed if `gateway_require_auth` is off, the presented X-API-KEY/Bearer equals the static `gateway_access_key` **or the auto-generated `gateway_internal_key`**, or the resolved user is an admin (`gateway_allow_admins`) or any user (`gateway_allow_users`).
|
||||
|
||||
**Per-user attribution:** the gateway also accepts a real user's `api_key` (Bearer / X-API-KEY / session), gated by `gateway_allow_users` (default **on**); `resolve_owner()` records `(owner_kind, owner_id)` per call, so usage is attributed and limitable per user. With `gateway_allow_users` off, only internal/access/admin keys work and per-user attribution never happens. Devii operating a signed-in user authenticates its LLM calls with that user's own `api_key` (set as the session's `ai_key` in `build_settings(..., owner_kind="user")`), so a user's full gateway spend (Devii and direct API calls) rolls up under their uid - surfaced admin-only on the profile page via `build_user_usage()` / `GET /admin/users/{uid}/ai-usage`. Guests keep the internal key.
|
||||
|
||||
## Config fields
|
||||
|
||||
All config is `config_fields` (upstream url/model/key, force-model, the Prompt group's `gateway_system_preamble`, timeout, instances, vision url/model/key/cache/toggle, the Embeddings group (`gateway_embed_enabled`/`_url`/`_model`/`_key`), the auth toggles + static key + internal key, plus the Pricing/Reliability/Tracking groups); live metrics (requests/errors/in-flight/vision-calls/embed-calls/latency plus 24h cost/tokens/success rollups) via `collect_metrics`.
|
||||
|
||||
## System-message composition (date awareness + operator preamble)
|
||||
|
||||
`system_message.py` (`apply_system_directives`) runs in `handle_chat` only (NOT `handle_embeddings`/`handle_passthrough`), after vision augmentation and before the upstream payload is assembled, so it reaches every chat consumer (news, bots, Devii guests, per-user direct API calls). It composes ONE system message in a fixed, cache-friendly order: **operator preamble (`gateway_system_preamble`) -> date line (`Current date: DD/MM/YYYY`) -> the client's original system content**.
|
||||
|
||||
Two behaviors:
|
||||
|
||||
- **Date awareness (EU, no time).** The current date is injected only when the composed text (preamble + client system content) contains NO date in any common format. Detection lives in `contains_date` via the `_DATE_PATTERNS` regex set: `YYYY-MM-DD` (ISO), `D/M/YYYY` and `DD/MM/YYYY` (also matches `MM/DD/YYYY`), `DD-MM-YYYY`, `DD.MM.YYYY`, and textual months (`14 June 2026`, `June 14, 2026`, `14 Jun 2026`, with abbreviations and optional ordinal/comma). The date is `datetime.now().strftime("%d/%m/%Y")` - **date only, never time**: omitting the time keeps the system message byte-stable for the whole day so upstream prompt/token caching stays effective.
|
||||
- **Operator preamble.** `gateway_system_preamble` (Prompt group, `type="text"` textarea, default empty) is prepended ahead of the client's system content on every chat call so all calls carry operator preferences. When the client sent NO system message, the composed content (preamble and/or date) becomes a brand-new leading system message; when the client DID send one, its content is replaced in place (preamble + date + original text). A blank/whitespace-only preamble with an already-dated request is a no-op (no empty system message is created). The date detection runs over the preamble too, so an operator preamble that already states a date suppresses the injected date line.
|
||||
|
||||
Embeddings/passthrough are untouched by this composition step.
|
||||
|
||||
## Usage, cost, latency, and reliability tracking
|
||||
|
||||
The gateway records one row per upstream call (chat, vision, passthrough) and surfaces per-hour and 24h analytics. The pieces:
|
||||
|
||||
- **`usage.py`** - `GatewayUsageLedger` writes to `gateway_usage_ledger` (one row per call, success and failure) and samples `gateway_concurrency_samples` each 30s tick. `normalize_usage` handles BOTH upstream shapes: DeepSeek (`prompt_cache_hit_tokens`/`prompt_cache_miss_tokens`, `completion_tokens_details.reasoning_tokens`) and OpenRouter (`prompt_tokens_details.cached_tokens`, native `cost`, `cost_details`). `compute_cost` PREFERS the upstream native `cost` (OpenRouter) and FALLS BACK to per-1M pricing for the chat backend (DeepSeek reports no cost); it also returns the input/output split (native cost is split by the modeled-rate share `input_cost / (input_cost + output_cost)` from the configured per-1M prices, not by token volume; a negative native cost is clamped to 0). `record()` is wrapped in try/except so a tracking failure never breaks the proxy.
|
||||
- **`gateway.py`** - `_send` is the gated/timed/retried path: a `CircuitBreaker` (`reliability.py`) short-circuits when open; `retry_send` retries timeouts/connection errors/5xx with linear backoff; semaphore acquire time is the `queue_wait_ms`; an httpx per-request `trace` extension captures `connect_ms` (0 on pooled reuse). `handle_chat`/`handle_passthrough` record every outcome with timings, tokens, cost, params, and the resolved owner. Vision calls record from `VisionAugmenter._describe_one`.
|
||||
- **Owner attribution** - `GatewayService.resolve_owner` maps the caller to `(owner_kind, owner_id)`: `internal:devii` (internal key), `key:access` (static key), `user:<uid>`/`admin:<uid>` (session or API key), else `anonymous`.
|
||||
- **`analytics.py`** - `build_analytics(hours, top_n, pricing)` pulls the bounded window once (clamped to `MAX_WINDOW_HOURS=168`) and computes everything in Python: volume/throughput, tokens (sums + avg + p50/p90/p95/p99 + max via `reliability.percentile`), latency (upstream/overhead/queue/connect/total + tokens/sec), errors by category, cost (per model/caller, input vs output, projected monthly, caching savings), behavior, and an hourly breakdown. `summary_metrics()` runs cheap SQL aggregates for the live service panel. TTFT/inter-token are returned `null` (gateway forwards non-streaming).
|
||||
- **Admin** - `/admin/ai-usage` (HTML, `templates/ai_usage.html` + `static/js/AiUsageMonitor.js`) and `/admin/ai-usage/data?hours=&top_n=` (JSON) in `routers/admin/` package, admin-gated. Schema `GatewayUsageOut`. Devii action `ai_usage` and docs `admin-ai-usage` expose the same endpoint. The `ai_usage` and `site_analytics` Devii catalog tools are both `requires_admin=True`.
|
||||
- **Per-user usage** - `analytics.build_user_usage(owner_id, hours=24, pricing)` filters the ledger by `owner_id` (a uid matches both `user:` and `admin:` rows) and returns requests, success/error %, token totals, cost (window / per-hour / per-request / 30-day projection = 24h spend x 30), avg latency + tps, per-model breakdown, and an hourly cost series. Served admin-only at `GET /admin/users/{uid}/ai-usage` (schema `UserAiUsageOut`, docs `admin-user-ai-usage`) and rendered on the profile page by `static/js/UserAiUsage.js` (the `[data-ai-usage]` card, gated `{% if is_admin(user) %}` in `templates/profile.html`). The 24h window is a sound projection basis (unlike a startup burst), so `cost_24h * 30` is honest.
|
||||
- **Profile visibility split** - `routers/profile/` package `_ai_quota(uid, include_cost=False)` builds the profile quota bar. **The displayed spend is the user's REAL total gateway spend, not the Devii-turn subtotal:** `spent_usd`/`used_pct`/`requests`/`last_used` come from `analytics.user_spend_24h(uid)`, the SAME `gateway_usage_ledger` source (summed by `owner_id`, matching both `user:` and `admin:` rows) that feeds the stat grid's `cost.window_usd`. So the server-rendered `spent_usd` is byte-for-byte the card's "Cost 24h" and the two halves of the card reconcile. `turns` is still the `devii_usage_ledger` `turns_24h` count (Devii conversation turns) and is labeled **"Devii turns"** in the template precisely because it is a *different metric* from the grid's total **API requests** (one Devii turn fans out into several gateway requests; direct API-key calls add requests with no turn, which is why `0 Devii turns` can sit beside `18 API requests`). The `limit`/`turns` still come from the Devii service (`daily_limit_for`/`turns_24h`); an **unlimited** limit (`<= 0`, the admin default) sets `unlimited=True` so the template renders "no cap" / "No daily spend cap" instead of a misleading `/ $0.00`. **This is display-only: the Devii daily-cap ENFORCEMENT and `/devii/usage` still read the Devii-scoped `devii_usage_ledger.spent_24h`** (per-turn, resettable, and the only per-guest spend record since guest Devii calls hit the gateway under the shared internal key, not the guest cookie). The profile page renders three ways: **admin** sees the full gateway stats card *plus* a server-rendered quota bar (with spend/limit/Devii turns); the **owner** (non-admin, `is_self`) sees a quota-**only** card showing just the percentage; everyone else sees nothing AI-related. `_ai_quota` is fail-safe (returns `None` on any error or when the Devii service is unregistered) and the result rides on `ProfileOut.ai_quota`. **Dollar figures (`spent_usd`/`limit_usd`) are included only when `include_cost=viewer_is_admin`** - never for a non-admin owner. This matters because the route serves the same context as JSON via `respond(..., model=ProfileOut)` and `ProfileOut.ai_quota` is a free-form dict, so hiding dollars in the HTML branch alone would still leak them to a member requesting their own profile with `Accept: application/json`; the server-side omission is the real control. Non-admins (HTML and JSON) get only `used_pct`/`turns`/`requests`.
|
||||
- **Retention** - `run_once` prunes both tables past `gateway_usage_retention_hours` (default 720 = 30 days).
|
||||
|
||||
**Financial data is admin-only everywhere.** Any monetary figure (USD cost, pricing, spend, limit) is restricted to administrators; members and guests see only the percentage of quota used - this rule is enforced consistently across the profile card, `ai_correction`/`ai_modifier` usage displays, and the Devii cost tools.
|
||||
|
||||
## Embeddings
|
||||
|
||||
`POST /openai/v1/embeddings` exposes an OpenAI-compatible text-embeddings model. Clients send the generic model `molodetz~embed` (`config.INTERNAL_EMBED_MODEL`), which `handle_embeddings` remaps to `gateway_embed_model` exactly like chat remaps `molodetz` -> `gateway_model` (also remapped when `gateway_force_model` is on or the model is empty). It defaults to OpenRouter's `qwen/qwen3-embedding-8b` at `https://openrouter.ai/api/v1/embeddings` (`config.EMBED_*_DEFAULT`, $0.01 per 1M input tokens). `handle_embeddings` mirrors `handle_chat` but is simpler: no vision augmentation and no streaming - build the payload, forward via `_send`, and record one ledger row through the same `finalize(...)` closure. The config fields are the **Embeddings** group (`gateway_embed_enabled` default on, `gateway_embed_url`, `gateway_embed_model`, `gateway_embed_key`) plus the Pricing-group `gateway_embed_price_input_per_m`. `effective_config()` falls the embed key back to `gateway_vision_key` then `OPENROUTER_API_KEY` (NOT `gateway_api_key`: that is the DeepSeek chat upstream key, whereas embeddings target OpenRouter like vision does). Usage is recorded with **`backend="embed"`**; `usage.compute_cost` adds an `embed` branch (input-only, completion always 0, native OpenRouter `cost` still preferred) and `Pricing` gained `embed_input_per_m`. `analytics.py` groups by `backend` generically, so embed rows roll up automatically; `caching_savings` counts only **non-native** chat rows (native-priced rows did not use the configured cache-hit/miss rates, so folding them in would report a fictional saving). When `gateway_embed_enabled` is off the endpoint returns 503 with no ledger row.
|
||||
|
||||
## Single point of truth for AI
|
||||
|
||||
The gateway is the only place that holds real provider URLs/models/keys. Every other AI consumer (news, bots, Devii guests) points at it by default and never touches a provider key:
|
||||
|
||||
- Defaults live in `config.py`: `INTERNAL_GATEWAY_URL` (`http://localhost:10500/openai/v1/chat/completions`, override with `DEVPLACE_INTERNAL_BASE_URL`) and `INTERNAL_MODEL` (`molodetz`). `news_ai_url`, `bot_api_url`, `devii_ai_url` default to `INTERNAL_GATEWAY_URL`; their model defaults to `molodetz`.
|
||||
- Each consumer's key falls back to `database.internal_gateway_key()` (reads the `gateway_internal_key` setting) when its own key field/env is unset - the provider-key fallbacks (`DEEPSEEK_API_KEY`/`OPENROUTER_API_KEY`) were removed from news and bots.
|
||||
- `gateway_force_model` (default on) and a `molodetz`/empty alias in `handle_chat` make the upstream always receive `gateway_model`, so `molodetz` is a stable generic alias.
|
||||
- `database.migrate_ai_gateway_settings()` (called at the end of `init_db()`, under the startup `init_lock`): generates `gateway_internal_key` (uuid4) if missing; migrates `DEEPSEEK_API_KEY`/`OPENROUTER_API_KEY` env into `gateway_api_key`/`gateway_vision_key` when the db value is empty; and rewrites any consumer AI URL still equal to the old `openai.app.molodetz.nl` default to the gateway, plus `bot_model` `deepseek-chat` -> `molodetz` (only uncustomized values).
|
||||
- The gateway's `gateway_api_key`/`gateway_vision_key`/`gateway_internal_key` fields are **non-secret** so the admin services page shows the value actually in use, editable.
|
||||
- Bot LLM calls are synchronous `urllib` but already run via `asyncio.to_thread` (`bot/bot.py`), so the local round-trip never blocks the event loop.
|
||||
- Real provider keys/URLs/models live ONLY in the gateway. The gateway is `default_enabled=True`.
|
||||
|
||||
## Provider and model routing (`routing.py`, admin `/admin/gateway`)
|
||||
|
||||
Layered ON TOP of the single-provider service config above, which stays THE implicit `default` provider (env migration, internal key, `molodetz`/`molodetz~embed`, vision, embeddings, ledger all unchanged).
|
||||
|
||||
**Storage.** Two dataset tables, ensured in `init_db` via `routing.ensure_tables`, cross-worker cache-invalidated under the `"gateway_routing"` cache-version name (module-level `provider_store`/`model_store` over a shared `_ROUTING_CACHE`; writes `bump_cache_version` and clear the cache). They are admin config, NOT in `SOFT_DELETE_TABLES` - hard CRUD, mirroring `site_settings`:
|
||||
|
||||
- `gateway_providers` - named upstreams: `name` + `base_url` (chat-completions URL) + `api_key` + `is_active`; the embeddings URL is derived by swapping `/chat/completions` -> `/embeddings`.
|
||||
- `gateway_models` - source->target routes: `source_model` (unique, what clients request) -> `provider` (blank = default) + `target_model` + `kind` (chat|embed) + optional `vision_provider`/`vision_model` for the **text+vision merge** (when set, image content is described by that vision model before forwarding) + `context_window` + its **own economy** (`price_cache_hit_per_m`/`price_cache_miss_per_m`/`price_output_per_m`/`price_input_per_m`, USD per 1M tokens) + `is_active`.
|
||||
|
||||
**Resolution is a per-request overlay, not a fork.** At request time `routing.chat_overlay(requested, cfg)` / `routing.embed_overlay(requested, cfg)` resolve an active route by the requested model name and return a per-request OVERLAY dict of `gateway_*` cfg keys (`gateway_force_model`+`gateway_model`=target, `gateway_upstream_url`/`gateway_api_key` from the provider, the price keys, an augmented `gateway_model_context_map`, and vision overlay keys); `handle_chat`/`handle_embeddings` merge it onto the base cfg (`cfg = {**cfg, **overlay}`) BEFORE everything else, so the existing model-selection / `pricing_from_cfg` / `parse_context_map` / vision / url+key paths transparently use the route's provider, target model, pricing, vision model and context window. `_ensure` (the httpx pool / semaphore / breaker / vision cache) reads only the non-overlaid pool keys, so the connection pool is never churned per request.
|
||||
|
||||
**No matching route = `None` overlay = byte-identical legacy behavior** - this is the "nobody feels the transformation" guarantee: `molodetz`/`molodetz~embed` and every existing caller are byte-identical when no route matches. A route with a blank provider overlays only model+pricing(+vision), keeping the default upstream url/key.
|
||||
|
||||
**CRUD.** Admin JSON at `/admin/gateway/{providers,models}` (`routers/admin/gateway_configs.py`, `require_admin`, Pydantic `ProviderIn`/`ModelRouteIn` validation, accepts both JSON from `static/js/GatewayAdmin.js` and form from Devii), audited under `gateway.provider.*`/`gateway.model.*` (category `ai`), included in the admin package with the page at `/admin/gateway` (`templates/admin_gateway.html`, sidebar link, `admin_section="gateway"`).
|
||||
|
||||
**Devii tools:** `gateway_providers`/`gateway_models` (read) and `gateway_provider_set`/`gateway_provider_delete`/`gateway_model_set`/`gateway_model_delete` (admin; the two deletes are `CONFIRM_REQUIRED`).
|
||||
|
||||
When adding a routed value, overlay it as the matching `gateway_*` cfg key so the runtime needs no new branch.
|
||||
|
||||
**Tests:** `tests/unit/services/openai_gateway/routing.py` (overlay/economy/kind isolation), `tests/unit/services/openai_gateway/gateway.py::test_model_route_overrides_upstream` (end-to-end through `handle_chat`), and `tests/api/admin/gateway/` (admin CRUD + validation + role gating).
|
||||
33
devplacepy/services/pubsub/CLAUDE.md
Normal file
33
devplacepy/services/pubsub/CLAUDE.md
Normal file
@ -0,0 +1,33 @@
|
||||
# Pub/Sub service (`devplacepy/services/pubsub/`, `devplacepy/routers/pubsub.py`)
|
||||
|
||||
This file documents the database-free publish/subscribe bus. Claude Code auto-loads it when a file under `devplacepy/services/pubsub/` is read or edited.
|
||||
|
||||
## Overview
|
||||
|
||||
`pubsub.py` (router, mounted at `/pubsub`) is a **database-free** publish/subscribe bus. `WS /pubsub/ws` (subscribe/unsubscribe/publish frames, `foo.*` wildcards), `POST /pubsub/publish` + `GET /pubsub/topics` (admin/internal). Lock-owner-gated WS (`4013` retry) so subscribers converge on one worker; topic authz in `services/pubsub/policy.py` (`user.{uid}.*` private, `public.*` shared, admin/internal anywhere, guests opt-in). In-process `services.pubsub.publish(topic, data)` for backends; frontend `app.pubsub`.
|
||||
|
||||
A general-purpose, **database-free** publish/subscribe bus mounted at `/pubsub`, for live broadcast between frontend tabs, background services, and Devii without polling.
|
||||
|
||||
## Hub (`services/pubsub/hub.py`)
|
||||
|
||||
`PubSubHub` keyed by topic pattern -> live websockets (no persisted buffer; pub/sub is fire-and-forget). `subscribe/unsubscribe/drop_socket`, `async publish(topic, frame) -> delivered` (drops dead sockets), `topics()`. Wildcards: a subscription to `foo.*` matches `foo` and `foo.bar` (`topic_matches`). Module singleton `pubsub`; `services.pubsub.publish(topic, data)` is the in-process helper for backends.
|
||||
|
||||
## Single-worker model
|
||||
|
||||
The `WS /pubsub/ws` is gated to the service lock-owner worker with the `4013` fast-retry close code (like `/devii/ws`), so every subscriber transparently converges on one worker and the in-process fan-out reaches everyone with no broker and no DB. Deliberate, documented trade-off: ephemeral and best-effort (a crash drops in-flight messages), matching the `background` queue's durability stance.
|
||||
|
||||
## Topic authz (`services/pubsub/policy.py`)
|
||||
|
||||
`resolve_actor` reuses the dbapi caller (admin/internal) then falls back to user/guest. `can_subscribe`/`can_publish`: admin/internal anywhere; users publish only to their own `user.{uid}.*` namespace and subscribe to that plus `public.*`; publishing to `public.*` is admin/internal only (prevents cross-user spam); guests get read-only `public.*` when `pubsub_allow_guests` is on. Topic names match `^[A-Za-z0-9_.*-]{1,128}$`; payloads capped at 64 KB.
|
||||
|
||||
## HTTP + introspection
|
||||
|
||||
`POST /pubsub/publish` (admin/internal; returns `409` on a non-owner worker, retry), `GET /pubsub/topics` (admin/internal). Admin/internal publishes record a `pubsub.publish` audit event.
|
||||
|
||||
## Frontend
|
||||
|
||||
`app.pubsub` (`static/js/PubSubClient.js`) is a lazy reconnecting client: `app.pubsub.subscribe(topic, cb)` / `app.pubsub.publish(topic, data)`, `4013`-aware fast retry, re-subscribes on reconnect, client-side wildcard dispatch.
|
||||
|
||||
## Service + nginx
|
||||
|
||||
`PubSubService` (`default_enabled=True`, no-op run loop) surfaces the `pubsub_allow_guests` toggle on `/admin/services` and gates the WS via `is_enabled()`. `/pubsub/ws` has its own nginx upgrade location.
|
||||
41
devplacepy/services/telegram/CLAUDE.md
Normal file
41
devplacepy/services/telegram/CLAUDE.md
Normal file
@ -0,0 +1,41 @@
|
||||
This file documents the Telegram bot subsystem. Claude Code auto-loads it when a file under `devplacepy/services/telegram/` is read or edited.
|
||||
|
||||
## Telegram bot (`services/telegram/`)
|
||||
|
||||
Puts Devii on Telegram. `TelegramService(BaseService)` (`services/telegram/service.py`, `default_enabled=False`, registered in `main.py`) supervises ONE long-poller subprocess (`python -m devplacepy.services.telegram.worker`). Because `BaseService` runs only on the service-lock owner, there is always exactly one poller - mandatory, since Telegram returns **409 Conflict** on concurrent `getUpdates` per token (a load-bearing invariant). The bot token is a **secret `ConfigField`** passed to the subprocess via the `TELEGRAM_BOT_TOKEN` env var, never baked or stored outside `site_settings`.
|
||||
|
||||
### Worker (`worker.py`, `backend.py`)
|
||||
|
||||
A pure Telegram I/O gateway: long-polls `getUpdates`, handles every Telegram API call, rate-limits, honours `429 retry_after`, treats `409` as transient backoff, and self-exits on stdin EOF (parent death). It imports only stdlib + `devplacepy.stealth` + `format.py` - **no FastAPI app / database import** (verify with `python -c "import devplacepy.services.telegram.worker"`). It speaks NDJSON over stdin/stdout: inbound `{type:"message", chat_id, from_id, text, images:[data-uri]}` on stdout, outbound `{cmd:"send"|"edit"|"chat_action", req_id, ...}` on stdin, plus `{type:"result", req_id, message_id}` and `{type:"log"}` frames. Private chats only. A sent photo is downloaded and inlined as a base64 `data:` URI (capped by `TELEGRAM_IMAGE_MAX_BYTES`); HTTP goes through the stealth client. `backend.py` has a `TelegramBackend` ABC + `HttpTelegramBackend` (real) + `FakeTelegramBackend` (tests).
|
||||
|
||||
### Bridge (`bridge.py`)
|
||||
|
||||
Runs in the parent (co-located with the Devii hub on the lock owner) and drives Devii **in-process** - NOT via the WebSocket, no WebSocket-from-subprocess. For each inbound message it resolves the chat to a pairing (`store.py`), then for a paired user opens a `hub.get_or_create("user", uid, ..., channel="telegram")` session, attaches a per-turn `TelegramConnection` (an object exposing `async send_json` - the same sink shape `session.attach` expects, reusing the `routers/devii.py` quota gate and the shared ledger), runs the same quota gate as `routers/devii.py` (`devii.daily_limit_for` / `spent_24h`, shared ledger), and calls `session.spawn_turn(content, audit_text=...)`. Turns serialize per chat via a lock and are bounded by a global semaphore. The connection sends a `Devii is thinking...` placeholder, refreshes a `typing` chat action every ~4s, throttle-edits the placeholder with tool progress, and on the `reply` frame edits it into the final answer (markdown -> Telegram HTML via `format.py`, split at 4096 with a plain-text fallback; overflow becomes extra messages) - so **a turn updates ONE Telegram message** (placeholder -> typing -> progress edits -> final answer) instead of spamming. Browser/avatar frames are resolved immediately as unsupported (no web client on Telegram). `_emit` frames never reach an open web terminal because the thread is its own channel.
|
||||
|
||||
### `channel="telegram"` isolation
|
||||
|
||||
A dedicated conversation thread, isolated from the web `main` thread, but `hub.get_or_create` is patched so this channel still uses the persistent `db` (shared lessons/behavior/tasks/quota with `main`) - `hub.get_or_create` treats `telegram` like `main` for `owned_db`, so lessons/behavior/tasks/quota are shared; only `devii_conversations` is keyed per channel. The scheduler only starts on `main`, so the Telegram session never double-runs it; scheduled-task pushes to Telegram go through the `telegram_send` tool from the user's `main` session.
|
||||
|
||||
### Pairing (`store.py`, `routers/profile/telegram.py`)
|
||||
|
||||
`POST /profile/{username}/telegram` (owner-or-admin, mirrors `ai_correction`) issues a single-use 4 digit code (sha256-hashed, unique among active codes, reissue invalidates prior, TTL `telegram_code_ttl_minutes`) into `telegram_pairings`, or unpairs. Sending the code to the bot binds `telegram_links` (chat_id+from_id -> user_uid); wrong attempts are throttled per chat in memory. Tables are ensured in `init_db`. Profile shows a Telegram card (`static/js/TelegramPairing.js`).
|
||||
|
||||
### `telegram_send` Devii tool
|
||||
|
||||
`services/devii/telegram/`, `handler="telegram"`, `requires_auth=True`, audited `telegram.send`. Looks up the paired chat, requires the service running on the lock owner, and delivers a markdown message. Use it for notifications from a turn or a scheduled task.
|
||||
|
||||
### Image vision (turn extension)
|
||||
|
||||
`session.spawn_turn`/`_run_turn` and `agent.respond` accept `str | list` for image vision; the bridge builds an OpenAI `[{type:text},{type:image_url,image_url:{url:<data-uri>}}]` content list, which the gateway `VisionAugmenter` describes when the route has a vision model. After the turn the structured message is redacted to a short `[image]` placeholder so `devii_conversations` never stores base64.
|
||||
|
||||
### Exact per-user cost (vision included)
|
||||
|
||||
The turn authenticates the gateway with the paired user's own api_key, so `resolve_owner` books both the chat call and the vision sub-call to `gateway_usage_ledger` under that user (authoritative). For Devii's own per-user ledger and 24h cap to also be exact, the gateway folds the vision sub-call cost into the chat response `X-Gateway-Cost-USD` (and exposes `X-Gateway-Vision-Cost-USD`), and `services/devii/llm.py` records that native cost via `cost.record_cost` instead of estimating from tokens (`CostTracker.has_native` makes `cost_usd()["total"]` the gateway's real cost). This applies to every gateway consumer, not just Telegram; with no gateway header it falls back to token-based pricing.
|
||||
|
||||
### Live logs and stats
|
||||
|
||||
Worker output streams to the pubsub topic `admin.services.telegram.logs` and the service detail page's live pane (`ServiceMonitor.appendLive`); it is never persisted. `collect_metrics` reports messages in/out, edits, errors, average response latency, uptime, and token presence.
|
||||
|
||||
### Audit and nginx
|
||||
|
||||
Audit events: `telegram.pair.request|success|failure`, `telegram.unpair`, `telegram.send` (category `telegram`). nginx note: the bot uses outbound long-polling, so no inbound WebSocket location is needed.
|
||||
21
devplacepy/services/xmlrpc/CLAUDE.md
Normal file
21
devplacepy/services/xmlrpc/CLAUDE.md
Normal file
@ -0,0 +1,21 @@
|
||||
# XML-RPC bridge (`devplacepy/services/xmlrpc/`, `devplacepy/routers/xmlrpc.py`)
|
||||
|
||||
This file documents the XML-RPC bridge. Claude Code auto-loads it when a file under `devplacepy/services/xmlrpc/` is read or edited.
|
||||
|
||||
## Overview
|
||||
|
||||
`xmlrpc.py` (the router, mounted at `/xmlrpc`) reverse-proxies XML-RPC calls to the forking XML-RPC bridge (`services/xmlrpc/`, supervised by `XmlrpcService` on loopback `config.XMLRPC_PORT`), which generates one XML-RPC method per documented REST endpoint from `docs_api.API_GROUPS` (`posts.create`, `feed.list`, ...). One struct of named params per call; auth via in-band `api_key`, `X-API-KEY`/`Bearer` header, or `http://user:pass@host/xmlrpc` Basic. Full introspection + `system.multicall`; REST errors become XML-RPC faults. Exempt from the rate limiter (enforced on the forwarded internal hop).
|
||||
|
||||
The whole REST surface is also exposed over XML-RPC at `/xmlrpc`, generated **automatically from `docs_api.API_GROUPS`** so it never drifts: add a documented endpoint and it becomes an XML-RPC method for free.
|
||||
|
||||
## Pieces
|
||||
|
||||
- **`registry.py`** walks `docs_api.API_GROUPS` and yields one `Method` per endpoint. Method name = the endpoint `id` with `-`->`.` (`posts-create` -> `posts.create`). It carries the HTTP method, path, auth, and the `param` list split by `location` (path/query/form). It also builds the `system.methodHelp` text and `system.methodSignature` from the same data.
|
||||
- **`bridge.py`** translates ONE XML-RPC call into an internal REST request. The single struct argument is a dict of named params; the bridge substitutes `{name}` path segments, routes the rest to query (GET) or form (write methods), sets auth headers, and calls `config.INTERNAL_BASE_URL` with a **synchronous** `stealth.stealth_sync_client` (the forking server is sync). It returns the decoded JSON; any REST `status >= 400` becomes an `xmlrpc.client.Fault` whose `faultCode` is the HTTP status. A missing required param faults before the call.
|
||||
- **`server.py`** is `ForkingXMLRPCServer(ForkingMixIn, BridgeDispatcher, SimpleXMLRPCServer)`, `allow_none=True`, with `register_introspection_functions()` + `register_multicall_functions()` and one registered function per generated method. The custom request handler captures auth from the POST into `server.current_auth` (safe because each request runs in its own forked child) and the bridge forwards it: `X-API-KEY` and `Bearer` become an api-key header; a raw `Authorization: Basic` header is **forwarded verbatim** so the REST layer does its normal `username/email:password` login (`utils._user_from_basic`) - this is what makes `xmlrpc.client.ServerProxy("http://user:pass@host/xmlrpc")` work. It also forwards `X-Real-IP`/`X-Forwarded-For` so **rate limiting is attributed to the real caller** on the internal REST hop. Run standalone with `python -m devplacepy.services.xmlrpc.server` or the `devplace-xmlrpc` console script.
|
||||
- **`XmlrpcService(BaseService)`** (`services/xmlrpc/__init__.py`) supervises the server as a **subprocess** (never fork the asyncio/uvicorn process): `on_enable` spawns, `run_once` health-checks and respawns, `on_disable` terminates. `default_enabled=True`, but like every service only the lock-owner worker runs it, so exactly one process binds the port.
|
||||
- **`routers/xmlrpc.py`** reverse-proxies `/xmlrpc` (POST = RPC, GET = usage banner) to the forking server on `config.XMLRPC_PORT`, forwarding the auth + real-IP headers. `/xmlrpc` is **exempt from the request rate limiter** in `main.py` (enforcement happens on the forwarded internal hop). nginx forwards `/xmlrpc` in production with longer timeouts.
|
||||
|
||||
## Auth
|
||||
|
||||
Auth has three equivalent forms: `api_key` inside the param struct (works with a vanilla `ServerProxy`), an `X-API-KEY`/`Bearer` header on the transport, or HTTP Basic via a credentialed URL `http://username:password@host/xmlrpc`. Public endpoints need none. Config: `config.XMLRPC_BIND`/`XMLRPC_PORT` (loopback, `10550`). Documented at `/docs/xmlrpc.html`.
|
||||
94
devplacepy/static/js/CLAUDE.md
Normal file
94
devplacepy/static/js/CLAUDE.md
Normal file
@ -0,0 +1,94 @@
|
||||
This file documents JS module organization, custom web components, and shared frontend utilities under `static/js/`. Claude Code auto-loads it whenever a file under this directory is read or edited.
|
||||
|
||||
## JS module organization (hard rules)
|
||||
|
||||
- All JS in `static/js/` is ES6 modules, one class per file, instantiated as `app` and reachable everywhere. The browser-side application root is `Application.js`.
|
||||
- Javascript files must be in module (ES6) format and imported as module format. It must be as object oriented as possible and a file per class. There must always be a main application class called `Application`, instantiated as `app`, accessible everywhere.
|
||||
- Never use JavaScript 3rd party frameworks unless specified.
|
||||
- CDN scripts referenced from `base.html` (marked, highlight.js, emoji-picker-element) MUST use `defer` or `type="module"` - otherwise `wait_until="domcontentloaded"` in Playwright tests will time out.
|
||||
- **Static asset URLs are boot-versioned for cache-busting.** Assets are served `public, immutable, max-age=31536000` (1 year, for the Lighthouse "efficient cache policy" score) while a restart still busts every browser. Never hardcode a bare `/static/...` href/src: templates wrap it in the `static_url(path)` Jinja global (`templating.py`) and runtime JS that builds an absolute static URL uses `assetUrl(path)` (`static/js/assetVersion.js`, reads `<meta name="asset-version">` rendered in `base.html`). Both emit a `/static/v<STATIC_VERSION>/...` path segment (`config.STATIC_VERSION` = `DEVPLACE_STATIC_VERSION` env or `int(time.time())` at process start), so a deploy = a restart = a new timestamp = a new URL for every asset. Both helpers no-op for non-`/static/` paths and for `/static/uploads/`, so they are safe to wrap around dynamic values.
|
||||
- **Path segment, never a query string - do NOT switch this to `?v=`.** The frontend is unbundled ES6 with ~130 relative imports (`./Http.js`) and zero absolute ones. A `?v=` only versions the entry `<script>`; its transitive relative imports would resolve to unversioned URLs and stay frozen under `immutable`, so a deploy would not propagate JS for a year. A path segment is inherited automatically by the entire module graph and by relative CSS `url()`.
|
||||
- **Serving:** `main.py` mounts a `CachedStaticFiles` at `/static/v{STATIC_VERSION}` (immutable 1y; `service-worker.js` -> `no-cache`) plus the plain `/static` mount as the unversioned fallback; nginx mirrors this with `location ~ ^/static/v\d+/` (immutable 1y) and a short `max-age=3600` on the unversioned `/static/`. `/static/uploads/` (user content, stable DB paths) is never versioned; `service-worker.js` keeps a stable unversioned URL with `no-cache` so its registration scope is stable and a new worker always deploys.
|
||||
- **Multi-worker:** the version is captured **once at launch** and shared via `DEVPLACE_STATIC_VERSION` (set in `make prod` = `$(date +%s)` and the Dockerfile's `sh -c` CMD) - otherwise two workers could compute timestamps a second apart and serve mismatched versioned mounts (a 404 on the other worker). Unset in dev (`--reload`) so each reload refreshes it.
|
||||
|
||||
## Custom web components (`static/js/components/`)
|
||||
|
||||
Self-contained, presentational UI is built as custom elements with the `dp-` prefix:
|
||||
`dp-avatar`, `dp-code`, `dp-content`, `dp-title`, `dp-toast`, `dp-dialog`, `dp-context-menu`, `dp-upload`, `dp-lightbox`.
|
||||
|
||||
Conventions:
|
||||
|
||||
- Each extends `Component` (`static/js/components/Component.js`), a thin `HTMLElement` base with `attr(name, fallback)`, `boolAttr(name)`, `intAttr(name, fallback)` helpers.
|
||||
- **Light DOM only** (no `attachShadow`), so the site's global CSS applies - matching the existing `devii-*` elements. `dp-dialog`/`dp-context-menu` reuse the existing `.dialog-*`/`.context-menu*` styles in `components.css`; `dp-toast` styles are in `components.css` (`.dp-toast-host`/`.dp-toast`).
|
||||
- Each file self-registers via `customElements.define` at the bottom. `components/index.js` imports them all and is imported by `Application.js`, so every element is defined on every page (including `/docs`).
|
||||
- The behavioural singletons are created once in `Application.js` and exposed as `app.dialog`, `app.contextMenu`, `app.toast`. The `dp-dialog`/`dp-context-menu` elements keep the exact pre-existing `Dialog`/`ContextMenu` APIs (`confirm`/`prompt`/`alert`/`open`, `attach`/`open`/`close`) so existing callers (`ProjectFiles`, `ApiKeyManager`, `ModalManager`, `PushManager`, `ApiTester`) are unchanged. `Avatar`/`CodeBlock`/`Toast`/`ContentRenderer` remain as static/service modules (widely imported); the matching elements (`dp-avatar`, `dp-code`, `dp-content`) reuse them. `dp-content` wraps the shared `contentRenderer` engine that also powers the `data-render` attribute (driven by `ContentEnhancer`) - the engine stays a reusable service, the element is its component face. `dp-content` captures its raw markdown source before `contentRenderer.applyTo` overwrites `textContent`, then mounts a hover/focus reveal `.content-copy-btn` in its top-right corner that copies that original source to the clipboard (mirroring `CodeBlock.copyButton`); CSS lives beside `.code-copy-btn` in `docs.css`.
|
||||
- **`dp-content` / `dp-title` are CLIENT-only now (live contexts).** `dp-content` (`AppContent.js`) and its inline companion `dp-title` (`AppTitle.js`, `contentRenderer.renderInline`) render markdown/emoji/media (and, for title, inline-only with an inline-tag allowlist) on the client from `textContent`. **They are no longer used for server-rendered content/titles** - those moved to the backend `render_content`/`render_title` globals (see the root CLAUDE.md "Content rendering pipeline" section) for SEO. The components remain defined and are used only where content is generated client-side after load: `dp-content` in the DeepSearch chat (`AppDeepsearchChat.js`) and the planning report (`PlanningGenerator.js` -> `admin_issues_planning.html`). Keep them for those live cases and future use; do NOT wrap server-known content/titles in them. Both are now used ONLY for live/client-generated content (chat, planning report, dynamically inserted bubbles); server-rendered content/titles use the backend `render_content`/`render_title` globals instead. The components are intentionally kept for those live contexts and future use.
|
||||
- **What is NOT a component (by design):** partial-bound controllers (`ReactionBar`, `VoteManager`, `BookmarkManager`, `PollManager`, `CommentManager`, etc.) enhance server-rendered `_*.html` markup rather than render standalone UI; pure utilities (`Http`, `DomUtils`, `FormManager`, `Poller`, `JobPoller`, `OptimisticAction`) have no UI. These stay plain ES6 modules. The four optimistic engagement controllers extend the shared `OptimisticAction` base - see "Shared frontend utilities" below. Self-contained presentational widgets become components; partial-bound controllers (votes, reactions, comments) and pure utilities stay plain modules because they enhance server-rendered markup rather than render standalone UI.
|
||||
- Documented in the public docs **Components** section (`templates/docs/component-*.html`, registered under `SECTION_COMPONENTS` in the `routers/docs` package). Each page has a `data-render` markdown block (with example markup HTML-escaped as `<dp-...>`, since `data-render` re-parses `textContent` and would otherwise instantiate the example) plus a live demo in a separate block whose `<script type="module">` imports its own component module (it runs before `Application.js`).
|
||||
- **`dp-lightbox` (`AppLightbox.js`, `app.lightbox`, `static/css/lightbox.css`)** is the single image lightbox. It is **attribute-wired, not call-wired**: the singleton installs ONE delegated `document` click listener and opens any clicked `img[data-lightbox]` (showing `data-full` when present, else the image's own src; `alt` as caption; ignored when the image sits inside an `<a>`). To make a thumbnail open it, just add `data-lightbox` - never attach a click handler. Attachment thumbnails carry `data-lightbox`+`data-full` (`_attachment_display.html`); markdown content images are marked centrally in `ContentEnhancer.initContentRenderer` (one place covers every `data-render` surface), so embedded images are always clickable. The old hand-rolled `.attachment-lightbox` markup/CSS was dead (never wired) and has been removed.
|
||||
- `dp-upload` (the single file-upload button used everywhere) replaced the old `AttachmentUploader`; see "The upload button (`dp-upload`)" below.
|
||||
|
||||
## Shared frontend utilities (do not re-implement these)
|
||||
|
||||
A small set of plain ES6 modules under `static/js/` own the cross-cutting patterns so feature code stays tiny. Reach for these instead of hand-rolling a loop, a fetch, or a click handler. `Http` (`static/js/Http.js`) is the single fetch helper (`getJson`, `sendForm`, and `send` which throws `error.message` on non-2xx or a `200 {ok:false}` body); every live-update loop uses `Poller` (`new Poller(fn, intervalMs, {pauseHidden})`); async-job status polls use `JobPoller.run(statusUrl, {onDone, onFailed, onTimeout})` (`ProjectForker`, `ZipDownloader`); click-to-POST engagement controllers (`VoteManager`/`ReactionBar`/`BookmarkManager`/`PollManager`) extend `OptimisticAction` and call `this.submit(url, params, errorTarget, render)`. Floating windows (container terminals and Devii) extend `FloatingWindow`. **Scroll restoration is `ScrollMemory`** (`static/js/ScrollMemory.js`, `app.scrollMemory`): per-tab (sessionStorage) positions keyed by exact `path+search`, restored ONLY on `back_forward`/`reload` navigations or a click on `a.back-link`/`[data-scroll-back]`/breadcrumb/previous-trail-URL links, applied via a layout-stable rAF loop that aborts on user input; it sets `history.scrollRestoration = "manual"` site-wide and upgrades query-less back-link hrefs to the exact previous URL - mark any "back to X" anchor with the `back-link` class and never hand-roll scroll persistence.
|
||||
|
||||
Detail on each utility:
|
||||
|
||||
- **`Http` (`static/js/Http.js`, global `window.Http`).** The single HTTP helper. `getJson(url)` (GET -> JSON, throws on non-2xx); `sendForm(url, params)` (POST form-encoded, follows the `/auth/login` redirect via `Http.toLogin()`, throws a bare status on failure, returns JSON); `send(url, params)` (POST form-encoded that throws `data.error.message` on `!ok` **or** a 200 body with `ok:false` - the manager-style error the container/admin UIs surface in a toast); `postJson`/`postForm`/`toLogin`. Container files (`ContainerManager`, `ContainerList`, `ContainerInstance`, `ContainerTerminal`), `ServiceMonitor`, and `ProjectFiles` all route through it - none re-implement `fetch`.
|
||||
- **`Poller` (`static/js/Poller.js`).** `new Poller(fn, intervalMs, { immediate = true, pauseHidden = false })` runs `fn` on an interval with `start()`/`stop()`/`tick()`; `tick()` swallows errors so one failed poll never kills the loop, and `pauseHidden` skips the tick while `document.hidden`. Used by every live-update loop: `CounterManager` (30s, `pauseHidden`), `ContainerManager` (3s), `ContainerList` (4s), `AiUsageMonitor`, `ServiceMonitor`, and `ContainerInstance`'s detail (4s) + logs (3s). Store the `Poller`, not a raw interval id.
|
||||
- **`JobPoller` (`static/js/JobPoller.js`).** `JobPoller.run(statusUrl, { onDone, onFailed, onTimeout, intervalMs = 1500, maxAttempts = 200 })` returns a Promise; it polls `Http.getJson(statusUrl)`, swallows transient fetch errors, and fires the matching callback on `status === "done"|"failed"` or timeout. This is the one place the async-job status-poll lives - `ProjectForker` and `ZipDownloader` both call it with their own navigate/download/toast callbacks.
|
||||
- **`OptimisticAction` (`static/js/OptimisticAction.js`).** Base with one method, `submit(url, params, errorTarget, render)`: `Http.sendForm` -> `render(result)` on success -> `console.error` + (when `errorTarget` is given) `Toast.flash(errorTarget, "Error", 1500)` on failure. `VoteManager`/`ReactionBar`/`BookmarkManager`/`PollManager` `extend` it and call `this.submit(...)` for their POST, **keeping their own event wiring** (so `ReactionBar`'s palette toggle and `PollManager`'s multi-action handlers and `VoteManager`'s per-button `stopPropagation` are untouched). Pass `errorTarget` only where the old code toasted (`VoteManager`); the others pass `null` to keep their console-only behaviour.
|
||||
- **`FloatingWindow` / `WindowManager` (`static/js/components/`).** The draggable window base and shared z-order manager - documented in the Container manager section; both the container terminals and the Devii terminal extend `FloatingWindow`.
|
||||
- **`ScrollMemory` (`static/js/ScrollMemory.js`, `app.scrollMemory`).** Site-wide, per-tab scroll restoration - the fix for the "back to feed jumps to top" class of bugs. It sets `history.scrollRestoration = "manual"` once (never rely on the browser's auto-restore, which fires before late-loading content settles and never applies to normal link navigations), so ALL scroll restoration flows through this one module. State lives in `sessionStorage` (per-tab by definition, exactly the required scope): a position map keyed by exact `pathname + search` (hash ignored; saved by a throttled passive scroll listener plus a final write on `pagehide`/hidden `visibilitychange`, pruned to 50 entries / 60 min), a visited-URL trail (capped at 20), and a one-shot click-intent flag (30s validity, consumed on every load).
|
||||
**When it restores** - only when the situation is genuinely "going back": (1) navigation type `back_forward` (browser back without bfcache; with bfcache - `pageshow` `persisted` - the frozen page already has its scroll, so it only re-syncs the trail and clears the intent flag), (2) `reload`, (3) a same-origin click on `a.back-link` / `a[data-scroll-back]` / a breadcrumb link / any link whose target equals the trail's previous URL, which stamps the intent flag the next load matches. A fresh visit (topnav, address bar, redirect after POST) never restores, and a URL with a `#fragment` always wins over restoration.
|
||||
**How it restores reliably**: a `requestAnimationFrame` loop re-applies the target position (clamped to the current `scrollHeight`, `behavior: "instant"` so the global `html { scroll-behavior: smooth }` never animates it) until the document height has been stable at the target for 10 frames or a 4s deadline passes, and aborts instantly on the first `wheel`/`touchstart`/`keydown`/`pointerdown` so it never fights the user. It also **upgrades bare back-links on load**: when the previous trail URL has the same pathname as a query-less `a.back-link`/`[data-scroll-back]` href (post page's `/feed` vs the `/feed?tab=recent&before=...` the user actually came from), the href is rewritten to the exact previous URL so tab/topic/cursor AND scroll survive the round trip. Give any new "back to X" anchor the `back-link` class (or `data-scroll-back`) and it participates automatically - never hand-roll `scrollTo` persistence per page. Guarded by `tests/e2e/feed.py::test_feed_scroll_restored_via_back_link` / `_via_browser_back` / `_not_restored_on_fresh_visit`.
|
||||
- **On-screen keyboard reflow (mobile).** `FloatingWindow` listens on `window.visualViewport` `resize`/`scroll` and, while fullscreen/maximized, sets the window inline `height`/`top` to the visual viewport (the area NOT covered by the phone keyboard) instead of letting the keyboard push the window off-screen; it also toggles the `fw-keyboard-visible` class whose CSS drops the fixed `bottom` so the inline size wins. The Devii terminal mirrors this in its own `_setState` / `_onVisualViewport2` (`devii-keyboard-visible`) and `_ensureInputVisible` keeps the focused `.devii-input` pinned just above the keyboard; `ContainerTerminal._onResize` re-fits xterm against the shrunken viewport and `scrollToBottom`s. `AppContextMenu.open` clamps into the visual-viewport rect (`offsetLeft/Top` + `width/height`) so Paste stays on-screen above the keyboard, and the menu no longer closes on a visualViewport-only height change (keyboard appearing) - only a true `window.innerWidth` change closes it. All paths fall back to `window.inner*` when `visualViewport` is absent, so desktop and old browsers are untouched.
|
||||
|
||||
## Clickable avatars and usernames
|
||||
|
||||
Every avatar and username in the UI links to the user's profile page. Use the `_avatar_link.html` and `_user_link.html` include components:
|
||||
|
||||
```html
|
||||
{% set _user = item.author %}
|
||||
{% set _size = 32 %}
|
||||
{% set _size_class = "sm" %}
|
||||
{% include "_avatar_link.html" %}
|
||||
|
||||
<a href="/profile/{{ user['username'] }}" class="post-author-link">{{ user['username'] }}</a>
|
||||
```
|
||||
|
||||
The include files expect: `_user` (dict), `_size` (pixels), `_size_class` ("sm"|"md"|"lg").
|
||||
|
||||
Affected templates: `base.html`, `feed.html`, `post.html`, `profile.html`, `messages.html`, `notifications.html`.
|
||||
|
||||
For clickable avatars/usernames, reuse `templates/_avatar_link.html` and `templates/_user_link.html` via `{% include %}` with `_user`/`_size`/`_size_class` locals.
|
||||
|
||||
## Image upload
|
||||
|
||||
When a user uploads an image during post creation, the markdown `` is appended to the post content. The ContentRenderer then renders it as an `<img>`. All URLs are relative.
|
||||
|
||||
```python
|
||||
content += f"\n\n"
|
||||
```
|
||||
|
||||
File validation: max 5MB, allowed extensions: `.png`, `.jpg`, `.jpeg`, `.gif`, `.webp`, `.svg`.
|
||||
|
||||
## Attachments and media
|
||||
|
||||
`attachments.py` is the single store for files referenced via `attachment_uids` (posts, comments, projects, gists, messages, issues). One gate, `is_extension_allowed(ext)`, decides what may be uploaded and is used by both `routers/uploads.py` and `store_attachment()` - do not reintroduce a second check. Its semantics: when the admin `allowed_file_types` setting is non-empty it is the **authoritative** allowlist (mime auto-detected via `mimetypes` for extensions outside the built-in `ALLOWED_UPLOAD_TYPES` map); when empty it falls back to the built-in map, which includes images plus the browser-playable video formats `.mp4/.webm/.ogv/.mov/.m4v`. (Before this, `allowed_file_types` could only *narrow* the hardcoded list, never add to it - which is why adding a video extension there had no effect.)
|
||||
|
||||
**Ingesting a file from a URL.** `store_attachment_from_url(url, user_uid, filename=None)` (async, in `attachments.py`) is the remote counterpart to `store_attachment`: it downloads the URL on the server through `fetch_remote_file()` - SSRF-guarded (`_guard_public_url` resolves the host and refuses private/loopback/reserved/multicast addresses, mirroring the Devii fetch guard) and size-capped (streams, aborting once `_get_max_upload_bytes()` is exceeded) - resolves a filename from the URL path or the response `Content-Type` (`MIME_TO_EXT`), then calls `store_attachment()` so the bytes land in the **exact same** pipeline (validation, thumbnailing, DB row). It raises `RemoteFetchError(message, status)` which the route maps to an HTTP status. It is exposed at `POST /uploads/upload-url` (`UploadUrlForm{url, filename?}`, `require_user_api`) and as the Devii catalog action `attach_url` (handler `http`, `requires_auth=True`); both return the same record as `/uploads/upload`. The returned `uid` binds to a resource the same way as any upload - via `attachment_uids` at create/edit time - so attaching a remote image is just `attach_url` then `create_post`/`create_project`/etc. with that uid. Do not re-download remote files in a router; reuse this helper so the guard and size cap stay in one place.
|
||||
|
||||
`_row_to_attachment()` / `store_attachment()` expose `is_image` and `is_video` (derived from the mime prefix). The shared partial `templates/_attachment_display.html` branches image -> `<img>`, video -> `<video controls preload="metadata" class="gallery-video">`, else download link; rendering through this one partial is what makes video work across every feature at once. `AttachmentOut` (`schemas.py`) carries both flags - add new display keys there too or JSON drops them.
|
||||
|
||||
Media is served **inline** (not forced-download) for known-safe types only. The set `INLINE_MEDIA_EXTENSIONS` in `main.py` (`UploadStaticFiles`) and the matching `map $uri $upload_disposition` in `nginx/nginx.conf.template` must stay in sync: images/video/audio -> `inline` (so `<video>` plays and seeks via Range), everything else -> `attachment`. SVG is deliberately excluded from both (stored-XSS defense). `ContentRenderer.js` embeds direct video URLs typed into content via `videoExtRe`, mirroring its image handling.
|
||||
|
||||
### The upload button (`dp-upload`)
|
||||
|
||||
Every file-upload UI is the one custom element `dp-upload` (`static/js/components/AppUpload.js`); it replaced `AttachmentUploader.js` (deleted). It shows a clean button with a count badge + removable filename chips (no thumbnail grid), and runs in one of three `mode`s:
|
||||
|
||||
- `attachment` (default, the 7 forms via `_attachment_form.html` / inline in `_comment_form.html` and `messages.html`): uploads each file to `/uploads/upload` on select and keeps the hidden `<input name="attachment_uids">` the routers already link - **server contract unchanged**. Limits come from `max-size`/`max-files`/`allowed-types` attributes (fed by the Jinja globals).
|
||||
- `direct` (the project file browser, `project_files.html`): uploads to a custom `endpoint` with `field-name` plus a settable `extraFields` (e.g. `{path}`) and emits `dp-upload:uploaded` / `dp-upload:done` / `dp-upload:error`. `ProjectFiles.uploadTo(dir)` sets `extraFields` then calls `widget.open()`, and refreshes the tree on `done`.
|
||||
- `field` (create-post image, `feed.html`): wraps a real `<input type="file" name="image">` that submits with the form - no AJAX, inline-image flow unchanged.
|
||||
|
||||
The component validates size/type/count and reports errors via `app.toast`. CSS is `.dp-upload-*` in `components.css`; the old `.attachment-upload-*` upload-widget styles were removed, but the `.attachment-gallery`/`.attachment-lightbox` display styles (for already-saved attachments) remain.
|
||||
79
devplacepy/templates/CLAUDE.md
Normal file
79
devplacepy/templates/CLAUDE.md
Normal file
@ -0,0 +1,79 @@
|
||||
This file documents CDN libraries, the emoji picker, the modal system, and shared template partials under `templates/`. Claude Code auto-loads it whenever a file under this directory is read or edited.
|
||||
|
||||
## Templates and frontend (general rules)
|
||||
|
||||
- All routers MUST import the shared `templates` from `devplacepy.templating`. Do NOT instantiate `Jinja2Templates` per router - globals (`avatar_url`, `get_unread_count`, `get_user_projects`, `format_date`) are registered on that single instance.
|
||||
- Templates extend `base.html`, add page CSS via `{% block extra_head %}`, page JS via `{% block extra_js %}`.
|
||||
- **Nav active state is the `nav_active` Jinja global**, never an inline path check. `nav_active(request, *prefixes, exact=False)` (`templating.py`) returns `"active"` (or `""`) when `request.url.path` equals a prefix or is a sub-path of it (`exact=True` matches only the exact path, e.g. the admin Dashboard root). Every topnav/sidebar link uses it (`class="topnav-link {{ nav_active(request, '/projects') }}"`); do NOT hand-write `{% if request.url.path == ... or request.url.path.startswith(...) %}active{% endif %}`.
|
||||
|
||||
## CDN Libraries
|
||||
|
||||
Loaded via `<script>` tags in `base.html`. ALL must use `defer` to avoid blocking DOMContentLoaded:
|
||||
|
||||
```html
|
||||
<script defer src="https://cdn.jsdelivr.net/npm/marked/lib/marked.umd.js"></script>
|
||||
<script defer src="https://cdn.jsdelivr.net/gh/highlightjs/cdn-release@11.11.1/build/highlight.min.js"></script>
|
||||
<script defer src="/static/vendor/purify.min.js"></script>
|
||||
<script type="module" src="https://cdn.jsdelivr.net/npm/emoji-picker-element@^1/index.js"></script>
|
||||
<script defer src="/static/js/ContentRenderer.js"></script>
|
||||
<script defer src="/static/js/EmojiPicker.js"></script>
|
||||
<script type="module" src="/static/js/Application.js"></script>
|
||||
```
|
||||
|
||||
**Never use `<script>` without `defer` for CDN libraries** - they block HTML parsing and cause `wait_until="domcontentloaded"` to timeout in Playwright tests.
|
||||
|
||||
## Emoji Picker
|
||||
|
||||
Uses `emoji-picker-element` web component (Discord-style, searchable, skin tones):
|
||||
- `EmojiPicker.js` wraps it with a toggle button and inserts unicode at cursor position
|
||||
- Added to all `.comment-form textarea` and `.emoji-picker-target` elements
|
||||
- The old `😀` emoji button has been removed from all templates
|
||||
|
||||
## Modal System
|
||||
|
||||
`Application.js` `initModals()` toggles a `.visible` CSS class on `.modal-overlay`; the CSS rule `.modal-overlay.visible { display: flex; }` handles visibility:
|
||||
|
||||
```javascript
|
||||
// CORRECT - toggle the .visible class on the modal:
|
||||
modal.classList.add("visible"); // show
|
||||
modal.classList.remove("visible"); // hide
|
||||
```
|
||||
|
||||
Triggers usually have `href="#"`, so call `e.preventDefault()` in click handlers. Always call `e.preventDefault()` on `[data-modal]` click handlers since trigger elements often have `href="#"`:
|
||||
|
||||
```javascript
|
||||
trigger.addEventListener("click", (e) => {
|
||||
e.preventDefault();
|
||||
modal.style.display = "flex";
|
||||
});
|
||||
```
|
||||
|
||||
`.modal-close` is wired generically by `Application.js` - no inline JS needed in templates for basic modals.
|
||||
|
||||
Do NOT hand-write the overlay/header markup. Use the shared macro in `templates/_macros.html`:
|
||||
|
||||
```jinja
|
||||
{% from "_macros.html" import modal %}
|
||||
{% call modal('create-post-modal', 'Create New Post') %}{# wide=true for modal-card-wide #}
|
||||
<form ...> ... <div class="modal-footer">...</div> </form>
|
||||
{% endcall %}
|
||||
```
|
||||
|
||||
## Shared template partials
|
||||
|
||||
Reuse these via `{% set _x = ... %}{% include %}` (the `_avatar_link.html` convention) instead of copy-pasting markup:
|
||||
|
||||
- `_post_votes.html` - post +/- vote bar. Locals: `_uid`, `_my_vote`, `_count`.
|
||||
- `_star_vote.html` - project/gist star button. Locals: `_type` (`project`|`gist`), `_uid`, `_my_vote`, `_count`, `_btn_class`, optional `_stop` (adds `data-stop-propagation`). The star glyph (`☆`→`★` when `.voted`) comes from the `vote-star` CSS class via `::before` (`base.css`) - do not put a literal star in markup.
|
||||
- `_post_header.html` - post author/avatar/time header (`.post-header`). Locals: `_author`, `_time`.
|
||||
- `_topic_selector.html` - topic radio group. Locals: `_topics`, `_selected`.
|
||||
|
||||
Vote button styles live ONCE in `feed.css` (`.post-action-btn`) - never redefine them in `post.css`. Page-specific CSS goes in a `static/css/*.css` file referenced from `{% block extra_head %}`, never an inline `<style>` block.
|
||||
|
||||
For clickable avatars/usernames, reuse `templates/_avatar_link.html` and `templates/_user_link.html` via `{% include %}` with `_user`, `_size`, `_size_class` locals.
|
||||
|
||||
The three public listings (`/feed`, `/gists`, `/projects`) share one search box: the `templates/_sidebar_search.html` partial (included at the top of `.sidebar-card` with `_action`/`_placeholder`/`_hidden` locals) plus the `database.text_search_clause(table, search, fields, author_field)` data helper. Each listing matches its text fields PLUS the author username: pass `author_field="user_uid"` and the helper resolves matching usernames to uids via `database.get_uids_by_username_match(search)` and OR-includes `user_uid IN (...)` (no JOIN, no separate `ilike`). Any new listing with a left filter panel reuses both - never re-inline an `ilike` search clause, never add a JOIN for author matching, and never hand-roll another search form. See `devplacepy/routers/CLAUDE.md` -> Listing search.
|
||||
|
||||
## Modal pattern
|
||||
|
||||
`Application.js` `initModals()` toggles a `.visible` CSS class on `.modal-overlay`; the CSS rule `.modal-overlay.visible { display: flex; }` handles visibility. Triggers usually have `href="#"`, so call `e.preventDefault()` in click handlers. `.modal-close` is wired generically - no inline JS needed.
|
||||
@ -8,7 +8,7 @@ How a change moves from understanding to shipped. The workflow is ordered by dat
|
||||
|
||||
## The feature workflow
|
||||
|
||||
1. **Understand.** Read the router for the area (a flat `routers/{area}.py`, or the leaf inside the `routers/{area}/` package that owns the endpoint), the template, the test file, and the matching `AGENTS.md` domain section before writing. Trace the existing input model, router, data helper, and response.
|
||||
1. **Understand.** Read the router for the area (a flat `routers/{area}.py`, or the leaf inside the `routers/{area}/` package that owns the endpoint), the template, the test file, and the matching nested `CLAUDE.md` before writing. Trace the existing input model, router, data helper, and response.
|
||||
2. **Data layer.** Add query and batch helpers in the `database/` package (never inline N+1). Add the Pydantic `Form` model to `models.py` and the `*Out` response model to the `schemas/` package. Schema auto-syncs via `dataset`; add indexes in `init_db()` with `CREATE INDEX IF NOT EXISTS`.
|
||||
3. **Server layer.** Write the handler with the right guard (`get_current_user` for public reads, `require_user` for member POSTs, `require_admin` for admin). Declare specific routes before catch-alls. Return HTML and JSON from one handler via `respond(..., model=XOut)`. Register any new router in `main.py`.
|
||||
4. **View layer.** Import the shared `templates`, extend `base.html`, put page CSS in `{% block extra_head %}` and page JS in `{% block extra_js %}`, and reuse partials and globals.
|
||||
@ -26,15 +26,14 @@ One handler is HTML, JSON, a Devii tool, and an API-docs entry at once. After to
|
||||
| Devii tool | the catalog `Action` | feature exists for humans but is invisible to the assistant |
|
||||
| API docs | `endpoint()` in the `docs_api/` package | endpoint undocumented |
|
||||
|
||||
## The documentation trio
|
||||
## The documentation set
|
||||
|
||||
Three files are kept current, each with a distinct role:
|
||||
Two kinds of files are kept current, each with a distinct role:
|
||||
|
||||
- **`README.md`** is product-facing. Update it when routes, config or env vars, dependencies, or user-visible features change. No development process, no begging, no emoticons.
|
||||
- **`AGENTS.md`** is the deep companion: every domain-specific mechanic, helper, pitfall, and cross-layer wiring, plus the full feature relationship map.
|
||||
- **`CLAUDE.md`** holds the rules and conventions. Update it only when a new architectural rule, convention, or workflow step is introduced, not per feature.
|
||||
- **`CLAUDE.md`** is split by scope: the root `CLAUDE.md` holds cross-cutting rules and conventions (update it only when a new architectural rule, convention, or workflow step is introduced, not per feature), while a nested `CLAUDE.md` inside a subsystem directory (e.g. `devplacepy/services/devii/CLAUDE.md`) holds that subsystem's domain-specific mechanics, helpers, pitfalls, and cross-layer wiring - it is loaded automatically only when a file in that directory is touched, so update the one nearest the code you changed.
|
||||
|
||||
This Architecture section in the in-app docs is the browsable, admin-facing reflection of those three: it explains the design and the process, while the trio remains the repository source of truth.
|
||||
This Architecture section in the in-app docs is the browsable, admin-facing reflection of that set: it explains the design and the process, while the `CLAUDE.md` files remain the repository source of truth.
|
||||
|
||||
## Validation gates
|
||||
|
||||
|
||||
@ -20,7 +20,7 @@ workflow run across every quality dimension, in canonical order.
|
||||
| `security-maintainer` | Authorization on every route, private-resource gating, read-only file guards, input validation, XSS controls. |
|
||||
| `audit-maintainer` | Every state-changing action emits an audit record; denials and failures carry the right result; the event catalogue is complete. |
|
||||
| `devii-maintainer` | The Devii assistant can do everything a role allows over REST, and exposes only the tools that role may call. |
|
||||
| `docs-maintainer` | CLAUDE.md, AGENTS.md, README, the API docs, and the prose pages agree with the source, with correct role gating. |
|
||||
| `docs-maintainer` | The root and nested CLAUDE.md files, README, the API docs, and the prose pages agree with the source, with correct role gating. |
|
||||
| `fanout-maintainer` | A feature is wired through every layer: form model, output schema, response, Devii tool, API docs, SEO, docs. |
|
||||
| `dry-maintainer` | Shared helpers are reused instead of duplicated logic re-implemented. |
|
||||
| `style-maintainer` | Naming, headers, typing, and formatting follow the project rules, applied with context so intentional patterns are left alone. |
|
||||
|
||||
@ -15,7 +15,7 @@ Arguments after the command name are available to the prompt; for example
|
||||
|
||||
| Command | What it does |
|
||||
|---------|--------------|
|
||||
| `/explain <area>` | Reads the relevant AGENTS.md section and the code, then summarizes the architecture, data flow, invariants, and entry points. Read-only. |
|
||||
| `/explain <area>` | Reads the relevant nested CLAUDE.md and the code, then summarizes the architecture, data flow, invariants, and entry points. Read-only. |
|
||||
| `/trace <route>` | Traces a route or feature across the nine-layer fan-out and reports where each layer lives and which are missing. Read-only. |
|
||||
|
||||
## Verify and test
|
||||
|
||||
@ -45,7 +45,7 @@ order so nothing is dropped:
|
||||
`respond(request, template, ctx, model=XOut)`.
|
||||
3. **View** - extend `base.html`; one ES6 class per file under `static/js/`.
|
||||
4. **Agent and docs** - add a Devii tool when a user could ask for the action, an
|
||||
`endpoint()` entry in `docs_api.py`, and update `README.md` and `AGENTS.md`.
|
||||
`endpoint()` entry in `docs_api.py`, and update `README.md` and the relevant nested `CLAUDE.md`.
|
||||
|
||||
## Validate before you finish
|
||||
|
||||
|
||||
114
devplacepy/utils/CLAUDE.md
Normal file
114
devplacepy/utils/CLAUDE.md
Normal file
@ -0,0 +1,114 @@
|
||||
This file documents notifications and gamification (XP/levels/badges/leaderboard), both implemented in devplacepy/utils.py. Claude Code loads it automatically whenever a file under devplacepy/utils/ is read or edited.
|
||||
|
||||
## Notification system
|
||||
|
||||
Notifications are created server-side in the route handlers and stored in the `notifications` table. Unread counts are cached per-process in `_unread_cache` (notifications) and `_messages_cache` (messages), both 10s TTL in `templating.py`. Mutating routes call `clear_unread_cache(uid)` / `clear_messages_cache(uid)` to invalidate.
|
||||
|
||||
### Live counter badges
|
||||
|
||||
The top-nav bell and Messages link carry `data-counter="notifications"` / `data-counter="messages"` with a child `<span data-counter-badge hidden>`. `CounterManager.js` polls `GET /notifications/counts` (returns `{"notifications", "messages"}`, zeros for guests) every 30s (via the shared `Poller`, `pauseHidden`) and on tab focus, then sets each badge's text and toggles its `hidden` attribute. Server-rendered counts seed the initial state, so the badges are correct on first paint and self-heal without a reload. Absolute badges use `.nav-badge`; inline badges (next to text links) add `.nav-badge-inline`.
|
||||
|
||||
### Notification types and trigger points
|
||||
|
||||
| Type | Trigger | Location | Condition |
|
||||
|------|---------|----------|-----------|
|
||||
| `comment` | Top-level comment on post | `comments.py` | `post["user_uid"] != user["uid"]` |
|
||||
| `reply` | Reply to a comment | `comments.py` | `parent["user_uid"] != user["uid"]` |
|
||||
| `vote` | Upvote on post or comment | `votes.py` | `value == 1` AND voter != owner |
|
||||
| `follow` | Follow another user | `follow.py` | Always (self-follow blocked upstream) |
|
||||
| `message` | Send a message | `messages.py` | Always (different user) |
|
||||
| `mention` | `@username` in content | `utils.py` `create_mention_notifications` | Mentioned user != actor |
|
||||
| `level` | Reach a new level | `utils.py` `award_xp` | `new_level > current_level` |
|
||||
| `badge` | Earn any badge | `utils.py` `notify_badge` | First grant only (`award_badge` returns `True`) |
|
||||
| `issue` | Issue filed/replied/status | `issues.py`, `services/gitea`, `services/jobs/issue_create_service.py` | Reporter or admins |
|
||||
|
||||
`level`/`badge` notifications have `related_uid == user_uid` (self), so the notifications template renders them with the recipient's own avatar; the template is type-agnostic (driven by `message`/`actor`), so no per-type handling is needed.
|
||||
|
||||
### Auto-mark-read on content view
|
||||
|
||||
Beyond the explicit clears (`/notifications/open/{uid}`, `POST /notifications/mark-read/{uid}`, `POST /notifications/mark-all-read`), a notification is **automatically marked read when the user opens the page where its referenced content is visible**. The single primitive is `database.mark_notifications_read_by_target(user_uid, target_url)`: it stamps `read = 1` on every unread notification of that user whose stored `target_url` equals `target_url` OR begins with `target_url + "#"` (so a post-page key clears its `#comment-{uid}` variants), then `clear_unread_cache(user_uid)`. It returns the count and short-circuits (no write, no cache bump) when nothing matches, so a detail view by a user with no relevant unread notifications costs only one indexed read. There is **no** `target_type`/`target_uid` column - the `target_url` string (produced by `resolve_object_url`) is the only content key, so callers build the page key with that same generator (or the identical literal the create-site used) to guarantee a match with zero drift.
|
||||
|
||||
Call sites (each only when a real user is present and the resource resolved, placed after the 404/canonical-redirect guards):
|
||||
|
||||
| GET handler | page key | clears |
|
||||
|-------------|----------|--------|
|
||||
| `posts.py::view_post` | `resolve_object_url("post", uid)` | comment, reply, vote, mention (post + its comments) |
|
||||
| `projects/index.py::project_detail` | `resolve_object_url("project", uid)` | vote, mention |
|
||||
| `gists.py::gist_detail` | `resolve_object_url("gist", uid)` | vote, mention |
|
||||
| `news.py::news_detail_page` | `resolve_object_url("news", uid)` | comment, vote, mention |
|
||||
| `messages.py::messages_page` | `/messages?with_uid={with_uid}` | message, DM mention (beside the existing `mark_conversation_read`) |
|
||||
| `profile/index.py::profile_page` | `/profile/{username}` | follow (follower's profile), badge/level (own profile) |
|
||||
| `issues/index.py::issue_detail` | `/issues?highlight={number}` | issue (the create-sites store the `?highlight=` URL, not the `/issues/{number}` detail path) |
|
||||
| `devii.py::devii_page` | `/devii` | reminder |
|
||||
| `game/index.py::game_home` | `/game` | harvest_stolen |
|
||||
|
||||
Deliberately excluded: `GET /notifications` (the list must stay readable) and `/feed` (not a content page). Reuse `mark_notifications_read_by_target` for any future surface; never hand-roll a per-handler `UPDATE notifications`.
|
||||
|
||||
### Time-grouped display
|
||||
|
||||
Notifications are grouped by time period in `notifications.py` `_group_label()`:
|
||||
- Same day - **Today**
|
||||
- Previous day - **Yesterday**
|
||||
- Within 7 days - **This week**
|
||||
- Older - **Older**
|
||||
|
||||
The template uses `notification_groups` (list of `{label, entries}` dicts). Jinja2 note: avoid `.items` as a dict key - it clashes with Python's `dict.items()` method.
|
||||
|
||||
### Vote notification messages
|
||||
|
||||
```python
|
||||
f"{user['username']} ++'d your post"
|
||||
f"{user['username']} ++'d your comment"
|
||||
```
|
||||
|
||||
### Per-user, per-channel preferences
|
||||
|
||||
Every notification type is independently toggleable on three channels - **in-app**, **push** and **telegram** - per user, with admin-set platform defaults. This is enforced at the single funnel, so there is no per-call-site work: a new `create_notification(user_uid, type, ...)` call is automatically subject to preferences.
|
||||
|
||||
- **Canonical types** = `database.NOTIFICATION_TYPES` (the nine listed above; `key`/`label`/`description`). This list drives the profile tab and the admin page; it is the single source of truth. The `test` type is intentionally absent, so it always delivers (the resolver defaults unknown types to on).
|
||||
- **Channels are data-driven.** Adding a channel is a constant change, never scattered edits: `NOTIFICATION_CHANNELS` (the tuple), `_NOTIFICATION_CHANNEL_COLUMNS` (channel -> table column), and `_NOTIFICATION_CHANNEL_DEFAULTS` (channel -> per-type fallback when no admin default row exists). `in_app`/`push` default `1` (on); **`telegram` defaults `0` (off for every type)**. The four helpers (`_notification_overrides`, `get_notification_prefs`, `set_notification_pref`, `_notification_default`) iterate these maps generically, so a new channel needs only the constant entry + the `notification_preferences` column in `init_db`.
|
||||
- **Resolution** (`database.notification_enabled(user_uid, type, channel)`): a live `notification_preferences` row for `(user_uid, type)` wins; else the global default `get_int_setting("notif_default_{type}_{channel}", _NOTIFICATION_CHANNEL_DEFAULTS[channel])`; else the channel fallback. So a user who never touched a setting inherits the admin default (or the channel fallback when no default row exists); a user who customized keeps their choice.
|
||||
- **Storage**: table `notification_preferences` (`uid`, `user_uid`, `notification_type`, `in_app_enabled`, `push_enabled`, `telegram_enabled`, timestamps, soft-delete columns), in `SOFT_DELETE_TABLES`, columns+indexes ensured in `init_db`. Cached under the `"notif_prefs"` cache-version name (`sync_local_cache`/`bump_cache_version`), keyed by `user_uid`. Helpers: `get_notification_prefs(user_uid)` (full resolved list for the UI/JSON, one boolean per channel keyed by channel name), `set_notification_pref(user_uid, type, channel, enabled)` (upsert one channel, leaving the others at their resolved value; revives a soft-deleted row), `reset_notification_prefs(user_uid)` (soft-delete the user's overrides -> fall back to defaults), `get_notification_default`/`set_notification_default` (the `notif_default_*` `site_settings` keys; defaults are NOT seeded so an untouched type reads its channel fallback: `1` for in_app/push, `0` for telegram).
|
||||
- **Enforcement (`create_notification` single funnel) is in `utils.create_notification`**: the in-app `notifications.insert` + `clear_unread_cache` run only when `notification_enabled(..., "in_app")`, `_schedule_push` runs only when `notification_enabled(..., "push")`, and `_schedule_telegram` runs only when `notification_enabled(..., "telegram")`. The `notification.create` audit row is unconditional and records which channels fired (`metadata.in_app`/`push`/`telegram`). **Because the DM path in `messages.py` now routes through `create_notification` (type `message`)** instead of inserting directly, direct messages are gated like everything else and gained a (default-on) push channel. **`create_notification` is a thin wrapper that defers the whole body** (preference reads + insert + push + telegram + audit) to the **background task queue** via `background.submit(_deliver_notification, ...)`, so it never blocks the request (inline under tests); call it directly, never wrap it in `background.submit`. See `devplacepy/services/CLAUDE.md` -> Background task queue.
|
||||
- **Telegram channel (cross-worker delivery).** `_schedule_telegram(user_uid, message)` calls `services/telegram/store.enqueue_outbox`, which no-ops when the user is not paired (`link_for_user` is None) and otherwise inserts a row into the `telegram_outbox` queue (`uid`, `user_uid`, `chat_id`, `text`, `status` pending|sent|failed, `attempts`, timestamps; transient, NOT in `SOFT_DELETE_TABLES`, columns+index ensured in `init_db`). This is needed because `_deliver_notification` runs on the background queue of **any** worker, but the Telegram bot worker subprocess (and `TelegramService.send_markdown`) lives **only on the service-lock owner**. `services/telegram/outbox_service.py` `TelegramOutboxService` (lock-owner `BaseService`, default-enabled, ~2s interval, registered in `main.py` after `TelegramService`) drains pending rows on the lock owner: if the Telegram service is missing/disabled/`not worker_alive()` it returns (rows wait, nothing is lost), else it sends each via `send_markdown`, `mark_outbox_sent` on success or `mark_outbox_failed` (retries up to `MAX_OUTBOX_ATTEMPTS=3`, then `failed`), and `prune_outbox` clears old sent/failed rows. The DB is the cross-worker bridge here exactly as in `NotificationRelayService`/`MessageRelay`; a dedicated service (not folded into the 1s toast relay) keeps the blocking 30s-per-send `send_markdown` off the toast path. Telegram defaults off, so absent any pairing the platform sends nothing until a paired user opts a type in. The profile Telegram column is disabled until the user pairs Telegram.
|
||||
- **Live pairing update (pub/sub).** When a user completes pairing from their phone, the bridge (`services/telegram/bridge.py` `_handle_unpaired`) runs on the **service-lock owner** - exactly where every `/pubsub/ws` subscriber converges - so right after the `telegram.pair.success` audit it calls `_publish_pairing(user_uid, True)` -> `services.pubsub.publish(f"user.{uid}.telegram", {"paired": True})` (best-effort, lazy import). No new table, no cross-worker hop. The topic is already authorized for that user by `services/pubsub/policy.py` `_own_namespace` (`user.{uid}.*`). Frontend: `TelegramPairing` and `NotificationPrefs` (both built with `this.pubsub` in `Application.js`, after `PubSubClient`) read `document.body.dataset.userUid` and subscribe to `user.{uid}.telegram` (the `LiveNotifications`/`CounterManager` pattern); on `{paired:true}` the pairing panel flips to Connected (`setPaired`/`hideCode`/`setMessage`) and the notifications Telegram column enables live (drop `disabled`+`notif-switch-disabled`, hide the hint). Cross-device **unpair** live update is intentionally not wired (HTTP unpair runs on any worker; same-page unpair updates locally).
|
||||
- **UI:** profile **Notifications** tab (`/profile/{username}?tab=notifications`, owner-or-admin, `POST /profile/{username}/notifications` + `/reset`) and admin global defaults at `/admin/notifications`. Devii tools `notification_list`/`notification_set`/`notification_reset` (`handler="notification"`, owner-scoped, `set`/`reset` in `CONFIRM_REQUIRED`). Reuse this canonical-list + resolver + override-table + single-funnel pattern for future per-user per-channel preferences.
|
||||
|
||||
### Live delivery (toasts and unread counts)
|
||||
|
||||
- **Live toasts ride the `in_app` channel** (no new channel). `services/notification_relay.py` `NotificationRelayService` is a lock-owner `BaseService` (registered in `main.py`, default-enabled, 1s interval) modeled on `MessageRelay`: it primes a watermark to `MAX(notifications.id)` on first tick, then each tick selects `notifications WHERE id > watermark` and publishes `{uid, type, message, target_url}` to the per-recipient pub/sub topic `user.{user_uid}.notifications`. It runs **only on the service lock owner**, which is exactly where every pub/sub WS subscriber converges (non-owner `/pubsub/ws` closes `4013`), so the in-process `services.pubsub.publish` reaches the recipient's browser. Because the `notifications` row exists only when the `in_app` channel is enabled (the insert above), a live toast fires **exactly** for the notifications a user has enabled - the relay needs no preference lookup of its own. The watermark always advances (boot-priming + every-tick) so a newly-connected subscriber never gets a backlog flood. Frontend: `base.html` exposes the viewer uid as `<body data-user-uid>`, `static/js/LiveNotifications.js` (`app.liveNotifications`, constructed with `this.pubsub`/`this.toast`) subscribes to that topic and calls `app.toast.show(message, {type:"info", ms, url})`. **The toast click reproduces a bell-item click exactly:** it targets `GET /notifications/open/{uid}` (which marks the row read and redirects to its `target_url`), not the bare `target_url` - the relay publishes `uid` for this. `AppToast` gained a generic click action via `_action(options)`: `options.onClick` (a function) or `options.url` (navigate); either adds the `.dp-toast-link` pointer cursor, so any caller can attach a click action. DMs toast too (type `message` is not excluded). Reuse this relay-on-the-lock-owner pattern for any future "live mirror of a DB-persisted, per-user event."
|
||||
- **Unread-count badges ride the same relay.** The same `NotificationRelayService` tick, after publishing the toast rows, also publishes the recipient's fresh unread counts `{notifications, messages}` to `user.{uid}.counts` (computed with a direct DB query on the lock owner, never the per-worker `_unread_cache`, which could be stale there). Because a new DM also inserts a `message`-type notification row through `persist_message -> create_notification`, this one publish point covers both new notifications and new messages, so the header badge bumps instantly. `static/js/CounterManager.js` (`app.counters`, constructed with `this.pubsub`) subscribes to `user.{uid}.counts` and applies the pushed counts; its HTTP poll of `/notifications/counts` stays as a 60s reconciliation fallback (covers decrements on read and any missed frame). Read events still clear the per-worker cache as before.
|
||||
|
||||
## Gamification (XP, levels, badges, leaderboard)
|
||||
|
||||
The progression engine lives in `utils.py` and is wired into the existing content-creation, vote, and follow hooks. Do NOT scatter XP/badge logic - go through these helpers.
|
||||
|
||||
### XP and levels (`utils.py`)
|
||||
|
||||
- `award_xp(user_uid, amount)` - adds XP (clamped to `>= 0`), recomputes and stores `level`, invalidates the per-process user cache (`clear_user_cache`), and on level-up fires a `level` notification plus any level-milestone badge. Returns `{"xp", "level", "leveled_up"}`.
|
||||
- `level_for_xp(xp)` -> `1 + max(0, xp) // LEVEL_XP` (`LEVEL_XP = 100`). `level` is stored on the user (not derived in the template) so `profile.html`'s `xp % 100` progress bar keeps working.
|
||||
- XP amounts are constants in `utils.py`: `XP_POST=10`, `XP_COMMENT=2`, `XP_PROJECT=15`, `XP_GIST=5`, `XP_UPVOTE=5`, `XP_FOLLOW=5`. Awards for received upvotes/followers go to the content owner / followed user and live **inside** the existing `owner_uid != user["uid"]` guards (`votes.py`, `follow.py`).
|
||||
|
||||
### Badges (`utils.py`)
|
||||
|
||||
- `award_badge(user_uid, name)` is idempotent (returns `True` only on first grant) - reuse it; never insert into `badges` directly.
|
||||
- `check_milestone_badges(user_uid)` recomputes **source-derived** count thresholds (counts of existing tables, so always accurate, no tracking needed) and awards + notifies any newly crossed badge. Call it after content creation and after an upvote/follow that changes the recipient's totals. The thresholds are the data-driven `_COUNT_MILESTONES` list `(badge, metric_key, threshold)` over metric keys `posts`/`comments`/`projects`/`gists`/`stars`/`followers`/`following`/`streak`, resolved lazily by `_milestone_metrics(user_uid)` (a memoized resolver that only queries a metric when an unheld badge needs it - badges already held are skipped before any query). To add a source-derived milestone, add a row to `_COUNT_MILESTONES` and the badge to `BADGE_CATALOG`. Streak badges additionally emit a `reward.streak.milestone` audit event.
|
||||
- `award_rewards(user_uid, amount, first_badge=None)` is the single entry point for the create/upvote/follow reward sequence: it awards the optional first-time badge, calls `award_xp`, then `check_milestone_badges`. Use it instead of calling the three separately (posts/projects/gists/comments/follow/votes all go through it) so milestone checks are never skipped. **`award_rewards` defers its whole body to the background task queue** (`background.submit(_apply_rewards, ...)`), so EVERY XP award on the platform leaves the request path - the badge/XP/milestone writes (and the reward-triggered level/badge notifications nested inside `award_xp`/`check_milestone_badges`) all run on the consumer. In tests the queue runs inline, so XP is awarded synchronously and assertions hold. Never re-wrap an `award_rewards` call in `background.submit` - it self-defers. See `devplacepy/services/CLAUDE.md` -> Background task queue.
|
||||
- Badge catalog + display metadata is `BADGE_CATALOG` in `utils.py` (55 badges), exposed to templates as the `badge_info(name)` global. Each entry has `icon`, `description`, and a `group` (one of `BADGE_GROUPS`: First steps, Explorer, Engagement, Content, Community, Reputation, Dedication, Levels, Milestones). Unknown names fall back to a default icon and the name as description. `LEVEL_BADGES` maps levels 5/10/25/50/100 to Level badges, awarded inside `award_xp` on level-up.
|
||||
|
||||
#### Activity tracking and feature-first-use badges
|
||||
|
||||
- For achievements that are **not** derivable from an existing table (first time a user uses a feature, "read N docs", number of DeepSearches, etc.), there is a generic per-user activity counter. `database.record_activity(user_uid, action) -> int` upserts a `user_activity` counter (unique `(user_uid, action)`, raw SQL `ON CONFLICT` inside `with db:`) and returns the new count; `database.record_unique_activity(user_uid, action, target) -> int | None` records a distinct `(user_uid, action, target)` in `user_activity_seen` (returns the new distinct count, or `None` if the target was already seen) for "N distinct things" achievements. Both tables + their unique indexes are ensured in `init_db()`.
|
||||
- `utils.ACHIEVEMENTS` maps `action -> [(threshold, badge_name), ...]` (threshold 1 = first use). `utils.UNIQUE_ACTIONS` is the set of actions counted distinctly (currently `docs.read`). `utils.track_action(user_uid, action, target=None)` is the single public hook: it returns immediately if the action is unknown, else `background.submit`s `_apply_achievements`, which records the activity (counter or unique), then awards + notifies every crossed threshold via `award_badge`/`notify_badge`. It is fully deferred (off the request path) and idempotent, so call it inline at the success point of any feature.
|
||||
- Wired actions and their hook sites: `vote`/`bookmark` (`content.py`), `reaction` (`routers/reactions.py`), `follow` (`routers/follow.py`, follower side), `message` (`services/messaging/persist.py`), `docs.read` (`routers/docs/views.py`, unique per slug, authenticated only), `fork`/`zip` (`routers/projects/index.py`), `zip`/`project_file` (`routers/projects/files.py`), `seo` (`routers/tools/seo.py`), `deepsearch` (`routers/tools/deepsearch.py`), `container` (`services/containers/api.create_instance`, user actor), `upload` (`routers/uploads.py`), `issue` (`routers/issues/create.py`), `poll` (`routers/polls.py`), `profile` (`routers/profile/index.py`, when a field is filled), `devii` (`services/devii/session.py` `_run_turn`, user owner on the non-docs channel only). To add a feature badge: add the badge to `BADGE_CATALOG`, an `ACHIEVEMENTS` entry, and a `track_action(...)` call at the feature's success point.
|
||||
- **Profile showcase:** `utils.build_achievements(held_names)` returns the full catalog grouped by `BADGE_GROUPS` with an `earned` flag per badge and `earned`/`total` per group. The profile route passes it as `achievements` (+ `badge_earned`/`badge_total`) and `profile.html` renders a collapsible **Achievements** showcase (`.profile-achievements`, styles in `profile.css`) above the tabs, so every badge - earned and locked - is discoverable.
|
||||
|
||||
### Rank and leaderboard (`database.py`)
|
||||
|
||||
- `_ranked_authors()` builds (and 60s-caches in `_authors_cache`) the full list of authors with positive total stars, ordered desc, enriched via `get_users_by_uids`. `get_top_authors(limit)`, `get_leaderboard(limit, offset)` (adds a 1-based `rank`), and `get_user_rank(user_uid)` all slice/scan this one list - keep them consistent.
|
||||
- `update_target_stars()` clears `_authors_cache` so a vote is reflected on the leaderboard and profile rank immediately (also keeps integration tests deterministic).
|
||||
- The leaderboard page (`/leaderboard`, `routers/leaderboard.py`) shows the **top 50 only - no pagination** (`TOP_LIMIT = 50`). It is public (`get_current_user`), highlights the current user's row (`.leaderboard-row-self`), and is listed in `sitemap.xml`.
|
||||
|
||||
### Backfill
|
||||
|
||||
`_backfill_gamification()` runs at the end of `init_db()`. It computes XP from prior activity (same amounts as above) for users still at the default `xp=0`, sets `level`, then runs `check_milestone_badges` per user. Guarded on `xp=0` so it is idempotent across restarts.
|
||||
130
tests/CLAUDE.md
Normal file
130
tests/CLAUDE.md
Normal file
@ -0,0 +1,130 @@
|
||||
# CLAUDE.md
|
||||
|
||||
This file documents detailed testing patterns, fixtures, and pitfalls for devplacepy/tests/ (Playwright + unit tests across api/e2e/unit tiers). Claude Code loads it automatically whenever a file under this directory is read or edited. Never run the test suite unless the user explicitly asks - validate with imports and manual checks instead.
|
||||
|
||||
## Never run tests unless explicitly asked (critical guardrail)
|
||||
|
||||
**NEVER run tests unless specifically asked by user.** Not the full suite, not a single file - do not run any tests unless the user explicitly requests it. Validate each touched language manually instead: Python (compile + import), JS (parse / bracket matching), CSS (brace matching), HTML (tag matching). Zero tolerance. A clean `python -c "from devplacepy.main import app"` import is the baseline check.
|
||||
|
||||
**Never run tests unless the user explicitly asks for it.** Not the full suite, not a single file. Validate code with a clean import (`python -c "from devplacepy.main import app"`) and per-language manual checks instead.
|
||||
|
||||
If you do need to run a single test on explicit request: `python -m pytest tests/e2e/feed.py::test_name -v --tb=line -x`.
|
||||
|
||||
## Fixture stack
|
||||
|
||||
- `app_server` (session-scoped): spawns uvicorn as a subprocess on port 10501 (the `PORT` constant) with a tempfile DB.
|
||||
- `browser_context` (session-scoped): one Playwright context shared by all tests.
|
||||
- `page` (function-scoped): `clear_cookies()` on the session context, then a fresh page.
|
||||
- `alice` / `bob`: seeded users `alice_test` / `bob_test` logged in via the login form. `bob` gets its own context for multi-user tests. Returns `(page, user_dict)`.
|
||||
|
||||
Tests run on port 10501 with a tempfile SQLite DB and a dedicated `DEVPLACE_DATA_DIR`, and `DEVPLACE_DISABLE_SERVICES=1` so background services don't start. `make test` runs **serially, one test at a time, in a single process**: one isolated DB + data dir + uvicorn subprocess + Chromium, shared across the whole session, so session fixtures (`seeded_db`) and intra-file ordering hold. Serial execution is enforced centrally in `pyproject.toml` (`[tool.pytest.ini_options]` `addopts = "--tb=line -p no:xdist"`); pytest-xdist is no longer a dependency and `-n` is rejected, so the suite can never run concurrently.
|
||||
|
||||
## General
|
||||
|
||||
- **Around 1959 tests across `tests/{unit,api,e2e}/`.** Playwright integration + HTTP + unit tests. All must pass before any merge.
|
||||
- **Tests use `-x` (fail-fast).** The suite stops at the first failure. Fix that test, then re-run.
|
||||
- **Validate each touched language manually:** Python (compile + import), JS (parse / bracket matching), CSS (brace matching), HTML (tag matching). Zero tolerance.
|
||||
- Playwright is used directly, NOT pytest-playwright (uninstall it if present - it conflicts).
|
||||
|
||||
## Playwright navigation
|
||||
|
||||
- **Every `page.goto()` must use `wait_until="domcontentloaded"`**, never the default `"load"`. CDN scripts and avatar images cause `load` to timeout.
|
||||
- **Every `page.wait_for_url()` must also use `wait_until="domcontentloaded"`** for the same reason.
|
||||
- **Prefer `page.locator(...).wait_for(state="visible")`** over bare `wait_for_selector` - it gives better error messages.
|
||||
- **Default timeout is 15 seconds** (increased from 10s for CDN script loading).
|
||||
|
||||
```python
|
||||
page.goto(f"{BASE_URL}/feed", wait_until="domcontentloaded")
|
||||
page.locator(".feed-fab").first.wait_for(state="visible", timeout=10000)
|
||||
```
|
||||
|
||||
## Delete button locator scoping
|
||||
|
||||
When both a post Delete and comment Delete button exist, always scope to the comment:
|
||||
|
||||
```python
|
||||
# CORRECT - scoped to comment:
|
||||
page.locator(".comment-action-btn:has-text('Delete')")
|
||||
|
||||
# WRONG - matches both post and comment Delete:
|
||||
page.locator("button:has-text('Delete')")
|
||||
```
|
||||
|
||||
## Confirm dialogs
|
||||
|
||||
Delete buttons carry `data-confirm`, which `ModalManager` turns into a native `confirm()`
|
||||
dialog. Playwright auto-dismisses dialogs (treated as Cancel), so the form never submits.
|
||||
Accept the dialog **before** clicking, or the delete silently does nothing:
|
||||
|
||||
```python
|
||||
page.once("dialog", lambda d: d.accept())
|
||||
page.locator(".comment-action-btn:has-text('Delete')").click()
|
||||
```
|
||||
|
||||
## Comment hash navigation
|
||||
|
||||
Opening a URL with a `#comment-<uid>` fragment scrolls the comment into view and adds the
|
||||
`comment-highlight` class (`NotificationManager.initHashScroll`). Tests assert on
|
||||
`#comment-<uid>.comment-highlight` after the page loads.
|
||||
|
||||
## Browser context
|
||||
|
||||
- **`browser_context` is session-scoped** (one per test session, shared by all tests in all files).
|
||||
- **Cookies are cleared per test via `browser_context.clear_cookies()`** in the `page` fixture.
|
||||
- **Each test gets a fresh `page`** from the shared context.
|
||||
- **`bob` fixture creates its own context** from the session `browser` - necessary for multi-user tests.
|
||||
- **Never share a page between two logged-in users** in the same test - use separate contexts.
|
||||
|
||||
## Test users
|
||||
|
||||
- **`alice_test` / `bob_test` are seeded once at session level** via HTTP POST to `/auth/signup`.
|
||||
- **`alice` fixture logs in alice_test** via the login form.
|
||||
- **`bob` fixture logs in bob_test** in a separate Playwright context.
|
||||
- **Use `alice` for single-user tests.** It returns `(page, user_dict)`.
|
||||
|
||||
## Required test patterns (summary)
|
||||
|
||||
- Every `page.goto(...)` and `page.wait_for_url(...)` MUST pass `wait_until="domcontentloaded"`. CDN libs and avatars cause the default `"load"` to time out.
|
||||
- Prefer `page.locator(...).wait_for(state="visible")` over `wait_for_selector`.
|
||||
- Scope ambiguous selectors: comment Delete is `.comment-action-btn:has-text('Delete')`; create-post Post button is `#create-post-modal button.btn-primary:has-text('Post')`; comment textarea is `.comment-form textarea[name='content']`.
|
||||
- Failure screenshots auto-save to `/tmp/devplace_test_screenshots/` via the `pytest_runtest_makereport` hook in `conftest.py`.
|
||||
|
||||
## Global `site_settings` tests
|
||||
|
||||
- **The uvicorn server is shared for the whole session.** A test that flips a global setting (maintenance mode, registration, rate limit, session length) changes behavior for every later test, so **restore the value in a `try/finally`** - see `tests/e2e/operational.py`.
|
||||
- **Saving through the admin form is the reliable mutation path**, not a direct DB write: the form handler runs in the server process and calls `clear_settings_cache()`, so the change takes effect immediately (a direct DB write waits out the 60s server-side cache).
|
||||
- **`conftest.py`'s `app_server` seeds `rate_limit_per_minute="1000000"`** so the suite is never throttled and the settings form round-trips a safe value instead of the template's `"60"` default.
|
||||
- **Rate-limit unit tests patch `m.get_int_setting`**, not the `RATE_LIMIT` constant - the constant is now only the fallback default passed to `get_int_setting`.
|
||||
- **Service enabled/disabled state is global state too - restore it.** Stopping a service through `POST /admin/services/{name}/stop` sets its `service_{name}_enabled` setting to `"0"` for the rest of the session. A test that stops a service (e.g. `test_gateway_disabled_returns_503` stops `openai` to assert the gateway returns 503) MUST restart it in a `try/finally` (`POST /admin/services/{name}/start`), and any `finally` that ends in `stop` should end in `start` instead - the default, expected state is enabled. Leaving the gateway stopped makes a later file's request hit `is_enabled()` first and get a `503` where it expected a `401`/`200`. The `openai` gateway's `handle()` checks `is_enabled()` (503) BEFORE `authorize()` (401), so a stopped gateway masks the auth result entirely.
|
||||
- **The 1,000,000/min rate-limit seed only applies after the server's 60s settings cache expires.** `conftest.py` seeds `rate_limit_per_minute` with a raw DB insert (no `clear_settings_cache()`), so for the first ~60s of server uptime the server still uses the `60`/min default. A high-volume request loop (e.g. `test_auth_matrix`) run in the FIRST minute can therefore trip a spurious `429`; in the full suite that test runs well past the window so it never does. If you reproduce a `429` only in a tiny isolated run, it is this warm-up artifact, not a real auth/rate-limit issue.
|
||||
- Operational tests live in `tests/e2e/operational.py` plus the `tests/api/admin/settings.py` family; rate-limit tests sit with the endpoint they probe (`tests/api/root.py`, `tests/api/auth/login.py`, `tests/api/robotstxt.py`).
|
||||
|
||||
## Failure handling
|
||||
|
||||
- **Failure screenshots auto-save** to `/tmp/devplace_test_screenshots/`.
|
||||
- **Tests stop at first failure** (`-x` flag in Makefile). No cascading failures.
|
||||
- **If the server won't start, kill leftover processes:** `kill -9 $(pgrep -f "uvicorn")`
|
||||
|
||||
## Common pitfalls
|
||||
|
||||
| Pitfall | Fix |
|
||||
|---------|------|
|
||||
| `goto`/`wait_for_url` times out | Add `wait_until="domcontentloaded"` |
|
||||
| CDN scripts block page load | Use `defer` on all `<script>` tags |
|
||||
| 500 on dataset `find()` | Use dict comparison syntax, not raw SQL |
|
||||
| N+1 query slowness | Use batch helpers: `get_users_by_uids()`, `get_comment_counts_by_post_uids()` |
|
||||
| Modal not opening/closing | Use `style.display`, not `classList.add/remove` |
|
||||
| Dual Delete buttons match | Scope to `.comment-action-btn` in tests |
|
||||
| Dual Post buttons match (feed inline comment) | Scope to `#create-post-modal button.btn-primary:has-text('Post')` in tests |
|
||||
| Edit modal textarea conflicts with comment textarea | Scope to `.comment-form textarea[name='content']` for comments |
|
||||
| Tests fail in sequence | Session-scoped context + `clear_cookies()` per test |
|
||||
| Double messages in chat | Deduplicate by message UID with `seen` set |
|
||||
| Avatar generation fails | Falls back to initial-based SVG - check multiavatar import |
|
||||
| `Http.getJson` returns HTML / `JSON.parse` throws | The helper sends `Accept: application/json`; a content-negotiated route (`respond(..., model=...)`) needs it. Don't hand-roll `fetch()` without that header against a negotiated route |
|
||||
| Page renders but data empty / 500 only after a CLI test ran earlier | A partial insert created the table with a reduced schema. Ensure the full column set in `init_db()` (see "Dataset rules -> `init_db()` MUST create every queried column") |
|
||||
| `no such column: X` 500 in the server log | Same root cause: a queried/indexed column is missing because the table was created lazily by a partial insert. Add the column to the `init_db()` ensure-block |
|
||||
| Action button click times out ("element is not visible") | The button moved into the `.project-actions-overflow` menu. Open it first: click `.project-actions-more`, then `.context-menu-item:has-text('<Label>')` (see `tests/e2e/projects/index.py`) |
|
||||
| `ImportError: cannot import name X from devplacepy.utils` in a test | The util was renamed (e.g. `badge_info` -> `get_badge`). Update the test import to the current name; grep `templating.py` globals for the new name |
|
||||
| Admin page heading assertion fails | Admin pages use `<h2>` inside `.admin-toolbar`, not `<h1>` - assert `text=` or `.admin-toolbar h2:has-text(...)` |
|
||||
| `test_auth_matrix` reports `422` instead of `401`/`403` | A POST route's form params are undocumented in `docs_api.py`, so the matrix sends an empty body and FastAPI validation (422) fires before the auth guard. Document every required form param so auth is reached and tested |
|
||||
| `test_auth_matrix` reports `503` on a gateway endpoint | A prior test left the `openai` service stopped. Restore it in `finally` (see "Global `site_settings` tests -> Service enabled/disabled state") |
|
||||
Loading…
Reference in New Issue
Block a user