docs: add block/mute user relations, emoji-sync CLI, and uid indexes
DevPlace CI / test (push) Failing after 2m7s
DevPlace CI / test (push) Failing after 2m7s
- Add `/block`, `/mute` endpoints with block/unblock and mute/unmute functionality in `routers/relations.py`, hiding blocked users' content everywhere except their own profile while muting only suppresses notifications - Introduce `devplace emoji-sync` CLI command to regenerate `static/js/emoji-shortcodes.js` from the emoji library, documented in `CLAUDE.md` and wired in `cli.py` - Create `get_blocked_uids()` database helper and apply it in `content.py` `load_detail()` to filter blocked users' posts from detail views - Implement `_uid_index()` and `_drop_index()` helpers in `database.py` for unique uid indexes across tables, with `user_relations` added to `SOFT_DELETE_TABLES` - Document new routes in `AGENTS.md` and `README.md`, including emoji shortcodes rendering behavior distinct from the emoji picker
This commit is contained in:
@@ -52,6 +52,7 @@ devplace containers reconcile # run one reconcile pass (desired vs docker ps)
|
||||
devplace containers prune # reap orphan containers + dangling images
|
||||
devplace containers prune-builds # remove legacy per-project images + clear dockerfiles/builds tables (one-time)
|
||||
devplace containers gc-workspaces # remove workspace dirs with no instances
|
||||
devplace emoji-sync # regenerate static/js/emoji-shortcodes.js from the emoji library (run after an emoji dep bump)
|
||||
devplace migrate-data # relocate legacy runtime files into data/ (idempotent; --dry-run to preview)
|
||||
```
|
||||
|
||||
@@ -126,6 +127,7 @@ Docs (`routers/docs/pages.py` `DOCS_PAGES`, re-exported from the `routers/docs`
|
||||
### Templates and frontend
|
||||
- All routers MUST import the shared `templates` from `devplacepy.templating`. Do NOT instantiate `Jinja2Templates` per router - globals (`avatar_url`, `get_unread_count`, `get_user_projects`, `format_date`) are registered on that single instance.
|
||||
- Templates extend `base.html`, add page CSS via `{% block extra_head %}`, page JS via `{% block extra_js %}`.
|
||||
- **Nav active state is the `nav_active` Jinja global**, never an inline path check. `nav_active(request, *prefixes, exact=False)` (`templating.py`) returns `"active"` (or `""`) when `request.url.path` equals a prefix or is a sub-path of it (`exact=True` matches only the exact path, e.g. the admin Dashboard root). Every topnav/sidebar link uses it (`class="topnav-link {{ nav_active(request, '/projects') }}"`); do NOT hand-write `{% if request.url.path == ... or request.url.path.startswith(...) %}active{% endif %}`.
|
||||
- **Static asset URLs are boot-versioned for cache-busting.** Never hardcode a bare `/static/...` href/src: templates wrap it in the `static_url(path)` Jinja global and runtime JS that builds an absolute static URL uses `assetUrl(path)` (`static/js/assetVersion.js`). Both emit a `/static/v<STATIC_VERSION>/...` path segment (`config.STATIC_VERSION` = `DEVPLACE_STATIC_VERSION` env or the boot unix timestamp), so a deploy busts every asset while it stays served `immutable, max-age=31536000`. The version is in the PATH (not a `?v=` query) so relative ES6 imports and relative CSS `url()` inherit it automatically. `/static/uploads/` (user content) and `service-worker.js` are excluded. See `AGENTS.md` -> "Static asset caching and versioning".
|
||||
- All JS in `static/js/` is ES6 modules, one class per file, instantiated as `app` and reachable everywhere. The browser-side application root is `Application.js`.
|
||||
- **Custom web components** live in `static/js/components/` (prefix `dp-`): `dp-avatar`, `dp-code`, `dp-content` (wraps the `contentRenderer` markdown/sanitize/media engine behind `data-render`) and `dp-title` (its inline title-sized companion) - **both are now used ONLY for live/client-generated content** (chat, planning report, dynamically inserted bubbles); server-rendered content/titles use the backend `render_content`/`render_title` globals instead (see Content rendering pipeline). The components are intentionally kept for those live contexts and future use, `dp-toast`, `dp-dialog`, `dp-context-menu`, `dp-upload` (the single file-upload button used everywhere; replaced the old `AttachmentUploader`), `dp-lightbox` (`app.lightbox`: the single image lightbox, attribute-wired via one delegated listener so any `img[data-lightbox]` opens full-size - `data-full` for a distinct full-res URL; attachment thumbnails and all `data-render` markdown images are marked centrally, the latter in `ContentEnhancer`). Each extends `Component` (a thin `HTMLElement` base with `attr`/`boolAttr`/`intAttr` helpers), renders into the **light DOM** (no shadow root, so global CSS applies) and self-registers via `customElements.define` at the bottom of its file. `components/index.js` imports them all and is imported by `Application.js`, so every element is defined site-wide. The singletons are created once in `Application.js` and exposed as `app.dialog` / `app.contextMenu` / `app.toast`. Self-contained presentational widgets become components; partial-bound controllers (votes, reactions, comments) and pure utilities stay plain modules because they enhance server-rendered markup rather than render standalone UI. They are documented in the public docs **Components** section.
|
||||
@@ -135,17 +137,21 @@ Docs (`routers/docs/pages.py` `DOCS_PAGES`, re-exported from the `routers/docs`
|
||||
- The three public listings (`/feed`, `/gists`, `/projects`) share one search box: the `templates/_sidebar_search.html` partial (included at the top of `.sidebar-card` with `_action`/`_placeholder`/`_hidden` locals) plus the `database.text_search_clause(table, search, fields)` data helper. Any new listing with a left filter panel reuses both - never re-inline an `ilike` search clause or hand-roll another search form. See `AGENTS.md` -> "Listing search".
|
||||
|
||||
### Content rendering pipeline
|
||||
**Server-rendered content and titles are rendered on the BACKEND for SEO** (`devplacepy/rendering.py`, exposed as the Jinja globals `render_content(text)` and `render_title(text)`, registered in `templating.py`). This is the default for all content that exists at request time: post bodies/cards, comments, news articles, project/gist descriptions, DM history, and EVERY content title. `render_content` mirrors the client `ContentRenderer.js` pipeline exactly using **mistune** (the markdown engine already used for docs prose): normalize em-dash -> hyphen, emoji shortcodes (the same `EMOJI_MAP` ported from JS), GFM markdown (`escape=True` so raw user HTML is escaped - the server XSS control, replacing client `DOMPurify`; `strikethrough`+`table` plugins, `hard_wrap`), then a stdlib `HTMLParser` media pass that walks text nodes (skipping `a`/`code`/`pre`) and turns bare URLs into YouTube iframes / `<img data-lightbox>` / `<video>` / `<audio>` / autolinks and `@mentions` into profile links - byte-for-byte the same markup the web component emits. Link/image URL schemes are validated (`javascript:`/`data:` -> `#`). `render_title` is the inline variant: emoji + inline markdown only, filtered to an inline-tag allowlist (no `a`/`img`/block tags), matching `dp-title`. Both are `@lru_cache`d (deterministic from input) so repeated/identical content across a feed page is rendered once - render speed. Server-rendered code blocks (`<pre><code class="language-x">`) are syntax-highlighted client-side and get the upper-right **Copy** button by `ContentEnhancer` (`highlightAll()` + `enhanceCodeBlocks()`, which calls the shared `CodeBlock.enhance(pre, {lineNumbers:false})` on every `.rendered-content pre` on load - the same primitive the docs/`dp-code` use; the generic `pre.code-has-copy > .code-copy-btn` styling lives in the global `markdown.css`). Content `<img data-lightbox>` is wired by the delegated `app.lightbox` listener - no per-element JS.
|
||||
**Server-rendered content and titles are rendered on the BACKEND for SEO** (`devplacepy/rendering.py`, exposed as the Jinja globals `render_content(text)` and `render_title(text)`, registered in `templating.py`). This is the default for all content that exists at request time: post bodies/cards, comments, news articles, project/gist descriptions, DM history, and EVERY content title. `render_content` mirrors the client `ContentRenderer.js` pipeline exactly using **mistune** (the markdown engine already used for docs prose): normalize em-dash -> hyphen, emoji shortcodes (the FULL GitHub/Discord `:name:` set, see **Emoji shortcodes** below), GFM markdown (`escape=True` so raw user HTML is escaped - the server XSS control, replacing client `DOMPurify`; `strikethrough`+`table` plugins, `hard_wrap`), then a stdlib `HTMLParser` media pass that walks text nodes (skipping `a`/`code`/`pre`) and turns bare URLs into YouTube iframes / `<img data-lightbox>` / `<video>` / `<audio>` / autolinks and `@mentions` into profile links - byte-for-byte the same markup the web component emits. Link/image URL schemes are validated (`javascript:`/`data:` -> `#`). `render_title` is the inline variant: emoji + inline markdown only, filtered to an inline-tag allowlist (no `a`/`img`/block tags), matching `dp-title`. Both are `@lru_cache`d (deterministic from input) so repeated/identical content across a feed page is rendered once - render speed. Server-rendered code blocks (`<pre><code class="language-x">`) are syntax-highlighted client-side and get the upper-right **Copy** button by `ContentEnhancer` (`highlightAll()` + `enhanceCodeBlocks()`, which calls the shared `CodeBlock.enhance(pre, {lineNumbers:false})` on every `.rendered-content pre` on load - the same primitive the docs/`dp-code` use; the generic `pre.code-has-copy > .code-copy-btn` styling lives in the global `markdown.css`). Content `<img data-lightbox>` is wired by the delegated `app.lightbox` listener - no per-element JS.
|
||||
|
||||
The CLIENT pipeline `ContentRenderer.js` (and its `dp-content`/`dp-title` web components) is **retained ONLY for genuinely live/dynamic content** that does not exist at request time: newly posted comments (`CommentManager`), live DM bubbles (`MessagesLayout`), Devii/DeepSearch/Docs chat, and the planning report. It runs on every element with `data-render`, in this exact order: emoji shortcode → markdown via `marked` → **`DOMPurify.sanitize`** → `highlight.js` on `<pre><code>` → image URL → YouTube URL → autolink. Uses a `NodeIterator` that skips `CODE`/`PRE`/`SCRIPT`/`STYLE` so URLs inside code blocks are never rewritten. **Do NOT add `data-render` to server-rendered content - call `render_content`/`render_title` instead** (SEO; the rule of thumb: if the text is known when the template renders, render it on the backend).
|
||||
|
||||
`DOMPurify.sanitize` runs on the raw `marked` output (untrusted) before `processMedia` injects trusted YouTube iframes, so user content cannot inject script/event-handler/iframe payloads while our own embeds survive. Sanitization is fail-closed: `render()` throws if `DOMPurify` is not loaded rather than emitting unsanitized HTML. Because content is rendered client-side from `element.textContent`, Jinja autoescaping alone does not protect it - this sanitize step is the XSS control. Server-side, JSON-LD emitted via `{{ page_schema | safe }}` is escaped in `seo.py` `_json_ld_dumps` (`<`/`>`/`&` and line separators → `\uXXXX`) to prevent `</script>` breakout.
|
||||
|
||||
**Emoji shortcodes (single source of truth).** `:name:` shortcodes are the **full GitHub/Discord set** (~4869 names, e.g. `:rocket:`->🚀, `:smiling_imp:`->😈), NOT a curated allowlist. `rendering.py` `build_emoji_shortcodes()` is the ONE generator: it walks the `emoji` library's `EMOJI_DATA` (every `en`/`alias` name of a fully-qualified emoji) and merges `CUSTOM_SHORTCODES` (the handful of legacy extras the library lacks, e.g. `party`/`tools`). The backend `EMOJI_MAP = build_emoji_shortcodes()` is built ONCE at import and used by `_replace_shortcodes` (first step of `render_content`/`render_title`). The frontend gets the **same** map as a generated plain-ES6 module `static/js/emoji-shortcodes.js` (`export const EMOJI_SHORTCODES`), written by `write_emoji_module()` and imported by `ContentRenderer.js` (so `dp-content`/`dp-title` and every live renderer share it). The module is a committed build artifact - regenerate it with `devplace emoji-sync` after bumping the `emoji` dependency, never hand-edit it. Unknown shortcodes pass through verbatim; shortcodes inside code spans/blocks are never expanded. The separate visual `emoji-picker-element` composer button is unrelated (it inserts the literal emoji char). User docs: `/docs/emoji-shortcodes`.
|
||||
|
||||
**Template em-dash normalization.** The shared `templates.env.template_class` is overridden (`templating.py` `_Template`) so EVERY rendered template runs its final HTML through `normalize_em_dash` - the em-dash character (U+2014) and its HTML entity forms (the named entity plus decimal 8212 / hex 2014) all become a plain `-`, application-wide, covering literal template text, `{{ variables }}`, includes/macros, and injected `render_content` HTML. One hook; never re-strip em-dash per template. (`render_content`/`render_title` also normalize em-dash in their own pipeline.)
|
||||
|
||||
### Auth
|
||||
Session cookies named `session`, value is a 64-char hex token from `secrets.token_hex(32)`. Passwords hashed with `pbkdf2_sha256` via passlib. `get_current_user(request)` returns user dict or `None`, with a per-process TTL cache (`_user_cache`, 300s) keyed by token. `require_user(request)` raises a 303 redirect to `/` for guests; `require_admin(request)` redirects to `/feed` for non-admins. Public pages (feed, news detail, project detail, gists) use `get_current_user` so guests can browse - POSTs are all guarded by `require_user`.
|
||||
|
||||
### Database (`devplacepy/database.py`)
|
||||
SQLite via `dataset.connect` with WAL + 30s busy timeout + 256MB mmap. `init_db()` is idempotent - every index is created via `CREATE INDEX IF NOT EXISTS` wrapped in try/except, and seed defaults for `site_settings` only insert when the key doesn't already exist. **`init_db()` also ensures the full column set of any table that code filters on** (`news`, `news_sync`, `attachments` use `get_table(name)` + `create_column_by_example` before their indexes). This is mandatory, not optional: dataset gives a lazily-created table ONLY the columns of its first insert, so a partial insert from a CLI tool/test/maintenance script would otherwise create the table with a reduced schema, and the long-running server caches that stale schema - making later queries on the missing column 500 with `no such column`. When adding a new filtered/indexed column, add it to the matching `init_db()` ensure-block (see `AGENTS.md` -> "Dataset rules -> `init_db()` MUST create every queried column").
|
||||
SQLite via `dataset.connect` with WAL + 30s busy timeout + 256MB mmap. `init_db()` is idempotent - every index is created via `CREATE INDEX IF NOT EXISTS` wrapped in try/except, and seed defaults for `site_settings` only insert when the key doesn't already exist. **`init_db()` also ensures the full column set of any table that code filters on** (`news`, `news_sync`, `attachments` use `get_table(name)` + `create_column_by_example` before their indexes). This is mandatory, not optional: dataset gives a lazily-created table ONLY the columns of its first insert, so a partial insert from a CLI tool/test/maintenance script would otherwise create the table with a reduced schema, and the long-running server caches that stale schema - making later queries on the missing column 500 with `no such column`. When adding a new filtered/indexed column, add it to the matching `init_db()` ensure-block (see `AGENTS.md` -> "Dataset rules -> `init_db()` MUST create every queried column"). The `_index(...)` helper supports `where=` (partial) and `unique=` indexes; every table with a `uid` column gets a UNIQUE `idx_<table>_uid`, soft-delete tables get a PARTIAL `idx_<table>_trash` (`WHERE deleted_at IS NOT NULL`) and NEVER a bare `deleted_at` index (it mis-steers the planner on live reads), and "live newest-first" listings need a composite/live-partial index that carries the sort column. `init_db()` finishes with `ANALYZE`/`PRAGMA optimize`. Verify any index change with `EXPLAIN QUERY PLAN` (see `AGENTS.md` -> "Indexing conventions (the soft-delete planner trap)").
|
||||
|
||||
**SQLite is synchronous by design and will never be made async (hard rule).** The `dataset`/SQLite layer is synchronous and is called directly inside the async route handlers. This is intentional and more than fast enough for this platform: the database is a local file tuned with WAL, `synchronous=NORMAL`, a 30s busy timeout, an 8MB page cache, and a 256MB mmap, so queries are sub-millisecond and never meaningfully stall the event loop. Do NOT convert the database layer to async, do NOT wrap DB calls in a threadpool/`run_in_executor`/`to_thread`, and do NOT raise the synchronous DB access as a performance, blocking-I/O, or "requests aren't fully async" concern. This is a settled, sophisticated design choice and is not open for revisiting.
|
||||
|
||||
@@ -236,7 +242,7 @@ A second REST surface mounted at `/api` that mimics the public devRant API on De
|
||||
- **All dates shown to users are DD/MM/YYYY** (European) and rendered **in the viewer's own timezone, client-side**. Timestamps are stored/emitted as UTC ISO; the display is localized in the browser. Use the `local_dt(iso, mode)` / `dt_ago(iso)` Jinja globals (`templating.py`) for any user-facing **instant** - they emit `<time datetime="UTC-ISO" data-dt data-dt-mode="date|datetime|ago">server-fallback</time>`, and `static/js/LocalTime.js` (`app.localTime`) reformats every `[data-dt]` to the browser's local timezone (`Intl`), with a `MutationObserver` so dynamically-inserted dates (feed, messages, comments) localize automatically and a 60s refresh for relative times. The server fallback (the `<time>` text content, via `format_date`/`time_ago`) is what no-JS clients see. **`format_date()`/`time_ago()` stay as plain-text helpers** for JSON responses, no-JS fallbacks, and the few non-timestamp or attribute-context dates (e.g. project `release_date`/`demo_date`, which are user-entered DD/MM/YYYY strings, and `value=`/`title=` attribute uses) - do NOT wrap those in `local_dt`. Pattern for converting a precomputed `time_ago` string display: `{{ dt_ago(x.created_at) if x.created_at else x.time_ago }}` (falls back to the original string if the raw ISO is absent). New user-facing timestamp displays should use `local_dt`/`dt_ago`.
|
||||
- **Slug + UUID lookup:** resources with slugs (posts, gists, news, projects) accept either the slug or the bare UUID - see `resolve_by_slug()` in `database.py`. Slugs are generated via `make_combined_slug(title, uid)` which prefixes the last 12 hex chars of the UUID (the uuid7 **random tail**, NOT its leading bytes - the first 48 bits are a millisecond timestamp, so a head prefix is identical for all rows written in the same ~65s window and collides with the same title, making `resolve_by_slug` ambiguous; same reasoning as the blob-shard tail rule).
|
||||
- **Username:** 3-32 chars, letters/numbers/hyphens/underscores. **Password:** min 6 chars.
|
||||
- **Roles are stored capitalized:** the `users.role` value is exactly `"Admin"` or `"Member"` (the first registered user becomes `"Admin"`; `auth.py`). Always test admin-ness through the `is_admin(user)` global (`utils.py`, also a Jinja global), which compares `user.get("role") == "Admin"` case-sensitively - never hand-roll a lowercase compare. The **only** lowercase surface is the CLI (`devplace role set <username> <member|admin>` accepts lowercase but writes `role.capitalize()`; `role get` prints `.lower()` for display). A lowercase role in the DB silently fails every admin check.
|
||||
- **Roles are stored capitalized:** the `users.role` value is exactly `"Admin"` or `"Member"` (the first registered user becomes `"Admin"`; `auth.py`). Always test admin-ness through the `is_admin(user)` global (`utils.py`, also a Jinja global), which compares `user.get("role") == "Admin"` case-sensitively - never hand-roll a lowercase compare. The **only** lowercase surface is the CLI (`devplace role set <username> <member|admin>` accepts lowercase but writes `role.capitalize()`; `role get` prints `.lower()` for display). A lowercase role in the DB silently fails every admin check. **Any write to `users.role` MUST call `database.invalidate_admins_cache()`** - the admin set is cached (`_admins_cache`, cross-worker `admins` version) behind `get_admin_uids`/`get_primary_admin_uid`; the three current role-write sites (`routers/admin/users.py`, `cli.py`, `utils._create_account` first-user) already do. See `AGENTS.md` -> "Multi-worker concurrency -> Admin-set cache invariant".
|
||||
- **Admin seniority: a junior admin cannot manage a more senior admin.** Every per-user mutation in `routers/admin/users.py` (role/password/toggle/reset-ai-quota) is gated by `_is_senior_admin(actor, target)` - it blocks (audits `result="denied"`) when the target is an Admin who registered earlier (`(created_at, id)` ascending, matching `get_primary_admin_uid()`). Members and junior admins stay manageable; the guard is server-side so it also covers Devii's admin tools. See AGENTS.md -> "Admin seniority guard".
|
||||
- **Never pass a `respond()` context key that collides with a Jinja global.** `respond(request, template, ctx, model=XOut)` feeds the **same** `ctx` dict to both the Pydantic model (JSON) and the template render. Jinja context variables shadow `env.globals` across the whole inheritance chain, so a key like `is_admin`/`avatar_url`/`format_date`/`is_self`/`owns`/`guest_disabled` holding a non-callable value turns `base.html`'s `{% if is_admin(user) %}` into `False(user)` -> `TypeError: 'bool' object is not callable`, a 500 that only fires for the branch that calls the global (e.g. logged-in users, not guests). Name viewer/permission flags distinctly (`viewer_is_admin`, not `is_admin`) in both the schema and the context. This was a real issue on `/issues/{number}`; `tests/api/issues/create.py::test_issue_detail_renders_for_{member,admin}` guard it.
|
||||
- **Avatars:** Multiavatar SVG generated locally from the username seed in <5ms, in-memory cached (cleared on restart). URL `/avatar/multiavatar/{seed}?size={size}`. Falls back to initial-based SVG on error.
|
||||
|
||||
Reference in New Issue
Block a user