devplacepy/AGENTS.md

# DevPlace - Agent Guide

## Quick start

```bash
make install      # pip install -e .
make dev          # uvicorn --reload on port 10500, backlog 4096
make prod         # uvicorn with 2 workers, backlog 8192 (production)
make test         # Playwright integration + unit tests (fail-fast -x)
make test-headed  # same tests in visible browser
make locust       # Locust load test (interactive web UI)
make locust-headless  # Locust in headless CLI mode (for CI)
```

**Env vars:** `DEVPLACE_DISABLE_SERVICES=1` prevents the NewsService (and future background services) from starting. Automatically set during tests.

## Architecture

- **FastAPI** backend serving **Jinja2 templates** (SSR). Pure ES6 JS for interactivity.
- **Database:** `dataset` (auto-syncs schema, uses `uid` for PKs). SQLite.
- **Auth:** Session cookies (`session` cookie), SHA256+SALT via passlib. No JWT.
- **Static:** `devplacepy/static/` mounted at `/static`
- **Templates:** `devplacepy/templates/`. Shared `templates` instance from `devplacepy.templating` - all routers import from there, do NOT create their own.
- **Ports:** 10500 (dev), 10501 (tests)
- **Username:** letters, numbers, hyphens, underscores only. 3-32 chars.
- **Password:** minimum 6 chars.
- **Avatars:** Multiavatar-based (local SVG generation, zero network). URL: `/avatar/multiavatar/{seed}`. Generation takes <5ms. Fallback to initial-based SVG on error. Cache is in-memory (cleared on restart).

## Routing

| Prefix | Router file |
|--------|-------------|
| `/auth` | `routers/auth.py` |
| `/feed` | `routers/feed.py` |
| `/news` | `routers/news.py` |
| `/posts` | `routers/posts.py` |
| `/comments` | `routers/comments.py` |
| `/projects` | `routers/projects.py` |
| `/profile` | `routers/profile.py` |
| `/messages` | `routers/messages.py` |
| `/notifications` | `routers/notifications.py` |
| `/votes` | `routers/votes.py` |
| `/reactions` | `routers/reactions.py` |
| `/bookmarks` | `routers/bookmarks.py` |
| `/polls` | `routers/polls.py` |
| `/avatar` | `routers/avatar.py` |
| `/follow` | `routers/follow.py` |
| `/leaderboard` | `routers/leaderboard.py` |
| `/admin` | `routers/admin.py` |
| `/bugs` | `routers/bugs.py` |
| `/gists` | `routers/gists.py` |
| `/admin/services` | `routers/services.py` |
| `(none)` | `routers/seo.py` (`/robots.txt`, `/sitemap.xml`) |

## Content Rendering Pipeline

`ContentRenderer.js` processes all user-generated text in this exact order:

1. **Emoji shortcodes** → Unicode emoji (`:fire:` → 🔥, 80+ shortcodes)
2. **Markdown parse** → via `marked` with GFM tables, line breaks
3. **Sanitize** → `DOMPurify.sanitize` strips script/event-handler/iframe/`javascript:` payloads from the marked output
4. **Code syntax highlight** → `highlight.js` on all `<pre><code>` blocks
5. **Image URLs** → standalone `.jpg/.png/.gif` URLs become `<img>` tags
6. **YouTube URLs** → `youtube.com/watch?v=` or `youtu.be/` become embedded iframe players
7. **All URLs** → become `<a>` links with `target="_blank"` and `rel="noopener"`

**Sanitization is the XSS control.** Content is rendered client-side from `element.textContent`, so Jinja autoescaping does not protect it. `DOMPurify.sanitize` (vendored at `static/vendor/purify.min.js`, loaded `defer` in `base.html`) runs on the raw `marked` output before `processMedia` injects the trusted YouTube iframes, so user payloads are removed while our embeds survive. It is fail-closed: `render()` throws if `DOMPurify` is missing rather than emitting unsanitized HTML — never relax this into a `typeof` skip.

**Code blocks are protected** - `NodeIterator` skips `CODE`, `PRE`, `SCRIPT`, `STYLE` elements during URL/media processing, so source code in markdown code blocks is never touched.

Elements with `data-render` attribute are auto-rendered by `Application.js` on page load. The `.rendered-content` CSS class provides table styles, code block backgrounds, and image sizing.

## CDN Libraries

Loaded via `<script>` tags in `base.html`. ALL must use `defer` to avoid blocking DOMContentLoaded:

```html
<script defer src="https://cdn.jsdelivr.net/npm/marked/lib/marked.umd.js"></script>
<script defer src="https://cdn.jsdelivr.net/gh/highlightjs/cdn-release@11.11.1/build/highlight.min.js"></script>
<script defer src="/static/vendor/purify.min.js"></script>
<script type="module" src="https://cdn.jsdelivr.net/npm/emoji-picker-element@^1/index.js"></script>
<script defer src="/static/js/ContentRenderer.js"></script>
<script defer src="/static/js/EmojiPicker.js"></script>
<script type="module" src="/static/js/Application.js"></script>
```

**Never use `<script>` without `defer` for CDN libraries** - they block HTML parsing and cause `wait_until="domcontentloaded"` to timeout in Playwright tests.

## Emoji Picker

Uses `emoji-picker-element` web component (Discord-style, searchable, skin tones):
- `EmojiPicker.js` wraps it with a toggle button and inserts unicode at cursor position
- Added to all `.comment-form textarea` and `.emoji-picker-target` elements
- The old `&#x1F600;` emoji button has been removed from all templates

## Modal System

`Application.js` `initModals()` toggles the `.visible` CSS class on the modal overlay. The CSS rule `.modal-overlay.visible { display: flex; }` handles visibility:

```javascript
// CORRECT - toggle the .visible class on the modal:
modal.classList.add("visible");    // show
modal.classList.remove("visible"); // hide
```

Always call `e.preventDefault()` on `[data-modal]` click handlers since trigger elements often have `href="#"`:

```javascript
trigger.addEventListener("click", (e) => {
    e.preventDefault();
    modal.style.display = "flex";
});
```

The `modal-close` class is handled by `Application.js` - no inline JS needed in templates for basic modals.

Do NOT hand-write the overlay/header markup. Use the shared macro in `templates/_macros.html`:

```jinja
{% from "_macros.html" import modal %}
{% call modal('create-post-modal', 'Create New Post') %}{# wide=true for modal-card-wide #}
    <form ...> ... <div class="modal-footer">...</div> </form>
{% endcall %}
```

## Shared template partials

Reuse these via `{% set _x = ... %}{% include %}` (the `_avatar_link.html` convention) instead of copy-pasting markup:

- `_post_votes.html` — post +/- vote bar. Locals: `_uid`, `_my_vote`, `_count`.
- `_star_vote.html` — project/gist star button. Locals: `_type` (`project`|`gist`), `_uid`, `_my_vote`, `_count`, `_btn_class`, optional `_stop` (adds `data-stop-propagation`). The star glyph (`☆`→`★` when `.voted`) comes from the `vote-star` CSS class via `::before` (`base.css`) — do not put a literal star in markup.
- `_post_header.html` — post author/avatar/time header (`.post-header`). Locals: `_author`, `_time`.
- `_topic_selector.html` — topic radio group. Locals: `_topics`, `_selected`.

Vote button styles live ONCE in `feed.css` (`.post-action-btn`) — never redefine them in `post.css`. Page-specific CSS goes in a `static/css/*.css` file referenced from `{% block extra_head %}`, never an inline `<style>` block.

## Database

SQLite via `dataset` with these pragmas on every connection:

```python
PRAGMA journal_mode=WAL;      -- concurrent readers + writers
PRAGMA synchronous=NORMAL;     -- safe with WAL mode
PRAGMA busy_timeout=30000;     -- wait 30s instead of failing on lock
PRAGMA cache_size=-8000;       -- 8MB page cache
PRAGMA temp_store=MEMORY;      -- temp tables in memory
```

Configured via `dataset.connect(engine_kwargs={"connect_args": {"timeout": 30, "check_same_thread": False}}, on_connect_statements=[...])`.

All indexes are created via `_index()` helper wrapped in try/except - safe to run on every startup regardless of table state.

## Dataset rules (hard-learned)

**`find()` does NOT accept raw SQL strings.** It takes keyword arguments for equality filters, dict comparison operators, or SQLAlchemy column expressions.

```python
# WRONG - causes 500 Internal Server Error:
table.find("created_at >= :start", {"start": today})
table.find(text("created_at >= :start"), start=today)

# CORRECT - dict comparison syntax:
table.find(created_at={">=": today})

# CORRECT - keyword equality:
table.find(country="France")

# CORRECT - SQLAlchemy column expression for IN clause:
table.find(table.table.columns.user_uid.in_(["uid1", "uid2"]))

# CORRECT - multiple equality filters combined:
table.find(topic="devlog", user_uid=some_uid)
```

**`update()` requires a key column list as second argument.** The first dict contains all fields including the key column.

```python
table.update({"uid": user_uid, "bio": "new bio"}, ["uid"])
```

**`db.query()` accepts raw SQL with named params as keyword arguments:**

```python
db.query("SELECT * FROM posts WHERE topic = :t", t="devlog")
# NOT: db.query("...", {"t": "devlog"})
```

**Always check `tables` list before raw SQL queries:**

```python
if "comments" not in db.tables:
    return {}  # table doesn't exist yet
```

**Batch queries eliminate N+1 problems.** Use `get_users_by_uids()`, `get_comment_counts_by_post_uids()`, and `get_vote_counts()` from `database.py` instead of per-row lookups in loops.

## FastAPI patterns

- **All routes are async.** Form data is validated via a typed Pydantic body param: `data: Annotated[SomeForm, Form()]` (models in `models.py`). Read raw `await request.form()` only when also handling an uploaded file (a separate `File()` param would embed the model under its parameter name).
- **Return `RedirectResponse(url=..., status_code=302)`** for redirects.
- **Return `templates.TemplateResponse("name.html", {...})`** from `devplacepy.templating` to render.
- **Never create your own `Jinja2Templates` instance.** Import the shared one: `from devplacepy.templating import templates`.
- **Register new routers in `main.py`:** `app.include_router(router_instance, prefix="/{path}")`
- **`require_user(request)` raises 303 redirect to `/`** if not authenticated. Only post/comment/vote/etc. routes use this - the feed is public.
- **For a missing detail resource, `raise not_found("X not found")`** (`utils.py`) — it returns an `HTTPException(404)` that the global handler renders as `error.html`. Do not return a bare `HTMLResponse(..., status_code=404)`.
- **Detail pages reuse `load_detail(table, target_type, slug, user)`** (`content.py`) for item+author+comments+attachments+`star_count`+`my_vote`; list pages reuse `enrich_items(items, key, authors, extra_maps, user=...)`. Prefer these over manual per-row loading (posts/projects/gists detail and feed/gists/profile lists already do).
- **`get_current_user(request)` is cached** in `_user_cache` dict by session token (per-process, no TTL). Use this for pages viewable by both auth guests (feed, news detail, projects).
- **Post deletion must cascade:** delete comments and votes first, then the post. Always check ownership: `post["user_uid"] == user["uid"]`.
- **Message deduplication needed** when `sender_uid == receiver_uid` (messaging yourself): `seen = set()` of message UIDs before appending to result list.

## Key conventions

- No comments/docstrings in source - code is self-documenting.
- Forbidden variable name patterns: `_new`, `_old`, `_temp`, `_v2`, `better_`, `my_`, `the_` (see CLAUDE.md for full list).
- Form validation uses Pydantic models in `models.py` via `Annotated[Model, Form()]` params; invalid input is caught by the global `RequestValidationError` handler in `main.py` (auth pages re-render with messages at 400, other routes redirect).
- Template globals: `get_unread_count(user_uid)`, `get_unread_messages(user_uid)`, `get_user_projects(user_uid)`, `avatar_url(style, seed, size)`, `format_date(dt_str, include_time=False)` (ISO → `DD/MM/YYYY` or `DD/MM/YYYY HH:MM`).
- `DEVPLACE_DATABASE_URL` env var overrides the SQLite path (used by tests).
- All `RedirectResponse` must use `status_code=302` (integer, not `status` module).
- All `dataset` operations are synchronous and run in the async event loop - keep them fast. No external HTTP calls in request handlers.
- For ownership-sensitive operations (delete, edit), always check `user["uid"]` against the resource's `user_uid`.

## Clickable Avatars & Usernames

Every avatar and username in the UI links to the user's profile page. Use the `_avatar_link.html` and `_user_link.html` include components:

```html
{% set _user = item.author %}
{% set _size = 32 %}
{% set _size_class = "sm" %}
{% include "_avatar_link.html" %}

<a href="/profile/{{ user['username'] }}" class="post-author-link">{{ user['username'] }}</a>
```

The include files expect: `_user` (dict), `_size` (pixels), `_size_class` ("sm"|"md"|"lg").

Affected templates: `base.html`, `feed.html`, `post.html`, `profile.html`, `messages.html`, `notifications.html`.

## Image Upload

When a user uploads an image during post creation, the markdown `![](/static/uploads/{filename})` is appended to the post content. The ContentRenderer then renders it as an `<img>`. All URLs are relative.

```python
content += f"\n\n![](/static/uploads/{image_filename})"
```

File validation: max 5MB, allowed extensions: `.png`, `.jpg`, `.jpeg`, `.gif`, `.webp`, `.svg`.

## Testing patterns

### General

- **148 tests across 14 files.** Playwright integration + unit tests. All must pass before any merge.
- **Tests use `-x` (fail-fast).** The suite stops at the first failure. Fix that test, then re-run.
- **NEVER run tests unless specifically asked by user.** Not the full suite, not a single file - do not run any tests unless the user explicitly requests it.
- **`hawk .` validates Python (compile + AST), JS (bracket matching), CSS (brace matching), HTML (tag matching).** Zero tolerance.

### Playwright navigation

- **Every `page.goto()` must use `wait_until="domcontentloaded"`**, never the default `"load"`. CDN scripts and avatar images cause `load` to timeout.
- **Every `page.wait_for_url()` must also use `wait_until="domcontentloaded"`** for the same reason.
- **Prefer `page.locator(...).wait_for(state="visible")`** over bare `wait_for_selector` - it gives better error messages.
- **Default timeout is 15 seconds** (increased from 10s for CDN script loading).

```python
page.goto(f"{BASE_URL}/feed", wait_until="domcontentloaded")
page.locator(".feed-fab").first.wait_for(state="visible", timeout=10000)
```

### Delete button locator scoping

When both a post Delete and comment Delete button exist, always scope to the comment:

```python
# CORRECT - scoped to comment:
page.locator(".comment-action-btn:has-text('Delete')")

# WRONG - matches both post and comment Delete:
page.locator("button:has-text('Delete')")
```

### Browser context

- **`browser_context` is session-scoped** (one per test session, shared by all tests in all files).
- **Cookies are cleared per test via `browser_context.clear_cookies()`** in the `page` fixture.
- **Each test gets a fresh `page`** from the shared context.
- **`bob` fixture creates its own context** from the session `browser` - necessary for multi-user tests.
- **Never share a page between two logged-in users** in the same test - use separate contexts.

### Test users

- **`alice_test` / `bob_test` are seeded once at session level** via HTTP POST to `/auth/signup`.
- **`alice` fixture logs in alice_test** via the login form.
- **`bob` fixture logs in bob_test** in a separate Playwright context.
- **Use `alice` for single-user tests.** It returns `(page, user_dict)`.

### Global `site_settings` tests

- **The uvicorn server is shared for the whole session.** A test that flips a global setting (maintenance mode, registration, rate limit, session length) changes behavior for every later test, so **restore the value in a `try/finally`** — see `tests/test_operational.py`.
- **Saving through the admin form is the reliable mutation path**, not a direct DB write: the form handler runs in the server process and calls `clear_settings_cache()`, so the change takes effect immediately (a direct DB write waits out the 60s server-side cache).
- **`conftest.py`'s `app_server` seeds `rate_limit_per_minute="1000000"`** so the suite is never throttled and the settings form round-trips a safe value instead of the template's `"60"` default.
- **Rate-limit unit tests patch `m.get_int_setting`**, not the `RATE_LIMIT` constant — the constant is now only the fallback default passed to `get_int_setting`.

### Failure handling

- **Failure screenshots auto-save** to `/tmp/devplace_test_screenshots/`.
- **Tests stop at first failure** (`-x` flag in Makefile). No cascading failures.
- **If the server won't start, kill leftover processes:** `kill -9 $(pgrep -f "uvicorn")`

### Common pitfalls

| Pitfall | Fix |
|---------|------|
| `goto`/`wait_for_url` times out | Add `wait_until="domcontentloaded"` |
| CDN scripts block page load | Use `defer` on all `<script>` tags |
| 500 on dataset `find()` | Use dict comparison syntax, not raw SQL |
| N+1 query slowness | Use batch helpers: `get_users_by_uids()`, `get_comment_counts_by_post_uids()` |
| Modal not opening/closing | Use `style.display`, not `classList.add/remove` |
| Dual Delete buttons match | Scope to `.comment-action-btn` in tests |
| Dual Post buttons match (feed inline comment) | Scope to `#create-post-modal button.btn-primary:has-text('Post')` in tests |
| Edit modal textarea conflicts with comment textarea | Scope to `.comment-form textarea[name='content']` for comments |
| Tests fail in sequence | Session-scoped context + `clear_cookies()` per test |
| Double messages in chat | Deduplicate by message UID with `seen` set |
| Avatar generation fails | Falls back to initial-based SVG - check multiavatar import |

## Feature Workflow (for automated agents)

### Step 1: Understand

- Read the router file for the feature area (`routers/{area}.py`)
- Read the template (`templates/{area}.html`)
- Read the existing test file (`tests/test_{area}.py`)
- Identify what data flows through: form fields → router → template → response

## Notification System

Notifications are created server-side in the route handlers and stored in the `notifications` table. Unread counts are cached per-process in `_unread_cache` (notifications) and `_messages_cache` (messages), both 10s TTL in `templating.py`. Mutating routes call `clear_unread_cache(uid)` / `clear_messages_cache(uid)` to invalidate.

### Live counter badges

The top-nav bell and Messages link carry `data-counter="notifications"` / `data-counter="messages"` with a child `<span data-counter-badge hidden>`. `CounterManager.js` polls `GET /notifications/counts` (returns `{"notifications", "messages"}`, zeros for guests) every 30s and on tab focus, then sets each badge's text and toggles its `hidden` attribute. Server-rendered counts seed the initial state, so the badges are correct on first paint and self-heal without a reload. Absolute badges use `.nav-badge`; inline badges (next to text links) add `.nav-badge-inline`.

### Notification types and trigger points

| Type | Trigger | Location | Condition |
|------|---------|----------|-----------|
| `comment` | Top-level comment on post | `comments.py` | `post["user_uid"] != user["uid"]` |
| `reply` | Reply to a comment | `comments.py` | `parent["user_uid"] != user["uid"]` |
| `vote` | Upvote on post or comment | `votes.py` | `value == 1` AND voter != owner |
| `follow` | Follow another user | `follow.py` | Always (self-follow blocked upstream) |
| `message` | Send a message | `messages.py` | Always (different user) |
| `level` | Reach a new level | `utils.py` `award_xp` | `new_level > current_level` |
| `badge` | Earn any badge | `utils.py` `notify_badge` | First grant only (`award_badge` returns `True`) |

`level`/`badge` notifications have `related_uid == user_uid` (self), so the notifications template renders them with the recipient's own avatar; the template is type-agnostic (driven by `message`/`actor`), so no per-type handling is needed.

### Time-grouped display

Notifications are grouped by time period in `notifications.py` `_group_label()`:
- Same day → **Today**
- Previous day → **Yesterday**
- Within 7 days → **This week**
- Older → **Older**

The template uses `notification_groups` (list of `{label, entries}` dicts). Jinja2 note: avoid `.items` as a dict key - it clashes with Python's `dict.items()` method.

### Vote notification messages

```python
f"{user['username']} ++'d your post"
f"{user['username']} ++'d your comment"
```

## Gamification (XP, levels, badges, leaderboard)

The progression engine lives in `utils.py` and is wired into the existing content-creation, vote, and follow hooks. Do NOT scatter XP/badge logic — go through these helpers.

### XP and levels (`utils.py`)

- `award_xp(user_uid, amount)` — adds XP (clamped to `>= 0`), recomputes and stores `level`, invalidates the per-process user cache (`clear_user_cache`), and on level-up fires a `level` notification plus any level-milestone badge. Returns `{"xp", "level", "leveled_up"}`.
- `level_for_xp(xp)` → `1 + max(0, xp) // LEVEL_XP` (`LEVEL_XP = 100`). `level` is stored on the user (not derived in the template) so `profile.html`'s `xp % 100` progress bar keeps working.
- XP amounts are constants in `utils.py`: `XP_POST=10`, `XP_COMMENT=2`, `XP_PROJECT=15`, `XP_GIST=5`, `XP_UPVOTE=5`, `XP_FOLLOW=5`. Awards for received upvotes/followers go to the content owner / followed user and live **inside** the existing `owner_uid != user["uid"]` guards (`votes.py`, `follow.py`).

### Badges (`utils.py`)

- `award_badge(user_uid, name)` is idempotent (returns `True` only on first grant) — reuse it; never insert into `badges` directly.
- `check_milestone_badges(user_uid)` recomputes count/star thresholds and awards + notifies any newly crossed badge. Call it after content creation and after an upvote/follow that changes the recipient's totals.
- `award_rewards(user_uid, amount, first_badge=None)` is the single entry point for the create/upvote/follow reward sequence: it awards the optional first-time badge, calls `award_xp`, then `check_milestone_badges`. Use it instead of calling the three separately (posts/projects/gists/comments/follow/votes all go through it) so milestone checks are never skipped.
- Badge catalog + display metadata is `BADGE_CATALOG` in `utils.py`, exposed to templates as the `badge_info(name)` global. Names: Member, First Post, First Comment, First Project, First Gist, Prolific (10 posts), Rising Star (25 stars), Star Author (100 stars), Popular (10 followers), Level 5, Level 10. Unknown names fall back to a default icon and the name as description.

### Rank and leaderboard (`database.py`)

- `_ranked_authors()` builds (and 60s-caches in `_authors_cache`) the full list of authors with positive total stars, ordered desc, enriched via `get_users_by_uids`. `get_top_authors(limit)`, `get_leaderboard(limit, offset)` (adds a 1-based `rank`), and `get_user_rank(user_uid)` all slice/scan this one list — keep them consistent.
- `update_target_stars()` clears `_authors_cache` so a vote is reflected on the leaderboard and profile rank immediately (also keeps integration tests deterministic).
- The leaderboard page (`/leaderboard`, `routers/leaderboard.py`) shows the **top 50 only — no pagination** (`TOP_LIMIT = 50`). It is public (`get_current_user`), highlights the current user's row (`.leaderboard-row-self`), and is listed in `sitemap.xml`.

### Backfill

`_backfill_gamification()` runs at the end of `init_db()`. It computes XP from prior activity (same amounts as above) for users still at the default `xp=0`, sets `level`, then runs `check_milestone_badges` per user. Guarded on `xp=0` so it is idempotent across restarts.

## Inline Comment on Feed Cards

Every post card on the feed now has an inline comment form (`.feed-comment-form`) beneath the post actions. It posts to `/comments/create` like the detail page comment form. Tests must scope Post button clicks to `#create-post-modal button.btn-primary:has-text('Post')` to avoid matching the inline comment's Post button.

## Post Editing

Post owners see an "Edit" button on the post detail page that opens `#edit-post-modal`. The edit form allows changing title, content, and topic. The POST route is `/posts/edit/{post_uid}` with ownership check. The edit modal's textarea has `id="edit-content"` - tests must scope to `.comment-form textarea[name='content']` for comment operations.

## Project Detail Page

Each project card links to `/projects/{project_uid}` showing full project details, author info, platforms, star count, and delete-for-owner. The route is `GET /projects/{project_uid}` in `routers/projects.py`. The sitemap generator links to this URL (not the old `?user_uid=` query param).

## Bug Reports

A dedicated `/bugs` page with create modal for authenticated users. Uses `bug_reports` table (auto-created by `dataset`). Registered in `main.py` with prefix `/bugs`. Footer link in `base.html` under `.site-footer`.

## Gists

A dedicated `/gists` page for sharing code snippets. Uses `gists` table (auto-created by `dataset`).

### Database columns

| Column | Type | Notes |
|--------|------|-------|
| `uid` | text | UUID |
| `user_uid` | text | FK → users.uid |
| `title` | text | Required, max 200 |
| `description` | text | Optional, max 5000, markdown (rendered by ContentRenderer) |
| `source_code` | text | Required, max 50000 |
| `language` | text | One of 27 supported languages |
| `slug` | text | `make_combined_slug(title, uid)` |
| `stars` | int | Net vote count (via `/votes/gist/{uid}`) |
| `created_at` | text | ISO datetime |

### Routes

| Method | Path | Handler | Auth |
|--------|------|---------|------|
| GET | `/gists` | `gists_page` | No |
| GET | `/gists/{slug}` | `gist_detail` | No |
| POST | `/gists/create` | `create_gist` | Yes |
| POST | `/gists/delete/{slug}` | `delete_gist` | Yes (owner) |

### Polymorphic reuse

- **Comments**: Uses `_comment_section.html` with `target_type="gist"` - same component as posts/projects
- **Voting**: Uses existing `/votes/gist/{uid}` route - updates `gists.stars`
- **Content rendering**: Description rendered via `ContentRenderer.js` (.rendered-content[data-render])
- **Profile tab**: "Gists" tab between Projects and Activity on profile pages

### CodeMirror editor

- CodeMirror 5 loaded from CDN in `gists.html` via `{% block extra_js %}`
- 22 language modes pre-loaded (Python, JS, TS, HTML, CSS, C, C++, Java, Go, Rust, SQL, Bash, YAML, Markdown, Swift, PHP, Ruby, Kotlin, Haskell, Lua, Perl, R, Dart, Scala)
- `GistEditor.js` initializes CodeMirror on `#gist-source-editor` textarea
- Language selector dropdown dynamically switches CodeMirror mode
- `Ctrl+S` shortcut saves and submits the form
- On form submit, `editor.save()` syncs CodeMirror content back to the hidden textarea

### Display

- Source code rendered in `<pre><code class="language-xxx">` block on detail page
- Syntax highlighting handled by existing `highlight.js` loaded globally in `base.html`
- Copy button uses `navigator.clipboard.writeText()`
- Cards in listing show language badge, title, truncated description, author, star count

### Sitemap

- Latest 500 gists included in sitemap, `changefreq="weekly"`, `priority="0.6"`

## Background Services

The `devplacepy/services/` package provides a generic framework for running background async services alongside the FastAPI server. Architecture:

### `BaseService` (`services/base.py`)
Abstract class for all services:
- **`name`** - unique identifier (used in routing, logs, and DB)
- **`interval_seconds`** - run interval (3600 for news), runs immediately on boot then every interval
- **`log_buffer`** - `deque(maxlen=20)` for log tail (served via `/services` page + auto-refresh)
- **`log(message)`** - writes to both the buffer and standard `logging`
- **`run_once()`** - abstract; override with actual work
- **`start()`** / **`stop()`** - asyncio task lifecycle with graceful cancellation (10s timeout)

### `ServiceManager` (`services/manager.py`)
Singleton that manages all registered services:
- `register(service)` - add a service
- `start_all()` - start all registered services
- `stop_all()` - cancel all tasks (called on server shutdown)
- `list_services()` → `list[dict]` with name, status, uptime, log buffer

### `NewsService` (`services/news.py`)
Implements `BaseService`:
- Fetches `GET {news_api_url}` → `{"articles": [...]}`
- Every article is graded via AI: `POST {news_ai_url}` with model `{news_ai_model}` (no auth needed)
- ALL articles are inserted into `news` table regardless of grade (never silently skipped)
- Each article gets a `status` field: `"published"` if grade >= threshold, `"draft"` otherwise
- Threshold configurable in admin settings
- Articles re-synced each run (upsert by `external_id`) - grade, status, images updated on every cycle
- Slugs generated via `make_combined_slug(title, uid)` - same format as posts/projects

### Database tables

| Table | Purpose |
|-------|---------|
| `news` | All synced articles with `status` (published/draft), `grade`, `slug`, `show_on_landing` |
| `news_images` | Images extracted from article URLs |
| `news_sync` | Sync state per article `guid` - tracks grading history |

### Site settings (seeded on startup)

| Key | Default | Purpose |
|-----|---------|---------|
| `site_name` / `site_description` / `site_tagline` | DevPlace branding | General site metadata |
| `news_grade_threshold` | `"7"` | Minimum AI grade for auto-publish |
| `news_api_url` | `"https://news.app.molodetz.nl/api"` | News source API |
| `news_ai_url` | `"https://openai.app.molodetz.nl/v1/chat/completions"` | AI grading endpoint |
| `news_ai_model` | `"molodetz"` | AI model identifier |
| `max_upload_size_mb` / `allowed_file_types` / `max_attachments_per_resource` | `"10"` / `""` / `"10"` | Upload limits |
| `rate_limit_per_minute` | `"60"` | Mutating requests per IP per window (`main.py` middleware) |
| `rate_limit_window_seconds` | `"60"` | Rolling window for the rate limit |
| `news_service_interval` | `"3600"` | Seconds between news fetch cycles (`NewsService.run_once` re-reads each cycle) |
| `session_max_age_days` | `"7"` | Standard session cookie + DB session lifetime |
| `session_remember_days` | `"30"` | Remember-me session lifetime |
| `registration_open` | `"1"` | When `"0"`, signup GET shows a closed notice and POST is rejected (`auth.py`) |
| `maintenance_mode` | `"0"` | When `"1"`, non-admins get a 503 (`main.py` maintenance middleware) |
| `maintenance_message` | scheduled-maintenance text | Body shown on the maintenance 503 page |

The seed block in `database.py` is guarded by `if "site_settings" in tables:` — on a brand-new DB the table does not exist yet (dataset creates tables lazily on first insert), so none of these rows are written until the table exists. Correct runtime behavior therefore relies on every consumer passing the production default to `get_setting`/`get_int_setting`, not on the seed.

### Operational settings — read sites and rules

| Setting(s) | Read at | Notes |
|-----------|---------|-------|
| `rate_limit_*` | `rate_limit_middleware` in `main.py` | `max(1, get_int_setting(...))` so `0` can't block all writes |
| `maintenance_mode` / `maintenance_message` | `maintenance_middleware` in `main.py` | Allows `/static`, `/avatar`, `/auth`, `/admin` and admins; everyone else gets `error.html` at 503 |
| `news_service_interval` | top of `NewsService.run_once` | `max(60, …)`; `BaseService._run_loop` reads `self.interval_seconds` for its sleep after each cycle, so a change applies next cycle |
| `session_max_age_days` / `session_remember_days` | `auth.py` signup + login | Multiplied by `SECONDS_PER_DAY`; passed to `create_session(uid, max_age)` so the cookie and the DB session row expire together |
| `registration_open` | `auth.py` `signup_page` (GET) and `signup` (POST) | POST returns before any DB write when closed |

**Booleans are `<select>`, never checkboxes.** The settings save handler (`admin.py`) skips empty form values so empty fields don't clobber existing rows. An unchecked checkbox submits nothing, so it could never be turned off — `registration_open` and `maintenance_mode` use `<option value="1">`/`<option value="0">` so a value is always submitted.

### Adding a new service

1. Create `devplacepy/services/your_service.py` with a class extending `BaseService`
2. Override `async def run_once(self) -> None`
3. Register in `main.py` startup event:
   ```python
   from devplacepy.services.your_service import YourService
   service_manager.register(YourService())
   ```
4. The service appears automatically on `/services` with log tail

### CLI

```bash
devplace news clear   # Delete all news from local database
```

## Signals Category

The `signals` topic is available as a feed filter sidebar item and post topic. Added CSS variable `--topic-signals: #00bcd4` in `variables.css`, badge class in `base.css`, and dot color in `feed.css`. Allowed in `posts.py` topic validation.

## Date Format

All dates displayed to users use European DD/MM/YYYY format. Implemented via:

- **`format_date(dt_str, include_time=False)`** in `utils.py` - converts ISO datetime → `DD/MM/YYYY` or `DD/MM/YYYY HH:MM`
- Registered as template global in `templating.py`: `{{ format_date(dt) }}`
- **`time_ago()`** returns `DD/MM/YYYY` for items older than 30 days (instead of `"Xmo ago"`)
- Services page has a JS `formatDate()` function for live polling updates

## Admin Pagination

Both `/admin/users` and `/admin/news` use offset-based pagination via a reusable component:

- **`templates/_pagination.html`** - numbered page links with ellipsis, Previous/Next buttons, total count
- Routes accept `?page=N` query param, clamped to valid range
- `per_page = 25`, pagination metadata computed server-side and passed as `pagination` dict
- Only renders when `total_pages > 1`
- CSS in `admin.css` (`.pagination`, `.pagination-btn`, `.pagination-page`, `.pagination-ellipsis`)

## News Detail & Comments

News articles have an internal detail page at `/news/{slug}` with full comment support:

- **Route:** `GET /news/{news_slug}` in `routers/news.py` - resolves by slug first, then UUID
- **Template:** `templates/news_detail.html` - shows image, source, grade, description, content, external link
- **Comments:** Uses `_comment_section.html` with `target_type="news"` - same component as posts/projects
- **`resolve_target_redirect()`** in `comments.py` handles `"news"` → `/news/{slug}`
- Listing links in `news.html` point to internal detail page; "Read on Source" still goes to external URL

## Landing Page News

Articles can be toggled to appear on the landing page via `/admin/news/{uid}/landing`:

- **`show_on_landing`** field on `news` table
- Landing route (`main.py` `GET /`) fetches up to 6 articles with `show_on_landing=1`
- Rendered as a 3-column card grid with image, source, title, date (responsive → 1 column on mobile)
- Toggleable individually from the admin news table

## Public Feed

The feed page (`GET /feed`) is accessible without authentication:

- Uses `get_current_user(request)` instead of `require_user()` - returns `None` for guests
- Guests see posts but not the FAB, create modal, inline comment forms, or following tab
- All POST routes (create, comment, vote) remain guarded by `require_user()`
- Topnav shows Login/Sign Up for unauthenticated visitors; Messages, Admin, notifications for authenticated

### Step 2: Implement backend

- Add/modify the route in `routers/{area}.py`
- Validate form input with a typed `Annotated[Model, Form()]` param (define the model in `models.py`); read raw `await request.form()` only for file uploads
- Use `templates.TemplateResponse(...)` from `devplacepy.templating`
- Redirect with `RedirectResponse(url=..., status_code=302)`
- Log every action: `logger.info(...)`
- New DB fields auto-sync via `dataset` - just add to the insert/update dict
- For ownership checks: `if resource["user_uid"] == user["uid"]`
- For deletion: cascade related data first (comments → votes → post)
- Register new routers in `main.py`: `app.include_router(router, prefix="/{path}")`

### Step 3: Implement frontend

- Template in `templates/`, CSS in `static/css/`, JS in `static/js/Application.js`
- Load CSS via `{% block extra_head %}` with `<link rel="stylesheet">`
- Use `{% extends "base.html" %}` and `{% block content %}`
- Jinja2 globals: `avatar_url()`, `get_unread_count()`, `get_user_projects()`
- For clickable avatars: `{% set _user = ... %}{% include "_avatar_link.html" %}`
- For rendered content: add `class="rendered-content"` and `data-render` attribute
- No NPM, no frameworks - pure ES6 modules

### Step 4: Validate code

```bash
hawk .
```

Zero errors required.

### Step 5: Run existing tests (only if asked by user)

```bash
make test
```

All tests must pass. Tests stop at first failure (`-x`).

### Step 6: Write new tests

- Add tests in `tests/test_{area}.py`
- Use `alice` for authenticated sessions, `bob` for multi-user
- Use `page`, `app_server` for unauthenticated page checks
- Assert on visible text/content, not internal state
- Use `wait_until="domcontentloaded"` on all `goto()` and `wait_for_url()` calls
- Test both success paths and error/validation paths
- For delete buttons, scope to the specific element type (e.g., `.comment-action-btn`)

### Step 7: Run full suite again (only if asked by user)

```bash
make test
make test-headed   # visual confirmation
```

### Step 8: Document

- Update `AGENTS.md` if new conventions introduced
- Update `README.md` if new routes, config, or dependencies added

## Autonomous Agentic CI Workflow

```python
while feature_not_complete:
    1. Plan: read existing code, design the change
    2. Implement: write code (router → template → CSS → JS)
    3. hawk .          # must pass
    4. falcon take + describe   # visual check for UI changes
    5. If visual fail: fix CSS/template → goto 3
    6. Update AGENTS.md if needed
```

Failures at any step block the workflow. Never skip a failed step.

## CI/CD

Gitea Actions workflow at `.gitea/workflows/test.yaml` runs on every push/PR to `main`. It installs dependencies, validates with `hawk`, runs all tests, and uploads failure screenshots. The CI must be green before merging.

## SEO Implementation

All SEO features are implemented across the following locations:

### Core SEO utilities
- `devplacepy/seo.py` - JSON-LD schema generators (WebSite, BreadcrumbList, DiscussionForumPosting, ProfilePage, SoftwareApplication), meta description truncation, schema combiner, sitemap XML generator
- `routers/seo.py` - robots.txt and sitemap.xml routes

### SEO template context
- Every router passes `page_title`, `meta_description`, `meta_robots`, `canonical_url`, `og_title`, `og_description`, `og_image`, `og_type`, `breadcrumbs`, `page_schema` via `base_seo_context()`
- Auth pages: `noindex,nofollow`
- Messages/Notifications: `noindex,nofollow`
- Profiles with < 2 posts: `noindex,follow`
- All other pages: `index,follow`

### Template layer
- `templates/base.html` - dynamic `<title>`, `<meta description>`, `<link canonical>`, `<meta robots>`, Open Graph, Twitter Cards, JSON-LD injection, breadcrumb nav, CDN `dns-prefetch`/`preconnect`
- `static/css/base.css` - `.breadcrumb` (aria-label breadcrumb nav), `.sr-only` (accessible hidden headings)

### Heading hierarchy
- `feed.html` - `<h1 class="sr-only">Feed</h1>`
- `profile.html` - username rendered as `<h1 class="profile-name">`
- `messages.html` - `<h1 class="sr-only">Messages</h1>`
- `projects.html` - `<h1>Projects</h1>`
- `post.html` - post title as `<h1>`, "Related Discussions" as `<h3>`

### Post slugs
- Slug generated on post creation via `slugify()` and stored in `posts.slug` column
- Posts can be looked up by slug or UUID
- Minimum content validation: post body >= 10 chars, comment >= 3 chars

### Related posts
- `templates/post.html` - "Related Discussions" widget at bottom of post page (queried by matching topic)

### Performance
- `loading="lazy"` on all avatar images
- `dns-prefetch` + `preconnect` for CDN resources in `<head>`
- Security headers middleware: `X-Robots-Tag`, `X-Content-Type-Options`

### Default OG image
- `static/og-default.svg` - 1200x630 SVG with DevPlace branding
- Used as fallback `og:image` on all pages

### SEO tests
- `tests/test_seo.py` - 13 tests covering: robots.txt, sitemap.xml, page titles, noindex, canonical URLs, OG tags, Twitter cards, structured data, security headers

## Engagement: reactions, bookmarks, polls, contribution heatmap

### Emoji reactions
- Curated palette only: `REACTION_EMOJI` in `constants.py` (registered as a template global). `ReactionForm` rejects anything outside it; free-text emoji are not allowed.
- Endpoint `POST /reactions/{target_type}/{target_uid}` (`routers/reactions.py`) toggles one `(user, target, emoji)` row in the `reactions` table. Target types: `post`, `comment`, `gist`, `project`. AJAX (`x-requested-with: fetch`) returns `{counts, mine}`.
- Reactions are **non-ranking** — they never touch `stars` or XP and intentionally send **no notifications** (votes already notify; reactions would be notification spam).
- Batch reads via `get_reactions_by_targets(target_type, uids, user)` in `database.py` (used by feed, profile, comment loader, `load_detail`) — never per-row. The `_reaction_bar.html` partial takes `_type`, `_uid`, `_reactions` ({counts, mine}) and renders the full palette as toggle chips; `ReactionBar.js` uses document-level click delegation.

### Bookmarks
- `POST /bookmarks/{target_type}/{target_uid}` toggles a `bookmarks` row; `GET /bookmarks/saved` renders the personal list (`saved.html`). Target types: `post`, `gist`, `project`, `news`.
- `_bookmark_button.html` takes `_type`, `_uid`, `_bookmarked`; `BookmarkManager.js` swaps the label/`bookmarked` class from the JSON `{saved}`. Batch state via `get_user_bookmarks(user_uid, target_type, uids)`.

### Polls
- A poll rides on a post (one `polls` row keyed by `post_uid`, options in `poll_options`, one-per-user votes in `poll_votes`). Created in `posts.py:create_poll` when `PostForm.poll_question` plus ≥2 non-empty `poll_options` are submitted (capped at 6). The create-post modal in `feed.html` has the builder (`data-poll-toggle` / `data-poll-add-option`).
- `POST /polls/{poll_uid}/vote` toggles the voter's choice and returns the full poll dict: voting on a new option switches, voting the **same** option again **retracts** (one vote per user, like reactions). Results (bars + `%`) are **always shown** to everyone; the chosen option gets `.chosen` + a `✓`. Guests see read-only bars with a "Log in to vote" hint (options `disabled`). Batch reads via `get_polls_by_post_uids(post_uids, user)` / `get_poll_for_post(post_uid, user)`.
- Composer poll builder (`feed.html`, `PollManager.js`): poll inputs are `disabled` until "Add poll" is opened (so a never-opened or "Remove poll"-collapsed builder submits **nothing**); a capture-phase `submit` listener blocks an incomplete poll (question + ≥2 options) with an inline error instead of silently dropping it — capture phase + `stopImmediatePropagation()` is required so it pre-empts `FormManager`'s bubble-phase submit handlers.

### Cascade
- `delete_engagement(target_type, uids)` in `database.py` removes reactions, bookmarks and (for posts) poll rows. It is called from `content.py:delete_content_item` (for the item and its comments) and `comments.py:delete_comment`. Add it to any new delete path.

### Contribution heatmap + streaks
- Derived, **no new table**: `get_activity_calendar(user_uid)` aggregates `date(created_at)` across `posts`/`comments`/`gists`/`projects` (cached 120s); `get_activity_heatmap` returns 53 Monday-aligned weeks of `{date, count, level}` (level 0–4); `get_streaks` returns `{current, longest}`. Rendered server-side in `profile.html` (`.heatmap-grid`, styles in `profile.css`). The `On Fire` badge is awarded in `check_milestone_badges` when the current streak ≥ 7.

### CSS
- Reactions, bookmarks, polls and the saved page live in `static/css/engagement.css`, loaded globally in `base.html` (the controls appear across feed, detail pages and comments). Heatmap styles are in `profile.css`.