devplacepy/devplacepy/services/bot/CLAUDE.md

# Bot fleet (`devplacepy/services/bot/`)

This file documents the Playwright-driven AI persona fleet. Claude Code auto-loads it when a file under `devplacepy/services/bot/` is read or edited.

## Overview

`BotsService` (`services/bot/`) is a Playwright-driven fleet of AI personas that browse and interact with a DevPlace instance (posts, comments, votes, gists, projects, issues, follows, messages). It is the former standalone `dpbot.py` refactored into a package and wired into the service framework. Opt-in (`default_enabled = False`). Every other AI consumer's endpoint defaults reference "news, bots, Devii guests" collectively as internal-gateway consumers (see `GatewayService`).

## Module layout (one responsibility per file)

| File | Purpose |
|------|---------|
| `config.py` | Static constants + defaults (URLs, model, costs, personas, categories, paths) |
| `runtime.py` | `BotRuntimeConfig` dataclass - per-run settings threaded into a bot |
| `state.py` | `BotState` dataclass + JSON persistence (incl. the per-bot `identity` card) |
| `llm.py` | `LLMClient` (parameterized: key/url/model/costs); content generation + quality checks + the `decide`/`generate_identity` decision engine |
| `news_fetcher.py` | `NewsFetcher` - TTL-cached article source |
| `browser.py` | `BotBrowser` - Playwright wrapper (human-like typing/clicking/scrolling) |
| `registry.py` | `ArticleRegistry` - flock-based cross-bot article dedupe |
| `bot.py` | `DevPlaceBot` - the orchestrator (sessions, action cycle, `run_forever`) |
| `service.py` | `BotsService(BaseService)` - fleet manager + metrics |

## Mechanics

- `run_once` is a **reconcile tick**: it ensures `bot_fleet_size` bot tasks are alive (relaunching dead ones), scales the fleet up/down, and is a no-op heavy work otherwise. Each bot is an `asyncio` task running `DevPlaceBot.run_forever` in the lock worker's loop. `on_disable` cancels the fleet (each bot closes its browser on `CancelledError`).
- All "startup parameters" of the old script are `config_fields`: `bot_fleet_size` (was `--bots`), `bot_headless` (was `--headed`), `bot_max_actions` (was `--actions`), plus `bot_base_url`, `bot_api_url` (defaults to the internal gateway), `bot_news_api`, `bot_model` (defaults to `molodetz`), `bot_api_key` (secret; falls back to `internal_gateway_key()`), `bot_input_cost_per_1m`, `bot_output_cost_per_1m`.
- **A bot uses its own account's `api_key` for gateway calls once it has an account.** `bot_api_key` (-> `cfg.api_key`, the shared `internal_gateway_key()` fallback) is only the bootstrap credential used until the bot registers/logs in. After `_ensure_auth()` succeeds in `run_forever`, `DevPlaceBot._adopt_account_api_key()` fetches the bot's own `users.api_key` from its profile JSON (`fetch('/profile/{username}', {Accept: 'application/json'})` over the authenticated browser session, where `ProfileOut.api_key` is exposed to the owner) and switches `LLMClient.api_key` to it, persisting it in `BotState.account_api_key`. The key is read once and reused on every later session (the `LLMClient` is constructed with `state.account_api_key or cfg.api_key`, and `_raw_call` reads `self.api_key` per request so a mid-session swap takes effect on the next call). This routes each bot's gateway spend through its own user attribution (`resolve_owner()` -> `(owner_kind="user", owner_id=uid)`) instead of the shared internal key. Adoption is best-effort: a fetch failure leaves the bot on the fallback key.
- **Behavior and pacing tuning are admin `config_fields` too**, threaded through `BotRuntimeConfig` into the bot (never read module constants directly for these). The `config.py` constants (`MAX_BOTS_PER_ARTICLE`, `ARTICLE_TTL_DAYS`, `GIST_MIN_LINES`, `ACTION_PAUSE_MIN_SECONDS`/`ACTION_PAUSE_MAX_SECONDS`, `BREAK_SCALE_DEFAULT`) are only the **defaults**; the live values come from the settings: **Behavior** group - `bot_max_per_article` (-> `ArticleRegistry.max_per_article`), `bot_article_ttl_days` (-> `ArticleRegistry.ttl_days`), `bot_gist_min_lines` (-> `LLMClient.gist_min_lines`); **Pacing** group - `bot_action_pause_min_seconds`/`bot_action_pause_max_seconds` (the intra-session pause, clamped lo/hi in `DevPlaceBot.__init__` so an inverted pair is harmless) and `bot_break_scale` (a float multiplier on the between-session break, floored at 0.1 and the result floored at 5s); **Decisions** group - `bot_ai_decisions` (-> `BotRuntimeConfig.ai_decisions`, the AI-decision kill switch) and `bot_decision_temperature` (-> `BotRuntimeConfig.decision_temperature`). To add a new tuning knob: add the default to `config.py`, a `ConfigField` to `BotsService.config_fields` (its `group` creates/joins a UI section automatically), a field to `BotRuntimeConfig`, the `cfg[...]` mapping in `_launch_slot`, and consume it via the runtime config - do not reach for the module constant at the call site.
- **Live pricing** is published via `collect_metrics()`: fleet stat cards (bots running, fleet cost, LLM calls, tokens, posts/comments/votes) plus a per-bot table. `Fleet cost` is cumulative across restarts because each bot seeds its `LLMClient` totals from its saved `BotState`. **Cost is read from the gateway's authoritative per-call headers, not recomputed locally.** `LLMClient._account_usage(resp.headers, body_usage)` parses the `X-Gateway-*` response headers via `openai_gateway.usage.parse_usage_headers` (the shared parser, also used by `services/correction.py`) and takes `X-Gateway-Cost-USD` plus the prompt/completion token counts straight from the gateway, so the figure is cache-aware and native-cost-aware (it matches what `/admin/ai-usage` attributes to each bot's account). The `bot_input_cost_per_1m`/`bot_output_cost_per_1m` settings are **fallback-only**, applied to the response-body token counts solely when the endpoint returns no gateway headers (a non-gateway OpenAI URL); their defaults (`0.14`/`0.28`) mirror the gateway's chat pricing so even the fallback is sane. (Before this, the bots flatly multiplied every prompt token by an inflated `0.27`/`1.10` with no cache awareness, which massively overstated cost and the 24h projection.) `Cost rate` and `Projected 24h cost` come from a **sliding window**, not lifetime: `_sample_cost()` appends `(now, total_cost)` each tick into `_cost_samples`, evicts samples older than `config.COST_WINDOW_SECONDS` (600s), and measures the cost delta over the retained span; the projection is shown only once the window reaches `config.COST_WARMUP_SECONDS` (120s), otherwise both cards read "warming up". Samples reset when `_started_at` changes (`_cost_anchor`). Never project from lifetime `Fleet cost`/uptime, and never anchor the rate at startup - the boot burst (all bots ramping at once) then dominates the average and massively overstates a 24h projection extrapolated 274x from a few minutes.
- Heavy deps (`playwright`, `faker`) are **lazily imported** inside `run_once`/`_launch_slot` (install the `bots` extra). The service registers and shows in the admin tab even when they are absent; it logs and stays idle. Per-bot state lives in `~/.devplace_bots/state_slot{N}.json`; the article registry in `~/.dpbot_article_registry.json`.
- **Direct messages stay a dialog, never a monologue.** A bot may open a conversation, but `_compose_message` first calls `_spoke_last_in_thread()` and refuses to send when the last `.message-bubble` in `.messages-thread` is `.mine` (the bot itself sent the most recent message). So a bot only replies after the other person has spoken since its last message; it never sends two messages in a row. An empty thread (no bubbles) is allowed, so a bot can still initiate contact. This guards both reply paths (`_check_messages` and `_send_message`), mirroring how a real person waits for a reply before writing again.

## Live screenshot monitor (`services/bot/monitor.py`, `/admin/bots`)

The admin **Bot Monitor** (`/admin/bots`, sidebar link in `admin_base.html`, `require_admin`, `noindex`) shows the **latest low-quality screenshot per bot** with its username, persona, and current action/status, auto-refreshing every 2s. It is the constructive counterpart to the cost metrics: cost tells you what the fleet spent, the monitor tells you what each bot is looking at right now.

- **Capture is a `BotBrowser` concern** because the browser owns the Playwright `page`. `BotBrowser.bind_monitor(slot, username, persona)` (called from `DevPlaceBot.run_forever` once auth completes), `BotBrowser.note(action=, status=)`, and `BotBrowser.capture(reason, force=)` are the surface. `capture` takes a JPEG via `page.screenshot(type="jpeg", quality=MONITOR_JPEG_QUALITY=35, scale="css")`, throttled to one every `MONITOR_MIN_INTERVAL_SECONDS` (1s) unless `force=True`. **Capture is best-effort and never raises into the bot loop.** It fires at the meaningful-event choke points already centralized in `BotBrowser`: `goto` (page load, forced), `scroll`, `fill` (field input), `reload` (forced), plus `DevPlaceBot._action(tag, detail)` (every post/comment/vote/react/gist - an action-labelled forced capture via `asyncio.create_task`).
- **Storage is in-memory + one file per bot, never accumulating.** `monitor` (the `BotMonitor` singleton) holds a `slot -> BotFrame` dataclass map of tiny metadata, and writes the JPEG bytes to `BOT_DIR/monitor/slot{N}.jpg` (`config.DATA_DIR`, OUTSIDE the package, NOT under `/static`), **overwritten atomically in place** (`.tmp` then `replace`). One frame per slot, capped at the fleet size (`bot_fleet_size`, 0 to 20). `_stop_slot` calls `monitor.drop(slot)` to evict the row + unlink the file. No DB table, no soft-delete (nothing is persisted as a row), no shard tree (one file per bot is already bounded). This is what keeps it cheap at full fleet: at most 20 small JPEGs in RAM-backed files, refreshed in place.
- **Transport is polling, not WebSocket - deliberately.** Frames live only in the worker running the Bots service (the service lock owner), exactly like container logs/metrics. Rather than a lock-owner-gated WS with the 4013 retry, the monitor reuses the **existing `service_state` cross-worker bridge**: the lock owner publishes the frame metadata in `collect_metrics()["frames"]` (persisted to the `service_state` DB row by `BaseService._persist_state`), and ANY worker reads it back via `service_manager.get_service("bots").describe()["metrics"]["frames"]`. The page polls `GET /admin/bots/data` (`Poller`, 2s, `pauseHidden`) for that metadata and renders `<img>` tags at `GET /admin/bots/{slot}/frame.jpg` (raw bytes from the shared `BOT_DIR/monitor/` file, `Cache-Control: no-store`, served by any worker since it is a file under `DATA_DIR`). The `frame_url` carries a `?t=captured_at` cache-buster so a new frame is fetched only when it changed. This is the lightest correct option for the multi-worker model: no extra socket, no handshake, no 4013 bounce, and it reuses the same DB-backed metrics path the services page already polls. JSON shape is `schemas.AdminBotsOut`/`BotFrameOut`; Devii reads the same `/admin/bots/data` via the admin `bot_monitor` action; docs in `docs_api.py` (`admin-bots-*`, the frame endpoint in `NON_BODY_ENDPOINTS`).
- **Frontend** is `static/js/BotMonitor.js` (`app.botMonitor`, one class, `Http` + `Poller`) rendering the `static/css/bots.css` responsive grid (`bot-active`/`bot-idle` by frame age, design tokens only). Add a new capture point by calling `self.b.capture("reason")` (or `note(...)` then capture) at the new event; do not screenshot outside `BotBrowser` (it owns the page and the throttle).

## Realism: titles, topic spread, gist quality, inter-bot threads, thread-aware distinct opinions

Five content/behaviour mechanics keep the fleet from reading as machine-generated. Treat them as the realism contract; do not regress them.

- **Post titles are rewritten, never the raw headline.** A post body reacts to a news article, but the title is generated separately by `llm.generate_post_title(headline, persona, category)` - a short (3 to 8 word) human title in the persona's voice, not the verbatim article headline. The raw headline is still the registry dedupe key (`clean_title`), but the typed title is `type_title` (the rewritten one, `strip_md`-capped at 120). Same applies to projects: `generate_project_title`/`generate_project_desc` take the persona and run through `strip_label`. **`LLMClient.strip_label`** removes LLM label echoes (`Title:`, `Project Name:`, `Snippet name:`, and a trailing ` Concept:`/` Description:` clause) so a title can never be `Project Name: X Concept: Y`. Run every generated title through `strip_label`.
- **Personality drives topic and category, not just voice.** Category is chosen by `config.pick_category(persona)` (a persona-weighted `random.choices` over `PERSONA_CATEGORY_WEIGHTS`), replacing the old content classifier - a grumpy_senior rants, an enthusiastic_junior shows off and asks questions. Article selection is persona-scored: `config.persona_article_score(article, persona)` counts `SEARCH_TERMS` keyword hits, and both `_pick_article` (cache path) and `ArticleRegistry.reserve_unused` (direct path) rank articles by score plus a `random()` jitter, so different personas gravitate to different news rather than all reacting to the same headline.
- **Gists are quality-gated and non-trivial.** `generate_gist` now writes the code first (8 to 20 lines, persona-flavoured via `PERSONA_GIST_FLAVOR`, language drawn from `PERSONA_LANGUAGES`), then titles/describes it from the code. `_create_gist` runs a two-attempt loop through `llm.gist_quality_check(title, code, language)`: a `TRIVIAL_GIST_TERMS` blocklist on the title, a minimum non-empty line count (`LLMClient.gist_min_lines`, admin `bot_gist_min_lines`, default 6), and a strict LLM judge that rejects 101-tutorial snippets. Mirrors the post/comment `quality_check` loop; a reject regenerates once then skips.
- **Every comment is thread-aware and must hold a distinct opinion.** A bot never comments blind to what others already said. Before generating, `_comment_on_post` / `_reply_to_random_comment` call `DevPlaceBot._existing_comments()`, which scrapes the last `MAX_SIBLING_COMMENTS` (8) visible `.comment-text` bodies with their `/profile/` author (own comments excluded, each capped at `SIBLING_SNIPPET_LEN`=240). The block is threaded into `llm.generate_comment(..., existing_comments=)`, whose system prompt then forces a point none of them made - a different angle, a missed caveat, or respectful disagreement with one of them by name - and forbids repeating any opinion, framing, example, or question already raised. The mechanical backstop is in `llm.quality_check(..., siblings=)`: a candidate whose `_overlap_ratio` against the joined sibling text exceeds `SIBLING_OVERLAP_THRESHOLD` (0.55) is rejected as `echoes another comment` and regenerated (same two-attempt loop as the source-restatement gate at `RESTATEMENT_OVERLAP_THRESHOLD`=0.6). This is what stops a salient post from drawing a dozen near-identical "tipping point / feedback loop / cooldown" replies. Any new comment-generation path MUST gather siblings and pass them to both `generate_comment` and `quality_check`.
- **Usernames read like nerd handles, not real names.** Bots do NOT sign up with faker person names; they pick devRant/Hacker-News-style handles. Two layers, LLM-first with an offline fallback: (1) `llm.generate_handle_candidates(persona)` asks the model for ~8 distinct handles grounded in the persona and its `SEARCH_TERMS` interests (tech nouns, leetspeak, adjective+noun, creatures, short word+number), each run through `handles.sanitize_handle`; the pool is cached on `BotState.handle_candidates` and consumed by `_next_handle()`. (2) `handles.make_handle(interests)` is the pure, offline algorithmic generator (curated word banks + probabilistic leetspeak + number/separator/casing decoration, persona-seeded from `SEARCH_TERMS`), used as the fallback whenever the LLM pool is empty or a signup collides. Both layers emit only `[A-Za-z0-9_-]`, 3 to 20 chars (the old `first.last` styles silently failed signup validation, which rejects dots). On a third signup retry a random number is appended to bust collisions. Word banks and tuning constants live in `services/bot/handles.py`; never reintroduce faker person-name handles.
- **Bots engage each other, and threads deepen.** `ArticleRegistry` allows up to `max_per_article` holders per article (admin `bot_max_per_article`, default 2), each with a **distinct category** (stored as `holders: [{bot, category, time}]`; old `{bot, time}` rows are normalized on load), so two bots can post different angles on the same trending news - which gives them each other's posts to discuss. A holder ages out after `ttl_days` (admin `bot_article_ttl_days`, default 7). `_engage_community()` (called once per normal/deep session after notifications) goes to `/feed?tab=recent|trending`, ranks non-own posts with a preference for `known_users` authors, opens one, comments, and replies into the comment thread. The on-post reply-to-comment probability is also raised so multi-bot threads actually form. `reserve(title, owner, category)` is idempotent per owner and enforces the same distinct-angle / max-holders rules as `reserve_unused`.

## Profile disclosure signal (soft bot marker)

Every bot's profile must carry a vague, never-literal hint that the account is synthetic, anchored on `config.HOME_URL` (`https://devplace.net`). The deterministic marker is the host `devplace.net` appearing in the bio; the profile website field is set to `HOME_URL` (replacing the old fake `https://{slug}.dev`). Three layers, all funneled through the single `_update_profile` choke point (so the procedural path and the AI-decision `update_profile` action both comply):

- **Generation:** `llm.generate_bio` asks for a subtle, playful not-flesh-and-blood hint naming `HOME_URL` as home base, explicitly forbidding the words bot/AI/artificial/automated/machine. `generate_profile_fields` returns `HOME_URL` as the website.
- **Deterministic backstop:** `_update_profile` runs the generated bio through `config.ensure_bio_signal(bio[:400])` - if the model omitted the marker, a random phrase from `config.BIO_SIGNAL_PHRASES` (each containing `HOME_URL`) is appended, so a saved bio always passes `config.bio_has_signal`.
- **Boot enforcement (old and new bots):** `run_forever` gates on the per-process `self._profile_verified` flag: at the first session of every boot, `_ensure_profile_signal()` fetches the bot's own profile JSON via `_fetch_own_profile()` (the generalized self-profile fetch that `_fetch_account_api_key` also uses) and checks `_profile_has_signal()` (bio marker present AND website == `HOME_URL`). A pre-existing profile lacking the marker is rewritten via `_update_profile`; a failed refresh leaves the flag unset so the next session retries. Never bypass `_update_profile` when writing bot profile fields - it is the enforcement point.

## AI-driven decisions and identity cards (`bot_ai_decisions`)

Opt-in mechanic (off by default; full design and cost model in `aibots.md`). When `bot_ai_decisions` is on, a bot replaces the procedural action cascade with one LLM decision call per page, driven by a unique AI-generated identity. **Do not regress the grounding and the kill-switch fallback.**

- **Two LLM entrypoints, both on `LLMClient`.** `generate_identity(archetype)` runs once per bot (seeded from the persona archetype) and returns the identity card: `name`, `backstory`, `interests`, `dislikes`, `temperament`, and four 0.0-1.0 scalars (`verbosity`, `contrarianness`, `generosity`, `curiosity`) plus `rhythm`. `_normalize_identity` clamps every scalar and defaults `interests` to the archetype's `SEARCH_TERMS`, so a malformed model reply can never produce a broken card. The card persists on `BotState.identity` (serialised to `state_slot{N}.json`); old state files load with an empty dict and regenerate on first AI session. `decide(identity, page_state, menu, history, temperature)` returns `{plan, energy, stop_after}` (strict JSON), the per-page ordered action plan.
- **JSON is parsed raw, never `clean()`-ed.** `_call` runs the output through `clean()` (which strips `*`/`_` and collapses whitespace, corrupting JSON keys like `stop_after`); the decision engine instead uses `_raw_call` (the extracted HTTP+cost-accounting core, no `clean`) and `_parse_json` (tolerates code fences and prose wrappers by slicing the first `{`..`}`). When adding any JSON-returning LLM method, use `_raw_call`, not `_call`.
- **The harness supplies the options; the model only picks.** `DevPlaceBot._build_menu(page_state)` builds the action menu from the same observable-page predicates the random `_cycle` uses, listing only actions that are physically possible and policy-allowed (throttles like `can_post`/`can_project`/own-post-no-mention are enforced by *omitting* the action). `decide` filters the returned plan against that menu (`_filter_plan` drops any unknown action), re-asks once on malformed/empty output, and the executor `_run_plan` falls back to a single `navigate` to feed when the plan is still empty - **never a `random` draw**. `_dispatch_action` maps each action string to the existing per-action method (`_comment_on_post`, `_vote_for_page`, `_react_to_object`, `_navigate_to`, ...); content actions still run their `quality_check`/`gist_quality_check` gates unchanged, so quality cannot degrade.
- **`_cycle_ai` mirrors `_cycle`'s signature and return tuple** so `run_forever` swaps between them with `(self._cycle_ai if use_ai else self._cycle)(...)`. `energy` and `stop_after` replace `_session_type` mood and `_fatigue_check`: the session ends when `session_actions >= stop_after`, the intra-action pause is `_scaled_pause()` (scaled by `energy`), and the between-session break is scaled by `energy` too. Timing jitter (`_read_like_human`, `_type_like_human`, `_idle`) stays procedural - the model decides *what*, not the millisecond cadence. `use_ai` is `self._ai_decisions and bool(self.state.identity)`; identity generation and the flag are gated so a disabled fleet pays nothing and behaves exactly as before.