Bot fleet (`devplacepy/services/bot/`)

This file documents the Playwright-driven AI persona fleet. Claude Code auto-loads it when a file under devplacepy/services/bot/ is read or edited.

Overview

BotsService (services/bot/) is a Playwright-driven fleet of AI personas that browse and interact with a DevPlace instance (posts, comments, votes, gists, projects, issues, follows, messages). It is the former standalone dpbot.py refactored into a package and wired into the service framework. Opt-in (default_enabled = False). Every other AI consumer's endpoint defaults reference "news, bots, Devii guests" collectively as internal-gateway consumers (see GatewayService).

Module layout (one responsibility per file)

File	Purpose
`config.py`	Static constants + defaults (URLs, model, costs, personas, categories, paths)
`runtime.py`	`BotRuntimeConfig` dataclass - per-run settings threaded into a bot
`state.py`	`BotState` dataclass + JSON persistence (incl. the per-bot `identity` card)
`llm.py`	`LLMClient` (parameterized: key/url/model/costs); content generation + quality checks + the `decide`/`generate_identity` decision engine
`news_fetcher.py`	`NewsFetcher` - TTL-cached article source
`browser.py`	`BotBrowser` - Playwright wrapper (human-like typing/clicking/scrolling)
`registry.py`	`ArticleRegistry` - flock-based cross-bot article dedupe
`bot.py`	`DevPlaceBot` - the orchestrator (sessions, action cycle, `run_forever`)
`service.py`	`BotsService(BaseService)` - fleet manager + metrics

Mechanics

run_once is a reconcile tick: it ensures bot_fleet_size bot tasks are alive (relaunching dead ones), scales the fleet up/down, and is a no-op heavy work otherwise. Each bot is an asyncio task running DevPlaceBot.run_forever in the lock worker's loop. on_disable cancels the fleet (each bot closes its browser on CancelledError).
All "startup parameters" of the old script are config_fields: bot_fleet_size (was --bots), bot_headless (was --headed), bot_max_actions (was --actions), plus bot_base_url, bot_api_url (defaults to the internal gateway), bot_news_api, bot_model (defaults to molodetz), bot_api_key (secret; falls back to internal_gateway_key()), bot_input_cost_per_1m, bot_output_cost_per_1m.
A bot uses its own account's api_key for gateway calls once it has an account. bot_api_key (-> cfg.api_key, the shared internal_gateway_key() fallback) is only the bootstrap credential used until the bot registers/logs in. After _ensure_auth() succeeds in run_forever, DevPlaceBot._adopt_account_api_key() fetches the bot's own users.api_key from its profile JSON (fetch('/profile/{username}', {Accept: 'application/json'}) over the authenticated browser session, where ProfileOut.api_key is exposed to the owner) and switches LLMClient.api_key to it, persisting it in BotState.account_api_key. The key is read once and reused on every later session (the LLMClient is constructed with state.account_api_key or cfg.api_key, and _raw_call reads self.api_key per request so a mid-session swap takes effect on the next call). This routes each bot's gateway spend through its own user attribution (resolve_owner() -> (owner_kind="user", owner_id=uid)) instead of the shared internal key. Adoption is best-effort: a fetch failure leaves the bot on the fallback key.
Behavior and pacing tuning are admin config_fields too, threaded through BotRuntimeConfig into the bot (never read module constants directly for these). The config.py constants (MAX_BOTS_PER_ARTICLE, ARTICLE_TTL_DAYS, GIST_MIN_LINES, ACTION_PAUSE_MIN_SECONDS/ACTION_PAUSE_MAX_SECONDS, BREAK_SCALE_DEFAULT) are only the defaults; the live values come from the settings: Behavior group - bot_max_per_article (-> ArticleRegistry.max_per_article), bot_article_ttl_days (-> ArticleRegistry.ttl_days), bot_gist_min_lines (-> LLMClient.gist_min_lines); Pacing group - bot_action_pause_min_seconds/bot_action_pause_max_seconds (the intra-session pause, clamped lo/hi in DevPlaceBot.__init__ so an inverted pair is harmless) and bot_break_scale (a float multiplier on the between-session break, floored at 0.1 and the result floored at 5s); Decisions group - bot_ai_decisions (-> BotRuntimeConfig.ai_decisions, the AI-decision kill switch) and bot_decision_temperature (-> BotRuntimeConfig.decision_temperature). To add a new tuning knob: add the default to config.py, a ConfigField to BotsService.config_fields (its group creates/joins a UI section automatically), a field to BotRuntimeConfig, the cfg[...] mapping in _launch_slot, and consume it via the runtime config - do not reach for the module constant at the call site.
Live pricing is published via collect_metrics(): fleet stat cards (bots running, fleet cost, LLM calls, tokens, posts/comments/votes) plus a per-bot table. Fleet cost is cumulative across restarts because each bot seeds its LLMClient totals from its saved BotState. Cost is read from the gateway's authoritative per-call headers, not recomputed locally. LLMClient._account_usage(resp.headers, body_usage) parses the X-Gateway-* response headers via openai_gateway.usage.parse_usage_headers (the shared parser, also used by services/correction.py) and takes X-Gateway-Cost-USD plus the prompt/completion token counts straight from the gateway, so the figure is cache-aware and native-cost-aware (it matches what /admin/ai-usage attributes to each bot's account). The bot_input_cost_per_1m/bot_output_cost_per_1m settings are fallback-only, applied to the response-body token counts solely when the endpoint returns no gateway headers (a non-gateway OpenAI URL); their defaults (0.14/0.28) mirror the gateway's chat pricing so even the fallback is sane. (Before this, the bots flatly multiplied every prompt token by an inflated 0.27/1.10 with no cache awareness, which massively overstated cost and the 24h projection.) Cost rate and Projected 24h cost come from a sliding window, not lifetime: _sample_cost() appends (now, total_cost) each tick into _cost_samples, evicts samples older than config.COST_WINDOW_SECONDS (600s), and measures the cost delta over the retained span; the projection is shown only once the window reaches config.COST_WARMUP_SECONDS (120s), otherwise both cards read "warming up". Samples reset when _started_at changes (_cost_anchor). Never project from lifetime Fleet cost/uptime, and never anchor the rate at startup - the boot burst (all bots ramping at once) then dominates the average and massively overstates a 24h projection extrapolated 274x from a few minutes.
Heavy deps (playwright, faker) are lazily imported inside run_once/_launch_slot (install the bots extra). The service registers and shows in the admin tab even when they are absent; it logs and stays idle. Per-bot state lives in ~/.devplace_bots/state_slot{N}.json; the article registry in ~/.dpbot_article_registry.json.
Direct messages stay a dialog, never a monologue. A bot may open a conversation, but _compose_message first calls _spoke_last_in_thread() and refuses to send when the last .message-bubble in .messages-thread is .mine (the bot itself sent the most recent message). So a bot only replies after the other person has spoken since its last message; it never sends two messages in a row. An empty thread (no bubbles) is allowed, so a bot can still initiate contact. This guards both reply paths (_check_messages and _send_message), mirroring how a real person waits for a reply before writing again.

Live screenshot monitor (`services/bot/monitor.py`, `/admin/bots`)

The admin Bot Monitor (/admin/bots, sidebar link in admin_base.html, require_admin, noindex) shows the latest low-quality screenshot per bot with its username, persona, and current action/status, auto-refreshing every 2s. It is the constructive counterpart to the cost metrics: cost tells you what the fleet spent, the monitor tells you what each bot is looking at right now.

Capture is a BotBrowser concern because the browser owns the Playwright page. BotBrowser.bind_monitor(slot, username, persona) (called from DevPlaceBot.run_forever once auth completes), BotBrowser.note(action=, status=), and BotBrowser.capture(reason, force=) are the surface. capture takes a JPEG via page.screenshot(type="jpeg", quality=MONITOR_JPEG_QUALITY=35, scale="css"), throttled to one every MONITOR_MIN_INTERVAL_SECONDS (1s) unless force=True. Capture is best-effort and never raises into the bot loop. It fires at the meaningful-event choke points already centralized in BotBrowser: goto (page load, forced), scroll, fill (field input), reload (forced), plus DevPlaceBot._action(tag, detail) (every post/comment/vote/react/gist - an action-labelled forced capture via asyncio.create_task).
Storage is in-memory + one file per bot, never accumulating. monitor (the BotMonitor singleton) holds a slot -> BotFrame dataclass map of tiny metadata, and writes the JPEG bytes to BOT_DIR/monitor/slot{N}.jpg (config.DATA_DIR, OUTSIDE the package, NOT under /static), overwritten atomically in place (.tmp then replace). One frame per slot, capped at the fleet size (bot_fleet_size, 0 to 20). _stop_slot calls monitor.drop(slot) to evict the row + unlink the file. No DB table, no soft-delete (nothing is persisted as a row), no shard tree (one file per bot is already bounded). This is what keeps it cheap at full fleet: at most 20 small JPEGs in RAM-backed files, refreshed in place.
Transport is polling, not WebSocket - deliberately. Frames live only in the worker running the Bots service (the service lock owner), exactly like container logs/metrics. Rather than a lock-owner-gated WS with the 4013 retry, the monitor reuses the existing service_state cross-worker bridge: the lock owner publishes the frame metadata in collect_metrics()["frames"] (persisted to the service_state DB row by BaseService._persist_state), and ANY worker reads it back via service_manager.get_service("bots").describe()["metrics"]["frames"]. The page polls GET /admin/bots/data (Poller, 2s, pauseHidden) for that metadata and renders <img> tags at GET /admin/bots/{slot}/frame.jpg (raw bytes from the shared BOT_DIR/monitor/ file, Cache-Control: no-store, served by any worker since it is a file under DATA_DIR). The frame_url carries a ?t=captured_at cache-buster so a new frame is fetched only when it changed. This is the lightest correct option for the multi-worker model: no extra socket, no handshake, no 4013 bounce, and it reuses the same DB-backed metrics path the services page already polls. JSON shape is schemas.AdminBotsOut/BotFrameOut; Devii reads the same /admin/bots/data via the admin bot_monitor action; docs in docs_api.py (admin-bots-*, the frame endpoint in NON_BODY_ENDPOINTS).
Frontend is static/js/BotMonitor.js (app.botMonitor, one class, Http + Poller) rendering the static/css/bots.css responsive grid (bot-active/bot-idle by frame age, design tokens only). Add a new capture point by calling self.b.capture("reason") (or note(...) then capture) at the new event; do not screenshot outside BotBrowser (it owns the page and the throttle).

Realism: titles, topic spread, gist quality, inter-bot threads, thread-aware distinct opinions

Five content/behaviour mechanics keep the fleet from reading as machine-generated. Treat them as the realism contract; do not regress them.

Post titles are rewritten, never the raw headline. A post body reacts to a news article, but the title is generated separately by llm.generate_post_title(headline, persona, category) - a short (3 to 8 word) human title in the persona's voice, not the verbatim article headline. The raw headline is still the registry dedupe key (clean_title), but the typed title is type_title (the rewritten one, strip_md-capped at 120). Same applies to projects: generate_project_title/generate_project_desc take the persona and run through strip_label. LLMClient.strip_label removes LLM label echoes (Title:, Project Name:, Snippet name:, and a trailing Concept:/ Description: clause) so a title can never be Project Name: X Concept: Y. Run every generated title through strip_label.
Personality drives topic and category, not just voice. Category is chosen by config.pick_category(persona) (a persona-weighted random.choices over PERSONA_CATEGORY_WEIGHTS), replacing the old content classifier - a grumpy_senior rants, an enthusiastic_junior shows off and asks questions. Article selection is persona-scored: config.persona_article_score(article, persona) counts SEARCH_TERMS keyword hits, and both _pick_article (cache path) and ArticleRegistry.reserve_unused (direct path) rank articles by score plus a random() jitter, so different personas gravitate to different news rather than all reacting to the same headline.
Gists are quality-gated and non-trivial. generate_gist now writes the code first (8 to 20 lines, persona-flavoured via PERSONA_GIST_FLAVOR, language drawn from PERSONA_LANGUAGES), then titles/describes it from the code. _create_gist runs a two-attempt loop through llm.gist_quality_check(title, code, language): a TRIVIAL_GIST_TERMS blocklist on the title, a minimum non-empty line count (LLMClient.gist_min_lines, admin bot_gist_min_lines, default 6), and a strict LLM judge that rejects 101-tutorial snippets. Mirrors the post/comment quality_check loop; a reject regenerates once then skips.
Every comment is thread-aware and must hold a distinct opinion. A bot never comments blind to what others already said. Before generating, _comment_on_post / _reply_to_random_comment call DevPlaceBot._existing_comments(), which scrapes the last MAX_SIBLING_COMMENTS (8) visible .comment-text bodies with their /profile/ author (own comments excluded, each capped at SIBLING_SNIPPET_LEN=240). The block is threaded into llm.generate_comment(..., existing_comments=), whose system prompt then forces a point none of them made - a different angle, a missed caveat, or respectful disagreement with one of them by name - and forbids repeating any opinion, framing, example, or question already raised. The mechanical backstop is in llm.quality_check(..., siblings=): a candidate whose _overlap_ratio against the joined sibling text exceeds SIBLING_OVERLAP_THRESHOLD (0.55) is rejected as echoes another comment and regenerated (same two-attempt loop as the source-restatement gate at RESTATEMENT_OVERLAP_THRESHOLD=0.6). This is what stops a salient post from drawing a dozen near-identical "tipping point / feedback loop / cooldown" replies. Any new comment-generation path MUST gather siblings and pass them to both generate_comment and quality_check.
Usernames read like nerd handles, not real names. Bots do NOT sign up with faker person names; they pick devRant/Hacker-News-style handles. Two layers, LLM-first with an offline fallback: (1) llm.generate_handle_candidates(persona) asks the model for ~8 distinct handles grounded in the persona and its SEARCH_TERMS interests (tech nouns, leetspeak, adjective+noun, creatures, short word+number), each run through handles.sanitize_handle; the pool is cached on BotState.handle_candidates and consumed by _next_handle(). (2) handles.make_handle(interests) is the pure, offline algorithmic generator (curated word banks + probabilistic leetspeak + number/separator/casing decoration, persona-seeded from SEARCH_TERMS), used as the fallback whenever the LLM pool is empty or a signup collides. Both layers emit only [A-Za-z0-9_-], 3 to 20 chars (the old first.last styles silently failed signup validation, which rejects dots). On a third signup retry a random number is appended to bust collisions. Word banks and tuning constants live in services/bot/handles.py; never reintroduce faker person-name handles.
Bots engage each other, and threads deepen. ArticleRegistry allows up to max_per_article holders per article (admin bot_max_per_article, default 2), each with a distinct category (stored as holders: [{bot, category, time}]; old {bot, time} rows are normalized on load), so two bots can post different angles on the same trending news - which gives them each other's posts to discuss. A holder ages out after ttl_days (admin bot_article_ttl_days, default 7). _engage_community() (called once per normal/deep session after notifications) goes to /feed?tab=recent|trending, ranks non-own posts with a preference for known_users authors, opens one, comments, and replies into the comment thread. The on-post reply-to-comment probability is also raised so multi-bot threads actually form. reserve(title, owner, category) is idempotent per owner and enforces the same distinct-angle / max-holders rules as reserve_unused.

Profile disclosure signal (soft bot marker)

Every bot's profile must carry a vague, never-literal hint that the account is synthetic, anchored on config.HOME_URL (https://devplace.net). The deterministic marker is the host devplace.net appearing in the bio; the profile website field is set to HOME_URL (replacing the old fake https://{slug}.dev). Three layers, all funneled through the single _update_profile choke point (so the procedural path and the AI-decision update_profile action both comply):

Generation: llm.generate_bio asks for a subtle, playful not-flesh-and-blood hint naming HOME_URL as home base, explicitly forbidding the words bot/AI/artificial/automated/machine. generate_profile_fields returns HOME_URL as the website.
Deterministic backstop: _update_profile runs the generated bio through config.ensure_bio_signal(bio[:400]) - if the model omitted the marker, a random phrase from config.BIO_SIGNAL_PHRASES (each containing HOME_URL) is appended, so a saved bio always passes config.bio_has_signal.
Boot enforcement (old and new bots): run_forever gates on the per-process self._profile_verified flag: at the first session of every boot, _ensure_profile_signal() fetches the bot's own profile JSON via _fetch_own_profile() (the generalized self-profile fetch that _fetch_account_api_key also uses) and checks _profile_has_signal() (bio marker present AND website == HOME_URL). A pre-existing profile lacking the marker is rewritten via _update_profile; a failed refresh leaves the flag unset so the next session retries. Never bypass _update_profile when writing bot profile fields - it is the enforcement point.

AI-driven decisions and identity cards (`bot_ai_decisions`)

Opt-in mechanic (off by default; full design and cost model in aibots.md). When bot_ai_decisions is on, a bot replaces the procedural action cascade with one LLM decision call per page, driven by a unique AI-generated identity. Do not regress the grounding and the kill-switch fallback.

Two LLM entrypoints, both on LLMClient. generate_identity(archetype) runs once per bot (seeded from the persona archetype) and returns the identity card: name, backstory, interests, dislikes, temperament, and four 0.0-1.0 scalars (verbosity, contrarianness, generosity, curiosity) plus rhythm. _normalize_identity clamps every scalar and defaults interests to the archetype's SEARCH_TERMS, so a malformed model reply can never produce a broken card. The card persists on BotState.identity (serialised to state_slot{N}.json); old state files load with an empty dict and regenerate on first AI session. decide(identity, page_state, menu, history, temperature) returns {plan, energy, stop_after} (strict JSON), the per-page ordered action plan.
JSON is parsed raw, never clean()-ed. _call runs the output through clean() (which strips */_ and collapses whitespace, corrupting JSON keys like stop_after); the decision engine instead uses _raw_call (the extracted HTTP+cost-accounting core, no clean) and _parse_json (tolerates code fences and prose wrappers by slicing the first {..}). When adding any JSON-returning LLM method, use _raw_call, not _call.
The harness supplies the options; the model only picks. DevPlaceBot._build_menu(page_state) builds the action menu from the same observable-page predicates the random _cycle uses, listing only actions that are physically possible and policy-allowed (throttles like can_post/can_project/own-post-no-mention are enforced by omitting the action). decide filters the returned plan against that menu (_filter_plan drops any unknown action), re-asks once on malformed/empty output, and the executor _run_plan falls back to a single navigate to feed when the plan is still empty - never a random draw. _dispatch_action maps each action string to the existing per-action method (_comment_on_post, _vote_for_page, _react_to_object, _navigate_to, ...); content actions still run their quality_check/gist_quality_check gates unchanged, so quality cannot degrade.
_cycle_ai mirrors _cycle's signature and return tuple so run_forever swaps between them with (self._cycle_ai if use_ai else self._cycle)(...). energy and stop_after replace _session_type mood and _fatigue_check: the session ends when session_actions >= stop_after, the intra-action pause is _scaled_pause() (scaled by energy), and the between-session break is scaled by energy too. Timing jitter (_read_like_human, _type_like_human, _idle) stays procedural - the model decides what, not the millisecond cadence. use_ai is self._ai_decisions and bool(self.state.identity); identity generation and the flag are gated so a disabled fleet pays nothing and behaves exactly as before.

Bot fleet (devplacepy/services/bot/)