# ref.md - Monster-file split plan (backend)

Author: retoor <retoor@molodetz.nl>

This document is the execution plan for splitting every backend "monster" file in
DevPlace into a package of cohesive modules, without changing behavior and without
breaking any importer or test. It is investigation output only; nothing here has
been executed. Each section is self-contained and can be executed independently in
the recommended order (Section 12).

---

## 1. Scope and definition

"Monster" = first-party backend Python file over 900 lines (excludes vendored JS,
generated artifacts, and the scratch/workspace copies under `data/`, `var/`, `tmp/`,
and the repo-root `maak.py`/`d.py`). The confirmed set:

| # | File | Lines | Split kind |
|---|------|------:|------------|
| 1 | `devplacepy/docs_api.py` | 5659 | package: one module per API group |
| 2 | `devplacepy/database.py` | 3764 | package: layered domain modules |
| 3 | `devplacepy/services/bot/bot.py` | 2999 | in-package: cooperative mixins |
| 4 | `devplacepy/services/containers/files/bot.py` | 2940 | SPECIAL - deployed single-file asset |
| 5 | `devplacepy/services/containers/files/d.py` | 2337 | SPECIAL - deployed single-file asset |
| 6 | `devplacepy/services/devii/actions/catalog.py` | 2036 | same-name package: one module per action domain |
| 7 | `devplacepy/schemas.py` | 1285 | package: domain schema modules |
| 8 | `devplacepy/cli.py` | 1080 | package: one module per command group |
| 9 | `devplacepy/utils.py` | 1024 | package: layered helper modules |
| 10 | `devplacepy/services/news.py` | 986 | package: pipeline-stage modules |
| 11 | `devplacepy/services/devii/session.py` | 962 | same-name package: extract stateless text/helpers |
| 12 | `devplacepy/services/game/store.py` | 948 | subpackage: cohesive store modules |

`services/bot/llm.py` (792) is below threshold and holds two realism mechanics; it is
explicitly **left intact** (Section 3.4).

---

## 2. Global rules for every split (apply to all sections)

These are invariants. Every section assumes them; deviations are called out per file.

1. **Same import path.** Convert `foo.py` into a package `foo/` (directory with
   `__init__.py`). Because Python resolves `foo/__init__.py` for `import ...foo`, every
   existing `from ...foo import X` and `import ...foo as m; m.X` keeps resolving with no
   caller edit. This is the ONLY compatibility mechanism used.

2. **Exhaustive re-export in `__init__.py`.** The package `__init__` must re-export the
   complete public surface the codebase and tests currently import - **including
   underscore-prefixed names that are imported elsewhere** (they are part of the effective
   surface; e.g. `database._now_iso`, `utils._resolve_user`, `docs_api._substitute`,
   `store._today`). Prefer **explicit** `from .module import name1, name2` blocks over
   `from .module import *` so the surface is auditable and star-import name-drops
   (underscore filtering, `__all__` gaps) cannot silently regress it.

3. **No behavior change.** Pure structural moves. No logic edits, no reordering of
   ordered data, no renames of public symbols, no signature changes.

4. **Preserve every function-local (lazy) import.** Many files defer imports inside
   function bodies specifically to break import cycles (`database` <-> `utils`,
   `utils` <-> `services.audit`/`templating`/`content`, `news` AI-key, game `leaderboard`).
   These MUST stay function-local in the new modules. Hoisting a lazy import to module
   top is the single most likely way this refactor introduces a boot-time cycle.

5. **File header.** Every new file starts on line 1 with `# retoor <retoor@molodetz.nl>`
   (project rule). Carry `from __future__ import annotations` into any module that uses
   `X | None` style annotations where the original had it.

6. **No em-dash.** Neither the em-dash character nor `&mdash;`/`&#8212;`/`&#x2014;`
   anywhere in new files. Hyphen only. Grep every new file before finishing.

7. **No forbidden names / no comments / no docstrings in source** (project rules): do not
   introduce `_new`/`_old`/`_temp`/`_v2`/`better_`/`my_`/`the_`, do not add comments or
   docstrings.

8. **`_shared.py` is the sanctioned package-private helper module.** The project already
   uses `_shared.py` in eight router packages for cross-leaf helpers. Use that exact name
   for genuinely package-private construction helpers (docs_api builders, catalog builders,
   cli `_audit_cli`). Everything else is a normal named module.

9. **Validation after each split (never run the test suite):**
   - `python -c "from devplacepy.main import app"` must import clean.
   - `python -m py_compile` every new file.
   - A public-surface diff: capture `sorted(n for n in dir(<module>) if not n.startswith('__'))`
     BEFORE the split (from git or a saved snapshot) and assert the post-split package
     exposes a superset. This is the definitive check that no re-export was missed.
   - em-dash grep over every new file.
   - The suite is run only if the user explicitly asks; the plan never runs it.

---

## 3. Per-file plans

### 3.1 `docs_api.py` (5659) -> `devplacepy/docs_api/`

**Nature.** A flat declarative module: a few construction helpers, one giant
`API_GROUPS` list literal (~5240 lines, `docs_api.py:143-5384`, 19 group dicts), then
a post-processing/registry/render tail. Endpoints are organized by API group; each group
is a self-contained dict literal built via the `endpoint()` factory.

**Public surface (only these are imported externally):**
`API_GROUPS`, `build_services_group`, `get_group`, `api_doc_pages`, `render_group`, and
`_substitute` (underscore but called by `docs_export.py:119`).

Importers: `docs_search.py:9,87,90`; `docs_export.py:10,118,119,121`;
`services/xmlrpc/registry.py:8,63`; `routers/docs/pages.py:3`; `routers/docs/views.py:14`.
Tests: `tests/e2e/docs.py:9`, `tests/unit/project_files.py:381-385`,
`tests/api/auth/matrix.py:7`, `tests/api/docs/{index,downloadhtml,downloadmd}.py`.

**Target layout:**

```
devplacepy/docs_api/
  __init__.py            # assembles API_GROUPS in order, runs the negotiation pass,
                         #   re-exports the public surface (incl. _substitute)
  _shared.py             # field, endpoint, ROLE_LABELS, NON_BODY_ENDPOINTS, and the
                         #   domain constant lists (VOTE_TARGETS, REACTION_TARGETS,
                         #   BOOKMARK_TARGETS, COMMENT_TARGETS, PROJECT_TYPES,
                         #   GIST_LANGUAGES, SERVICE_ACTIONS)
  negotiation.py         # _PAGE_RESPONSES, _ACTION_RESPONSES, _classify,
                         #   _apply_negotiated_responses(groups)   (schemas/docs_examples live here)
  services_group.py      # _service_control_endpoints, _field_to_param, _service_section,
                         #   build_services_group
  render.py              # _GROUPS_BY_SLUG, get_group, api_doc_pages, _substitute, render_group
  groups/
    __init__.py          # ORDERED_GROUPS = [conventions.GROUP, auth.GROUP, ... game.GROUP]
    conventions.py       # docs_api.py:145-293
    auth.py              # 295-427
    lookups.py           # 428-484
    social_actions.py    # 485-674
    content.py           # 675-1615   (largest group, ~940 lines)
    profiles.py          # 1616-2294
    messaging.py         # 2295-2365
    notifications.py     # 2366-2450
    uploads.py           # 2451-2552
    project_files.py     # 2553-3098
    containers.py        # 3099-3673  (admin: True)
    tools.py             # 3674-3902
    push.py              # 3903-3964
    issues.py            # 3965-4258
    gateway.py           # 4259-4449  (admin: True)
    services.py          # 4450-4456  (dynamic placeholder group dict - keep hollow)
    admin.py             # 4458-5193  (admin: True, ~735 lines)
    game.py              # 5194-5383
```

Each `groups/*.py` holds one `GROUP = {...}` literal importing `field`, `endpoint`,
`ROLE_LABELS` and the constant lists from `.._shared` (and `constants.TOPICS`/
`REACTION_EMOJI` where the literal uses them - re-scan each block for free names before
cutting).

**`__init__.py` must, in this exact order:**
1. `from .groups import ORDERED_GROUPS as API_GROUPS` (order is load-bearing).
2. `from .negotiation import _apply_negotiated_responses` then call
   `_apply_negotiated_responses(API_GROUPS)` - reproduces the current import-time mutation
   at `docs_api.py:5494`. Must run AFTER `API_GROUPS` is assembled.
3. Re-export `API_GROUPS`, `build_services_group`, `get_group`, `api_doc_pages`,
   `render_group`, `_substitute` (and, for safety, `endpoint`, `field`, `ROLE_LABELS`,
   the constant lists).

**Risks / ordering hazards (must be handled):**
- **Import-time negotiation mutation.** `_apply_negotiated_responses()` runs at module top
  today (`docs_api.py:5494`) and mutates each endpoint dict in place (fills
  `sample_response`, stamps `negotiation`). Centralize this in `__init__` after assembly;
  if a group is read before the pass runs, endpoints lack `negotiation`/`sample_response`.
- **Forward reference.** `_classify` (`105-112`) reads `_PAGE_RESPONSES`/`_ACTION_RESPONSES`
  defined 5280 lines later. Colocate `_classify` + both maps + the pass in `negotiation.py`
  to make the dependency explicit.
- **Assembly-before-render cycle.** `render.py` (`get_group`/`render_group`/`api_doc_pages`/
  `_GROUPS_BY_SLUG`) needs the final `API_GROUPS`. Build `API_GROUPS` in `groups/__init__.py`
  and import it into both package `__init__` and `render.py`; do NOT let `render.py` import
  the package `__init__` at module top (cycle).
- **`build_services_group` reuses core helpers** (`endpoint`, `field` via `_field_to_param`,
  `_classify`, `_service_control_endpoints`) and stamps `negotiation` itself for its runtime
  group; `services_group.py` imports `endpoint`/`field` from `_shared` and `_classify` from
  `negotiation`; do NOT double-apply negotiation to it.
- **`_substitute` name must be preserved** in the re-export (called as
  `docs_api._substitute` by `docs_export.py:119`).
- **Two groups slug `services`** (the hollow placeholder in the list + the runtime
  `build_services_group` output). Keep the placeholder exactly where it is in the ordered list.

**Test invariant:** `API_GROUPS` order/content, group flags (`slug/title/admin/dynamic/
endpoints`), and every endpoint's post-negotiation shape (`id/method/path/auth/min_role/
params/negotiation/sample_response`) must be identical after the split. `tests/api/auth/
matrix.py` asserts `len(endpoints) >= 50` and drives each endpoint's documented auth against
live behavior; `tests/unit/project_files.py:381-385` asserts the `project-files` group's exact
endpoint id set.

---

### 3.2 `database.py` (3764) -> `devplacepy/database/`

**Nature.** The core data layer, imported nearly everywhere (677 `from ...database import`
statements across 329 files; hottest names `get_table` x404, `refresh_snapshot` x92,
`set_setting` x67, `db` x33). The `__init__` MUST re-export essentially the entire namespace
including underscore helpers (`_now_iso` x6, `_index`, and whatever tests reach for).

**The linchpin.** A single `core.py` owns the `db` singleton, the import-time connection
side effects, the cross-worker cache-version primitives, index helpers, and trivial
accessors. Every other module imports only from `core` (and strictly-lower layers). Because
`core.py` is imported once, Python's module cache guarantees the single `db` object, exactly
as today.

**Target layout (`devplacepy/database/`):**

```
core.py            # db, refresh_snapshot; import-time ensure_data_dirs + sqlite mkdir +
                   #   dataset.connect (database.py:20-42); cache-version primitives
                   #   (_ensure_cache_state, get_cache_version, bump_cache_version,
                   #   sync_local_cache, _local_cache_versions, _cache_version_cache,
                   #   _cache_state_ready); _index, _drop_index, _uid_index;
                   #   get_table, _in_clause, _now_iso          <- imported by ALL
settings.py        # _settings_cache, get_setting, get_int_setting, set_setting,
                   #   clear_settings_cache, internal_gateway_key
users.py           # get_users_by_uids, search_users_by_username, set_user_timezone,
                   #   admins cache (_admins_cache, invalidate_admins_cache,
                   #   get_admin_uids, get_primary_admin_uid)
relations.py       # user block/mute cache + accessors (get_user_relations, get_blocked_uids,
                   #   get_muted_uids, get_silenced_uids, invalidate_user_relations)
pagination.py      # PAGE_SIZE, build_pagination, paginate, paginate_diverse,
                   #   interleave_by_author, get_user_post_count   (imports core + relations)
soft_delete.py     # SOFT_DELETE_TABLES, ensure_soft_delete_columns, soft_delete,
                   #   soft_delete_in, restore, purge, list_deleted, count_deleted,
                   #   restore_event, purge_event   (imports core + pagination.build_pagination)
engagement.py      # comment-count/vote/reaction/bookmark batch helpers, polls
comments.py        # _drop_blocked, _build_comment_items, load_comments, recent/by-target loaders
attachments_data.py# get_attachments*, get_news_images_by_uids, delete_attachment*,
                   #   _delete_attachment_file, get_user_media, get_deleted_media
usage.py           # _add_usage, _get_usage, add/get_*_usage + *_USAGE_KEY constants
seo_meta.py        # SEO_META_TYPES + seo_metadata CRUD
activity.py        # user_activity counters + calendar/heatmap/streaks
ranking.py         # VOTABLE_TARGETS, STAR_TARGETS, authors/leaderboard/stars,
                   #   update_target_stars, get_target_owner_uid, soft_delete_engagement,
                   #   delete_engagement
customization.py   # customization prefs + override CRUD
email.py           # email account CRUD
notifications.py   # notification prefs + mark_notifications_read_by_target
forks.py           # record_fork, get_fork_parent, count_forks, *_fork_relations
content.py         # resolve_by_slug, resolve_object_url, text_search_clause,
                   #   get_uids_by_username_match, get_daily_topic, get_featured_news
follows.py         # get_follow_counts, get_follow_list, get_following_among
stats.py           # get_site_stats, get_platform_analytics, get_gist_languages
deepsearch.py      # _ds_now + deepsearch session/message/url-cache helpers
schema.py          # init_db, _refresh_query_planner_stats, migrate_bug_tables_to_issue_tables,
                   #   BUG_TABLE_RENAMES, migrate_ai_gateway_settings, OLD_GATEWAY_URL,
                   #   backfill_api_keys, _backfill_gamification    <- imports EVERYTHING; loaded last
__init__.py        # explicit re-export of every module's public + test-touched underscore names
```

**Where shared state lives (critical).** The only truly global object is `db` (+ the
cache-version machinery), which lives in `core.py`. Every domain `TTLCache`/dict singleton
(`_admins_cache`, `_settings_cache`, `_relations_cache`, `_customizations_cache`,
`_notification_prefs_cache`, `_stats_cache`, `_analytics_cache`, `_activity_cache`,
`_gist_languages_cache`, `_authors_cache`, `_comment_count_cache`) is referenced only by its
own domain, so it moves WITH that domain's module - no shared-cache problem. Never re-create
a cache in `__init__`; only re-export. Two import paths to one cache would create two caches.

**Layering (acyclic).** `core` <- everything; `pagination -> relations`;
`soft_delete -> pagination`; `comments -> users + engagement + relations`;
`ranking -> users`; `stats -> ranking`; `attachments_data -> content`; `content -> users`;
`schema` imports any sibling at module top (nothing imports `schema`; loaded last in `__init__`).

**Cycle-avoidance rules:**
- `core.py` imports nothing from siblings.
- Strict layer order: `core -> settings/users/relations -> pagination -> soft_delete ->
  domain modules -> schema`.
- **Preserve every existing function-local import verbatim** - the `utils`, `attachments`,
  `templating`, `services.audit.store`/`services.backup.store`/`services.openai_gateway.routing`
  imports inside functions (e.g. `init_db` at `database.py:951-961`) exist to break the real
  `database <-> utils` cycle. Do not hoist them.
- `utils.py` uses `database`'s cache primitive under the `"auth"` name; those primitives are in
  `core.py` and remain reachable as `database.bump_cache_version`/`database.sync_local_cache`
  via re-export.

**Risks:**
- **`init_db()` (503-1534) is one idempotent ordered routine** - move it WHOLE into `schema.py`;
  do not reorder its internal steps (migration-before-index, backfills/ANALYZE last at 1516-1519).
- **Import-time side effects (20-42)** must live in `core.py` and run exactly once; `core` must
  be the first thing `__init__` imports.
- **Underscore re-export gap** is the highest-probability regression (`import *` drops `_now_iso`/
  `_index`). Use explicit re-exports or per-module `__all__` including the imported underscores.
- **`get_table` (404 importers) and `refresh_snapshot` (92)** - any miss breaks a huge fraction of
  the app at import; the public-surface diff (Section 2.9) is the guard.

**Test footprint.** `tests/unit/database.py` is the primary direct consumer (it also hits a live
server, so it is effectively api-tier); `tests/unit/services/gitea.py` and
`tests/unit/services/openai_gateway/analytics.py` import it for setup; dozens of api/e2e files use
`get_table`/`set_setting`/usage helpers indirectly. With a complete re-export, zero test edits are
needed.

---

### 3.3 `services/bot/bot.py` (2999) -> cooperative mixins in `services/bot/`

**Nature.** ONE class `DevPlaceBot` (`bot.py:48`), ~90 methods, every method reads/writes
shared `self` state (`self.state` persisted `BotState`, `self.b` browser, `self.llm`,
`self._session_*` counters, the `_post_cache/_project_cache/_issue_cache/_session_history`
caches). The only zero-behavior-change decomposition is **cooperative mixins** (Python MRO gives
byte-identical `self` semantics; the unbound-method unit test still resolves via inheritance).
There is NO `@tool` registry in this file (the closest analogue is the AI decision/menu engine).

**Target layout (new modules beside the existing `services/bot/` files):**

| New module | Mixin | Moves from bot.py | Contents |
|------------|-------|-------------------|----------|
| `helpers.py` | `BotHelpersMixin` | 103-245 | instrumentation, cost/session banners, `_thread_id`, `_pick_article`, cache warm |
| `auth.py` | `BotAuthMixin` | 246-355 | handles, `_ensure_auth`, account-key adoption |
| `human.py` | `BotHumanMixin` | 356-522 | read/type/fatigue simulation, session type, page inspection |
| `agent.py` | `BotAgentMixin` | 523-757 | AI decision/menu/plan engine (`_build_menu`, `_run_plan`, `_cycle_ai`, dispatch, nav) |
| `engage.py` | `BotEngageMixin` | 753-1436 | voting, commenting, reacting (realism #4b) |
| `posting.py` | `BotPostingMixin` | 1438-1770 | gist + post creation, content open/click (realism #1, #3 call-sites) |
| `social.py` | `BotSocialMixin` | 1771-2378 | profile/project/issue/follow, notifications, mentions (#4b), messaging |
| `browse.py` | `BotBrowseMixin` | 2379-2502 | scroll, nav, browse, search, `_engage_community` |
| `loop.py` | `BotLoopMixin` | 2503-2999 | `_cycle`, `run_forever` (scheduling) |

`bot.py` shrinks to ~110 lines: the mixin imports, `__init__`, and:

```python
class DevPlaceBot(
    BotHelpersMixin, BotAuthMixin, BotHumanMixin, BotAgentMixin,
    BotEngageMixin, BotPostingMixin, BotSocialMixin, BotBrowseMixin, BotLoopMixin,
):
    ...
```

`DevPlaceBot` stays importable from `devplacepy.services.bot.bot` (do NOT move the class file;
only lift methods into inherited mixins). Method names are already globally unique across the
90 methods - verify no collision when composing.

**Realism contract (MUST NOT regress) - exact locations to preserve:**
1. Title rewrite: `LLMClient.generate_post`/`generate_post_title` (in `llm.py`, untouched);
   call-sites `bot.py:205,211,1571,1577,1597` move into `helpers.py`/`posting.py` unchanged.
2. Persona topic spread: `persona_article_score`/`pick_category` (`config.py`, untouched);
   call-sites in `helpers.py`/`posting.py`.
3. Gist quality gate: `LLMClient.gist_quality_check`/`quality_check` (`llm.py`, untouched);
   enforced in `posting.py` (`_create_gist`) and `engage.py` (`_comment_on_post`).
4. Inter-bot threads: `ArticleRegistry` (own module, untouched) PLUS in-bot taper
   (`state.thread_reply_counts`/`MAX_THREAD_REPLIES` `bot.py:2163-2166,2272-2277`,
   `MAX_SIBLING_COMMENTS` `912,997-1020,1155-1164`, `_respond_to_mention` 2143,
   `_reply_to_random_comment` 1100, `_spoke_last_in_thread` 2305) - keep this logic together
   and unchanged across `engage.py`/`social.py`.

**`__init__.py` - keep exporting ONLY `BotsService` (lazy-Playwright boundary).** `service.py`
lazy-imports `bot.py` and treats `ImportError` as "install the bots extra". `bot.py` imports
`faker` and `BotBrowser` (Playwright) at module top. If the package `__init__` eagerly imported
`DevPlaceBot`, importing `devplacepy.services.bot` (for `BotsService`, used by `main.py`) would
hard-fail on boxes without the extra. So: leave `__init__.py` exporting only `BotsService`; keep
`DevPlaceBot`/`LLMClient` importable from their own modules (all live importers already use those
paths - `service.py:263`, `bot.py:39,59`).

**Stealth rule - already compliant, do not regress.** The only outbound HTTP is
`LLMClient._raw_call` (`llm.py:69`) via `stealth.stealth_sync_client`. `bot.py` makes no bare
`httpx` calls (all page "fetch" is inside Playwright). No new module may introduce a bare
`httpx.Client`.

**Test footprint.** `tests/unit/services/bot/bot.py` imports `bot as bot_module` and
`DevPlaceBot`, calling `DevPlaceBot._adopt_account_api_key(fake)` as an unbound method - safe
under mixins (method stays an attribute via inheritance). Admin fleet api/e2e tests go through
`BotsService`, unaffected.

#### 3.4 `services/bot/llm.py` (792) - LEAVE INTACT (explicit decision)

Below the 900-line threshold, a single cohesive `LLMClient` with shared `_call`/cost counters,
and it holds two realism mechanics (title rewrite, gist gate). A mixin split would be
net-negative and risk the contract. Keep it whole. (If ever reduced later, the only safe cut is
lifting the static text cleaners `clean`/`strip_md`/`strip_label`/`strip_code_fences`/`_parse_json`/
`_clamp01`/`_as_int`/`_as_terms`/`_overlap_ratio` into an `llm_text.py` of module functions that
`LLMClient` staticmethods delegate to, preserving `LLMClient.clean` as a public alias. Not part of
this plan.)

---

### 3.5 `services/devii/actions/catalog.py` (2036) -> same-name package `catalog/`

**Nature.** NOT the `Catalog`/`Action` classes (those live in `actions/spec.py`) and NOT the
live `CATALOG` (that is `registry.py:26-46`, `Catalog(actions=ACTIONS + TASK_ACTIONS + ...)`,
with `_assert_confirm_params(CATALOG)` an import-time side effect at `registry.py:68`). This file
only defines `ACTIONS` (161 `Action` literals, `catalog.py:43-2034`), five private builder helpers
(`path/query/body/upload/confirm`, `8-37`), two description constants (`ATTACHMENTS`, `TARGET_TYPE`,
`40-41`), and `PLATFORM_CATALOG = Catalog(actions=ACTIONS)` (`2036`). No logic.

**Public surface:** `ACTIONS` and `PLATFORM_CATALOG` only. Importers: `actions/__init__.py:3,7`,
`registry.py:9,27`, and tests `tests/unit/project_files.py:79,366,368`, `tests/unit/schemas.py:117,
123,132`, `tests/unit/services/devii/actions/dispatcher.py:6`,
`tests/unit/services/devii/actions/catalog.py:3`, `tests/api/projects/files/{lines,replacelines}.py:79`.

**Target layout (`services/devii/actions/catalog/`):**

```
__init__.py        # imports each domain tuple, ACTIONS = (AUTH_ACTIONS + POSTS_ACTIONS + ...),
                   #   PLATFORM_CATALOG = Catalog(actions=ACTIONS); __all__ = ["ACTIONS","PLATFORM_CATALOG"]
_shared.py         # path(), query(), body(), upload(), confirm(), ATTACHMENTS, TARGET_TYPE
auth.py            # AUTH_ACTIONS            (catalog.py:44-105)
posts.py           # POSTS_ACTIONS          (106-185, incl. view_feed)
comments.py        # COMMENTS_ACTIONS       (186-216)
projects.py        # PROJECTS_ACTIONS       (217-342)
project_files.py   # PROJECT_FILE_ACTIONS   (343-511)
jobs.py            # JOB_ACTIONS (zip+fork) (512-578)
tools.py           # TOOLS_ACTIONS          (579-679)
profile.py         # PROFILE_ACTIONS        (680-748)
messages.py        # MESSAGE_ACTIONS        (749-776)
notifications.py   # NOTIFICATION_ACTIONS   (777-810)
engagement.py      # ENGAGEMENT_ACTIONS     (811-880)
social.py          # SOCIAL_ACTIONS         (881-976, incl. leaderboard)
issues.py          # ISSUE_ACTIONS          (977-1122)
gists.py           # GIST_ACTIONS           (1123-1190)
news.py            # NEWS_ACTIONS           (1191-1211)
uploads.py         # UPLOAD_ACTIONS         (1212-1246)
admin.py           # ADMIN_ACTIONS          (1247-1661, ~415 lines; optional further sub-split)
dbapi.py           # DBAPI_ACTIONS          (1662-1778)
gateway.py         # GATEWAY_ACTIONS        (1779-1874)
game.py            # GAME_ACTIONS           (1875-2033)
```

Each domain module does `from ._shared import path, query, body, confirm, ATTACHMENTS, TARGET_TYPE`
and `from ..spec import Action`. **`dbapi.py`, `gateway.py`, `game.py` must ALSO
`from ..spec import Param`** - they construct `Param(...)` directly for boolean/integer/number
params. `admin.py` at ~415 lines may optionally be sub-split into `admin_users.py`/`admin_services.py`/
`admin_backups.py`/`admin_analytics.py`, but a single `admin.py` is already a large improvement.

**Risks:**
- **Action ordering is behavior.** `Catalog.tool_schemas()` emits schemas in `self.actions` order
  (the LLM tool-list order). The `__init__` concatenation must reproduce the exact source order in
  the table above. Verify `[a.name for a in ACTIONS]` is identical before/after (order + set).
- **Import-time singleton + validation.** `PLATFORM_CATALOG` builds at import; `registry.py:68`
  runs `_assert_confirm_params(CATALOG)` at import. `ACTIONS` must be fully assembled when the
  package `__init__` finishes. Domain modules import only from `.._shared`/`..spec` (leaf-level, no
  cycle).
- **`confirm`-param invariant.** `registry._assert_confirm_params` (`49-65`) requires every name in
  `dispatcher.CONFIRM_REQUIRED`/`CONDITIONAL_CONFIRM` to declare a `confirm` param; dropping any
  `confirm()`/inline-confirm param makes import `RuntimeError`. Watch the inline
  `body("confirm", ...)` cases at `catalog.py:310-314,336-340,727-731`.

Nothing outside the package changes; `actions/__init__.py` and `registry.py` keep their exact
imports.

---

### 3.6 `schemas.py` (1285) -> `devplacepy/schemas/`

**Nature.** Pure Pydantic `*Out` models, all subclassing one shared base `_Out`
(`schemas.py:10-11`). One import-time side effect: `CommentItemOut.model_rebuild()` at
`schemas.py:1023` (resolves the self-referential `children` forward ref). No runtime logic.
48 files import from `devplacepy.schemas` (all `from ...import X`), each router pulling the models
it feeds to `respond(..., model=XOut)`; the `__init__` must re-export ALL public model names.

**Model dependency graph (drives module order).** `UserOut` is depended on by nearly every
wrapper; `AttachmentOut/VotesOut/ReactionsOut/PollOut/BadgeOut/CommentItemOut` feed the item
wrappers; cross-module inheritance: `AdminUserOut(UserOut)`, `AdminMediaItemOut(MediaItemOut)`,
`ProjectListItemOut(ProjectOut)`, `ProjectFileRawOut(ProjectFileOut)`. So the shared primitives
live in `content.py`/`profile.py`, imported first by the domain modules.

**Target layout (`devplacepy/schemas/`):**

```
__init__.py   # explicit re-export of every *Out (+ ActionResult, ErrorOut, ValidationErrorOut)
base.py       # _Out + generic envelopes (ActionResult, ErrorOut, ValidationErrorOut)
content.py    # UserOut, AttachmentOut, VotesOut, ReactionsOut, Poll*, BadgeOut, PostOut, GistOut,
              #   ProjectOut, ProjectFile*, NewsOut, CommentOut, CommentEditOut, CommentItemOut,
              #   NotificationOut, MessageOut   (+ CommentItemOut.model_rebuild() at bottom)
listings.py   # FeedItemOut/GistItemOut/ProjectListItemOut/NewsListItemOut, Conversation,
              #   MessageItemOut, Notification*Out, Saved*, Leaderboard*, and the page-level
              #   FeedOut/PostDetailOut/ProjectsOut/ProjectDetailOut/Gists*/News*/Messages*/
              #   Notifications*/Saved*
profile.py    # MediaItemOut, ProfileOut, TelegramPairOut
issues.py     # IssueItemOut, IssueCommentOut, IssueDetailOut, IssueAttachmentsOut, IssueJobOut, IssuesOut
containers.py # InstanceOut, ScheduleOut, ContainersOut, AdminContainers*, Bot*
jobs.py       # ZipJobOut, PlanningJobOut, ForkJobOut, SeoJobOut, SeoReportOut, SeoMetaOut,
              #   DeepsearchJobOut, DeepsearchSessionOut, DbQueryJobOut
backups.py    # BackupStoragePathOut, BackupOut, BackupJobOut, BackupScheduleOut, BackupDashboardOut
admin.py      # AdminUserOut, AdminUsersOut, AdminNews*, AdminSettingsOut, AdminNotificationsOut,
              #   AdminMedia*, AdminTrashOut/TrashItemOut
gateway.py    # GatewayUsageOut, UserAiUsageOut
auth.py       # AuthPageOut, Landing*, DeviiPageOut
audit.py      # AuditLinkOut, AuditEntryOut, AuditLogOut, AuditEventOut
dbapi.py      # DbTableOut..NlQueryOut
game.py       # GameCropOut..GameLeaderboardOut
```

Every module does `from devplacepy.schemas.base import _Out` and imports the primitive models it
subclasses/references from `content.py`/`profile.py`. Use explicit re-export (not `*`) so ordering
does not matter and the surface is auditable.

**Risks:**
- **Forward-ref rebuild.** Keep `CommentItemOut` and the whole content-primitive cluster in
  `content.py` and call `CommentItemOut.model_rebuild()` at the bottom of that module (after all
  referencing types are importable).
- **Cross-module inheritance** requires the base imported before the subclass; per-file
  `from ...import` handles it since all base classes live in `content.py`/`profile.py`.
- Models reach routers only via `respond(model=...)`; a dropped export silently degrades JSON
  responses (Pydantic drops unknown keys). Re-export completeness is the whole risk.

**Test footprint.** `tests/unit/schemas.py` imports `MediaItemOut`, `AdminMediaItemOut`,
`GameLegacyOut` inside test functions - all must survive re-export.

---

### 3.7 `cli.py` (1080) -> `devplacepy/cli/`

**Nature.** Plain argparse. One `main()` builds the parser tree; each subcommand sets
`func=cmd_*`; `main()` dispatches `args.func(args)`. Console-script entry `pyproject.toml:40`
`devplace = "devplacepy.cli:main"`. Heavy imports are lazy inside each command body (containers,
jobs.queue, backup.store) - this keeps CLI import cheap and MUST be preserved.

**Target layout (`devplacepy/cli/`):**

```
__init__.py    # from .main import main; re-export EVERY cmd_* (tests call cli.cmd_role_get etc.)
_shared.py     # _audit_cli (cli.py:9-23)
roles.py       # cmd_role_get/set + register_roles(sub)        (26-61)
apikeys.py     # cmd_apikey_* + register_apikeys(sub)          (64-106)
tokens.py      # cmd_token_* + register_tokens(sub)            (109-211)
devii.py       # cmd_devii_reset_quota + register_devii(sub)   (214-254)
news.py        # cmd_news_* + register_news(sub)               (257-290)
attachments.py # cmd_attachments_prune + register(sub)         (293-315)
jobs.py        # zips/forks/seo/seo-meta/deepsearch prune-clear + _remove_*_artifacts (318-576)
backups.py     # cmd_backups_* + register_backups(sub)         (393-445)
containers.py  # cmd_containers_* + register_containers(sub)   (579-659)
migrate.py     # _crc32/_migrate_*/_db helpers + cmd_migrate_data + cmd_emoji_sync (662-870)
main.py        # build_parser()/main(): imports each module's register_* and dispatches;
               #   holds the `if __name__ == "__main__": main()` guard
```

Each leaf exposes `register_<group>(subparsers)` adding its parser(s) + `set_defaults(func=...)`.
`main.py` calls them in order. `__init__.py` re-exports `main` AND every `cmd_*` because
`tests/unit/cli.py` (`from devplacepy import cli`) calls `cli.cmd_role_get/set`, `cli.cmd_news_clear/
sanitize`, `cli.cmd_attachments_prune`, `cli.cmd_apikey_get/reset/backfill`, `cli.main` directly.

**Risks:**
- **Console-script entry** `devplacepy.cli:main` must keep resolving - satisfied by
  `from .main import main` in `__init__.py`.
- **Preserve lazy in-function imports** (avoid import-time cycles / heavy deps). Do not hoist.
- Move the `__main__` guard into `main.py` (or `__init__` calling `main()`) so `python -m devplacepy.cli`
  behavior is unchanged.

No production module imports `devplacepy.cli` (only tests). Sibling entrypoints
`devplacepy.services.devii.cli:main` and `devplacepy.services.xmlrpc.server:main` are unaffected.

---

### 3.8 `utils.py` (1024) -> `devplacepy/utils/`

**Nature.** The most import-sensitive file: 224 files import from it; source of several Jinja
globals and the notification/rewards/achievement funnels. Most-used: `make_combined_slug` (144),
`generate_uid` (127), `clear_user_cache` (32), `not_found` (22), `get_current_user` (20),
`require_admin` (19), `require_user` (16), `is_admin` (12), `track_action` (10). Several
underscore helpers are part of the effective surface (`_resolve_user`, `_user_from_session`,
`_user_from_api_key`, `_user_from_devrant_token`) and MUST be re-exported.

**Target layout (`devplacepy/utils/`):**

```
__init__.py       # explicit re-export of the FULL public API incl. imported underscore helpers
passwords.py      # hash_password/verify_password (+async), create_session               (40-70)
authcache.py      # _user_cache, _sync_auth_cache, clear_user_cache, clear_session_cache (73-90)
auth.py           # _UNSET, _user_from_*, _resolve_user, get_current_user                (92-249)
guards.py         # safe_next, redirect_back, cookie_secure, client_ip, require_user,
                  #   require_admin, require_user_api, not_found, is_admin, is_primary_admin (252-376)
text.py           # strip_html, plain_float, _mark_floats, pretty_json, slugify, generate_uid,
                  #   make_combined_slug, time_ago, format_date, extract_mentions          (379-459, 1015-1024)
notifications.py  # PUSH_ICON, DEFAULT_PUSH_URL, _push_tasks, _safe_notify, _schedule_push,
                  #   _schedule_telegram, create_notification, _deliver_notification,
                  #   create_mention_notifications, _deliver_mention_notifications         (462-575, 989-1013)
badges.py         # BADGE_CATALOG, BADGE_GROUPS, LEVEL_BADGES, get_badge, build_achievements,
                  #   award_badge, notify_badge                                            (671-745, 806-848)
rewards.py        # LEVEL_XP, XP_*, level_for_xp, award_xp, _COUNT_MILESTONES, _milestone_metrics,
                  #   check_milestone_badges, award_rewards, _apply_rewards, ACHIEVEMENTS,
                  #   UNIQUE_ACTIONS, track_action, _apply_achievements                     (654-987)
accounts.py       # _create_account, register_account, register_account_async             (578-641)
```

**Circular-import sensitivity (must preserve exactly):**
- `clear_user_cache` is lazily imported by `services/correction.py:183` and by 32 other files +
  `cli.py:74`; it MUST stay importable as `devplacepy.utils.clear_user_cache` via re-export.
- Every cross-cutting lazy import inside utils stays function-local: `is_primary_admin` ->
  `database.get_primary_admin_uid` (`328`); `require_user`/`require_admin` -> `responses.wants_json`
  + `services.audit.record` (`303-304,336-337`); `_deliver_notification` -> `templating.clear_unread_cache`
  (`531`) + `services.audit` (`557`); `award_xp`/`check_milestone_badges`/`award_badge`/`_create_account`
  -> `services.audit`; `_milestone_metrics` -> `database.get_streaks`; `_schedule_telegram` ->
  `services.telegram.store`; `_safe_notify` -> `devplacepy.push`; `_user_from_devrant_token`/
  `_user_from_access_token` -> `services.devrant.tokens`/`services.access_tokens`. Do NOT hoist any.
- **Cache-name coupling:** the auth cache uses `sync_local_cache("auth", ...)`/`bump_cache_version("auth")`
  - do not rename the `"auth"` key.

**Funnel pairing (do not split apart, do not double-wrap):** each public funnel and its
`background.submit` worker must live in the same module: `create_notification`/`_deliver_notification`
and `create_mention_notifications`/`_deliver_mention_notifications` in `notifications.py`;
`award_rewards`/`_apply_rewards` and `track_action`/`_apply_achievements` in `rewards.py`. Both
`notifications.py` and `rewards.py` need the module-level `from devplacepy.services.background import
background` import.

**Jinja globals (register-once, from `templating.py`).** `templating.py:13-15,62-63,112,132-136`
imports `format_date`, `time_ago`, `get_badge`, `is_admin`, `is_primary_admin`, `pretty_json` and
registers `is_admin`/`is_primary_admin`/`format_date` as callable globals (load-bearing per the
`respond()` collision rule). The registration stays in `templating.py`; the split must only keep
those import paths resolving via the package.

**Cache invalidation invariant.** `_create_account` calls `database.invalidate_admins_cache()` on the
first user (`640`) - keep that import in `accounts.py`.

**Test footprint.** `tests/unit/utils.py` imports `award_badge, award_xp, get_badge,
check_milestone_badges, create_mention_notifications, extract_mentions, generate_uid, hash_password,
is_primary_admin, level_for_xp, safe_next, slugify, strip_html, time_ago, verify_password` - every one
must survive re-export. After the split, diff `dir(devplacepy.utils)` public names against the
pre-split snapshot.

---

### 3.9 `services/news.py` (986) -> `devplacepy/services/news/`

**Nature.** `NewsService(BaseService)` plus a set of pure free functions and constants for the
import pipeline. `main.py:88,236` imports and registers `NewsService`. The CLI does NOT import this
module (its `news clear`/`news sanitize` operate on raw tables via dataset + `utils.strip_html`).

**Target layout (`devplacepy/services/news/`):**

```
__init__.py    # re-exports the full public + test surface; also re-binds `net_guard` and `httpx`
constants.py   # all tuning constants + JUNK_PATTERNS + WHITESPACE_PATTERN + prose blocks
               #   (NEWS_SUMMARY, GRADE_PROMPT_SPEC, FORMAT_PROMPT_SPEC, GRADING_RULES_DESCRIPTION);
               #   numeric constants MUST precede GRADING_RULES_DESCRIPTION (it is an f-string over them)
models.py      # ImageCandidate, ArticleGrade
clean.py       # clean_news_text, _caps_ratio, _is_valid_url, reliability_reason, effective_score,
               #   _extract_grade, _strip_md_fence
images.py      # _decode_phash, _get_article_images, _fetch_image_candidate,
               #   _flag_shared_placeholders, _primary_image
service.py     # NewsService (run_once/collect_metrics/_store_*/_apply_landing_selection/
               #   _grade_article/_format_article/_grade_article_full/_resolve_uid) + _get_ai_key
```

`__init__.py` re-export must include at minimum: `NewsService`, `ImageCandidate`, `clean_news_text`,
`reliability_reason`, `effective_score`, `_extract_grade`, `_get_ai_key`, `_get_article_images`,
`_flag_shared_placeholders`, `_primary_image`, `_strip_md_fence`, and constants `LANDING_MIN_SCORE`,
`LANDING_MAX`, `UNIQUE_IMAGE_BONUS`, `THIN_CONTENT_PENALTY`; plus re-bind `net_guard` and `httpx` as
package attributes.

**Load-bearing risk (dictates where `_get_ai_key` lives).** `tests/unit/services/news.py`
monkeypatches names in the news-module namespace: `monkeypatch.setattr(news_mod, "get_setting", ...)`
(x10) where `news_mod = devplacepy.services.news`, and `setattr(news_mod.net_guard, "guard_public_url",
...)`, and reads `news_mod.httpx`. `_get_ai_key` resolves `get_setting` from ITS OWN module globals. To
keep these tests passing with NO test edits, keep `_get_ai_key` and its `from ...database import
get_setting` in the SAME module the test patches - i.e. keep them in `service.py` and ensure the test's
`monkeypatch.setattr(news_mod, "get_setting", ...)` still targets the name `_get_ai_key` reads. The safe
form: keep `_get_ai_key` + the `get_setting` import together in `service.py`, and have `__init__` do
`from .service import get_setting` re-binding so the package-namespace patch and the resolving namespace
are the same object. Simplest conservative variant: peel off only the genuinely pure modules
(`constants.py`, `models.py`, `clean.py`, `images.py`) and keep `NewsService` + `_get_ai_key` +
`get_setting`/`net_guard`/`httpx` imports in `service.py` (which `__init__` re-exports wholesale). This
removes most of the size while eliminating the namespace hazard.

**Other risks:**
- `GRADING_RULES_DESCRIPTION` f-string ordering (constants first).
- `_decode_phash` runs via `run_in_executor` - pure, safe to relocate to `images.py`.
- BaseService contract: `run_once`/`collect_metrics`/`config_fields`/`interval_key`/`default_enabled`/
  `__init__(name="news", interval_seconds=3600)` stay on `NewsService`. Methods using `self.log`/
  `self.get_config` cannot become free functions.
- `_grade_article_full` is pure but called as `self._grade_article_full(...)`; leave it a method
  (lowest risk).

---

### 3.10 `services/devii/session.py` (962) -> same-name package `session/` (extract, do not decompose)

**Nature.** ONE heavily-shared-mutable-state class `DeviiSession` (`73-851`) plus ~150 lines of
stateless constants/pure helpers. Constructed by `hub.py:70-85` with an exact 11-positional +
3-keyword arg signature. `_parse_topic` is imported by `tests/unit/services/devii/session.py:6`.

**Recommendation: extract the stateless text/helpers into a package, keep `DeviiSession` whole.**
A true class decomposition would require threading `_lock`/`_send_lock`/`_conns`/`_conn_meta`/
`agent._messages`/`self.tools`/`_pending`/`_buffer`/`_turn_epoch` across objects and risks the
`self.tools[:]` aliasing and `_turn_epoch` interrupt guard. Not worth it.

**Target layout (`services/devii/session/`):**

```
__init__.py   # from .core import DeviiSession; from .prompts import _parse_topic (+ optional constants)
prompts.py    # DOCS_SYSTEM_PROMPT, DOCS_GREETING, DOCS_TOOLS, TOPIC_CLASSIFIER_PROMPT,
              #   NON_ADMIN_COST_RULE, LOGIN_REQUEST, BEHAVIOR_HEADER, _parse_topic(), _system_prompt_for()
              #   (session.py:854-947 + 40 + 67)
_helpers.py   # _repair_history(), _now_iso(), _format_offset()   (43-65, 950-962)
core.py       # class DeviiSession + the numeric constants only it uses (27-31, 39)
```

This lifts ~150 lines out of the class file with zero behavior risk. `__init__.py` re-exports
`DeviiSession` (used by `hub.py:15`) and `_parse_topic` (test). `prompts.py`/`_helpers.py` import
nothing from `core.py` (no cycle). All prompt constants are used only inside the class today, so they
move freely.

Optional (secondary, only if further reduction is wanted): a mixin split of `DeviiSession` into
`ConnectionMixin`/`PromptComposition`/`TurnControl`/`SpendTracking`/`Scheduling` operating on one
shared `self`. Given the density of cross-group `self.` references (e.g. `_run_turn` touches
`_refresh_tools`, `_refresh_system_prompt`, `_docs_topic_gate`, `_cost_snapshot`, `_record_spend`,
`_persist_history`, `_emit`, `_turn_epoch`), this trades one long file for cross-mixin coupling - do it
only if the primary extraction is not enough.

**Invariants that must not break (whichever depth is chosen):**
- **`self.tools` mutated in place, never reassigned.** `_refresh_tools` uses `self.tools[:] = ...`
  because the same list object is bound into `AgenticController`/`Agent`/`react_loop`. Reassigning
  freezes the tool list (virtual tools stop taking effect). Same in-place discipline for
  `self.agent._messages` rewrites (`_refresh_system_prompt` `544-547`, `reset_conversation` `357-375`,
  `_docs_topic_gate` `549-577`, `stop`/`_repair_history` `408-426`).
- **Turn-epoch interrupt guard** (`_turn_epoch` at `184`, incremented in `cancel_turns:384`, checked
  in `_run_turn:451,459,463`) - keep `spawn_turn`/`cancel_turns`/`_run_turn` with the shared
  `_turn_epoch`/`_turns` state (MEMORY: devii-browser-and-interrupt; do not regress).
- **`__init__` positional-arg contract** with `hub.py:70-85` must be preserved exactly.
- `session.py` imports `CATALOG` from `.registry` (import-time `_assert_confirm_params`); the package
  rename must not introduce a cycle - `prompts.py`/`_helpers.py` import nothing from `core.py`.
- `from __future__ import annotations` in every extracted module (annotations use `X | None`).

---

### 3.11 `services/game/store.py` (948) -> subpackage `services/game/store/`

**Nature.** The DB/stateful layer of the already-package `services/game/` (siblings: `economy.py`
pure rules - untouched; `__init__.py` does `from . import economy, store; from .store import GameError`).
Only `store.py` needs splitting. Every function funnels through four `get_table` accessors
(`_farms/_plots/_quests/_steals`) and `_update_farm`; the only module-level state is `PERK_COLUMN`
(built from `economy.PERKS` at import). The game tables carry no `deleted_at` (outside
`SOFT_DELETE_TABLES`) - no soft-delete concerns.

**Target layout (convert `store.py` -> `store/` package, keeping path `services.game.store`):**

```
store/
  __init__.py   # re-exports EVERY name currently on store.py, incl. private helpers tests call
  common.py     # GameError, _now, _iso, _parse, _today, _parse_date, _farms, _plots, _quests,
                #   _steals, _lvl, PERK_COLUMN, _update_farm, last_steal_at, steal_cooldown_remaining
  farm.py       # get_farm, ensure_farm, _create_plot, get_plots, _plot_at, buy_plot, upgrade_ci, leaderboard
  serialize.py  # _watered_by, _plot_state, serialize_plot, serialize_farm, _serialize_perks,
                #   _serialize_legacy, _serialize_quests, _daily_available
  quests.py     # ensure_quests, advance_quests
  actions.py    # plant, harvest, water, steal, fertilize, prestige, upgrade_perk, upgrade_legacy,
                #   claim_daily, claim_quest, _auto_harvest
```

`__init__.py` re-export must include the router surface (`ensure_farm`, `serialize_farm`, `plant`,
`harvest`, `buy_plot`, `upgrade_ci`, `water`, `steal`, `leaderboard`, `claim_daily`, `claim_quest`,
`upgrade_perk`, `upgrade_legacy`, `prestige`, `fertilize`, `GameError`) **plus the private symbols the
unit test calls directly**: `get_farm`, `get_plots`, `ensure_quests`, `advance_quests`, `_plot_at`,
`_today`. `services/game/__init__.py` keeps working unchanged.

Importers: `routers/game/index.py:18`, `routers/game/farm.py:11`, `routers/game/_shared.py:6` (all
`from devplacepy.services.game import GameError, store` then `store.X`). No CLI, no `main.py`.

**Risks:**
- **Re-export private helpers** (`_plot_at`, `_today`, + `get_farm`/`get_plots`/`ensure_quests`/
  `advance_quests`) or the unit test breaks. No monkeypatch-of-module-name coupling here (unlike news),
  so the only hazard is a missing re-export.
- **`PERK_COLUMN`** built from `economy.PERKS` at import must live in the low-level shared module
  (`common.py`) imported by `serialize.py`/`actions.py`; `common.py` imports `economy`, higher modules
  import `common` + `economy` (no cycle).
- **Dense intra-module call graph** (`serialize_farm -> _auto_harvest -> advance_quests`;
  `actions -> ensure_farm/_update_farm/advance_quests`) - group so lower-level modules
  (`common`/`quests`/`farm`) are imported by higher-level (`serialize`/`actions`), not vice-versa.
- **`leaderboard` lazy import** `from devplacepy.database import get_users_by_uids` at `store.py:593` -
  keep function-local when relocating.

---

## 4. SPECIAL CASE - container standalone scripts (files 4 and 5)

`services/containers/files/bot.py` (2940, `COPY bot.py /usr/bin/botje.py`) and
`services/containers/files/d.py` (2337, `COPY d.py /usr/bin/d.py`) are NOT server backend. They are
standalone single-file programs baked into the `ppy` image (`ppy.Dockerfile:47-48`, build context
`devplacepy/services/containers/files`, invoked `python /usr/bin/botje.py`). Their only hard
third-party dep is `httpx`. **Nothing in `devplacepy/` or `tests/` imports either file** (confirmed
grep) - they are not on `sys.path` as modules and have no test coverage. The only breakage surface is
the Docker build's COPY path and the documented invocation.

**Binding constraint.** A normal package split (like the server-side files) is **incompatible** with
the single-file COPY + bare-`python` exec contract: a package leaf COPY'd to `/usr/bin/botje.py` would
fail at the first cross-module `from botpkg... import` (no package root on the container's path). So the
in-place package approach used for every other monster does not apply here.

**Options (pick per user direction; this plan does not choose for you):**

- **(a) Keep as single-file monoliths.** Zero Dockerfile/Makefile change; the contract stays intact.
  Trade-off: the two files remain large and duplicate ~2000 lines of agent code between them
  (`files/bot.py` = `files/d.py`'s agent + a DevPlace XML-RPC bot loop; `files/d.py`'s stealth half is
  a pure-httpx sibling of `devplacepy/stealth.py`). This is the lowest-risk option.

- **(b) Split into a source package + add a build step** that re-stitches the single deployed artifact.
  Two sub-options, each with concrete required edits:
  - **zipapp:** author `files/agent/` as a package, build with
    `python -m zipapp files/agent -m "agent.cli:main" -o build/botje.pyz`. Requires changing
    `ppy.Dockerfile:47-48` COPY source + extension (`.pyz`), the `chmod`/`ln` lines (`:52`), the
    invocation in docs (`templates/docs/getting-started-vibing.html:128-162`) from `python /usr/bin/
    botje.py` to `python /usr/bin/botje.pyz`, and the `make ppy` target (`Makefile:129-130`) to run the
    bundler before `docker build`. Downside: breaks the documented `cp /usr/bin/botje.py /app/mybot.py`
    copy-and-edit single-file flow.
  - **concatenation/stitch:** a `make` recipe concatenates the package modules (deduped imports, in
    dependency order) into one `build/bot.py`, which the Dockerfile then COPYs. Keeps the `.py`
    extension and `python /usr/bin/botje.py` intact; users edit generated output, not source. Requires
    `Makefile:129` (add the bundler step + widen the rebuild-trigger set) and `ppy.Dockerfile:47-48`
    (COPY from `build/`).

**Note (not a recommendation, for context):** the largest issue with these two files is duplication,
not size - `files/bot.py` and `files/d.py` share the entire agent (tool framework, `react_loop`,
`CorpusIndex`, embedded `ChromeStealthClient`). If the user later wants these deduplicated rather than
split, that is a different task; this plan records the fact only.

---

## 5. Cross-file execution order (recommended)

Order by blast radius (lowest first) so a mistake surfaces on a small surface before the wide ones,
and so the widely-imported files are split last against a known-green tree:

1. **`services/game/store.py`** - self-contained subpackage, 3 importers, no cycles. Warm-up.
2. **`services/devii/actions/catalog.py`** - pure data, 2 exported symbols, ordering-only risk.
3. **`schemas.py`** - pure models, mechanical, wide but simple re-export.
4. **`services/news.py`** - watch the monkeypatch namespace; conservative variant recommended.
5. **`services/devii/session.py`** - extract stateless helpers only (keep class whole).
6. **`services/bot/bot.py`** - mixin lift; keep `__init__` exporting only `BotsService`.
7. **`cli.py`** - package with per-group modules; keep the console-script entry.
8. **`docs_api.py`** - large but declarative; handle the three ordering hazards.
9. **`utils.py`** - 224 importers, funnel + lazy-import + Jinja-global sensitivity.
10. **`database.py`** - 329 importers, the core layer; do it last against a green tree.
11. **Container `files/bot.py` / `files/d.py`** - only if the user opts into option (b); otherwise
    leave as-is per Section 4.

Each step is independent: split one file, run the Section-2.9 validation, commit, move on. Do not batch
multiple monster splits into one unvalidated change.

---

## 6. Convention alignment (why packages, not `_`-suffixed files)

This plan uses the project's existing structural idiom, verified against the current tree:
- Router packages (`routers/issues/`, `routers/admin/`, `routers/profile/`, `routers/projects/`) are
  directories whose `__init__.py` aggregates leaf modules and re-exports the public symbol, each new
  file headed `# retoor <retoor@molodetz.nl>`. Every split above mirrors that.
- `_shared.py` is the sanctioned package-private helper module (already used in eight router packages);
  it is used here only for genuine cross-leaf construction helpers (`docs_api/_shared.py`,
  `catalog/_shared.py`, `cli/_shared.py`), never as a dumping ground.
- No `_v2`/`_refactored`/`_new` suffixes anywhere (forbidden by project rules). The compatibility
  mechanism is the same-name package + re-export, not a parallel file.
- `services/` already contains packages that expose a `store` submodule (`services/audit/`,
  `services/backup/`, `services/containers/`, `services/game/`), so `services/news/` and
  `services/game/store/` and `services/devii/session/` and `services/devii/actions/catalog/` all follow
  precedent.

---

## 7. Documentation follow-up (after any split is executed)

Per the project feature workflow, once a split is executed the structural change is documented in the
three canonical files - but ONLY if a rule/convention changes. A pure module move that preserves every
import path introduces no new architectural rule, so:
- `README.md`: no change (no user-visible/route/env/dep change).
- `AGENTS.md`: add a short note under the relevant domain only if the new module boundary is worth
  navigational guidance (e.g. "docs_api is now a package, one module per API group").
- `CLAUDE.md`: no change unless the split establishes a NEW convention (it does not - it follows the
  existing package idiom).

This section is a reminder, not a task; the plan itself does not edit docs.

---

## 8. Summary table - target packages

| Source file | Target | Split style | Key invariant to protect |
|-------------|--------|-------------|--------------------------|
| `docs_api.py` | `docs_api/` (+`groups/`) | one module per API group | import-time negotiation pass; group order; `_substitute` re-export |
| `database.py` | `database/` | layered domain modules on `core.py` | `db` singleton in `core`; underscore re-exports; keep lazy imports |
| `services/bot/bot.py` | mixins in `services/bot/` | cooperative mixins | realism contract; `__init__` exports only `BotsService`; stealth rule |
| `services/devii/actions/catalog.py` | `catalog/` (+`_shared.py`) | one module per action domain | `ACTIONS` order; `confirm`-param invariant; `Param` import in dbapi/gateway/game |
| `schemas.py` | `schemas/` | domain schema modules | `CommentItemOut.model_rebuild()`; cross-module inheritance order |
| `cli.py` | `cli/` (+`_shared.py`) | one module per command group | console-script `cli:main`; re-export all `cmd_*`; lazy imports |
| `utils.py` | `utils/` | layered helper modules | funnel pairing; lazy imports; Jinja-global paths; `"auth"` cache key |
| `services/news.py` | `services/news/` | pipeline-stage modules | monkeypatch namespace (`_get_ai_key`/`get_setting`); f-string const order |
| `services/devii/session.py` | `session/` | extract stateless text/helpers | `self.tools[:]` in-place; `_turn_epoch` guard; `__init__` arg contract |
| `services/game/store.py` | `services/game/store/` | cohesive store modules | re-export private test helpers; `PERK_COLUMN` placement |
| `services/containers/files/bot.py` | SPECIAL (Section 4) | single-file OR build step | COPY-single-file + bare-exec contract |
| `services/containers/files/d.py` | SPECIAL (Section 4) | single-file OR build step | COPY-single-file + bare-exec contract |
