diff --git a/ref.md b/ref.md new file mode 100644 index 00000000..df0ec510 --- /dev/null +++ b/ref.md @@ -0,0 +1,875 @@ +# ref.md - Monster-file split plan (backend) + +Author: retoor + +This document is the execution plan for splitting every backend "monster" file in +DevPlace into a package of cohesive modules, without changing behavior and without +breaking any importer or test. It is investigation output only; nothing here has +been executed. Each section is self-contained and can be executed independently in +the recommended order (Section 12). + +--- + +## 1. Scope and definition + +"Monster" = first-party backend Python file over 900 lines (excludes vendored JS, +generated artifacts, and the scratch/workspace copies under `data/`, `var/`, `tmp/`, +and the repo-root `maak.py`/`d.py`). The confirmed set: + +| # | File | Lines | Split kind | +|---|------|------:|------------| +| 1 | `devplacepy/docs_api.py` | 5659 | package: one module per API group | +| 2 | `devplacepy/database.py` | 3764 | package: layered domain modules | +| 3 | `devplacepy/services/bot/bot.py` | 2999 | in-package: cooperative mixins | +| 4 | `devplacepy/services/containers/files/bot.py` | 2940 | SPECIAL - deployed single-file asset | +| 5 | `devplacepy/services/containers/files/d.py` | 2337 | SPECIAL - deployed single-file asset | +| 6 | `devplacepy/services/devii/actions/catalog.py` | 2036 | same-name package: one module per action domain | +| 7 | `devplacepy/schemas.py` | 1285 | package: domain schema modules | +| 8 | `devplacepy/cli.py` | 1080 | package: one module per command group | +| 9 | `devplacepy/utils.py` | 1024 | package: layered helper modules | +| 10 | `devplacepy/services/news.py` | 986 | package: pipeline-stage modules | +| 11 | `devplacepy/services/devii/session.py` | 962 | same-name package: extract stateless text/helpers | +| 12 | `devplacepy/services/game/store.py` | 948 | subpackage: cohesive store modules | + +`services/bot/llm.py` (792) is below threshold and holds two realism mechanics; it is +explicitly **left intact** (Section 3.4). + +--- + +## 2. Global rules for every split (apply to all sections) + +These are invariants. Every section assumes them; deviations are called out per file. + +1. **Same import path.** Convert `foo.py` into a package `foo/` (directory with + `__init__.py`). Because Python resolves `foo/__init__.py` for `import ...foo`, every + existing `from ...foo import X` and `import ...foo as m; m.X` keeps resolving with no + caller edit. This is the ONLY compatibility mechanism used. + +2. **Exhaustive re-export in `__init__.py`.** The package `__init__` must re-export the + complete public surface the codebase and tests currently import - **including + underscore-prefixed names that are imported elsewhere** (they are part of the effective + surface; e.g. `database._now_iso`, `utils._resolve_user`, `docs_api._substitute`, + `store._today`). Prefer **explicit** `from .module import name1, name2` blocks over + `from .module import *` so the surface is auditable and star-import name-drops + (underscore filtering, `__all__` gaps) cannot silently regress it. + +3. **No behavior change.** Pure structural moves. No logic edits, no reordering of + ordered data, no renames of public symbols, no signature changes. + +4. **Preserve every function-local (lazy) import.** Many files defer imports inside + function bodies specifically to break import cycles (`database` <-> `utils`, + `utils` <-> `services.audit`/`templating`/`content`, `news` AI-key, game `leaderboard`). + These MUST stay function-local in the new modules. Hoisting a lazy import to module + top is the single most likely way this refactor introduces a boot-time cycle. + +5. **File header.** Every new file starts on line 1 with `# retoor ` + (project rule). Carry `from __future__ import annotations` into any module that uses + `X | None` style annotations where the original had it. + +6. **No em-dash.** Neither the em-dash character nor `—`/`—`/`—` + anywhere in new files. Hyphen only. Grep every new file before finishing. + +7. **No forbidden names / no comments / no docstrings in source** (project rules): do not + introduce `_new`/`_old`/`_temp`/`_v2`/`better_`/`my_`/`the_`, do not add comments or + docstrings. + +8. **`_shared.py` is the sanctioned package-private helper module.** The project already + uses `_shared.py` in eight router packages for cross-leaf helpers. Use that exact name + for genuinely package-private construction helpers (docs_api builders, catalog builders, + cli `_audit_cli`). Everything else is a normal named module. + +9. **Validation after each split (never run the test suite):** + - `python -c "from devplacepy.main import app"` must import clean. + - `python -m py_compile` every new file. + - A public-surface diff: capture `sorted(n for n in dir() if not n.startswith('__'))` + BEFORE the split (from git or a saved snapshot) and assert the post-split package + exposes a superset. This is the definitive check that no re-export was missed. + - em-dash grep over every new file. + - The suite is run only if the user explicitly asks; the plan never runs it. + +--- + +## 3. Per-file plans + +### 3.1 `docs_api.py` (5659) -> `devplacepy/docs_api/` + +**Nature.** A flat declarative module: a few construction helpers, one giant +`API_GROUPS` list literal (~5240 lines, `docs_api.py:143-5384`, 19 group dicts), then +a post-processing/registry/render tail. Endpoints are organized by API group; each group +is a self-contained dict literal built via the `endpoint()` factory. + +**Public surface (only these are imported externally):** +`API_GROUPS`, `build_services_group`, `get_group`, `api_doc_pages`, `render_group`, and +`_substitute` (underscore but called by `docs_export.py:119`). + +Importers: `docs_search.py:9,87,90`; `docs_export.py:10,118,119,121`; +`services/xmlrpc/registry.py:8,63`; `routers/docs/pages.py:3`; `routers/docs/views.py:14`. +Tests: `tests/e2e/docs.py:9`, `tests/unit/project_files.py:381-385`, +`tests/api/auth/matrix.py:7`, `tests/api/docs/{index,downloadhtml,downloadmd}.py`. + +**Target layout:** + +``` +devplacepy/docs_api/ + __init__.py # assembles API_GROUPS in order, runs the negotiation pass, + # re-exports the public surface (incl. _substitute) + _shared.py # field, endpoint, ROLE_LABELS, NON_BODY_ENDPOINTS, and the + # domain constant lists (VOTE_TARGETS, REACTION_TARGETS, + # BOOKMARK_TARGETS, COMMENT_TARGETS, PROJECT_TYPES, + # GIST_LANGUAGES, SERVICE_ACTIONS) + negotiation.py # _PAGE_RESPONSES, _ACTION_RESPONSES, _classify, + # _apply_negotiated_responses(groups) (schemas/docs_examples live here) + services_group.py # _service_control_endpoints, _field_to_param, _service_section, + # build_services_group + render.py # _GROUPS_BY_SLUG, get_group, api_doc_pages, _substitute, render_group + groups/ + __init__.py # ORDERED_GROUPS = [conventions.GROUP, auth.GROUP, ... game.GROUP] + conventions.py # docs_api.py:145-293 + auth.py # 295-427 + lookups.py # 428-484 + social_actions.py # 485-674 + content.py # 675-1615 (largest group, ~940 lines) + profiles.py # 1616-2294 + messaging.py # 2295-2365 + notifications.py # 2366-2450 + uploads.py # 2451-2552 + project_files.py # 2553-3098 + containers.py # 3099-3673 (admin: True) + tools.py # 3674-3902 + push.py # 3903-3964 + issues.py # 3965-4258 + gateway.py # 4259-4449 (admin: True) + services.py # 4450-4456 (dynamic placeholder group dict - keep hollow) + admin.py # 4458-5193 (admin: True, ~735 lines) + game.py # 5194-5383 +``` + +Each `groups/*.py` holds one `GROUP = {...}` literal importing `field`, `endpoint`, +`ROLE_LABELS` and the constant lists from `.._shared` (and `constants.TOPICS`/ +`REACTION_EMOJI` where the literal uses them - re-scan each block for free names before +cutting). + +**`__init__.py` must, in this exact order:** +1. `from .groups import ORDERED_GROUPS as API_GROUPS` (order is load-bearing). +2. `from .negotiation import _apply_negotiated_responses` then call + `_apply_negotiated_responses(API_GROUPS)` - reproduces the current import-time mutation + at `docs_api.py:5494`. Must run AFTER `API_GROUPS` is assembled. +3. Re-export `API_GROUPS`, `build_services_group`, `get_group`, `api_doc_pages`, + `render_group`, `_substitute` (and, for safety, `endpoint`, `field`, `ROLE_LABELS`, + the constant lists). + +**Risks / ordering hazards (must be handled):** +- **Import-time negotiation mutation.** `_apply_negotiated_responses()` runs at module top + today (`docs_api.py:5494`) and mutates each endpoint dict in place (fills + `sample_response`, stamps `negotiation`). Centralize this in `__init__` after assembly; + if a group is read before the pass runs, endpoints lack `negotiation`/`sample_response`. +- **Forward reference.** `_classify` (`105-112`) reads `_PAGE_RESPONSES`/`_ACTION_RESPONSES` + defined 5280 lines later. Colocate `_classify` + both maps + the pass in `negotiation.py` + to make the dependency explicit. +- **Assembly-before-render cycle.** `render.py` (`get_group`/`render_group`/`api_doc_pages`/ + `_GROUPS_BY_SLUG`) needs the final `API_GROUPS`. Build `API_GROUPS` in `groups/__init__.py` + and import it into both package `__init__` and `render.py`; do NOT let `render.py` import + the package `__init__` at module top (cycle). +- **`build_services_group` reuses core helpers** (`endpoint`, `field` via `_field_to_param`, + `_classify`, `_service_control_endpoints`) and stamps `negotiation` itself for its runtime + group; `services_group.py` imports `endpoint`/`field` from `_shared` and `_classify` from + `negotiation`; do NOT double-apply negotiation to it. +- **`_substitute` name must be preserved** in the re-export (called as + `docs_api._substitute` by `docs_export.py:119`). +- **Two groups slug `services`** (the hollow placeholder in the list + the runtime + `build_services_group` output). Keep the placeholder exactly where it is in the ordered list. + +**Test invariant:** `API_GROUPS` order/content, group flags (`slug/title/admin/dynamic/ +endpoints`), and every endpoint's post-negotiation shape (`id/method/path/auth/min_role/ +params/negotiation/sample_response`) must be identical after the split. `tests/api/auth/ +matrix.py` asserts `len(endpoints) >= 50` and drives each endpoint's documented auth against +live behavior; `tests/unit/project_files.py:381-385` asserts the `project-files` group's exact +endpoint id set. + +--- + +### 3.2 `database.py` (3764) -> `devplacepy/database/` + +**Nature.** The core data layer, imported nearly everywhere (677 `from ...database import` +statements across 329 files; hottest names `get_table` x404, `refresh_snapshot` x92, +`set_setting` x67, `db` x33). The `__init__` MUST re-export essentially the entire namespace +including underscore helpers (`_now_iso` x6, `_index`, and whatever tests reach for). + +**The linchpin.** A single `core.py` owns the `db` singleton, the import-time connection +side effects, the cross-worker cache-version primitives, index helpers, and trivial +accessors. Every other module imports only from `core` (and strictly-lower layers). Because +`core.py` is imported once, Python's module cache guarantees the single `db` object, exactly +as today. + +**Target layout (`devplacepy/database/`):** + +``` +core.py # db, refresh_snapshot; import-time ensure_data_dirs + sqlite mkdir + + # dataset.connect (database.py:20-42); cache-version primitives + # (_ensure_cache_state, get_cache_version, bump_cache_version, + # sync_local_cache, _local_cache_versions, _cache_version_cache, + # _cache_state_ready); _index, _drop_index, _uid_index; + # get_table, _in_clause, _now_iso <- imported by ALL +settings.py # _settings_cache, get_setting, get_int_setting, set_setting, + # clear_settings_cache, internal_gateway_key +users.py # get_users_by_uids, search_users_by_username, set_user_timezone, + # admins cache (_admins_cache, invalidate_admins_cache, + # get_admin_uids, get_primary_admin_uid) +relations.py # user block/mute cache + accessors (get_user_relations, get_blocked_uids, + # get_muted_uids, get_silenced_uids, invalidate_user_relations) +pagination.py # PAGE_SIZE, build_pagination, paginate, paginate_diverse, + # interleave_by_author, get_user_post_count (imports core + relations) +soft_delete.py # SOFT_DELETE_TABLES, ensure_soft_delete_columns, soft_delete, + # soft_delete_in, restore, purge, list_deleted, count_deleted, + # restore_event, purge_event (imports core + pagination.build_pagination) +engagement.py # comment-count/vote/reaction/bookmark batch helpers, polls +comments.py # _drop_blocked, _build_comment_items, load_comments, recent/by-target loaders +attachments_data.py# get_attachments*, get_news_images_by_uids, delete_attachment*, + # _delete_attachment_file, get_user_media, get_deleted_media +usage.py # _add_usage, _get_usage, add/get_*_usage + *_USAGE_KEY constants +seo_meta.py # SEO_META_TYPES + seo_metadata CRUD +activity.py # user_activity counters + calendar/heatmap/streaks +ranking.py # VOTABLE_TARGETS, STAR_TARGETS, authors/leaderboard/stars, + # update_target_stars, get_target_owner_uid, soft_delete_engagement, + # delete_engagement +customization.py # customization prefs + override CRUD +email.py # email account CRUD +notifications.py # notification prefs + mark_notifications_read_by_target +forks.py # record_fork, get_fork_parent, count_forks, *_fork_relations +content.py # resolve_by_slug, resolve_object_url, text_search_clause, + # get_uids_by_username_match, get_daily_topic, get_featured_news +follows.py # get_follow_counts, get_follow_list, get_following_among +stats.py # get_site_stats, get_platform_analytics, get_gist_languages +deepsearch.py # _ds_now + deepsearch session/message/url-cache helpers +schema.py # init_db, _refresh_query_planner_stats, migrate_bug_tables_to_issue_tables, + # BUG_TABLE_RENAMES, migrate_ai_gateway_settings, OLD_GATEWAY_URL, + # backfill_api_keys, _backfill_gamification <- imports EVERYTHING; loaded last +__init__.py # explicit re-export of every module's public + test-touched underscore names +``` + +**Where shared state lives (critical).** The only truly global object is `db` (+ the +cache-version machinery), which lives in `core.py`. Every domain `TTLCache`/dict singleton +(`_admins_cache`, `_settings_cache`, `_relations_cache`, `_customizations_cache`, +`_notification_prefs_cache`, `_stats_cache`, `_analytics_cache`, `_activity_cache`, +`_gist_languages_cache`, `_authors_cache`, `_comment_count_cache`) is referenced only by its +own domain, so it moves WITH that domain's module - no shared-cache problem. Never re-create +a cache in `__init__`; only re-export. Two import paths to one cache would create two caches. + +**Layering (acyclic).** `core` <- everything; `pagination -> relations`; +`soft_delete -> pagination`; `comments -> users + engagement + relations`; +`ranking -> users`; `stats -> ranking`; `attachments_data -> content`; `content -> users`; +`schema` imports any sibling at module top (nothing imports `schema`; loaded last in `__init__`). + +**Cycle-avoidance rules:** +- `core.py` imports nothing from siblings. +- Strict layer order: `core -> settings/users/relations -> pagination -> soft_delete -> + domain modules -> schema`. +- **Preserve every existing function-local import verbatim** - the `utils`, `attachments`, + `templating`, `services.audit.store`/`services.backup.store`/`services.openai_gateway.routing` + imports inside functions (e.g. `init_db` at `database.py:951-961`) exist to break the real + `database <-> utils` cycle. Do not hoist them. +- `utils.py` uses `database`'s cache primitive under the `"auth"` name; those primitives are in + `core.py` and remain reachable as `database.bump_cache_version`/`database.sync_local_cache` + via re-export. + +**Risks:** +- **`init_db()` (503-1534) is one idempotent ordered routine** - move it WHOLE into `schema.py`; + do not reorder its internal steps (migration-before-index, backfills/ANALYZE last at 1516-1519). +- **Import-time side effects (20-42)** must live in `core.py` and run exactly once; `core` must + be the first thing `__init__` imports. +- **Underscore re-export gap** is the highest-probability regression (`import *` drops `_now_iso`/ + `_index`). Use explicit re-exports or per-module `__all__` including the imported underscores. +- **`get_table` (404 importers) and `refresh_snapshot` (92)** - any miss breaks a huge fraction of + the app at import; the public-surface diff (Section 2.9) is the guard. + +**Test footprint.** `tests/unit/database.py` is the primary direct consumer (it also hits a live +server, so it is effectively api-tier); `tests/unit/services/gitea.py` and +`tests/unit/services/openai_gateway/analytics.py` import it for setup; dozens of api/e2e files use +`get_table`/`set_setting`/usage helpers indirectly. With a complete re-export, zero test edits are +needed. + +--- + +### 3.3 `services/bot/bot.py` (2999) -> cooperative mixins in `services/bot/` + +**Nature.** ONE class `DevPlaceBot` (`bot.py:48`), ~90 methods, every method reads/writes +shared `self` state (`self.state` persisted `BotState`, `self.b` browser, `self.llm`, +`self._session_*` counters, the `_post_cache/_project_cache/_issue_cache/_session_history` +caches). The only zero-behavior-change decomposition is **cooperative mixins** (Python MRO gives +byte-identical `self` semantics; the unbound-method unit test still resolves via inheritance). +There is NO `@tool` registry in this file (the closest analogue is the AI decision/menu engine). + +**Target layout (new modules beside the existing `services/bot/` files):** + +| New module | Mixin | Moves from bot.py | Contents | +|------------|-------|-------------------|----------| +| `helpers.py` | `BotHelpersMixin` | 103-245 | instrumentation, cost/session banners, `_thread_id`, `_pick_article`, cache warm | +| `auth.py` | `BotAuthMixin` | 246-355 | handles, `_ensure_auth`, account-key adoption | +| `human.py` | `BotHumanMixin` | 356-522 | read/type/fatigue simulation, session type, page inspection | +| `agent.py` | `BotAgentMixin` | 523-757 | AI decision/menu/plan engine (`_build_menu`, `_run_plan`, `_cycle_ai`, dispatch, nav) | +| `engage.py` | `BotEngageMixin` | 753-1436 | voting, commenting, reacting (realism #4b) | +| `posting.py` | `BotPostingMixin` | 1438-1770 | gist + post creation, content open/click (realism #1, #3 call-sites) | +| `social.py` | `BotSocialMixin` | 1771-2378 | profile/project/issue/follow, notifications, mentions (#4b), messaging | +| `browse.py` | `BotBrowseMixin` | 2379-2502 | scroll, nav, browse, search, `_engage_community` | +| `loop.py` | `BotLoopMixin` | 2503-2999 | `_cycle`, `run_forever` (scheduling) | + +`bot.py` shrinks to ~110 lines: the mixin imports, `__init__`, and: + +```python +class DevPlaceBot( + BotHelpersMixin, BotAuthMixin, BotHumanMixin, BotAgentMixin, + BotEngageMixin, BotPostingMixin, BotSocialMixin, BotBrowseMixin, BotLoopMixin, +): + ... +``` + +`DevPlaceBot` stays importable from `devplacepy.services.bot.bot` (do NOT move the class file; +only lift methods into inherited mixins). Method names are already globally unique across the +90 methods - verify no collision when composing. + +**Realism contract (MUST NOT regress) - exact locations to preserve:** +1. Title rewrite: `LLMClient.generate_post`/`generate_post_title` (in `llm.py`, untouched); + call-sites `bot.py:205,211,1571,1577,1597` move into `helpers.py`/`posting.py` unchanged. +2. Persona topic spread: `persona_article_score`/`pick_category` (`config.py`, untouched); + call-sites in `helpers.py`/`posting.py`. +3. Gist quality gate: `LLMClient.gist_quality_check`/`quality_check` (`llm.py`, untouched); + enforced in `posting.py` (`_create_gist`) and `engage.py` (`_comment_on_post`). +4. Inter-bot threads: `ArticleRegistry` (own module, untouched) PLUS in-bot taper + (`state.thread_reply_counts`/`MAX_THREAD_REPLIES` `bot.py:2163-2166,2272-2277`, + `MAX_SIBLING_COMMENTS` `912,997-1020,1155-1164`, `_respond_to_mention` 2143, + `_reply_to_random_comment` 1100, `_spoke_last_in_thread` 2305) - keep this logic together + and unchanged across `engage.py`/`social.py`. + +**`__init__.py` - keep exporting ONLY `BotsService` (lazy-Playwright boundary).** `service.py` +lazy-imports `bot.py` and treats `ImportError` as "install the bots extra". `bot.py` imports +`faker` and `BotBrowser` (Playwright) at module top. If the package `__init__` eagerly imported +`DevPlaceBot`, importing `devplacepy.services.bot` (for `BotsService`, used by `main.py`) would +hard-fail on boxes without the extra. So: leave `__init__.py` exporting only `BotsService`; keep +`DevPlaceBot`/`LLMClient` importable from their own modules (all live importers already use those +paths - `service.py:263`, `bot.py:39,59`). + +**Stealth rule - already compliant, do not regress.** The only outbound HTTP is +`LLMClient._raw_call` (`llm.py:69`) via `stealth.stealth_sync_client`. `bot.py` makes no bare +`httpx` calls (all page "fetch" is inside Playwright). No new module may introduce a bare +`httpx.Client`. + +**Test footprint.** `tests/unit/services/bot/bot.py` imports `bot as bot_module` and +`DevPlaceBot`, calling `DevPlaceBot._adopt_account_api_key(fake)` as an unbound method - safe +under mixins (method stays an attribute via inheritance). Admin fleet api/e2e tests go through +`BotsService`, unaffected. + +#### 3.4 `services/bot/llm.py` (792) - LEAVE INTACT (explicit decision) + +Below the 900-line threshold, a single cohesive `LLMClient` with shared `_call`/cost counters, +and it holds two realism mechanics (title rewrite, gist gate). A mixin split would be +net-negative and risk the contract. Keep it whole. (If ever reduced later, the only safe cut is +lifting the static text cleaners `clean`/`strip_md`/`strip_label`/`strip_code_fences`/`_parse_json`/ +`_clamp01`/`_as_int`/`_as_terms`/`_overlap_ratio` into an `llm_text.py` of module functions that +`LLMClient` staticmethods delegate to, preserving `LLMClient.clean` as a public alias. Not part of +this plan.) + +--- + +### 3.5 `services/devii/actions/catalog.py` (2036) -> same-name package `catalog/` + +**Nature.** NOT the `Catalog`/`Action` classes (those live in `actions/spec.py`) and NOT the +live `CATALOG` (that is `registry.py:26-46`, `Catalog(actions=ACTIONS + TASK_ACTIONS + ...)`, +with `_assert_confirm_params(CATALOG)` an import-time side effect at `registry.py:68`). This file +only defines `ACTIONS` (161 `Action` literals, `catalog.py:43-2034`), five private builder helpers +(`path/query/body/upload/confirm`, `8-37`), two description constants (`ATTACHMENTS`, `TARGET_TYPE`, +`40-41`), and `PLATFORM_CATALOG = Catalog(actions=ACTIONS)` (`2036`). No logic. + +**Public surface:** `ACTIONS` and `PLATFORM_CATALOG` only. Importers: `actions/__init__.py:3,7`, +`registry.py:9,27`, and tests `tests/unit/project_files.py:79,366,368`, `tests/unit/schemas.py:117, +123,132`, `tests/unit/services/devii/actions/dispatcher.py:6`, +`tests/unit/services/devii/actions/catalog.py:3`, `tests/api/projects/files/{lines,replacelines}.py:79`. + +**Target layout (`services/devii/actions/catalog/`):** + +``` +__init__.py # imports each domain tuple, ACTIONS = (AUTH_ACTIONS + POSTS_ACTIONS + ...), + # PLATFORM_CATALOG = Catalog(actions=ACTIONS); __all__ = ["ACTIONS","PLATFORM_CATALOG"] +_shared.py # path(), query(), body(), upload(), confirm(), ATTACHMENTS, TARGET_TYPE +auth.py # AUTH_ACTIONS (catalog.py:44-105) +posts.py # POSTS_ACTIONS (106-185, incl. view_feed) +comments.py # COMMENTS_ACTIONS (186-216) +projects.py # PROJECTS_ACTIONS (217-342) +project_files.py # PROJECT_FILE_ACTIONS (343-511) +jobs.py # JOB_ACTIONS (zip+fork) (512-578) +tools.py # TOOLS_ACTIONS (579-679) +profile.py # PROFILE_ACTIONS (680-748) +messages.py # MESSAGE_ACTIONS (749-776) +notifications.py # NOTIFICATION_ACTIONS (777-810) +engagement.py # ENGAGEMENT_ACTIONS (811-880) +social.py # SOCIAL_ACTIONS (881-976, incl. leaderboard) +issues.py # ISSUE_ACTIONS (977-1122) +gists.py # GIST_ACTIONS (1123-1190) +news.py # NEWS_ACTIONS (1191-1211) +uploads.py # UPLOAD_ACTIONS (1212-1246) +admin.py # ADMIN_ACTIONS (1247-1661, ~415 lines; optional further sub-split) +dbapi.py # DBAPI_ACTIONS (1662-1778) +gateway.py # GATEWAY_ACTIONS (1779-1874) +game.py # GAME_ACTIONS (1875-2033) +``` + +Each domain module does `from ._shared import path, query, body, confirm, ATTACHMENTS, TARGET_TYPE` +and `from ..spec import Action`. **`dbapi.py`, `gateway.py`, `game.py` must ALSO +`from ..spec import Param`** - they construct `Param(...)` directly for boolean/integer/number +params. `admin.py` at ~415 lines may optionally be sub-split into `admin_users.py`/`admin_services.py`/ +`admin_backups.py`/`admin_analytics.py`, but a single `admin.py` is already a large improvement. + +**Risks:** +- **Action ordering is behavior.** `Catalog.tool_schemas()` emits schemas in `self.actions` order + (the LLM tool-list order). The `__init__` concatenation must reproduce the exact source order in + the table above. Verify `[a.name for a in ACTIONS]` is identical before/after (order + set). +- **Import-time singleton + validation.** `PLATFORM_CATALOG` builds at import; `registry.py:68` + runs `_assert_confirm_params(CATALOG)` at import. `ACTIONS` must be fully assembled when the + package `__init__` finishes. Domain modules import only from `.._shared`/`..spec` (leaf-level, no + cycle). +- **`confirm`-param invariant.** `registry._assert_confirm_params` (`49-65`) requires every name in + `dispatcher.CONFIRM_REQUIRED`/`CONDITIONAL_CONFIRM` to declare a `confirm` param; dropping any + `confirm()`/inline-confirm param makes import `RuntimeError`. Watch the inline + `body("confirm", ...)` cases at `catalog.py:310-314,336-340,727-731`. + +Nothing outside the package changes; `actions/__init__.py` and `registry.py` keep their exact +imports. + +--- + +### 3.6 `schemas.py` (1285) -> `devplacepy/schemas/` + +**Nature.** Pure Pydantic `*Out` models, all subclassing one shared base `_Out` +(`schemas.py:10-11`). One import-time side effect: `CommentItemOut.model_rebuild()` at +`schemas.py:1023` (resolves the self-referential `children` forward ref). No runtime logic. +48 files import from `devplacepy.schemas` (all `from ...import X`), each router pulling the models +it feeds to `respond(..., model=XOut)`; the `__init__` must re-export ALL public model names. + +**Model dependency graph (drives module order).** `UserOut` is depended on by nearly every +wrapper; `AttachmentOut/VotesOut/ReactionsOut/PollOut/BadgeOut/CommentItemOut` feed the item +wrappers; cross-module inheritance: `AdminUserOut(UserOut)`, `AdminMediaItemOut(MediaItemOut)`, +`ProjectListItemOut(ProjectOut)`, `ProjectFileRawOut(ProjectFileOut)`. So the shared primitives +live in `content.py`/`profile.py`, imported first by the domain modules. + +**Target layout (`devplacepy/schemas/`):** + +``` +__init__.py # explicit re-export of every *Out (+ ActionResult, ErrorOut, ValidationErrorOut) +base.py # _Out + generic envelopes (ActionResult, ErrorOut, ValidationErrorOut) +content.py # UserOut, AttachmentOut, VotesOut, ReactionsOut, Poll*, BadgeOut, PostOut, GistOut, + # ProjectOut, ProjectFile*, NewsOut, CommentOut, CommentEditOut, CommentItemOut, + # NotificationOut, MessageOut (+ CommentItemOut.model_rebuild() at bottom) +listings.py # FeedItemOut/GistItemOut/ProjectListItemOut/NewsListItemOut, Conversation, + # MessageItemOut, Notification*Out, Saved*, Leaderboard*, and the page-level + # FeedOut/PostDetailOut/ProjectsOut/ProjectDetailOut/Gists*/News*/Messages*/ + # Notifications*/Saved* +profile.py # MediaItemOut, ProfileOut, TelegramPairOut +issues.py # IssueItemOut, IssueCommentOut, IssueDetailOut, IssueAttachmentsOut, IssueJobOut, IssuesOut +containers.py # InstanceOut, ScheduleOut, ContainersOut, AdminContainers*, Bot* +jobs.py # ZipJobOut, PlanningJobOut, ForkJobOut, SeoJobOut, SeoReportOut, SeoMetaOut, + # DeepsearchJobOut, DeepsearchSessionOut, DbQueryJobOut +backups.py # BackupStoragePathOut, BackupOut, BackupJobOut, BackupScheduleOut, BackupDashboardOut +admin.py # AdminUserOut, AdminUsersOut, AdminNews*, AdminSettingsOut, AdminNotificationsOut, + # AdminMedia*, AdminTrashOut/TrashItemOut +gateway.py # GatewayUsageOut, UserAiUsageOut +auth.py # AuthPageOut, Landing*, DeviiPageOut +audit.py # AuditLinkOut, AuditEntryOut, AuditLogOut, AuditEventOut +dbapi.py # DbTableOut..NlQueryOut +game.py # GameCropOut..GameLeaderboardOut +``` + +Every module does `from devplacepy.schemas.base import _Out` and imports the primitive models it +subclasses/references from `content.py`/`profile.py`. Use explicit re-export (not `*`) so ordering +does not matter and the surface is auditable. + +**Risks:** +- **Forward-ref rebuild.** Keep `CommentItemOut` and the whole content-primitive cluster in + `content.py` and call `CommentItemOut.model_rebuild()` at the bottom of that module (after all + referencing types are importable). +- **Cross-module inheritance** requires the base imported before the subclass; per-file + `from ...import` handles it since all base classes live in `content.py`/`profile.py`. +- Models reach routers only via `respond(model=...)`; a dropped export silently degrades JSON + responses (Pydantic drops unknown keys). Re-export completeness is the whole risk. + +**Test footprint.** `tests/unit/schemas.py` imports `MediaItemOut`, `AdminMediaItemOut`, +`GameLegacyOut` inside test functions - all must survive re-export. + +--- + +### 3.7 `cli.py` (1080) -> `devplacepy/cli/` + +**Nature.** Plain argparse. One `main()` builds the parser tree; each subcommand sets +`func=cmd_*`; `main()` dispatches `args.func(args)`. Console-script entry `pyproject.toml:40` +`devplace = "devplacepy.cli:main"`. Heavy imports are lazy inside each command body (containers, +jobs.queue, backup.store) - this keeps CLI import cheap and MUST be preserved. + +**Target layout (`devplacepy/cli/`):** + +``` +__init__.py # from .main import main; re-export EVERY cmd_* (tests call cli.cmd_role_get etc.) +_shared.py # _audit_cli (cli.py:9-23) +roles.py # cmd_role_get/set + register_roles(sub) (26-61) +apikeys.py # cmd_apikey_* + register_apikeys(sub) (64-106) +tokens.py # cmd_token_* + register_tokens(sub) (109-211) +devii.py # cmd_devii_reset_quota + register_devii(sub) (214-254) +news.py # cmd_news_* + register_news(sub) (257-290) +attachments.py # cmd_attachments_prune + register(sub) (293-315) +jobs.py # zips/forks/seo/seo-meta/deepsearch prune-clear + _remove_*_artifacts (318-576) +backups.py # cmd_backups_* + register_backups(sub) (393-445) +containers.py # cmd_containers_* + register_containers(sub) (579-659) +migrate.py # _crc32/_migrate_*/_db helpers + cmd_migrate_data + cmd_emoji_sync (662-870) +main.py # build_parser()/main(): imports each module's register_* and dispatches; + # holds the `if __name__ == "__main__": main()` guard +``` + +Each leaf exposes `register_(subparsers)` adding its parser(s) + `set_defaults(func=...)`. +`main.py` calls them in order. `__init__.py` re-exports `main` AND every `cmd_*` because +`tests/unit/cli.py` (`from devplacepy import cli`) calls `cli.cmd_role_get/set`, `cli.cmd_news_clear/ +sanitize`, `cli.cmd_attachments_prune`, `cli.cmd_apikey_get/reset/backfill`, `cli.main` directly. + +**Risks:** +- **Console-script entry** `devplacepy.cli:main` must keep resolving - satisfied by + `from .main import main` in `__init__.py`. +- **Preserve lazy in-function imports** (avoid import-time cycles / heavy deps). Do not hoist. +- Move the `__main__` guard into `main.py` (or `__init__` calling `main()`) so `python -m devplacepy.cli` + behavior is unchanged. + +No production module imports `devplacepy.cli` (only tests). Sibling entrypoints +`devplacepy.services.devii.cli:main` and `devplacepy.services.xmlrpc.server:main` are unaffected. + +--- + +### 3.8 `utils.py` (1024) -> `devplacepy/utils/` + +**Nature.** The most import-sensitive file: 224 files import from it; source of several Jinja +globals and the notification/rewards/achievement funnels. Most-used: `make_combined_slug` (144), +`generate_uid` (127), `clear_user_cache` (32), `not_found` (22), `get_current_user` (20), +`require_admin` (19), `require_user` (16), `is_admin` (12), `track_action` (10). Several +underscore helpers are part of the effective surface (`_resolve_user`, `_user_from_session`, +`_user_from_api_key`, `_user_from_devrant_token`) and MUST be re-exported. + +**Target layout (`devplacepy/utils/`):** + +``` +__init__.py # explicit re-export of the FULL public API incl. imported underscore helpers +passwords.py # hash_password/verify_password (+async), create_session (40-70) +authcache.py # _user_cache, _sync_auth_cache, clear_user_cache, clear_session_cache (73-90) +auth.py # _UNSET, _user_from_*, _resolve_user, get_current_user (92-249) +guards.py # safe_next, redirect_back, cookie_secure, client_ip, require_user, + # require_admin, require_user_api, not_found, is_admin, is_primary_admin (252-376) +text.py # strip_html, plain_float, _mark_floats, pretty_json, slugify, generate_uid, + # make_combined_slug, time_ago, format_date, extract_mentions (379-459, 1015-1024) +notifications.py # PUSH_ICON, DEFAULT_PUSH_URL, _push_tasks, _safe_notify, _schedule_push, + # _schedule_telegram, create_notification, _deliver_notification, + # create_mention_notifications, _deliver_mention_notifications (462-575, 989-1013) +badges.py # BADGE_CATALOG, BADGE_GROUPS, LEVEL_BADGES, get_badge, build_achievements, + # award_badge, notify_badge (671-745, 806-848) +rewards.py # LEVEL_XP, XP_*, level_for_xp, award_xp, _COUNT_MILESTONES, _milestone_metrics, + # check_milestone_badges, award_rewards, _apply_rewards, ACHIEVEMENTS, + # UNIQUE_ACTIONS, track_action, _apply_achievements (654-987) +accounts.py # _create_account, register_account, register_account_async (578-641) +``` + +**Circular-import sensitivity (must preserve exactly):** +- `clear_user_cache` is lazily imported by `services/correction.py:183` and by 32 other files + + `cli.py:74`; it MUST stay importable as `devplacepy.utils.clear_user_cache` via re-export. +- Every cross-cutting lazy import inside utils stays function-local: `is_primary_admin` -> + `database.get_primary_admin_uid` (`328`); `require_user`/`require_admin` -> `responses.wants_json` + + `services.audit.record` (`303-304,336-337`); `_deliver_notification` -> `templating.clear_unread_cache` + (`531`) + `services.audit` (`557`); `award_xp`/`check_milestone_badges`/`award_badge`/`_create_account` + -> `services.audit`; `_milestone_metrics` -> `database.get_streaks`; `_schedule_telegram` -> + `services.telegram.store`; `_safe_notify` -> `devplacepy.push`; `_user_from_devrant_token`/ + `_user_from_access_token` -> `services.devrant.tokens`/`services.access_tokens`. Do NOT hoist any. +- **Cache-name coupling:** the auth cache uses `sync_local_cache("auth", ...)`/`bump_cache_version("auth")` + - do not rename the `"auth"` key. + +**Funnel pairing (do not split apart, do not double-wrap):** each public funnel and its +`background.submit` worker must live in the same module: `create_notification`/`_deliver_notification` +and `create_mention_notifications`/`_deliver_mention_notifications` in `notifications.py`; +`award_rewards`/`_apply_rewards` and `track_action`/`_apply_achievements` in `rewards.py`. Both +`notifications.py` and `rewards.py` need the module-level `from devplacepy.services.background import +background` import. + +**Jinja globals (register-once, from `templating.py`).** `templating.py:13-15,62-63,112,132-136` +imports `format_date`, `time_ago`, `get_badge`, `is_admin`, `is_primary_admin`, `pretty_json` and +registers `is_admin`/`is_primary_admin`/`format_date` as callable globals (load-bearing per the +`respond()` collision rule). The registration stays in `templating.py`; the split must only keep +those import paths resolving via the package. + +**Cache invalidation invariant.** `_create_account` calls `database.invalidate_admins_cache()` on the +first user (`640`) - keep that import in `accounts.py`. + +**Test footprint.** `tests/unit/utils.py` imports `award_badge, award_xp, get_badge, +check_milestone_badges, create_mention_notifications, extract_mentions, generate_uid, hash_password, +is_primary_admin, level_for_xp, safe_next, slugify, strip_html, time_ago, verify_password` - every one +must survive re-export. After the split, diff `dir(devplacepy.utils)` public names against the +pre-split snapshot. + +--- + +### 3.9 `services/news.py` (986) -> `devplacepy/services/news/` + +**Nature.** `NewsService(BaseService)` plus a set of pure free functions and constants for the +import pipeline. `main.py:88,236` imports and registers `NewsService`. The CLI does NOT import this +module (its `news clear`/`news sanitize` operate on raw tables via dataset + `utils.strip_html`). + +**Target layout (`devplacepy/services/news/`):** + +``` +__init__.py # re-exports the full public + test surface; also re-binds `net_guard` and `httpx` +constants.py # all tuning constants + JUNK_PATTERNS + WHITESPACE_PATTERN + prose blocks + # (NEWS_SUMMARY, GRADE_PROMPT_SPEC, FORMAT_PROMPT_SPEC, GRADING_RULES_DESCRIPTION); + # numeric constants MUST precede GRADING_RULES_DESCRIPTION (it is an f-string over them) +models.py # ImageCandidate, ArticleGrade +clean.py # clean_news_text, _caps_ratio, _is_valid_url, reliability_reason, effective_score, + # _extract_grade, _strip_md_fence +images.py # _decode_phash, _get_article_images, _fetch_image_candidate, + # _flag_shared_placeholders, _primary_image +service.py # NewsService (run_once/collect_metrics/_store_*/_apply_landing_selection/ + # _grade_article/_format_article/_grade_article_full/_resolve_uid) + _get_ai_key +``` + +`__init__.py` re-export must include at minimum: `NewsService`, `ImageCandidate`, `clean_news_text`, +`reliability_reason`, `effective_score`, `_extract_grade`, `_get_ai_key`, `_get_article_images`, +`_flag_shared_placeholders`, `_primary_image`, `_strip_md_fence`, and constants `LANDING_MIN_SCORE`, +`LANDING_MAX`, `UNIQUE_IMAGE_BONUS`, `THIN_CONTENT_PENALTY`; plus re-bind `net_guard` and `httpx` as +package attributes. + +**Load-bearing risk (dictates where `_get_ai_key` lives).** `tests/unit/services/news.py` +monkeypatches names in the news-module namespace: `monkeypatch.setattr(news_mod, "get_setting", ...)` +(x10) where `news_mod = devplacepy.services.news`, and `setattr(news_mod.net_guard, "guard_public_url", +...)`, and reads `news_mod.httpx`. `_get_ai_key` resolves `get_setting` from ITS OWN module globals. To +keep these tests passing with NO test edits, keep `_get_ai_key` and its `from ...database import +get_setting` in the SAME module the test patches - i.e. keep them in `service.py` and ensure the test's +`monkeypatch.setattr(news_mod, "get_setting", ...)` still targets the name `_get_ai_key` reads. The safe +form: keep `_get_ai_key` + the `get_setting` import together in `service.py`, and have `__init__` do +`from .service import get_setting` re-binding so the package-namespace patch and the resolving namespace +are the same object. Simplest conservative variant: peel off only the genuinely pure modules +(`constants.py`, `models.py`, `clean.py`, `images.py`) and keep `NewsService` + `_get_ai_key` + +`get_setting`/`net_guard`/`httpx` imports in `service.py` (which `__init__` re-exports wholesale). This +removes most of the size while eliminating the namespace hazard. + +**Other risks:** +- `GRADING_RULES_DESCRIPTION` f-string ordering (constants first). +- `_decode_phash` runs via `run_in_executor` - pure, safe to relocate to `images.py`. +- BaseService contract: `run_once`/`collect_metrics`/`config_fields`/`interval_key`/`default_enabled`/ + `__init__(name="news", interval_seconds=3600)` stay on `NewsService`. Methods using `self.log`/ + `self.get_config` cannot become free functions. +- `_grade_article_full` is pure but called as `self._grade_article_full(...)`; leave it a method + (lowest risk). + +--- + +### 3.10 `services/devii/session.py` (962) -> same-name package `session/` (extract, do not decompose) + +**Nature.** ONE heavily-shared-mutable-state class `DeviiSession` (`73-851`) plus ~150 lines of +stateless constants/pure helpers. Constructed by `hub.py:70-85` with an exact 11-positional + +3-keyword arg signature. `_parse_topic` is imported by `tests/unit/services/devii/session.py:6`. + +**Recommendation: extract the stateless text/helpers into a package, keep `DeviiSession` whole.** +A true class decomposition would require threading `_lock`/`_send_lock`/`_conns`/`_conn_meta`/ +`agent._messages`/`self.tools`/`_pending`/`_buffer`/`_turn_epoch` across objects and risks the +`self.tools[:]` aliasing and `_turn_epoch` interrupt guard. Not worth it. + +**Target layout (`services/devii/session/`):** + +``` +__init__.py # from .core import DeviiSession; from .prompts import _parse_topic (+ optional constants) +prompts.py # DOCS_SYSTEM_PROMPT, DOCS_GREETING, DOCS_TOOLS, TOPIC_CLASSIFIER_PROMPT, + # NON_ADMIN_COST_RULE, LOGIN_REQUEST, BEHAVIOR_HEADER, _parse_topic(), _system_prompt_for() + # (session.py:854-947 + 40 + 67) +_helpers.py # _repair_history(), _now_iso(), _format_offset() (43-65, 950-962) +core.py # class DeviiSession + the numeric constants only it uses (27-31, 39) +``` + +This lifts ~150 lines out of the class file with zero behavior risk. `__init__.py` re-exports +`DeviiSession` (used by `hub.py:15`) and `_parse_topic` (test). `prompts.py`/`_helpers.py` import +nothing from `core.py` (no cycle). All prompt constants are used only inside the class today, so they +move freely. + +Optional (secondary, only if further reduction is wanted): a mixin split of `DeviiSession` into +`ConnectionMixin`/`PromptComposition`/`TurnControl`/`SpendTracking`/`Scheduling` operating on one +shared `self`. Given the density of cross-group `self.` references (e.g. `_run_turn` touches +`_refresh_tools`, `_refresh_system_prompt`, `_docs_topic_gate`, `_cost_snapshot`, `_record_spend`, +`_persist_history`, `_emit`, `_turn_epoch`), this trades one long file for cross-mixin coupling - do it +only if the primary extraction is not enough. + +**Invariants that must not break (whichever depth is chosen):** +- **`self.tools` mutated in place, never reassigned.** `_refresh_tools` uses `self.tools[:] = ...` + because the same list object is bound into `AgenticController`/`Agent`/`react_loop`. Reassigning + freezes the tool list (virtual tools stop taking effect). Same in-place discipline for + `self.agent._messages` rewrites (`_refresh_system_prompt` `544-547`, `reset_conversation` `357-375`, + `_docs_topic_gate` `549-577`, `stop`/`_repair_history` `408-426`). +- **Turn-epoch interrupt guard** (`_turn_epoch` at `184`, incremented in `cancel_turns:384`, checked + in `_run_turn:451,459,463`) - keep `spawn_turn`/`cancel_turns`/`_run_turn` with the shared + `_turn_epoch`/`_turns` state (MEMORY: devii-browser-and-interrupt; do not regress). +- **`__init__` positional-arg contract** with `hub.py:70-85` must be preserved exactly. +- `session.py` imports `CATALOG` from `.registry` (import-time `_assert_confirm_params`); the package + rename must not introduce a cycle - `prompts.py`/`_helpers.py` import nothing from `core.py`. +- `from __future__ import annotations` in every extracted module (annotations use `X | None`). + +--- + +### 3.11 `services/game/store.py` (948) -> subpackage `services/game/store/` + +**Nature.** The DB/stateful layer of the already-package `services/game/` (siblings: `economy.py` +pure rules - untouched; `__init__.py` does `from . import economy, store; from .store import GameError`). +Only `store.py` needs splitting. Every function funnels through four `get_table` accessors +(`_farms/_plots/_quests/_steals`) and `_update_farm`; the only module-level state is `PERK_COLUMN` +(built from `economy.PERKS` at import). The game tables carry no `deleted_at` (outside +`SOFT_DELETE_TABLES`) - no soft-delete concerns. + +**Target layout (convert `store.py` -> `store/` package, keeping path `services.game.store`):** + +``` +store/ + __init__.py # re-exports EVERY name currently on store.py, incl. private helpers tests call + common.py # GameError, _now, _iso, _parse, _today, _parse_date, _farms, _plots, _quests, + # _steals, _lvl, PERK_COLUMN, _update_farm, last_steal_at, steal_cooldown_remaining + farm.py # get_farm, ensure_farm, _create_plot, get_plots, _plot_at, buy_plot, upgrade_ci, leaderboard + serialize.py # _watered_by, _plot_state, serialize_plot, serialize_farm, _serialize_perks, + # _serialize_legacy, _serialize_quests, _daily_available + quests.py # ensure_quests, advance_quests + actions.py # plant, harvest, water, steal, fertilize, prestige, upgrade_perk, upgrade_legacy, + # claim_daily, claim_quest, _auto_harvest +``` + +`__init__.py` re-export must include the router surface (`ensure_farm`, `serialize_farm`, `plant`, +`harvest`, `buy_plot`, `upgrade_ci`, `water`, `steal`, `leaderboard`, `claim_daily`, `claim_quest`, +`upgrade_perk`, `upgrade_legacy`, `prestige`, `fertilize`, `GameError`) **plus the private symbols the +unit test calls directly**: `get_farm`, `get_plots`, `ensure_quests`, `advance_quests`, `_plot_at`, +`_today`. `services/game/__init__.py` keeps working unchanged. + +Importers: `routers/game/index.py:18`, `routers/game/farm.py:11`, `routers/game/_shared.py:6` (all +`from devplacepy.services.game import GameError, store` then `store.X`). No CLI, no `main.py`. + +**Risks:** +- **Re-export private helpers** (`_plot_at`, `_today`, + `get_farm`/`get_plots`/`ensure_quests`/ + `advance_quests`) or the unit test breaks. No monkeypatch-of-module-name coupling here (unlike news), + so the only hazard is a missing re-export. +- **`PERK_COLUMN`** built from `economy.PERKS` at import must live in the low-level shared module + (`common.py`) imported by `serialize.py`/`actions.py`; `common.py` imports `economy`, higher modules + import `common` + `economy` (no cycle). +- **Dense intra-module call graph** (`serialize_farm -> _auto_harvest -> advance_quests`; + `actions -> ensure_farm/_update_farm/advance_quests`) - group so lower-level modules + (`common`/`quests`/`farm`) are imported by higher-level (`serialize`/`actions`), not vice-versa. +- **`leaderboard` lazy import** `from devplacepy.database import get_users_by_uids` at `store.py:593` - + keep function-local when relocating. + +--- + +## 4. SPECIAL CASE - container standalone scripts (files 4 and 5) + +`services/containers/files/bot.py` (2940, `COPY bot.py /usr/bin/botje.py`) and +`services/containers/files/d.py` (2337, `COPY d.py /usr/bin/d.py`) are NOT server backend. They are +standalone single-file programs baked into the `ppy` image (`ppy.Dockerfile:47-48`, build context +`devplacepy/services/containers/files`, invoked `python /usr/bin/botje.py`). Their only hard +third-party dep is `httpx`. **Nothing in `devplacepy/` or `tests/` imports either file** (confirmed +grep) - they are not on `sys.path` as modules and have no test coverage. The only breakage surface is +the Docker build's COPY path and the documented invocation. + +**Binding constraint.** A normal package split (like the server-side files) is **incompatible** with +the single-file COPY + bare-`python` exec contract: a package leaf COPY'd to `/usr/bin/botje.py` would +fail at the first cross-module `from botpkg... import` (no package root on the container's path). So the +in-place package approach used for every other monster does not apply here. + +**Options (pick per user direction; this plan does not choose for you):** + +- **(a) Keep as single-file monoliths.** Zero Dockerfile/Makefile change; the contract stays intact. + Trade-off: the two files remain large and duplicate ~2000 lines of agent code between them + (`files/bot.py` = `files/d.py`'s agent + a DevPlace XML-RPC bot loop; `files/d.py`'s stealth half is + a pure-httpx sibling of `devplacepy/stealth.py`). This is the lowest-risk option. + +- **(b) Split into a source package + add a build step** that re-stitches the single deployed artifact. + Two sub-options, each with concrete required edits: + - **zipapp:** author `files/agent/` as a package, build with + `python -m zipapp files/agent -m "agent.cli:main" -o build/botje.pyz`. Requires changing + `ppy.Dockerfile:47-48` COPY source + extension (`.pyz`), the `chmod`/`ln` lines (`:52`), the + invocation in docs (`templates/docs/getting-started-vibing.html:128-162`) from `python /usr/bin/ + botje.py` to `python /usr/bin/botje.pyz`, and the `make ppy` target (`Makefile:129-130`) to run the + bundler before `docker build`. Downside: breaks the documented `cp /usr/bin/botje.py /app/mybot.py` + copy-and-edit single-file flow. + - **concatenation/stitch:** a `make` recipe concatenates the package modules (deduped imports, in + dependency order) into one `build/bot.py`, which the Dockerfile then COPYs. Keeps the `.py` + extension and `python /usr/bin/botje.py` intact; users edit generated output, not source. Requires + `Makefile:129` (add the bundler step + widen the rebuild-trigger set) and `ppy.Dockerfile:47-48` + (COPY from `build/`). + +**Note (not a recommendation, for context):** the largest issue with these two files is duplication, +not size - `files/bot.py` and `files/d.py` share the entire agent (tool framework, `react_loop`, +`CorpusIndex`, embedded `ChromeStealthClient`). If the user later wants these deduplicated rather than +split, that is a different task; this plan records the fact only. + +--- + +## 5. Cross-file execution order (recommended) + +Order by blast radius (lowest first) so a mistake surfaces on a small surface before the wide ones, +and so the widely-imported files are split last against a known-green tree: + +1. **`services/game/store.py`** - self-contained subpackage, 3 importers, no cycles. Warm-up. +2. **`services/devii/actions/catalog.py`** - pure data, 2 exported symbols, ordering-only risk. +3. **`schemas.py`** - pure models, mechanical, wide but simple re-export. +4. **`services/news.py`** - watch the monkeypatch namespace; conservative variant recommended. +5. **`services/devii/session.py`** - extract stateless helpers only (keep class whole). +6. **`services/bot/bot.py`** - mixin lift; keep `__init__` exporting only `BotsService`. +7. **`cli.py`** - package with per-group modules; keep the console-script entry. +8. **`docs_api.py`** - large but declarative; handle the three ordering hazards. +9. **`utils.py`** - 224 importers, funnel + lazy-import + Jinja-global sensitivity. +10. **`database.py`** - 329 importers, the core layer; do it last against a green tree. +11. **Container `files/bot.py` / `files/d.py`** - only if the user opts into option (b); otherwise + leave as-is per Section 4. + +Each step is independent: split one file, run the Section-2.9 validation, commit, move on. Do not batch +multiple monster splits into one unvalidated change. + +--- + +## 6. Convention alignment (why packages, not `_`-suffixed files) + +This plan uses the project's existing structural idiom, verified against the current tree: +- Router packages (`routers/issues/`, `routers/admin/`, `routers/profile/`, `routers/projects/`) are + directories whose `__init__.py` aggregates leaf modules and re-exports the public symbol, each new + file headed `# retoor `. Every split above mirrors that. +- `_shared.py` is the sanctioned package-private helper module (already used in eight router packages); + it is used here only for genuine cross-leaf construction helpers (`docs_api/_shared.py`, + `catalog/_shared.py`, `cli/_shared.py`), never as a dumping ground. +- No `_v2`/`_refactored`/`_new` suffixes anywhere (forbidden by project rules). The compatibility + mechanism is the same-name package + re-export, not a parallel file. +- `services/` already contains packages that expose a `store` submodule (`services/audit/`, + `services/backup/`, `services/containers/`, `services/game/`), so `services/news/` and + `services/game/store/` and `services/devii/session/` and `services/devii/actions/catalog/` all follow + precedent. + +--- + +## 7. Documentation follow-up (after any split is executed) + +Per the project feature workflow, once a split is executed the structural change is documented in the +three canonical files - but ONLY if a rule/convention changes. A pure module move that preserves every +import path introduces no new architectural rule, so: +- `README.md`: no change (no user-visible/route/env/dep change). +- `AGENTS.md`: add a short note under the relevant domain only if the new module boundary is worth + navigational guidance (e.g. "docs_api is now a package, one module per API group"). +- `CLAUDE.md`: no change unless the split establishes a NEW convention (it does not - it follows the + existing package idiom). + +This section is a reminder, not a task; the plan itself does not edit docs. + +--- + +## 8. Summary table - target packages + +| Source file | Target | Split style | Key invariant to protect | +|-------------|--------|-------------|--------------------------| +| `docs_api.py` | `docs_api/` (+`groups/`) | one module per API group | import-time negotiation pass; group order; `_substitute` re-export | +| `database.py` | `database/` | layered domain modules on `core.py` | `db` singleton in `core`; underscore re-exports; keep lazy imports | +| `services/bot/bot.py` | mixins in `services/bot/` | cooperative mixins | realism contract; `__init__` exports only `BotsService`; stealth rule | +| `services/devii/actions/catalog.py` | `catalog/` (+`_shared.py`) | one module per action domain | `ACTIONS` order; `confirm`-param invariant; `Param` import in dbapi/gateway/game | +| `schemas.py` | `schemas/` | domain schema modules | `CommentItemOut.model_rebuild()`; cross-module inheritance order | +| `cli.py` | `cli/` (+`_shared.py`) | one module per command group | console-script `cli:main`; re-export all `cmd_*`; lazy imports | +| `utils.py` | `utils/` | layered helper modules | funnel pairing; lazy imports; Jinja-global paths; `"auth"` cache key | +| `services/news.py` | `services/news/` | pipeline-stage modules | monkeypatch namespace (`_get_ai_key`/`get_setting`); f-string const order | +| `services/devii/session.py` | `session/` | extract stateless text/helpers | `self.tools[:]` in-place; `_turn_epoch` guard; `__init__` arg contract | +| `services/game/store.py` | `services/game/store/` | cohesive store modules | re-export private test helpers; `PERK_COLUMN` placement | +| `services/containers/files/bot.py` | SPECIAL (Section 4) | single-file OR build step | COPY-single-file + bare-exec contract | +| `services/containers/files/d.py` | SPECIAL (Section 4) | single-file OR build step | COPY-single-file + bare-exec contract |