# ref.md - Monster-file split plan (backend) Author: retoor This document is the execution plan for splitting every backend "monster" file in DevPlace into a package of cohesive modules, without changing behavior and without breaking any importer or test. It is investigation output only; nothing here has been executed. Each section is self-contained and can be executed independently in the recommended order (Section 12). --- ## 1. Scope and definition "Monster" = first-party backend Python file over 900 lines (excludes vendored JS, generated artifacts, and the scratch/workspace copies under `data/`, `var/`, `tmp/`, and the repo-root `maak.py`/`d.py`). The confirmed set: | # | File | Lines | Split kind | |---|------|------:|------------| | 1 | `devplacepy/docs_api.py` | 5659 | package: one module per API group | | 2 | `devplacepy/database.py` | 3764 | package: layered domain modules | | 3 | `devplacepy/services/bot/bot.py` | 2999 | in-package: cooperative mixins | | 4 | `devplacepy/services/containers/files/bot.py` | 2940 | SPECIAL - deployed single-file asset | | 5 | `devplacepy/services/containers/files/d.py` | 2337 | SPECIAL - deployed single-file asset | | 6 | `devplacepy/services/devii/actions/catalog.py` | 2036 | same-name package: one module per action domain | | 7 | `devplacepy/schemas.py` | 1285 | package: domain schema modules | | 8 | `devplacepy/cli.py` | 1080 | package: one module per command group | | 9 | `devplacepy/utils.py` | 1024 | package: layered helper modules | | 10 | `devplacepy/services/news.py` | 986 | package: pipeline-stage modules | | 11 | `devplacepy/services/devii/session.py` | 962 | same-name package: extract stateless text/helpers | | 12 | `devplacepy/services/game/store.py` | 948 | subpackage: cohesive store modules | `services/bot/llm.py` (792) is below threshold and holds two realism mechanics; it is explicitly **left intact** (Section 3.4). --- ## 2. Global rules for every split (apply to all sections) These are invariants. Every section assumes them; deviations are called out per file. 1. **Same import path.** Convert `foo.py` into a package `foo/` (directory with `__init__.py`). Because Python resolves `foo/__init__.py` for `import ...foo`, every existing `from ...foo import X` and `import ...foo as m; m.X` keeps resolving with no caller edit. This is the ONLY compatibility mechanism used. 2. **Exhaustive re-export in `__init__.py`.** The package `__init__` must re-export the complete public surface the codebase and tests currently import - **including underscore-prefixed names that are imported elsewhere** (they are part of the effective surface; e.g. `database._now_iso`, `utils._resolve_user`, `docs_api._substitute`, `store._today`). Prefer **explicit** `from .module import name1, name2` blocks over `from .module import *` so the surface is auditable and star-import name-drops (underscore filtering, `__all__` gaps) cannot silently regress it. 3. **No behavior change.** Pure structural moves. No logic edits, no reordering of ordered data, no renames of public symbols, no signature changes. 4. **Preserve every function-local (lazy) import.** Many files defer imports inside function bodies specifically to break import cycles (`database` <-> `utils`, `utils` <-> `services.audit`/`templating`/`content`, `news` AI-key, game `leaderboard`). These MUST stay function-local in the new modules. Hoisting a lazy import to module top is the single most likely way this refactor introduces a boot-time cycle. 5. **File header.** Every new file starts on line 1 with `# retoor ` (project rule). Carry `from __future__ import annotations` into any module that uses `X | None` style annotations where the original had it. 6. **No em-dash.** Neither the em-dash character nor `—`/`—`/`—` anywhere in new files. Hyphen only. Grep every new file before finishing. 7. **No forbidden names / no comments / no docstrings in source** (project rules): do not introduce `_new`/`_old`/`_temp`/`_v2`/`better_`/`my_`/`the_`, do not add comments or docstrings. 8. **`_shared.py` is the sanctioned package-private helper module.** The project already uses `_shared.py` in eight router packages for cross-leaf helpers. Use that exact name for genuinely package-private construction helpers (docs_api builders, catalog builders, cli `_audit_cli`). Everything else is a normal named module. 9. **Validation after each split (never run the test suite):** - `python -c "from devplacepy.main import app"` must import clean. - `python -m py_compile` every new file. - A public-surface diff: capture `sorted(n for n in dir() if not n.startswith('__'))` BEFORE the split (from git or a saved snapshot) and assert the post-split package exposes a superset. This is the definitive check that no re-export was missed. - em-dash grep over every new file. - The suite is run only if the user explicitly asks; the plan never runs it. --- ## 3. Per-file plans ### 3.1 `docs_api.py` (5659) -> `devplacepy/docs_api/` **Nature.** A flat declarative module: a few construction helpers, one giant `API_GROUPS` list literal (~5240 lines, `docs_api.py:143-5384`, 19 group dicts), then a post-processing/registry/render tail. Endpoints are organized by API group; each group is a self-contained dict literal built via the `endpoint()` factory. **Public surface (only these are imported externally):** `API_GROUPS`, `build_services_group`, `get_group`, `api_doc_pages`, `render_group`, and `_substitute` (underscore but called by `docs_export.py:119`). Importers: `docs_search.py:9,87,90`; `docs_export.py:10,118,119,121`; `services/xmlrpc/registry.py:8,63`; `routers/docs/pages.py:3`; `routers/docs/views.py:14`. Tests: `tests/e2e/docs.py:9`, `tests/unit/project_files.py:381-385`, `tests/api/auth/matrix.py:7`, `tests/api/docs/{index,downloadhtml,downloadmd}.py`. **Target layout:** ``` devplacepy/docs_api/ __init__.py # assembles API_GROUPS in order, runs the negotiation pass, # re-exports the public surface (incl. _substitute) _shared.py # field, endpoint, ROLE_LABELS, NON_BODY_ENDPOINTS, and the # domain constant lists (VOTE_TARGETS, REACTION_TARGETS, # BOOKMARK_TARGETS, COMMENT_TARGETS, PROJECT_TYPES, # GIST_LANGUAGES, SERVICE_ACTIONS) negotiation.py # _PAGE_RESPONSES, _ACTION_RESPONSES, _classify, # _apply_negotiated_responses(groups) (schemas/docs_examples live here) services_group.py # _service_control_endpoints, _field_to_param, _service_section, # build_services_group render.py # _GROUPS_BY_SLUG, get_group, api_doc_pages, _substitute, render_group groups/ __init__.py # ORDERED_GROUPS = [conventions.GROUP, auth.GROUP, ... game.GROUP] conventions.py # docs_api.py:145-293 auth.py # 295-427 lookups.py # 428-484 social_actions.py # 485-674 content.py # 675-1615 (largest group, ~940 lines) profiles.py # 1616-2294 messaging.py # 2295-2365 notifications.py # 2366-2450 uploads.py # 2451-2552 project_files.py # 2553-3098 containers.py # 3099-3673 (admin: True) tools.py # 3674-3902 push.py # 3903-3964 issues.py # 3965-4258 gateway.py # 4259-4449 (admin: True) services.py # 4450-4456 (dynamic placeholder group dict - keep hollow) admin.py # 4458-5193 (admin: True, ~735 lines) game.py # 5194-5383 ``` Each `groups/*.py` holds one `GROUP = {...}` literal importing `field`, `endpoint`, `ROLE_LABELS` and the constant lists from `.._shared` (and `constants.TOPICS`/ `REACTION_EMOJI` where the literal uses them - re-scan each block for free names before cutting). **`__init__.py` must, in this exact order:** 1. `from .groups import ORDERED_GROUPS as API_GROUPS` (order is load-bearing). 2. `from .negotiation import _apply_negotiated_responses` then call `_apply_negotiated_responses(API_GROUPS)` - reproduces the current import-time mutation at `docs_api.py:5494`. Must run AFTER `API_GROUPS` is assembled. 3. Re-export `API_GROUPS`, `build_services_group`, `get_group`, `api_doc_pages`, `render_group`, `_substitute` (and, for safety, `endpoint`, `field`, `ROLE_LABELS`, the constant lists). **Risks / ordering hazards (must be handled):** - **Import-time negotiation mutation.** `_apply_negotiated_responses()` runs at module top today (`docs_api.py:5494`) and mutates each endpoint dict in place (fills `sample_response`, stamps `negotiation`). Centralize this in `__init__` after assembly; if a group is read before the pass runs, endpoints lack `negotiation`/`sample_response`. - **Forward reference.** `_classify` (`105-112`) reads `_PAGE_RESPONSES`/`_ACTION_RESPONSES` defined 5280 lines later. Colocate `_classify` + both maps + the pass in `negotiation.py` to make the dependency explicit. - **Assembly-before-render cycle.** `render.py` (`get_group`/`render_group`/`api_doc_pages`/ `_GROUPS_BY_SLUG`) needs the final `API_GROUPS`. Build `API_GROUPS` in `groups/__init__.py` and import it into both package `__init__` and `render.py`; do NOT let `render.py` import the package `__init__` at module top (cycle). - **`build_services_group` reuses core helpers** (`endpoint`, `field` via `_field_to_param`, `_classify`, `_service_control_endpoints`) and stamps `negotiation` itself for its runtime group; `services_group.py` imports `endpoint`/`field` from `_shared` and `_classify` from `negotiation`; do NOT double-apply negotiation to it. - **`_substitute` name must be preserved** in the re-export (called as `docs_api._substitute` by `docs_export.py:119`). - **Two groups slug `services`** (the hollow placeholder in the list + the runtime `build_services_group` output). Keep the placeholder exactly where it is in the ordered list. **Test invariant:** `API_GROUPS` order/content, group flags (`slug/title/admin/dynamic/ endpoints`), and every endpoint's post-negotiation shape (`id/method/path/auth/min_role/ params/negotiation/sample_response`) must be identical after the split. `tests/api/auth/ matrix.py` asserts `len(endpoints) >= 50` and drives each endpoint's documented auth against live behavior; `tests/unit/project_files.py:381-385` asserts the `project-files` group's exact endpoint id set. --- ### 3.2 `database.py` (3764) -> `devplacepy/database/` **Nature.** The core data layer, imported nearly everywhere (677 `from ...database import` statements across 329 files; hottest names `get_table` x404, `refresh_snapshot` x92, `set_setting` x67, `db` x33). The `__init__` MUST re-export essentially the entire namespace including underscore helpers (`_now_iso` x6, `_index`, and whatever tests reach for). **The linchpin.** A single `core.py` owns the `db` singleton, the import-time connection side effects, the cross-worker cache-version primitives, index helpers, and trivial accessors. Every other module imports only from `core` (and strictly-lower layers). Because `core.py` is imported once, Python's module cache guarantees the single `db` object, exactly as today. **Target layout (`devplacepy/database/`):** ``` core.py # db, refresh_snapshot; import-time ensure_data_dirs + sqlite mkdir + # dataset.connect (database.py:20-42); cache-version primitives # (_ensure_cache_state, get_cache_version, bump_cache_version, # sync_local_cache, _local_cache_versions, _cache_version_cache, # _cache_state_ready); _index, _drop_index, _uid_index; # get_table, _in_clause, _now_iso <- imported by ALL settings.py # _settings_cache, get_setting, get_int_setting, set_setting, # clear_settings_cache, internal_gateway_key users.py # get_users_by_uids, search_users_by_username, set_user_timezone, # admins cache (_admins_cache, invalidate_admins_cache, # get_admin_uids, get_primary_admin_uid) relations.py # user block/mute cache + accessors (get_user_relations, get_blocked_uids, # get_muted_uids, get_silenced_uids, invalidate_user_relations) pagination.py # PAGE_SIZE, build_pagination, paginate, paginate_diverse, # interleave_by_author, get_user_post_count (imports core + relations) soft_delete.py # SOFT_DELETE_TABLES, ensure_soft_delete_columns, soft_delete, # soft_delete_in, restore, purge, list_deleted, count_deleted, # restore_event, purge_event (imports core + pagination.build_pagination) engagement.py # comment-count/vote/reaction/bookmark batch helpers, polls comments.py # _drop_blocked, _build_comment_items, load_comments, recent/by-target loaders attachments_data.py# get_attachments*, get_news_images_by_uids, delete_attachment*, # _delete_attachment_file, get_user_media, get_deleted_media usage.py # _add_usage, _get_usage, add/get_*_usage + *_USAGE_KEY constants seo_meta.py # SEO_META_TYPES + seo_metadata CRUD activity.py # user_activity counters + calendar/heatmap/streaks ranking.py # VOTABLE_TARGETS, STAR_TARGETS, authors/leaderboard/stars, # update_target_stars, get_target_owner_uid, soft_delete_engagement, # delete_engagement customization.py # customization prefs + override CRUD email.py # email account CRUD notifications.py # notification prefs + mark_notifications_read_by_target forks.py # record_fork, get_fork_parent, count_forks, *_fork_relations content.py # resolve_by_slug, resolve_object_url, text_search_clause, # get_uids_by_username_match, get_daily_topic, get_featured_news follows.py # get_follow_counts, get_follow_list, get_following_among stats.py # get_site_stats, get_platform_analytics, get_gist_languages deepsearch.py # _ds_now + deepsearch session/message/url-cache helpers schema.py # init_db, _refresh_query_planner_stats, migrate_bug_tables_to_issue_tables, # BUG_TABLE_RENAMES, migrate_ai_gateway_settings, OLD_GATEWAY_URL, # backfill_api_keys, _backfill_gamification <- imports EVERYTHING; loaded last __init__.py # explicit re-export of every module's public + test-touched underscore names ``` **Where shared state lives (critical).** The only truly global object is `db` (+ the cache-version machinery), which lives in `core.py`. Every domain `TTLCache`/dict singleton (`_admins_cache`, `_settings_cache`, `_relations_cache`, `_customizations_cache`, `_notification_prefs_cache`, `_stats_cache`, `_analytics_cache`, `_activity_cache`, `_gist_languages_cache`, `_authors_cache`, `_comment_count_cache`) is referenced only by its own domain, so it moves WITH that domain's module - no shared-cache problem. Never re-create a cache in `__init__`; only re-export. Two import paths to one cache would create two caches. **Layering (acyclic).** `core` <- everything; `pagination -> relations`; `soft_delete -> pagination`; `comments -> users + engagement + relations`; `ranking -> users`; `stats -> ranking`; `attachments_data -> content`; `content -> users`; `schema` imports any sibling at module top (nothing imports `schema`; loaded last in `__init__`). **Cycle-avoidance rules:** - `core.py` imports nothing from siblings. - Strict layer order: `core -> settings/users/relations -> pagination -> soft_delete -> domain modules -> schema`. - **Preserve every existing function-local import verbatim** - the `utils`, `attachments`, `templating`, `services.audit.store`/`services.backup.store`/`services.openai_gateway.routing` imports inside functions (e.g. `init_db` at `database.py:951-961`) exist to break the real `database <-> utils` cycle. Do not hoist them. - `utils.py` uses `database`'s cache primitive under the `"auth"` name; those primitives are in `core.py` and remain reachable as `database.bump_cache_version`/`database.sync_local_cache` via re-export. **Risks:** - **`init_db()` (503-1534) is one idempotent ordered routine** - move it WHOLE into `schema.py`; do not reorder its internal steps (migration-before-index, backfills/ANALYZE last at 1516-1519). - **Import-time side effects (20-42)** must live in `core.py` and run exactly once; `core` must be the first thing `__init__` imports. - **Underscore re-export gap** is the highest-probability regression (`import *` drops `_now_iso`/ `_index`). Use explicit re-exports or per-module `__all__` including the imported underscores. - **`get_table` (404 importers) and `refresh_snapshot` (92)** - any miss breaks a huge fraction of the app at import; the public-surface diff (Section 2.9) is the guard. **Test footprint.** `tests/unit/database.py` is the primary direct consumer (it also hits a live server, so it is effectively api-tier); `tests/unit/services/gitea.py` and `tests/unit/services/openai_gateway/analytics.py` import it for setup; dozens of api/e2e files use `get_table`/`set_setting`/usage helpers indirectly. With a complete re-export, zero test edits are needed. --- ### 3.3 `services/bot/bot.py` (2999) -> cooperative mixins in `services/bot/` **Nature.** ONE class `DevPlaceBot` (`bot.py:48`), ~90 methods, every method reads/writes shared `self` state (`self.state` persisted `BotState`, `self.b` browser, `self.llm`, `self._session_*` counters, the `_post_cache/_project_cache/_issue_cache/_session_history` caches). The only zero-behavior-change decomposition is **cooperative mixins** (Python MRO gives byte-identical `self` semantics; the unbound-method unit test still resolves via inheritance). There is NO `@tool` registry in this file (the closest analogue is the AI decision/menu engine). **Target layout (new modules beside the existing `services/bot/` files):** | New module | Mixin | Moves from bot.py | Contents | |------------|-------|-------------------|----------| | `helpers.py` | `BotHelpersMixin` | 103-245 | instrumentation, cost/session banners, `_thread_id`, `_pick_article`, cache warm | | `auth.py` | `BotAuthMixin` | 246-355 | handles, `_ensure_auth`, account-key adoption | | `human.py` | `BotHumanMixin` | 356-522 | read/type/fatigue simulation, session type, page inspection | | `agent.py` | `BotAgentMixin` | 523-757 | AI decision/menu/plan engine (`_build_menu`, `_run_plan`, `_cycle_ai`, dispatch, nav) | | `engage.py` | `BotEngageMixin` | 753-1436 | voting, commenting, reacting (realism #4b) | | `posting.py` | `BotPostingMixin` | 1438-1770 | gist + post creation, content open/click (realism #1, #3 call-sites) | | `social.py` | `BotSocialMixin` | 1771-2378 | profile/project/issue/follow, notifications, mentions (#4b), messaging | | `browse.py` | `BotBrowseMixin` | 2379-2502 | scroll, nav, browse, search, `_engage_community` | | `loop.py` | `BotLoopMixin` | 2503-2999 | `_cycle`, `run_forever` (scheduling) | `bot.py` shrinks to ~110 lines: the mixin imports, `__init__`, and: ```python class DevPlaceBot( BotHelpersMixin, BotAuthMixin, BotHumanMixin, BotAgentMixin, BotEngageMixin, BotPostingMixin, BotSocialMixin, BotBrowseMixin, BotLoopMixin, ): ... ``` `DevPlaceBot` stays importable from `devplacepy.services.bot.bot` (do NOT move the class file; only lift methods into inherited mixins). Method names are already globally unique across the 90 methods - verify no collision when composing. **Realism contract (MUST NOT regress) - exact locations to preserve:** 1. Title rewrite: `LLMClient.generate_post`/`generate_post_title` (in `llm.py`, untouched); call-sites `bot.py:205,211,1571,1577,1597` move into `helpers.py`/`posting.py` unchanged. 2. Persona topic spread: `persona_article_score`/`pick_category` (`config.py`, untouched); call-sites in `helpers.py`/`posting.py`. 3. Gist quality gate: `LLMClient.gist_quality_check`/`quality_check` (`llm.py`, untouched); enforced in `posting.py` (`_create_gist`) and `engage.py` (`_comment_on_post`). 4. Inter-bot threads: `ArticleRegistry` (own module, untouched) PLUS in-bot taper (`state.thread_reply_counts`/`MAX_THREAD_REPLIES` `bot.py:2163-2166,2272-2277`, `MAX_SIBLING_COMMENTS` `912,997-1020,1155-1164`, `_respond_to_mention` 2143, `_reply_to_random_comment` 1100, `_spoke_last_in_thread` 2305) - keep this logic together and unchanged across `engage.py`/`social.py`. **`__init__.py` - keep exporting ONLY `BotsService` (lazy-Playwright boundary).** `service.py` lazy-imports `bot.py` and treats `ImportError` as "install the bots extra". `bot.py` imports `faker` and `BotBrowser` (Playwright) at module top. If the package `__init__` eagerly imported `DevPlaceBot`, importing `devplacepy.services.bot` (for `BotsService`, used by `main.py`) would hard-fail on boxes without the extra. So: leave `__init__.py` exporting only `BotsService`; keep `DevPlaceBot`/`LLMClient` importable from their own modules (all live importers already use those paths - `service.py:263`, `bot.py:39,59`). **Stealth rule - already compliant, do not regress.** The only outbound HTTP is `LLMClient._raw_call` (`llm.py:69`) via `stealth.stealth_sync_client`. `bot.py` makes no bare `httpx` calls (all page "fetch" is inside Playwright). No new module may introduce a bare `httpx.Client`. **Test footprint.** `tests/unit/services/bot/bot.py` imports `bot as bot_module` and `DevPlaceBot`, calling `DevPlaceBot._adopt_account_api_key(fake)` as an unbound method - safe under mixins (method stays an attribute via inheritance). Admin fleet api/e2e tests go through `BotsService`, unaffected. #### 3.4 `services/bot/llm.py` (792) - LEAVE INTACT (explicit decision) Below the 900-line threshold, a single cohesive `LLMClient` with shared `_call`/cost counters, and it holds two realism mechanics (title rewrite, gist gate). A mixin split would be net-negative and risk the contract. Keep it whole. (If ever reduced later, the only safe cut is lifting the static text cleaners `clean`/`strip_md`/`strip_label`/`strip_code_fences`/`_parse_json`/ `_clamp01`/`_as_int`/`_as_terms`/`_overlap_ratio` into an `llm_text.py` of module functions that `LLMClient` staticmethods delegate to, preserving `LLMClient.clean` as a public alias. Not part of this plan.) --- ### 3.5 `services/devii/actions/catalog.py` (2036) -> same-name package `catalog/` **Nature.** NOT the `Catalog`/`Action` classes (those live in `actions/spec.py`) and NOT the live `CATALOG` (that is `registry.py:26-46`, `Catalog(actions=ACTIONS + TASK_ACTIONS + ...)`, with `_assert_confirm_params(CATALOG)` an import-time side effect at `registry.py:68`). This file only defines `ACTIONS` (161 `Action` literals, `catalog.py:43-2034`), five private builder helpers (`path/query/body/upload/confirm`, `8-37`), two description constants (`ATTACHMENTS`, `TARGET_TYPE`, `40-41`), and `PLATFORM_CATALOG = Catalog(actions=ACTIONS)` (`2036`). No logic. **Public surface:** `ACTIONS` and `PLATFORM_CATALOG` only. Importers: `actions/__init__.py:3,7`, `registry.py:9,27`, and tests `tests/unit/project_files.py:79,366,368`, `tests/unit/schemas.py:117, 123,132`, `tests/unit/services/devii/actions/dispatcher.py:6`, `tests/unit/services/devii/actions/catalog.py:3`, `tests/api/projects/files/{lines,replacelines}.py:79`. **Target layout (`services/devii/actions/catalog/`):** ``` __init__.py # imports each domain tuple, ACTIONS = (AUTH_ACTIONS + POSTS_ACTIONS + ...), # PLATFORM_CATALOG = Catalog(actions=ACTIONS); __all__ = ["ACTIONS","PLATFORM_CATALOG"] _shared.py # path(), query(), body(), upload(), confirm(), ATTACHMENTS, TARGET_TYPE auth.py # AUTH_ACTIONS (catalog.py:44-105) posts.py # POSTS_ACTIONS (106-185, incl. view_feed) comments.py # COMMENTS_ACTIONS (186-216) projects.py # PROJECTS_ACTIONS (217-342) project_files.py # PROJECT_FILE_ACTIONS (343-511) jobs.py # JOB_ACTIONS (zip+fork) (512-578) tools.py # TOOLS_ACTIONS (579-679) profile.py # PROFILE_ACTIONS (680-748) messages.py # MESSAGE_ACTIONS (749-776) notifications.py # NOTIFICATION_ACTIONS (777-810) engagement.py # ENGAGEMENT_ACTIONS (811-880) social.py # SOCIAL_ACTIONS (881-976, incl. leaderboard) issues.py # ISSUE_ACTIONS (977-1122) gists.py # GIST_ACTIONS (1123-1190) news.py # NEWS_ACTIONS (1191-1211) uploads.py # UPLOAD_ACTIONS (1212-1246) admin.py # ADMIN_ACTIONS (1247-1661, ~415 lines; optional further sub-split) dbapi.py # DBAPI_ACTIONS (1662-1778) gateway.py # GATEWAY_ACTIONS (1779-1874) game.py # GAME_ACTIONS (1875-2033) ``` Each domain module does `from ._shared import path, query, body, confirm, ATTACHMENTS, TARGET_TYPE` and `from ..spec import Action`. **`dbapi.py`, `gateway.py`, `game.py` must ALSO `from ..spec import Param`** - they construct `Param(...)` directly for boolean/integer/number params. `admin.py` at ~415 lines may optionally be sub-split into `admin_users.py`/`admin_services.py`/ `admin_backups.py`/`admin_analytics.py`, but a single `admin.py` is already a large improvement. **Risks:** - **Action ordering is behavior.** `Catalog.tool_schemas()` emits schemas in `self.actions` order (the LLM tool-list order). The `__init__` concatenation must reproduce the exact source order in the table above. Verify `[a.name for a in ACTIONS]` is identical before/after (order + set). - **Import-time singleton + validation.** `PLATFORM_CATALOG` builds at import; `registry.py:68` runs `_assert_confirm_params(CATALOG)` at import. `ACTIONS` must be fully assembled when the package `__init__` finishes. Domain modules import only from `.._shared`/`..spec` (leaf-level, no cycle). - **`confirm`-param invariant.** `registry._assert_confirm_params` (`49-65`) requires every name in `dispatcher.CONFIRM_REQUIRED`/`CONDITIONAL_CONFIRM` to declare a `confirm` param; dropping any `confirm()`/inline-confirm param makes import `RuntimeError`. Watch the inline `body("confirm", ...)` cases at `catalog.py:310-314,336-340,727-731`. Nothing outside the package changes; `actions/__init__.py` and `registry.py` keep their exact imports. --- ### 3.6 `schemas.py` (1285) -> `devplacepy/schemas/` **Nature.** Pure Pydantic `*Out` models, all subclassing one shared base `_Out` (`schemas.py:10-11`). One import-time side effect: `CommentItemOut.model_rebuild()` at `schemas.py:1023` (resolves the self-referential `children` forward ref). No runtime logic. 48 files import from `devplacepy.schemas` (all `from ...import X`), each router pulling the models it feeds to `respond(..., model=XOut)`; the `__init__` must re-export ALL public model names. **Model dependency graph (drives module order).** `UserOut` is depended on by nearly every wrapper; `AttachmentOut/VotesOut/ReactionsOut/PollOut/BadgeOut/CommentItemOut` feed the item wrappers; cross-module inheritance: `AdminUserOut(UserOut)`, `AdminMediaItemOut(MediaItemOut)`, `ProjectListItemOut(ProjectOut)`, `ProjectFileRawOut(ProjectFileOut)`. So the shared primitives live in `content.py`/`profile.py`, imported first by the domain modules. **Target layout (`devplacepy/schemas/`):** ``` __init__.py # explicit re-export of every *Out (+ ActionResult, ErrorOut, ValidationErrorOut) base.py # _Out + generic envelopes (ActionResult, ErrorOut, ValidationErrorOut) content.py # UserOut, AttachmentOut, VotesOut, ReactionsOut, Poll*, BadgeOut, PostOut, GistOut, # ProjectOut, ProjectFile*, NewsOut, CommentOut, CommentEditOut, CommentItemOut, # NotificationOut, MessageOut (+ CommentItemOut.model_rebuild() at bottom) listings.py # FeedItemOut/GistItemOut/ProjectListItemOut/NewsListItemOut, Conversation, # MessageItemOut, Notification*Out, Saved*, Leaderboard*, and the page-level # FeedOut/PostDetailOut/ProjectsOut/ProjectDetailOut/Gists*/News*/Messages*/ # Notifications*/Saved* profile.py # MediaItemOut, ProfileOut, TelegramPairOut issues.py # IssueItemOut, IssueCommentOut, IssueDetailOut, IssueAttachmentsOut, IssueJobOut, IssuesOut containers.py # InstanceOut, ScheduleOut, ContainersOut, AdminContainers*, Bot* jobs.py # ZipJobOut, PlanningJobOut, ForkJobOut, SeoJobOut, SeoReportOut, SeoMetaOut, # DeepsearchJobOut, DeepsearchSessionOut, DbQueryJobOut backups.py # BackupStoragePathOut, BackupOut, BackupJobOut, BackupScheduleOut, BackupDashboardOut admin.py # AdminUserOut, AdminUsersOut, AdminNews*, AdminSettingsOut, AdminNotificationsOut, # AdminMedia*, AdminTrashOut/TrashItemOut gateway.py # GatewayUsageOut, UserAiUsageOut auth.py # AuthPageOut, Landing*, DeviiPageOut audit.py # AuditLinkOut, AuditEntryOut, AuditLogOut, AuditEventOut dbapi.py # DbTableOut..NlQueryOut game.py # GameCropOut..GameLeaderboardOut ``` Every module does `from devplacepy.schemas.base import _Out` and imports the primitive models it subclasses/references from `content.py`/`profile.py`. Use explicit re-export (not `*`) so ordering does not matter and the surface is auditable. **Risks:** - **Forward-ref rebuild.** Keep `CommentItemOut` and the whole content-primitive cluster in `content.py` and call `CommentItemOut.model_rebuild()` at the bottom of that module (after all referencing types are importable). - **Cross-module inheritance** requires the base imported before the subclass; per-file `from ...import` handles it since all base classes live in `content.py`/`profile.py`. - Models reach routers only via `respond(model=...)`; a dropped export silently degrades JSON responses (Pydantic drops unknown keys). Re-export completeness is the whole risk. **Test footprint.** `tests/unit/schemas.py` imports `MediaItemOut`, `AdminMediaItemOut`, `GameLegacyOut` inside test functions - all must survive re-export. --- ### 3.7 `cli.py` (1080) -> `devplacepy/cli/` **Nature.** Plain argparse. One `main()` builds the parser tree; each subcommand sets `func=cmd_*`; `main()` dispatches `args.func(args)`. Console-script entry `pyproject.toml:40` `devplace = "devplacepy.cli:main"`. Heavy imports are lazy inside each command body (containers, jobs.queue, backup.store) - this keeps CLI import cheap and MUST be preserved. **Target layout (`devplacepy/cli/`):** ``` __init__.py # from .main import main; re-export EVERY cmd_* (tests call cli.cmd_role_get etc.) _shared.py # _audit_cli (cli.py:9-23) roles.py # cmd_role_get/set + register_roles(sub) (26-61) apikeys.py # cmd_apikey_* + register_apikeys(sub) (64-106) tokens.py # cmd_token_* + register_tokens(sub) (109-211) devii.py # cmd_devii_reset_quota + register_devii(sub) (214-254) news.py # cmd_news_* + register_news(sub) (257-290) attachments.py # cmd_attachments_prune + register(sub) (293-315) jobs.py # zips/forks/seo/seo-meta/deepsearch prune-clear + _remove_*_artifacts (318-576) backups.py # cmd_backups_* + register_backups(sub) (393-445) containers.py # cmd_containers_* + register_containers(sub) (579-659) migrate.py # _crc32/_migrate_*/_db helpers + cmd_migrate_data + cmd_emoji_sync (662-870) main.py # build_parser()/main(): imports each module's register_* and dispatches; # holds the `if __name__ == "__main__": main()` guard ``` Each leaf exposes `register_(subparsers)` adding its parser(s) + `set_defaults(func=...)`. `main.py` calls them in order. `__init__.py` re-exports `main` AND every `cmd_*` because `tests/unit/cli.py` (`from devplacepy import cli`) calls `cli.cmd_role_get/set`, `cli.cmd_news_clear/ sanitize`, `cli.cmd_attachments_prune`, `cli.cmd_apikey_get/reset/backfill`, `cli.main` directly. **Risks:** - **Console-script entry** `devplacepy.cli:main` must keep resolving - satisfied by `from .main import main` in `__init__.py`. - **Preserve lazy in-function imports** (avoid import-time cycles / heavy deps). Do not hoist. - Move the `__main__` guard into `main.py` (or `__init__` calling `main()`) so `python -m devplacepy.cli` behavior is unchanged. No production module imports `devplacepy.cli` (only tests). Sibling entrypoints `devplacepy.services.devii.cli:main` and `devplacepy.services.xmlrpc.server:main` are unaffected. --- ### 3.8 `utils.py` (1024) -> `devplacepy/utils/` **Nature.** The most import-sensitive file: 224 files import from it; source of several Jinja globals and the notification/rewards/achievement funnels. Most-used: `make_combined_slug` (144), `generate_uid` (127), `clear_user_cache` (32), `not_found` (22), `get_current_user` (20), `require_admin` (19), `require_user` (16), `is_admin` (12), `track_action` (10). Several underscore helpers are part of the effective surface (`_resolve_user`, `_user_from_session`, `_user_from_api_key`, `_user_from_devrant_token`) and MUST be re-exported. **Target layout (`devplacepy/utils/`):** ``` __init__.py # explicit re-export of the FULL public API incl. imported underscore helpers passwords.py # hash_password/verify_password (+async), create_session (40-70) authcache.py # _user_cache, _sync_auth_cache, clear_user_cache, clear_session_cache (73-90) auth.py # _UNSET, _user_from_*, _resolve_user, get_current_user (92-249) guards.py # safe_next, redirect_back, cookie_secure, client_ip, require_user, # require_admin, require_user_api, not_found, is_admin, is_primary_admin (252-376) text.py # strip_html, plain_float, _mark_floats, pretty_json, slugify, generate_uid, # make_combined_slug, time_ago, format_date, extract_mentions (379-459, 1015-1024) notifications.py # PUSH_ICON, DEFAULT_PUSH_URL, _push_tasks, _safe_notify, _schedule_push, # _schedule_telegram, create_notification, _deliver_notification, # create_mention_notifications, _deliver_mention_notifications (462-575, 989-1013) badges.py # BADGE_CATALOG, BADGE_GROUPS, LEVEL_BADGES, get_badge, build_achievements, # award_badge, notify_badge (671-745, 806-848) rewards.py # LEVEL_XP, XP_*, level_for_xp, award_xp, _COUNT_MILESTONES, _milestone_metrics, # check_milestone_badges, award_rewards, _apply_rewards, ACHIEVEMENTS, # UNIQUE_ACTIONS, track_action, _apply_achievements (654-987) accounts.py # _create_account, register_account, register_account_async (578-641) ``` **Circular-import sensitivity (must preserve exactly):** - `clear_user_cache` is lazily imported by `services/correction.py:183` and by 32 other files + `cli.py:74`; it MUST stay importable as `devplacepy.utils.clear_user_cache` via re-export. - Every cross-cutting lazy import inside utils stays function-local: `is_primary_admin` -> `database.get_primary_admin_uid` (`328`); `require_user`/`require_admin` -> `responses.wants_json` + `services.audit.record` (`303-304,336-337`); `_deliver_notification` -> `templating.clear_unread_cache` (`531`) + `services.audit` (`557`); `award_xp`/`check_milestone_badges`/`award_badge`/`_create_account` -> `services.audit`; `_milestone_metrics` -> `database.get_streaks`; `_schedule_telegram` -> `services.telegram.store`; `_safe_notify` -> `devplacepy.push`; `_user_from_devrant_token`/ `_user_from_access_token` -> `services.devrant.tokens`/`services.access_tokens`. Do NOT hoist any. - **Cache-name coupling:** the auth cache uses `sync_local_cache("auth", ...)`/`bump_cache_version("auth")` - do not rename the `"auth"` key. **Funnel pairing (do not split apart, do not double-wrap):** each public funnel and its `background.submit` worker must live in the same module: `create_notification`/`_deliver_notification` and `create_mention_notifications`/`_deliver_mention_notifications` in `notifications.py`; `award_rewards`/`_apply_rewards` and `track_action`/`_apply_achievements` in `rewards.py`. Both `notifications.py` and `rewards.py` need the module-level `from devplacepy.services.background import background` import. **Jinja globals (register-once, from `templating.py`).** `templating.py:13-15,62-63,112,132-136` imports `format_date`, `time_ago`, `get_badge`, `is_admin`, `is_primary_admin`, `pretty_json` and registers `is_admin`/`is_primary_admin`/`format_date` as callable globals (load-bearing per the `respond()` collision rule). The registration stays in `templating.py`; the split must only keep those import paths resolving via the package. **Cache invalidation invariant.** `_create_account` calls `database.invalidate_admins_cache()` on the first user (`640`) - keep that import in `accounts.py`. **Test footprint.** `tests/unit/utils.py` imports `award_badge, award_xp, get_badge, check_milestone_badges, create_mention_notifications, extract_mentions, generate_uid, hash_password, is_primary_admin, level_for_xp, safe_next, slugify, strip_html, time_ago, verify_password` - every one must survive re-export. After the split, diff `dir(devplacepy.utils)` public names against the pre-split snapshot. --- ### 3.9 `services/news.py` (986) -> `devplacepy/services/news/` **Nature.** `NewsService(BaseService)` plus a set of pure free functions and constants for the import pipeline. `main.py:88,236` imports and registers `NewsService`. The CLI does NOT import this module (its `news clear`/`news sanitize` operate on raw tables via dataset + `utils.strip_html`). **Target layout (`devplacepy/services/news/`):** ``` __init__.py # re-exports the full public + test surface; also re-binds `net_guard` and `httpx` constants.py # all tuning constants + JUNK_PATTERNS + WHITESPACE_PATTERN + prose blocks # (NEWS_SUMMARY, GRADE_PROMPT_SPEC, FORMAT_PROMPT_SPEC, GRADING_RULES_DESCRIPTION); # numeric constants MUST precede GRADING_RULES_DESCRIPTION (it is an f-string over them) models.py # ImageCandidate, ArticleGrade clean.py # clean_news_text, _caps_ratio, _is_valid_url, reliability_reason, effective_score, # _extract_grade, _strip_md_fence images.py # _decode_phash, _get_article_images, _fetch_image_candidate, # _flag_shared_placeholders, _primary_image service.py # NewsService (run_once/collect_metrics/_store_*/_apply_landing_selection/ # _grade_article/_format_article/_grade_article_full/_resolve_uid) + _get_ai_key ``` `__init__.py` re-export must include at minimum: `NewsService`, `ImageCandidate`, `clean_news_text`, `reliability_reason`, `effective_score`, `_extract_grade`, `_get_ai_key`, `_get_article_images`, `_flag_shared_placeholders`, `_primary_image`, `_strip_md_fence`, and constants `LANDING_MIN_SCORE`, `LANDING_MAX`, `UNIQUE_IMAGE_BONUS`, `THIN_CONTENT_PENALTY`; plus re-bind `net_guard` and `httpx` as package attributes. **Load-bearing risk (dictates where `_get_ai_key` lives).** `tests/unit/services/news.py` monkeypatches names in the news-module namespace: `monkeypatch.setattr(news_mod, "get_setting", ...)` (x10) where `news_mod = devplacepy.services.news`, and `setattr(news_mod.net_guard, "guard_public_url", ...)`, and reads `news_mod.httpx`. `_get_ai_key` resolves `get_setting` from ITS OWN module globals. To keep these tests passing with NO test edits, keep `_get_ai_key` and its `from ...database import get_setting` in the SAME module the test patches - i.e. keep them in `service.py` and ensure the test's `monkeypatch.setattr(news_mod, "get_setting", ...)` still targets the name `_get_ai_key` reads. The safe form: keep `_get_ai_key` + the `get_setting` import together in `service.py`, and have `__init__` do `from .service import get_setting` re-binding so the package-namespace patch and the resolving namespace are the same object. Simplest conservative variant: peel off only the genuinely pure modules (`constants.py`, `models.py`, `clean.py`, `images.py`) and keep `NewsService` + `_get_ai_key` + `get_setting`/`net_guard`/`httpx` imports in `service.py` (which `__init__` re-exports wholesale). This removes most of the size while eliminating the namespace hazard. **Other risks:** - `GRADING_RULES_DESCRIPTION` f-string ordering (constants first). - `_decode_phash` runs via `run_in_executor` - pure, safe to relocate to `images.py`. - BaseService contract: `run_once`/`collect_metrics`/`config_fields`/`interval_key`/`default_enabled`/ `__init__(name="news", interval_seconds=3600)` stay on `NewsService`. Methods using `self.log`/ `self.get_config` cannot become free functions. - `_grade_article_full` is pure but called as `self._grade_article_full(...)`; leave it a method (lowest risk). --- ### 3.10 `services/devii/session.py` (962) -> same-name package `session/` (extract, do not decompose) **Nature.** ONE heavily-shared-mutable-state class `DeviiSession` (`73-851`) plus ~150 lines of stateless constants/pure helpers. Constructed by `hub.py:70-85` with an exact 11-positional + 3-keyword arg signature. `_parse_topic` is imported by `tests/unit/services/devii/session.py:6`. **Recommendation: extract the stateless text/helpers into a package, keep `DeviiSession` whole.** A true class decomposition would require threading `_lock`/`_send_lock`/`_conns`/`_conn_meta`/ `agent._messages`/`self.tools`/`_pending`/`_buffer`/`_turn_epoch` across objects and risks the `self.tools[:]` aliasing and `_turn_epoch` interrupt guard. Not worth it. **Target layout (`services/devii/session/`):** ``` __init__.py # from .core import DeviiSession; from .prompts import _parse_topic (+ optional constants) prompts.py # DOCS_SYSTEM_PROMPT, DOCS_GREETING, DOCS_TOOLS, TOPIC_CLASSIFIER_PROMPT, # NON_ADMIN_COST_RULE, LOGIN_REQUEST, BEHAVIOR_HEADER, _parse_topic(), _system_prompt_for() # (session.py:854-947 + 40 + 67) _helpers.py # _repair_history(), _now_iso(), _format_offset() (43-65, 950-962) core.py # class DeviiSession + the numeric constants only it uses (27-31, 39) ``` This lifts ~150 lines out of the class file with zero behavior risk. `__init__.py` re-exports `DeviiSession` (used by `hub.py:15`) and `_parse_topic` (test). `prompts.py`/`_helpers.py` import nothing from `core.py` (no cycle). All prompt constants are used only inside the class today, so they move freely. Optional (secondary, only if further reduction is wanted): a mixin split of `DeviiSession` into `ConnectionMixin`/`PromptComposition`/`TurnControl`/`SpendTracking`/`Scheduling` operating on one shared `self`. Given the density of cross-group `self.` references (e.g. `_run_turn` touches `_refresh_tools`, `_refresh_system_prompt`, `_docs_topic_gate`, `_cost_snapshot`, `_record_spend`, `_persist_history`, `_emit`, `_turn_epoch`), this trades one long file for cross-mixin coupling - do it only if the primary extraction is not enough. **Invariants that must not break (whichever depth is chosen):** - **`self.tools` mutated in place, never reassigned.** `_refresh_tools` uses `self.tools[:] = ...` because the same list object is bound into `AgenticController`/`Agent`/`react_loop`. Reassigning freezes the tool list (virtual tools stop taking effect). Same in-place discipline for `self.agent._messages` rewrites (`_refresh_system_prompt` `544-547`, `reset_conversation` `357-375`, `_docs_topic_gate` `549-577`, `stop`/`_repair_history` `408-426`). - **Turn-epoch interrupt guard** (`_turn_epoch` at `184`, incremented in `cancel_turns:384`, checked in `_run_turn:451,459,463`) - keep `spawn_turn`/`cancel_turns`/`_run_turn` with the shared `_turn_epoch`/`_turns` state (MEMORY: devii-browser-and-interrupt; do not regress). - **`__init__` positional-arg contract** with `hub.py:70-85` must be preserved exactly. - `session.py` imports `CATALOG` from `.registry` (import-time `_assert_confirm_params`); the package rename must not introduce a cycle - `prompts.py`/`_helpers.py` import nothing from `core.py`. - `from __future__ import annotations` in every extracted module (annotations use `X | None`). --- ### 3.11 `services/game/store.py` (948) -> subpackage `services/game/store/` **Nature.** The DB/stateful layer of the already-package `services/game/` (siblings: `economy.py` pure rules - untouched; `__init__.py` does `from . import economy, store; from .store import GameError`). Only `store.py` needs splitting. Every function funnels through four `get_table` accessors (`_farms/_plots/_quests/_steals`) and `_update_farm`; the only module-level state is `PERK_COLUMN` (built from `economy.PERKS` at import). The game tables carry no `deleted_at` (outside `SOFT_DELETE_TABLES`) - no soft-delete concerns. **Target layout (convert `store.py` -> `store/` package, keeping path `services.game.store`):** ``` store/ __init__.py # re-exports EVERY name currently on store.py, incl. private helpers tests call common.py # GameError, _now, _iso, _parse, _today, _parse_date, _farms, _plots, _quests, # _steals, _lvl, PERK_COLUMN, _update_farm, last_steal_at, steal_cooldown_remaining farm.py # get_farm, ensure_farm, _create_plot, get_plots, _plot_at, buy_plot, upgrade_ci, leaderboard serialize.py # _watered_by, _plot_state, serialize_plot, serialize_farm, _serialize_perks, # _serialize_legacy, _serialize_quests, _daily_available quests.py # ensure_quests, advance_quests actions.py # plant, harvest, water, steal, fertilize, prestige, upgrade_perk, upgrade_legacy, # claim_daily, claim_quest, _auto_harvest ``` `__init__.py` re-export must include the router surface (`ensure_farm`, `serialize_farm`, `plant`, `harvest`, `buy_plot`, `upgrade_ci`, `water`, `steal`, `leaderboard`, `claim_daily`, `claim_quest`, `upgrade_perk`, `upgrade_legacy`, `prestige`, `fertilize`, `GameError`) **plus the private symbols the unit test calls directly**: `get_farm`, `get_plots`, `ensure_quests`, `advance_quests`, `_plot_at`, `_today`. `services/game/__init__.py` keeps working unchanged. Importers: `routers/game/index.py:18`, `routers/game/farm.py:11`, `routers/game/_shared.py:6` (all `from devplacepy.services.game import GameError, store` then `store.X`). No CLI, no `main.py`. **Risks:** - **Re-export private helpers** (`_plot_at`, `_today`, + `get_farm`/`get_plots`/`ensure_quests`/ `advance_quests`) or the unit test breaks. No monkeypatch-of-module-name coupling here (unlike news), so the only hazard is a missing re-export. - **`PERK_COLUMN`** built from `economy.PERKS` at import must live in the low-level shared module (`common.py`) imported by `serialize.py`/`actions.py`; `common.py` imports `economy`, higher modules import `common` + `economy` (no cycle). - **Dense intra-module call graph** (`serialize_farm -> _auto_harvest -> advance_quests`; `actions -> ensure_farm/_update_farm/advance_quests`) - group so lower-level modules (`common`/`quests`/`farm`) are imported by higher-level (`serialize`/`actions`), not vice-versa. - **`leaderboard` lazy import** `from devplacepy.database import get_users_by_uids` at `store.py:593` - keep function-local when relocating. --- ## 4. SPECIAL CASE - container standalone scripts (files 4 and 5) `services/containers/files/bot.py` (2940, `COPY bot.py /usr/bin/botje.py`) and `services/containers/files/d.py` (2337, `COPY d.py /usr/bin/d.py`) are NOT server backend. They are standalone single-file programs baked into the `ppy` image (`ppy.Dockerfile:47-48`, build context `devplacepy/services/containers/files`, invoked `python /usr/bin/botje.py`). Their only hard third-party dep is `httpx`. **Nothing in `devplacepy/` or `tests/` imports either file** (confirmed grep) - they are not on `sys.path` as modules and have no test coverage. The only breakage surface is the Docker build's COPY path and the documented invocation. **Binding constraint.** A normal package split (like the server-side files) is **incompatible** with the single-file COPY + bare-`python` exec contract: a package leaf COPY'd to `/usr/bin/botje.py` would fail at the first cross-module `from botpkg... import` (no package root on the container's path). So the in-place package approach used for every other monster does not apply here. **Options (pick per user direction; this plan does not choose for you):** - **(a) Keep as single-file monoliths.** Zero Dockerfile/Makefile change; the contract stays intact. Trade-off: the two files remain large and duplicate ~2000 lines of agent code between them (`files/bot.py` = `files/d.py`'s agent + a DevPlace XML-RPC bot loop; `files/d.py`'s stealth half is a pure-httpx sibling of `devplacepy/stealth.py`). This is the lowest-risk option. - **(b) Split into a source package + add a build step** that re-stitches the single deployed artifact. Two sub-options, each with concrete required edits: - **zipapp:** author `files/agent/` as a package, build with `python -m zipapp files/agent -m "agent.cli:main" -o build/botje.pyz`. Requires changing `ppy.Dockerfile:47-48` COPY source + extension (`.pyz`), the `chmod`/`ln` lines (`:52`), the invocation in docs (`templates/docs/getting-started-vibing.html:128-162`) from `python /usr/bin/ botje.py` to `python /usr/bin/botje.pyz`, and the `make ppy` target (`Makefile:129-130`) to run the bundler before `docker build`. Downside: breaks the documented `cp /usr/bin/botje.py /app/mybot.py` copy-and-edit single-file flow. - **concatenation/stitch:** a `make` recipe concatenates the package modules (deduped imports, in dependency order) into one `build/bot.py`, which the Dockerfile then COPYs. Keeps the `.py` extension and `python /usr/bin/botje.py` intact; users edit generated output, not source. Requires `Makefile:129` (add the bundler step + widen the rebuild-trigger set) and `ppy.Dockerfile:47-48` (COPY from `build/`). **Note (not a recommendation, for context):** the largest issue with these two files is duplication, not size - `files/bot.py` and `files/d.py` share the entire agent (tool framework, `react_loop`, `CorpusIndex`, embedded `ChromeStealthClient`). If the user later wants these deduplicated rather than split, that is a different task; this plan records the fact only. --- ## 5. Cross-file execution order (recommended) Order by blast radius (lowest first) so a mistake surfaces on a small surface before the wide ones, and so the widely-imported files are split last against a known-green tree: 1. **`services/game/store.py`** - self-contained subpackage, 3 importers, no cycles. Warm-up. 2. **`services/devii/actions/catalog.py`** - pure data, 2 exported symbols, ordering-only risk. 3. **`schemas.py`** - pure models, mechanical, wide but simple re-export. 4. **`services/news.py`** - watch the monkeypatch namespace; conservative variant recommended. 5. **`services/devii/session.py`** - extract stateless helpers only (keep class whole). 6. **`services/bot/bot.py`** - mixin lift; keep `__init__` exporting only `BotsService`. 7. **`cli.py`** - package with per-group modules; keep the console-script entry. 8. **`docs_api.py`** - large but declarative; handle the three ordering hazards. 9. **`utils.py`** - 224 importers, funnel + lazy-import + Jinja-global sensitivity. 10. **`database.py`** - 329 importers, the core layer; do it last against a green tree. 11. **Container `files/bot.py` / `files/d.py`** - only if the user opts into option (b); otherwise leave as-is per Section 4. Each step is independent: split one file, run the Section-2.9 validation, commit, move on. Do not batch multiple monster splits into one unvalidated change. --- ## 6. Convention alignment (why packages, not `_`-suffixed files) This plan uses the project's existing structural idiom, verified against the current tree: - Router packages (`routers/issues/`, `routers/admin/`, `routers/profile/`, `routers/projects/`) are directories whose `__init__.py` aggregates leaf modules and re-exports the public symbol, each new file headed `# retoor `. Every split above mirrors that. - `_shared.py` is the sanctioned package-private helper module (already used in eight router packages); it is used here only for genuine cross-leaf construction helpers (`docs_api/_shared.py`, `catalog/_shared.py`, `cli/_shared.py`), never as a dumping ground. - No `_v2`/`_refactored`/`_new` suffixes anywhere (forbidden by project rules). The compatibility mechanism is the same-name package + re-export, not a parallel file. - `services/` already contains packages that expose a `store` submodule (`services/audit/`, `services/backup/`, `services/containers/`, `services/game/`), so `services/news/` and `services/game/store/` and `services/devii/session/` and `services/devii/actions/catalog/` all follow precedent. --- ## 7. Documentation follow-up (after any split is executed) Per the project feature workflow, once a split is executed the structural change is documented in the three canonical files - but ONLY if a rule/convention changes. A pure module move that preserves every import path introduces no new architectural rule, so: - `README.md`: no change (no user-visible/route/env/dep change). - `AGENTS.md`: add a short note under the relevant domain only if the new module boundary is worth navigational guidance (e.g. "docs_api is now a package, one module per API group"). - `CLAUDE.md`: no change unless the split establishes a NEW convention (it does not - it follows the existing package idiom). This section is a reminder, not a task; the plan itself does not edit docs. --- ## 8. Summary table - target packages | Source file | Target | Split style | Key invariant to protect | |-------------|--------|-------------|--------------------------| | `docs_api.py` | `docs_api/` (+`groups/`) | one module per API group | import-time negotiation pass; group order; `_substitute` re-export | | `database.py` | `database/` | layered domain modules on `core.py` | `db` singleton in `core`; underscore re-exports; keep lazy imports | | `services/bot/bot.py` | mixins in `services/bot/` | cooperative mixins | realism contract; `__init__` exports only `BotsService`; stealth rule | | `services/devii/actions/catalog.py` | `catalog/` (+`_shared.py`) | one module per action domain | `ACTIONS` order; `confirm`-param invariant; `Param` import in dbapi/gateway/game | | `schemas.py` | `schemas/` | domain schema modules | `CommentItemOut.model_rebuild()`; cross-module inheritance order | | `cli.py` | `cli/` (+`_shared.py`) | one module per command group | console-script `cli:main`; re-export all `cmd_*`; lazy imports | | `utils.py` | `utils/` | layered helper modules | funnel pairing; lazy imports; Jinja-global paths; `"auth"` cache key | | `services/news.py` | `services/news/` | pipeline-stage modules | monkeypatch namespace (`_get_ai_key`/`get_setting`); f-string const order | | `services/devii/session.py` | `session/` | extract stateless text/helpers | `self.tools[:]` in-place; `_turn_epoch` guard; `__init__` arg contract | | `services/game/store.py` | `services/game/store/` | cohesive store modules | re-export private test helpers; `PERK_COLUMN` placement | | `services/containers/files/bot.py` | SPECIAL (Section 4) | single-file OR build step | COPY-single-file + bare-exec contract | | `services/containers/files/d.py` | SPECIAL (Section 4) | single-file OR build step | COPY-single-file + bare-exec contract |