feat: enforce hard test-tier requirement across all DevPlace workflow agents and feature-builder docs
DevPlace CI / test (push) Failing after 21m53s

Update the feature-builder agent prompt, test-maintainer agent, and all four workflow JS files (devii-tool, endpoint, feature, job-service) to codify the DevPlace test standard as a non-optional project requirement: one test file per endpoint, directory tree mirroring the URL/source path, split into three tiers (unit, api, e2e). Add explicit Test phases to devii-tool, endpoint, feature, and job-service workflows, and embed tier-specific test instructions (path mapping, fixture choice, coverage scope) directly in each workflow's meta description and TESTS constant.
This commit is contained in:
2026-06-15 12:10:14 +00:00
parent 3c7527988e
commit e1874f9b6a
71 changed files with 2198 additions and 145 deletions
+7 -1
View File
@@ -35,7 +35,7 @@ A DevPlace feature is one data source fanning out into several consumers, all fr
6. **Agent + docs (the most-forgotten layers)** - if a user could ask Devii to do it, add an `Action` in `services/devii/actions/catalog.py` with auth flags matched to the route guard (and a declared `confirm` boolean for any irreversible action added to `CONFIRM_REQUIRED`). Add a `docs_api.py` `endpoint()` entry (params + `sample_response`) for every public/auth endpoint; add a prose page to `routers/docs/pages.py` `DOCS_PAGES` when warranted. State-changing actions need an audit event (`events.md` key, `category_for`, recorder call at the mutation point).
7. **SEO** - public pages build `base_seo_context` and the right JSON-LD; add to `routers/seo.py` sitemap when indexable.
8. **Docs of record** - update `README.md` (product-facing) and `AGENTS.md` (deep companion) for any new route/config/dependency/mechanic; update `CLAUDE.md` only when a NEW architectural rule or convention is introduced.
9. **Tests** - add tests in the correct tier and path (`tests/{unit,api,e2e}/<endpoint>.py`, directory tree mirrors the URL/source path) following the required Playwright patterns. WRITE them; NEVER run them.
9. **Tests (a hard project requirement, never optional)** - the DevPlace suite is one test file per endpoint, ~932 tests, split into three tiers with the directory tree mirroring the URL/source path. Every feature gets a test in EVERY tier it exercises: `tests/unit/` for a new data/query helper (pure in-process, `local_db` or no fixture, path mirrors the SOURCE module - `devplacepy/utils.py` -> `tests/unit/utils.py`); `tests/api/` for a new JSON or HTML route (HTTP integration against the live uvicorn subprocess via `app_server`/`seeded_db`, path mirrors the endpoint - `POST /auth/login` -> `tests/api/auth/login.py`); `tests/e2e/` for a new interactive UI flow (Playwright `page`/`alice`/`bob`, path mirrors the endpoint - `GET /admin/ai-usage` -> `tests/e2e/admin/aiusage.py`). A route or feature with no test in any tier is incomplete. Follow the required patterns (`wait_until="domcontentloaded"` on every `goto`/`wait_for_url`, scoped selectors, `try/finally` restore of any flipped global setting, the shared fixtures, `test_`-prefixed functions in non-prefixed files, born-live `deleted_at`/`deleted_by` on raw soft-delete inserts) and create any missing package directories (`__init__.py`). WRITE them; validate each by a clean import only; NEVER run them.
When a layer is intentionally absent (an internal route with no public docs, a route Devii must never call), say so explicitly in the plan with the rationale rather than fabricating the layer.
@@ -51,6 +51,12 @@ No comments or docstrings in source you author; full typing on every signature a
## Validation (after implementing; never skip)
Run `hawk .` and confirm zero errors. Confirm `python -c "from devplacepy.main import app"` imports clean. Grep your touched files for em-dashes and confirm none. Do NOT run the test suite. Then hand off: name which maintainer dimensions are most relevant to the change (e.g. fanout, security, dry, docs, seo, audit, frontend, style, test) so the fleet can verify it.
## Live verification of UI/API changes (mandatory for visual work)
A structurally valid template can still render broken - `hawk` and the import check never open a browser. Per CLAUDE.md this project treats live verification as non-negotiable for any layout, styling, component, responsive, or backend change:
- When your change touches `templates/` or `static/`, the rendered result MUST be visually verified: start the dev server (`make dev` in the background; confirm with `mole check http://localhost:10500`), capture each new/changed route with headless Playwright (`wait_until="domcontentloaded"`), run `falcon describe <png>`, and compare the description against the intended UI and the surrounding design system (tokens, spacing, responsiveness). Tear down any server you started.
- When your change touches `routers/`, verify the endpoints over HTTP with a `hound` JSON spec against the live server.
- The `/feature` workflow performs this live `Verify` phase for you automatically; when you are invoked standalone for UI/API work, perform it yourself before declaring the work complete, or explicitly state it is the caller's responsibility and name the routes to check.
## Output
- In PLAN mode: a short situation summary of the area, then the ordered layer-by-layer plan (each layer: what file, what change, or "n/a - rationale"), then the list of maintainer dimensions that will need to verify it. End by asking for approval to implement.
- In IMPLEMENT mode: a concise summary of what was built per layer (`file:line` references), the validation results (`hawk`, import, em-dash scan), and the recommended maintainer hand-off.
+13 -5
View File
@@ -36,15 +36,23 @@ Default to **REPORT** mode: record coverage gaps and pattern violations, do NOT
No comments or docstrings in source you author; no em-dashes (use a hyphen); keep `retoor <retoor@molodetz.nl>` as the first line of any file you create.
## Your dimension
Keep integration-test coverage in step with the routes and features, writing tests that follow the project's required patterns.
Keep integration-test coverage in step with the routes and features. The DevPlace suite is a hard project standard, not a nicety: one test file per endpoint, ~932 tests, split into three tiers with the directory tree mirroring the URL/source path. A route or feature that exercises a tier with no test in it is a coverage gap.
DETECT: routes and features with no corresponding test under `tests/{api,e2e,unit}/<endpoint-path>.py` (per the directory-mirrors-path naming rule). The project prefers integration tests over unit tests and tests the interface and API.
The three tiers and which one a change belongs to (decided by what it exercises, mirroring the existing files):
- **`tests/unit/`** - pure in-process tests of library functions (`local_db` or no fixture); the path mirrors the SOURCE module (`devplacepy/utils.py` -> `tests/unit/utils.py`, `devplacepy/services/audit/store.py` -> `tests/unit/services/audit/store.py`). The right tier for a new data/query/serialization helper.
- **`tests/api/`** - HTTP integration tests against the live uvicorn subprocess (`app_server`/`seeded_db`, `requests`/`httpx` vs `BASE_URL`, no browser); the path mirrors the endpoint (`POST /auth/login` -> `tests/api/auth/login.py`). The right tier for a JSON or HTML route, auth/role gating, and Devii actions.
- **`tests/e2e/`** - Playwright browser tests (`page`/`alice`/`bob`); the path mirrors the endpoint (`GET /admin/ai-usage` -> `tests/e2e/admin/aiusage.py`). The right tier for an interactive UI flow. The project prefers the interface/API tiers over unit where either fits.
FIX: write the missing integration test following the required patterns: every `page.goto` and `page.wait_for_url` passes `wait_until="domcontentloaded"`; selectors are scoped; a test that flips a global `site_settings` value restores it in `try/finally`; the shared fixtures (`alice`, `bob`, `app_server`) are used; test functions are `test_`-prefixed though files are not.
A feature that adds a data helper AND a JSON route AND a UI flow needs a test in all three tiers. Choose the tier(s) by what the change actually touches; never leave a new route or helper untested.
DETECT: routes, features, data helpers, and Devii actions with no corresponding test in the tier(s) they exercise under `tests/{unit,api,e2e}/<path>.py` (per the directory-mirrors-path naming rule). A collection path that also parents deeper paths uses `index.py` in its own directory.
FIX: write the missing test in the correct tier, creating any missing package directories (`__init__.py`), following the required patterns: every `page.goto` and `page.wait_for_url` passes `wait_until="domcontentloaded"`; selectors are scoped; a test that flips a global `site_settings` value restores it in `try/finally`; the shared fixtures (`alice`, `bob`, `app_server`, `seeded_db`) are used; test functions are `test_`-prefixed though files are not; a raw insert into a `SOFT_DELETE_TABLES` table sets `deleted_at`/`deleted_by`; a test that mutates a cross-process cached value (settings/roles) polls the endpoint rather than asserting immediately.
## Scope units
- **coverage-gaps**: `routers/*.py` routes with no referencing test under `tests/{api,e2e,unit}/`.
- **pattern-lint**: `tests/*.py` use `domcontentloaded`, scoped selectors, try/finally global restore, shared fixtures.
- **coverage-gaps**: `routers/*.py` routes, `database.py`/service data helpers, and `services/devii/actions/catalog.py` actions with no referencing test in the tier(s) they exercise under `tests/{unit,api,e2e}/`.
- **tier-fit**: a feature exercising a tier (a UI flow with only an api test, a data helper with no unit test) where that tier's test is missing.
- **pattern-lint**: `tests/*.py` use `domcontentloaded`, scoped selectors, try/finally global restore, shared fixtures, born-live soft-delete inserts.
## Output
Return a markdown report: a one-line summary line, then one bullet per finding with severity (`error`/`warning`/`info`), `file:line`, the rule name, the message, and (in fix mode) whether the test was written.