feat: enforce hard test-tier requirement across all DevPlace workflow agents and feature-builder docs
DevPlace CI / test (push) Failing after 21m53s
DevPlace CI / test (push) Failing after 21m53s
Update the feature-builder agent prompt, test-maintainer agent, and all four workflow JS files (devii-tool, endpoint, feature, job-service) to codify the DevPlace test standard as a non-optional project requirement: one test file per endpoint, directory tree mirroring the URL/source path, split into three tiers (unit, api, e2e). Add explicit Test phases to devii-tool, endpoint, feature, and job-service workflows, and embed tier-specific test instructions (path mapping, fixture choice, coverage scope) directly in each workflow's meta description and TESTS constant.
This commit is contained in:
+148
-26
@@ -1,13 +1,15 @@
|
||||
// retoor <retoor@molodetz.nl>
|
||||
export const meta = {
|
||||
name: 'feature',
|
||||
description: 'Add a feature across the full DevPlace fan-out: understand the area, plan the layers, implement coherently in the repo, audit completeness and security, fix gaps and write tests, then verify',
|
||||
description: 'Add a feature across the full DevPlace fan-out: understand the area, plan the layers, build via the feature-builder agent, audit every quality dimension with adversarial verification, verify live in the browser and over HTTP, close gaps, then write the integration tests across every applicable tier (unit, api, e2e)',
|
||||
phases: [
|
||||
{ title: 'Understand', detail: 'map the target area and a similar existing feature' },
|
||||
{ title: 'Plan', detail: 'a per-layer implementation plan across the nine touchpoints' },
|
||||
{ title: 'Implement', detail: 'build all layers coherently in the repo' },
|
||||
{ title: 'Audit', detail: 'completeness and security review of the changed files' },
|
||||
{ title: 'Fix', detail: 'close audit gaps and write missing integration tests' },
|
||||
{ title: 'Implement', detail: 'build all layers coherently via the feature-builder agent' },
|
||||
{ title: 'Audit', detail: 'every relevant quality dimension, each finding adversarially verified against source' },
|
||||
{ title: 'Verify', detail: 'live dev-server visual (falcon) and API (hound) verification of the change' },
|
||||
{ title: 'Fix', detail: 'close confirmed gaps from the audit and live verification' },
|
||||
{ title: 'Test', detail: 'write integration tests across every applicable tier (unit, api, e2e), one file per endpoint mirroring the path' },
|
||||
],
|
||||
}
|
||||
|
||||
@@ -17,10 +19,10 @@ const RULES = [
|
||||
'- First line of any NEW file is the header: Python "# retoor <retoor@molodetz.nl>", JS "// retoor <retoor@molodetz.nl>", CSS "/* retoor <retoor@molodetz.nl> */".',
|
||||
'- No em-dash characters; use a hyphen. Source is English only.',
|
||||
'- Full type hints on Python signatures and variables; pathlib over os; Pydantic Form input with explicit max lengths; sanitize and bound all user input.',
|
||||
'- Reuse shared helpers: templating.templates (never a per-router Jinja2Templates), database.py batch helpers (no inline N+1), the respond() negotiator, _avatar_link.html / _user_link.html, and on the frontend Http / Poller / JobPoller / OptimisticAction / FloatingWindow.',
|
||||
'- Reuse shared helpers: templating.templates (never a per-router Jinja2Templates), database.py batch helpers (no inline N+1), the respond() negotiator, _avatar_link.html / _user_link.html, and on the frontend Http / Poller / JobPoller / OptimisticAction / FloatingWindow and the dp-* components.',
|
||||
'- Auth guards: get_current_user (public read), require_user (member write), require_admin (admin). Every POST/PUT/DELETE is guarded; deletes are soft and owner-or-admin.',
|
||||
'- Never pass a respond() context key that collides with a Jinja global (use viewer_is_admin, not is_admin). Dates are DD/MM/YYYY via format_date.',
|
||||
'- Validate with "hawk ." and "python -c \\"from devplacepy.main import app\\"". NEVER run the test suite. Never perform any git write.',
|
||||
'- Validate with "hawk ." and "python -c \\"from devplacepy.main import app\\"". NEVER run the pytest suite. Never perform any git write.',
|
||||
].join('\n')
|
||||
|
||||
const FANOUT = [
|
||||
@@ -36,6 +38,16 @@ const FANOUT = [
|
||||
'9. README.md (product) + AGENTS.md (mechanics) + CLAUDE.md (only for a genuinely new architectural rule).',
|
||||
].join('\n')
|
||||
|
||||
const TESTS = [
|
||||
'DevPlace test standard (a hard project requirement, NOT optional - the suite is one test file per endpoint, ~932 tests, with the directory tree mirroring the URL/source path):',
|
||||
'- tests/unit/ - pure in-process tests of library functions (local_db or no fixture); the path mirrors the SOURCE module (devplacepy.utils -> tests/unit/utils.py).',
|
||||
'- tests/api/ - HTTP integration tests against the live uvicorn subprocess (app_server/seeded_db, requests/httpx vs BASE_URL, no browser); the path mirrors the endpoint (POST /auth/login -> tests/api/auth/login.py).',
|
||||
'- tests/e2e/ - Playwright browser tests (page/alice/bob); the path mirrors the endpoint (GET /admin/ai-usage -> tests/e2e/admin/aiusage.py).',
|
||||
'A feature MUST get every tier it exercises: a new data/query helper -> a unit test; a new JSON or HTML route -> an api test; a new interactive UI flow -> an e2e test. Pick tiers by what the change actually touches; never ship a route or feature with no test in any tier.',
|
||||
'Required patterns: every page.goto/page.wait_for_url passes wait_until="domcontentloaded"; selectors are scoped; a test that flips a global site_settings value restores it in try/finally; reuse the shared fixtures (alice, bob, app_server, seeded_db); test FUNCTIONS are test_-prefixed though files are not; raw inserts into a soft-delete table set deleted_at/deleted_by.',
|
||||
'Validate each new test module by a clean import only (python -c "import ..." or python -m py_compile). NEVER run the suite, not the full suite and not one file - that is the human-only /test path.',
|
||||
].join('\n')
|
||||
|
||||
function featureBrief() {
|
||||
if (!args) return ''
|
||||
if (typeof args === 'string') return args
|
||||
@@ -81,6 +93,7 @@ const PLAN_SCHEMA = {
|
||||
},
|
||||
},
|
||||
},
|
||||
routes: { type: 'array', items: { type: 'string' } },
|
||||
outOfScope: { type: 'array', items: { type: 'string' } },
|
||||
},
|
||||
}
|
||||
@@ -94,6 +107,7 @@ const BUILD_SCHEMA = {
|
||||
filesChanged: { type: 'array', items: { type: 'string' } },
|
||||
validatorPassed: { type: 'boolean' },
|
||||
importOk: { type: 'boolean' },
|
||||
routes: { type: 'array', items: { type: 'string' } },
|
||||
notes: { type: 'string' },
|
||||
},
|
||||
}
|
||||
@@ -122,6 +136,42 @@ const FINDINGS_SCHEMA = {
|
||||
},
|
||||
}
|
||||
|
||||
const VERDICT_SCHEMA = {
|
||||
type: 'object',
|
||||
additionalProperties: false,
|
||||
required: ['isReal', 'reason'],
|
||||
properties: {
|
||||
isReal: { type: 'boolean' },
|
||||
reason: { type: 'string' },
|
||||
severity: { type: 'string', enum: ['error', 'warning', 'info'] },
|
||||
},
|
||||
}
|
||||
|
||||
const LIVE_SCHEMA = {
|
||||
type: 'object',
|
||||
additionalProperties: false,
|
||||
required: ['ran', 'summary'],
|
||||
properties: {
|
||||
ran: { type: 'boolean' },
|
||||
summary: { type: 'string' },
|
||||
pagesChecked: { type: 'array', items: { type: 'string' } },
|
||||
apiChecked: { type: 'array', items: { type: 'string' } },
|
||||
issues: {
|
||||
type: 'array',
|
||||
items: {
|
||||
type: 'object',
|
||||
additionalProperties: false,
|
||||
required: ['severity', 'where', 'message'],
|
||||
properties: {
|
||||
severity: { type: 'string', enum: ['error', 'warning', 'info'] },
|
||||
where: { type: 'string' },
|
||||
message: { type: 'string' },
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
log(`Feature: ${ask}`)
|
||||
|
||||
const map = await agent(
|
||||
@@ -130,16 +180,17 @@ const map = await agent(
|
||||
)
|
||||
|
||||
const plan = await agent(
|
||||
`Produce a precise, per-layer implementation plan for this DevPlace feature. One step per file with the exact touchpoint to add or change. Mark layers that are intentionally not needed as outOfScope with a reason. Do not write code.\n\nFeature request: ${ask}\n\nArea map:\n${JSON.stringify(map, null, 2)}\n\n${FANOUT}`,
|
||||
`Produce a precise, per-layer implementation plan for this DevPlace feature. One step per file with the exact touchpoint to add or change. List the user-facing routes (URL paths) the feature adds or changes in "routes". Mark layers that are intentionally not needed as outOfScope with a reason. Do not write code.\n\nFeature request: ${ask}\n\nArea map:\n${JSON.stringify(map, null, 2)}\n\n${FANOUT}`,
|
||||
{ agentType: 'Plan', label: 'plan', phase: 'Plan', schema: PLAN_SCHEMA }
|
||||
)
|
||||
|
||||
const build = await agent(
|
||||
`Implement this DevPlace feature coherently and completely, editing files directly in the repo. Follow the plan; keep every layer in agreement (the *Out schema must carry every JSON key the handler returns; the Devii action auth flags must match the route guard). When done, run "hawk ." and "python -c \\"from devplacepy.main import app\\"" and report whether each passed. Do not write tests in this step. Do not run the test suite. Do not commit.\n\nFeature request: ${ask}\n\nPlan:\n${JSON.stringify(plan, null, 2)}\n\n${FANOUT}\n\n${RULES}\n\nReturn the list of files you changed or created, whether the validator and the import passed, and a short summary.`,
|
||||
{ label: 'implement', phase: 'Implement', schema: BUILD_SCHEMA }
|
||||
`Implement directly - no plan, no approval needed, this is implement mode. Build this DevPlace feature coherently and completely, editing files in the repo, following the plan. Keep every layer in agreement (the *Out schema must carry every JSON key the handler returns; the Devii action auth flags must match the route guard; a respond() context key must never shadow a Jinja global). Do NOT write pytest tests in this step (a later phase owns that). When done, run "hawk ." and "python -c \\"from devplacepy.main import app\\"" and report whether each passed, and list the user-facing routes the feature exposes.\n\nFeature request: ${ask}\n\nPlan:\n${JSON.stringify(plan, null, 2)}\n\n${FANOUT}\n\n${RULES}\n\nReturn the list of files you changed or created, whether the validator and the import passed, the routes, and a short summary.`,
|
||||
{ agentType: 'feature-builder', label: 'implement', phase: 'Implement', schema: BUILD_SCHEMA }
|
||||
)
|
||||
|
||||
const changed = (build && build.filesChanged) || []
|
||||
const routes = (build && build.routes && build.routes.length ? build.routes : (plan && plan.routes) || [])
|
||||
const scopeNote = changed.length
|
||||
? `\n\nRestrict your findings to these changed files (read others only for cross-reference):\n${changed.join('\n')}`
|
||||
: ''
|
||||
@@ -147,36 +198,107 @@ const scopeNote = changed.length
|
||||
const AUDITORS = [
|
||||
{ key: 'fanout', agent: 'fanout-maintainer' },
|
||||
{ key: 'security', agent: 'security-maintainer' },
|
||||
{ key: 'style', agent: 'style-maintainer' },
|
||||
{ key: 'dry', agent: 'dry-maintainer' },
|
||||
{ key: 'frontend', agent: 'frontend-maintainer' },
|
||||
{ key: 'seo', agent: 'seo-maintainer' },
|
||||
{ key: 'audit', agent: 'audit-maintainer' },
|
||||
{ key: 'devii', agent: 'devii-maintainer' },
|
||||
{ key: 'docs', agent: 'docs-maintainer' },
|
||||
]
|
||||
|
||||
const audits = await parallel(
|
||||
AUDITORS.map((a) => () =>
|
||||
agent(
|
||||
`Operate in REPORT mode (read-only). Audit the just-implemented feature for your single dimension. Confirm each finding against the source.${scopeNote}\n\nFeature request: ${ask}`,
|
||||
{ agentType: a.agent, label: `audit:${a.key}`, phase: 'Audit', schema: FINDINGS_SCHEMA }
|
||||
).then((r) => ({ key: a.key, findings: (r && r.findings) || [] }))
|
||||
function verifyPrompt(dimension, finding) {
|
||||
return (
|
||||
`Adversarially verify a candidate "${dimension}" finding against the just-built feature. Your goal is to REFUTE it. ` +
|
||||
`Open the exact file and read enough surrounding context (the whole function, the caller, the contract) to judge intent. ` +
|
||||
`It is REAL only if it survives refutation as a genuine violation of the ${dimension} dimension introduced by this change. ` +
|
||||
`Rule it out (isReal=false) if it is a contract identifier, DATA rather than authored prose, generated/vendored/third-party, ` +
|
||||
`pre-existing and untouched by this feature, or already correct under a known exemption. When uncertain, default to isReal=false.\n\n` +
|
||||
`Candidate finding:\n- file: ${finding.file}\n- line: ${finding.line == null ? 'unspecified' : finding.line}\n` +
|
||||
`- severity: ${finding.severity}\n- rule: ${finding.rule}\n- message: ${finding.message}\n\nReturn isReal and a one-line reason.`
|
||||
)
|
||||
}
|
||||
|
||||
const reviewed = await pipeline(
|
||||
AUDITORS,
|
||||
(auditor) =>
|
||||
agent(
|
||||
`Operate in REPORT mode (read-only). Do not modify any file. Audit the just-implemented feature for your single quality dimension, following your mandate and accuracy doctrine. Confirm each candidate against the actual source before recording it.${scopeNote}\n\nFeature request: ${ask}`,
|
||||
{ agentType: auditor.agent, label: `audit:${auditor.key}`, phase: 'Audit', schema: FINDINGS_SCHEMA }
|
||||
),
|
||||
(review, auditor) =>
|
||||
parallel(
|
||||
((review && review.findings) || []).map((finding) => () =>
|
||||
agent(verifyPrompt(auditor.key, finding), {
|
||||
agentType: auditor.agent,
|
||||
label: `verify:${auditor.key}`,
|
||||
phase: 'Audit',
|
||||
schema: VERDICT_SCHEMA,
|
||||
}).then((verdict) => ({ ...finding, dimension: auditor.key, verdict }))
|
||||
)
|
||||
)
|
||||
)
|
||||
|
||||
const gaps = audits
|
||||
.filter(Boolean)
|
||||
.flatMap((r) => r.findings.map((f) => ({ dimension: r.key, ...f })))
|
||||
.filter((f) => f.severity !== 'info')
|
||||
const auditCandidates = reviewed.flat().filter(Boolean)
|
||||
const auditConfirmed = auditCandidates.filter((f) => f.verdict && f.verdict.isReal)
|
||||
log(`Audit: ${auditConfirmed.length} confirmed of ${auditCandidates.length} candidate finding(s) across ${AUDITORS.length} dimensions`)
|
||||
|
||||
let gapFix = 'no actionable gaps from the audit'
|
||||
const touchedFrontend = changed.some((f) => f.includes('/templates/') || f.includes('/static/'))
|
||||
const touchedApi = changed.some((f) => f.includes('/routers/'))
|
||||
|
||||
let live = { ran: false, summary: 'no frontend or API files changed; live verification skipped', issues: [] }
|
||||
if (touchedFrontend || touchedApi) {
|
||||
const kinds = [touchedFrontend ? 'visual (falcon)' : null, touchedApi ? 'API (hound)' : null].filter(Boolean).join(' and ')
|
||||
live = await agent(
|
||||
`Operate the MANDATORY DevPlace live verification (${kinds}) for the just-built feature, exactly per CLAUDE.md.\n\n` +
|
||||
`Procedure:\n` +
|
||||
`1. Check if the dev server already answers: "mole check http://localhost:10500". If it does NOT, start it yourself with "make dev" as a BACKGROUND process, then poll "mole check http://localhost:10500" until healthy (give uvicorn a few seconds to boot). Remember whether YOU started it.\n` +
|
||||
(touchedFrontend
|
||||
? `2. VISUAL: for each user-facing route the feature adds or changes, capture a screenshot with the installed Playwright (chromium, headless) navigating to "http://localhost:10500<route>" with wait_until="domcontentloaded", saving a PNG under /tmp/, then run "falcon describe <png>". Compare each AI description against the intended UI and the surrounding design system (layout, spacing, design tokens, responsiveness). Record any mismatch, broken layout, missing element, or visual regression as an issue. Authenticated routes: log in via the /auth/login form first (seeded users may not exist on a fresh dev DB - if a route needs auth and you cannot reach it, record that as an info issue rather than failing).\n`
|
||||
: '') +
|
||||
(touchedApi
|
||||
? `3. API: write a hound JSON spec (tests: name/method/path/expect_status[/expect_body/expect_headers]) covering the feature's endpoints with realistic expected statuses, then run "hound <spec>.json --base-url http://localhost:10500". Record every failing assertion as an issue.\n`
|
||||
: '') +
|
||||
`4. TEARDOWN: if YOU started the server, kill it now (do not leave a stray uvicorn running). If it was already running, leave it.\n\n` +
|
||||
`Routes for this feature: ${routes.length ? routes.join(', ') : '(infer from the changed routers/templates below)'}\n` +
|
||||
`Changed files:\n${changed.join('\n')}\n\n` +
|
||||
`Return ran=true, the pages and api endpoints you checked, and one issue per real visual/functional defect (severity/where/message). Do not edit feature source in this phase; only report.`,
|
||||
{ label: 'live-verify', phase: 'Verify', schema: LIVE_SCHEMA }
|
||||
)
|
||||
log(`Live verify: ${(live && live.issues && live.issues.length) || 0} issue(s) over ${((live && live.pagesChecked) || []).length} page(s)`)
|
||||
}
|
||||
|
||||
const gaps = []
|
||||
for (const f of auditConfirmed) {
|
||||
if (f.severity !== 'info') gaps.push({ source: f.dimension, file: f.file, line: f.line, rule: f.rule, message: f.message })
|
||||
}
|
||||
for (const i of (live && live.issues) || []) {
|
||||
if (i.severity !== 'info') gaps.push({ source: 'live-verify', file: i.where, rule: 'live', message: i.message })
|
||||
}
|
||||
|
||||
let gapFix = 'no actionable gaps from the audit or live verification'
|
||||
if (gaps.length) {
|
||||
gapFix = await agent(
|
||||
`Close these completeness and security gaps found by the audit of the new feature. Apply minimal root-cause fixes directly in the repo, keeping all layers in agreement. Re-run "hawk ." afterward. Do not run the test suite. Do not commit.\n\nGaps:\n${JSON.stringify(gaps, null, 2)}\n\n${RULES}`,
|
||||
{ label: 'fix-gaps', phase: 'Fix' }
|
||||
`Close these confirmed completeness, security, style, frontend, and live-rendering gaps found in the new feature. Apply minimal root-cause fixes directly in the repo, keeping all layers in agreement and the styling consistent with the design system. Re-run "hawk ." afterward. Do not run the pytest suite. Do not commit.\n\nGaps:\n${JSON.stringify(gaps, null, 2)}\n\n${RULES}`,
|
||||
{ agentType: 'feature-builder', label: 'fix-gaps', phase: 'Fix' }
|
||||
)
|
||||
}
|
||||
|
||||
const tests = await agent(
|
||||
`Operate in FIX mode. Write the missing integration tests for this new feature following the required Playwright patterns and the directory-mirrors-path layout. Validate each new test module by a clean import only. NEVER run the suite, not the full suite and not one file.\n\nFeature request: ${ask}\nFiles changed:\n${changed.join('\n')}`,
|
||||
{ agentType: 'test-maintainer', label: 'tests', phase: 'Fix' }
|
||||
`Operate in FIX mode. Write the missing integration tests for this new feature across EVERY tier it exercises, per the DevPlace test standard below. This is mandatory, not a nicety: the feature is incomplete until each route and helper it adds has a test in the appropriate tier (unit for new data/query helpers, api for new JSON/HTML routes, e2e for new interactive UI flows), in the correct file under the directory-mirrors-path layout. Decide the tiers from the changed files and routes; create the package directories (with __init__.py) the new test paths require. Validate each new test module by a clean import only. NEVER run the suite, not the full suite and not one file.\n\n${TESTS}\n\nFeature request: ${ask}\nRoutes: ${routes.join(', ')}\nFiles changed:\n${changed.join('\n')}\n\nReturn the test files you wrote, the tier of each, and which routes/helpers remain uncovered (with the reason).`,
|
||||
{ agentType: 'test-maintainer', label: 'tests', phase: 'Test' }
|
||||
)
|
||||
|
||||
log(`Feature build complete: ${changed.length} file(s), ${gaps.length} audit gap(s) addressed`)
|
||||
log(`Feature build complete: ${changed.length} file(s), ${gaps.length} gap(s) addressed`)
|
||||
|
||||
return { ask, map, plan, build, audit: gaps, gapFix, tests }
|
||||
return {
|
||||
ask,
|
||||
map,
|
||||
plan,
|
||||
build,
|
||||
routes,
|
||||
audit: { candidates: auditCandidates.length, confirmed: auditConfirmed, gaps },
|
||||
liveVerify: live,
|
||||
gapFix,
|
||||
tests,
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user