chore: reorganize test files into domain-specific subdirectories under tests/

Split the monolithic test directory into three tiers (unit, api, e2e) with a path-mirroring directory structure. Added corresponding Makefile targets (test-unit, test-api, test-e2e) and updated all documentation references (CLAUDE.md, README.md, testing-cicd.html, testing-framework.html, testing-make.html) to reflect the new layout and naming conventions.
This commit is contained in:
2026-06-13 14:32:33 +00:00
parent 014c4c6bb3
commit 43f4011005
268 changed files with 20634 additions and 9206 deletions
+16 -5
View File
@@ -21,7 +21,7 @@ make locust-headless # Locust CLI mode for CI
Validate code without running the suite: confirm `python -c "from devplacepy.main import app"` imports clean and check each touched language manually (Python compiles/imports, JS parses, CSS braces and HTML tags balance).
Single test: `python -m pytest tests/test_feed.py::test_name -v --tb=line -x`
Single test: `python -m pytest tests/e2e/feed.py::test_name -v --tb=line -x`
CLI (installed as `devplace`):
```bash
@@ -180,7 +180,7 @@ Devii partially configures its OWN system message: every system prompt ends with
- **Slug + UUID lookup:** resources with slugs (posts, gists, news, projects) accept either the slug or the bare UUID - see `resolve_by_slug()` in `database.py`. Slugs are generated via `make_combined_slug(title, uid)` which prefixes the first 8 chars of the UUID.
- **Username:** 3-32 chars, letters/numbers/hyphens/underscores. **Password:** min 6 chars.
- **Roles are stored capitalized:** the `users.role` value is exactly `"Admin"` or `"Member"` (the first registered user becomes `"Admin"`; `auth.py`). Always test admin-ness through the `is_admin(user)` global (`utils.py`, also a Jinja global), which compares `user.get("role") == "Admin"` case-sensitively - never hand-roll a lowercase compare. The **only** lowercase surface is the CLI (`devplace role set <username> <member|admin>` accepts lowercase but writes `role.capitalize()`; `role get` prints `.lower()` for display). A lowercase role in the DB silently fails every admin check.
- **Never pass a `respond()` context key that collides with a Jinja global.** `respond(request, template, ctx, model=XOut)` feeds the **same** `ctx` dict to both the Pydantic model (JSON) and the template render. Jinja context variables shadow `env.globals` across the whole inheritance chain, so a key like `is_admin`/`avatar_url`/`format_date`/`is_self`/`owns`/`guest_disabled` holding a non-callable value turns `base.html`'s `{% if is_admin(user) %}` into `False(user)` -> `TypeError: 'bool' object is not callable`, a 500 that only fires for the branch that calls the global (e.g. logged-in users, not guests). Name viewer/permission flags distinctly (`viewer_is_admin`, not `is_admin`) in both the schema and the context. This was a real bug on `/bugs/{number}`; `tests/test_bugs_gitea.py::test_bug_detail_renders_for_{member,admin}` guard it.
- **Never pass a `respond()` context key that collides with a Jinja global.** `respond(request, template, ctx, model=XOut)` feeds the **same** `ctx` dict to both the Pydantic model (JSON) and the template render. Jinja context variables shadow `env.globals` across the whole inheritance chain, so a key like `is_admin`/`avatar_url`/`format_date`/`is_self`/`owns`/`guest_disabled` holding a non-callable value turns `base.html`'s `{% if is_admin(user) %}` into `False(user)` -> `TypeError: 'bool' object is not callable`, a 500 that only fires for the branch that calls the global (e.g. logged-in users, not guests). Name viewer/permission flags distinctly (`viewer_is_admin`, not `is_admin`) in both the schema and the context. This was a real bug on `/bugs/{number}`; `tests/api/bugs/create.py::test_bug_detail_renders_for_{member,admin}` guard it.
- **Avatars:** Multiavatar SVG generated locally from the username seed in <5ms, in-memory cached (cleared on restart). URL `/avatar/multiavatar/{seed}?size={size}`. Falls back to initial-based SVG on error.
- **Project visibility (`is_private`) and read-only (`read_only`)** are owner-controlled integer flags on the `projects` row (see `AGENTS.md` -> "Project visibility and read-only"). Two invariants: (1) private-read access is gated by the single `content.can_view_project(project, user)` predicate (owner or admin) at EVERY read surface - detail, file routes (`_load_viewable_project`), zip, listing, profile, sitemap - never re-implement the check inline; (2) read-only is enforced as a single data-layer guard `project_files._guard_writable(project_uid)` at the top of every mutation entrypoint in `project_files.py`, so it covers the HTTP routes, Devii (via the routes), and container sync at once - add the guard to any NEW file-mutating function, and never enforce read-only only in a route. Devii may flip read-only or visibility (private/public) only after explicit user confirmation, enforced by `dispatcher.confirmation_error` (the `CONFIRM_REQUIRED` set) before the HTTP call; mirror that pattern for any future irreversible agent action. Both owner toggle buttons (`project_detail.html`) likewise carry `data-confirm` so the human UI gates them through `app.dialog` too. **Deletions ALWAYS require confirmation:** `confirmation_error` gates EVERY content delete tool via the `CONFIRM_REQUIRED` set - `delete_post`, `delete_comment`, `delete_gist`, `delete_project`, `project_delete_file`, `delete_media`, `delete_attachment`, `admin_delete_news` - plus `container_instance_action` with `action="delete"` and any `container_exec` whose command matches `dispatcher.DESTRUCTIVE_COMMAND` (rm/rmdir/unlink/shred/truncate/dd/mkfs/wipefs, `find -delete`, redirect over a file, SQL `drop`/`delete from`). The first call is refused with a message; the agent must show the user the exact target and only then pass `confirm=true` (the `DELETING IS ALWAYS CONFIRMED` system-prompt rule mirrors this). Any new name added to `CONFIRM_REQUIRED` is auto-confirmed by a generic fallback in `confirmation_error`. **Load-bearing: every confirmation-gated tool MUST also declare a `confirm` boolean param in its catalog spec** (use the `confirm()` helper in `catalog.py`, or `arg("confirm", ..., kind="boolean")` for container actions). Tool schemas set `additionalProperties: false`, so a gated tool WITHOUT a declared `confirm` param can never receive `confirm=true` from the model - the gate refuses every call and the agent loops forever (this was a real bug on all the delete tools). The param is harmlessly forwarded to the HTTP endpoint (the routes do not declare it; FastAPI ignores undeclared form fields) and ignored by the LOCAL container/customization controllers. The delete tools stay `requires_auth` (not `requires_admin`) - members delete their own, the endpoint's owner-or-admin guard lets admins delete any.
@@ -211,7 +211,18 @@ Rules when touching this area:
## Testing
Playwright (NOT pytest-playwright - uninstall if present, it conflicts). 849+ tests across 63 files in `tests/` (mixed Playwright UI tests and in-process `TestClient`/asyncio unit tests). The fixture stack:
Playwright (NOT pytest-playwright - uninstall if present, it conflicts). 932 tests in `tests/`, split into **three category directories** by *what they exercise*, one file per endpoint:
- `tests/api/` - HTTP integration tests against the live uvicorn subprocess (`app_server`/`seeded_db`, `requests`/`httpx` vs `BASE_URL`); no browser.
- `tests/e2e/` - Playwright browser tests (`page`/`alice`/`bob`).
- `tests/unit/` - pure in-process tests (the `local_db` fixture or no fixture); call library functions directly, no running server.
**The directory tree mirrors the path** - one segment per directory, the last segment is the file (NOT a flat `_`-joined name):
- **api/e2e are organised by endpoint path.** Each route segment becomes a directory, the final segment becomes the file; `{param}` segments are dropped and each segment is lowercased with non-alphanumerics stripped. `GET /admin/ai-usage` -> `tests/e2e/admin/aiusage.py`, `POST /auth/login` -> `tests/api/auth/login.py`, `GET /projects/{slug}/files/lines` -> `tests/api/projects/files/lines.py`. A collection/bare path that is also a parent of deeper paths uses `index.py` inside its own directory (`/posts` -> `tests/e2e/posts/index.py` alongside `/posts/create` -> `tests/e2e/posts/create.py`); `GET /` -> `tests/<tier>/root.py`. Multiple verbs of one path share one file.
- **unit mirrors the SOURCE module path** of the code under test (strip the `devplacepy.` prefix, dots become directories): tests for `devplacepy/utils.py` -> `tests/unit/utils.py`, for `devplacepy/services/audit/store.py` -> `tests/unit/services/audit/store.py`, for `devplacepy/services/openai_gateway/analytics.py` -> `tests/unit/services/openai_gateway/analytics.py`.
A test's tier is decided by its fixtures: a browser fixture (`page`/`alice`/`bob`) = e2e; `app_server`/`seeded_db` or any HTTP call = api; `local_db`-only or no fixture = unit. Every directory is a package (`__init__.py`). `pyproject.toml` sets `testpaths = ["tests/unit","tests/api","tests/e2e"]` and `python_files = ["*.py"]` (files are NOT `test_`-prefixed; test *functions* still are). `conftest.py` stays at `tests/` root so its fixtures inherit into all subdirs. Helpers shared across files import from the canonical new module path (e.g. `from tests.e2e.post import create_post`). Run a tier with `make test-unit` / `make test-api` / `make test-e2e`; `make test` runs all three. The fixture stack:
- `app_server` (session-scoped): spawns uvicorn as a subprocess on port 10501 (the `PORT` constant) with a tempfile DB.
- `browser_context` (session-scoped): one Playwright context shared by all tests.
@@ -223,7 +234,7 @@ Required test patterns:
- Prefer `page.locator(...).wait_for(state="visible")` over `wait_for_selector`.
- Scope ambiguous selectors: comment Delete is `.comment-action-btn:has-text('Delete')`; create-post Post button is `#create-post-modal button.btn-primary:has-text('Post')`; comment textarea is `.comment-form textarea[name='content']`.
- Failure screenshots auto-save to `/tmp/devplace_test_screenshots/` via the `pytest_runtest_makereport` hook in `conftest.py`.
- The server is shared session-wide, so a test that flips a global `site_settings` value (maintenance mode, registration, rate limit, session length) MUST restore it in a `try/finally`. `conftest.py`'s `app_server` seeds `rate_limit_per_minute="1000000"` so the suite is never throttled; rate-limit unit tests patch `m.get_int_setting` rather than the `RATE_LIMIT` constant (the constant is only a fallback now). See `tests/test_operational.py` and `tests/test_ratelimit.py`.
- The server is shared session-wide, so a test that flips a global `site_settings` value (maintenance mode, registration, rate limit, session length) MUST restore it in a `try/finally`. `conftest.py`'s `app_server` seeds `rate_limit_per_minute="1000000"` so the suite is never throttled; rate-limit tests patch `m.get_int_setting` rather than the `RATE_LIMIT` constant (the constant is only a fallback now). Operational tests live in `tests/e2e/operational.py` plus the `tests/api/admin/settings.py` family; rate-limit tests sit with the endpoint they probe (`tests/api/root.py`, `tests/api/auth/login.py`, `tests/api/robotstxt.py`).
**Never run tests unless the user explicitly asks for it.** Not the full suite, not a single file. Validate code with a clean import (`python -c "from devplacepy.main import app"`) and per-language manual checks instead.
@@ -232,7 +243,7 @@ Required test patterns:
Every feature in DevPlace is **one data source fanning out into several consumers**. A single route is rendered as HTML, served as JSON, called by the Devii agent, and described in the API docs - all from the same handler. The cardinal failure mode is changing one layer and forgetting a connected one (add a route but no JSON schema; add a JSON endpoint but no Devii tool; ship a feature but never document it). The checklist below is ordered by data flow so nothing is dropped. See `AGENTS.md` -> "Anatomy of a feature" for the deep relationship map and a worked example.
### 1. Understand (read before writing)
Read the router (`routers/{area}.py`), template (`templates/{area}.html`), test file (`tests/test_{area}.py`), and the matching `AGENTS.md` domain section. Trace the existing data flow: input model -> router -> data helper -> response (HTML and JSON).
Read the router (`routers/{area}.py`), template (`templates/{area}.html`), the matching tests (`tests/{unit,api,e2e}/<endpoint>.py` - per the naming rule in the Testing section), and the matching `AGENTS.md` domain section. Trace the existing data flow: input model -> router -> data helper -> response (HTML and JSON).
### 2. Data layer (`database.py`, `models.py`, `schemas.py`)
- **`database.py`** - add query/batch helpers here, never inline N+1 loops in routers (use `get_users_by_uids`, `build_pagination`, `_in_clause`). Guard raw SQL with `if "table" in db.tables`. Schema auto-syncs via `dataset`; add indexes in `init_db()` with `CREATE INDEX IF NOT EXISTS`.