## Implementation Plan ### Changes Required 1. **`devplacepy/routers/seo.py`** – Add `Disallow: /game/` after line 19 (after `/notifications/`). *Insert line:* `Disallow: /game/` 2. **`devplacepy/main.py`** – Insert a new middleware **after line 466** (after `await_pending_corrections`) and **before line 492** (rate limiter). Add an async function `crawler_detection_middleware` that: - Reads `request.headers.get("user-agent", "")`. - If UA matches any of a small hardcoded set of known crawlers (e.g. `"Google-Read-Aloud"`, `"Googlebot"`, `"Bingbot"`, `"Slurp"`), and `request.url.path` starts with one of the auth-gated paths (`/messages`, `/notifications`, `/game`, `/admin`), then immediately return `Response(status_code=204)` (no audit, no route handler). - Otherwise call `await call_next(request)`. Register this middleware via `app.middleware("http")` decorator at module level after the existing middleware block (around line 592) or insert inline – simplest is to add a new `@app.middleware("http")` at the end of the file, but to maintain ordering it should be placed before rate limiter. Place just before line 492 with a comment. 3. **`tests/api/robotstxt.py`** – Update `test_robots_txt_exists` (or add a new assertion) to verify that `Disallow: /game/` is present in the response body. *Minimal change:* add one line in the existing test: ```python assert b"Disallow: /game/" in response.content ``` 4. **`devplacepy/tests/api/test_crawler_middleware.py`** – *Optional but recommended:* Add a unit test verifying that a request with `User-Agent: Google-Read-Aloud` to `/game` returns 204 (no audit). Not strictly required for the fix but improves confidence. If omitted, Definition of Done still passes via existing tests. --- ## Definition of Done - [ ] `devplacepy/routers/seo.py` contains the line `Disallow: /game/` in the robots.txt response. - [ ] `devplacepy/main.py` includes a middleware that returns HTTP 204 for known crawler User-Agents on auth-gated paths, and the middleware is placed before the rate limiter. - [ ] Updated robotstxt test (`tests/api/robotstxt.py`) asserts that `/game/` appears in the disallow list. - [ ] The project’s test command (`pip install -e '.[dev]' -q && make test-unit`) passes with exit code 0. - [ ] The project’s lint command (`pip install -q ruff && ruff check .`) passes with exit code 0. - [ ] No new warnings or errors introduced by the changes.