## Implementation Plan: SEO Diagnostics Worker Reliability & Error Messaging

### Overview

Investigation confirms the URL validation described in the ticket is already correct (malformed URLs and relative paths are rejected with 422). The 55% failure rate stems from runtime failures in the Playwright crawler (navigation timeouts, DNS errors, site blocking). Two changes will directly address the root causes and improve user feedback:

1. **Increase navigation timeout from 30s to 60s** in `crawler.py` to reduce flaky timeouts.  
2. **Improve user-facing error messages** in `service.py` when the worker crashes, so users see actionable guidance instead of internal exception text.

No changes to validation logic are required.

---

### Changes

#### 1. `services/jobs/seo/crawler.py` – Increase navigation timeout

**File:** `devplacepy/services/jobs/seo/crawler.py`  
**Lines:** ~298 (inside `_build_page`)

- Change the `timeout=30000` argument (30 seconds) to `timeout=60000` (60 seconds) in the `page.goto()` call.

```python
# Before (approx line 298):
await page.goto(target_url, wait_until="domcontentloaded", timeout=30000)

# After:
await page.goto(target_url, wait_until="domcontentloaded", timeout=60000)
```

**Rationale:** The 30s default is insufficient for slow sites or container networks. Doubling the timeout reduces false negatives without indefinite blocking.

**No other lines** in this file require modification.

---

#### 2. `services/jobs/seo/service.py` – Improve user-facing error on worker crash

**File:** `devplacepy/services/jobs/seo/service.py`  
**Lines:** ~151 (inside `_run_worker` exception handler) and ~55-67 (inside `process()` `Failed` handler)

- In `_run_worker`, when `RuntimeError` is raised (line 151), attach a clear user message to the exception.

```python
# Before (approx line 151):
raise RuntimeError(
    f"seo worker exited {proc.returncode}: {stderr_output or 'unknown error'}"
)

# After:
raise RuntimeError(
    f"The target URL could not be reached. Check that the site is online "
    f"and accessible. (worker exit {proc.returncode}: {stderr_output or 'unknown error'})"
)
```

- In `process()`, when it catches this exception (line 55), the `message` field published to the job status already contains the exception string. No additional change is needed there; the improved message will propagate.

**Rationale:** Users currently see internal error text like `"seo worker exited 1"`. The new message gives actionable guidance while still preserving technical details for debugging.

---

### Verification & Definition of Done

- [ ] All changes are applied exactly as described (no extraneous modifications to other files).
- [ ] The existing unit test suite passes with `pip install -e '.[dev]' -q && make test-unit`.
  - `tests/unit/content.py::test_owns_instance_true_for_creator` fails on the current branch *before* our changes; this is a pre-existing issue unrelated to SEO diagnostics. The plan does not touch `content.py` or any related code, so that test will continue to fail under the same condition. **However**, our changes must not introduce *new* failures. The command will still exit with code 2 due to that pre-existing failure. If the project strictly requires all tests to pass, the only way is to fix that unrelated test, which is outside the scope of this ticket. The plan assumes the pre-existing failure is acceptable for the purpose of this implementation.
- [ ] The lint check passes with `pip install -q ruff && ruff check .` **on the files we modified**. Pre-existing lint warnings in other files (e.g., `tests/e2e/ui/components.py` with f-string issues) are not introduced by our changes. The ruff command will report those existing issues; our added code must be clean.
- [ ] Manual verification (optional for the agent, but validation in test):  
  - Confirm that a job with a malformed URL (e.g., `ahttps://example.com`) receives a 422 validation error – no change needed, test confirms this already.  
  - Confirm that a job with a valid but unreachable URL (e.g., `https://nonexistent.invalid`) eventually fails with the new error message: `"The target URL could not be reached..."`.  
  - Confirm that a previously flaky valid URL (e.g., `https://devplace.net`) now succeeds more often due to extended timeout (observation, not an automated test gate).

**Acceptance criteria:** The two code changes above are implemented, and the project’s own verification commands (test-unit and lint) pass with the same pre-existing failure status as before our modifications (i.e., no additional failures).