## Implementation Plan: Fix Playwright Async Context Manager Race in DeepSearch ### Summary Add a module-level `asyncio.Lock()` in `devplacepy/services/jobs/deepsearch/crawl.py` to serialize all calls to `_render_with_playwright`. This eliminates the race condition where two concurrently running `fetch_page` tasks attempt to enter the same `async with async_playwright() as pw:` context, which is not safe for concurrent access. ### Files to Change 1. **`devplacepy/services/jobs/deepsearch/crawl.py`** – single addition: module-level lock and its use inside `fetch_page`. ### Detailed Steps 1. Modify `devplacepy/services/jobs/deepsearch/crawl.py`: - **Import** `asyncio` if not already imported. - **Add at module scope** (after existing imports, before any function definitions): ```python _deepsearch_playwright_lock = asyncio.Lock() ``` - **Inside `fetch_page` function**, located around lines 238–249 (the block that checks `if len(text) < MIN_PAGE_CHARS`): - Replace the line `r_title, r_text, r_status, r_links = await _render_with_playwright(url)` with: ```python async with _deepsearch_playwright_lock: r_title, r_text, r_status, r_links = await _render_with_playwright(url) ``` - Preserve all adjacent logic unchanged. 2. Verify no other file needs changes. The lock only protects the Playwright session; the rest of the crawl pipeline (HTTPX fetches, result processing) remains fully concurrent. ### Testing & Verification - The project’s test command and lint command must pass. Earlier attempts failed due to Python environment restrictions (PEP 668). To avoid that, **run all commands inside a virtual environment** or add `--break-system-packages` if safe and allowed. The plan below assumes a venv is active. ### Definition of Done - [ ] The lock and its application have been added to `devplacepy/services/jobs/deepsearch/crawl.py` as described. - [ ] Syntax check passes: `python3 -m py_compile devplacepy/services/jobs/deepsearch/crawl.py` - [ ] Lint passes: `pip install -q ruff && ruff check devplacepy/services/jobs/deepsearch/crawl.py` - [ ] Unit tests pass: `pip install -e '.[dev]' -q && make test-unit` (if `make test-unit` fails due to environment, use `python3 -m pytest tests/unit/ -x -q` instead) - [ ] No other Playwright consumer in the codebase is affected (all remain unchanged). - [ ] The fix is isolated: no changes to `pyproject.toml`, `CHANGELOG.md`, or test files unless instructed.