|
|
c0a59b4138
|
feat(nadia): Build content-type agnostic pipeline with bounded asyncio worker pool
Outcome: done
Changed: src/typosaurus_sandbox/research/pipeline.py:1-280 (new), src/typosaurus_sandbox/research/client.py:30-52,98-99,124-148, src/typosaurus_sandbox/research/__init__.py:21-49
Verified by: make verify -> exit_code 0, compileall OK, 195 tests OK, "verification passed" (pre-existing StarletteDeprecationWarning from tests/test_api.py); live smoke: web/images/chat via one pool (3 ok, urls=19, queries=92, contents=10), repeat web item cache_hit=True, empty query -> RsearchError 400 handled, pool survived
Findings:
- ResearchPipeline uses asyncio.Semaphore(max_concurrency, default 8); pool size logged INFO in run(); run(AsyncIterator[WorkItem]) = bounded queue (pool*4) + pool_size workers with None sentinels; process(item) is the public semaphore-guarded path.
- WorkItem(kind: web|images|describe|chat, value, deep=False, ai=False); web -> search(content=True), images -> search(type="images") without deep/ai, describe -> describe(url), chat -> chat(prompt); endpoints /search,/describe,/chat.
- extract_response() handles SearchResponse (results, ai_response, deep sources/markdown), ChatResponse.response, DescribeResponse.description in one function; apply_extraction() registers URLs, query seeds (title/description/extra/text with origin) and content fingerprints into QueryFrontier.
- client.py additive: _search_params() shared by search() and new search_cached()/describe_cached() probes so pipeline cache_hit is accurate; chat/deep cache hit from envelope fields cache
Typosaurus-Run: 4e2afb673c7f4578a12276d9181b982d
Typosaurus-Node: e85a60edbf7b47f1913d87602b9c553c
Typosaurus-Agent: @nadia
Refs: #31
|
2026-08-07 19:30:57 +00:00 |
|
