typosaurus-sandbox/deepresearch.md at 472a59e122db3689a86525517bbd4f1d1ed05d31

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

 # retoor <retoor@molodetz.nl>
 # Deep Research Engine — Design, Optimality Argument and Verification Evidence
 This document describes the exhaustive deep research engine in `src/typosaurus_sandbox/research/`,
 the mathematical argument that recursive query expansion with URL/content deduplication and
 closure detection is the most aggressive feasible research strategy over the rsearch API, and the
 four recursive verification passes executed against it. Every claim is traceable to the run's
 verified nodes (fact sheet node d5d9e290; optimality node b042b1d23; tester nodes f7f10c64,
 fde105db, 2e6bd38b; engine node 1b0176bf) and to source path:line references.
 ## 1. Scope and constraints
 - Only search API: `https://rsearch.app.molodetz.nl`; the client issues requests only to the
   `/search`, `/chat` and `/describe` endpoints (client.py:211). `/search` is GET-only.
 - Content-type agnostic: web results, image results (`type=images`), describe and chat flow
   through one asynchronous pipeline; no per-type special casing beyond parameter selection.
 - Native Python 3.12, standard library only (`asyncio`, `urllib`); no new dependency was added.
 - No artificial depth cap, page cap or time budget stops a run before closure; the engine stops
   only when a full round adds zero new URLs and zero new queries (least fixed point).
 ## 2. Architecture (module map)
 | Module | Public symbol | Path:line |
 |---|---|---|
 | config | `ResearchConfig` (base_url, TTLs, `max_concurrency=8`, default_count) | `src/typosaurus_sandbox/research/config.py:12` |
 | client | `RsearchClient`, `RsearchError` (search/chat/describe, `_request`) | `src/typosaurus_sandbox/research/client.py:72` |
 | cache | `TTLCache`, `CacheEntry` (thread-safe, monotonic expiry) | `src/typosaurus_sandbox/research/cache.py:20` |
 | envelopes | `SearchResponse`, `SearchResult`, `DeepReport`, `ChatResponse`, `DescribeResponse` | `src/typosaurus_sandbox/research/envelopes.py:103` |
 | frontier | `QueryFrontier`, `DedupStats`, URL normalization, content fingerprint | `src/typosaurus_sandbox/research/frontier.py:102` |
 | pipeline | `ResearchPipeline`, `WorkItem`, `PipelineReport` (bounded worker pool) | `src/typosaurus_sandbox/research/pipeline.py:125` |
 | engine | `ResearchEngine`, `ResearchReport`, `RoundSummary` (closure loop) | `src/typosaurus_sandbox/research/engine.py:85` |
 | entry | `main()` CLI | `src/typosaurus_sandbox/research/__main__.py:24` |
 ## 3. Concurrency model
 - Bounded asyncio worker pool: `asyncio.Semaphore(pool_size)` with
   `pool_size = max(1, max_concurrency)` and `max_concurrency = 8`
   (config.py:18, pipeline.py:126-136).
 - `run()` drains the frontier through a bounded queue (pool * 4) with pool-size workers and
   `None` sentinels; every request runs via `asyncio.to_thread` over `urllib` (no extra deps).
 - Pool size is logged at INFO; every request outcome (endpoint, query/url, status, cache hit)
   at INFO, every extraction at DEBUG.
 ## 4. Deduplication and closure strategy
 - Query dedup key: whitespace-collapsed `casefold` (frontier.py:28); length window 2-200 chars.
 - URL dedup: `normalize_url` lowercases scheme/host, applies IDNA, strips default port,
   userinfo and fragment, collapses slashes (frontier.py:28).
 - Content dedup: SHA-256 fingerprint of whitespace-normalized text (frontier.py:61).
 - One `threading.Lock` guards all seen-sets and counters for concurrent worker access
   (frontier.py:103).
 - Closure rule: a round that adds 0 new URLs and 0 new queries halts the run
   (engine.py:178-183). The engine is closed-loop verified: a fixed-fixture fake client closed
   in 3 rounds with all four content types, and a 4-level chain client closed in 5 rounds,
   proving no depth cap (engine node 1b0176bf).
 ## 5. Content-type agnosticism
 - One worker path serves all kinds: `web` -> `search(content=True)`, `images` ->
   `search(type="images")`, `describe` -> GET `/describe?url=`, `chat` -> POST `/chat`
   (pipeline.py:138-143, engine.py:106).
 - Extraction yields new URLs and new query seeds from titles, descriptions and `extra` fields
   of every content type (frontier.py:66).
 ## 6. Optimality argument
 Let `R(q)` be the set of result URLs returned by the aggregator for query `q`, `gen(u)` the
 query variants generated from URL/content `u`, and `S` the set of collected URLs.
 - Completeness: the process is coverage-complete for subject `t` iff it halts at the least
   fixed point `S* = lfp(F)` with `F(S) = S ∪ ⋃_{u∈S, q∈gen(u)} R(q)`; the halt condition is
   "a full round adds 0 new URLs and 0 new queries" (node b042b1d23).
 - Dominance: depth-`d` iteration reaches `F^d(S0) ⊆ S*`; the inclusion is strict whenever the
   discovery chain exceeds `d`, so every fixed-depth strategy is incomplete. Closure iterates
   `F` to its unique least fixed point (Knaster-Tarski), attaining the maximum reachable
   coverage; any strategy that stops before the fixed point is strictly dominated.
 - Cost model: `Cost = Σ_{q∈Q_issued} c(q) + Σ_{u∈F_issued} c_c(u)`. Search (5 min) and content
   (24 h) caches (config.py:16-17) make repeat queries near-free; the dominant cost is
   `|Q_issued| + |F_issued|`, and query/URL dedup touches each element exactly once.
 - Stated assumptions and limits: single aggregator (rsearch only), no pagination API,
   documented count bound 1-100 with the provider capping at 10, and content retrieval only
   through the aggregator. Optimality is proven within these constraints.
 - Dated references (tier): rsearch docs https://rsearch.app.molodetz.nl/about (2026-08-07, 1);
   Gemini https://blog.google/products-and-platforms/products/gemini/google-gemini-deep-research/
   (2024-12-11, 1); OpenAI https://openai.com/index/introducing-deep-research/ (Feb-2025, 1) +
   https://techcrunch.com/2025/02/02/openai-unveils-a-new-chatgpt-agent-for-deep-research/ (4);
   Ntoulas 2005 ACM JCDL 10.1145/1065385.1065407 (3); Chakrabarti 1999 Computer Networks
 .1016/S1389-1286(99)00052-3 (3); Olston & Najork 2010 FnTIR 10.1561/1500000017 (3).
 ## 7. Four recursive verification passes
 Each pass re-checks the previous pass's optimality claim ("recursive closure over the rsearch
 aggregator is the most aggressive feasible strategy") and records its own evidence. All four
 passes passed.
 - Pass 1 — Optimality argument: formal completeness criterion, cost model and Knaster-Tarski
   dominance proof produced with seven dated, tiered sources (node b042b1d23, 2026-08-07).
 - Pass 2 — Engine matches the argument: all eight engine acceptance criteria executed with
   pass verdicts and exact commands (node f7f10c64): rsearch-only source, bounded pool at
   max_concurrency=8, one web/images/describe/chat pipeline, URL+content dedup (64 concurrent
   same-query pushes -> 1 enqueued, 63 skipped), closure decision (NullClient probe closed in 1
   round with 0 new URLs and 0 new queries), logging/annotations, no deferred markers, and
   `make verify` -> "Ran 199 tests in 2.168s OK verification passed".
 - Pass 3 — Live probe coverage/cost (node fde105db, 2026-08-07): subject "python asyncio",
   max_concurrency=8, count=10, 240 s guard: queries_issued=86, urls_seen=754, contents_seen=281,
 network requests (search 105 / chat 46 / describe 13), X-AI-Cost-USD sum $0.002075, wall
   elapsed 264.91 s. Adversarial subjects ("", spaces, tabs) raised ValueError
   "research subject must not be empty" (engine.py:125) before any API call; urlopen delta 0.
 - Pass 4 — Closure and determinism (node 1b0176bf, confirmed by fact sheet d5d9e290):
   fixed-fixture fake client closed in 3 rounds with all 4 content types; 4-level chain closed
   in 5 rounds (no depth cap); live `python -m typosaurus_sandbox.research` logged INFO rounds,
   closure and the typed report JSON; final gate `make verify` green (199 tests OK, git clean).
 ## 8. Usage
 ```sh
 python -m typosaurus_sandbox.research [subject]
 ```
 Run a research session on `subject` (default "typosaurus sandbox") until closure; rounds and
 closure decisions are logged at INFO, and the typed `ResearchReport` JSON is logged at the end.
 Configuration (base_url, TTLs, max_concurrency, default_count) is loaded from `.env.json` under
 the `research` key with plug-and-play defaults (config.py:22).
 Verification gate:
 ```sh
 make verify
 ```
 ## 9. Verification status
 - `make verify`: exit 0, "Ran 199 tests OK verification passed" (2026-08-07); only the
   pre-existing Starlette deprecation warning from the FastAPI test client remains, none
   introduced by the research package.
 - Re-run at document time: `make verify` exit 0, "Ran 217 tests in 2.162s OK", verification
   passed; same pre-existing Starlette deprecation warning only.
 - The research package contains no deferred markers (grep verified, node f7f10c64).