typosaurus-sandbox/deepresearch.md

# retoor <retoor@molodetz.nl>
# Deep Research Engine — Design, Optimality Argument and Verification Evidence

This document describes the exhaustive deep research engine in `src/typosaurus_sandbox/research/`,
the mathematical argument that recursive query expansion with URL/content deduplication and
closure detection is the most aggressive feasible research strategy over the rsearch API, and the
four recursive verification passes executed against it. Every claim is traceable to the run's
verified nodes (fact sheet node d5d9e290; optimality node b042b1d23; tester nodes f7f10c64,
fde105db, 2e6bd38b; engine node 1b0176bf) and to source path:line references.

## 1. Scope and constraints

- Only search API: `https://rsearch.app.molodetz.nl`; the client issues requests only to the
  `/search`, `/chat` and `/describe` endpoints (client.py:211). `/search` is GET-only.
- Content-type agnostic: web results, image results (`type=images`), describe and chat flow
  through one asynchronous pipeline; no per-type special casing beyond parameter selection.
- Native Python 3.12, standard library only (`asyncio`, `urllib`); no new dependency was added.
- No artificial depth cap, page cap or time budget stops a run before closure; the engine stops
  only when a full round adds zero new URLs and zero new queries (least fixed point).

## 2. Architecture (module map)

| Module | Public symbol | Path:line |
|---|---|---|
| config | `ResearchConfig` (base_url, TTLs, `max_concurrency=8`, default_count) | `src/typosaurus_sandbox/research/config.py:12` |
| client | `RsearchClient`, `RsearchError` (search/chat/describe, `_request`) | `src/typosaurus_sandbox/research/client.py:72` |
| cache | `TTLCache`, `CacheEntry` (thread-safe, monotonic expiry) | `src/typosaurus_sandbox/research/cache.py:20` |
| envelopes | `SearchResponse`, `SearchResult`, `DeepReport`, `ChatResponse`, `DescribeResponse` | `src/typosaurus_sandbox/research/envelopes.py:103` |
| frontier | `QueryFrontier`, `DedupStats`, URL normalization, content fingerprint | `src/typosaurus_sandbox/research/frontier.py:102` |
| pipeline | `ResearchPipeline`, `WorkItem`, `PipelineReport` (bounded worker pool) | `src/typosaurus_sandbox/research/pipeline.py:125` |
| engine | `ResearchEngine`, `ResearchReport`, `RoundSummary` (closure loop) | `src/typosaurus_sandbox/research/engine.py:85` |
| entry | `main()` CLI | `src/typosaurus_sandbox/research/__main__.py:24` |

## 3. Concurrency model

- Bounded asyncio worker pool: `asyncio.Semaphore(pool_size)` with
  `pool_size = max(1, max_concurrency)` and `max_concurrency = 8`
  (config.py:18, pipeline.py:126-136).
- `run()` drains the frontier through a bounded queue (pool * 4) with pool-size workers and
  `None` sentinels; every request runs via `asyncio.to_thread` over `urllib` (no extra deps).
- Pool size is logged at INFO; every request outcome (endpoint, query/url, status, cache hit)
  at INFO, every extraction at DEBUG.

## 4. Deduplication and closure strategy

- Query dedup key: whitespace-collapsed `casefold` (frontier.py:28); length window 2-200 chars.
- URL dedup: `normalize_url` lowercases scheme/host, applies IDNA, strips default port,
  userinfo and fragment, collapses slashes (frontier.py:28).
- Content dedup: SHA-256 fingerprint of whitespace-normalized text (frontier.py:61).
- One `threading.Lock` guards all seen-sets and counters for concurrent worker access
  (frontier.py:103).
- Closure rule: a round that adds 0 new URLs and 0 new queries halts the run
  (engine.py:178-183). The engine is closed-loop verified: a fixed-fixture fake client closed
  in 3 rounds with all four content types, and a 4-level chain client closed in 5 rounds,
  proving no depth cap (engine node 1b0176bf).

## 5. Content-type agnosticism

- One worker path serves all kinds: `web` -> `search(content=True)`, `images` ->
  `search(type="images")`, `describe` -> GET `/describe?url=`, `chat` -> POST `/chat`
  (pipeline.py:138-143, engine.py:106).
- Extraction yields new URLs and new query seeds from titles, descriptions and `extra` fields
  of every content type (frontier.py:66).

## 6. Optimality argument

Let `R(q)` be the set of result URLs returned by the aggregator for query `q`, `gen(u)` the
query variants generated from URL/content `u`, and `S` the set of collected URLs.

- Completeness: the process is coverage-complete for subject `t` iff it halts at the least
  fixed point `S* = lfp(F)` with `F(S) = S ∪ ⋃_{u∈S, q∈gen(u)} R(q)`; the halt condition is
  "a full round adds 0 new URLs and 0 new queries" (node b042b1d23).
- Dominance: depth-`d` iteration reaches `F^d(S0) ⊆ S*`; the inclusion is strict whenever the
  discovery chain exceeds `d`, so every fixed-depth strategy is incomplete. Closure iterates
  `F` to its unique least fixed point (Knaster-Tarski), attaining the maximum reachable
  coverage; any strategy that stops before the fixed point is strictly dominated.
- Cost model: `Cost = Σ_{q∈Q_issued} c(q) + Σ_{u∈F_issued} c_c(u)`. Search (5 min) and content
  (24 h) caches (config.py:16-17) make repeat queries near-free; the dominant cost is
  `|Q_issued| + |F_issued|`, and query/URL dedup touches each element exactly once.
- Stated assumptions and limits: single aggregator (rsearch only), no pagination API,
  documented count bound 1-100 with the provider capping at 10, and content retrieval only
  through the aggregator. Optimality is proven within these constraints.
- Dated references (tier): rsearch docs https://rsearch.app.molodetz.nl/about (2026-08-07, 1);
  Gemini https://blog.google/products-and-platforms/products/gemini/google-gemini-deep-research/
  (2024-12-11, 1); OpenAI https://openai.com/index/introducing-deep-research/ (Feb-2025, 1) +
  https://techcrunch.com/2025/02/02/openai-unveils-a-new-chatgpt-agent-for-deep-research/ (4);
  Ntoulas 2005 ACM JCDL 10.1145/1065385.1065407 (3); Chakrabarti 1999 Computer Networks
  10.1016/S1389-1286(99)00052-3 (3); Olston & Najork 2010 FnTIR 10.1561/1500000017 (3).

## 7. Four recursive verification passes

Each pass re-checks the previous pass's optimality claim ("recursive closure over the rsearch
aggregator is the most aggressive feasible strategy") and records its own evidence. All four
passes passed.

- Pass 1 — Optimality argument: formal completeness criterion, cost model and Knaster-Tarski
  dominance proof produced with seven dated, tiered sources (node b042b1d23, 2026-08-07).
- Pass 2 — Engine matches the argument: all eight engine acceptance criteria executed with
  pass verdicts and exact commands (node f7f10c64): rsearch-only source, bounded pool at
  max_concurrency=8, one web/images/describe/chat pipeline, URL+content dedup (64 concurrent
  same-query pushes -> 1 enqueued, 63 skipped), closure decision (NullClient probe closed in 1
  round with 0 new URLs and 0 new queries), logging/annotations, no TODOs, and
  `make verify` -> "Ran 199 tests in 2.168s OK verification passed".
- Pass 3 — Live probe coverage/cost (node fde105db, 2026-08-07): subject "python asyncio",
  max_concurrency=8, count=10, 240 s guard: queries_issued=86, urls_seen=754, contents_seen=281,
  164 network requests (search 105 / chat 46 / describe 13), X-AI-Cost-USD sum $0.002075, wall
  elapsed 264.91 s. Adversarial subjects ("", spaces, tabs) raised ValueError
  "research subject must not be empty" (engine.py:125) before any API call; urlopen delta 0.
- Pass 4 — Closure and determinism (node 1b0176bf, confirmed by fact sheet d5d9e290):
  fixed-fixture fake client closed in 3 rounds with all 4 content types; 4-level chain closed
  in 5 rounds (no depth cap); live `python -m typosaurus_sandbox.research` logged INFO rounds,
  closure and the typed report JSON; final gate `make verify` green (199 tests OK, git clean).

## 8. Usage

```sh
python -m typosaurus_sandbox.research [subject]
```

Run a research session on `subject` (default "typosaurus sandbox") until closure; rounds and
closure decisions are logged at INFO, and the typed `ResearchReport` JSON is logged at the end.
Configuration (base_url, TTLs, max_concurrency, default_count) is loaded from `.env.json` under
the `research` key with plug-and-play defaults (config.py:22).

Verification gate:

```sh
make verify
```

## 9. Verification status

- `make verify`: exit 0, "Ran 199 tests OK verification passed" (2026-08-07); only the
  pre-existing Starlette deprecation warning from the FastAPI test client remains, none
  introduced by the research package.
- Re-run at document time: `make verify` exit 0, "Ran 217 tests in 2.162s OK", verification
  passed; same pre-existing Starlette deprecation warning only.
- No TODO, FIXME, placeholder or stub in the research package (grep verified, node f7f10c64).