retoor retoor@molodetz.nl

Deep Research Engine — Design, Optimality Argument and Verification Evidence

This document describes the exhaustive deep research engine in src/typosaurus_sandbox/research/, the mathematical argument that recursive query expansion with URL/content deduplication and closure detection is the most aggressive feasible research strategy over the rsearch API, and the four recursive verification passes executed against it. Every claim is traceable to the run's verified nodes (fact sheet node d5d9e290; optimality node b042b1d23; tester nodes f7f10c64, fde105db, 2e6bd38b; engine node 1b0176bf) and to source path:line references.

1. Scope and constraints

  • Only search API: https://rsearch.app.molodetz.nl; the client issues requests only to the /search, /chat and /describe endpoints (client.py:211). /search is GET-only.
  • Content-type agnostic: web results, image results (type=images), describe and chat flow through one asynchronous pipeline; no per-type special casing beyond parameter selection.
  • Native Python 3.12, standard library only (asyncio, urllib); no new dependency was added.
  • No artificial depth cap, page cap or time budget stops a run before closure; the engine stops only when a full round adds zero new URLs and zero new queries (least fixed point).

2. Architecture (module map)

Module Public symbol Path:line
config ResearchConfig (base_url, TTLs, max_concurrency=8, default_count) src/typosaurus_sandbox/research/config.py:12
client RsearchClient, RsearchError (search/chat/describe, _request) src/typosaurus_sandbox/research/client.py:72
cache TTLCache, CacheEntry (thread-safe, monotonic expiry) src/typosaurus_sandbox/research/cache.py:20
envelopes SearchResponse, SearchResult, DeepReport, ChatResponse, DescribeResponse src/typosaurus_sandbox/research/envelopes.py:103
frontier QueryFrontier, DedupStats, URL normalization, content fingerprint src/typosaurus_sandbox/research/frontier.py:102
pipeline ResearchPipeline, WorkItem, PipelineReport (bounded worker pool) src/typosaurus_sandbox/research/pipeline.py:125
engine ResearchEngine, ResearchReport, RoundSummary (closure loop) src/typosaurus_sandbox/research/engine.py:85
entry main() CLI src/typosaurus_sandbox/research/__main__.py:24

3. Concurrency model

  • Bounded asyncio worker pool: asyncio.Semaphore(pool_size) with pool_size = max(1, max_concurrency) and max_concurrency = 8 (config.py:18, pipeline.py:126-136).
  • run() drains the frontier through a bounded queue (pool * 4) with pool-size workers and None sentinels; every request runs via asyncio.to_thread over urllib (no extra deps).
  • Pool size is logged at INFO; every request outcome (endpoint, query/url, status, cache hit) at INFO, every extraction at DEBUG.

4. Deduplication and closure strategy

  • Query dedup key: whitespace-collapsed casefold (frontier.py:28); length window 2-200 chars.
  • URL dedup: normalize_url lowercases scheme/host, applies IDNA, strips default port, userinfo and fragment, collapses slashes (frontier.py:28).
  • Content dedup: SHA-256 fingerprint of whitespace-normalized text (frontier.py:61).
  • One threading.Lock guards all seen-sets and counters for concurrent worker access (frontier.py:103).
  • Closure rule: a round that adds 0 new URLs and 0 new queries halts the run (engine.py:178-183). The engine is closed-loop verified: a fixed-fixture fake client closed in 3 rounds with all four content types, and a 4-level chain client closed in 5 rounds, proving no depth cap (engine node 1b0176bf).

5. Content-type agnosticism

  • One worker path serves all kinds: web -> search(content=True), images -> search(type="images"), describe -> GET /describe?url=, chat -> POST /chat (pipeline.py:138-143, engine.py:106).
  • Extraction yields new URLs and new query seeds from titles, descriptions and extra fields of every content type (frontier.py:66).

6. Optimality argument

Let R(q) be the set of result URLs returned by the aggregator for query q, gen(u) the query variants generated from URL/content u, and S the set of collected URLs.

  • Completeness: the process is coverage-complete for subject t iff it halts at the least fixed point S* = lfp(F) with F(S) = S ∪ ⋃_{u∈S, q∈gen(u)} R(q); the halt condition is "a full round adds 0 new URLs and 0 new queries" (node b042b1d23).
  • Dominance: depth-d iteration reaches F^d(S0) ⊆ S*; the inclusion is strict whenever the discovery chain exceeds d, so every fixed-depth strategy is incomplete. Closure iterates F to its unique least fixed point (Knaster-Tarski), attaining the maximum reachable coverage; any strategy that stops before the fixed point is strictly dominated.
  • Cost model: Cost = Σ_{q∈Q_issued} c(q) + Σ_{u∈F_issued} c_c(u). Search (5 min) and content (24 h) caches (config.py:16-17) make repeat queries near-free; the dominant cost is |Q_issued| + |F_issued|, and query/URL dedup touches each element exactly once.
  • Stated assumptions and limits: single aggregator (rsearch only), no pagination API, documented count bound 1-100 with the provider capping at 10, and content retrieval only through the aggregator. Optimality is proven within these constraints.
  • Dated references (tier): rsearch docs https://rsearch.app.molodetz.nl/about (2026-08-07, 1); Gemini https://blog.google/products-and-platforms/products/gemini/google-gemini-deep-research/ (2024-12-11, 1); OpenAI https://openai.com/index/introducing-deep-research/ (Feb-2025, 1) + https://techcrunch.com/2025/02/02/openai-unveils-a-new-chatgpt-agent-for-deep-research/ (4); Ntoulas 2005 ACM JCDL 10.1145/1065385.1065407 (3); Chakrabarti 1999 Computer Networks 10.1016/S1389-1286(99)00052-3 (3); Olston & Najork 2010 FnTIR 10.1561/1500000017 (3).

7. Four recursive verification passes

Each pass re-checks the previous pass's optimality claim ("recursive closure over the rsearch aggregator is the most aggressive feasible strategy") and records its own evidence. All four passes passed.

  • Pass 1 — Optimality argument: formal completeness criterion, cost model and Knaster-Tarski dominance proof produced with seven dated, tiered sources (node b042b1d23, 2026-08-07).
  • Pass 2 — Engine matches the argument: all eight engine acceptance criteria executed with pass verdicts and exact commands (node f7f10c64): rsearch-only source, bounded pool at max_concurrency=8, one web/images/describe/chat pipeline, URL+content dedup (64 concurrent same-query pushes -> 1 enqueued, 63 skipped), closure decision (NullClient probe closed in 1 round with 0 new URLs and 0 new queries), logging/annotations, no deferred markers, and make verify -> "Ran 199 tests in 2.168s OK verification passed".
  • Pass 3 — Live probe coverage/cost (node fde105db, 2026-08-07): subject "python asyncio", max_concurrency=8, count=10, 240 s guard: queries_issued=86, urls_seen=754, contents_seen=281, 164 network requests (search 105 / chat 46 / describe 13), X-AI-Cost-USD sum $0.002075, wall elapsed 264.91 s. Adversarial subjects ("", spaces, tabs) raised ValueError "research subject must not be empty" (engine.py:125) before any API call; urlopen delta 0.
  • Pass 4 — Closure and determinism (node 1b0176bf, confirmed by fact sheet d5d9e290): fixed-fixture fake client closed in 3 rounds with all 4 content types; 4-level chain closed in 5 rounds (no depth cap); live python -m typosaurus_sandbox.research logged INFO rounds, closure and the typed report JSON; final gate make verify green (199 tests OK, git clean).

8. Usage

python -m typosaurus_sandbox.research [subject]

Run a research session on subject (default "typosaurus sandbox") until closure; rounds and closure decisions are logged at INFO, and the typed ResearchReport JSON is logged at the end. Configuration (base_url, TTLs, max_concurrency, default_count) is loaded from .env.json under the research key with plug-and-play defaults (config.py:22).

Verification gate:

make verify

9. Verification status

  • make verify: exit 0, "Ran 199 tests OK verification passed" (2026-08-07); only the pre-existing Starlette deprecation warning from the FastAPI test client remains, none introduced by the research package.
  • Re-run at document time: make verify exit 0, "Ran 217 tests in 2.162s OK", verification passed; same pre-existing Starlette deprecation warning only.
  • The research package contains no deferred markers (grep verified, node f7f10c64).