stealthii/README.md

stealthii

retoor <retoor@molodetz.nl>

An aiohttp/httpx-shaped async HTTP client backed by a single, persistent, stealth-patched Chromium instance. Every request goes out through the real browser's own network stack, so it carries a genuine Chrome TLS/HTTP2/UA fingerprint instead of Python's OpenSSL-based one.

This README is the complete documentation. Two companion documents go deeper on specific topics: STEALTH_PATCHES.md (every anti-automation patch applied, and why) and PERFORMANCE.md (measured latency and concurrency overhead against aiohttp).

Why this exists

requests, httpx, and aiohttp all negotiate TLS through Python's ssl module (OpenSSL). Real browsers negotiate through their own TLS stack (Chrome: BoringSSL). The resulting ClientHello - cipher suites, extensions, curves, and their order - is a fingerprint (JA3/JA4) that anti-bot systems read directly off the handshake, independent of whatever User-Agent header the client claims. A Python client cannot fake this from the application layer; the only way to present a real Chrome fingerprint is to make the request from an actual Chrome process.

Playwright's BrowserContext.request API does exactly that: it issues plain HTTP requests through the same browser process backing a real page, without needing a tab or JS execution - as fast as a normal HTTP call, but with the browser's authentic identity. Measured directly (tests/test_stealth_detection.py::test_fast_path_uses_real_chromium_fingerprint) against a plain aiohttp client hitting the same TLS-fingerprint echo endpoint:

Client	JA3	JA4
stealthii (context.request)	944d1e1858cd278718f8a46b65d3212f	t13d5211_b262b3658495_8e6e362c5eac
aiohttp	304734bb1c086c3453b387400cf83f11	t13d1812h1_85036bcba153_d41ae481755e

Distinct hashes, produced by the same physical request path our get/post/etc. use - proof the fast path is not just claiming to be Chrome, it is Chrome's own network stack. Measured latency and concurrency overhead of that same fast path against aiohttp - steady state, and 5×/20× concurrent - are in PERFORMANCE.md.

On top of that, stealthii removes the JS-level automation tells a stock headless Chromium exposes - navigator.webdriver, a missing window.chrome, an empty plugin list, a SwiftShader WebGL renderer, a permissions-query mismatch. Every patch and its exact rationale is documented in STEALTH_PATCHES.md; the result is verified clean against bot.sannysoft.com's full detection suite on every test run.

What this does not do

TLS/HTTP2 impersonation defeats fingerprint-based detection. It does not defeat IP-reputation-based rate limiting, nor the behavioural layer of a system like DataDome or a Cloudflare managed challenge - those need a clean IP and, for the hardest targets, genuine human-like interaction. Layer stealthii with residential proxies and session warm-up (session=, below) for those targets; do not expect it alone to be a universal bypass. See "Known limitations" in STEALTH_PATCHES.md for the specific gaps, including that the fast request path negotiates HTTP/1.1 rather than HTTP/2.

Dependencies

Exactly one: playwright (>=1.40.0), declared in pyproject.toml. No other third-party runtime dependency - no requests, no aiohttp, no stealth-plugin package. The Chromium binary itself is installed separately via playwright install chromium (see Install, below); it is Playwright's own dependency, not stealthii's.

Install

pip install -e .
playwright install chromium

The Stealthii object: construction, laziness, lifetime

Stealthii() does no I/O. It only records configuration. The browser process, its stealth-patched contexts, and every tab are created lazily, on the first call that actually needs them - the first get(), post(), render(), page(), or an explicit start().

from stealthii import Stealthii

stealth = Stealthii()   # cheap - nothing has launched yet

An instance is not a Python singleton in the enforced sense (nothing stops you from constructing several) - but a Stealthii instance owns one Chromium process and every context derived from it, so the intended usage is exactly like an aiohttp.ClientSession or an httpx.Client: construct one, hand it to whatever needs it, and share it for the lifetime of your application. Constructing a second instance launches a second, entirely separate Chromium process; there is no coordination between instances.

Call start() once at application boot if you want the browser launch latency paid up front rather than by whichever request happens to be first:

await stealth.start()      # optional - every method below also does this lazily

Call shutdown() once when your application exits:

await stealth.shutdown()   # closes every context and the browser process

In an aiohttp web application, wire these into on_startup/on_cleanup; in any other async app, wherever your own startup/shutdown hooks live.

HTTP methods

get, post, put, patch, delete, head all accept the request shape you already know from aiohttp/httpx:

resp = await stealth.get(url, params={"q": "x"}, headers={...}, timeout=15)
resp.status
await resp.json()
await resp.text()
await resp.read()          # raw bytes

• params - query string parameters (dict).
• headers - request headers (dict).
• data - a dict is sent form-urlencoded; bytes/str is sent as the raw body. Matches both libraries' data= behaviour.
• json - JSON-encoded body; sets Content-Type: application/json.
• files - multipart file upload, httpx-style: files={"upload": ("name.txt", b"content")}. Combine with data for the plain form fields alongside the file(s).
• cookies - a per-call dict, layered on top of whatever the context's own cookie jar already holds for that URL, without mutating the jar. Matches aiohttp/httpx's per-call cookies=.
• auth - an (username, password) tuple, sent as HTTP Basic auth.
• allow_redirects (the aiohttp spelling) and follow_redirects (the httpx spelling) are both accepted; either disables following redirects when passed False.
• max_redirects - cap when following redirects (default 20).
• timeout - seconds; falls back to the instance's request_timeout.
• session - see "Named sessions", below.

await stealth.post(url, json={"hello": "world"})
await stealth.post(url, data={"x": "1"})                       # form-urlencoded
await stealth.post(url, data={"field": "value"},
                   files={"upload": ("name.txt", b"content")}) # multipart
await stealth.get(url, cookies={"flavor": "choc"})
await stealth.get(url, auth=("user", "pass"))
await stealth.get(url, allow_redirects=False)

None of these open a tab - they go through the shared context's context.request fast path, described above.

download()

Raw bytes, for images, PDFs, or any other binary attachment - the same fast, no-tab path as get():

png_bytes = await stealth.download("https://example.com/photo.png")

render() and screenshot(): full page execution

For targets that need a JS challenge solved - an SPA that only renders results client-side, a Cloudflare-style "verifying your browser" interstitial - escalate to a real page load with render():

result = await stealth.render(url, collect_links=True)
result["status"]   # HTTP status of the final navigation
result["text"]     # document.body.innerText after JS has run and settled
result["links"]    # only populated when collect_links=True

screenshot() renders the same way and captures a PNG:

png_b64 = await stealth.screenshot(url)                    # base64 string
path = await stealth.screenshot(url, path="shot.png")       # saved to disk, path returned

With path given, the file is written and path is returned. Without it, the PNG bytes come back base64-encoded - convenient for embedding directly in a data URI or a JSON API response.

Raw page access: page() and async with stealth as page

For anything the high-level API does not cover, page() yields a real Playwright Page - already stealth-patched - and guarantees it (and, for an anonymous call, its context) closes again even if your code raises:

async with stealth.page() as pg:
    await pg.goto(url)
    await pg.click("#accept-cookies")
    await pg.fill("#search", "query")

Stealthii also supports using the instance itself as the context manager, claiming a tab directly:

async with stealth as pg:
    await pg.goto(url)
    title = await pg.title()

This is exactly page() under the hood, with one guarantee on top: it is task-safe. Two coroutines running async with stealth as pg at the same time on the same shared Stealthii instance each get, and independently release, their own tab - the claim is tracked per asyncio task, not on the instance itself, so concurrent use never cross-wires which pg belongs to which caller.

Named sessions: cookie persistence and warm-up

Every call above is anonymous by default - it uses one shared context that is transparently recycled (see below), with no cookie continuity implied between calls. Pass session="some-name" to any of get/post/put/patch/delete/head/download/render/ screenshot/page to use a dedicated, reused context under that name instead. Cookies set under one name persist across every subsequent call using that same name, in either direction - a render() call can warm up a session that a later get() then reuses, or vice versa, because both operate on the very same underlying browser context:

await stealth.render("https://example.com/login", session="acme")   # solves a JS challenge, sets cookies
resp = await stealth.get("https://example.com/api/data", session="acme")  # reuses those cookies

Anonymous calls and named-session calls never share cookies with each other - each name (including the implicit None for anonymous calls) is its own isolated context.

Configuration

Everything is set once, at construction:

stealth = Stealthii(
    max_tabs=4,                            # concurrent render()/page()/screenshot() tabs
    fingerprint="ja4",                     # "ja3" or "ja4" - see fingerprint_info()
    request_timeout=15.0,                  # default seconds for get/post/... 
    max_retries=3,                         # attempts before raising, for both request() and render()/goto
    disk_cache_bytes=1,                    # Chromium's own on-disk cache size, kept near-zero
    recycle_context_after=(500, 1800.0),   # (max requests, max seconds) per context, named or not
    recycle_browser_after_seconds=6*3600,  # full relaunch cadence, or None to disable
)

fingerprint selects which hash fingerprint_info() reports - it does not change what is sent on the wire; both values describe the same genuine Chromium handshake, just in a different published format:

await stealth.fingerprint_info()
# {"mode": "ja4", "hash": "t13d5211_...", "http_version": "HTTP/1.1"}

Why the recycle settings exist

A browser that runs for the entire lifetime of a process otherwise accumulates cookies and an on-disk HTTP cache without bound - a slow, silent resource leak in anything long-running. stealthii closes and recreates any context, anonymous or named, once it crosses recycle_context_after (request count or age, whichever comes first), and fully relaunches the browser process - discarding its whole profile directory - every recycle_browser_after_seconds. A named session that needs to stay warm longer than the default 30 minutes / 500 requests should get a larger recycle_context_after passed at construction.

Error handling

• StealthUnavailableError - the browser itself could not be started (Playwright not installed, or the Chromium launch failed). Raised from start() and anything that needs the browser.
• StealthRequestError - a get/post/put/patch/delete/head/ download call failed outright after max_retries attempts, each separated by a short backoff. The retry is recursive and bounded, not an unbounded loop.
• render()/screenshot()/page()'s navigation retries the same way, bounded by max_retries, but returns a status: None result on final failure rather than raising, since a page render already produces a body of best-effort content even when the final navigation attempt did not fully succeed.
• Both the tab (page()) and, for anonymous calls, its context are guaranteed to close in a finally, even when your code inside async with stealth.page() as pg: (or async with stealth as pg:) raises.

One instance, one event loop

A Stealthii instance binds its lock, semaphore, browser process, and every context to whichever asyncio event loop is running the first time it is actually used. Using the same instance again from a different event loop raises StealthUnavailableError immediately (rather than hanging) - the fix is either a fresh Stealthii() per loop, or making sure your event loop does not change across the calls that share one instance. This matters most for test suites: pytest-asyncio defaults to a new event loop per test function, which breaks a module-level Stealthii singleton shared across tests. Set:

[pytest]
asyncio_default_test_loop_scope = session
asyncio_default_fixture_loop_scope = session

so the whole test session shares one loop - matching how a real application actually runs (one process, one loop, for its entire lifetime), and how Stealthii is meant to be used.

Testing

make test-unit          # no network required
make test-integration   # hits real sites; requires Chromium installed

tests/test_client.py covers construction, configuration, and the pure request-shaping logic without any network access. tests/test_http_methods.py and tests/test_stealth_detection.py are integration tests (@pytest.mark.online) that exercise every method above - including named-session persistence, per-call cookies, auth, multipart upload, retries against an unreachable host, and the sannysoft stealth regression suite - against real endpoints. scripts/smoke_live.py is a plain manual script covering the same ground end-to-end, useful for a quick sanity check outside pytest.