stealthii/README.md

stealthii

retoor <retoor@molodetz.nl>

An aiohttp/httpx-shaped async HTTP client backed by a single, persistent, stealth-patched Chromium instance. Every request goes out through the real browser's own network stack, so it carries a genuine Chrome TLS/HTTP2/UA fingerprint instead of Python's OpenSSL-based one — the signal most anti-bot systems key on first, before they ever look at a header.

Why

requests, httpx, and aiohttp all negotiate TLS through Python's ssl module (OpenSSL). Real browsers negotiate through their own TLS stack (Chrome: BoringSSL). The resulting ClientHello — cipher suites, extensions, curves, and their order — is a fingerprint (JA3/JA4) that anti-bot systems read directly off the handshake, independent of whatever User-Agent header the client claims. A Python client cannot fake this from the application layer; the only way to get a real Chrome fingerprint is to make the request from an actual Chrome process.

Playwright's BrowserContext.request API does exactly that: it issues plain HTTP requests through the same browser process backing a real page, without needing a tab or JS execution — as fast as a normal HTTP call, but with the browser's authentic identity. stealthii wraps that fast path (get/post/put/patch/delete/head/download) behind a familiar API, and adds full page rendering (render/screenshot/page) as a heavier fallback for targets that require solving a JS challenge.

On top of that, stealthii removes the JS-level automation tells a stock headless Chromium exposes (navigator.webdriver, missing window.chrome, an empty plugin list, a SwiftShader WebGL renderer, a permissions-query mismatch) — verified clean against bot.sannysoft.com's full check suite.

What it does not do

TLS/HTTP2 impersonation defeats fingerprint-based detection. It does not defeat IP-reputation-based rate limiting, nor the behavioural layer of a system like DataDome or a Cloudflare managed challenge — those need a clean IP and, for the hardest targets, genuine human-like interaction. Layer stealthii with residential proxies and session warm-up (session=, see below) for those targets; don't expect it alone to be a universal bypass.

Install

pip install -e .
playwright install chromium

Usage

from stealthii import Stealthii

stealth = Stealthii()          # cheap, lazy — launches nothing yet
resp = await stealth.get("https://example.com/api", params={"q": "x"})
resp.status
await resp.json()
await resp.text()

await stealth.shutdown()       # call once at host process shutdown

get/post/put/patch/delete/head accept the same shapes as aiohttp/httpx: params, headers, data (dict → form-urlencoded, bytes/str → raw body), json, files (multipart), cookies (per-call, layered on the context's own jar), auth as an (user, password) Basic tuple, allow_redirects / follow_redirects (both spellings accepted), max_redirects, and timeout.

data = await stealth.download("https://example.com/photo.png")  # raw bytes

png_b64 = await stealth.screenshot("https://example.com")        # base64 PNG
await stealth.screenshot("https://example.com", path="shot.png")  # saved to disk

rendered = await stealth.render("https://example.com", collect_links=True)
rendered["text"]; rendered["links"]; rendered["status"]

Named sessions (cookie persistence / warm-up)

Every call is anonymous by default (a shared, periodically recycled context). Pass session="name" to use a dedicated, reused context instead — cookies set by one call under that name are seen by the next:

await stealth.render("https://example.com/", session="acme")       # picks up cookies
resp = await stealth.get("https://example.com/api", session="acme") # reuses them

Raw page access

For anything the high-level API doesn't cover, page() yields a real Playwright Page (already stealth-patched), guaranteed to close even on exception:

async with stealth.page() as pg:
    await pg.goto("https://example.com")
    await pg.click("#accept-cookies")

Configuration

All configuration is passed once, at construction — Stealthii(...) does no I/O itself; the browser launches lazily on first real use:

stealth = Stealthii(
    max_tabs=4,                            # concurrent render()/page() tabs
    fingerprint="ja4",                     # "ja3" or "ja4" — see fingerprint_info()
    request_timeout=15.0,
    max_retries=3,
    disk_cache_bytes=1,                    # keep Chromium's own on-disk cache tiny
    recycle_context_after=(500, 1800.0),   # (max requests, max seconds) per context
    recycle_browser_after_seconds=6*3600,  # full relaunch cadence, or None to disable
)

fingerprint selects which hash fingerprint_info() reports (JA3 or JA4) — it does not change what's sent on the wire; both describe the same genuine Chromium handshake.

Why the recycle settings exist

A browser that runs for the process's entire lifetime otherwise accumulates cookies and an on-disk HTTP cache without bound. stealthii closes and recreates any context — anonymous or named — once it crosses recycle_context_after, and fully relaunches the browser (discarding its whole profile directory) every recycle_browser_after_seconds. Named sessions meant to stay warm longer should get a larger recycle_context_after at construction time.

Testing

make test-unit          # no network required
make test-integration   # hits real sites; requires Chromium installed