stealthii/README.md

# stealthii

`retoor <retoor@molodetz.nl>`

An `aiohttp`/`httpx`-shaped async HTTP client backed by a single,
persistent, stealth-patched Chromium instance. Every request goes out
through the real browser's own network stack, so it carries a genuine
Chrome TLS/HTTP2/UA fingerprint instead of Python's OpenSSL-based one.

This README is the complete documentation. Two companion documents go
deeper on specific topics: [`STEALTH_PATCHES.md`](./STEALTH_PATCHES.md)
(every anti-automation patch applied, and why) and
[`PERFORMANCE.md`](./PERFORMANCE.md) (measured latency and concurrency
overhead against `aiohttp`).

## Why this exists

`requests`, `httpx`, and `aiohttp` all negotiate TLS through Python's
`ssl` module (OpenSSL). Real browsers negotiate through their own TLS
stack (Chrome: BoringSSL). The resulting ClientHello - cipher suites,
extensions, curves, and their order - is a fingerprint (JA3/JA4) that
anti-bot systems read directly off the handshake, independent of
whatever `User-Agent` header the client claims. A Python client cannot
fake this from the application layer; the only way to present a real
Chrome fingerprint is to make the request from an actual Chrome process.

Playwright's `BrowserContext.request` API does exactly that: it issues
plain HTTP requests through the same browser process backing a real
page, without needing a tab or JS execution - as fast as a normal HTTP
call, but with the browser's authentic identity. Measured directly
(`tests/test_stealth_detection.py::test_fast_path_uses_real_chromium_fingerprint`)
against a plain `aiohttp` client hitting the same TLS-fingerprint echo
endpoint:

| Client | JA3 | JA4 |
|---|---|---|
| stealthii (`context.request`) | `944d1e1858cd278718f8a46b65d3212f` | `t13d5211_b262b3658495_8e6e362c5eac` |
| aiohttp | `304734bb1c086c3453b387400cf83f11` | `t13d1812h1_85036bcba153_d41ae481755e` |

Distinct hashes, produced by the same physical request path our
`get`/`post`/etc. use - proof the fast path is not just claiming to be
Chrome, it is Chrome's own network stack. Measured latency and
concurrency overhead of that same fast path against `aiohttp` - steady
state, and 5×/20× concurrent - are in [`PERFORMANCE.md`](./PERFORMANCE.md).

On top of that, `stealthii` removes the JS-level automation tells a
stock headless Chromium exposes - `navigator.webdriver`, a missing
`window.chrome`, an empty plugin list, a SwiftShader WebGL renderer, a
permissions-query mismatch. Every patch and its exact rationale is
documented in [`STEALTH_PATCHES.md`](./STEALTH_PATCHES.md); the result
is verified clean against `bot.sannysoft.com`'s full detection suite on
every test run.

## What this does not do

TLS/HTTP2 impersonation defeats fingerprint-based detection. It does
not defeat IP-reputation-based rate limiting, nor the behavioural layer
of a system like DataDome or a Cloudflare managed challenge - those
need a clean IP and, for the hardest targets, genuine human-like
interaction. Layer `stealthii` with residential proxies and session
warm-up (`session=`, below) for those targets; do not expect it alone
to be a universal bypass. See "Known limitations" in
[`STEALTH_PATCHES.md`](./STEALTH_PATCHES.md) for the specific gaps,
including that the fast request path negotiates HTTP/1.1 rather than
HTTP/2.

## Dependencies

Exactly one: [`playwright`](https://playwright.dev/python/) (`>=1.40.0`),
declared in `pyproject.toml`. No other third-party runtime dependency -
no `requests`, no `aiohttp`, no stealth-plugin package. The Chromium
binary itself is installed separately via `playwright install chromium`
(see Install, below); it is Playwright's own dependency, not
stealthii's.

## Install

```bash
pip install -e .
playwright install chromium
```

## The `Stealthii` object: construction, laziness, lifetime

`Stealthii()` does no I/O. It only records configuration. The browser
process, its stealth-patched contexts, and every tab are created lazily,
on the first call that actually needs them - the first `get()`,
`post()`, `render()`, `page()`, or an explicit `start()`.

```python
from stealthii import Stealthii

stealth = Stealthii()   # cheap - nothing has launched yet
```

An instance is *not* a Python singleton in the enforced sense (nothing
stops you from constructing several) - but a `Stealthii` instance owns
one Chromium process and every context derived from it, so the intended
usage is exactly like an `aiohttp.ClientSession` or an `httpx.Client`:
construct one, hand it to whatever needs it, and share it for the
lifetime of your application. Constructing a second instance launches a
second, entirely separate Chromium process; there is no coordination
between instances.

Call `start()` once at application boot if you want the browser launch
latency paid up front rather than by whichever request happens to be
first:

```python
await stealth.start()      # optional - every method below also does this lazily
```

Call `shutdown()` once when your application exits:

```python
await stealth.shutdown()   # closes every context and the browser process
```

In an `aiohttp` web application, wire these into `on_startup`/`on_cleanup`;
in any other async app, wherever your own startup/shutdown hooks live.

## HTTP methods

`get`, `post`, `put`, `patch`, `delete`, `head` all accept the request
shape you already know from `aiohttp`/`httpx`:

```python
resp = await stealth.get(url, params={"q": "x"}, headers={...}, timeout=15)
resp.status
await resp.json()
await resp.text()
await resp.read()          # raw bytes
```

- `params` - query string parameters (dict).
- `headers` - request headers (dict).
- `data` - a `dict` is sent form-urlencoded; `bytes`/`str` is sent as
  the raw body. Matches both libraries' `data=` behaviour.
- `json` - JSON-encoded body; sets `Content-Type: application/json`.
- `files` - multipart file upload, `httpx`-style:
  `files={"upload": ("name.txt", b"content")}`. Combine with `data` for
  the plain form fields alongside the file(s).
- `cookies` - a per-call `dict`, layered on top of whatever the
  context's own cookie jar already holds for that URL, without
  mutating the jar. Matches `aiohttp`/`httpx`'s per-call `cookies=`.
- `auth` - an `(username, password)` tuple, sent as HTTP Basic auth.
- `allow_redirects` (the `aiohttp` spelling) and `follow_redirects`
  (the `httpx` spelling) are both accepted; either disables following
  redirects when passed `False`.
- `max_redirects` - cap when following redirects (default `20`).
- `timeout` - seconds; falls back to the instance's `request_timeout`.
- `session` - see "Named sessions", below.

```python
await stealth.post(url, json={"hello": "world"})
await stealth.post(url, data={"x": "1"})                       # form-urlencoded
await stealth.post(url, data={"field": "value"},
                   files={"upload": ("name.txt", b"content")}) # multipart
await stealth.get(url, cookies={"flavor": "choc"})
await stealth.get(url, auth=("user", "pass"))
await stealth.get(url, allow_redirects=False)
```

None of these open a tab - they go through the shared context's
`context.request` fast path, described above.

### `download()`

Raw bytes, for images, PDFs, or any other binary attachment - the same
fast, no-tab path as `get()`:

```python
png_bytes = await stealth.download("https://example.com/photo.png")
```

## `render()` and `screenshot()`: full page execution

For targets that need a JS challenge solved - an SPA that only renders
results client-side, a Cloudflare-style "verifying your browser"
interstitial - escalate to a real page load with `render()`:

```python
result = await stealth.render(url, collect_links=True)
result["status"]   # HTTP status of the final navigation
result["text"]     # document.body.innerText after JS has run and settled
result["links"]    # only populated when collect_links=True
```

`screenshot()` renders the same way and captures a PNG:

```python
png_b64 = await stealth.screenshot(url)                    # base64 string
path = await stealth.screenshot(url, path="shot.png")       # saved to disk, path returned
```

With `path` given, the file is written and `path` is returned. Without
it, the PNG bytes come back base64-encoded - convenient for embedding
directly in a data URI or a JSON API response.

## Raw page access: `page()` and `async with stealth as page`

For anything the high-level API does not cover, `page()` yields a real
Playwright `Page` - already stealth-patched - and guarantees it (and,
for an anonymous call, its context) closes again even if your code
raises:

```python
async with stealth.page() as pg:
    await pg.goto(url)
    await pg.click("#accept-cookies")
    await pg.fill("#search", "query")
```

`Stealthii` also supports using the instance itself as the context
manager, claiming a tab directly:

```python
async with stealth as pg:
    await pg.goto(url)
    title = await pg.title()
```

This is exactly `page()` under the hood, with one guarantee on top:
it is task-safe. Two coroutines running `async with stealth as pg` at
the same time on the same shared `Stealthii` instance each get, and
independently release, their own tab - the claim is tracked per
`asyncio` task, not on the instance itself, so concurrent use never
cross-wires which `pg` belongs to which caller.

## Named sessions: cookie persistence and warm-up

Every call above is anonymous by default - it uses one shared context
that is transparently recycled (see below), with no cookie continuity
implied between calls. Pass `session="some-name"` to any of
`get`/`post`/`put`/`patch`/`delete`/`head`/`download`/`render`/
`screenshot`/`page` to use a dedicated, reused context under that name
instead. Cookies set under one name persist across every subsequent
call using that same name, in either direction - a `render()` call can
warm up a session that a later `get()` then reuses, or vice versa,
because both operate on the very same underlying browser context:

```python
await stealth.render("https://example.com/login", session="acme")   # solves a JS challenge, sets cookies
resp = await stealth.get("https://example.com/api/data", session="acme")  # reuses those cookies
```

Anonymous calls and named-session calls never share cookies with each
other - each name (including the implicit `None` for anonymous calls)
is its own isolated context.

## Configuration

Everything is set once, at construction:

```python
stealth = Stealthii(
    max_tabs=4,                            # concurrent render()/page()/screenshot() tabs
    fingerprint="ja4",                     # "ja3" or "ja4" - see fingerprint_info()
    request_timeout=15.0,                  # default seconds for get/post/... 
    max_retries=3,                         # attempts before raising, for both request() and render()/goto
    disk_cache_bytes=1,                    # Chromium's own on-disk cache size, kept near-zero
    recycle_context_after=(500, 1800.0),   # (max requests, max seconds) per context, named or not
    recycle_browser_after_seconds=6*3600,  # full relaunch cadence, or None to disable
)
```

`fingerprint` selects which hash `fingerprint_info()` reports - it does
not change what is sent on the wire; both values describe the same
genuine Chromium handshake, just in a different published format:

```python
await stealth.fingerprint_info()
# {"mode": "ja4", "hash": "t13d5211_...", "http_version": "HTTP/1.1"}
```

### Why the recycle settings exist

A browser that runs for the entire lifetime of a process otherwise
accumulates cookies and an on-disk HTTP cache without bound - a slow,
silent resource leak in anything long-running. `stealthii` closes and
recreates any context, anonymous or named, once it crosses
`recycle_context_after` (request count or age, whichever comes first),
and fully relaunches the browser process - discarding its whole profile
directory - every `recycle_browser_after_seconds`. A named session that
needs to stay warm longer than the default 30 minutes / 500 requests
should get a larger `recycle_context_after` passed at construction.

## Error handling

- `StealthUnavailableError` - the browser itself could not be started
  (Playwright not installed, or the Chromium launch failed). Raised
  from `start()` and anything that needs the browser.
- `StealthRequestError` - a `get`/`post`/`put`/`patch`/`delete`/`head`/
  `download` call failed outright after `max_retries` attempts, each
  separated by a short backoff. The retry is recursive and bounded, not
  an unbounded loop.
- `render()`/`screenshot()`/`page()`'s navigation retries the same way,
  bounded by `max_retries`, but returns a `status: None` result on
  final failure rather than raising, since a page render already
  produces a body of best-effort content even when the final navigation
  attempt did not fully succeed.
- Both the tab (`page()`) and, for anonymous calls, its context are
  guaranteed to close in a `finally`, even when your code inside
  `async with stealth.page() as pg:` (or `async with stealth as pg:`)
  raises.

## One instance, one event loop

A `Stealthii` instance binds its lock, semaphore, browser process, and
every context to whichever `asyncio` event loop is running the first
time it is actually used. Using the same instance again from a
*different* event loop raises `StealthUnavailableError` immediately
(rather than hanging) - the fix is either a fresh `Stealthii()` per loop,
or making sure your event loop does not change across the calls that
share one instance. This matters most for test suites: `pytest-asyncio`
defaults to a new event loop per test function, which breaks a
module-level `Stealthii` singleton shared across tests. Set:

```ini
[pytest]
asyncio_default_test_loop_scope = session
asyncio_default_fixture_loop_scope = session
```

so the whole test session shares one loop - matching how a real
application actually runs (one process, one loop, for its entire
lifetime), and how `Stealthii` is meant to be used.

## Testing

```bash
make test-unit          # no network required
make test-integration   # hits real sites; requires Chromium installed
```

`tests/test_client.py` covers construction, configuration, and the
pure request-shaping logic without any network access.
`tests/test_http_methods.py` and `tests/test_stealth_detection.py` are
integration tests (`@pytest.mark.online`) that exercise every method
above - including named-session persistence, per-call cookies, auth,
multipart upload, retries against an unreachable host, and the
sannysoft stealth regression suite - against real endpoints.
`scripts/smoke_live.py` is a plain manual script covering the same
ground end-to-end, useful for a quick sanity check outside pytest.