retoor/stealthii
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add PUT/PATCH/DELETE/HEAD, full aiohttp/httpx param parity, screenshots,

Browse Source 
async-with tab claiming, and full documentation

- get/post/put/patch/delete/head now accept the same request shape as
  aiohttp/httpx: data/json/files/cookies/auth/allow_redirects/
  follow_redirects/max_redirects, plus the existing session= for named
  cookie-persisting contexts.
- screenshot(url, path=None) captures a PNG, saved to disk or returned
  as base64.
- `async with stealth as page` claims a tab directly, task-safe for
  concurrent use on one shared instance (keyed off asyncio.current_task,
  not instance state).
- Strip all docstrings/comments from the source (author line excepted),
  per house style.
- README.md rewritten as the complete documentation: rationale, JA3/JA4
  comparison, dependencies, construction/laziness/lifetime, every method
  with examples, named sessions, configuration, error handling, testing.
- STEALTH_PATCHES.md: a literal, patch-by-patch account of every launch
  argument and JS patch applied, with rationale and verification method.
- 20 new tests: full HTTP-method coverage, cookies, auth, multipart,
  download, render, screenshot, named-session isolation, retry/error
  paths, and guaranteed page/context cleanup under exception.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0116i7dYLNZTXNR7Lqf439bV
This commit is contained in:
retoor
2026-09-09 13:56:28 +02:00
co-authored by Claude Sonnet 5
parent a3bb7639d2
commit 86c3a95cb8
7 changed files with 540 additions and 225 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+250 -78
View File
@@ -2,45 +2,68 @@
`retoor <retoor@molodetz.nl>`
An `aiohttp`/`httpx`-shaped async HTTP client backed by a single, persistent,
stealth-patched Chromium instance. Every request goes out through the real
browser's own network stack, so it carries a genuine Chrome TLS/HTTP2/UA
fingerprint instead of Python's OpenSSL-based one — the signal most
anti-bot systems key on first, before they ever look at a header.
An `aiohttp`/`httpx`-shaped async HTTP client backed by a single,
persistent, stealth-patched Chromium instance. Every request goes out
through the real browser's own network stack, so it carries a genuine
Chrome TLS/HTTP2/UA fingerprint instead of Python's OpenSSL-based one.
## Why
## Why this exists
`requests`, `httpx`, and `aiohttp` all negotiate TLS through Python's `ssl`
module (OpenSSL). Real browsers negotiate through their own TLS stack
(Chrome: BoringSSL). The resulting ClientHello — cipher suites, extensions,
curves, and their order — is a fingerprint (JA3/JA4) that anti-bot systems
read directly off the handshake, independent of whatever `User-Agent`
header the client claims. A Python client cannot fake this from the
application layer; the only way to get a real Chrome fingerprint is to
make the request from an actual Chrome process.
`requests`, `httpx`, and `aiohttp` all negotiate TLS through Python's
`ssl` module (OpenSSL). Real browsers negotiate through their own TLS
stack (Chrome: BoringSSL). The resulting ClientHello — cipher suites,
extensions, curves, and their order — is a fingerprint (JA3/JA4) that
anti-bot systems read directly off the handshake, independent of
whatever `User-Agent` header the client claims. A Python client cannot
fake this from the application layer; the only way to present a real
Chrome fingerprint is to make the request from an actual Chrome process.
Playwright's `BrowserContext.request` API does exactly that: it issues
plain HTTP requests through the same browser process backing a real page,
without needing a tab or JS execution — as fast as a normal HTTP call, but
with the browser's authentic identity. `stealthii` wraps that fast path
(`get`/`post`/`put`/`patch`/`delete`/`head`/`download`) behind a familiar
API, and adds full page rendering (`render`/`screenshot`/`page`) as a
heavier fallback for targets that require solving a JS challenge.
plain HTTP requests through the same browser process backing a real
page, without needing a tab or JS execution — as fast as a normal HTTP
call, but with the browser's authentic identity. Measured directly
(`tests/test_stealth_detection.py::test_fast_path_uses_real_chromium_fingerprint`)
against a plain `aiohttp` client hitting the same TLS-fingerprint echo
endpoint:
On top of that, `stealthii` removes the JS-level automation tells a stock
headless Chromium exposes (`navigator.webdriver`, missing `window.chrome`,
an empty plugin list, a SwiftShader WebGL renderer, a permissions-query
mismatch) — verified clean against `bot.sannysoft.com`'s full check suite.
| Client | JA3 | JA4 |
|---|---|---|
| stealthii (`context.request`) | `944d1e1858cd278718f8a46b65d3212f` | `t13d5211_b262b3658495_8e6e362c5eac` |
| aiohttp | `304734bb1c086c3453b387400cf83f11` | `t13d1812h1_85036bcba153_d41ae481755e` |
## What it does not do
Distinct hashes, produced by the same physical request path our
`get`/`post`/etc. use — proof the fast path is not just claiming to be
Chrome, it is Chrome's own network stack.
TLS/HTTP2 impersonation defeats fingerprint-based detection. It does not
defeat IP-reputation-based rate limiting, nor the behavioural layer of a
system like DataDome or a Cloudflare managed challenge — those need a
clean IP and, for the hardest targets, genuine human-like interaction.
Layer `stealthii` with residential proxies and session warm-up
(`session=`, see below) for those targets; don't expect it alone to be a
universal bypass.
On top of that, `stealthii` removes the JS-level automation tells a
stock headless Chromium exposes — `navigator.webdriver`, a missing
`window.chrome`, an empty plugin list, a SwiftShader WebGL renderer, a
permissions-query mismatch. Every patch and its exact rationale is
documented in [`STEALTH_PATCHES.md`](./STEALTH_PATCHES.md); the result
is verified clean against `bot.sannysoft.com`'s full detection suite on
every test run.
## What this does not do
TLS/HTTP2 impersonation defeats fingerprint-based detection. It does
not defeat IP-reputation-based rate limiting, nor the behavioural layer
of a system like DataDome or a Cloudflare managed challenge — those
need a clean IP and, for the hardest targets, genuine human-like
interaction. Layer `stealthii` with residential proxies and session
warm-up (`session=`, below) for those targets; do not expect it alone
to be a universal bypass. See "Known limitations" in
[`STEALTH_PATCHES.md`](./STEALTH_PATCHES.md) for the specific gaps,
including that the fast request path negotiates HTTP/1.1 rather than
HTTP/2.
## Dependencies
Exactly one: [`playwright`](https://playwright.dev/python/) (`>=1.40.0`),
declared in `pyproject.toml`. No other third-party runtime dependency —
no `requests`, no `aiohttp`, no stealth-plugin package. The Chromium
binary itself is installed separately via `playwright install chromium`
(see Install, below); it is Playwright's own dependency, not
stealthii's.
## Install
@@ -49,90 +72,229 @@ pip install -e .
playwright install chromium
```
## Usage
## The `Stealthii` object: construction, laziness, lifetime
`Stealthii()` does no I/O. It only records configuration. The browser
process, its stealth-patched contexts, and every tab are created lazily,
on the first call that actually needs them — the first `get()`,
`post()`, `render()`, `page()`, or an explicit `start()`.
```python
from stealthii import Stealthii
stealth = Stealthii() # cheap, lazy — launches nothing yet
resp = await stealth.get("https://example.com/api", params={"q": "x"})
stealth = Stealthii() # cheap — nothing has launched yet
```
An instance is *not* a Python singleton in the enforced sense (nothing
stops you from constructing several) — but a `Stealthii` instance owns
one Chromium process and every context derived from it, so the intended
usage is exactly like an `aiohttp.ClientSession` or an `httpx.Client`:
construct one, hand it to whatever needs it, and share it for the
lifetime of your application. Constructing a second instance launches a
second, entirely separate Chromium process; there is no coordination
between instances.
Call `start()` once at application boot if you want the browser launch
latency paid up front rather than by whichever request happens to be
first:
```python
await stealth.start() # optional — every method below also does this lazily
```
Call `shutdown()` once when your application exits:
```python
await stealth.shutdown() # closes every context and the browser process
```
In an `aiohttp` web application, wire these into `on_startup`/`on_cleanup`;
in any other async app, wherever your own startup/shutdown hooks live.
## HTTP methods
`get`, `post`, `put`, `patch`, `delete`, `head` all accept the request
shape you already know from `aiohttp`/`httpx`:
```python
resp = await stealth.get(url, params={"q": "x"}, headers={...}, timeout=15)
resp.status
await resp.json()
await resp.text()
await stealth.shutdown() # call once at host process shutdown
await resp.read() # raw bytes
```
`get`/`post`/`put`/`patch`/`delete`/`head` accept the same shapes as
`aiohttp`/`httpx`: `params`, `headers`, `data` (dict → form-urlencoded,
bytes/str → raw body), `json`, `files` (multipart), `cookies` (per-call,
layered on the context's own jar), `auth` as an `(user, password)` Basic
tuple, `allow_redirects` / `follow_redirects` (both spellings accepted),
`max_redirects`, and `timeout`.
- `params` — query string parameters (dict).
- `headers` — request headers (dict).
- `data` — a `dict` is sent form-urlencoded; `bytes`/`str` is sent as
the raw body. Matches both libraries' `data=` behaviour.
- `json` — JSON-encoded body; sets `Content-Type: application/json`.
- `files` — multipart file upload, `httpx`-style:
`files={"upload": ("name.txt", b"content")}`. Combine with `data` for
the plain form fields alongside the file(s).
- `cookies` — a per-call `dict`, layered on top of whatever the
context's own cookie jar already holds for that URL, without
mutating the jar. Matches `aiohttp`/`httpx`'s per-call `cookies=`.
- `auth` — an `(username, password)` tuple, sent as HTTP Basic auth.
- `allow_redirects` (the `aiohttp` spelling) and `follow_redirects`
(the `httpx` spelling) are both accepted; either disables following
redirects when passed `False`.
- `max_redirects` — cap when following redirects (default `20`).
- `timeout` — seconds; falls back to the instance's `request_timeout`.
- `session` — see "Named sessions", below.
```python
data = await stealth.download("https://example.com/photo.png") # raw bytes
png_b64 = await stealth.screenshot("https://example.com") # base64 PNG
await stealth.screenshot("https://example.com", path="shot.png") # saved to disk
rendered = await stealth.render("https://example.com", collect_links=True)
rendered["text"]; rendered["links"]; rendered["status"]
await stealth.post(url, json={"hello": "world"})
await stealth.post(url, data={"x": "1"}) # form-urlencoded
await stealth.post(url, data={"field": "value"},
files={"upload": ("name.txt", b"content")}) # multipart
await stealth.get(url, cookies={"flavor": "choc"})
await stealth.get(url, auth=("user", "pass"))
await stealth.get(url, allow_redirects=False)
```
### Named sessions (cookie persistence / warm-up)
None of these open a tab — they go through the shared context's
`context.request` fast path, described above.
Every call is anonymous by default (a shared, periodically recycled
context). Pass `session="name"` to use a dedicated, reused context instead
— cookies set by one call under that name are seen by the next:
### `download()`
Raw bytes, for images, PDFs, or any other binary attachment — the same
fast, no-tab path as `get()`:
```python
await stealth.render("https://example.com/", session="acme") # picks up cookies
resp = await stealth.get("https://example.com/api", session="acme") # reuses them
png_bytes = await stealth.download("https://example.com/photo.png")
```
### Raw page access
## `render()` and `screenshot()`: full page execution
For anything the high-level API doesn't cover, `page()` yields a real
Playwright `Page` (already stealth-patched), guaranteed to close even on
exception:
For targets that need a JS challenge solved — an SPA that only renders
results client-side, a Cloudflare-style "verifying your browser"
interstitial — escalate to a real page load with `render()`:
```python
result = await stealth.render(url, collect_links=True)
result["status"] # HTTP status of the final navigation
result["text"] # document.body.innerText after JS has run and settled
result["links"] # only populated when collect_links=True
```
`screenshot()` renders the same way and captures a PNG:
```python
png_b64 = await stealth.screenshot(url) # base64 string
path = await stealth.screenshot(url, path="shot.png") # saved to disk, path returned
```
With `path` given, the file is written and `path` is returned. Without
it, the PNG bytes come back base64-encoded — convenient for embedding
directly in a data URI or a JSON API response.
## Raw page access: `page()` and `async with stealth as page`
For anything the high-level API does not cover, `page()` yields a real
Playwright `Page` — already stealth-patched — and guarantees it (and,
for an anonymous call, its context) closes again even if your code
raises:
```python
async with stealth.page() as pg:
await pg.goto("https://example.com")
await pg.goto(url)
await pg.click("#accept-cookies")
await pg.fill("#search", "query")
```
`Stealthii` also supports using the instance itself as the context
manager, claiming a tab directly:
```python
async with stealth as pg:
await pg.goto(url)
title = await pg.title()
```
This is exactly `page()` under the hood, with one guarantee on top:
it is task-safe. Two coroutines running `async with stealth as pg` at
the same time on the same shared `Stealthii` instance each get, and
independently release, their own tab — the claim is tracked per
`asyncio` task, not on the instance itself, so concurrent use never
cross-wires which `pg` belongs to which caller.
## Named sessions: cookie persistence and warm-up
Every call above is anonymous by default — it uses one shared context
that is transparently recycled (see below), with no cookie continuity
implied between calls. Pass `session="some-name"` to any of
`get`/`post`/`put`/`patch`/`delete`/`head`/`download`/`render`/
`screenshot`/`page` to use a dedicated, reused context under that name
instead. Cookies set under one name persist across every subsequent
call using that same name, in either direction — a `render()` call can
warm up a session that a later `get()` then reuses, or vice versa,
because both operate on the very same underlying browser context:
```python
await stealth.render("https://example.com/login", session="acme") # solves a JS challenge, sets cookies
resp = await stealth.get("https://example.com/api/data", session="acme") # reuses those cookies
```
Anonymous calls and named-session calls never share cookies with each
other — each name (including the implicit `None` for anonymous calls)
is its own isolated context.
## Configuration
All configuration is passed once, at construction — `Stealthii(...)` does
no I/O itself; the browser launches lazily on first real use:
Everything is set once, at construction:
```python
stealth = Stealthii(
max_tabs=4, # concurrent render()/page() tabs
max_tabs=4, # concurrent render()/page()/screenshot() tabs
fingerprint="ja4", # "ja3" or "ja4" — see fingerprint_info()
request_timeout=15.0,
max_retries=3,
disk_cache_bytes=1, # keep Chromium's own on-disk cache tiny
recycle_context_after=(500, 1800.0), # (max requests, max seconds) per context
request_timeout=15.0, # default seconds for get/post/...
max_retries=3, # attempts before raising, for both request() and render()/goto
disk_cache_bytes=1, # Chromium's own on-disk cache size, kept near-zero
recycle_context_after=(500, 1800.0), # (max requests, max seconds) per context, named or not
recycle_browser_after_seconds=6*3600, # full relaunch cadence, or None to disable
)
```
`fingerprint` selects which hash `fingerprint_info()` reports (JA3 or
JA4) — it does not change what's sent on the wire; both describe the same
genuine Chromium handshake.
`fingerprint` selects which hash `fingerprint_info()` reports — it does
not change what is sent on the wire; both values describe the same
genuine Chromium handshake, just in a different published format:
```python
await stealth.fingerprint_info()
# {"mode": "ja4", "hash": "t13d5211_...", "http_version": "HTTP/1.1"}
```
### Why the recycle settings exist
A browser that runs for the process's entire lifetime otherwise
accumulates cookies and an on-disk HTTP cache without bound. `stealthii`
closes and recreates any context — anonymous or named — once it crosses
`recycle_context_after`, and fully relaunches the browser (discarding its
whole profile directory) every `recycle_browser_after_seconds`. Named
sessions meant to stay warm longer should get a larger
`recycle_context_after` at construction time.
A browser that runs for the entire lifetime of a process otherwise
accumulates cookies and an on-disk HTTP cache without bound — a slow,
silent resource leak in anything long-running. `stealthii` closes and
recreates any context, anonymous or named, once it crosses
`recycle_context_after` (request count or age, whichever comes first),
and fully relaunches the browser process — discarding its whole profile
directory — every `recycle_browser_after_seconds`. A named session that
needs to stay warm longer than the default 30 minutes / 500 requests
should get a larger `recycle_context_after` passed at construction.
## Error handling
- `StealthUnavailableError` — the browser itself could not be started
(Playwright not installed, or the Chromium launch failed). Raised
from `start()` and anything that needs the browser.
- `StealthRequestError` — a `get`/`post`/`put`/`patch`/`delete`/`head`/
`download` call failed outright after `max_retries` attempts, each
separated by a short backoff. The retry is recursive and bounded, not
an unbounded loop.
- `render()`/`screenshot()`/`page()`'s navigation retries the same way,
bounded by `max_retries`, but returns a `status: None` result on
final failure rather than raising, since a page render already
produces a body of best-effort content even when the final navigation
attempt did not fully succeed.
- Both the tab (`page()`) and, for anonymous calls, its context are
guaranteed to close in a `finally`, even when your code inside
`async with stealth.page() as pg:` (or `async with stealth as pg:`)
raises.
## Testing
@@ -140,3 +302,13 @@ sessions meant to stay warm longer should get a larger
make test-unit # no network required
make test-integration # hits real sites; requires Chromium installed
```
`tests/test_client.py` covers construction, configuration, and the
pure request-shaping logic without any network access.
`tests/test_http_methods.py` and `tests/test_stealth_detection.py` are
integration tests (`@pytest.mark.online`) that exercise every method
above — including named-session persistence, per-call cookies, auth,
multipart upload, retries against an unreachable host, and the
sannysoft stealth regression suite — against real endpoints.
`scripts/smoke_live.py` is a plain manual script covering the same
ground end-to-end, useful for a quick sanity check outside pytest.