retoor/stealthii
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fail fast with a clear error on event loop reuse instead of hanging

Browse Source 
A Stealthii instance binds its lock, browser, and contexts to whichever
asyncio event loop is running the first time it is used. Using it again
from a different loop previously hung forever (the lock/semaphore/
Playwright transport are all bound to the dead first loop and never
wake the waiting coroutine). Detected in rsearch's test suite, where
pytest-asyncio's default per-test-function event loop broke the
module-level Stealthii singleton shared across tests.

Now raises StealthUnavailableError immediately with an actionable
message. README documents the constraint and the pytest-asyncio config
fix (asyncio_default_test_loop_scope = session).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0116i7dYLNZTXNR7Lqf439bV
This commit is contained in:
retoor
2026-09-10 01:58:55 +02:00
co-authored by Claude Sonnet 5
parent 23cdad8966
commit 423170e1bb
5 changed files with 105 additions and 61 deletions
Download Patch File Download Diff File Expand all files Collapse all files
PERFORMANCE.md
+11 -11
View File
@@ -7,7 +7,7 @@ fast path (`get`/`post`/`put`/`patch`/`delete`/`head`/`download`) against
a plain `aiohttp.ClientSession`, and the fast path's behaviour under
concurrency. All numbers below were measured against public, internet-hosted
endpoints (`httpbin.org`), so absolute times include normal internet
latency and third-party server jitter — the comparison between the two
latency and third-party server jitter - the comparison between the two
clients on the same run is the meaningful signal, not the absolute
seconds.
@@ -15,13 +15,13 @@ seconds.
| | One-time | Per request (steady state, n=10) |
|---|---|---|
| Browser launch (`start()`) | ~0.96s | — |
| First request (context warm-up) | ~0.47s | — |
| stealthii `get()` | — | avg 0.184s (min 0.101s, max 0.791s) |
| aiohttp `get()` | — | avg 0.168s (min 0.102s, max 0.441s) |
| Browser launch (`start()`) | ~0.96s | - |
| First request (context warm-up) | ~0.47s | - |
| stealthii `get()` | - | avg 0.184s (min 0.101s, max 0.791s) |
| aiohttp `get()` | - | avg 0.168s (min 0.102s, max 0.441s) |
Browser launch and the shared context's warm-up are each paid exactly
once per process — in a host application, at `start()` during startup,
once per process - in a host application, at `start()` during startup,
not per request. Steady-state, the fast path costs roughly 15-20ms more
per request than `aiohttp` on average, which is within normal network
jitter for a single request (the observed max-min spread on both clients
@@ -40,7 +40,7 @@ rather than approaching `1s`:
| 20× | 2.033s, 20/20 succeeded | 1.867s, 20/20 succeeded |
Both clients complete 20 concurrent 1-second-delay requests in
~2 seconds total, not 20 seconds — confirming the fast path's requests
~2 seconds total, not 20 seconds - confirming the fast path's requests
against the one shared `BrowserContext` are not serialized through some
hidden single-flight point (the underlying CDP connection, the Python
process, or the browser's own request handling). The 5× round-to-round
@@ -56,12 +56,12 @@ requests), the fast path's overhead relative to `aiohttp` is negligible
next to ordinary network latency, and it does not degrade under
concurrency up to at least 20 simultaneous requests on one shared
context. The one meaningfully different cost is the one-time browser
launch (~1s) — pay it once at application startup (`await
launch (~1s) - pay it once at application startup (`await
stealth.start()` in an `on_startup` hook, or accept it landing on
whichever request happens to be first) rather than per request.
`render()`/`screenshot()`/`page()` (full page navigation) were not
benchmarked here — they are inherently much heavier than the fast path
benchmarked here - they are inherently much heavier than the fast path
(loading a real document, executing its JS, subject to `max_tabs`), by
design and by necessity for the targets that require them. Reach for the
fast path by default; escalate to page rendering only for targets that
@@ -70,7 +70,7 @@ actually need JS execution.
## Methodology notes
- Measured with Python's `time.monotonic()` around `asyncio.gather()`
batches, one process, one machine, one network path — not a controlled
batches, one process, one machine, one network path - not a controlled
lab benchmark. Re-running these will not reproduce the exact numbers,
only the same qualitative shape (comparable per-request cost, no
concurrency bottleneck).
@@ -79,6 +79,6 @@ actually need JS execution.
earlier attempt at this same measurement against
`html.duckduckgo.com/html/` was discarded after DuckDuckGo started
returning `202` instead of `200` after a handful of identical rapid
requests — a server-side response to request pattern, not a
requests - a server-side response to request pattern, not a
client-timing artifact, but one that would have corrupted a latency
comparison.
README.md
+64 -42
View File
@@ -17,8 +17,8 @@ overhead against `aiohttp`).
`requests`, `httpx`, and `aiohttp` all negotiate TLS through Python's
`ssl` module (OpenSSL). Real browsers negotiate through their own TLS
stack (Chrome: BoringSSL). The resulting ClientHello — cipher suites,
extensions, curves, and their order — is a fingerprint (JA3/JA4) that
stack (Chrome: BoringSSL). The resulting ClientHello - cipher suites,
extensions, curves, and their order - is a fingerprint (JA3/JA4) that
anti-bot systems read directly off the handshake, independent of
whatever `User-Agent` header the client claims. A Python client cannot
fake this from the application layer; the only way to present a real
@@ -26,7 +26,7 @@ Chrome fingerprint is to make the request from an actual Chrome process.
Playwright's `BrowserContext.request` API does exactly that: it issues
plain HTTP requests through the same browser process backing a real
page, without needing a tab or JS execution — as fast as a normal HTTP
page, without needing a tab or JS execution - as fast as a normal HTTP
call, but with the browser's authentic identity. Measured directly
(`tests/test_stealth_detection.py::test_fast_path_uses_real_chromium_fingerprint`)
against a plain `aiohttp` client hitting the same TLS-fingerprint echo
@@ -38,13 +38,13 @@ endpoint:
| aiohttp | `304734bb1c086c3453b387400cf83f11` | `t13d1812h1_85036bcba153_d41ae481755e` |
Distinct hashes, produced by the same physical request path our
`get`/`post`/etc. use — proof the fast path is not just claiming to be
`get`/`post`/etc. use - proof the fast path is not just claiming to be
Chrome, it is Chrome's own network stack. Measured latency and
concurrency overhead of that same fast path against `aiohttp` — steady
state, and 5×/20× concurrent — are in [`PERFORMANCE.md`](./PERFORMANCE.md).
concurrency overhead of that same fast path against `aiohttp` - steady
state, and 5×/20× concurrent - are in [`PERFORMANCE.md`](./PERFORMANCE.md).
On top of that, `stealthii` removes the JS-level automation tells a
stock headless Chromium exposes — `navigator.webdriver`, a missing
stock headless Chromium exposes - `navigator.webdriver`, a missing
`window.chrome`, an empty plugin list, a SwiftShader WebGL renderer, a
permissions-query mismatch. Every patch and its exact rationale is
documented in [`STEALTH_PATCHES.md`](./STEALTH_PATCHES.md); the result
@@ -55,7 +55,7 @@ every test run.
TLS/HTTP2 impersonation defeats fingerprint-based detection. It does
not defeat IP-reputation-based rate limiting, nor the behavioural layer
of a system like DataDome or a Cloudflare managed challenge — those
of a system like DataDome or a Cloudflare managed challenge - those
need a clean IP and, for the hardest targets, genuine human-like
interaction. Layer `stealthii` with residential proxies and session
warm-up (`session=`, below) for those targets; do not expect it alone
@@ -67,7 +67,7 @@ HTTP/2.
## Dependencies
Exactly one: [`playwright`](https://playwright.dev/python/) (`>=1.40.0`),
declared in `pyproject.toml`. No other third-party runtime dependency —
declared in `pyproject.toml`. No other third-party runtime dependency -
no `requests`, no `aiohttp`, no stealth-plugin package. The Chromium
binary itself is installed separately via `playwright install chromium`
(see Install, below); it is Playwright's own dependency, not
@@ -84,17 +84,17 @@ playwright install chromium
`Stealthii()` does no I/O. It only records configuration. The browser
process, its stealth-patched contexts, and every tab are created lazily,
on the first call that actually needs them — the first `get()`,
on the first call that actually needs them - the first `get()`,
`post()`, `render()`, `page()`, or an explicit `start()`.
```python
from stealthii import Stealthii
stealth = Stealthii() # cheap — nothing has launched yet
stealth = Stealthii() # cheap - nothing has launched yet
```
An instance is *not* a Python singleton in the enforced sense (nothing
stops you from constructing several) — but a `Stealthii` instance owns
stops you from constructing several) - but a `Stealthii` instance owns
one Chromium process and every context derived from it, so the intended
usage is exactly like an `aiohttp.ClientSession` or an `httpx.Client`:
construct one, hand it to whatever needs it, and share it for the
@@ -107,7 +107,7 @@ latency paid up front rather than by whichever request happens to be
first:
```python
await stealth.start() # optional — every method below also does this lazily
await stealth.start() # optional - every method below also does this lazily
```
Call `shutdown()` once when your application exits:
@@ -132,24 +132,24 @@ await resp.text()
await resp.read() # raw bytes
```
- `params` — query string parameters (dict).
- `headers` — request headers (dict).
- `data` — a `dict` is sent form-urlencoded; `bytes`/`str` is sent as
- `params` - query string parameters (dict).
- `headers` - request headers (dict).
- `data` - a `dict` is sent form-urlencoded; `bytes`/`str` is sent as
the raw body. Matches both libraries' `data=` behaviour.
- `json` — JSON-encoded body; sets `Content-Type: application/json`.
- `files` — multipart file upload, `httpx`-style:
- `json` - JSON-encoded body; sets `Content-Type: application/json`.
- `files` - multipart file upload, `httpx`-style:
`files={"upload": ("name.txt", b"content")}`. Combine with `data` for
the plain form fields alongside the file(s).
- `cookies` — a per-call `dict`, layered on top of whatever the
- `cookies` - a per-call `dict`, layered on top of whatever the
context's own cookie jar already holds for that URL, without
mutating the jar. Matches `aiohttp`/`httpx`'s per-call `cookies=`.
- `auth` — an `(username, password)` tuple, sent as HTTP Basic auth.
- `auth` - an `(username, password)` tuple, sent as HTTP Basic auth.
- `allow_redirects` (the `aiohttp` spelling) and `follow_redirects`
(the `httpx` spelling) are both accepted; either disables following
redirects when passed `False`.
- `max_redirects` — cap when following redirects (default `20`).
- `timeout` — seconds; falls back to the instance's `request_timeout`.
- `session` — see "Named sessions", below.
- `max_redirects` - cap when following redirects (default `20`).
- `timeout` - seconds; falls back to the instance's `request_timeout`.
- `session` - see "Named sessions", below.
```python
await stealth.post(url, json={"hello": "world"})
@@ -161,12 +161,12 @@ await stealth.get(url, auth=("user", "pass"))
await stealth.get(url, allow_redirects=False)
```
None of these open a tab — they go through the shared context's
None of these open a tab - they go through the shared context's
`context.request` fast path, described above.
### `download()`
Raw bytes, for images, PDFs, or any other binary attachment — the same
Raw bytes, for images, PDFs, or any other binary attachment - the same
fast, no-tab path as `get()`:
```python
@@ -175,9 +175,9 @@ png_bytes = await stealth.download("https://example.com/photo.png")
## `render()` and `screenshot()`: full page execution
For targets that need a JS challenge solved — an SPA that only renders
For targets that need a JS challenge solved - an SPA that only renders
results client-side, a Cloudflare-style "verifying your browser"
interstitial — escalate to a real page load with `render()`:
interstitial - escalate to a real page load with `render()`:
```python
result = await stealth.render(url, collect_links=True)
@@ -194,13 +194,13 @@ path = await stealth.screenshot(url, path="shot.png") # saved to disk, pat
```
With `path` given, the file is written and `path` is returned. Without
it, the PNG bytes come back base64-encoded — convenient for embedding
it, the PNG bytes come back base64-encoded - convenient for embedding
directly in a data URI or a JSON API response.
## Raw page access: `page()` and `async with stealth as page`
For anything the high-level API does not cover, `page()` yields a real
Playwright `Page` — already stealth-patched — and guarantees it (and,
Playwright `Page` - already stealth-patched - and guarantees it (and,
for an anonymous call, its context) closes again even if your code
raises:
@@ -223,19 +223,19 @@ async with stealth as pg:
This is exactly `page()` under the hood, with one guarantee on top:
it is task-safe. Two coroutines running `async with stealth as pg` at
the same time on the same shared `Stealthii` instance each get, and
independently release, their own tab — the claim is tracked per
independently release, their own tab - the claim is tracked per
`asyncio` task, not on the instance itself, so concurrent use never
cross-wires which `pg` belongs to which caller.
## Named sessions: cookie persistence and warm-up
Every call above is anonymous by default — it uses one shared context
Every call above is anonymous by default - it uses one shared context
that is transparently recycled (see below), with no cookie continuity
implied between calls. Pass `session="some-name"` to any of
`get`/`post`/`put`/`patch`/`delete`/`head`/`download`/`render`/
`screenshot`/`page` to use a dedicated, reused context under that name
instead. Cookies set under one name persist across every subsequent
call using that same name, in either direction — a `render()` call can
call using that same name, in either direction - a `render()` call can
warm up a session that a later `get()` then reuses, or vice versa,
because both operate on the very same underlying browser context:
@@ -245,7 +245,7 @@ resp = await stealth.get("https://example.com/api/data", session="acme") # reus
```
Anonymous calls and named-session calls never share cookies with each
other — each name (including the implicit `None` for anonymous calls)
other - each name (including the implicit `None` for anonymous calls)
is its own isolated context.
## Configuration
@@ -255,7 +255,7 @@ Everything is set once, at construction:
```python
stealth = Stealthii(
max_tabs=4, # concurrent render()/page()/screenshot() tabs
fingerprint="ja4", # "ja3" or "ja4" — see fingerprint_info()
fingerprint="ja4", # "ja3" or "ja4" - see fingerprint_info()
request_timeout=15.0, # default seconds for get/post/...
max_retries=3, # attempts before raising, for both request() and render()/goto
disk_cache_bytes=1, # Chromium's own on-disk cache size, kept near-zero
@@ -264,7 +264,7 @@ stealth = Stealthii(
)
```
`fingerprint` selects which hash `fingerprint_info()` reports — it does
`fingerprint` selects which hash `fingerprint_info()` reports - it does
not change what is sent on the wire; both values describe the same
genuine Chromium handshake, just in a different published format:
@@ -276,21 +276,21 @@ await stealth.fingerprint_info()
### Why the recycle settings exist
A browser that runs for the entire lifetime of a process otherwise
accumulates cookies and an on-disk HTTP cache without bound — a slow,
accumulates cookies and an on-disk HTTP cache without bound - a slow,
silent resource leak in anything long-running. `stealthii` closes and
recreates any context, anonymous or named, once it crosses
`recycle_context_after` (request count or age, whichever comes first),
and fully relaunches the browser process — discarding its whole profile
directory — every `recycle_browser_after_seconds`. A named session that
and fully relaunches the browser process - discarding its whole profile
directory - every `recycle_browser_after_seconds`. A named session that
needs to stay warm longer than the default 30 minutes / 500 requests
should get a larger `recycle_context_after` passed at construction.
## Error handling
- `StealthUnavailableError` — the browser itself could not be started
- `StealthUnavailableError` - the browser itself could not be started
(Playwright not installed, or the Chromium launch failed). Raised
from `start()` and anything that needs the browser.
- `StealthRequestError` — a `get`/`post`/`put`/`patch`/`delete`/`head`/
- `StealthRequestError` - a `get`/`post`/`put`/`patch`/`delete`/`head`/
`download` call failed outright after `max_retries` attempts, each
separated by a short backoff. The retry is recursive and bounded, not
an unbounded loop.
@@ -304,6 +304,28 @@ should get a larger `recycle_context_after` passed at construction.
`async with stealth.page() as pg:` (or `async with stealth as pg:`)
raises.
## One instance, one event loop
A `Stealthii` instance binds its lock, semaphore, browser process, and
every context to whichever `asyncio` event loop is running the first
time it is actually used. Using the same instance again from a
*different* event loop raises `StealthUnavailableError` immediately
(rather than hanging) - the fix is either a fresh `Stealthii()` per loop,
or making sure your event loop does not change across the calls that
share one instance. This matters most for test suites: `pytest-asyncio`
defaults to a new event loop per test function, which breaks a
module-level `Stealthii` singleton shared across tests. Set:
```ini
[pytest]
asyncio_default_test_loop_scope = session
asyncio_default_fixture_loop_scope = session
```
so the whole test session shares one loop - matching how a real
application actually runs (one process, one loop, for its entire
lifetime), and how `Stealthii` is meant to be used.
## Testing
```bash
@@ -315,8 +337,8 @@ make test-integration # hits real sites; requires Chromium installed
pure request-shaping logic without any network access.
`tests/test_http_methods.py` and `tests/test_stealth_detection.py` are
integration tests (`@pytest.mark.online`) that exercise every method
above — including named-session persistence, per-call cookies, auth,
above - including named-session persistence, per-call cookies, auth,
multipart upload, retries against an unreachable host, and the
sannysoft stealth regression suite — against real endpoints.
sannysoft stealth regression suite - against real endpoints.
`scripts/smoke_live.py` is a plain manual script covering the same
ground end-to-end, useful for a quick sanity check outside pytest.
STEALTH_PATCHES.md
+7 -7
View File
@@ -17,7 +17,7 @@ Applied in `Stealthii._ensure_browser_locked()` at browser launch.
| `ignore_default_args=['--enable-automation']` | Playwright passes `--enable-automation` to Chromium by default, which enables the automation infobar and related detectable behaviour. Suppressed. |
| `--no-sandbox` | Required to run headless Chromium as a non-root, unprivileged process in this deployment environment. Not a stealth measure. |
| `--disable-dev-shm-usage` | Avoids `/dev/shm` exhaustion under containerized/limited-memory environments. Not a stealth measure. |
| `--disk-cache-size=<bytes>` / `--media-cache-size=<bytes>` | Bounds Chromium's own on-disk HTTP/media cache, configurable via `disk_cache_bytes` (default near-zero). Not a stealth measure — an operational safeguard against unbounded disk growth in a long-lived process. |
| `--disk-cache-size=<bytes>` / `--media-cache-size=<bytes>` | Bounds Chromium's own on-disk HTTP/media cache, configurable via `disk_cache_bytes` (default near-zero). Not a stealth measure - an operational safeguard against unbounded disk growth in a long-lived process. |
## JS-level patches
@@ -42,7 +42,7 @@ and `app` members shaped like the real object's public surface.
A stock headless Chromium reports `denied` for a `notifications`
permission query while `Notification.permission` itself still reports
`default` — a mismatch a real browser never produces. The query is
`default` - a mismatch a real browser never produces. The query is
patched to always return the same value as `Notification.permission`
for that one permission name; all other permission names are passed
through to the original implementation unmodified.
@@ -55,7 +55,7 @@ A stock headless Chromium reports an empty plugin list and a
set (`PDF Viewer`, `Chrome PDF Viewer`, `Chromium PDF Viewer`,
`Microsoft Edge PDF Viewer`, `WebKit built-in PDF`) are constructed, each
individually reassigned the real `Plugin.prototype`, and the containing
array reassigned `PluginArray.prototype` — including
array reassigned `PluginArray.prototype` - including
`Symbol.toStringTag`, so both a property count check and a
`Object.prototype.toString.call(...)` type check pass. `navigator.mimeTypes`
is patched the same way for the corresponding `application/pdf` MIME type.
@@ -71,7 +71,7 @@ already reports plausible values here in most cases.
A stock headless Chromium renders WebGL through SwiftShader (a software
rasterizer), which reports `UNMASKED_VENDOR_WEBGL` (constant `37445`) as
`Google Inc. (Google)` and `UNMASKED_RENDERER_WEBGL` (constant `37446`)
containing the literal string `SwiftShader` — an unambiguous headless
containing the literal string `SwiftShader` - an unambiguous headless
signal checked by essentially every fingerprinting script. Both
constants are intercepted and rewritten to report a real discrete GPU
(`Google Inc. (NVIDIA)` / an ANGLE-wrapped `NVIDIA GeForce RTX 3060`
@@ -80,7 +80,7 @@ string); every other parameter is passed through unmodified.
### `HTMLIFrameElement.prototype.contentWindow`
A known secondary check reads `window.chrome` from inside a freshly
created, same-origin iframe rather than the top-level document — since
created, same-origin iframe rather than the top-level document - since
each document gets its own JS globals, an iframe's `window.chrome` can
be absent even when the top-level one was patched. The `contentWindow`
getter is wrapped so that, whenever the returned window is missing
@@ -94,7 +94,7 @@ Every function or getter installed by the patches above is wrapped in a
real native function produces (`function name() { [native code] }`).
Without this, a page can distinguish a patched native method from a real
one by calling `.toString()` on it and observing actual JavaScript source
instead of the native-code marker — a check used by several stealth-
instead of the native-code marker - a check used by several stealth-
detection scripts specifically to unmask other stealth patches.
## Verification
@@ -115,7 +115,7 @@ the browser's own TLS stack rather than Python's.
## Known limitations
- The `context.request` fast path negotiates HTTP/1.1, not HTTP/2 — a
- The `context.request` fast path negotiates HTTP/1.1, not HTTP/2 - a
real Chrome page load always uses HTTP/2 against an HTTP/2-capable
server. A system that cross-checks the TLS fingerprint against the
negotiated protocol can detect this specific inconsistency. `render()`
stealthii/client.py
+13
View File
@@ -92,10 +92,23 @@ class Stealthii:
self._user_agent = None
self._tab_semaphore = None
self._lock: Optional[asyncio.Lock] = None
self._loop = None
self._attempted = False
self._claimed_pages: Dict[Any, Any] = {}
def _get_lock(self) -> asyncio.Lock:
loop = asyncio.get_running_loop()
if self._loop is None:
self._loop = loop
elif self._loop is not loop:
raise StealthUnavailableError(
'this Stealthii instance was first used on a different asyncio '
'event loop and cannot be reused on a new one (its browser '
'process, lock, and contexts are all bound to that first loop). '
'Construct a fresh Stealthii() per event loop, or make sure your '
'test runner uses one event loop for the whole session (e.g. '
'pytest-asyncio: asyncio_default_test_loop_scope = "session").'
)
if self._lock is None:
self._lock = asyncio.Lock()
return self._lock
tests/test_client.py
+10 -1
View File
@@ -1,7 +1,7 @@
# retoor <retoor@molodetz.nl>
import pytest
from stealthii import Stealthii, StealthResponse
from stealthii import Stealthii, StealthResponse, StealthUnavailableError
from stealthii.client import _ContextSlot
@@ -69,6 +69,15 @@ async def test_stealth_response_shapes_like_aiohttp():
assert r is resp
@pytest.mark.asyncio
async def test_get_lock_raises_clearly_on_event_loop_mismatch():
s = Stealthii()
s._get_lock()
s._loop = object()
with pytest.raises(StealthUnavailableError, match='different asyncio event loop'):
s._get_lock()
def test_context_slot_starts_fresh():
slot = _ContextSlot(context=object())
assert slot.count == 0