diff --git a/PERFORMANCE.md b/PERFORMANCE.md index a84afa6..843b344 100644 --- a/PERFORMANCE.md +++ b/PERFORMANCE.md @@ -7,7 +7,7 @@ fast path (`get`/`post`/`put`/`patch`/`delete`/`head`/`download`) against a plain `aiohttp.ClientSession`, and the fast path's behaviour under concurrency. All numbers below were measured against public, internet-hosted endpoints (`httpbin.org`), so absolute times include normal internet -latency and third-party server jitter — the comparison between the two +latency and third-party server jitter - the comparison between the two clients on the same run is the meaningful signal, not the absolute seconds. @@ -15,13 +15,13 @@ seconds. | | One-time | Per request (steady state, n=10) | |---|---|---| -| Browser launch (`start()`) | ~0.96s | — | -| First request (context warm-up) | ~0.47s | — | -| stealthii `get()` | — | avg 0.184s (min 0.101s, max 0.791s) | -| aiohttp `get()` | — | avg 0.168s (min 0.102s, max 0.441s) | +| Browser launch (`start()`) | ~0.96s | - | +| First request (context warm-up) | ~0.47s | - | +| stealthii `get()` | - | avg 0.184s (min 0.101s, max 0.791s) | +| aiohttp `get()` | - | avg 0.168s (min 0.102s, max 0.441s) | Browser launch and the shared context's warm-up are each paid exactly -once per process — in a host application, at `start()` during startup, +once per process - in a host application, at `start()` during startup, not per request. Steady-state, the fast path costs roughly 15-20ms more per request than `aiohttp` on average, which is within normal network jitter for a single request (the observed max-min spread on both clients @@ -40,7 +40,7 @@ rather than approaching `1s`: | 20× | 2.033s, 20/20 succeeded | 1.867s, 20/20 succeeded | Both clients complete 20 concurrent 1-second-delay requests in -~2 seconds total, not 20 seconds — confirming the fast path's requests +~2 seconds total, not 20 seconds - confirming the fast path's requests against the one shared `BrowserContext` are not serialized through some hidden single-flight point (the underlying CDP connection, the Python process, or the browser's own request handling). The 5× round-to-round @@ -56,12 +56,12 @@ requests), the fast path's overhead relative to `aiohttp` is negligible next to ordinary network latency, and it does not degrade under concurrency up to at least 20 simultaneous requests on one shared context. The one meaningfully different cost is the one-time browser -launch (~1s) — pay it once at application startup (`await +launch (~1s) - pay it once at application startup (`await stealth.start()` in an `on_startup` hook, or accept it landing on whichever request happens to be first) rather than per request. `render()`/`screenshot()`/`page()` (full page navigation) were not -benchmarked here — they are inherently much heavier than the fast path +benchmarked here - they are inherently much heavier than the fast path (loading a real document, executing its JS, subject to `max_tabs`), by design and by necessity for the targets that require them. Reach for the fast path by default; escalate to page rendering only for targets that @@ -70,7 +70,7 @@ actually need JS execution. ## Methodology notes - Measured with Python's `time.monotonic()` around `asyncio.gather()` - batches, one process, one machine, one network path — not a controlled + batches, one process, one machine, one network path - not a controlled lab benchmark. Re-running these will not reproduce the exact numbers, only the same qualitative shape (comparable per-request cost, no concurrency bottleneck). @@ -79,6 +79,6 @@ actually need JS execution. earlier attempt at this same measurement against `html.duckduckgo.com/html/` was discarded after DuckDuckGo started returning `202` instead of `200` after a handful of identical rapid - requests — a server-side response to request pattern, not a + requests - a server-side response to request pattern, not a client-timing artifact, but one that would have corrupted a latency comparison. diff --git a/README.md b/README.md index 04f78ba..2dc09be 100644 --- a/README.md +++ b/README.md @@ -17,8 +17,8 @@ overhead against `aiohttp`). `requests`, `httpx`, and `aiohttp` all negotiate TLS through Python's `ssl` module (OpenSSL). Real browsers negotiate through their own TLS -stack (Chrome: BoringSSL). The resulting ClientHello — cipher suites, -extensions, curves, and their order — is a fingerprint (JA3/JA4) that +stack (Chrome: BoringSSL). The resulting ClientHello - cipher suites, +extensions, curves, and their order - is a fingerprint (JA3/JA4) that anti-bot systems read directly off the handshake, independent of whatever `User-Agent` header the client claims. A Python client cannot fake this from the application layer; the only way to present a real @@ -26,7 +26,7 @@ Chrome fingerprint is to make the request from an actual Chrome process. Playwright's `BrowserContext.request` API does exactly that: it issues plain HTTP requests through the same browser process backing a real -page, without needing a tab or JS execution — as fast as a normal HTTP +page, without needing a tab or JS execution - as fast as a normal HTTP call, but with the browser's authentic identity. Measured directly (`tests/test_stealth_detection.py::test_fast_path_uses_real_chromium_fingerprint`) against a plain `aiohttp` client hitting the same TLS-fingerprint echo @@ -38,13 +38,13 @@ endpoint: | aiohttp | `304734bb1c086c3453b387400cf83f11` | `t13d1812h1_85036bcba153_d41ae481755e` | Distinct hashes, produced by the same physical request path our -`get`/`post`/etc. use — proof the fast path is not just claiming to be +`get`/`post`/etc. use - proof the fast path is not just claiming to be Chrome, it is Chrome's own network stack. Measured latency and -concurrency overhead of that same fast path against `aiohttp` — steady -state, and 5×/20× concurrent — are in [`PERFORMANCE.md`](./PERFORMANCE.md). +concurrency overhead of that same fast path against `aiohttp` - steady +state, and 5×/20× concurrent - are in [`PERFORMANCE.md`](./PERFORMANCE.md). On top of that, `stealthii` removes the JS-level automation tells a -stock headless Chromium exposes — `navigator.webdriver`, a missing +stock headless Chromium exposes - `navigator.webdriver`, a missing `window.chrome`, an empty plugin list, a SwiftShader WebGL renderer, a permissions-query mismatch. Every patch and its exact rationale is documented in [`STEALTH_PATCHES.md`](./STEALTH_PATCHES.md); the result @@ -55,7 +55,7 @@ every test run. TLS/HTTP2 impersonation defeats fingerprint-based detection. It does not defeat IP-reputation-based rate limiting, nor the behavioural layer -of a system like DataDome or a Cloudflare managed challenge — those +of a system like DataDome or a Cloudflare managed challenge - those need a clean IP and, for the hardest targets, genuine human-like interaction. Layer `stealthii` with residential proxies and session warm-up (`session=`, below) for those targets; do not expect it alone @@ -67,7 +67,7 @@ HTTP/2. ## Dependencies Exactly one: [`playwright`](https://playwright.dev/python/) (`>=1.40.0`), -declared in `pyproject.toml`. No other third-party runtime dependency — +declared in `pyproject.toml`. No other third-party runtime dependency - no `requests`, no `aiohttp`, no stealth-plugin package. The Chromium binary itself is installed separately via `playwright install chromium` (see Install, below); it is Playwright's own dependency, not @@ -84,17 +84,17 @@ playwright install chromium `Stealthii()` does no I/O. It only records configuration. The browser process, its stealth-patched contexts, and every tab are created lazily, -on the first call that actually needs them — the first `get()`, +on the first call that actually needs them - the first `get()`, `post()`, `render()`, `page()`, or an explicit `start()`. ```python from stealthii import Stealthii -stealth = Stealthii() # cheap — nothing has launched yet +stealth = Stealthii() # cheap - nothing has launched yet ``` An instance is *not* a Python singleton in the enforced sense (nothing -stops you from constructing several) — but a `Stealthii` instance owns +stops you from constructing several) - but a `Stealthii` instance owns one Chromium process and every context derived from it, so the intended usage is exactly like an `aiohttp.ClientSession` or an `httpx.Client`: construct one, hand it to whatever needs it, and share it for the @@ -107,7 +107,7 @@ latency paid up front rather than by whichever request happens to be first: ```python -await stealth.start() # optional — every method below also does this lazily +await stealth.start() # optional - every method below also does this lazily ``` Call `shutdown()` once when your application exits: @@ -132,24 +132,24 @@ await resp.text() await resp.read() # raw bytes ``` -- `params` — query string parameters (dict). -- `headers` — request headers (dict). -- `data` — a `dict` is sent form-urlencoded; `bytes`/`str` is sent as +- `params` - query string parameters (dict). +- `headers` - request headers (dict). +- `data` - a `dict` is sent form-urlencoded; `bytes`/`str` is sent as the raw body. Matches both libraries' `data=` behaviour. -- `json` — JSON-encoded body; sets `Content-Type: application/json`. -- `files` — multipart file upload, `httpx`-style: +- `json` - JSON-encoded body; sets `Content-Type: application/json`. +- `files` - multipart file upload, `httpx`-style: `files={"upload": ("name.txt", b"content")}`. Combine with `data` for the plain form fields alongside the file(s). -- `cookies` — a per-call `dict`, layered on top of whatever the +- `cookies` - a per-call `dict`, layered on top of whatever the context's own cookie jar already holds for that URL, without mutating the jar. Matches `aiohttp`/`httpx`'s per-call `cookies=`. -- `auth` — an `(username, password)` tuple, sent as HTTP Basic auth. +- `auth` - an `(username, password)` tuple, sent as HTTP Basic auth. - `allow_redirects` (the `aiohttp` spelling) and `follow_redirects` (the `httpx` spelling) are both accepted; either disables following redirects when passed `False`. -- `max_redirects` — cap when following redirects (default `20`). -- `timeout` — seconds; falls back to the instance's `request_timeout`. -- `session` — see "Named sessions", below. +- `max_redirects` - cap when following redirects (default `20`). +- `timeout` - seconds; falls back to the instance's `request_timeout`. +- `session` - see "Named sessions", below. ```python await stealth.post(url, json={"hello": "world"}) @@ -161,12 +161,12 @@ await stealth.get(url, auth=("user", "pass")) await stealth.get(url, allow_redirects=False) ``` -None of these open a tab — they go through the shared context's +None of these open a tab - they go through the shared context's `context.request` fast path, described above. ### `download()` -Raw bytes, for images, PDFs, or any other binary attachment — the same +Raw bytes, for images, PDFs, or any other binary attachment - the same fast, no-tab path as `get()`: ```python @@ -175,9 +175,9 @@ png_bytes = await stealth.download("https://example.com/photo.png") ## `render()` and `screenshot()`: full page execution -For targets that need a JS challenge solved — an SPA that only renders +For targets that need a JS challenge solved - an SPA that only renders results client-side, a Cloudflare-style "verifying your browser" -interstitial — escalate to a real page load with `render()`: +interstitial - escalate to a real page load with `render()`: ```python result = await stealth.render(url, collect_links=True) @@ -194,13 +194,13 @@ path = await stealth.screenshot(url, path="shot.png") # saved to disk, pat ``` With `path` given, the file is written and `path` is returned. Without -it, the PNG bytes come back base64-encoded — convenient for embedding +it, the PNG bytes come back base64-encoded - convenient for embedding directly in a data URI or a JSON API response. ## Raw page access: `page()` and `async with stealth as page` For anything the high-level API does not cover, `page()` yields a real -Playwright `Page` — already stealth-patched — and guarantees it (and, +Playwright `Page` - already stealth-patched - and guarantees it (and, for an anonymous call, its context) closes again even if your code raises: @@ -223,19 +223,19 @@ async with stealth as pg: This is exactly `page()` under the hood, with one guarantee on top: it is task-safe. Two coroutines running `async with stealth as pg` at the same time on the same shared `Stealthii` instance each get, and -independently release, their own tab — the claim is tracked per +independently release, their own tab - the claim is tracked per `asyncio` task, not on the instance itself, so concurrent use never cross-wires which `pg` belongs to which caller. ## Named sessions: cookie persistence and warm-up -Every call above is anonymous by default — it uses one shared context +Every call above is anonymous by default - it uses one shared context that is transparently recycled (see below), with no cookie continuity implied between calls. Pass `session="some-name"` to any of `get`/`post`/`put`/`patch`/`delete`/`head`/`download`/`render`/ `screenshot`/`page` to use a dedicated, reused context under that name instead. Cookies set under one name persist across every subsequent -call using that same name, in either direction — a `render()` call can +call using that same name, in either direction - a `render()` call can warm up a session that a later `get()` then reuses, or vice versa, because both operate on the very same underlying browser context: @@ -245,7 +245,7 @@ resp = await stealth.get("https://example.com/api/data", session="acme") # reus ``` Anonymous calls and named-session calls never share cookies with each -other — each name (including the implicit `None` for anonymous calls) +other - each name (including the implicit `None` for anonymous calls) is its own isolated context. ## Configuration @@ -255,7 +255,7 @@ Everything is set once, at construction: ```python stealth = Stealthii( max_tabs=4, # concurrent render()/page()/screenshot() tabs - fingerprint="ja4", # "ja3" or "ja4" — see fingerprint_info() + fingerprint="ja4", # "ja3" or "ja4" - see fingerprint_info() request_timeout=15.0, # default seconds for get/post/... max_retries=3, # attempts before raising, for both request() and render()/goto disk_cache_bytes=1, # Chromium's own on-disk cache size, kept near-zero @@ -264,7 +264,7 @@ stealth = Stealthii( ) ``` -`fingerprint` selects which hash `fingerprint_info()` reports — it does +`fingerprint` selects which hash `fingerprint_info()` reports - it does not change what is sent on the wire; both values describe the same genuine Chromium handshake, just in a different published format: @@ -276,21 +276,21 @@ await stealth.fingerprint_info() ### Why the recycle settings exist A browser that runs for the entire lifetime of a process otherwise -accumulates cookies and an on-disk HTTP cache without bound — a slow, +accumulates cookies and an on-disk HTTP cache without bound - a slow, silent resource leak in anything long-running. `stealthii` closes and recreates any context, anonymous or named, once it crosses `recycle_context_after` (request count or age, whichever comes first), -and fully relaunches the browser process — discarding its whole profile -directory — every `recycle_browser_after_seconds`. A named session that +and fully relaunches the browser process - discarding its whole profile +directory - every `recycle_browser_after_seconds`. A named session that needs to stay warm longer than the default 30 minutes / 500 requests should get a larger `recycle_context_after` passed at construction. ## Error handling -- `StealthUnavailableError` — the browser itself could not be started +- `StealthUnavailableError` - the browser itself could not be started (Playwright not installed, or the Chromium launch failed). Raised from `start()` and anything that needs the browser. -- `StealthRequestError` — a `get`/`post`/`put`/`patch`/`delete`/`head`/ +- `StealthRequestError` - a `get`/`post`/`put`/`patch`/`delete`/`head`/ `download` call failed outright after `max_retries` attempts, each separated by a short backoff. The retry is recursive and bounded, not an unbounded loop. @@ -304,6 +304,28 @@ should get a larger `recycle_context_after` passed at construction. `async with stealth.page() as pg:` (or `async with stealth as pg:`) raises. +## One instance, one event loop + +A `Stealthii` instance binds its lock, semaphore, browser process, and +every context to whichever `asyncio` event loop is running the first +time it is actually used. Using the same instance again from a +*different* event loop raises `StealthUnavailableError` immediately +(rather than hanging) - the fix is either a fresh `Stealthii()` per loop, +or making sure your event loop does not change across the calls that +share one instance. This matters most for test suites: `pytest-asyncio` +defaults to a new event loop per test function, which breaks a +module-level `Stealthii` singleton shared across tests. Set: + +```ini +[pytest] +asyncio_default_test_loop_scope = session +asyncio_default_fixture_loop_scope = session +``` + +so the whole test session shares one loop - matching how a real +application actually runs (one process, one loop, for its entire +lifetime), and how `Stealthii` is meant to be used. + ## Testing ```bash @@ -315,8 +337,8 @@ make test-integration # hits real sites; requires Chromium installed pure request-shaping logic without any network access. `tests/test_http_methods.py` and `tests/test_stealth_detection.py` are integration tests (`@pytest.mark.online`) that exercise every method -above — including named-session persistence, per-call cookies, auth, +above - including named-session persistence, per-call cookies, auth, multipart upload, retries against an unreachable host, and the -sannysoft stealth regression suite — against real endpoints. +sannysoft stealth regression suite - against real endpoints. `scripts/smoke_live.py` is a plain manual script covering the same ground end-to-end, useful for a quick sanity check outside pytest. diff --git a/STEALTH_PATCHES.md b/STEALTH_PATCHES.md index 342e441..d4b132f 100644 --- a/STEALTH_PATCHES.md +++ b/STEALTH_PATCHES.md @@ -17,7 +17,7 @@ Applied in `Stealthii._ensure_browser_locked()` at browser launch. | `ignore_default_args=['--enable-automation']` | Playwright passes `--enable-automation` to Chromium by default, which enables the automation infobar and related detectable behaviour. Suppressed. | | `--no-sandbox` | Required to run headless Chromium as a non-root, unprivileged process in this deployment environment. Not a stealth measure. | | `--disable-dev-shm-usage` | Avoids `/dev/shm` exhaustion under containerized/limited-memory environments. Not a stealth measure. | -| `--disk-cache-size=` / `--media-cache-size=` | Bounds Chromium's own on-disk HTTP/media cache, configurable via `disk_cache_bytes` (default near-zero). Not a stealth measure — an operational safeguard against unbounded disk growth in a long-lived process. | +| `--disk-cache-size=` / `--media-cache-size=` | Bounds Chromium's own on-disk HTTP/media cache, configurable via `disk_cache_bytes` (default near-zero). Not a stealth measure - an operational safeguard against unbounded disk growth in a long-lived process. | ## JS-level patches @@ -42,7 +42,7 @@ and `app` members shaped like the real object's public surface. A stock headless Chromium reports `denied` for a `notifications` permission query while `Notification.permission` itself still reports -`default` — a mismatch a real browser never produces. The query is +`default` - a mismatch a real browser never produces. The query is patched to always return the same value as `Notification.permission` for that one permission name; all other permission names are passed through to the original implementation unmodified. @@ -55,7 +55,7 @@ A stock headless Chromium reports an empty plugin list and a set (`PDF Viewer`, `Chrome PDF Viewer`, `Chromium PDF Viewer`, `Microsoft Edge PDF Viewer`, `WebKit built-in PDF`) are constructed, each individually reassigned the real `Plugin.prototype`, and the containing -array reassigned `PluginArray.prototype` — including +array reassigned `PluginArray.prototype` - including `Symbol.toStringTag`, so both a property count check and a `Object.prototype.toString.call(...)` type check pass. `navigator.mimeTypes` is patched the same way for the corresponding `application/pdf` MIME type. @@ -71,7 +71,7 @@ already reports plausible values here in most cases. A stock headless Chromium renders WebGL through SwiftShader (a software rasterizer), which reports `UNMASKED_VENDOR_WEBGL` (constant `37445`) as `Google Inc. (Google)` and `UNMASKED_RENDERER_WEBGL` (constant `37446`) -containing the literal string `SwiftShader` — an unambiguous headless +containing the literal string `SwiftShader` - an unambiguous headless signal checked by essentially every fingerprinting script. Both constants are intercepted and rewritten to report a real discrete GPU (`Google Inc. (NVIDIA)` / an ANGLE-wrapped `NVIDIA GeForce RTX 3060` @@ -80,7 +80,7 @@ string); every other parameter is passed through unmodified. ### `HTMLIFrameElement.prototype.contentWindow` A known secondary check reads `window.chrome` from inside a freshly -created, same-origin iframe rather than the top-level document — since +created, same-origin iframe rather than the top-level document - since each document gets its own JS globals, an iframe's `window.chrome` can be absent even when the top-level one was patched. The `contentWindow` getter is wrapped so that, whenever the returned window is missing @@ -94,7 +94,7 @@ Every function or getter installed by the patches above is wrapped in a real native function produces (`function name() { [native code] }`). Without this, a page can distinguish a patched native method from a real one by calling `.toString()` on it and observing actual JavaScript source -instead of the native-code marker — a check used by several stealth- +instead of the native-code marker - a check used by several stealth- detection scripts specifically to unmask other stealth patches. ## Verification @@ -115,7 +115,7 @@ the browser's own TLS stack rather than Python's. ## Known limitations -- The `context.request` fast path negotiates HTTP/1.1, not HTTP/2 — a +- The `context.request` fast path negotiates HTTP/1.1, not HTTP/2 - a real Chrome page load always uses HTTP/2 against an HTTP/2-capable server. A system that cross-checks the TLS fingerprint against the negotiated protocol can detect this specific inconsistency. `render()` diff --git a/stealthii/client.py b/stealthii/client.py index c131864..d76c765 100644 --- a/stealthii/client.py +++ b/stealthii/client.py @@ -92,10 +92,23 @@ class Stealthii: self._user_agent = None self._tab_semaphore = None self._lock: Optional[asyncio.Lock] = None + self._loop = None self._attempted = False self._claimed_pages: Dict[Any, Any] = {} def _get_lock(self) -> asyncio.Lock: + loop = asyncio.get_running_loop() + if self._loop is None: + self._loop = loop + elif self._loop is not loop: + raise StealthUnavailableError( + 'this Stealthii instance was first used on a different asyncio ' + 'event loop and cannot be reused on a new one (its browser ' + 'process, lock, and contexts are all bound to that first loop). ' + 'Construct a fresh Stealthii() per event loop, or make sure your ' + 'test runner uses one event loop for the whole session (e.g. ' + 'pytest-asyncio: asyncio_default_test_loop_scope = "session").' + ) if self._lock is None: self._lock = asyncio.Lock() return self._lock diff --git a/tests/test_client.py b/tests/test_client.py index 9b69ea9..9c8ce64 100644 --- a/tests/test_client.py +++ b/tests/test_client.py @@ -1,7 +1,7 @@ # retoor import pytest -from stealthii import Stealthii, StealthResponse +from stealthii import Stealthii, StealthResponse, StealthUnavailableError from stealthii.client import _ContextSlot @@ -69,6 +69,15 @@ async def test_stealth_response_shapes_like_aiohttp(): assert r is resp +@pytest.mark.asyncio +async def test_get_lock_raises_clearly_on_event_loop_mismatch(): + s = Stealthii() + s._get_lock() + s._loop = object() + with pytest.raises(StealthUnavailableError, match='different asyncio event loop'): + s._get_lock() + + def test_context_slot_starts_fresh(): slot = _ContextSlot(context=object()) assert slot.count == 0