Fail fast with a clear error on event loop reuse instead of hanging
A Stealthii instance binds its lock, browser, and contexts to whichever asyncio event loop is running the first time it is used. Using it again from a different loop previously hung forever (the lock/semaphore/ Playwright transport are all bound to the dead first loop and never wake the waiting coroutine). Detected in rsearch's test suite, where pytest-asyncio's default per-test-function event loop broke the module-level Stealthii singleton shared across tests. Now raises StealthUnavailableError immediately with an actionable message. README documents the constraint and the pytest-asyncio config fix (asyncio_default_test_loop_scope = session). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0116i7dYLNZTXNR7Lqf439bV
This commit is contained in:
+11
-11
@@ -7,7 +7,7 @@ fast path (`get`/`post`/`put`/`patch`/`delete`/`head`/`download`) against
|
|||||||
a plain `aiohttp.ClientSession`, and the fast path's behaviour under
|
a plain `aiohttp.ClientSession`, and the fast path's behaviour under
|
||||||
concurrency. All numbers below were measured against public, internet-hosted
|
concurrency. All numbers below were measured against public, internet-hosted
|
||||||
endpoints (`httpbin.org`), so absolute times include normal internet
|
endpoints (`httpbin.org`), so absolute times include normal internet
|
||||||
latency and third-party server jitter — the comparison between the two
|
latency and third-party server jitter - the comparison between the two
|
||||||
clients on the same run is the meaningful signal, not the absolute
|
clients on the same run is the meaningful signal, not the absolute
|
||||||
seconds.
|
seconds.
|
||||||
|
|
||||||
@@ -15,13 +15,13 @@ seconds.
|
|||||||
|
|
||||||
| | One-time | Per request (steady state, n=10) |
|
| | One-time | Per request (steady state, n=10) |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| Browser launch (`start()`) | ~0.96s | — |
|
| Browser launch (`start()`) | ~0.96s | - |
|
||||||
| First request (context warm-up) | ~0.47s | — |
|
| First request (context warm-up) | ~0.47s | - |
|
||||||
| stealthii `get()` | — | avg 0.184s (min 0.101s, max 0.791s) |
|
| stealthii `get()` | - | avg 0.184s (min 0.101s, max 0.791s) |
|
||||||
| aiohttp `get()` | — | avg 0.168s (min 0.102s, max 0.441s) |
|
| aiohttp `get()` | - | avg 0.168s (min 0.102s, max 0.441s) |
|
||||||
|
|
||||||
Browser launch and the shared context's warm-up are each paid exactly
|
Browser launch and the shared context's warm-up are each paid exactly
|
||||||
once per process — in a host application, at `start()` during startup,
|
once per process - in a host application, at `start()` during startup,
|
||||||
not per request. Steady-state, the fast path costs roughly 15-20ms more
|
not per request. Steady-state, the fast path costs roughly 15-20ms more
|
||||||
per request than `aiohttp` on average, which is within normal network
|
per request than `aiohttp` on average, which is within normal network
|
||||||
jitter for a single request (the observed max-min spread on both clients
|
jitter for a single request (the observed max-min spread on both clients
|
||||||
@@ -40,7 +40,7 @@ rather than approaching `1s`:
|
|||||||
| 20× | 2.033s, 20/20 succeeded | 1.867s, 20/20 succeeded |
|
| 20× | 2.033s, 20/20 succeeded | 1.867s, 20/20 succeeded |
|
||||||
|
|
||||||
Both clients complete 20 concurrent 1-second-delay requests in
|
Both clients complete 20 concurrent 1-second-delay requests in
|
||||||
~2 seconds total, not 20 seconds — confirming the fast path's requests
|
~2 seconds total, not 20 seconds - confirming the fast path's requests
|
||||||
against the one shared `BrowserContext` are not serialized through some
|
against the one shared `BrowserContext` are not serialized through some
|
||||||
hidden single-flight point (the underlying CDP connection, the Python
|
hidden single-flight point (the underlying CDP connection, the Python
|
||||||
process, or the browser's own request handling). The 5× round-to-round
|
process, or the browser's own request handling). The 5× round-to-round
|
||||||
@@ -56,12 +56,12 @@ requests), the fast path's overhead relative to `aiohttp` is negligible
|
|||||||
next to ordinary network latency, and it does not degrade under
|
next to ordinary network latency, and it does not degrade under
|
||||||
concurrency up to at least 20 simultaneous requests on one shared
|
concurrency up to at least 20 simultaneous requests on one shared
|
||||||
context. The one meaningfully different cost is the one-time browser
|
context. The one meaningfully different cost is the one-time browser
|
||||||
launch (~1s) — pay it once at application startup (`await
|
launch (~1s) - pay it once at application startup (`await
|
||||||
stealth.start()` in an `on_startup` hook, or accept it landing on
|
stealth.start()` in an `on_startup` hook, or accept it landing on
|
||||||
whichever request happens to be first) rather than per request.
|
whichever request happens to be first) rather than per request.
|
||||||
|
|
||||||
`render()`/`screenshot()`/`page()` (full page navigation) were not
|
`render()`/`screenshot()`/`page()` (full page navigation) were not
|
||||||
benchmarked here — they are inherently much heavier than the fast path
|
benchmarked here - they are inherently much heavier than the fast path
|
||||||
(loading a real document, executing its JS, subject to `max_tabs`), by
|
(loading a real document, executing its JS, subject to `max_tabs`), by
|
||||||
design and by necessity for the targets that require them. Reach for the
|
design and by necessity for the targets that require them. Reach for the
|
||||||
fast path by default; escalate to page rendering only for targets that
|
fast path by default; escalate to page rendering only for targets that
|
||||||
@@ -70,7 +70,7 @@ actually need JS execution.
|
|||||||
## Methodology notes
|
## Methodology notes
|
||||||
|
|
||||||
- Measured with Python's `time.monotonic()` around `asyncio.gather()`
|
- Measured with Python's `time.monotonic()` around `asyncio.gather()`
|
||||||
batches, one process, one machine, one network path — not a controlled
|
batches, one process, one machine, one network path - not a controlled
|
||||||
lab benchmark. Re-running these will not reproduce the exact numbers,
|
lab benchmark. Re-running these will not reproduce the exact numbers,
|
||||||
only the same qualitative shape (comparable per-request cost, no
|
only the same qualitative shape (comparable per-request cost, no
|
||||||
concurrency bottleneck).
|
concurrency bottleneck).
|
||||||
@@ -79,6 +79,6 @@ actually need JS execution.
|
|||||||
earlier attempt at this same measurement against
|
earlier attempt at this same measurement against
|
||||||
`html.duckduckgo.com/html/` was discarded after DuckDuckGo started
|
`html.duckduckgo.com/html/` was discarded after DuckDuckGo started
|
||||||
returning `202` instead of `200` after a handful of identical rapid
|
returning `202` instead of `200` after a handful of identical rapid
|
||||||
requests — a server-side response to request pattern, not a
|
requests - a server-side response to request pattern, not a
|
||||||
client-timing artifact, but one that would have corrupted a latency
|
client-timing artifact, but one that would have corrupted a latency
|
||||||
comparison.
|
comparison.
|
||||||
|
|||||||
@@ -17,8 +17,8 @@ overhead against `aiohttp`).
|
|||||||
|
|
||||||
`requests`, `httpx`, and `aiohttp` all negotiate TLS through Python's
|
`requests`, `httpx`, and `aiohttp` all negotiate TLS through Python's
|
||||||
`ssl` module (OpenSSL). Real browsers negotiate through their own TLS
|
`ssl` module (OpenSSL). Real browsers negotiate through their own TLS
|
||||||
stack (Chrome: BoringSSL). The resulting ClientHello — cipher suites,
|
stack (Chrome: BoringSSL). The resulting ClientHello - cipher suites,
|
||||||
extensions, curves, and their order — is a fingerprint (JA3/JA4) that
|
extensions, curves, and their order - is a fingerprint (JA3/JA4) that
|
||||||
anti-bot systems read directly off the handshake, independent of
|
anti-bot systems read directly off the handshake, independent of
|
||||||
whatever `User-Agent` header the client claims. A Python client cannot
|
whatever `User-Agent` header the client claims. A Python client cannot
|
||||||
fake this from the application layer; the only way to present a real
|
fake this from the application layer; the only way to present a real
|
||||||
@@ -26,7 +26,7 @@ Chrome fingerprint is to make the request from an actual Chrome process.
|
|||||||
|
|
||||||
Playwright's `BrowserContext.request` API does exactly that: it issues
|
Playwright's `BrowserContext.request` API does exactly that: it issues
|
||||||
plain HTTP requests through the same browser process backing a real
|
plain HTTP requests through the same browser process backing a real
|
||||||
page, without needing a tab or JS execution — as fast as a normal HTTP
|
page, without needing a tab or JS execution - as fast as a normal HTTP
|
||||||
call, but with the browser's authentic identity. Measured directly
|
call, but with the browser's authentic identity. Measured directly
|
||||||
(`tests/test_stealth_detection.py::test_fast_path_uses_real_chromium_fingerprint`)
|
(`tests/test_stealth_detection.py::test_fast_path_uses_real_chromium_fingerprint`)
|
||||||
against a plain `aiohttp` client hitting the same TLS-fingerprint echo
|
against a plain `aiohttp` client hitting the same TLS-fingerprint echo
|
||||||
@@ -38,13 +38,13 @@ endpoint:
|
|||||||
| aiohttp | `304734bb1c086c3453b387400cf83f11` | `t13d1812h1_85036bcba153_d41ae481755e` |
|
| aiohttp | `304734bb1c086c3453b387400cf83f11` | `t13d1812h1_85036bcba153_d41ae481755e` |
|
||||||
|
|
||||||
Distinct hashes, produced by the same physical request path our
|
Distinct hashes, produced by the same physical request path our
|
||||||
`get`/`post`/etc. use — proof the fast path is not just claiming to be
|
`get`/`post`/etc. use - proof the fast path is not just claiming to be
|
||||||
Chrome, it is Chrome's own network stack. Measured latency and
|
Chrome, it is Chrome's own network stack. Measured latency and
|
||||||
concurrency overhead of that same fast path against `aiohttp` — steady
|
concurrency overhead of that same fast path against `aiohttp` - steady
|
||||||
state, and 5×/20× concurrent — are in [`PERFORMANCE.md`](./PERFORMANCE.md).
|
state, and 5×/20× concurrent - are in [`PERFORMANCE.md`](./PERFORMANCE.md).
|
||||||
|
|
||||||
On top of that, `stealthii` removes the JS-level automation tells a
|
On top of that, `stealthii` removes the JS-level automation tells a
|
||||||
stock headless Chromium exposes — `navigator.webdriver`, a missing
|
stock headless Chromium exposes - `navigator.webdriver`, a missing
|
||||||
`window.chrome`, an empty plugin list, a SwiftShader WebGL renderer, a
|
`window.chrome`, an empty plugin list, a SwiftShader WebGL renderer, a
|
||||||
permissions-query mismatch. Every patch and its exact rationale is
|
permissions-query mismatch. Every patch and its exact rationale is
|
||||||
documented in [`STEALTH_PATCHES.md`](./STEALTH_PATCHES.md); the result
|
documented in [`STEALTH_PATCHES.md`](./STEALTH_PATCHES.md); the result
|
||||||
@@ -55,7 +55,7 @@ every test run.
|
|||||||
|
|
||||||
TLS/HTTP2 impersonation defeats fingerprint-based detection. It does
|
TLS/HTTP2 impersonation defeats fingerprint-based detection. It does
|
||||||
not defeat IP-reputation-based rate limiting, nor the behavioural layer
|
not defeat IP-reputation-based rate limiting, nor the behavioural layer
|
||||||
of a system like DataDome or a Cloudflare managed challenge — those
|
of a system like DataDome or a Cloudflare managed challenge - those
|
||||||
need a clean IP and, for the hardest targets, genuine human-like
|
need a clean IP and, for the hardest targets, genuine human-like
|
||||||
interaction. Layer `stealthii` with residential proxies and session
|
interaction. Layer `stealthii` with residential proxies and session
|
||||||
warm-up (`session=`, below) for those targets; do not expect it alone
|
warm-up (`session=`, below) for those targets; do not expect it alone
|
||||||
@@ -67,7 +67,7 @@ HTTP/2.
|
|||||||
## Dependencies
|
## Dependencies
|
||||||
|
|
||||||
Exactly one: [`playwright`](https://playwright.dev/python/) (`>=1.40.0`),
|
Exactly one: [`playwright`](https://playwright.dev/python/) (`>=1.40.0`),
|
||||||
declared in `pyproject.toml`. No other third-party runtime dependency —
|
declared in `pyproject.toml`. No other third-party runtime dependency -
|
||||||
no `requests`, no `aiohttp`, no stealth-plugin package. The Chromium
|
no `requests`, no `aiohttp`, no stealth-plugin package. The Chromium
|
||||||
binary itself is installed separately via `playwright install chromium`
|
binary itself is installed separately via `playwright install chromium`
|
||||||
(see Install, below); it is Playwright's own dependency, not
|
(see Install, below); it is Playwright's own dependency, not
|
||||||
@@ -84,17 +84,17 @@ playwright install chromium
|
|||||||
|
|
||||||
`Stealthii()` does no I/O. It only records configuration. The browser
|
`Stealthii()` does no I/O. It only records configuration. The browser
|
||||||
process, its stealth-patched contexts, and every tab are created lazily,
|
process, its stealth-patched contexts, and every tab are created lazily,
|
||||||
on the first call that actually needs them — the first `get()`,
|
on the first call that actually needs them - the first `get()`,
|
||||||
`post()`, `render()`, `page()`, or an explicit `start()`.
|
`post()`, `render()`, `page()`, or an explicit `start()`.
|
||||||
|
|
||||||
```python
|
```python
|
||||||
from stealthii import Stealthii
|
from stealthii import Stealthii
|
||||||
|
|
||||||
stealth = Stealthii() # cheap — nothing has launched yet
|
stealth = Stealthii() # cheap - nothing has launched yet
|
||||||
```
|
```
|
||||||
|
|
||||||
An instance is *not* a Python singleton in the enforced sense (nothing
|
An instance is *not* a Python singleton in the enforced sense (nothing
|
||||||
stops you from constructing several) — but a `Stealthii` instance owns
|
stops you from constructing several) - but a `Stealthii` instance owns
|
||||||
one Chromium process and every context derived from it, so the intended
|
one Chromium process and every context derived from it, so the intended
|
||||||
usage is exactly like an `aiohttp.ClientSession` or an `httpx.Client`:
|
usage is exactly like an `aiohttp.ClientSession` or an `httpx.Client`:
|
||||||
construct one, hand it to whatever needs it, and share it for the
|
construct one, hand it to whatever needs it, and share it for the
|
||||||
@@ -107,7 +107,7 @@ latency paid up front rather than by whichever request happens to be
|
|||||||
first:
|
first:
|
||||||
|
|
||||||
```python
|
```python
|
||||||
await stealth.start() # optional — every method below also does this lazily
|
await stealth.start() # optional - every method below also does this lazily
|
||||||
```
|
```
|
||||||
|
|
||||||
Call `shutdown()` once when your application exits:
|
Call `shutdown()` once when your application exits:
|
||||||
@@ -132,24 +132,24 @@ await resp.text()
|
|||||||
await resp.read() # raw bytes
|
await resp.read() # raw bytes
|
||||||
```
|
```
|
||||||
|
|
||||||
- `params` — query string parameters (dict).
|
- `params` - query string parameters (dict).
|
||||||
- `headers` — request headers (dict).
|
- `headers` - request headers (dict).
|
||||||
- `data` — a `dict` is sent form-urlencoded; `bytes`/`str` is sent as
|
- `data` - a `dict` is sent form-urlencoded; `bytes`/`str` is sent as
|
||||||
the raw body. Matches both libraries' `data=` behaviour.
|
the raw body. Matches both libraries' `data=` behaviour.
|
||||||
- `json` — JSON-encoded body; sets `Content-Type: application/json`.
|
- `json` - JSON-encoded body; sets `Content-Type: application/json`.
|
||||||
- `files` — multipart file upload, `httpx`-style:
|
- `files` - multipart file upload, `httpx`-style:
|
||||||
`files={"upload": ("name.txt", b"content")}`. Combine with `data` for
|
`files={"upload": ("name.txt", b"content")}`. Combine with `data` for
|
||||||
the plain form fields alongside the file(s).
|
the plain form fields alongside the file(s).
|
||||||
- `cookies` — a per-call `dict`, layered on top of whatever the
|
- `cookies` - a per-call `dict`, layered on top of whatever the
|
||||||
context's own cookie jar already holds for that URL, without
|
context's own cookie jar already holds for that URL, without
|
||||||
mutating the jar. Matches `aiohttp`/`httpx`'s per-call `cookies=`.
|
mutating the jar. Matches `aiohttp`/`httpx`'s per-call `cookies=`.
|
||||||
- `auth` — an `(username, password)` tuple, sent as HTTP Basic auth.
|
- `auth` - an `(username, password)` tuple, sent as HTTP Basic auth.
|
||||||
- `allow_redirects` (the `aiohttp` spelling) and `follow_redirects`
|
- `allow_redirects` (the `aiohttp` spelling) and `follow_redirects`
|
||||||
(the `httpx` spelling) are both accepted; either disables following
|
(the `httpx` spelling) are both accepted; either disables following
|
||||||
redirects when passed `False`.
|
redirects when passed `False`.
|
||||||
- `max_redirects` — cap when following redirects (default `20`).
|
- `max_redirects` - cap when following redirects (default `20`).
|
||||||
- `timeout` — seconds; falls back to the instance's `request_timeout`.
|
- `timeout` - seconds; falls back to the instance's `request_timeout`.
|
||||||
- `session` — see "Named sessions", below.
|
- `session` - see "Named sessions", below.
|
||||||
|
|
||||||
```python
|
```python
|
||||||
await stealth.post(url, json={"hello": "world"})
|
await stealth.post(url, json={"hello": "world"})
|
||||||
@@ -161,12 +161,12 @@ await stealth.get(url, auth=("user", "pass"))
|
|||||||
await stealth.get(url, allow_redirects=False)
|
await stealth.get(url, allow_redirects=False)
|
||||||
```
|
```
|
||||||
|
|
||||||
None of these open a tab — they go through the shared context's
|
None of these open a tab - they go through the shared context's
|
||||||
`context.request` fast path, described above.
|
`context.request` fast path, described above.
|
||||||
|
|
||||||
### `download()`
|
### `download()`
|
||||||
|
|
||||||
Raw bytes, for images, PDFs, or any other binary attachment — the same
|
Raw bytes, for images, PDFs, or any other binary attachment - the same
|
||||||
fast, no-tab path as `get()`:
|
fast, no-tab path as `get()`:
|
||||||
|
|
||||||
```python
|
```python
|
||||||
@@ -175,9 +175,9 @@ png_bytes = await stealth.download("https://example.com/photo.png")
|
|||||||
|
|
||||||
## `render()` and `screenshot()`: full page execution
|
## `render()` and `screenshot()`: full page execution
|
||||||
|
|
||||||
For targets that need a JS challenge solved — an SPA that only renders
|
For targets that need a JS challenge solved - an SPA that only renders
|
||||||
results client-side, a Cloudflare-style "verifying your browser"
|
results client-side, a Cloudflare-style "verifying your browser"
|
||||||
interstitial — escalate to a real page load with `render()`:
|
interstitial - escalate to a real page load with `render()`:
|
||||||
|
|
||||||
```python
|
```python
|
||||||
result = await stealth.render(url, collect_links=True)
|
result = await stealth.render(url, collect_links=True)
|
||||||
@@ -194,13 +194,13 @@ path = await stealth.screenshot(url, path="shot.png") # saved to disk, pat
|
|||||||
```
|
```
|
||||||
|
|
||||||
With `path` given, the file is written and `path` is returned. Without
|
With `path` given, the file is written and `path` is returned. Without
|
||||||
it, the PNG bytes come back base64-encoded — convenient for embedding
|
it, the PNG bytes come back base64-encoded - convenient for embedding
|
||||||
directly in a data URI or a JSON API response.
|
directly in a data URI or a JSON API response.
|
||||||
|
|
||||||
## Raw page access: `page()` and `async with stealth as page`
|
## Raw page access: `page()` and `async with stealth as page`
|
||||||
|
|
||||||
For anything the high-level API does not cover, `page()` yields a real
|
For anything the high-level API does not cover, `page()` yields a real
|
||||||
Playwright `Page` — already stealth-patched — and guarantees it (and,
|
Playwright `Page` - already stealth-patched - and guarantees it (and,
|
||||||
for an anonymous call, its context) closes again even if your code
|
for an anonymous call, its context) closes again even if your code
|
||||||
raises:
|
raises:
|
||||||
|
|
||||||
@@ -223,19 +223,19 @@ async with stealth as pg:
|
|||||||
This is exactly `page()` under the hood, with one guarantee on top:
|
This is exactly `page()` under the hood, with one guarantee on top:
|
||||||
it is task-safe. Two coroutines running `async with stealth as pg` at
|
it is task-safe. Two coroutines running `async with stealth as pg` at
|
||||||
the same time on the same shared `Stealthii` instance each get, and
|
the same time on the same shared `Stealthii` instance each get, and
|
||||||
independently release, their own tab — the claim is tracked per
|
independently release, their own tab - the claim is tracked per
|
||||||
`asyncio` task, not on the instance itself, so concurrent use never
|
`asyncio` task, not on the instance itself, so concurrent use never
|
||||||
cross-wires which `pg` belongs to which caller.
|
cross-wires which `pg` belongs to which caller.
|
||||||
|
|
||||||
## Named sessions: cookie persistence and warm-up
|
## Named sessions: cookie persistence and warm-up
|
||||||
|
|
||||||
Every call above is anonymous by default — it uses one shared context
|
Every call above is anonymous by default - it uses one shared context
|
||||||
that is transparently recycled (see below), with no cookie continuity
|
that is transparently recycled (see below), with no cookie continuity
|
||||||
implied between calls. Pass `session="some-name"` to any of
|
implied between calls. Pass `session="some-name"` to any of
|
||||||
`get`/`post`/`put`/`patch`/`delete`/`head`/`download`/`render`/
|
`get`/`post`/`put`/`patch`/`delete`/`head`/`download`/`render`/
|
||||||
`screenshot`/`page` to use a dedicated, reused context under that name
|
`screenshot`/`page` to use a dedicated, reused context under that name
|
||||||
instead. Cookies set under one name persist across every subsequent
|
instead. Cookies set under one name persist across every subsequent
|
||||||
call using that same name, in either direction — a `render()` call can
|
call using that same name, in either direction - a `render()` call can
|
||||||
warm up a session that a later `get()` then reuses, or vice versa,
|
warm up a session that a later `get()` then reuses, or vice versa,
|
||||||
because both operate on the very same underlying browser context:
|
because both operate on the very same underlying browser context:
|
||||||
|
|
||||||
@@ -245,7 +245,7 @@ resp = await stealth.get("https://example.com/api/data", session="acme") # reus
|
|||||||
```
|
```
|
||||||
|
|
||||||
Anonymous calls and named-session calls never share cookies with each
|
Anonymous calls and named-session calls never share cookies with each
|
||||||
other — each name (including the implicit `None` for anonymous calls)
|
other - each name (including the implicit `None` for anonymous calls)
|
||||||
is its own isolated context.
|
is its own isolated context.
|
||||||
|
|
||||||
## Configuration
|
## Configuration
|
||||||
@@ -255,7 +255,7 @@ Everything is set once, at construction:
|
|||||||
```python
|
```python
|
||||||
stealth = Stealthii(
|
stealth = Stealthii(
|
||||||
max_tabs=4, # concurrent render()/page()/screenshot() tabs
|
max_tabs=4, # concurrent render()/page()/screenshot() tabs
|
||||||
fingerprint="ja4", # "ja3" or "ja4" — see fingerprint_info()
|
fingerprint="ja4", # "ja3" or "ja4" - see fingerprint_info()
|
||||||
request_timeout=15.0, # default seconds for get/post/...
|
request_timeout=15.0, # default seconds for get/post/...
|
||||||
max_retries=3, # attempts before raising, for both request() and render()/goto
|
max_retries=3, # attempts before raising, for both request() and render()/goto
|
||||||
disk_cache_bytes=1, # Chromium's own on-disk cache size, kept near-zero
|
disk_cache_bytes=1, # Chromium's own on-disk cache size, kept near-zero
|
||||||
@@ -264,7 +264,7 @@ stealth = Stealthii(
|
|||||||
)
|
)
|
||||||
```
|
```
|
||||||
|
|
||||||
`fingerprint` selects which hash `fingerprint_info()` reports — it does
|
`fingerprint` selects which hash `fingerprint_info()` reports - it does
|
||||||
not change what is sent on the wire; both values describe the same
|
not change what is sent on the wire; both values describe the same
|
||||||
genuine Chromium handshake, just in a different published format:
|
genuine Chromium handshake, just in a different published format:
|
||||||
|
|
||||||
@@ -276,21 +276,21 @@ await stealth.fingerprint_info()
|
|||||||
### Why the recycle settings exist
|
### Why the recycle settings exist
|
||||||
|
|
||||||
A browser that runs for the entire lifetime of a process otherwise
|
A browser that runs for the entire lifetime of a process otherwise
|
||||||
accumulates cookies and an on-disk HTTP cache without bound — a slow,
|
accumulates cookies and an on-disk HTTP cache without bound - a slow,
|
||||||
silent resource leak in anything long-running. `stealthii` closes and
|
silent resource leak in anything long-running. `stealthii` closes and
|
||||||
recreates any context, anonymous or named, once it crosses
|
recreates any context, anonymous or named, once it crosses
|
||||||
`recycle_context_after` (request count or age, whichever comes first),
|
`recycle_context_after` (request count or age, whichever comes first),
|
||||||
and fully relaunches the browser process — discarding its whole profile
|
and fully relaunches the browser process - discarding its whole profile
|
||||||
directory — every `recycle_browser_after_seconds`. A named session that
|
directory - every `recycle_browser_after_seconds`. A named session that
|
||||||
needs to stay warm longer than the default 30 minutes / 500 requests
|
needs to stay warm longer than the default 30 minutes / 500 requests
|
||||||
should get a larger `recycle_context_after` passed at construction.
|
should get a larger `recycle_context_after` passed at construction.
|
||||||
|
|
||||||
## Error handling
|
## Error handling
|
||||||
|
|
||||||
- `StealthUnavailableError` — the browser itself could not be started
|
- `StealthUnavailableError` - the browser itself could not be started
|
||||||
(Playwright not installed, or the Chromium launch failed). Raised
|
(Playwright not installed, or the Chromium launch failed). Raised
|
||||||
from `start()` and anything that needs the browser.
|
from `start()` and anything that needs the browser.
|
||||||
- `StealthRequestError` — a `get`/`post`/`put`/`patch`/`delete`/`head`/
|
- `StealthRequestError` - a `get`/`post`/`put`/`patch`/`delete`/`head`/
|
||||||
`download` call failed outright after `max_retries` attempts, each
|
`download` call failed outright after `max_retries` attempts, each
|
||||||
separated by a short backoff. The retry is recursive and bounded, not
|
separated by a short backoff. The retry is recursive and bounded, not
|
||||||
an unbounded loop.
|
an unbounded loop.
|
||||||
@@ -304,6 +304,28 @@ should get a larger `recycle_context_after` passed at construction.
|
|||||||
`async with stealth.page() as pg:` (or `async with stealth as pg:`)
|
`async with stealth.page() as pg:` (or `async with stealth as pg:`)
|
||||||
raises.
|
raises.
|
||||||
|
|
||||||
|
## One instance, one event loop
|
||||||
|
|
||||||
|
A `Stealthii` instance binds its lock, semaphore, browser process, and
|
||||||
|
every context to whichever `asyncio` event loop is running the first
|
||||||
|
time it is actually used. Using the same instance again from a
|
||||||
|
*different* event loop raises `StealthUnavailableError` immediately
|
||||||
|
(rather than hanging) - the fix is either a fresh `Stealthii()` per loop,
|
||||||
|
or making sure your event loop does not change across the calls that
|
||||||
|
share one instance. This matters most for test suites: `pytest-asyncio`
|
||||||
|
defaults to a new event loop per test function, which breaks a
|
||||||
|
module-level `Stealthii` singleton shared across tests. Set:
|
||||||
|
|
||||||
|
```ini
|
||||||
|
[pytest]
|
||||||
|
asyncio_default_test_loop_scope = session
|
||||||
|
asyncio_default_fixture_loop_scope = session
|
||||||
|
```
|
||||||
|
|
||||||
|
so the whole test session shares one loop - matching how a real
|
||||||
|
application actually runs (one process, one loop, for its entire
|
||||||
|
lifetime), and how `Stealthii` is meant to be used.
|
||||||
|
|
||||||
## Testing
|
## Testing
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
@@ -315,8 +337,8 @@ make test-integration # hits real sites; requires Chromium installed
|
|||||||
pure request-shaping logic without any network access.
|
pure request-shaping logic without any network access.
|
||||||
`tests/test_http_methods.py` and `tests/test_stealth_detection.py` are
|
`tests/test_http_methods.py` and `tests/test_stealth_detection.py` are
|
||||||
integration tests (`@pytest.mark.online`) that exercise every method
|
integration tests (`@pytest.mark.online`) that exercise every method
|
||||||
above — including named-session persistence, per-call cookies, auth,
|
above - including named-session persistence, per-call cookies, auth,
|
||||||
multipart upload, retries against an unreachable host, and the
|
multipart upload, retries against an unreachable host, and the
|
||||||
sannysoft stealth regression suite — against real endpoints.
|
sannysoft stealth regression suite - against real endpoints.
|
||||||
`scripts/smoke_live.py` is a plain manual script covering the same
|
`scripts/smoke_live.py` is a plain manual script covering the same
|
||||||
ground end-to-end, useful for a quick sanity check outside pytest.
|
ground end-to-end, useful for a quick sanity check outside pytest.
|
||||||
|
|||||||
+7
-7
@@ -17,7 +17,7 @@ Applied in `Stealthii._ensure_browser_locked()` at browser launch.
|
|||||||
| `ignore_default_args=['--enable-automation']` | Playwright passes `--enable-automation` to Chromium by default, which enables the automation infobar and related detectable behaviour. Suppressed. |
|
| `ignore_default_args=['--enable-automation']` | Playwright passes `--enable-automation` to Chromium by default, which enables the automation infobar and related detectable behaviour. Suppressed. |
|
||||||
| `--no-sandbox` | Required to run headless Chromium as a non-root, unprivileged process in this deployment environment. Not a stealth measure. |
|
| `--no-sandbox` | Required to run headless Chromium as a non-root, unprivileged process in this deployment environment. Not a stealth measure. |
|
||||||
| `--disable-dev-shm-usage` | Avoids `/dev/shm` exhaustion under containerized/limited-memory environments. Not a stealth measure. |
|
| `--disable-dev-shm-usage` | Avoids `/dev/shm` exhaustion under containerized/limited-memory environments. Not a stealth measure. |
|
||||||
| `--disk-cache-size=<bytes>` / `--media-cache-size=<bytes>` | Bounds Chromium's own on-disk HTTP/media cache, configurable via `disk_cache_bytes` (default near-zero). Not a stealth measure — an operational safeguard against unbounded disk growth in a long-lived process. |
|
| `--disk-cache-size=<bytes>` / `--media-cache-size=<bytes>` | Bounds Chromium's own on-disk HTTP/media cache, configurable via `disk_cache_bytes` (default near-zero). Not a stealth measure - an operational safeguard against unbounded disk growth in a long-lived process. |
|
||||||
|
|
||||||
## JS-level patches
|
## JS-level patches
|
||||||
|
|
||||||
@@ -42,7 +42,7 @@ and `app` members shaped like the real object's public surface.
|
|||||||
|
|
||||||
A stock headless Chromium reports `denied` for a `notifications`
|
A stock headless Chromium reports `denied` for a `notifications`
|
||||||
permission query while `Notification.permission` itself still reports
|
permission query while `Notification.permission` itself still reports
|
||||||
`default` — a mismatch a real browser never produces. The query is
|
`default` - a mismatch a real browser never produces. The query is
|
||||||
patched to always return the same value as `Notification.permission`
|
patched to always return the same value as `Notification.permission`
|
||||||
for that one permission name; all other permission names are passed
|
for that one permission name; all other permission names are passed
|
||||||
through to the original implementation unmodified.
|
through to the original implementation unmodified.
|
||||||
@@ -55,7 +55,7 @@ A stock headless Chromium reports an empty plugin list and a
|
|||||||
set (`PDF Viewer`, `Chrome PDF Viewer`, `Chromium PDF Viewer`,
|
set (`PDF Viewer`, `Chrome PDF Viewer`, `Chromium PDF Viewer`,
|
||||||
`Microsoft Edge PDF Viewer`, `WebKit built-in PDF`) are constructed, each
|
`Microsoft Edge PDF Viewer`, `WebKit built-in PDF`) are constructed, each
|
||||||
individually reassigned the real `Plugin.prototype`, and the containing
|
individually reassigned the real `Plugin.prototype`, and the containing
|
||||||
array reassigned `PluginArray.prototype` — including
|
array reassigned `PluginArray.prototype` - including
|
||||||
`Symbol.toStringTag`, so both a property count check and a
|
`Symbol.toStringTag`, so both a property count check and a
|
||||||
`Object.prototype.toString.call(...)` type check pass. `navigator.mimeTypes`
|
`Object.prototype.toString.call(...)` type check pass. `navigator.mimeTypes`
|
||||||
is patched the same way for the corresponding `application/pdf` MIME type.
|
is patched the same way for the corresponding `application/pdf` MIME type.
|
||||||
@@ -71,7 +71,7 @@ already reports plausible values here in most cases.
|
|||||||
A stock headless Chromium renders WebGL through SwiftShader (a software
|
A stock headless Chromium renders WebGL through SwiftShader (a software
|
||||||
rasterizer), which reports `UNMASKED_VENDOR_WEBGL` (constant `37445`) as
|
rasterizer), which reports `UNMASKED_VENDOR_WEBGL` (constant `37445`) as
|
||||||
`Google Inc. (Google)` and `UNMASKED_RENDERER_WEBGL` (constant `37446`)
|
`Google Inc. (Google)` and `UNMASKED_RENDERER_WEBGL` (constant `37446`)
|
||||||
containing the literal string `SwiftShader` — an unambiguous headless
|
containing the literal string `SwiftShader` - an unambiguous headless
|
||||||
signal checked by essentially every fingerprinting script. Both
|
signal checked by essentially every fingerprinting script. Both
|
||||||
constants are intercepted and rewritten to report a real discrete GPU
|
constants are intercepted and rewritten to report a real discrete GPU
|
||||||
(`Google Inc. (NVIDIA)` / an ANGLE-wrapped `NVIDIA GeForce RTX 3060`
|
(`Google Inc. (NVIDIA)` / an ANGLE-wrapped `NVIDIA GeForce RTX 3060`
|
||||||
@@ -80,7 +80,7 @@ string); every other parameter is passed through unmodified.
|
|||||||
### `HTMLIFrameElement.prototype.contentWindow`
|
### `HTMLIFrameElement.prototype.contentWindow`
|
||||||
|
|
||||||
A known secondary check reads `window.chrome` from inside a freshly
|
A known secondary check reads `window.chrome` from inside a freshly
|
||||||
created, same-origin iframe rather than the top-level document — since
|
created, same-origin iframe rather than the top-level document - since
|
||||||
each document gets its own JS globals, an iframe's `window.chrome` can
|
each document gets its own JS globals, an iframe's `window.chrome` can
|
||||||
be absent even when the top-level one was patched. The `contentWindow`
|
be absent even when the top-level one was patched. The `contentWindow`
|
||||||
getter is wrapped so that, whenever the returned window is missing
|
getter is wrapped so that, whenever the returned window is missing
|
||||||
@@ -94,7 +94,7 @@ Every function or getter installed by the patches above is wrapped in a
|
|||||||
real native function produces (`function name() { [native code] }`).
|
real native function produces (`function name() { [native code] }`).
|
||||||
Without this, a page can distinguish a patched native method from a real
|
Without this, a page can distinguish a patched native method from a real
|
||||||
one by calling `.toString()` on it and observing actual JavaScript source
|
one by calling `.toString()` on it and observing actual JavaScript source
|
||||||
instead of the native-code marker — a check used by several stealth-
|
instead of the native-code marker - a check used by several stealth-
|
||||||
detection scripts specifically to unmask other stealth patches.
|
detection scripts specifically to unmask other stealth patches.
|
||||||
|
|
||||||
## Verification
|
## Verification
|
||||||
@@ -115,7 +115,7 @@ the browser's own TLS stack rather than Python's.
|
|||||||
|
|
||||||
## Known limitations
|
## Known limitations
|
||||||
|
|
||||||
- The `context.request` fast path negotiates HTTP/1.1, not HTTP/2 — a
|
- The `context.request` fast path negotiates HTTP/1.1, not HTTP/2 - a
|
||||||
real Chrome page load always uses HTTP/2 against an HTTP/2-capable
|
real Chrome page load always uses HTTP/2 against an HTTP/2-capable
|
||||||
server. A system that cross-checks the TLS fingerprint against the
|
server. A system that cross-checks the TLS fingerprint against the
|
||||||
negotiated protocol can detect this specific inconsistency. `render()`
|
negotiated protocol can detect this specific inconsistency. `render()`
|
||||||
|
|||||||
@@ -92,10 +92,23 @@ class Stealthii:
|
|||||||
self._user_agent = None
|
self._user_agent = None
|
||||||
self._tab_semaphore = None
|
self._tab_semaphore = None
|
||||||
self._lock: Optional[asyncio.Lock] = None
|
self._lock: Optional[asyncio.Lock] = None
|
||||||
|
self._loop = None
|
||||||
self._attempted = False
|
self._attempted = False
|
||||||
self._claimed_pages: Dict[Any, Any] = {}
|
self._claimed_pages: Dict[Any, Any] = {}
|
||||||
|
|
||||||
def _get_lock(self) -> asyncio.Lock:
|
def _get_lock(self) -> asyncio.Lock:
|
||||||
|
loop = asyncio.get_running_loop()
|
||||||
|
if self._loop is None:
|
||||||
|
self._loop = loop
|
||||||
|
elif self._loop is not loop:
|
||||||
|
raise StealthUnavailableError(
|
||||||
|
'this Stealthii instance was first used on a different asyncio '
|
||||||
|
'event loop and cannot be reused on a new one (its browser '
|
||||||
|
'process, lock, and contexts are all bound to that first loop). '
|
||||||
|
'Construct a fresh Stealthii() per event loop, or make sure your '
|
||||||
|
'test runner uses one event loop for the whole session (e.g. '
|
||||||
|
'pytest-asyncio: asyncio_default_test_loop_scope = "session").'
|
||||||
|
)
|
||||||
if self._lock is None:
|
if self._lock is None:
|
||||||
self._lock = asyncio.Lock()
|
self._lock = asyncio.Lock()
|
||||||
return self._lock
|
return self._lock
|
||||||
|
|||||||
+10
-1
@@ -1,7 +1,7 @@
|
|||||||
# retoor <retoor@molodetz.nl>
|
# retoor <retoor@molodetz.nl>
|
||||||
import pytest
|
import pytest
|
||||||
|
|
||||||
from stealthii import Stealthii, StealthResponse
|
from stealthii import Stealthii, StealthResponse, StealthUnavailableError
|
||||||
from stealthii.client import _ContextSlot
|
from stealthii.client import _ContextSlot
|
||||||
|
|
||||||
|
|
||||||
@@ -69,6 +69,15 @@ async def test_stealth_response_shapes_like_aiohttp():
|
|||||||
assert r is resp
|
assert r is resp
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_get_lock_raises_clearly_on_event_loop_mismatch():
|
||||||
|
s = Stealthii()
|
||||||
|
s._get_lock()
|
||||||
|
s._loop = object()
|
||||||
|
with pytest.raises(StealthUnavailableError, match='different asyncio event loop'):
|
||||||
|
s._get_lock()
|
||||||
|
|
||||||
|
|
||||||
def test_context_slot_starts_fresh():
|
def test_context_slot_starts_fresh():
|
||||||
slot = _ContextSlot(context=object())
|
slot = _ContextSlot(context=object())
|
||||||
assert slot.count == 0
|
assert slot.count == 0
|
||||||
|
|||||||
Reference in New Issue
Block a user