Waw
This commit is contained in:
@@ -12,7 +12,9 @@ PRAGMA cache_size=-8000; -- 8MB page cache
|
||||
PRAGMA temp_store=MEMORY; -- temp tables in memory
|
||||
```
|
||||
|
||||
Configured via `dataset.connect(engine_kwargs={"connect_args": {"timeout": 30, "check_same_thread": False}}, on_connect_statements=[...])`. In addition to the pragmas above, the connection is tuned with a 30s busy timeout, an 8MB page cache, and a 256MB mmap.
|
||||
Configured via `dataset.connect(engine_kwargs={"connect_args": {"timeout": 30, "check_same_thread": False}, "poolclass": NullPool}, on_connect_statements=[...])`. In addition to the pragmas above, the connection is tuned with a 30s busy timeout, an 8MB page cache, and a 256MB mmap.
|
||||
|
||||
**`poolclass=NullPool` is load-bearing - never revert it to SQLAlchemy's default `QueuePool` (caused a production outage).** `dataset.Database.executable` caches ONE DBAPI connection per OS thread ID **forever** and never returns it to the pool except via `db.close()`, which nothing in this codebase calls (`dataset/database.py`: `self.connections[tid] = self.engine.connect()`). That is fine as long as the same handful of threads ever touch the DB - but FastAPI runs every sync route dependency (`get_setting` and friends, hit on nearly every request) through `anyio.to_thread.run_sync`, whose worker pool scales up and recycles threads elastically under load, and container sync (`asyncio.to_thread`) adds more. Each new thread's first query permanently claims one pool slot. With the default bounded `QueuePool` (`pool_size=5, max_overflow=10` = 15 total), a burst of concurrent load creates enough new threads that the pool fills for good within minutes, and every request thereafter - including the Docker healthcheck's own probe - blocks the full 30s pool timeout and then raises `sqlalchemy.exc.TimeoutError: QueuePool limit of size 5 overflow 10 reached`, wedging the whole app (nginx waits on an app that is waiting on itself; every external caller sees a bare connection timeout, not an HTTP error). `NullPool` removes the artificial ceiling: each `engine.connect()` opens a real, unpooled SQLite connection, so the existing "one connection cached per thread forever" behavior just works, exactly as WAL mode is designed to support. Never pass `pool_size`/`max_overflow` alongside `NullPool` (SQLAlchemy rejects them). Do not "fix" the underlying thread churn instead - that means touching the sync-dependency/threadpool model, which the hard rule below forbids.
|
||||
|
||||
`init_db()` is idempotent - every index is created via `CREATE INDEX IF NOT EXISTS` (through the `_index()` helper, wrapped in try/except so it is safe to run on every startup regardless of table state), and seed defaults for `site_settings` only insert when the key doesn't already exist.
|
||||
|
||||
|
||||
@@ -4,6 +4,7 @@ import dataset
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from sqlalchemy import or_
|
||||
from sqlalchemy.pool import NullPool
|
||||
from collections import defaultdict
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from devplacepy.cache import TTLCache
|
||||
@@ -30,6 +31,7 @@ db = dataset.connect(
|
||||
"timeout": 30,
|
||||
"check_same_thread": False,
|
||||
},
|
||||
"poolclass": NullPool,
|
||||
},
|
||||
on_connect_statements=[
|
||||
"PRAGMA journal_mode=WAL",
|
||||
|
||||
Reference in New Issue
Block a user