This commit is contained in:
2026-08-13 12:59:53 +02:00
parent 45ad8e79ed
commit b97b5a7854
5 changed files with 42 additions and 32 deletions
+3 -1
View File
@@ -12,7 +12,9 @@ PRAGMA cache_size=-8000; -- 8MB page cache
PRAGMA temp_store=MEMORY; -- temp tables in memory
```
Configured via `dataset.connect(engine_kwargs={"connect_args": {"timeout": 30, "check_same_thread": False}}, on_connect_statements=[...])`. In addition to the pragmas above, the connection is tuned with a 30s busy timeout, an 8MB page cache, and a 256MB mmap.
Configured via `dataset.connect(engine_kwargs={"connect_args": {"timeout": 30, "check_same_thread": False}, "poolclass": NullPool}, on_connect_statements=[...])`. In addition to the pragmas above, the connection is tuned with a 30s busy timeout, an 8MB page cache, and a 256MB mmap.
**`poolclass=NullPool` is load-bearing - never revert it to SQLAlchemy's default `QueuePool` (caused a production outage).** `dataset.Database.executable` caches ONE DBAPI connection per OS thread ID **forever** and never returns it to the pool except via `db.close()`, which nothing in this codebase calls (`dataset/database.py`: `self.connections[tid] = self.engine.connect()`). That is fine as long as the same handful of threads ever touch the DB - but FastAPI runs every sync route dependency (`get_setting` and friends, hit on nearly every request) through `anyio.to_thread.run_sync`, whose worker pool scales up and recycles threads elastically under load, and container sync (`asyncio.to_thread`) adds more. Each new thread's first query permanently claims one pool slot. With the default bounded `QueuePool` (`pool_size=5, max_overflow=10` = 15 total), a burst of concurrent load creates enough new threads that the pool fills for good within minutes, and every request thereafter - including the Docker healthcheck's own probe - blocks the full 30s pool timeout and then raises `sqlalchemy.exc.TimeoutError: QueuePool limit of size 5 overflow 10 reached`, wedging the whole app (nginx waits on an app that is waiting on itself; every external caller sees a bare connection timeout, not an HTTP error). `NullPool` removes the artificial ceiling: each `engine.connect()` opens a real, unpooled SQLite connection, so the existing "one connection cached per thread forever" behavior just works, exactly as WAL mode is designed to support. Never pass `pool_size`/`max_overflow` alongside `NullPool` (SQLAlchemy rejects them). Do not "fix" the underlying thread churn instead - that means touching the sync-dependency/threadpool model, which the hard rule below forbids.
`init_db()` is idempotent - every index is created via `CREATE INDEX IF NOT EXISTS` (through the `_index()` helper, wrapped in try/except so it is safe to run on every startup regardless of table state), and seed defaults for `site_settings` only insert when the key doesn't already exist.
+2
View File
@@ -4,6 +4,7 @@ import dataset
import logging
from pathlib import Path
from sqlalchemy import or_
from sqlalchemy.pool import NullPool
from collections import defaultdict
from datetime import datetime, timedelta, timezone
from devplacepy.cache import TTLCache
@@ -30,6 +31,7 @@ db = dataset.connect(
"timeout": 30,
"check_same_thread": False,
},
"poolclass": NullPool,
},
on_connect_statements=[
"PRAGMA journal_mode=WAL",