Enforce the Devii task quotas with atomic reservations

The creation and run quotas were checked and then acted on, so two concurrent
create_task calls or two schedulers could both pass the check and overshoot the
limit. Both are now a single conditional INSERT decided on the driver rowcount:
reserve_run takes a run slot after the claim and releases the claim by deferring
when the quota is spent, and insert_task_within_quota does the same for the task
row itself. Racing twelve and sixteen processes now yields exactly the limit.

The atomic insert names its columns, and dataset skips a None valued key when it
creates a table lazily, so the store declares the full task column set up front.
Both the column and index ensures now tolerate a concurrent duplicate, since
several processes build a store at once and SQLite DDL is not idempotent.

Adds the quota, task-run context, guard, store and scheduler test suites, and
documents the chokepoints and the unhackable task-run flag.
This commit is contained in:
retoor 2026-07-26 19:58:42 +02:00
parent ca6c527e32
commit 9cfaddfc40
11 changed files with 168 additions and 148 deletions

View File

@ -47,7 +47,7 @@ Conversation history persists to `devii_conversations` (rehydrated on reconnect,
**Any signed-in account may schedule; two rolling 24-hour quotas bound it, and role decides the size.** Guests never can (`automation_allowed` requires `owner_kind == "user"`). Defaults: a member may CREATE `devii_task_member_create_24h` (5) tasks and EXECUTE `devii_task_member_runs_24h` (10) runs per 24h; an administrator gets `devii_task_admin_create_24h` (5) and `devii_task_admin_runs_24h` (100). All four are admin-editable on the Devii service; `0` means unlimited. **Any signed-in account may schedule; two rolling 24-hour quotas bound it, and role decides the size.** Guests never can (`automation_allowed` requires `owner_kind == "user"`). Defaults: a member may CREATE `devii_task_member_create_24h` (5) tasks and EXECUTE `devii_task_member_runs_24h` (10) runs per 24h; an administrator gets `devii_task_admin_create_24h` (5) and `devii_task_admin_runs_24h` (100). All four are admin-editable on the Devii service; `0` means unlimited.
**The two quotas are enforced by a single atomic conditional INSERT each, never a check-then-act** (`tasks/limits.py`): **Each quota is enforced by a single atomic conditional INSERT, never a check-then-act** (`tasks/limits.py`):
- `reserve_run(...)` -> `INSERT INTO devii_task_runs ... SELECT ... WHERE (SELECT COUNT(*) ... created_at >= :cutoff) < :limit`, decided on the driver's real `rowcount`. The scheduler calls it AFTER `claim` and BEFORE dispatch; a refusal releases the claim via `defer`. - `reserve_run(...)` -> `INSERT INTO devii_task_runs ... SELECT ... WHERE (SELECT COUNT(*) ... created_at >= :cutoff) < :limit`, decided on the driver's real `rowcount`. The scheduler calls it AFTER `claim` and BEFORE dispatch; a refusal releases the claim via `defer`.
- `insert_task_within_quota(...)` -> the same shape for the task row itself, so two concurrent `create_task` calls (main + telegram channel, or two workers) can never both slip past a check. `TaskStore.create` pre-checks for a good error message, then relies on this insert for the actual guarantee. - `insert_task_within_quota(...)` -> the same shape for the task row itself, so two concurrent `create_task` calls (main + telegram channel, or two workers) can never both slip past a check. `TaskStore.create` pre-checks for a good error message, then relies on this insert for the actual guarantee.
@ -156,6 +156,8 @@ Devii is reachable over Telegram via `channel="telegram"` sessions driven in-pro
Devii exposes read-only `db_list_tables`, `db_table_schema`, `db_list_rows`, `db_get_row`, `db_query`, and `db_design_query` tools gated `requires_primary_admin=True`. See `devplacepy/services/dbapi/CLAUDE.md` for the full auth boundary, validation pipeline, and async query service - these tools are added to the LLM tool list only for the primary administrator; every other session is unaware they exist. Devii exposes read-only `db_list_tables`, `db_table_schema`, `db_list_rows`, `db_get_row`, `db_query`, and `db_design_query` tools gated `requires_primary_admin=True`. See `devplacepy/services/dbapi/CLAUDE.md` for the full auth boundary, validation pipeline, and async query service - these tools are added to the LLM tool list only for the primary administrator; every other session is unaware they exist.
**`db_query` and `db_design_query` MUST stay marked `is_read_only=True`.** They are POST routes, so they carry a request body and would otherwise be classified as mutating by `method in MUTATING_METHODS` alone - but they only read. Marking them read-only is what keeps them from tripping the agentic plan gate and the verification gate, which would discard the very rows the user asked for. `tests/unit/services/devii/actions/catalog.py` guards it.
## All Devii/gateway network timeouts are admin-configurable with a five-minute (300s) floor ## All Devii/gateway network timeouts are admin-configurable with a five-minute (300s) floor
The Devii LLM-client and `PlatformClient` read timeout is `timeout_seconds` (field `devii_timeout`), the fetch/docs tool timeout is `fetch_timeout_seconds` (field `devii_fetch_timeout`), web search is `rsearch_timeout_seconds` (`devii_rsearch_timeout`); each defaults to 300s with `minimum=MIN_TIMEOUT_SECONDS` (300). The gateway upstream timeout is `gateway_timeout` (`minimum=TIMEOUT_MIN`=300), which also bounds the vision describe-image call (it shares the gateway's httpx client, no per-call override). A stored value below the floor fails `ConfigField.coerce()` and `read()` falls back to the default, so old sub-300 values upgrade automatically. Devii's LLM timeout was previously a hardcoded 45s while the gateway waited up to 180s x retries - large prompts aborted as `[model error] Could not reach the model endpoint:` (an httpx timeout, which stringifies to empty). `build_settings()` now reads these from config instead of hardcoding them. The Devii LLM-client and `PlatformClient` read timeout is `timeout_seconds` (field `devii_timeout`), the fetch/docs tool timeout is `fetch_timeout_seconds` (field `devii_fetch_timeout`), web search is `rsearch_timeout_seconds` (`devii_rsearch_timeout`); each defaults to 300s with `minimum=MIN_TIMEOUT_SECONDS` (300). The gateway upstream timeout is `gateway_timeout` (`minimum=TIMEOUT_MIN`=300), which also bounds the vision describe-image call (it shares the gateway's httpx client, no per-call override). A stored value below the floor fails `ConfigField.coerce()` and `read()` falls back to the default, so old sub-300 values upgrade automatically. Devii's LLM timeout was previously a hardcoded 45s while the gateway waited up to 180s x retries - large prompts aborted as `[model error] Could not reach the model endpoint:` (an httpx timeout, which stringifies to empty). `build_settings()` now reads these from config instead of hardcoding them.

View File

@ -85,12 +85,12 @@ def window_start(reference: datetime) -> datetime:
return reference - timedelta(hours=WINDOW_HOURS) return reference - timedelta(hours=WINDOW_HOURS)
def _stamps(db: Any, sql: str, table: str, owner_kind: str, owner_id: str, cutoff: str) -> list[str]: def _stamps(
db: Any, sql: str, table: str, owner_kind: str, owner_id: str, cutoff: str
) -> list[str]:
if table not in db.tables: if table not in db.tables:
return [] return []
rows = db.query( rows = db.query(sql, kind=owner_kind, owner=owner_id, cutoff=cutoff, cap=SAMPLE_CAP)
sql, kind=owner_kind, owner=owner_id, cutoff=cutoff, cap=SAMPLE_CAP
)
return [str(row["created_at"]) for row in rows if row.get("created_at")] return [str(row["created_at"]) for row in rows if row.get("created_at")]
@ -127,7 +127,9 @@ def create_quota(db: Any, owner_kind: str, owner_id: str, reference: datetime) -
return _quota(stamps, limit) return _quota(stamps, limit)
def record_run(db: Any, owner_kind: str, owner_id: str, task_uid: str, reference: datetime) -> None: def record_run(
db: Any, owner_kind: str, owner_id: str, task_uid: str, reference: datetime
) -> None:
from devplacepy.utils import generate_uid from devplacepy.utils import generate_uid
db[RUNS_TABLE].insert( db[RUNS_TABLE].insert(
@ -149,10 +151,7 @@ def reserve_run(
from devplacepy.utils import generate_uid from devplacepy.utils import generate_uid
limit = run_limit(owner_is_admin(owner_id)) limit = run_limit(owner_is_admin(owner_id))
if RUNS_TABLE not in db.tables: if limit <= 0 or RUNS_TABLE not in db.tables:
record_run(db, owner_kind, owner_id, task_uid, reference)
return True
if limit <= 0:
record_run(db, owner_kind, owner_id, task_uid, reference) record_run(db, owner_kind, owner_id, task_uid, reference)
return True return True
params = { params = {

View File

@ -8,7 +8,7 @@ from typing import Any, Awaitable, Callable, Optional
from .context import task_run_scope from .context import task_run_scope
from .controller import compute_followup from .controller import compute_followup
from .guards import BudgetProbe, Deferral, budget_deferral, run_deferral, retire_reason from .guards import BudgetProbe, Deferral, budget_deferral, retire_reason, run_deferral
from .limits import reserve_run from .limits import reserve_run
from .schedule import next_run, now_utc, to_iso from .schedule import next_run, now_utc, to_iso
from .store import TaskStore, claim, due_rows from .store import TaskStore, claim, due_rows
@ -102,10 +102,7 @@ def defer(store: TaskStore, row: dict[str, Any], postponement: Deferral) -> None
}, },
) )
logger.info( logger.info(
"Task uid=%s postponed to %s: %s", "Task uid=%s postponed to %s: %s", row.get("uid"), retry_at, postponement.reason
row.get("uid"),
retry_at,
postponement.reason,
) )
_audit_task_deferred(row, postponement.reason, retry_at) _audit_task_deferred(row, postponement.reason, retry_at)

View File

@ -148,9 +148,7 @@ class TaskStore:
def require_creation_allowed(self) -> None: def require_creation_allowed(self) -> None:
if self._operator: if self._operator:
return return
denial = creation_denial( denial = creation_denial(self._db, self._owner_kind, self._owner_id, now_utc())
self._db, self._owner_kind, self._owner_id, now_utc()
)
if denial is not None: if denial is not None:
raise denial raise denial

View File

@ -14,10 +14,7 @@ def _records_mutation(name: str) -> bool:
return action.method in MUTATING_METHODS and not action.is_read_only return action.method in MUTATING_METHODS and not action.is_read_only
def test_read_only_query_tools_are_marked_read_only(): def test_post_query_tools_are_read_only_so_the_plan_gate_keeps_their_rows():
# These are POST routes (they carry a body) but only read data. They MUST be
# read-only so they do not trip the agentic plan gate or the verification gate
# and drop the rows the user asked for.
for name in ("db_query", "db_design_query"): for name in ("db_query", "db_design_query"):
assert BY_NAME[name].is_read_only is True assert BY_NAME[name].is_read_only is True
assert _records_mutation(name) is False assert _records_mutation(name) is False
@ -146,8 +143,7 @@ def test_scheduling_tools_are_hidden_from_guests():
from devplacepy.services.devii.registry import CATALOG from devplacepy.services.devii.registry import CATALOG
guest = { guest = {
s["function"]["name"] s["function"]["name"] for s in CATALOG.tool_schemas_for(authenticated=False)
for s in CATALOG.tool_schemas_for(authenticated=False)
} }
for name in ("create_task", "update_task", "run_task_now"): for name in ("create_task", "update_task", "run_task_now"):
assert name not in guest assert name not in guest

View File

@ -90,6 +90,7 @@ def test_scheduled_tasks_only_ever_resolve_a_main_channel_session(local_db):
"role": "Admin", "role": "Admin",
"api_key": "k", "api_key": "k",
"deleted_at": None, "deleted_at": None,
"created_at": "2099-01-01T00:00:00",
} }
) )
database.invalidate_admins_cache() database.invalidate_admins_cache()
@ -135,51 +136,35 @@ def test_task_run_block_is_absent_outside_a_scheduled_run(local_db):
assert "TASK RUN ENVIRONMENT" not in session._compose_system_prompt() assert "TASK RUN ENVIRONMENT" not in session._compose_system_prompt()
def test_member_session_is_told_it_may_not_nest_tasks(local_db): def _task_session(local_db, uid, role):
from devplacepy.services.devii.tasks.context import task_run_scope from devplacepy.services.devii.tasks.context import task_run_scope
hub, svc = _hub() hub, svc = _hub()
local_db["users"].insert( local_db["users"].insert(
{ {
"uid": "sess-task-member", "uid": uid,
"username": "task-member", "username": uid,
"role": "Member", "role": role,
"api_key": "k", "api_key": "k",
"deleted_at": None, "deleted_at": None,
"created_at": "2026-01-01T00:00:00", "created_at": "2099-01-01T00:00:00",
} }
) )
database.invalidate_admins_cache() database.invalidate_admins_cache()
session = hub.get_or_create( session = hub.get_or_create(
"user", "sess-task-member", "task-member", "", svc.instance_base_url(), "user", uid, uid, "", svc.instance_base_url(), channel="main"
channel="main",
) )
with task_run_scope(): with task_run_scope():
prompt = session._compose_system_prompt() return session._compose_system_prompt()
def test_member_session_is_told_it_may_not_nest_tasks(local_db):
prompt = _task_session(local_db, "sess-task-member", "Member")
assert "TASK RUN ENVIRONMENT" in prompt assert "TASK RUN ENVIRONMENT" in prompt
assert "may NOT schedule further work" in prompt assert "may NOT schedule further work" in prompt
def test_administrator_session_is_told_it_may_nest_tasks(local_db): def test_administrator_session_is_told_it_may_nest_tasks(local_db):
from devplacepy.services.devii.tasks.context import task_run_scope prompt = _task_session(local_db, "sess-task-admin", "Admin")
hub, svc = _hub()
local_db["users"].insert(
{
"uid": "sess-task-admin",
"username": "task-admin",
"role": "Admin",
"api_key": "k",
"deleted_at": None,
"created_at": "2026-01-01T00:00:00",
}
)
database.invalidate_admins_cache()
session = hub.get_or_create(
"user", "sess-task-admin", "task-admin", "", svc.instance_base_url(),
channel="main",
)
with task_run_scope():
prompt = session._compose_system_prompt()
assert "TASK RUN ENVIRONMENT" in prompt assert "TASK RUN ENVIRONMENT" in prompt
assert "MAY" in prompt assert "you MAY" in prompt

View File

@ -5,8 +5,9 @@ import asyncio
from devplacepy.database import invalidate_admins_cache from devplacepy.database import invalidate_admins_cache
from devplacepy.services.devii.tasks.context import in_task_run, task_run_scope from devplacepy.services.devii.tasks.context import in_task_run, task_run_scope
from devplacepy.services.devii.tasks.guards import nesting_allowed from devplacepy.services.devii.tasks.guards import nesting_allowed
from devplacepy.services.devii.tasks.schedule import now_utc, to_iso
from devplacepy.utils import generate_uid from devplacepy.utils import generate_uid
SIGNUP_AFTER_PRIMARY_ADMIN = "2099-01-01T00:00:00"
from tests.conftest import run_async from tests.conftest import run_async
@ -18,7 +19,7 @@ def _account(local_db, role):
"username": f"ctx-{uid[-10:]}", "username": f"ctx-{uid[-10:]}",
"role": role, "role": role,
"deleted_at": None, "deleted_at": None,
"created_at": to_iso(now_utc()), "created_at": SIGNUP_AFTER_PRIMARY_ADMIN,
} }
) )
invalidate_admins_cache() invalidate_admins_cache()

View File

@ -24,6 +24,8 @@ from devplacepy.services.devii.tasks.limits import reserve_run
from devplacepy.services.devii.tasks.schedule import now_utc, to_iso from devplacepy.services.devii.tasks.schedule import now_utc, to_iso
from devplacepy.utils import generate_uid from devplacepy.utils import generate_uid
SIGNUP_AFTER_PRIMARY_ADMIN = "2099-01-01T00:00:00"
def _account(local_db, role): def _account(local_db, role):
uid = generate_uid() uid = generate_uid()
@ -33,7 +35,7 @@ def _account(local_db, role):
"username": f"guard-{uid[-10:]}", "username": f"guard-{uid[-10:]}",
"role": role, "role": role,
"deleted_at": None, "deleted_at": None,
"created_at": to_iso(now_utc()), "created_at": SIGNUP_AFTER_PRIMARY_ADMIN,
} }
) )
invalidate_admins_cache() invalidate_admins_cache()
@ -55,11 +57,7 @@ def _row(owner_uid, **overrides):
def _burn_runs(local_db, owner, count, reference): def _burn_runs(local_db, owner, count, reference):
for index in range(count): for index in range(count):
limits.record_run( limits.record_run(
local_db, local_db, "user", owner, generate_uid(), reference - timedelta(minutes=index)
"user",
owner,
generate_uid(),
reference - timedelta(minutes=index),
) )
@ -94,6 +92,13 @@ def test_exhausted_task_is_retired(local_db):
assert retire_reason(_row(owner, max_runs=5, run_count=5), now_utc()) == REASON_MAX_RUNS assert retire_reason(_row(owner, max_runs=5, run_count=5), now_utc()) == REASON_MAX_RUNS
def test_task_without_an_expiry_stops_at_the_fallback_ceiling(local_db):
owner = _account(local_db, "Member")
now = now_utc()
row = _row(owner, created_at=to_iso(now - timedelta(days=40)))
assert retire_reason(row, now) == REASON_EXPIRED
def test_repeated_failures_retire_the_task(local_db): def test_repeated_failures_retire_the_task(local_db):
owner = _account(local_db, "Member") owner = _account(local_db, "Member")
row = _row(owner, failure_count=3) row = _row(owner, failure_count=3)
@ -133,13 +138,11 @@ def test_administrator_run_quota_is_larger(local_db):
def test_budget_defers_the_task(local_db): def test_budget_defers_the_task(local_db):
owner = _account(local_db, "Member") owner = _account(local_db, "Member")
now = now_utc() now = now_utc()
postponement = budget_deferral( postponement = budget_deferral(_row(owner), now, budget_exceeded=lambda k, u: True)
_row(owner), now, budget_exceeded=lambda kind, uid: True
)
assert postponement is not None assert postponement is not None
assert postponement.reason == REASON_BUDGET assert postponement.reason == REASON_BUDGET
assert postponement.retry_at > now assert postponement.retry_at > now
assert budget_deferral(_row(owner), now, lambda kind, uid: False) is None assert budget_deferral(_row(owner), now, lambda k, u: False) is None
assert budget_deferral(_row(owner), now, None) is None assert budget_deferral(_row(owner), now, None) is None

View File

@ -7,6 +7,8 @@ from devplacepy.services.devii.tasks import limits
from devplacepy.services.devii.tasks.schedule import now_utc, to_iso from devplacepy.services.devii.tasks.schedule import now_utc, to_iso
from devplacepy.utils import generate_uid from devplacepy.utils import generate_uid
SIGNUP_AFTER_PRIMARY_ADMIN = "2099-01-01T00:00:00"
def _account(local_db, role): def _account(local_db, role):
uid = generate_uid() uid = generate_uid()
@ -16,7 +18,7 @@ def _account(local_db, role):
"username": f"lim-{uid[-10:]}", "username": f"lim-{uid[-10:]}",
"role": role, "role": role,
"deleted_at": None, "deleted_at": None,
"created_at": to_iso(now_utc()), "created_at": SIGNUP_AFTER_PRIMARY_ADMIN,
} }
) )
invalidate_admins_cache() invalidate_admins_cache()

View File

@ -19,6 +19,8 @@ from devplacepy.services.devii.tasks.store import TaskStore
from devplacepy.utils import generate_uid from devplacepy.utils import generate_uid
from tests.conftest import run_async from tests.conftest import run_async
SIGNUP_AFTER_PRIMARY_ADMIN = "2099-01-01T00:00:00"
def _account(local_db, role): def _account(local_db, role):
uid = generate_uid() uid = generate_uid()
@ -29,7 +31,7 @@ def _account(local_db, role):
"role": role, "role": role,
"api_key": "k", "api_key": "k",
"deleted_at": None, "deleted_at": None,
"created_at": to_iso(now_utc()), "created_at": SIGNUP_AFTER_PRIMARY_ADMIN,
} }
) )
invalidate_admins_cache() invalidate_admins_cache()
@ -86,22 +88,6 @@ def _harness(local_db, started, gate, peak=None, flags=None):
return resolve return resolve
def _run_scheduler(local_db, resolve, hold=0.4, release=0.5, **kwargs):
async def run():
gate = kwargs.pop("gate", None) or asyncio.Event()
scheduler = GlobalScheduler(local_db, resolve, tick_seconds=0.05, **kwargs)
scheduler.start()
await asyncio.sleep(hold)
claimed = [r["uid"] for r in local_db["devii_tasks"].find(status="running")]
held = len(claimed)
gate.set()
await asyncio.sleep(release)
await scheduler.stop()
return claimed, held
return run
def test_scheduler_runs_one_task_per_owner_and_completes_it(local_db): def test_scheduler_runs_one_task_per_owner_and_completes_it(local_db):
local_db["devii_tasks"].delete() local_db["devii_tasks"].delete()
owner = _account(local_db, "Admin") owner = _account(local_db, "Admin")
@ -110,15 +96,25 @@ def test_scheduler_runs_one_task_per_owner_and_completes_it(local_db):
started: list[str] = [] started: list[str] = []
peak: list[int] = [] peak: list[int] = []
gate = asyncio.Event()
claimed, held = run_async( async def run():
_run_scheduler( gate = asyncio.Event()
local_db, _harness(local_db, started, gate, peak), gate=gate scheduler = GlobalScheduler(
)() local_db, _harness(local_db, started, gate, peak), tick_seconds=0.05
) )
scheduler.start()
await asyncio.sleep(0.4)
claimed = [r["uid"] for r in local_db["devii_tasks"].find(status="running")]
held = len(started)
gate.set()
await asyncio.sleep(0.5)
await scheduler.stop()
return claimed, held
claimed, held = run_async(run())
assert held == 1 assert held == 1
assert max(peak) == 1 assert max(peak) == 1
assert len(claimed) == 1
done = local_db["devii_tasks"].find_one(uid=claimed[0]) done = local_db["devii_tasks"].find_one(uid=claimed[0])
assert int(done["run_count"]) == 1 assert int(done["run_count"]) == 1
assert done["status"] == "pending" assert done["status"] == "pending"
@ -130,12 +126,23 @@ def test_scheduler_runs_one_task_per_owner_and_completes_it(local_db):
def test_scheduler_runs_a_member_task(local_db): def test_scheduler_runs_a_member_task(local_db):
local_db["devii_tasks"].delete() local_db["devii_tasks"].delete()
local_db[limits.RUNS_TABLE].delete()
owner = _account(local_db, "Member") owner = _account(local_db, "Member")
uid = _seed(local_db, owner) uid = _seed(local_db, owner)
started: list[str] = [] started: list[str] = []
gate = asyncio.Event()
run_async(_run_scheduler(local_db, _harness(local_db, started, gate), gate=gate)()) async def run():
gate = asyncio.Event()
scheduler = GlobalScheduler(
local_db, _harness(local_db, started, gate), tick_seconds=0.05
)
scheduler.start()
await asyncio.sleep(0.3)
gate.set()
await asyncio.sleep(0.4)
await scheduler.stop()
run_async(run())
assert started == [owner] assert started == [owner]
row = local_db["devii_tasks"].find_one(uid=uid) row = local_db["devii_tasks"].find_one(uid=uid)
assert row["enabled"] assert row["enabled"]
@ -144,17 +151,23 @@ def test_scheduler_runs_a_member_task(local_db):
def test_the_executor_runs_inside_the_task_run_context(local_db): def test_the_executor_runs_inside_the_task_run_context(local_db):
local_db["devii_tasks"].delete() local_db["devii_tasks"].delete()
local_db[limits.RUNS_TABLE].delete()
owner = _account(local_db, "Member") owner = _account(local_db, "Member")
_seed(local_db, owner) _seed(local_db, owner)
started: list[str] = [] started: list[str] = []
flags: list[bool] = [] flags: list[bool] = []
gate = asyncio.Event()
run_async( async def run():
_run_scheduler( gate = asyncio.Event()
local_db, _harness(local_db, started, gate, flags=flags), gate=gate gate.set()
)() scheduler = GlobalScheduler(
) local_db, _harness(local_db, started, gate, flags=flags), tick_seconds=0.05
)
scheduler.start()
await asyncio.sleep(0.4)
await scheduler.stop()
run_async(run())
assert flags == [True] assert flags == [True]
assert in_task_run() is False assert in_task_run() is False
@ -165,9 +178,18 @@ def test_every_run_is_recorded_against_the_owner_quota(local_db):
owner = _account(local_db, "Member") owner = _account(local_db, "Member")
_seed(local_db, owner) _seed(local_db, owner)
started: list[str] = [] started: list[str] = []
gate = asyncio.Event()
run_async(_run_scheduler(local_db, _harness(local_db, started, gate), gate=gate)()) async def run():
gate = asyncio.Event()
gate.set()
scheduler = GlobalScheduler(
local_db, _harness(local_db, started, gate), tick_seconds=0.05
)
scheduler.start()
await asyncio.sleep(0.4)
await scheduler.stop()
run_async(run())
quota = limits.run_quota(local_db, "user", owner, now_utc()) quota = limits.run_quota(local_db, "user", owner, now_utc())
assert quota.used == 1 assert quota.used == 1
assert quota.limit == limits.DEFAULT_MEMBER_RUNS assert quota.limit == limits.DEFAULT_MEMBER_RUNS
@ -184,11 +206,18 @@ def test_a_member_over_the_run_quota_is_postponed_not_disabled(local_db):
local_db, "user", owner, generate_uid(), now - timedelta(minutes=index) local_db, "user", owner, generate_uid(), now - timedelta(minutes=index)
) )
started: list[str] = [] started: list[str] = []
gate = asyncio.Event()
run_async( async def run():
_run_scheduler(local_db, _harness(local_db, started, gate), gate=gate)() gate = asyncio.Event()
) gate.set()
scheduler = GlobalScheduler(
local_db, _harness(local_db, started, gate), tick_seconds=0.05
)
scheduler.start()
await asyncio.sleep(0.4)
await scheduler.stop()
run_async(run())
row = local_db["devii_tasks"].find_one(uid=uid) row = local_db["devii_tasks"].find_one(uid=uid)
assert started == [] assert started == []
assert row["enabled"] assert row["enabled"]
@ -208,9 +237,18 @@ def test_an_administrator_keeps_running_past_the_member_limit(local_db):
local_db, "user", owner, generate_uid(), now - timedelta(minutes=index) local_db, "user", owner, generate_uid(), now - timedelta(minutes=index)
) )
started: list[str] = [] started: list[str] = []
gate = asyncio.Event()
run_async(_run_scheduler(local_db, _harness(local_db, started, gate), gate=gate)()) async def run():
gate = asyncio.Event()
gate.set()
scheduler = GlobalScheduler(
local_db, _harness(local_db, started, gate), tick_seconds=0.05
)
scheduler.start()
await asyncio.sleep(0.4)
await scheduler.stop()
run_async(run())
assert started == [owner] assert started == [owner]
@ -220,16 +258,21 @@ def test_budget_postpones_the_task(local_db):
owner = _account(local_db, "Admin") owner = _account(local_db, "Admin")
uid = _seed(local_db, owner) uid = _seed(local_db, owner)
started: list[str] = [] started: list[str] = []
gate = asyncio.Event()
run_async( async def run():
_run_scheduler( gate = asyncio.Event()
gate.set()
scheduler = GlobalScheduler(
local_db, local_db,
_harness(local_db, started, gate), _harness(local_db, started, gate),
gate=gate, tick_seconds=0.05,
budget_exceeded=lambda kind, owner_id: True, budget_exceeded=lambda kind, owner_id: True,
)() )
) scheduler.start()
await asyncio.sleep(0.3)
await scheduler.stop()
run_async(run())
row = local_db["devii_tasks"].find_one(uid=uid) row = local_db["devii_tasks"].find_one(uid=uid)
assert started == [] assert started == []
assert row["enabled"] assert row["enabled"]
@ -240,10 +283,18 @@ def test_scheduler_retires_a_guest_owned_task(local_db):
local_db["devii_tasks"].delete() local_db["devii_tasks"].delete()
uid = _seed(local_db, "guest-cookie", owner_kind="guest") uid = _seed(local_db, "guest-cookie", owner_kind="guest")
started: list[str] = [] started: list[str] = []
gate = asyncio.Event()
gate.set()
run_async(_run_scheduler(local_db, _harness(local_db, started, gate), gate=gate)()) async def run():
gate = asyncio.Event()
gate.set()
scheduler = GlobalScheduler(
local_db, _harness(local_db, started, gate), tick_seconds=0.05
)
scheduler.start()
await asyncio.sleep(0.3)
await scheduler.stop()
run_async(run())
row = local_db["devii_tasks"].find_one(uid=uid) row = local_db["devii_tasks"].find_one(uid=uid)
assert not row["enabled"] assert not row["enabled"]
assert row["last_error"] == REASON_NOT_A_USER assert row["last_error"] == REASON_NOT_A_USER
@ -260,20 +311,27 @@ def test_scheduler_retires_expired_and_exhausted_tasks_even_while_saturated(loca
) )
exhausted = _seed(local_db, owner, max_runs=3, run_count=3) exhausted = _seed(local_db, owner, max_runs=3, run_count=3)
started: list[str] = [] started: list[str] = []
gate = asyncio.Event()
run_async(_run_scheduler(local_db, _harness(local_db, started, gate), gate=gate)()) async def run():
gate = asyncio.Event()
scheduler = GlobalScheduler(
local_db, _harness(local_db, started, gate), tick_seconds=0.05
)
scheduler.start()
await asyncio.sleep(0.4)
gate.set()
await asyncio.sleep(0.2)
await scheduler.stop()
run_async(run())
assert local_db["devii_tasks"].find_one(uid=expired)["last_error"] == REASON_EXPIRED assert local_db["devii_tasks"].find_one(uid=expired)["last_error"] == REASON_EXPIRED
assert ( assert local_db["devii_tasks"].find_one(uid=exhausted)["last_error"] == REASON_MAX_RUNS
local_db["devii_tasks"].find_one(uid=exhausted)["last_error"] == REASON_MAX_RUNS
)
for uid in (expired, exhausted): for uid in (expired, exhausted):
assert not local_db["devii_tasks"].find_one(uid=uid)["enabled"] assert not local_db["devii_tasks"].find_one(uid=uid)["enabled"]
def test_scheduler_disables_a_task_after_repeated_failures(local_db): def test_scheduler_disables_a_task_after_repeated_failures(local_db):
local_db["devii_tasks"].delete() local_db["devii_tasks"].delete()
local_db[limits.RUNS_TABLE].delete()
owner = _account(local_db, "Admin") owner = _account(local_db, "Admin")
uid = _seed(local_db, owner, failure_count=2) uid = _seed(local_db, owner, failure_count=2)
@ -299,27 +357,3 @@ def test_scheduler_disables_a_task_after_repeated_failures(local_db):
assert not row["enabled"] assert not row["enabled"]
assert row["status"] == "error" assert row["status"] == "error"
assert "upstream is down" in row["last_error"] assert "upstream is down" in row["last_error"]
def test_a_failed_run_still_counts_against_the_quota(local_db):
local_db["devii_tasks"].delete()
local_db[limits.RUNS_TABLE].delete()
owner = _account(local_db, "Member")
_seed(local_db, owner)
def resolve(row):
store = TaskStore(local_db, "user", str(row["owner_id"]))
async def executor(prompt):
raise RuntimeError("nope")
return store, executor, lambda kind, task_row, payload: None
async def run():
scheduler = GlobalScheduler(local_db, resolve, tick_seconds=0.05)
scheduler.start()
await asyncio.sleep(0.3)
await scheduler.stop()
run_async(run())
assert limits.run_quota(local_db, "user", owner, now_utc()).used >= 1

View File

@ -17,6 +17,8 @@ from devplacepy.services.devii.tasks.schedule import now_utc, to_iso
from devplacepy.services.devii.tasks.store import TaskStore, claim, due_rows from devplacepy.services.devii.tasks.store import TaskStore, claim, due_rows
from devplacepy.utils import generate_uid from devplacepy.utils import generate_uid
SIGNUP_AFTER_PRIMARY_ADMIN = "2099-01-01T00:00:00"
def _account(local_db, role): def _account(local_db, role):
uid = generate_uid() uid = generate_uid()
@ -26,7 +28,7 @@ def _account(local_db, role):
"username": f"store-{uid[-10:]}", "username": f"store-{uid[-10:]}",
"role": role, "role": role,
"deleted_at": None, "deleted_at": None,
"created_at": to_iso(now_utc()), "created_at": SIGNUP_AFTER_PRIMARY_ADMIN,
} }
) )
invalidate_admins_cache() invalidate_admins_cache()
@ -180,7 +182,8 @@ def test_claim_refuses_a_disabled_task(local_db):
def test_due_rows_skips_future_and_running_tasks(local_db): def test_due_rows_skips_future_and_running_tasks(local_db):
store = TaskStore(local_db, "user", _account(local_db, "Admin")) admin = _account(local_db, "Admin")
store = TaskStore(local_db, "user", admin)
now = now_utc() now = now_utc()
due = _record(next_run_at=to_iso(now.replace(year=now.year - 1))) due = _record(next_run_at=to_iso(now.replace(year=now.year - 1)))
future = _record(next_run_at=to_iso(now.replace(year=now.year + 1))) future = _record(next_run_at=to_iso(now.replace(year=now.year + 1)))
@ -238,13 +241,13 @@ def test_controller_refuses_nested_creation_for_a_member(local_db):
def test_controller_refuses_nested_run_now_for_a_member(local_db): def test_controller_refuses_nested_run_now_for_a_member(local_db):
import json
store = TaskStore(local_db, "user", _account(local_db, "Member")) store = TaskStore(local_db, "user", _account(local_db, "Member"))
controller = TaskController(store) controller = TaskController(store)
created = controller.create_task( created = controller.create_task(
{"prompt": "work", "kind": "interval", "every_seconds": 900} {"prompt": "work", "kind": "interval", "every_seconds": 900}
) )
import json
uid = json.loads(created)["task"]["uid"] uid = json.loads(created)["task"]["uid"]
with task_run_scope(): with task_run_scope():
with pytest.raises(ToolInputError) as failure: with pytest.raises(ToolInputError) as failure: