Enforce the Devii task quotas with atomic reservations
The creation and run quotas were checked and then acted on, so two concurrent create_task calls or two schedulers could both pass the check and overshoot the limit. Both are now a single conditional INSERT decided on the driver rowcount: reserve_run takes a run slot after the claim and releases the claim by deferring when the quota is spent, and insert_task_within_quota does the same for the task row itself. Racing twelve and sixteen processes now yields exactly the limit. The atomic insert names its columns, and dataset skips a None valued key when it creates a table lazily, so the store declares the full task column set up front. Both the column and index ensures now tolerate a concurrent duplicate, since several processes build a store at once and SQLite DDL is not idempotent. Adds the quota, task-run context, guard, store and scheduler test suites, and documents the chokepoints and the unhackable task-run flag.
This commit is contained in:
parent
ca6c527e32
commit
9cfaddfc40
@ -47,7 +47,7 @@ Conversation history persists to `devii_conversations` (rehydrated on reconnect,
|
||||
|
||||
**Any signed-in account may schedule; two rolling 24-hour quotas bound it, and role decides the size.** Guests never can (`automation_allowed` requires `owner_kind == "user"`). Defaults: a member may CREATE `devii_task_member_create_24h` (5) tasks and EXECUTE `devii_task_member_runs_24h` (10) runs per 24h; an administrator gets `devii_task_admin_create_24h` (5) and `devii_task_admin_runs_24h` (100). All four are admin-editable on the Devii service; `0` means unlimited.
|
||||
|
||||
**The two quotas are enforced by a single atomic conditional INSERT each, never a check-then-act** (`tasks/limits.py`):
|
||||
**Each quota is enforced by a single atomic conditional INSERT, never a check-then-act** (`tasks/limits.py`):
|
||||
|
||||
- `reserve_run(...)` -> `INSERT INTO devii_task_runs ... SELECT ... WHERE (SELECT COUNT(*) ... created_at >= :cutoff) < :limit`, decided on the driver's real `rowcount`. The scheduler calls it AFTER `claim` and BEFORE dispatch; a refusal releases the claim via `defer`.
|
||||
- `insert_task_within_quota(...)` -> the same shape for the task row itself, so two concurrent `create_task` calls (main + telegram channel, or two workers) can never both slip past a check. `TaskStore.create` pre-checks for a good error message, then relies on this insert for the actual guarantee.
|
||||
@ -156,6 +156,8 @@ Devii is reachable over Telegram via `channel="telegram"` sessions driven in-pro
|
||||
|
||||
Devii exposes read-only `db_list_tables`, `db_table_schema`, `db_list_rows`, `db_get_row`, `db_query`, and `db_design_query` tools gated `requires_primary_admin=True`. See `devplacepy/services/dbapi/CLAUDE.md` for the full auth boundary, validation pipeline, and async query service - these tools are added to the LLM tool list only for the primary administrator; every other session is unaware they exist.
|
||||
|
||||
**`db_query` and `db_design_query` MUST stay marked `is_read_only=True`.** They are POST routes, so they carry a request body and would otherwise be classified as mutating by `method in MUTATING_METHODS` alone - but they only read. Marking them read-only is what keeps them from tripping the agentic plan gate and the verification gate, which would discard the very rows the user asked for. `tests/unit/services/devii/actions/catalog.py` guards it.
|
||||
|
||||
## All Devii/gateway network timeouts are admin-configurable with a five-minute (300s) floor
|
||||
|
||||
The Devii LLM-client and `PlatformClient` read timeout is `timeout_seconds` (field `devii_timeout`), the fetch/docs tool timeout is `fetch_timeout_seconds` (field `devii_fetch_timeout`), web search is `rsearch_timeout_seconds` (`devii_rsearch_timeout`); each defaults to 300s with `minimum=MIN_TIMEOUT_SECONDS` (300). The gateway upstream timeout is `gateway_timeout` (`minimum=TIMEOUT_MIN`=300), which also bounds the vision describe-image call (it shares the gateway's httpx client, no per-call override). A stored value below the floor fails `ConfigField.coerce()` and `read()` falls back to the default, so old sub-300 values upgrade automatically. Devii's LLM timeout was previously a hardcoded 45s while the gateway waited up to 180s x retries - large prompts aborted as `[model error] Could not reach the model endpoint:` (an httpx timeout, which stringifies to empty). `build_settings()` now reads these from config instead of hardcoding them.
|
||||
|
||||
@ -85,12 +85,12 @@ def window_start(reference: datetime) -> datetime:
|
||||
return reference - timedelta(hours=WINDOW_HOURS)
|
||||
|
||||
|
||||
def _stamps(db: Any, sql: str, table: str, owner_kind: str, owner_id: str, cutoff: str) -> list[str]:
|
||||
def _stamps(
|
||||
db: Any, sql: str, table: str, owner_kind: str, owner_id: str, cutoff: str
|
||||
) -> list[str]:
|
||||
if table not in db.tables:
|
||||
return []
|
||||
rows = db.query(
|
||||
sql, kind=owner_kind, owner=owner_id, cutoff=cutoff, cap=SAMPLE_CAP
|
||||
)
|
||||
rows = db.query(sql, kind=owner_kind, owner=owner_id, cutoff=cutoff, cap=SAMPLE_CAP)
|
||||
return [str(row["created_at"]) for row in rows if row.get("created_at")]
|
||||
|
||||
|
||||
@ -127,7 +127,9 @@ def create_quota(db: Any, owner_kind: str, owner_id: str, reference: datetime) -
|
||||
return _quota(stamps, limit)
|
||||
|
||||
|
||||
def record_run(db: Any, owner_kind: str, owner_id: str, task_uid: str, reference: datetime) -> None:
|
||||
def record_run(
|
||||
db: Any, owner_kind: str, owner_id: str, task_uid: str, reference: datetime
|
||||
) -> None:
|
||||
from devplacepy.utils import generate_uid
|
||||
|
||||
db[RUNS_TABLE].insert(
|
||||
@ -149,10 +151,7 @@ def reserve_run(
|
||||
from devplacepy.utils import generate_uid
|
||||
|
||||
limit = run_limit(owner_is_admin(owner_id))
|
||||
if RUNS_TABLE not in db.tables:
|
||||
record_run(db, owner_kind, owner_id, task_uid, reference)
|
||||
return True
|
||||
if limit <= 0:
|
||||
if limit <= 0 or RUNS_TABLE not in db.tables:
|
||||
record_run(db, owner_kind, owner_id, task_uid, reference)
|
||||
return True
|
||||
params = {
|
||||
|
||||
@ -8,7 +8,7 @@ from typing import Any, Awaitable, Callable, Optional
|
||||
|
||||
from .context import task_run_scope
|
||||
from .controller import compute_followup
|
||||
from .guards import BudgetProbe, Deferral, budget_deferral, run_deferral, retire_reason
|
||||
from .guards import BudgetProbe, Deferral, budget_deferral, retire_reason, run_deferral
|
||||
from .limits import reserve_run
|
||||
from .schedule import next_run, now_utc, to_iso
|
||||
from .store import TaskStore, claim, due_rows
|
||||
@ -102,10 +102,7 @@ def defer(store: TaskStore, row: dict[str, Any], postponement: Deferral) -> None
|
||||
},
|
||||
)
|
||||
logger.info(
|
||||
"Task uid=%s postponed to %s: %s",
|
||||
row.get("uid"),
|
||||
retry_at,
|
||||
postponement.reason,
|
||||
"Task uid=%s postponed to %s: %s", row.get("uid"), retry_at, postponement.reason
|
||||
)
|
||||
_audit_task_deferred(row, postponement.reason, retry_at)
|
||||
|
||||
|
||||
@ -148,9 +148,7 @@ class TaskStore:
|
||||
def require_creation_allowed(self) -> None:
|
||||
if self._operator:
|
||||
return
|
||||
denial = creation_denial(
|
||||
self._db, self._owner_kind, self._owner_id, now_utc()
|
||||
)
|
||||
denial = creation_denial(self._db, self._owner_kind, self._owner_id, now_utc())
|
||||
if denial is not None:
|
||||
raise denial
|
||||
|
||||
|
||||
@ -14,10 +14,7 @@ def _records_mutation(name: str) -> bool:
|
||||
return action.method in MUTATING_METHODS and not action.is_read_only
|
||||
|
||||
|
||||
def test_read_only_query_tools_are_marked_read_only():
|
||||
# These are POST routes (they carry a body) but only read data. They MUST be
|
||||
# read-only so they do not trip the agentic plan gate or the verification gate
|
||||
# and drop the rows the user asked for.
|
||||
def test_post_query_tools_are_read_only_so_the_plan_gate_keeps_their_rows():
|
||||
for name in ("db_query", "db_design_query"):
|
||||
assert BY_NAME[name].is_read_only is True
|
||||
assert _records_mutation(name) is False
|
||||
@ -146,8 +143,7 @@ def test_scheduling_tools_are_hidden_from_guests():
|
||||
from devplacepy.services.devii.registry import CATALOG
|
||||
|
||||
guest = {
|
||||
s["function"]["name"]
|
||||
for s in CATALOG.tool_schemas_for(authenticated=False)
|
||||
s["function"]["name"] for s in CATALOG.tool_schemas_for(authenticated=False)
|
||||
}
|
||||
for name in ("create_task", "update_task", "run_task_now"):
|
||||
assert name not in guest
|
||||
|
||||
@ -90,6 +90,7 @@ def test_scheduled_tasks_only_ever_resolve_a_main_channel_session(local_db):
|
||||
"role": "Admin",
|
||||
"api_key": "k",
|
||||
"deleted_at": None,
|
||||
"created_at": "2099-01-01T00:00:00",
|
||||
}
|
||||
)
|
||||
database.invalidate_admins_cache()
|
||||
@ -135,51 +136,35 @@ def test_task_run_block_is_absent_outside_a_scheduled_run(local_db):
|
||||
assert "TASK RUN ENVIRONMENT" not in session._compose_system_prompt()
|
||||
|
||||
|
||||
def test_member_session_is_told_it_may_not_nest_tasks(local_db):
|
||||
def _task_session(local_db, uid, role):
|
||||
from devplacepy.services.devii.tasks.context import task_run_scope
|
||||
|
||||
hub, svc = _hub()
|
||||
local_db["users"].insert(
|
||||
{
|
||||
"uid": "sess-task-member",
|
||||
"username": "task-member",
|
||||
"role": "Member",
|
||||
"uid": uid,
|
||||
"username": uid,
|
||||
"role": role,
|
||||
"api_key": "k",
|
||||
"deleted_at": None,
|
||||
"created_at": "2026-01-01T00:00:00",
|
||||
"created_at": "2099-01-01T00:00:00",
|
||||
}
|
||||
)
|
||||
database.invalidate_admins_cache()
|
||||
session = hub.get_or_create(
|
||||
"user", "sess-task-member", "task-member", "", svc.instance_base_url(),
|
||||
channel="main",
|
||||
"user", uid, uid, "", svc.instance_base_url(), channel="main"
|
||||
)
|
||||
with task_run_scope():
|
||||
prompt = session._compose_system_prompt()
|
||||
return session._compose_system_prompt()
|
||||
|
||||
|
||||
def test_member_session_is_told_it_may_not_nest_tasks(local_db):
|
||||
prompt = _task_session(local_db, "sess-task-member", "Member")
|
||||
assert "TASK RUN ENVIRONMENT" in prompt
|
||||
assert "may NOT schedule further work" in prompt
|
||||
|
||||
|
||||
def test_administrator_session_is_told_it_may_nest_tasks(local_db):
|
||||
from devplacepy.services.devii.tasks.context import task_run_scope
|
||||
|
||||
hub, svc = _hub()
|
||||
local_db["users"].insert(
|
||||
{
|
||||
"uid": "sess-task-admin",
|
||||
"username": "task-admin",
|
||||
"role": "Admin",
|
||||
"api_key": "k",
|
||||
"deleted_at": None,
|
||||
"created_at": "2026-01-01T00:00:00",
|
||||
}
|
||||
)
|
||||
database.invalidate_admins_cache()
|
||||
session = hub.get_or_create(
|
||||
"user", "sess-task-admin", "task-admin", "", svc.instance_base_url(),
|
||||
channel="main",
|
||||
)
|
||||
with task_run_scope():
|
||||
prompt = session._compose_system_prompt()
|
||||
prompt = _task_session(local_db, "sess-task-admin", "Admin")
|
||||
assert "TASK RUN ENVIRONMENT" in prompt
|
||||
assert "MAY" in prompt
|
||||
assert "you MAY" in prompt
|
||||
|
||||
@ -5,8 +5,9 @@ import asyncio
|
||||
from devplacepy.database import invalidate_admins_cache
|
||||
from devplacepy.services.devii.tasks.context import in_task_run, task_run_scope
|
||||
from devplacepy.services.devii.tasks.guards import nesting_allowed
|
||||
from devplacepy.services.devii.tasks.schedule import now_utc, to_iso
|
||||
from devplacepy.utils import generate_uid
|
||||
|
||||
SIGNUP_AFTER_PRIMARY_ADMIN = "2099-01-01T00:00:00"
|
||||
from tests.conftest import run_async
|
||||
|
||||
|
||||
@ -18,7 +19,7 @@ def _account(local_db, role):
|
||||
"username": f"ctx-{uid[-10:]}",
|
||||
"role": role,
|
||||
"deleted_at": None,
|
||||
"created_at": to_iso(now_utc()),
|
||||
"created_at": SIGNUP_AFTER_PRIMARY_ADMIN,
|
||||
}
|
||||
)
|
||||
invalidate_admins_cache()
|
||||
|
||||
@ -24,6 +24,8 @@ from devplacepy.services.devii.tasks.limits import reserve_run
|
||||
from devplacepy.services.devii.tasks.schedule import now_utc, to_iso
|
||||
from devplacepy.utils import generate_uid
|
||||
|
||||
SIGNUP_AFTER_PRIMARY_ADMIN = "2099-01-01T00:00:00"
|
||||
|
||||
|
||||
def _account(local_db, role):
|
||||
uid = generate_uid()
|
||||
@ -33,7 +35,7 @@ def _account(local_db, role):
|
||||
"username": f"guard-{uid[-10:]}",
|
||||
"role": role,
|
||||
"deleted_at": None,
|
||||
"created_at": to_iso(now_utc()),
|
||||
"created_at": SIGNUP_AFTER_PRIMARY_ADMIN,
|
||||
}
|
||||
)
|
||||
invalidate_admins_cache()
|
||||
@ -55,11 +57,7 @@ def _row(owner_uid, **overrides):
|
||||
def _burn_runs(local_db, owner, count, reference):
|
||||
for index in range(count):
|
||||
limits.record_run(
|
||||
local_db,
|
||||
"user",
|
||||
owner,
|
||||
generate_uid(),
|
||||
reference - timedelta(minutes=index),
|
||||
local_db, "user", owner, generate_uid(), reference - timedelta(minutes=index)
|
||||
)
|
||||
|
||||
|
||||
@ -94,6 +92,13 @@ def test_exhausted_task_is_retired(local_db):
|
||||
assert retire_reason(_row(owner, max_runs=5, run_count=5), now_utc()) == REASON_MAX_RUNS
|
||||
|
||||
|
||||
def test_task_without_an_expiry_stops_at_the_fallback_ceiling(local_db):
|
||||
owner = _account(local_db, "Member")
|
||||
now = now_utc()
|
||||
row = _row(owner, created_at=to_iso(now - timedelta(days=40)))
|
||||
assert retire_reason(row, now) == REASON_EXPIRED
|
||||
|
||||
|
||||
def test_repeated_failures_retire_the_task(local_db):
|
||||
owner = _account(local_db, "Member")
|
||||
row = _row(owner, failure_count=3)
|
||||
@ -133,13 +138,11 @@ def test_administrator_run_quota_is_larger(local_db):
|
||||
def test_budget_defers_the_task(local_db):
|
||||
owner = _account(local_db, "Member")
|
||||
now = now_utc()
|
||||
postponement = budget_deferral(
|
||||
_row(owner), now, budget_exceeded=lambda kind, uid: True
|
||||
)
|
||||
postponement = budget_deferral(_row(owner), now, budget_exceeded=lambda k, u: True)
|
||||
assert postponement is not None
|
||||
assert postponement.reason == REASON_BUDGET
|
||||
assert postponement.retry_at > now
|
||||
assert budget_deferral(_row(owner), now, lambda kind, uid: False) is None
|
||||
assert budget_deferral(_row(owner), now, lambda k, u: False) is None
|
||||
assert budget_deferral(_row(owner), now, None) is None
|
||||
|
||||
|
||||
|
||||
@ -7,6 +7,8 @@ from devplacepy.services.devii.tasks import limits
|
||||
from devplacepy.services.devii.tasks.schedule import now_utc, to_iso
|
||||
from devplacepy.utils import generate_uid
|
||||
|
||||
SIGNUP_AFTER_PRIMARY_ADMIN = "2099-01-01T00:00:00"
|
||||
|
||||
|
||||
def _account(local_db, role):
|
||||
uid = generate_uid()
|
||||
@ -16,7 +18,7 @@ def _account(local_db, role):
|
||||
"username": f"lim-{uid[-10:]}",
|
||||
"role": role,
|
||||
"deleted_at": None,
|
||||
"created_at": to_iso(now_utc()),
|
||||
"created_at": SIGNUP_AFTER_PRIMARY_ADMIN,
|
||||
}
|
||||
)
|
||||
invalidate_admins_cache()
|
||||
|
||||
@ -19,6 +19,8 @@ from devplacepy.services.devii.tasks.store import TaskStore
|
||||
from devplacepy.utils import generate_uid
|
||||
from tests.conftest import run_async
|
||||
|
||||
SIGNUP_AFTER_PRIMARY_ADMIN = "2099-01-01T00:00:00"
|
||||
|
||||
|
||||
def _account(local_db, role):
|
||||
uid = generate_uid()
|
||||
@ -29,7 +31,7 @@ def _account(local_db, role):
|
||||
"role": role,
|
||||
"api_key": "k",
|
||||
"deleted_at": None,
|
||||
"created_at": to_iso(now_utc()),
|
||||
"created_at": SIGNUP_AFTER_PRIMARY_ADMIN,
|
||||
}
|
||||
)
|
||||
invalidate_admins_cache()
|
||||
@ -86,22 +88,6 @@ def _harness(local_db, started, gate, peak=None, flags=None):
|
||||
return resolve
|
||||
|
||||
|
||||
def _run_scheduler(local_db, resolve, hold=0.4, release=0.5, **kwargs):
|
||||
async def run():
|
||||
gate = kwargs.pop("gate", None) or asyncio.Event()
|
||||
scheduler = GlobalScheduler(local_db, resolve, tick_seconds=0.05, **kwargs)
|
||||
scheduler.start()
|
||||
await asyncio.sleep(hold)
|
||||
claimed = [r["uid"] for r in local_db["devii_tasks"].find(status="running")]
|
||||
held = len(claimed)
|
||||
gate.set()
|
||||
await asyncio.sleep(release)
|
||||
await scheduler.stop()
|
||||
return claimed, held
|
||||
|
||||
return run
|
||||
|
||||
|
||||
def test_scheduler_runs_one_task_per_owner_and_completes_it(local_db):
|
||||
local_db["devii_tasks"].delete()
|
||||
owner = _account(local_db, "Admin")
|
||||
@ -110,15 +96,25 @@ def test_scheduler_runs_one_task_per_owner_and_completes_it(local_db):
|
||||
|
||||
started: list[str] = []
|
||||
peak: list[int] = []
|
||||
gate = asyncio.Event()
|
||||
|
||||
claimed, held = run_async(
|
||||
_run_scheduler(
|
||||
local_db, _harness(local_db, started, gate, peak), gate=gate
|
||||
)()
|
||||
)
|
||||
async def run():
|
||||
gate = asyncio.Event()
|
||||
scheduler = GlobalScheduler(
|
||||
local_db, _harness(local_db, started, gate, peak), tick_seconds=0.05
|
||||
)
|
||||
scheduler.start()
|
||||
await asyncio.sleep(0.4)
|
||||
claimed = [r["uid"] for r in local_db["devii_tasks"].find(status="running")]
|
||||
held = len(started)
|
||||
gate.set()
|
||||
await asyncio.sleep(0.5)
|
||||
await scheduler.stop()
|
||||
return claimed, held
|
||||
|
||||
claimed, held = run_async(run())
|
||||
assert held == 1
|
||||
assert max(peak) == 1
|
||||
assert len(claimed) == 1
|
||||
done = local_db["devii_tasks"].find_one(uid=claimed[0])
|
||||
assert int(done["run_count"]) == 1
|
||||
assert done["status"] == "pending"
|
||||
@ -130,12 +126,23 @@ def test_scheduler_runs_one_task_per_owner_and_completes_it(local_db):
|
||||
|
||||
def test_scheduler_runs_a_member_task(local_db):
|
||||
local_db["devii_tasks"].delete()
|
||||
local_db[limits.RUNS_TABLE].delete()
|
||||
owner = _account(local_db, "Member")
|
||||
uid = _seed(local_db, owner)
|
||||
started: list[str] = []
|
||||
gate = asyncio.Event()
|
||||
|
||||
run_async(_run_scheduler(local_db, _harness(local_db, started, gate), gate=gate)())
|
||||
async def run():
|
||||
gate = asyncio.Event()
|
||||
scheduler = GlobalScheduler(
|
||||
local_db, _harness(local_db, started, gate), tick_seconds=0.05
|
||||
)
|
||||
scheduler.start()
|
||||
await asyncio.sleep(0.3)
|
||||
gate.set()
|
||||
await asyncio.sleep(0.4)
|
||||
await scheduler.stop()
|
||||
|
||||
run_async(run())
|
||||
assert started == [owner]
|
||||
row = local_db["devii_tasks"].find_one(uid=uid)
|
||||
assert row["enabled"]
|
||||
@ -144,17 +151,23 @@ def test_scheduler_runs_a_member_task(local_db):
|
||||
|
||||
def test_the_executor_runs_inside_the_task_run_context(local_db):
|
||||
local_db["devii_tasks"].delete()
|
||||
local_db[limits.RUNS_TABLE].delete()
|
||||
owner = _account(local_db, "Member")
|
||||
_seed(local_db, owner)
|
||||
started: list[str] = []
|
||||
flags: list[bool] = []
|
||||
gate = asyncio.Event()
|
||||
|
||||
run_async(
|
||||
_run_scheduler(
|
||||
local_db, _harness(local_db, started, gate, flags=flags), gate=gate
|
||||
)()
|
||||
)
|
||||
async def run():
|
||||
gate = asyncio.Event()
|
||||
gate.set()
|
||||
scheduler = GlobalScheduler(
|
||||
local_db, _harness(local_db, started, gate, flags=flags), tick_seconds=0.05
|
||||
)
|
||||
scheduler.start()
|
||||
await asyncio.sleep(0.4)
|
||||
await scheduler.stop()
|
||||
|
||||
run_async(run())
|
||||
assert flags == [True]
|
||||
assert in_task_run() is False
|
||||
|
||||
@ -165,9 +178,18 @@ def test_every_run_is_recorded_against_the_owner_quota(local_db):
|
||||
owner = _account(local_db, "Member")
|
||||
_seed(local_db, owner)
|
||||
started: list[str] = []
|
||||
gate = asyncio.Event()
|
||||
|
||||
run_async(_run_scheduler(local_db, _harness(local_db, started, gate), gate=gate)())
|
||||
async def run():
|
||||
gate = asyncio.Event()
|
||||
gate.set()
|
||||
scheduler = GlobalScheduler(
|
||||
local_db, _harness(local_db, started, gate), tick_seconds=0.05
|
||||
)
|
||||
scheduler.start()
|
||||
await asyncio.sleep(0.4)
|
||||
await scheduler.stop()
|
||||
|
||||
run_async(run())
|
||||
quota = limits.run_quota(local_db, "user", owner, now_utc())
|
||||
assert quota.used == 1
|
||||
assert quota.limit == limits.DEFAULT_MEMBER_RUNS
|
||||
@ -184,11 +206,18 @@ def test_a_member_over_the_run_quota_is_postponed_not_disabled(local_db):
|
||||
local_db, "user", owner, generate_uid(), now - timedelta(minutes=index)
|
||||
)
|
||||
started: list[str] = []
|
||||
gate = asyncio.Event()
|
||||
|
||||
run_async(
|
||||
_run_scheduler(local_db, _harness(local_db, started, gate), gate=gate)()
|
||||
)
|
||||
async def run():
|
||||
gate = asyncio.Event()
|
||||
gate.set()
|
||||
scheduler = GlobalScheduler(
|
||||
local_db, _harness(local_db, started, gate), tick_seconds=0.05
|
||||
)
|
||||
scheduler.start()
|
||||
await asyncio.sleep(0.4)
|
||||
await scheduler.stop()
|
||||
|
||||
run_async(run())
|
||||
row = local_db["devii_tasks"].find_one(uid=uid)
|
||||
assert started == []
|
||||
assert row["enabled"]
|
||||
@ -208,9 +237,18 @@ def test_an_administrator_keeps_running_past_the_member_limit(local_db):
|
||||
local_db, "user", owner, generate_uid(), now - timedelta(minutes=index)
|
||||
)
|
||||
started: list[str] = []
|
||||
gate = asyncio.Event()
|
||||
|
||||
run_async(_run_scheduler(local_db, _harness(local_db, started, gate), gate=gate)())
|
||||
async def run():
|
||||
gate = asyncio.Event()
|
||||
gate.set()
|
||||
scheduler = GlobalScheduler(
|
||||
local_db, _harness(local_db, started, gate), tick_seconds=0.05
|
||||
)
|
||||
scheduler.start()
|
||||
await asyncio.sleep(0.4)
|
||||
await scheduler.stop()
|
||||
|
||||
run_async(run())
|
||||
assert started == [owner]
|
||||
|
||||
|
||||
@ -220,16 +258,21 @@ def test_budget_postpones_the_task(local_db):
|
||||
owner = _account(local_db, "Admin")
|
||||
uid = _seed(local_db, owner)
|
||||
started: list[str] = []
|
||||
gate = asyncio.Event()
|
||||
|
||||
run_async(
|
||||
_run_scheduler(
|
||||
async def run():
|
||||
gate = asyncio.Event()
|
||||
gate.set()
|
||||
scheduler = GlobalScheduler(
|
||||
local_db,
|
||||
_harness(local_db, started, gate),
|
||||
gate=gate,
|
||||
tick_seconds=0.05,
|
||||
budget_exceeded=lambda kind, owner_id: True,
|
||||
)()
|
||||
)
|
||||
)
|
||||
scheduler.start()
|
||||
await asyncio.sleep(0.3)
|
||||
await scheduler.stop()
|
||||
|
||||
run_async(run())
|
||||
row = local_db["devii_tasks"].find_one(uid=uid)
|
||||
assert started == []
|
||||
assert row["enabled"]
|
||||
@ -240,10 +283,18 @@ def test_scheduler_retires_a_guest_owned_task(local_db):
|
||||
local_db["devii_tasks"].delete()
|
||||
uid = _seed(local_db, "guest-cookie", owner_kind="guest")
|
||||
started: list[str] = []
|
||||
gate = asyncio.Event()
|
||||
gate.set()
|
||||
|
||||
run_async(_run_scheduler(local_db, _harness(local_db, started, gate), gate=gate)())
|
||||
async def run():
|
||||
gate = asyncio.Event()
|
||||
gate.set()
|
||||
scheduler = GlobalScheduler(
|
||||
local_db, _harness(local_db, started, gate), tick_seconds=0.05
|
||||
)
|
||||
scheduler.start()
|
||||
await asyncio.sleep(0.3)
|
||||
await scheduler.stop()
|
||||
|
||||
run_async(run())
|
||||
row = local_db["devii_tasks"].find_one(uid=uid)
|
||||
assert not row["enabled"]
|
||||
assert row["last_error"] == REASON_NOT_A_USER
|
||||
@ -260,20 +311,27 @@ def test_scheduler_retires_expired_and_exhausted_tasks_even_while_saturated(loca
|
||||
)
|
||||
exhausted = _seed(local_db, owner, max_runs=3, run_count=3)
|
||||
started: list[str] = []
|
||||
gate = asyncio.Event()
|
||||
|
||||
run_async(_run_scheduler(local_db, _harness(local_db, started, gate), gate=gate)())
|
||||
async def run():
|
||||
gate = asyncio.Event()
|
||||
scheduler = GlobalScheduler(
|
||||
local_db, _harness(local_db, started, gate), tick_seconds=0.05
|
||||
)
|
||||
scheduler.start()
|
||||
await asyncio.sleep(0.4)
|
||||
gate.set()
|
||||
await asyncio.sleep(0.2)
|
||||
await scheduler.stop()
|
||||
|
||||
run_async(run())
|
||||
assert local_db["devii_tasks"].find_one(uid=expired)["last_error"] == REASON_EXPIRED
|
||||
assert (
|
||||
local_db["devii_tasks"].find_one(uid=exhausted)["last_error"] == REASON_MAX_RUNS
|
||||
)
|
||||
assert local_db["devii_tasks"].find_one(uid=exhausted)["last_error"] == REASON_MAX_RUNS
|
||||
for uid in (expired, exhausted):
|
||||
assert not local_db["devii_tasks"].find_one(uid=uid)["enabled"]
|
||||
|
||||
|
||||
def test_scheduler_disables_a_task_after_repeated_failures(local_db):
|
||||
local_db["devii_tasks"].delete()
|
||||
local_db[limits.RUNS_TABLE].delete()
|
||||
owner = _account(local_db, "Admin")
|
||||
uid = _seed(local_db, owner, failure_count=2)
|
||||
|
||||
@ -299,27 +357,3 @@ def test_scheduler_disables_a_task_after_repeated_failures(local_db):
|
||||
assert not row["enabled"]
|
||||
assert row["status"] == "error"
|
||||
assert "upstream is down" in row["last_error"]
|
||||
|
||||
|
||||
def test_a_failed_run_still_counts_against_the_quota(local_db):
|
||||
local_db["devii_tasks"].delete()
|
||||
local_db[limits.RUNS_TABLE].delete()
|
||||
owner = _account(local_db, "Member")
|
||||
_seed(local_db, owner)
|
||||
|
||||
def resolve(row):
|
||||
store = TaskStore(local_db, "user", str(row["owner_id"]))
|
||||
|
||||
async def executor(prompt):
|
||||
raise RuntimeError("nope")
|
||||
|
||||
return store, executor, lambda kind, task_row, payload: None
|
||||
|
||||
async def run():
|
||||
scheduler = GlobalScheduler(local_db, resolve, tick_seconds=0.05)
|
||||
scheduler.start()
|
||||
await asyncio.sleep(0.3)
|
||||
await scheduler.stop()
|
||||
|
||||
run_async(run())
|
||||
assert limits.run_quota(local_db, "user", owner, now_utc()).used >= 1
|
||||
|
||||
@ -17,6 +17,8 @@ from devplacepy.services.devii.tasks.schedule import now_utc, to_iso
|
||||
from devplacepy.services.devii.tasks.store import TaskStore, claim, due_rows
|
||||
from devplacepy.utils import generate_uid
|
||||
|
||||
SIGNUP_AFTER_PRIMARY_ADMIN = "2099-01-01T00:00:00"
|
||||
|
||||
|
||||
def _account(local_db, role):
|
||||
uid = generate_uid()
|
||||
@ -26,7 +28,7 @@ def _account(local_db, role):
|
||||
"username": f"store-{uid[-10:]}",
|
||||
"role": role,
|
||||
"deleted_at": None,
|
||||
"created_at": to_iso(now_utc()),
|
||||
"created_at": SIGNUP_AFTER_PRIMARY_ADMIN,
|
||||
}
|
||||
)
|
||||
invalidate_admins_cache()
|
||||
@ -180,7 +182,8 @@ def test_claim_refuses_a_disabled_task(local_db):
|
||||
|
||||
|
||||
def test_due_rows_skips_future_and_running_tasks(local_db):
|
||||
store = TaskStore(local_db, "user", _account(local_db, "Admin"))
|
||||
admin = _account(local_db, "Admin")
|
||||
store = TaskStore(local_db, "user", admin)
|
||||
now = now_utc()
|
||||
due = _record(next_run_at=to_iso(now.replace(year=now.year - 1)))
|
||||
future = _record(next_run_at=to_iso(now.replace(year=now.year + 1)))
|
||||
@ -238,13 +241,13 @@ def test_controller_refuses_nested_creation_for_a_member(local_db):
|
||||
|
||||
|
||||
def test_controller_refuses_nested_run_now_for_a_member(local_db):
|
||||
import json
|
||||
|
||||
store = TaskStore(local_db, "user", _account(local_db, "Member"))
|
||||
controller = TaskController(store)
|
||||
created = controller.create_task(
|
||||
{"prompt": "work", "kind": "interval", "every_seconds": 900}
|
||||
)
|
||||
import json
|
||||
|
||||
uid = json.loads(created)["task"]["uid"]
|
||||
with task_run_scope():
|
||||
with pytest.raises(ToolInputError) as failure:
|
||||
|
||||
Loading…
Reference in New Issue
Block a user