Order the workspace editor certificate at creation

Browse Source 
A new tunnel row was inert at pending until the next WorkspaceService
tick picked it up, so the editor certificate waited up to 30 seconds
before the ACME order even started. Opening a workspace and reaching its
public hostname in that window served the wildcard default certificate
and produced ERR_CERT_COMMON_NAME_INVALID in the browser.

certs.certify now owns the whole transition - provisioning, issue,
active or failed, then the owner notification - and both callers use it.
provision.ensure schedules it for the editor tunnel the moment the
workspace exists, claiming the row synchronously so the tick can never
issue the same host twice and burn a duplicate-certificate slot. The
service loop keeps calling certify per pending row as the safety net for
user-created tunnels and for rows created while molohttp was
unconfigured.

Issuance is a loop task rather than an await: an order takes about ten
seconds and certs.issue allows up to 180, which would hang the workspace
POST. The task is held until it completes so it cannot be collected mid
flight.

resume also restarts the idle clock, and the workspace routes gain
HTTP-level tests covering every refusal path they had none for.
This commit is contained in:
retoor 2026-08-08 13:52:02 +02:00
parent 56becfb3f7
commit 372067bbe4
5 changed files with 528 additions and 60 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

40
devplacepy/services/containers/CLAUDE.md
View File

@ -252,12 +252,40 @@ wildcard over `http-01`. So a `*.tunnel.pravda.education` **certificate** is imp
site describes **routing**, which is correct and already in place (one enabled molohttp site
`*.tunnel.pravda.education -> http://127.0.0.1:10500`, no per-tunnel site object). Only the cert is
per-host. `tunnels.create` writes a row at `STATUS_PENDING` and nothing else; the row is inert until
`_issue_tunnel_certificates` picks it up - `pending` is not in `SERVING_STATUSES`, so an
un-provisioned tunnel 404s. The phase reads `tunnels.awaiting_certificate()` (`status=pending` +
`desired_state=present` + not deleted), flips the row to `provisioning`, calls
`certs.issue(hostname)` (`workspace/certs.py`, `POST {workspace_molohttp_base_url}/api/v1/certs/issue`
with the `x-api-key` header, Basic as fallback), and lands on `active` or on `failed` with
`last_error`. **Only `pending` is retried** - a `failed` row stays failed until the user recreates
something certifies it - `pending` is not in `SERVING_STATUSES`, so an un-provisioned tunnel 404s.
**One state machine, two callers.** `certs.certify(row, log=None)` owns the whole transition:
`provisioning` -> `certs.issue(hostname)` (`POST {workspace_molohttp_base_url}/api/v1/certs/issue`
with the `x-api-key` header, Basic as fallback) -> `active` or `failed` with `last_error`, then
`certs.announce(row, hostname)` sends the owner the live URL (plus the password for the editor
tunnel). Never re-implement those transitions at a call site - both paths call `certify`:
- **Eagerly, at workspace creation.** `provision.ensure` ends with
`schedule_certificate(publish_editor_tunnel(...))`, which claims the row as `provisioning`
**synchronously** and then spawns `certify` as a loop task, so the editor certificate is ordered
the instant the workspace exists rather than up to one service tick later. It is fire-and-forget
by necessity: an ACME `http-01` order takes ~10s and `certs.issue` allows up to 180s, so awaiting
it would hang the `POST /projects/{slug}/workspace` response. The task is held in
`provision._pending_certificates` and discarded on completion - a bare `create_task` reference can
be garbage collected mid-flight. The synchronous claim is what closes the race with the tick: the
service selects only `pending`, so a row already claimed can never be issued twice and burn a
Let's Encrypt duplicate-certificate slot. `schedule_certificate` no-ops (leaving the row `pending`
for the tick) when molohttp is unconfigured or there is no running loop.
- **On the tick, as the safety net.** `WorkspaceService._issue_tunnel_certificates` reads
`tunnels.awaiting_certificate()` (`status=pending` + `desired_state=present` + not deleted) and
calls `certify` per row. This covers user-created tunnels, rows predating the eager path, and any
creation that happened with molohttp unconfigured.
The residual cost is the ACME round trip itself, which is irreducible under `http-01`. The tick
interval (`service_workspace_interval`, default 30s, floor `min_interval = 5`) no longer sits in
front of the editor tunnel, but still bounds how long a user-created tunnel waits.
**A tunnel serves before its certificate exists.** `SERVING_STATUSES = (provisioning, active)`, so
`routers/tunnel.py` routes the host the moment the row is claimed, while the order is still in
flight. The wildcard DNS already resolves, so molohttp terminates TLS with its default certificate
and the browser shows `ERR_CERT_COMMON_NAME_INVALID` for that window rather than failing to connect.
That is deliberate (a reachable host beats a dead one) and is the reason eager issuance matters: the
window is only as long as issuance takes. **Only `pending` is retried** - a `failed` row stays failed until the user recreates
the tunnel (`create` revives it to `pending`), which keeps a broken host from burning the Let's
Encrypt failure rate limit every 30s. **Renewal is molohttp's job, not DevPlace's**: once a host is
issued, `AcmeRenewalService` re-issues it against its own expiry threshold forever, so DevPlace never

53
devplacepy/services/containers/workspace/certs.py
View File

@ -2,9 +2,14 @@
from __future__ import annotations
from collections.abc import Callable
from devplacepy.database import get_setting
from devplacepy.services.containers import store
from devplacepy.stealth import stealth_async_client
from . import tunnels
API_PATH = "/api/v1"
ISSUE_TIMEOUT_SECONDS = 180.0
@ -62,3 +67,51 @@ async def issue(hostname: str) -> None:
f"molohttp cert issue failed ({response.status_code}): "
f"{response.text[:200]}"
)
async def certify(row: dict, log: Callable[[str], None] | None = None) -> bool:
hostname = row.get("hostname", "")
tunnels.update(
row["uid"], {"status": tunnels.STATUS_PROVISIONING, "last_error": ""}
)
try:
await issue(hostname)
except Exception as error:
tunnels.update(
row["uid"],
{"status": tunnels.STATUS_FAILED, "last_error": str(error)[:500]},
)
if log:
log(f"tunnel certificate for {hostname} failed: {error}")
return False
tunnels.update(row["uid"], {"status": tunnels.STATUS_ACTIVE, "last_error": ""})
if log:
log(f"tunnel certificate issued for {hostname}")
announce(row, hostname)
return True
def announce(row: dict, hostname: str) -> None:
from devplacepy.utils import create_notification
from . import provision
instance = store.get_instance(row.get("instance_uid", ""))
if not instance:
return
owner_uid = instance.get("workspace_owner_uid", "")
if not owner_uid:
return
url = f"https://{hostname}"
if provision.is_editor_tunnel(instance, row):
password = instance.get("editor_password") or ""
message = f"Your VS Code workspace is live at {url} (password: {password})."
else:
message = f"Tunnel {row.get('label') or hostname} is live at {url}."
create_notification(
owner_uid,
"workspace",
message,
instance["uid"],
f"/projects/{instance.get('project_uid', '')}/workspace",
)

30
devplacepy/services/containers/workspace/provision.py
View File

@ -2,6 +2,7 @@
from __future__ import annotations
import asyncio
import json
from pathlib import Path
@ -14,6 +15,8 @@ from . import flags, naming, quota, tunnels
MANIFEST_DIRECTORY = ".devplace"
MANIFEST_NAME = "tunnels.json"
_pending_certificates: set[asyncio.Task] = set()
class WorkspaceError(Exception):
pass
@ -63,10 +66,28 @@ async def ensure(project: dict, user: dict) -> dict:
)
instance = store.get_instance(instance["uid"])
api.ensure_editor_password(instance)
publish_editor_tunnel(instance, owner_uid)
schedule_certificate(publish_editor_tunnel(instance, owner_uid))
return store.get_instance(instance["uid"])
def schedule_certificate(tunnel: dict | None) -> bool:
from . import certs
if not tunnel or not certs.configured():
return False
try:
loop = asyncio.get_running_loop()
except RuntimeError:
return False
tunnels.update(
tunnel["uid"], {"status": tunnels.STATUS_PROVISIONING, "last_error": ""}
)
task = loop.create_task(certs.certify(tunnel))
_pending_certificates.add(task)
task.add_done_callback(_pending_certificates.discard)
return True
def publish_editor_tunnel(instance: dict, owner_uid: str) -> dict | None:
port = int(instance.get("editor_port") or api.EDITOR_DEFAULT_PORT)
if not port or not instance.get("tunnel_name"):
@ -90,7 +111,12 @@ def resume(instance: dict) -> dict:
)
store.update_instance(
instance["uid"],
{"desired_state": "running", "idle_warned_at": "", "delete_warned_at": ""},
{
"desired_state": "running",
"last_active_at": store.now(),
"idle_warned_at": "",
"delete_warned_at": "",
},
)
return store.get_instance(instance["uid"])

50
devplacepy/services/containers/workspace_service.py
View File

@ -14,7 +14,6 @@ from devplacepy.services.containers import store
from devplacepy.services.containers.workspace import (
certs,
flags,
provision,
quota,
tunnels,
)
@ -176,54 +175,7 @@ class WorkspaceService(BaseService):
if not certs.configured():
return
for row in tunnels.awaiting_certificate():
hostname = row.get("hostname", "")
tunnels.update(
row["uid"],
{"status": tunnels.STATUS_PROVISIONING, "last_error": ""},
)
try:
await certs.issue(hostname)
except Exception as error:
tunnels.update(
row["uid"],
{
"status": tunnels.STATUS_FAILED,
"last_error": str(error)[:500],
},
)
self.log(f"tunnel certificate for {hostname} failed: {error}")
continue
tunnels.update(
row["uid"], {"status": tunnels.STATUS_ACTIVE, "last_error": ""}
)
self.log(f"tunnel certificate issued for {hostname}")
self._announce_tunnel(row, hostname)
def _announce_tunnel(self, row: dict, hostname: str) -> None:
from devplacepy.utils import create_notification
instance = store.get_instance(row.get("instance_uid", ""))
if not instance:
return
owner_uid = instance.get("workspace_owner_uid", "")
if not owner_uid:
return
url = f"https://{hostname}"
if provision.is_editor_tunnel(instance, row):
password = instance.get("editor_password") or ""
message = (
f"Your VS Code workspace is live at {url} "
f"(password: {password})."
)
else:
message = f"Tunnel {row.get('label') or hostname} is live at {url}."
create_notification(
owner_uid,
"workspace",
message,
instance["uid"],
f"/projects/{instance.get('project_uid', '')}/workspace",
)
await certs.certify(row, self.log)
def _sample_disk(self, rows: list[dict], cfg: dict) -> None:
interval = int(cfg.get("workspace_disk_sample_minutes") or

415
tests/api/projects/workspace.py
View File

@ -15,10 +15,12 @@ from devplacepy.services.containers.workspace import (
tunnels,
)
from devplacepy.services.containers.workspace.provision import WorkspaceError
from tests.conftest import run_async
from tests.conftest import BASE_URL, run_async
OWNER = "user-owner"
OTHER = "user-other"
# A workspace route that answers wrongly must fail the test, never wedge the serial suite.
HTTP_TIMEOUT = 30
@pytest.fixture(autouse=True)
@ -216,6 +218,120 @@ def test_devii_workspace_tools_are_role_gated():
assert names <= admin
def _idle_for(minutes: float) -> str:
from datetime import datetime, timedelta, timezone
return (
datetime.now(timezone.utc) - timedelta(minutes=minutes)
).isoformat()
def _lifecycle(rows, stop="auto", delete="auto"):
from devplacepy.services.containers.workspace_service import WorkspaceService
WorkspaceService()._advance_lifecycle(
rows, {"workspace_auto_stop": stop, "workspace_auto_delete": delete}
)
def _reload(uid: str) -> dict:
return store.get_instance(uid)
def test_resuming_restarts_the_idle_clock():
instance = _instance(status="stopped", desired_state="stopped")
store.update_instance(instance["uid"], {"last_active_at": _idle_for(600)})
resumed = provision.resume(_reload(instance["uid"]))
assert resumed["desired_state"] == "running"
running = dict(resumed, status="running")
_lifecycle([running])
assert _reload(instance["uid"])["desired_state"] == "running"
def test_idle_workspace_is_warned_then_stopped():
instance = _instance(status="running", desired_state="running")
store.update_instance(instance["uid"], {"last_active_at": _idle_for(50)})
_lifecycle([_reload(instance["uid"])])
warned = _reload(instance["uid"])
assert warned["idle_warned_at"]
assert warned["desired_state"] == "running"
store.update_instance(instance["uid"], {"last_active_at": _idle_for(90)})
_lifecycle([_reload(instance["uid"])])
stopped = _reload(instance["uid"])
assert stopped["desired_state"] == "stopped"
assert not stopped["idle_warned_at"]
def test_fresh_workspace_is_left_alone():
instance = _instance(status="running", desired_state="running")
store.update_instance(instance["uid"], {"last_active_at": _idle_for(1)})
_lifecycle([_reload(instance["uid"])])
fresh = _reload(instance["uid"])
assert fresh["desired_state"] == "running"
assert not fresh["idle_warned_at"]
def test_auto_stop_off_never_stops_an_idle_workspace():
instance = _instance(status="running", desired_state="running")
store.update_instance(instance["uid"], {"last_active_at": _idle_for(900)})
_lifecycle([_reload(instance["uid"])], stop="off")
assert _reload(instance["uid"])["desired_state"] == "running"
def test_suspended_workspace_is_skipped_entirely():
instance = _instance(status="running", desired_state="running")
store.update_instance(
instance["uid"],
{
"last_active_at": _idle_for(60 * 24 * 400),
"suspended_at": "2026-01-01T00:00:00+00:00",
},
)
_lifecycle([_reload(instance["uid"])])
survivor = _reload(instance["uid"])
assert survivor is not None
assert survivor["desired_state"] == "running"
def test_workspace_without_activity_is_never_touched():
instance = _instance(status="running", desired_state="running")
store.update_instance(instance["uid"], {"last_active_at": ""})
_lifecycle([_reload(instance["uid"])])
assert _reload(instance["uid"])["desired_state"] == "running"
def test_stopped_workspace_past_retention_is_deleted_with_its_tunnels():
instance = _instance(status="stopped", desired_state="stopped")
tunnels.create(instance, "web", 8080, OWNER)
store.update_instance(
instance["uid"], {"last_active_at": _idle_for(60 * 24 * 20)}
)
_lifecycle([_reload(instance["uid"])])
assert _reload(instance["uid"]) is None
for row in tunnels.list_for_instance(instance["uid"]):
assert row["status"] == tunnels.STATUS_SUSPENDED
def test_running_workspace_is_never_deleted_by_retention():
instance = _instance(status="running", desired_state="running")
store.update_instance(
instance["uid"], {"last_active_at": _idle_for(60 * 24 * 400)}
)
_lifecycle([_reload(instance["uid"])], stop="off")
assert _reload(instance["uid"]) is not None
def test_auto_delete_off_keeps_an_expired_workspace():
instance = _instance(status="stopped", desired_state="stopped")
store.update_instance(
instance["uid"], {"last_active_at": _idle_for(60 * 24 * 400)}
)
_lifecycle([_reload(instance["uid"])], delete="off")
assert _reload(instance["uid"]) is not None
def test_opening_a_workspace_publishes_the_editor_tunnel_automatically():
project = _project()
user = {"uid": OWNER, "username": "owner"}
@ -231,13 +347,13 @@ def test_opening_a_workspace_publishes_the_editor_tunnel_automatically():
def test_active_editor_tunnel_notifies_the_owner_with_url_and_password():
from devplacepy.services.containers.workspace_service import WorkspaceService
from devplacepy.services.containers.workspace import certs
project = _project()
instance = run_async(provision.ensure(project, {"uid": OWNER, "username": "o"}))
row = tunnels.list_for_instance(instance["uid"])[0]
try:
WorkspaceService()._announce_tunnel(row, row["hostname"])
certs.announce(row, row["hostname"])
sent = list(
get_table("notifications").find(
user_uid=OWNER, type="workspace", order_by=["-id"]
@ -395,3 +511,296 @@ def test_tunnel_host_detection_never_matches_the_site():
assert naming.is_tunnel_host("abc." + naming.domain())
assert not naming.is_tunnel_host("pravda.education")
assert not naming.is_tunnel_host("")
def _seeded_user(username: str) -> dict:
from devplacepy.database import db, refresh_snapshot
refresh_snapshot()
rows = list(
db.query(
"SELECT uid, username, api_key FROM users WHERE username = :username",
username=username,
)
)
assert rows, f"user {username} was not created by the server"
return rows[0]
def _fresh_member() -> dict:
import time as _time
import requests
name = f"wsmem{int(_time.time() * 1000)}"
requests.post(
f"{BASE_URL}/auth/signup",
data={
"username": name,
"email": f"{name}@t.dev",
"password": "secret123",
"confirm_password": "secret123",
},
allow_redirects=True,
timeout=HTTP_TIMEOUT,
)
return _seeded_user(name)
def _http_project(owner_uid: str, title: str) -> dict:
from devplacepy.utils import generate_uid, make_combined_slug
uid = generate_uid()
row = {
"uid": uid,
"user_uid": owner_uid,
"title": title,
"description": "workspace host project",
"slug": make_combined_slug(title, uid),
"project_type": "software",
"status": "In Development",
"platforms": "",
"is_private": 0,
"read_only": 0,
"stars": 0,
"created_at": "2026-01-01T00:00:00+00:00",
"deleted_at": None,
"deleted_by": None,
}
get_table("projects").insert(row)
_CREATED_PROJECTS.append(uid)
return row
_CREATED_PROJECTS: list = []
@pytest.fixture(autouse=True)
def _drop_created_projects():
yield
while _CREATED_PROJECTS:
get_table("projects").delete(uid=_CREATED_PROJECTS.pop())
def _await_workspaces_enabled(slug: str, key: str) -> bool:
import time as _time
import requests
deadline = _time.time() + 10.0
headers = {"Accept": "application/json", "X-API-KEY": key}
while _time.time() < deadline:
body = requests.get(
f"{BASE_URL}/projects/{slug}", headers=headers, timeout=HTTP_TIMEOUT
).json()
if body.get("viewer_can_workspace"):
return True
_time.sleep(0.2)
return False
def _post(path: str, key: str, **kwargs):
import requests
return requests.post(
f"{BASE_URL}{path}",
headers={"Accept": "application/json", "X-API-KEY": key},
timeout=HTTP_TIMEOUT,
**kwargs,
)
def test_opening_a_workspace_over_quota_answers_400_not_500(app_server, seeded_db):
owner = _seeded_user("bob_test")
limit = quota.resolve(owner["uid"]).max_workspaces
for index in range(limit):
_instance(
project_uid=f"proj-quota-{index}",
name=f"ws-quota-{index}",
workspace_owner_uid=owner["uid"],
)
project = _http_project(owner["uid"], "WS Quota Http")
assert _await_workspaces_enabled(project["slug"], owner["api_key"])
response = _post(f"/projects/{project['slug']}/workspace", owner["api_key"])
assert response.status_code == 400, response.text[:400]
assert "limit" in response.json()["error"]["message"]
assert provision.count_for_owner(owner["uid"]) == limit
def test_opening_a_workspace_under_quota_is_not_refused(app_server, seeded_db):
owner = _seeded_user("bob_test")
project = _http_project(owner["uid"], "WS Room Http")
assert _await_workspaces_enabled(project["slug"], owner["api_key"])
response = _post(f"/projects/{project['slug']}/workspace", owner["api_key"])
assert response.status_code != 500, response.text[:400]
assert response.status_code != 400 or "limit" not in response.json()["error"]["message"]
def test_tunnel_creation_rejects_a_zero_port_with_400(app_server, seeded_db):
owner = _seeded_user("bob_test")
project = _http_project(owner["uid"], "WS Port Http")
_instance(project_uid=project["uid"], workspace_owner_uid=owner["uid"])
assert _await_workspaces_enabled(project["slug"], owner["api_key"])
response = _post(
f"/projects/{project['slug']}/workspace/tunnels",
owner["api_key"],
data={"label": "web", "container_port": "0"},
)
assert response.status_code == 400, response.text[:400]
assert "container_port" in response.json()["error"]["message"]
def test_tunnel_limit_answers_400(app_server, seeded_db):
owner = _seeded_user("bob_test")
project = _http_project(owner["uid"], "WS Tunnel Cap Http")
instance = _instance(project_uid=project["uid"], workspace_owner_uid=owner["uid"])
limit = quota.resolve(owner["uid"], instance).max_tunnels
for port in range(9000, 9000 + limit):
tunnels.create(instance, f"t{port}", port, owner["uid"])
assert _await_workspaces_enabled(project["slug"], owner["api_key"])
response = _post(
f"/projects/{project['slug']}/workspace/tunnels",
owner["api_key"],
data={"label": "over", "container_port": "9999"},
)
assert response.status_code == 400, response.text[:400]
assert "tunnel limit reached" in response.json()["error"]["message"]
assert tunnels.count_for_instance(instance["uid"]) == limit
def test_editor_proxy_denies_a_stranger_with_403(app_server, seeded_db):
import requests
owner = _seeded_user("bob_test")
stranger = _fresh_member()
project = _http_project(owner["uid"], "WS Editor Http")
instance = _instance(project_uid=project["uid"], workspace_owner_uid=owner["uid"])
response = requests.get(
f"{BASE_URL}/projects/{project['slug']}/containers/instances/{instance['uid']}/code/",
headers={"Accept": "application/json", "X-API-KEY": stranger["api_key"]},
timeout=HTTP_TIMEOUT,
)
assert response.status_code == 403, response.text[:400]
def test_admin_suspend_without_a_reason_answers_400(app_server, seeded_db):
admin = _seeded_user("alice_test")
owner = _seeded_user("bob_test")
project = _http_project(owner["uid"], "WS Suspend Http")
instance = _instance(project_uid=project["uid"], workspace_owner_uid=owner["uid"])
response = _post(
f"/admin/workspaces/{instance['uid']}/suspend",
admin["api_key"],
data={"reason": " "},
)
assert response.status_code == 400, response.text[:400]
assert "reason" in response.json()["error"]["message"]
assert not store.get_instance(instance["uid"])["suspended_at"]
def test_admin_quota_rule_without_an_owner_answers_400(app_server, seeded_db):
admin = _seeded_user("alice_test")
response = _post(
"/admin/workspaces/quota", admin["api_key"], data={"owner_id": ""}
)
assert response.status_code == 400, response.text[:400]
assert "owner_id" in response.json()["error"]["message"]
def _certified(project: dict, monkeypatch, issue) -> dict:
from devplacepy.services.containers.workspace import certs
monkeypatch.setattr(certs, "configured", lambda: True)
monkeypatch.setattr(certs, "issue", issue)
async def _open_and_drain() -> dict:
import asyncio
instance = await provision.ensure(project, {"uid": OWNER, "username": "o"})
pending = list(provision._pending_certificates)
if pending:
await asyncio.gather(*pending, return_exceptions=True)
return instance
return run_async(_open_and_drain())
def test_opening_a_workspace_certifies_its_editor_tunnel_without_waiting_for_a_tick(
monkeypatch,
):
ordered = []
async def _ok(hostname: str) -> None:
ordered.append(hostname)
instance = _certified(_project(), monkeypatch, _ok)
row = tunnels.list_for_instance(instance["uid"])[0]
assert ordered == [row["hostname"]], "the editor certificate was not ordered eagerly"
assert row["status"] == tunnels.STATUS_ACTIVE
assert not tunnels.awaiting_certificate()
def test_eagerly_certified_editor_tunnel_still_announces_url_and_password(monkeypatch):
async def _ok(hostname: str) -> None:
return None
instance = _certified(_project(), monkeypatch, _ok)
row = tunnels.list_for_instance(instance["uid"])[0]
try:
sent = list(
get_table("notifications").find(
user_uid=OWNER, type="workspace", order_by=["-id"]
)
)
assert sent, "owner was not notified after eager issuance"
assert f"https://{row['hostname']}" in sent[0]["message"]
assert instance["editor_password"] in sent[0]["message"]
finally:
get_table("notifications").delete(user_uid=OWNER)
def test_an_eager_certificate_failure_leaves_the_tunnel_failed_not_pending(monkeypatch):
from devplacepy.services.containers.workspace import certs
async def _boom(hostname: str) -> None:
raise certs.CertError("molohttp cert issue failed (503): upstream down")
instance = _certified(_project(), monkeypatch, _boom)
row = tunnels.list_for_instance(instance["uid"])[0]
assert row["status"] == tunnels.STATUS_FAILED
assert "upstream down" in row["last_error"]
assert not tunnels.awaiting_certificate()
def test_no_certificate_is_ordered_when_molohttp_is_unconfigured(monkeypatch):
from devplacepy.services.containers.workspace import certs
ordered = []
async def _ok(hostname: str) -> None:
ordered.append(hostname)
monkeypatch.setattr(certs, "configured", lambda: False)
monkeypatch.setattr(certs, "issue", _ok)
instance = run_async(provision.ensure(_project(), {"uid": OWNER, "username": "o"}))
assert ordered == []
row = tunnels.list_for_instance(instance["uid"])[0]
assert row["status"] == tunnels.STATUS_PENDING
assert [r["uid"] for r in tunnels.awaiting_certificate()] == [row["uid"]]