Order the workspace editor certificate at creation

A new tunnel row was inert at pending until the next WorkspaceService
tick picked it up, so the editor certificate waited up to 30 seconds
before the ACME order even started. Opening a workspace and reaching its
public hostname in that window served the wildcard default certificate
and produced ERR_CERT_COMMON_NAME_INVALID in the browser.

certs.certify now owns the whole transition - provisioning, issue,
active or failed, then the owner notification - and both callers use it.
provision.ensure schedules it for the editor tunnel the moment the
workspace exists, claiming the row synchronously so the tick can never
issue the same host twice and burn a duplicate-certificate slot. The
service loop keeps calling certify per pending row as the safety net for
user-created tunnels and for rows created while molohttp was
unconfigured.

Issuance is a loop task rather than an await: an order takes about ten
seconds and certs.issue allows up to 180, which would hang the workspace
POST. The task is held until it completes so it cannot be collected mid
flight.

resume also restarts the idle clock, and the workspace routes gain
HTTP-level tests covering every refusal path they had none for.
This commit is contained in:
2026-08-08 13:52:02 +02:00
parent 56becfb3f7
commit 372067bbe4
5 changed files with 528 additions and 60 deletions
+34 -6
View File
@@ -252,12 +252,40 @@ wildcard over `http-01`. So a `*.tunnel.pravda.education` **certificate** is imp
site describes **routing**, which is correct and already in place (one enabled molohttp site
`*.tunnel.pravda.education -> http://127.0.0.1:10500`, no per-tunnel site object). Only the cert is
per-host. `tunnels.create` writes a row at `STATUS_PENDING` and nothing else; the row is inert until
`_issue_tunnel_certificates` picks it up - `pending` is not in `SERVING_STATUSES`, so an
un-provisioned tunnel 404s. The phase reads `tunnels.awaiting_certificate()` (`status=pending` +
`desired_state=present` + not deleted), flips the row to `provisioning`, calls
`certs.issue(hostname)` (`workspace/certs.py`, `POST {workspace_molohttp_base_url}/api/v1/certs/issue`
with the `x-api-key` header, Basic as fallback), and lands on `active` or on `failed` with
`last_error`. **Only `pending` is retried** - a `failed` row stays failed until the user recreates
something certifies it - `pending` is not in `SERVING_STATUSES`, so an un-provisioned tunnel 404s.
**One state machine, two callers.** `certs.certify(row, log=None)` owns the whole transition:
`provisioning` -> `certs.issue(hostname)` (`POST {workspace_molohttp_base_url}/api/v1/certs/issue`
with the `x-api-key` header, Basic as fallback) -> `active` or `failed` with `last_error`, then
`certs.announce(row, hostname)` sends the owner the live URL (plus the password for the editor
tunnel). Never re-implement those transitions at a call site - both paths call `certify`:
- **Eagerly, at workspace creation.** `provision.ensure` ends with
`schedule_certificate(publish_editor_tunnel(...))`, which claims the row as `provisioning`
**synchronously** and then spawns `certify` as a loop task, so the editor certificate is ordered
the instant the workspace exists rather than up to one service tick later. It is fire-and-forget
by necessity: an ACME `http-01` order takes ~10s and `certs.issue` allows up to 180s, so awaiting
it would hang the `POST /projects/{slug}/workspace` response. The task is held in
`provision._pending_certificates` and discarded on completion - a bare `create_task` reference can
be garbage collected mid-flight. The synchronous claim is what closes the race with the tick: the
service selects only `pending`, so a row already claimed can never be issued twice and burn a
Let's Encrypt duplicate-certificate slot. `schedule_certificate` no-ops (leaving the row `pending`
for the tick) when molohttp is unconfigured or there is no running loop.
- **On the tick, as the safety net.** `WorkspaceService._issue_tunnel_certificates` reads
`tunnels.awaiting_certificate()` (`status=pending` + `desired_state=present` + not deleted) and
calls `certify` per row. This covers user-created tunnels, rows predating the eager path, and any
creation that happened with molohttp unconfigured.
The residual cost is the ACME round trip itself, which is irreducible under `http-01`. The tick
interval (`service_workspace_interval`, default 30s, floor `min_interval = 5`) no longer sits in
front of the editor tunnel, but still bounds how long a user-created tunnel waits.
**A tunnel serves before its certificate exists.** `SERVING_STATUSES = (provisioning, active)`, so
`routers/tunnel.py` routes the host the moment the row is claimed, while the order is still in
flight. The wildcard DNS already resolves, so molohttp terminates TLS with its default certificate
and the browser shows `ERR_CERT_COMMON_NAME_INVALID` for that window rather than failing to connect.
That is deliberate (a reachable host beats a dead one) and is the reason eager issuance matters: the
window is only as long as issuance takes. **Only `pending` is retried** - a `failed` row stays failed until the user recreates
the tunnel (`create` revives it to `pending`), which keeps a broken host from burning the Let's
Encrypt failure rate limit every 30s. **Renewal is molohttp's job, not DevPlace's**: once a host is
issued, `AcmeRenewalService` re-issues it against its own expiry threshold forever, so DevPlace never
@@ -2,9 +2,14 @@
from __future__ import annotations
from collections.abc import Callable
from devplacepy.database import get_setting
from devplacepy.services.containers import store
from devplacepy.stealth import stealth_async_client
from . import tunnels
API_PATH = "/api/v1"
ISSUE_TIMEOUT_SECONDS = 180.0
@@ -62,3 +67,51 @@ async def issue(hostname: str) -> None:
f"molohttp cert issue failed ({response.status_code}): "
f"{response.text[:200]}"
)
async def certify(row: dict, log: Callable[[str], None] | None = None) -> bool:
hostname = row.get("hostname", "")
tunnels.update(
row["uid"], {"status": tunnels.STATUS_PROVISIONING, "last_error": ""}
)
try:
await issue(hostname)
except Exception as error:
tunnels.update(
row["uid"],
{"status": tunnels.STATUS_FAILED, "last_error": str(error)[:500]},
)
if log:
log(f"tunnel certificate for {hostname} failed: {error}")
return False
tunnels.update(row["uid"], {"status": tunnels.STATUS_ACTIVE, "last_error": ""})
if log:
log(f"tunnel certificate issued for {hostname}")
announce(row, hostname)
return True
def announce(row: dict, hostname: str) -> None:
from devplacepy.utils import create_notification
from . import provision
instance = store.get_instance(row.get("instance_uid", ""))
if not instance:
return
owner_uid = instance.get("workspace_owner_uid", "")
if not owner_uid:
return
url = f"https://{hostname}"
if provision.is_editor_tunnel(instance, row):
password = instance.get("editor_password") or ""
message = f"Your VS Code workspace is live at {url} (password: {password})."
else:
message = f"Tunnel {row.get('label') or hostname} is live at {url}."
create_notification(
owner_uid,
"workspace",
message,
instance["uid"],
f"/projects/{instance.get('project_uid', '')}/workspace",
)
@@ -2,6 +2,7 @@
from __future__ import annotations
import asyncio
import json
from pathlib import Path
@@ -14,6 +15,8 @@ from . import flags, naming, quota, tunnels
MANIFEST_DIRECTORY = ".devplace"
MANIFEST_NAME = "tunnels.json"
_pending_certificates: set[asyncio.Task] = set()
class WorkspaceError(Exception):
pass
@@ -63,10 +66,28 @@ async def ensure(project: dict, user: dict) -> dict:
)
instance = store.get_instance(instance["uid"])
api.ensure_editor_password(instance)
publish_editor_tunnel(instance, owner_uid)
schedule_certificate(publish_editor_tunnel(instance, owner_uid))
return store.get_instance(instance["uid"])
def schedule_certificate(tunnel: dict | None) -> bool:
from . import certs
if not tunnel or not certs.configured():
return False
try:
loop = asyncio.get_running_loop()
except RuntimeError:
return False
tunnels.update(
tunnel["uid"], {"status": tunnels.STATUS_PROVISIONING, "last_error": ""}
)
task = loop.create_task(certs.certify(tunnel))
_pending_certificates.add(task)
task.add_done_callback(_pending_certificates.discard)
return True
def publish_editor_tunnel(instance: dict, owner_uid: str) -> dict | None:
port = int(instance.get("editor_port") or api.EDITOR_DEFAULT_PORT)
if not port or not instance.get("tunnel_name"):
@@ -90,7 +111,12 @@ def resume(instance: dict) -> dict:
)
store.update_instance(
instance["uid"],
{"desired_state": "running", "idle_warned_at": "", "delete_warned_at": ""},
{
"desired_state": "running",
"last_active_at": store.now(),
"idle_warned_at": "",
"delete_warned_at": "",
},
)
return store.get_instance(instance["uid"])
@@ -14,7 +14,6 @@ from devplacepy.services.containers import store
from devplacepy.services.containers.workspace import (
certs,
flags,
provision,
quota,
tunnels,
)
@@ -176,54 +175,7 @@ class WorkspaceService(BaseService):
if not certs.configured():
return
for row in tunnels.awaiting_certificate():
hostname = row.get("hostname", "")
tunnels.update(
row["uid"],
{"status": tunnels.STATUS_PROVISIONING, "last_error": ""},
)
try:
await certs.issue(hostname)
except Exception as error:
tunnels.update(
row["uid"],
{
"status": tunnels.STATUS_FAILED,
"last_error": str(error)[:500],
},
)
self.log(f"tunnel certificate for {hostname} failed: {error}")
continue
tunnels.update(
row["uid"], {"status": tunnels.STATUS_ACTIVE, "last_error": ""}
)
self.log(f"tunnel certificate issued for {hostname}")
self._announce_tunnel(row, hostname)
def _announce_tunnel(self, row: dict, hostname: str) -> None:
from devplacepy.utils import create_notification
instance = store.get_instance(row.get("instance_uid", ""))
if not instance:
return
owner_uid = instance.get("workspace_owner_uid", "")
if not owner_uid:
return
url = f"https://{hostname}"
if provision.is_editor_tunnel(instance, row):
password = instance.get("editor_password") or ""
message = (
f"Your VS Code workspace is live at {url} "
f"(password: {password})."
)
else:
message = f"Tunnel {row.get('label') or hostname} is live at {url}."
create_notification(
owner_uid,
"workspace",
message,
instance["uid"],
f"/projects/{instance.get('project_uid', '')}/workspace",
)
await certs.certify(row, self.log)
def _sample_disk(self, rows: list[dict], cfg: dict) -> None:
interval = int(cfg.get("workspace_disk_sample_minutes") or