Order the workspace editor certificate at creation
A new tunnel row was inert at pending until the next WorkspaceService tick picked it up, so the editor certificate waited up to 30 seconds before the ACME order even started. Opening a workspace and reaching its public hostname in that window served the wildcard default certificate and produced ERR_CERT_COMMON_NAME_INVALID in the browser. certs.certify now owns the whole transition - provisioning, issue, active or failed, then the owner notification - and both callers use it. provision.ensure schedules it for the editor tunnel the moment the workspace exists, claiming the row synchronously so the tick can never issue the same host twice and burn a duplicate-certificate slot. The service loop keeps calling certify per pending row as the safety net for user-created tunnels and for rows created while molohttp was unconfigured. Issuance is a loop task rather than an await: an order takes about ten seconds and certs.issue allows up to 180, which would hang the workspace POST. The task is held until it completes so it cannot be collected mid flight. resume also restarts the idle clock, and the workspace routes gain HTTP-level tests covering every refusal path they had none for.
This commit is contained in:
@@ -252,12 +252,40 @@ wildcard over `http-01`. So a `*.tunnel.pravda.education` **certificate** is imp
|
||||
site describes **routing**, which is correct and already in place (one enabled molohttp site
|
||||
`*.tunnel.pravda.education -> http://127.0.0.1:10500`, no per-tunnel site object). Only the cert is
|
||||
per-host. `tunnels.create` writes a row at `STATUS_PENDING` and nothing else; the row is inert until
|
||||
`_issue_tunnel_certificates` picks it up - `pending` is not in `SERVING_STATUSES`, so an
|
||||
un-provisioned tunnel 404s. The phase reads `tunnels.awaiting_certificate()` (`status=pending` +
|
||||
`desired_state=present` + not deleted), flips the row to `provisioning`, calls
|
||||
`certs.issue(hostname)` (`workspace/certs.py`, `POST {workspace_molohttp_base_url}/api/v1/certs/issue`
|
||||
with the `x-api-key` header, Basic as fallback), and lands on `active` or on `failed` with
|
||||
`last_error`. **Only `pending` is retried** - a `failed` row stays failed until the user recreates
|
||||
something certifies it - `pending` is not in `SERVING_STATUSES`, so an un-provisioned tunnel 404s.
|
||||
|
||||
**One state machine, two callers.** `certs.certify(row, log=None)` owns the whole transition:
|
||||
`provisioning` -> `certs.issue(hostname)` (`POST {workspace_molohttp_base_url}/api/v1/certs/issue`
|
||||
with the `x-api-key` header, Basic as fallback) -> `active` or `failed` with `last_error`, then
|
||||
`certs.announce(row, hostname)` sends the owner the live URL (plus the password for the editor
|
||||
tunnel). Never re-implement those transitions at a call site - both paths call `certify`:
|
||||
|
||||
- **Eagerly, at workspace creation.** `provision.ensure` ends with
|
||||
`schedule_certificate(publish_editor_tunnel(...))`, which claims the row as `provisioning`
|
||||
**synchronously** and then spawns `certify` as a loop task, so the editor certificate is ordered
|
||||
the instant the workspace exists rather than up to one service tick later. It is fire-and-forget
|
||||
by necessity: an ACME `http-01` order takes ~10s and `certs.issue` allows up to 180s, so awaiting
|
||||
it would hang the `POST /projects/{slug}/workspace` response. The task is held in
|
||||
`provision._pending_certificates` and discarded on completion - a bare `create_task` reference can
|
||||
be garbage collected mid-flight. The synchronous claim is what closes the race with the tick: the
|
||||
service selects only `pending`, so a row already claimed can never be issued twice and burn a
|
||||
Let's Encrypt duplicate-certificate slot. `schedule_certificate` no-ops (leaving the row `pending`
|
||||
for the tick) when molohttp is unconfigured or there is no running loop.
|
||||
- **On the tick, as the safety net.** `WorkspaceService._issue_tunnel_certificates` reads
|
||||
`tunnels.awaiting_certificate()` (`status=pending` + `desired_state=present` + not deleted) and
|
||||
calls `certify` per row. This covers user-created tunnels, rows predating the eager path, and any
|
||||
creation that happened with molohttp unconfigured.
|
||||
|
||||
The residual cost is the ACME round trip itself, which is irreducible under `http-01`. The tick
|
||||
interval (`service_workspace_interval`, default 30s, floor `min_interval = 5`) no longer sits in
|
||||
front of the editor tunnel, but still bounds how long a user-created tunnel waits.
|
||||
|
||||
**A tunnel serves before its certificate exists.** `SERVING_STATUSES = (provisioning, active)`, so
|
||||
`routers/tunnel.py` routes the host the moment the row is claimed, while the order is still in
|
||||
flight. The wildcard DNS already resolves, so molohttp terminates TLS with its default certificate
|
||||
and the browser shows `ERR_CERT_COMMON_NAME_INVALID` for that window rather than failing to connect.
|
||||
That is deliberate (a reachable host beats a dead one) and is the reason eager issuance matters: the
|
||||
window is only as long as issuance takes. **Only `pending` is retried** - a `failed` row stays failed until the user recreates
|
||||
the tunnel (`create` revives it to `pending`), which keeps a broken host from burning the Let's
|
||||
Encrypt failure rate limit every 30s. **Renewal is molohttp's job, not DevPlace's**: once a host is
|
||||
issued, `AcmeRenewalService` re-issues it against its own expiry threshold forever, so DevPlace never
|
||||
|
||||
@@ -2,9 +2,14 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Callable
|
||||
|
||||
from devplacepy.database import get_setting
|
||||
from devplacepy.services.containers import store
|
||||
from devplacepy.stealth import stealth_async_client
|
||||
|
||||
from . import tunnels
|
||||
|
||||
API_PATH = "/api/v1"
|
||||
ISSUE_TIMEOUT_SECONDS = 180.0
|
||||
|
||||
@@ -62,3 +67,51 @@ async def issue(hostname: str) -> None:
|
||||
f"molohttp cert issue failed ({response.status_code}): "
|
||||
f"{response.text[:200]}"
|
||||
)
|
||||
|
||||
|
||||
async def certify(row: dict, log: Callable[[str], None] | None = None) -> bool:
|
||||
hostname = row.get("hostname", "")
|
||||
tunnels.update(
|
||||
row["uid"], {"status": tunnels.STATUS_PROVISIONING, "last_error": ""}
|
||||
)
|
||||
try:
|
||||
await issue(hostname)
|
||||
except Exception as error:
|
||||
tunnels.update(
|
||||
row["uid"],
|
||||
{"status": tunnels.STATUS_FAILED, "last_error": str(error)[:500]},
|
||||
)
|
||||
if log:
|
||||
log(f"tunnel certificate for {hostname} failed: {error}")
|
||||
return False
|
||||
tunnels.update(row["uid"], {"status": tunnels.STATUS_ACTIVE, "last_error": ""})
|
||||
if log:
|
||||
log(f"tunnel certificate issued for {hostname}")
|
||||
announce(row, hostname)
|
||||
return True
|
||||
|
||||
|
||||
def announce(row: dict, hostname: str) -> None:
|
||||
from devplacepy.utils import create_notification
|
||||
|
||||
from . import provision
|
||||
|
||||
instance = store.get_instance(row.get("instance_uid", ""))
|
||||
if not instance:
|
||||
return
|
||||
owner_uid = instance.get("workspace_owner_uid", "")
|
||||
if not owner_uid:
|
||||
return
|
||||
url = f"https://{hostname}"
|
||||
if provision.is_editor_tunnel(instance, row):
|
||||
password = instance.get("editor_password") or ""
|
||||
message = f"Your VS Code workspace is live at {url} (password: {password})."
|
||||
else:
|
||||
message = f"Tunnel {row.get('label') or hostname} is live at {url}."
|
||||
create_notification(
|
||||
owner_uid,
|
||||
"workspace",
|
||||
message,
|
||||
instance["uid"],
|
||||
f"/projects/{instance.get('project_uid', '')}/workspace",
|
||||
)
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
@@ -14,6 +15,8 @@ from . import flags, naming, quota, tunnels
|
||||
MANIFEST_DIRECTORY = ".devplace"
|
||||
MANIFEST_NAME = "tunnels.json"
|
||||
|
||||
_pending_certificates: set[asyncio.Task] = set()
|
||||
|
||||
|
||||
class WorkspaceError(Exception):
|
||||
pass
|
||||
@@ -63,10 +66,28 @@ async def ensure(project: dict, user: dict) -> dict:
|
||||
)
|
||||
instance = store.get_instance(instance["uid"])
|
||||
api.ensure_editor_password(instance)
|
||||
publish_editor_tunnel(instance, owner_uid)
|
||||
schedule_certificate(publish_editor_tunnel(instance, owner_uid))
|
||||
return store.get_instance(instance["uid"])
|
||||
|
||||
|
||||
def schedule_certificate(tunnel: dict | None) -> bool:
|
||||
from . import certs
|
||||
|
||||
if not tunnel or not certs.configured():
|
||||
return False
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
except RuntimeError:
|
||||
return False
|
||||
tunnels.update(
|
||||
tunnel["uid"], {"status": tunnels.STATUS_PROVISIONING, "last_error": ""}
|
||||
)
|
||||
task = loop.create_task(certs.certify(tunnel))
|
||||
_pending_certificates.add(task)
|
||||
task.add_done_callback(_pending_certificates.discard)
|
||||
return True
|
||||
|
||||
|
||||
def publish_editor_tunnel(instance: dict, owner_uid: str) -> dict | None:
|
||||
port = int(instance.get("editor_port") or api.EDITOR_DEFAULT_PORT)
|
||||
if not port or not instance.get("tunnel_name"):
|
||||
@@ -90,7 +111,12 @@ def resume(instance: dict) -> dict:
|
||||
)
|
||||
store.update_instance(
|
||||
instance["uid"],
|
||||
{"desired_state": "running", "idle_warned_at": "", "delete_warned_at": ""},
|
||||
{
|
||||
"desired_state": "running",
|
||||
"last_active_at": store.now(),
|
||||
"idle_warned_at": "",
|
||||
"delete_warned_at": "",
|
||||
},
|
||||
)
|
||||
return store.get_instance(instance["uid"])
|
||||
|
||||
|
||||
@@ -14,7 +14,6 @@ from devplacepy.services.containers import store
|
||||
from devplacepy.services.containers.workspace import (
|
||||
certs,
|
||||
flags,
|
||||
provision,
|
||||
quota,
|
||||
tunnels,
|
||||
)
|
||||
@@ -176,54 +175,7 @@ class WorkspaceService(BaseService):
|
||||
if not certs.configured():
|
||||
return
|
||||
for row in tunnels.awaiting_certificate():
|
||||
hostname = row.get("hostname", "")
|
||||
tunnels.update(
|
||||
row["uid"],
|
||||
{"status": tunnels.STATUS_PROVISIONING, "last_error": ""},
|
||||
)
|
||||
try:
|
||||
await certs.issue(hostname)
|
||||
except Exception as error:
|
||||
tunnels.update(
|
||||
row["uid"],
|
||||
{
|
||||
"status": tunnels.STATUS_FAILED,
|
||||
"last_error": str(error)[:500],
|
||||
},
|
||||
)
|
||||
self.log(f"tunnel certificate for {hostname} failed: {error}")
|
||||
continue
|
||||
tunnels.update(
|
||||
row["uid"], {"status": tunnels.STATUS_ACTIVE, "last_error": ""}
|
||||
)
|
||||
self.log(f"tunnel certificate issued for {hostname}")
|
||||
self._announce_tunnel(row, hostname)
|
||||
|
||||
def _announce_tunnel(self, row: dict, hostname: str) -> None:
|
||||
from devplacepy.utils import create_notification
|
||||
|
||||
instance = store.get_instance(row.get("instance_uid", ""))
|
||||
if not instance:
|
||||
return
|
||||
owner_uid = instance.get("workspace_owner_uid", "")
|
||||
if not owner_uid:
|
||||
return
|
||||
url = f"https://{hostname}"
|
||||
if provision.is_editor_tunnel(instance, row):
|
||||
password = instance.get("editor_password") or ""
|
||||
message = (
|
||||
f"Your VS Code workspace is live at {url} "
|
||||
f"(password: {password})."
|
||||
)
|
||||
else:
|
||||
message = f"Tunnel {row.get('label') or hostname} is live at {url}."
|
||||
create_notification(
|
||||
owner_uid,
|
||||
"workspace",
|
||||
message,
|
||||
instance["uid"],
|
||||
f"/projects/{instance.get('project_uid', '')}/workspace",
|
||||
)
|
||||
await certs.certify(row, self.log)
|
||||
|
||||
def _sample_disk(self, rows: list[dict], cfg: dict) -> None:
|
||||
interval = int(cfg.get("workspace_disk_sample_minutes") or
|
||||
|
||||
Reference in New Issue
Block a user