forked from retoor/devplacepy
Inline YouTube Music embeds with one shared embed spec
YouTube Music has no player of its own and music.youtube.com sends X-Frame-Options SAMEORIGIN, so music links render YouTube's own player with an Open in YouTube Music caption. Tracks, playlists and OLAK5uy albums embed; browse, channel, search and library stay plain links. - embeds.py is the single URL spec (exact host allowlists, query-order independent, validated ids only); embed-spec.js is generated from it by the embeds-sync CLI command and drift is tested - rendering.py and ContentRenderer.js both follow it; the old duplicate YouTube regexes are removed - max 3 music embeds per content item, overflow stays a plain link - fixes evilyoutube.com matching and watch?feature=share&v=ID links - fixes the invalid unicode CSS escape that printed u2197 after every external link in rendered content - docs, privacy third-party row, parser table tests, server/client parity Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -105,6 +105,7 @@ devplace containers prune # reap orphan containers + dangling images
|
||||
devplace containers prune-builds # remove legacy per-project images + clear dockerfiles/builds tables (one-time)
|
||||
devplace containers gc-workspaces # remove workspace dirs with no instances
|
||||
devplace emoji-sync # regenerate static/js/emoji-shortcodes.js from the emoji library (run after an emoji dep bump)
|
||||
devplace embeds-sync # regenerate static/js/embed-spec.js from devplacepy/embeds.py (run after editing the embed table)
|
||||
devplace migrate-data # relocate legacy runtime files into data/ (idempotent; --dry-run to preview)
|
||||
```
|
||||
|
||||
@@ -240,6 +241,8 @@ The CLIENT pipeline (`ContentRenderer.js`, `dp-content`/`dp-title`) is retained
|
||||
|
||||
**Emoji shortcodes are the full GitHub/Discord `:name:` set** (~4869 names), generated once from the `emoji` library by `rendering.py` `build_emoji_shortcodes()`; the frontend gets the identical map via the generated `static/js/emoji-shortcodes.js` (regenerate with `devplace emoji-sync` after bumping the dependency, never hand-edit).
|
||||
|
||||
**YouTube and YouTube Music embeds** have ONE specification, `devplacepy/embeds.py` (`EmbedParser.parse(url)` -> `Embed(kind, video_id, list_id, start, music)`, `EmbedParser.src(embed)`): exact host allowlists (never substring matching), query-order independent parsing, validated ids only (`[A-Za-z0-9_-]{11}` video, `{13,64}` list) so nothing but validated characters can reach an iframe `src`. YouTube Music has no player of its own and `music.youtube.com` sends `X-Frame-Options: SAMEORIGIN`, so a music link is YouTube's player (`www.youtube.com/embed/...`) wrapped in `<div class="embed-music">` with an "Open in YouTube Music" caption link; playlists/albums use `embed/videoseries?list=`. `/browse`, `/channel`, `/search`, `/library` carry no embeddable id and stay plain links. At most `MAX_MUSIC_EMBEDS` (3) music embeds render per content item, the overflow stays a plain link; plain YouTube is uncapped. `rendering.py` (`_youtube_embed`, per-`_MediaProcessor` `_EmbedBudget`) and `ContentRenderer.js` (`youtubeEmbed`, via `static/js/EmbedParser.js`) both follow it; the JS constants come from the generated `static/js/embed-spec.js` (`devplace embeds-sync`, never hand-edit). Never add another YouTube regex.
|
||||
|
||||
**Email anonymization:** both `_render_content` and `_render_title` mask email addresses to prevent doxing. `_mask_emails` in `rendering.py` runs on rendered text nodes only (inside `_transform_text`, `_MediaProcessor`, and `_InlineFilter` - never before mistune, or the `*` mask characters would be parsed as emphasis) and stars ~80% of the local part (keeps a leading 20%, minimum one visible char). Addresses on `molodetz.nl` (and its subdomains) are exempt and render verbatim.
|
||||
|
||||
**Em-dash normalization:** `_normalize_dashes` in `rendering.py` replaces all forms (em dash `\u2014`, en dash `\u2013`, and their HTML entities) with a hyphen BEFORE mistune processes the text. Runs inside `_render_content`/`_render_title` which are `@lru_cache`d, so each unique text is normalized once. No per-template overhead.
|
||||
|
||||
@@ -1188,6 +1188,19 @@ Two background services bridge persisted state onto the bus so the interface upd
|
||||
- Session-scoped browser context with per-test cookie clearing
|
||||
- `page.locator(...).wait_for(state="visible")` preferred over bare selectors
|
||||
|
||||
## Media embeds (YouTube and YouTube Music)
|
||||
|
||||
A bare YouTube or YouTube Music link in a post, comment, message or description renders as an inline player. YouTube Music has no player of its own: its track and playlist ids are YouTube's, so the embed is YouTube's own player, shown with an "Open in YouTube Music" caption that links back to the original.
|
||||
|
||||
| Link | Result |
|
||||
|------|--------|
|
||||
| `youtube.com/watch?v=ID`, `youtu.be/ID`, `/embed/ID`, `/v/ID`, `/shorts/ID`, `m.` and `-nocookie` hosts | Video player; `t=` / `start=` seconds are kept |
|
||||
| `music.youtube.com/watch?v=ID[&list=ID]` | Track player with the caption, queue kept when `list=` is a valid playlist id |
|
||||
| `music.youtube.com/playlist?list=ID`, `youtube.com/playlist?list=ID` | Playlist or album player (`OLAK5uy...` album ids work) |
|
||||
| `music.youtube.com/browse/...`, `/channel/...`, `/search`, `/library` | Plain link: these URLs carry no playlist id |
|
||||
|
||||
Parsing is by exact host allowlist and query parameter, in any order, so look-alike hosts such as `evilyoutube.com` never embed. A single post renders at most 3 music embeds; further music links stay plain links. The URL rules live in one place, `devplacepy/embeds.py`; the browser gets them through the generated `static/js/embed-spec.js` (`devplace embeds-sync` after changing the table, never hand-edit) and `static/js/EmbedParser.js`. A unit test fails when the generated file drifts and an e2e test runs one shared case table through both parsers.
|
||||
|
||||
## Avatars
|
||||
|
||||
Uses [Multiavatar](https://github.com/multiavatar/multiavatar-python) - generates deterministic SVG avatars locally from a seed string (the username). No external API calls, no network dependency. Generation takes <5ms. Results are cached in-memory (cleared on server restart).
|
||||
|
||||
@@ -44,7 +44,7 @@ from devplacepy.cli.containers import (
|
||||
cmd_containers_gc_workspaces,
|
||||
)
|
||||
from devplacepy.cli.quiz import cmd_quiz_prune
|
||||
from devplacepy.cli.migrate import cmd_emoji_sync, cmd_migrate_data
|
||||
from devplacepy.cli.migrate import cmd_embeds_sync, cmd_emoji_sync, cmd_migrate_data
|
||||
from devplacepy.cli.system import cmd_system_prune
|
||||
|
||||
__all__ = [
|
||||
@@ -90,6 +90,7 @@ __all__ = [
|
||||
"cmd_containers_prune_builds",
|
||||
"cmd_containers_gc_workspaces",
|
||||
"cmd_quiz_prune",
|
||||
"cmd_embeds_sync",
|
||||
"cmd_emoji_sync",
|
||||
"cmd_migrate_data",
|
||||
"cmd_system_prune",
|
||||
|
||||
@@ -131,6 +131,14 @@ def cmd_emoji_sync(args):
|
||||
print(f"Wrote {count} emoji shortcodes to {EMOJI_JS_PATH}")
|
||||
|
||||
|
||||
def cmd_embeds_sync(args):
|
||||
from devplacepy.embeds import EMBED_JS_PATH, EmbedParser
|
||||
|
||||
count = EmbedParser.write_module()
|
||||
_audit_cli("cli.embeds.sync", f"CLI regenerated the embed spec ({count} keys)", metadata={"count": count})
|
||||
print(f"Wrote {count} embed spec keys to {EMBED_JS_PATH}")
|
||||
|
||||
|
||||
def cmd_migrate_data(args):
|
||||
import os
|
||||
from pathlib import Path
|
||||
@@ -221,6 +229,11 @@ def register_migrate(subparsers):
|
||||
help="Regenerate static/js/emoji-shortcodes.js from the emoji library",
|
||||
).set_defaults(func=cmd_emoji_sync)
|
||||
|
||||
subparsers.add_parser(
|
||||
"embeds-sync",
|
||||
help="Regenerate static/js/embed-spec.js from devplacepy/embeds.py",
|
||||
).set_defaults(func=cmd_embeds_sync)
|
||||
|
||||
migrate = subparsers.add_parser(
|
||||
"migrate-data",
|
||||
help="Relocate legacy runtime files into the consolidated data/ directory",
|
||||
|
||||
@@ -0,0 +1,163 @@
|
||||
# retoor <retoor@molodetz.nl>
|
||||
|
||||
import json
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from urllib.parse import parse_qs, urlencode, urlparse
|
||||
|
||||
EMBED_JS_PATH = Path(__file__).parent / "static" / "js" / "embed-spec.js"
|
||||
|
||||
YOUTUBE_HOSTS = frozenset(
|
||||
{
|
||||
"youtube.com",
|
||||
"www.youtube.com",
|
||||
"m.youtube.com",
|
||||
"youtube-nocookie.com",
|
||||
"www.youtube-nocookie.com",
|
||||
}
|
||||
)
|
||||
MUSIC_HOSTS = frozenset({"music.youtube.com"})
|
||||
SHORT_HOSTS = frozenset({"youtu.be", "www.youtu.be"})
|
||||
|
||||
VIDEO_ID_PATTERN = "[A-Za-z0-9_-]{11}"
|
||||
LIST_ID_PATTERN = "[A-Za-z0-9_-]{13,64}"
|
||||
START_PATTERN = "[0-9]{1,6}s?"
|
||||
MAX_START_SECONDS = 86400
|
||||
MAX_MUSIC_EMBEDS = 3
|
||||
|
||||
EMBED_BASE = "https://www.youtube.com/embed"
|
||||
PLAYLIST_SLUG = "videoseries"
|
||||
EMBED_ALLOW = (
|
||||
"accelerometer; autoplay; clipboard-write; encrypted-media; "
|
||||
"gyroscope; picture-in-picture"
|
||||
)
|
||||
MUSIC_CAPTION = "Open in YouTube Music"
|
||||
PATH_VIDEO_PREFIXES = ("embed", "v", "shorts")
|
||||
|
||||
_VIDEO_ID_RE = re.compile(VIDEO_ID_PATTERN)
|
||||
_LIST_ID_RE = re.compile(LIST_ID_PATTERN)
|
||||
_START_RE = re.compile(START_PATTERN)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Embed:
|
||||
kind: str
|
||||
video_id: str
|
||||
list_id: str
|
||||
start: int
|
||||
music: bool
|
||||
|
||||
|
||||
class EmbedParser:
|
||||
@staticmethod
|
||||
def parse(url: str) -> Embed | None:
|
||||
try:
|
||||
parts = urlparse(url)
|
||||
except ValueError:
|
||||
return None
|
||||
if parts.scheme not in ("http", "https"):
|
||||
return None
|
||||
host = (parts.hostname or "").lower()
|
||||
query = parse_qs(parts.query)
|
||||
segments = [segment for segment in parts.path.split("/") if segment]
|
||||
start = EmbedParser._start(query)
|
||||
if host in SHORT_HOSTS:
|
||||
if len(segments) != 1 or not _VIDEO_ID_RE.fullmatch(segments[0]):
|
||||
return None
|
||||
return Embed("video", segments[0], "", start, False)
|
||||
music = host in MUSIC_HOSTS
|
||||
if not music and host not in YOUTUBE_HOSTS:
|
||||
return None
|
||||
list_id = EmbedParser._list_id(query)
|
||||
if segments == ["watch"]:
|
||||
return EmbedParser._from_watch(query, list_id, start, music)
|
||||
if segments == ["playlist"]:
|
||||
return EmbedParser._playlist(list_id, music)
|
||||
if music:
|
||||
return None
|
||||
if segments == ["embed"] and query.get("listType", [""])[0] == "playlist":
|
||||
return EmbedParser._playlist(list_id, False)
|
||||
if len(segments) == 2 and segments[0] in PATH_VIDEO_PREFIXES:
|
||||
return EmbedParser._from_path(segments, list_id, start)
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def src(embed: Embed) -> str:
|
||||
if embed.kind == "playlist":
|
||||
return f"{EMBED_BASE}/{PLAYLIST_SLUG}?{urlencode({'list': embed.list_id})}"
|
||||
params: dict[str, str | int] = {}
|
||||
if embed.list_id:
|
||||
params["list"] = embed.list_id
|
||||
if embed.start:
|
||||
params["start"] = embed.start
|
||||
suffix = f"?{urlencode(params)}" if params else ""
|
||||
return f"{EMBED_BASE}/{embed.video_id}{suffix}"
|
||||
|
||||
@staticmethod
|
||||
def spec() -> dict:
|
||||
return {
|
||||
"youtubeHosts": sorted(YOUTUBE_HOSTS),
|
||||
"musicHosts": sorted(MUSIC_HOSTS),
|
||||
"shortHosts": sorted(SHORT_HOSTS),
|
||||
"videoIdPattern": VIDEO_ID_PATTERN,
|
||||
"listIdPattern": LIST_ID_PATTERN,
|
||||
"startPattern": START_PATTERN,
|
||||
"maxStartSeconds": MAX_START_SECONDS,
|
||||
"maxMusicEmbeds": MAX_MUSIC_EMBEDS,
|
||||
"embedBase": EMBED_BASE,
|
||||
"playlistSlug": PLAYLIST_SLUG,
|
||||
"allow": EMBED_ALLOW,
|
||||
"musicCaption": MUSIC_CAPTION,
|
||||
"pathVideoPrefixes": list(PATH_VIDEO_PREFIXES),
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def write_module(path: Path = EMBED_JS_PATH) -> int:
|
||||
spec = EmbedParser.spec()
|
||||
body = json.dumps(spec, ensure_ascii=False, sort_keys=True, indent=4)
|
||||
path.write_text(
|
||||
f"// retoor <retoor@molodetz.nl>\n\nexport const EMBED_SPEC = {body};\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
return len(spec)
|
||||
|
||||
@staticmethod
|
||||
def _first(query: dict, key: str) -> str:
|
||||
values = query.get(key)
|
||||
return values[0] if values else ""
|
||||
|
||||
@staticmethod
|
||||
def _list_id(query: dict) -> str:
|
||||
value = EmbedParser._first(query, "list")
|
||||
return value if _LIST_ID_RE.fullmatch(value) else ""
|
||||
|
||||
@staticmethod
|
||||
def _start(query: dict) -> int:
|
||||
raw = EmbedParser._first(query, "start") or EmbedParser._first(query, "t")
|
||||
if not _START_RE.fullmatch(raw):
|
||||
return 0
|
||||
seconds = int(raw.rstrip("s"))
|
||||
return seconds if seconds <= MAX_START_SECONDS else 0
|
||||
|
||||
@staticmethod
|
||||
def _playlist(list_id: str, music: bool) -> Embed | None:
|
||||
if not list_id:
|
||||
return None
|
||||
return Embed("playlist", "", list_id, 0, music)
|
||||
|
||||
@staticmethod
|
||||
def _from_watch(query: dict, list_id: str, start: int, music: bool) -> Embed | None:
|
||||
video_id = EmbedParser._first(query, "v")
|
||||
if _VIDEO_ID_RE.fullmatch(video_id):
|
||||
return Embed("video", video_id, list_id, start, music)
|
||||
return EmbedParser._playlist(list_id, music)
|
||||
|
||||
@staticmethod
|
||||
def _from_path(segments: list[str], list_id: str, start: int) -> Embed | None:
|
||||
slug = segments[1]
|
||||
if slug == PLAYLIST_SLUG:
|
||||
return EmbedParser._playlist(list_id, False)
|
||||
if not _VIDEO_ID_RE.fullmatch(slug):
|
||||
return None
|
||||
return Embed("video", slug, list_id, start, False)
|
||||
+43
-22
@@ -12,6 +12,8 @@ import emoji
|
||||
import mistune
|
||||
from markupsafe import Markup
|
||||
|
||||
from devplacepy.embeds import EMBED_ALLOW, MAX_MUSIC_EMBEDS, MUSIC_CAPTION, EmbedParser
|
||||
|
||||
EMOJI_JS_PATH = Path(__file__).parent / "static" / "js" / "emoji-shortcodes.js"
|
||||
|
||||
CUSTOM_SHORTCODES = {
|
||||
@@ -55,11 +57,6 @@ _WIDGET_RE = re.compile(r"<dp-widget>(.*?)</dp-widget>", re.DOTALL | re.IGNORECA
|
||||
_WIDGET_PH = "\x00WIDGET_{}\x00"
|
||||
|
||||
_SHORTCODE_RE = re.compile(r":([A-Za-z0-9_+\-]+):")
|
||||
_YOUTUBE_RE = re.compile(
|
||||
r"(?:https?://)?(?:www\.)?"
|
||||
r"(?:youtube\.com/(?:watch\?v=|embed/|v/|shorts/)|youtu\.be/)"
|
||||
r"([A-Za-z0-9_-]{11})"
|
||||
)
|
||||
_IMAGE_RE = re.compile(r"\.(jpg|jpeg|png|gif|webp|svg|bmp)(\?.*)?$", re.I)
|
||||
_VIDEO_RE = re.compile(r"\.(mp4|webm|ogg|ogv|mov|m4v)(\?.*)?$", re.I)
|
||||
_AUDIO_RE = re.compile(r"\.(mp3|wav|flac|ogg|aac|m4a|wma|opus)(\?.*)?$", re.I)
|
||||
@@ -90,11 +87,6 @@ def _split_trailing_punct(url: str) -> tuple[str, str]:
|
||||
break
|
||||
return url, "".join(reversed(trailing))
|
||||
|
||||
_YOUTUBE_ALLOW = (
|
||||
"accelerometer; autoplay; clipboard-write; encrypted-media; "
|
||||
"gyroscope; picture-in-picture"
|
||||
)
|
||||
|
||||
_EMAIL_RE = re.compile(r"\b[A-Za-z0-9._%+\-]+@[A-Za-z0-9.\-]+\.[A-Za-z]{2,}\b")
|
||||
_EMAIL_KEEP_DOMAIN = "molodetz.nl"
|
||||
|
||||
@@ -192,16 +184,44 @@ def _alt_from_url(url: str) -> str:
|
||||
return name if name else "Embedded image"
|
||||
|
||||
|
||||
def _embed_url(url: str) -> str:
|
||||
youtube = _YOUTUBE_RE.search(url)
|
||||
class _EmbedBudget:
|
||||
def __init__(self) -> None:
|
||||
self.music = 0
|
||||
|
||||
def allows(self, music: bool) -> bool:
|
||||
if not music:
|
||||
return True
|
||||
if self.music >= MAX_MUSIC_EMBEDS:
|
||||
return False
|
||||
self.music += 1
|
||||
return True
|
||||
|
||||
|
||||
def _youtube_embed(url: str, budget: _EmbedBudget) -> str | None:
|
||||
embed = EmbedParser.parse(url)
|
||||
if embed is None or not budget.allows(embed.music):
|
||||
return None
|
||||
lazy = " loading=\"lazy\"" if embed.music else ""
|
||||
frame = (
|
||||
f'<div class="embed-youtube"><iframe '
|
||||
f'src="{html.escape(EmbedParser.src(embed), quote=True)}" '
|
||||
f'frameborder="0" allowfullscreen{lazy} allow="{EMBED_ALLOW}">'
|
||||
f"</iframe></div>"
|
||||
)
|
||||
if not embed.music:
|
||||
return frame
|
||||
return (
|
||||
f'<div class="embed-music">{frame}'
|
||||
f'<a class="embed-music-caption" href="{html.escape(url, quote=True)}" '
|
||||
f'target="_blank" rel="noopener noreferrer">{html.escape(MUSIC_CAPTION)}</a>'
|
||||
f"</div>"
|
||||
)
|
||||
|
||||
|
||||
def _embed_url(url: str, budget: _EmbedBudget) -> str:
|
||||
youtube = _youtube_embed(url, budget)
|
||||
if youtube:
|
||||
video_id = youtube.group(1)
|
||||
return (
|
||||
f'<div class="embed-youtube"><iframe '
|
||||
f'src="https://www.youtube.com/embed/{video_id}" '
|
||||
f'frameborder="0" allowfullscreen allow="{_YOUTUBE_ALLOW}">'
|
||||
f"</iframe></div>"
|
||||
)
|
||||
return youtube
|
||||
escaped = html.escape(url, quote=True)
|
||||
if _IMAGE_RE.search(url):
|
||||
alt = html.escape(_alt_from_url(url), quote=True)
|
||||
@@ -230,7 +250,7 @@ def _mask_emails(text: str) -> str:
|
||||
return _EMAIL_RE.sub(_mask_email, text)
|
||||
|
||||
|
||||
def _transform_text(text: str) -> str:
|
||||
def _transform_text(text: str, budget: _EmbedBudget) -> str:
|
||||
out: list[str] = []
|
||||
pos = 0
|
||||
for match in _TOKEN_RE.finditer(text):
|
||||
@@ -238,7 +258,7 @@ def _transform_text(text: str) -> str:
|
||||
out.append(html.escape(_mask_emails(text[pos:match.start()])))
|
||||
if match.group("url"):
|
||||
url, trailing = _split_trailing_punct(match.group("url"))
|
||||
out.append(_embed_url(url))
|
||||
out.append(_embed_url(url, budget))
|
||||
if trailing:
|
||||
out.append(html.escape(trailing))
|
||||
else:
|
||||
@@ -269,6 +289,7 @@ class _MediaProcessor(HTMLParser):
|
||||
super().__init__(convert_charrefs=True)
|
||||
self._out: list[str] = []
|
||||
self._skip_depth = 0
|
||||
self._budget = _EmbedBudget()
|
||||
|
||||
def handle_starttag(self, tag: str, attrs: list) -> None:
|
||||
self._out.append(_emit_tag(tag, attrs))
|
||||
@@ -287,7 +308,7 @@ class _MediaProcessor(HTMLParser):
|
||||
if self._skip_depth > 0:
|
||||
self._out.append(html.escape(_mask_emails(data)))
|
||||
else:
|
||||
self._out.append(_transform_text(data))
|
||||
self._out.append(_transform_text(data, self._budget))
|
||||
|
||||
def result(self) -> str:
|
||||
return "".join(self._out)
|
||||
|
||||
@@ -77,7 +77,7 @@ Presence is handled entirely by the generic, cross-worker-correct mechanism docu
|
||||
|
||||
## Renderer integration (XSS control)
|
||||
|
||||
Live/echoed message bubbles are NEVER injected as raw HTML. `AppChat._buildBubble` (`static/js/components/AppChat.js`) creates a `<dp-content>` element and sets its `textContent` to the message body - never `innerHTML` - so the element's own client-side pipeline (emoji shortcode -> marked -> `DOMPurify.sanitize` -> highlight.js -> image/YouTube/autolink) renders it safely, exactly like every other live-content surface that uses `<dp-content>`. This gives Discord-style emoji, image and YouTube embeds, autolinking, and sanitization for free.
|
||||
Live/echoed message bubbles are NEVER injected as raw HTML. `AppChat._buildBubble` (`static/js/components/AppChat.js`) creates a `<dp-content>` element and sets its `textContent` to the message body - never `innerHTML` - so the element's own client-side pipeline (emoji shortcode -> marked -> `DOMPurify.sanitize` -> highlight.js -> image/YouTube and YouTube Music/autolink) renders it safely, exactly like every other live-content surface that uses `<dp-content>`. This gives Discord-style emoji, image, YouTube and YouTube Music embeds (`static/js/EmbedParser.js`, spec in `devplacepy/embeds.py`), autolinking, and sanitization for free.
|
||||
|
||||
## Frontend
|
||||
|
||||
|
||||
@@ -193,6 +193,25 @@ pre.code-has-copy > .code-copy-btn:hover {
|
||||
border: none;
|
||||
}
|
||||
|
||||
.embed-music {
|
||||
margin: 0.75rem 0;
|
||||
}
|
||||
|
||||
.embed-music .embed-youtube {
|
||||
margin-bottom: var(--space-xs);
|
||||
}
|
||||
|
||||
.embed-music-caption {
|
||||
display: inline-block;
|
||||
font-size: 0.75rem;
|
||||
color: var(--text-muted);
|
||||
text-decoration: none;
|
||||
}
|
||||
|
||||
.embed-music-caption:hover {
|
||||
color: var(--accent);
|
||||
}
|
||||
|
||||
.rendered-content img:not(.avatar-img) {
|
||||
max-width: 100%;
|
||||
border-radius: var(--radius);
|
||||
@@ -209,7 +228,7 @@ pre.code-has-copy > .code-copy-btn:hover {
|
||||
}
|
||||
|
||||
.rendered-content a[href^="http"]::after {
|
||||
content: "\u2197";
|
||||
content: "\2197";
|
||||
display: inline-block;
|
||||
margin-left: 0.125rem;
|
||||
font-size: 0.75em;
|
||||
|
||||
@@ -46,6 +46,7 @@ Detail on each utility:
|
||||
`Http.suspend()` is the named "we are resolving this elsewhere, do not let the caller render an error" idiom (a promise that never settles). It replaced three inline `new Promise(() => {})` copies and is what `toLogin()` and the terms gate both return.
|
||||
- **`TermsGate` (`static/js/TermsGate.js`, `app.termsGate`).** The single client-side handler for the terms-acceptance refusal. `Http` calls `Http._gate(data, options, retry)` on **every** POST helper (`sendForm`, `send`, `postJson`); when the error payload carries `code: "terms_acceptance_required"` it hands off to `app.termsGate.intercept(error, retry)`, which shows one dialog (Accept and continue / Not now, with the Terms, Guidelines and Privacy links), POSTs `/auth/accept-terms` on accept, and then **re-runs the original request** so the click the user made actually happens. Declining returns `Http.suspend()`.
|
||||
Load-bearing details: the handoff runs **before** the `options.silent` check, because the four `OptimisticAction` controllers (vote/react/bookmark/poll) pass `silent: true` and would otherwise swallow a blocking gate into a 1.5s "Error" flash; `options.termsRetry` bounds the retry to exactly one pass; and `confirm()`/`accept()` are each deduped by a stored promise so N concurrent gated requests produce one dialog and one acceptance POST. `dp-upload` bypasses `Http` (it needs `FormData`), so it checks `app.termsGate.matches(data)` itself - any other raw-`fetch` caller must do the same. Never add a per-caller terms check: the backend contract lives in `routers/auth/terms.py` `TERMS_ACCEPTANCE_CODE` and is documented in `devplacepy/services/moderation/CLAUDE.md`.
|
||||
- **`EmbedParser` (`static/js/EmbedParser.js`).** The single client-side YouTube / YouTube Music URL parser (`EmbedParser.parse(url)` -> `{kind, videoId, listId, start, music}` or `null`, `EmbedParser.src(embed)`). Its constants (host allowlists, id patterns, caps, the `allow` string, the caption) come from the generated `static/js/embed-spec.js`, written from `devplacepy/embeds.py` by `devplace embeds-sync` - never hand-edit it. `ContentRenderer.youtubeEmbed` is its only consumer and enforces the per-render music cap; never add another YouTube regex.
|
||||
- **`Poller` (`static/js/Poller.js`).** `new Poller(fn, intervalMs, { immediate = true, pauseHidden = false })` runs `fn` on an interval with `start()`/`stop()`/`tick()`; `tick()` swallows errors so one failed poll never kills the loop, and `pauseHidden` skips the tick while `document.hidden`. Used by every live-update loop: `CounterManager` (30s, `pauseHidden`), `ContainerManager` (3s), `ContainerList` (4s), `AiUsageMonitor`, `ServiceMonitor`, and `ContainerInstance`'s detail (4s) + logs (3s). Store the `Poller`, not a raw interval id.
|
||||
- **`JobPoller` (`static/js/JobPoller.js`).** `JobPoller.run(statusUrl, { onDone, onFailed, onTimeout, intervalMs = 1500, maxAttempts = 200 })` returns a Promise; it polls `Http.getJson(statusUrl)`, swallows transient fetch errors, and fires the matching callback on `status === "done"|"failed"` or timeout. This is the one place the async-job status-poll lives - `ProjectForker` and `ZipDownloader` both call it with their own navigate/download/toast callbacks.
|
||||
- **`OptimisticAction` (`static/js/OptimisticAction.js`).** Base with one method, `submit(url, params, errorTarget, render)`: `Http.sendForm` -> `render(result)` on success -> `console.error` + (when `errorTarget` is given) `Toast.flash(errorTarget, "Error", 1500)` on failure. `VoteManager`/`ReactionBar`/`BookmarkManager`/`PollManager` `extend` it and call `this.submit(...)` for their POST, **keeping their own event wiring** (so `ReactionBar`'s palette toggle and `PollManager`'s multi-action handlers and `VoteManager`'s per-button `stopPropagation` are untouched). Pass `errorTarget` only where the old code toasted (`VoteManager`); the others pass `null` to keep their console-only behaviour.
|
||||
|
||||
@@ -1,5 +1,8 @@
|
||||
// retoor <retoor@molodetz.nl>
|
||||
|
||||
import { EMBED_SPEC } from "./embed-spec.js";
|
||||
import { EmbedParser } from "./EmbedParser.js";
|
||||
|
||||
export class ContentRenderer {
|
||||
constructor() {
|
||||
this.emojiMap = {};
|
||||
@@ -7,7 +10,6 @@ export class ContentRenderer {
|
||||
this.imageExtRe = /\.(jpg|jpeg|png|gif|webp|svg|bmp|webp)(\?.*)?$/i;
|
||||
this.videoExtRe = /\.(mp4|webm|ogg|ogv|mov|m4v)(\?.*)?$/i;
|
||||
this.audioExtRe = /\.(mp3|wav|flac|ogg|aac|m4a|wma|opus)(\?.*)?$/i;
|
||||
this.youtubeRe = /(?:https?:\/\/)?(?:www\.)?(?:youtube\.com\/(?:watch\?v=|embed\/|v\/|shorts\/)|youtu\.be\/)([a-zA-Z0-9_-]{11})/;
|
||||
}
|
||||
|
||||
replaceShortcodes(text) {
|
||||
@@ -151,6 +153,8 @@ export class ContentRenderer {
|
||||
const temp = document.createElement("div");
|
||||
temp.innerHTML = html;
|
||||
|
||||
const budget = { music: 0 };
|
||||
|
||||
this.walkNodes(temp, (node) => {
|
||||
if (node.nodeType !== 3) return;
|
||||
const text = node.textContent;
|
||||
@@ -188,7 +192,7 @@ export class ContentRenderer {
|
||||
a.textContent = "@" + part.username;
|
||||
fragment.appendChild(a);
|
||||
} else {
|
||||
const el = this.urlToEmbed(part.value);
|
||||
const el = this.urlToEmbed(part.value, budget);
|
||||
if (el) {
|
||||
fragment.appendChild(el);
|
||||
} else {
|
||||
@@ -240,20 +244,39 @@ export class ContentRenderer {
|
||||
}
|
||||
}
|
||||
|
||||
urlToEmbed(url) {
|
||||
const ytMatch = url.match(this.youtubeRe);
|
||||
if (ytMatch) {
|
||||
const videoId = ytMatch[1];
|
||||
const wrap = document.createElement("div");
|
||||
wrap.className = "embed-youtube";
|
||||
wrap.innerHTML = (
|
||||
'<iframe src="https://www.youtube.com/embed/' + videoId + '" ' +
|
||||
'frameborder="0" allowfullscreen ' +
|
||||
'allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture">' +
|
||||
'</iframe>'
|
||||
);
|
||||
return wrap;
|
||||
youtubeEmbed(url, budget) {
|
||||
const embed = EmbedParser.parse(url);
|
||||
if (!embed) return null;
|
||||
if (embed.music) {
|
||||
if (budget.music >= EMBED_SPEC.maxMusicEmbeds) return null;
|
||||
budget.music += 1;
|
||||
}
|
||||
const wrap = document.createElement("div");
|
||||
wrap.className = "embed-youtube";
|
||||
const frame = document.createElement("iframe");
|
||||
frame.src = EmbedParser.src(embed);
|
||||
frame.setAttribute("frameborder", "0");
|
||||
frame.setAttribute("allowfullscreen", "");
|
||||
frame.setAttribute("allow", EMBED_SPEC.allow);
|
||||
if (embed.music) frame.setAttribute("loading", "lazy");
|
||||
wrap.appendChild(frame);
|
||||
if (!embed.music) return wrap;
|
||||
const box = document.createElement("div");
|
||||
box.className = "embed-music";
|
||||
const caption = document.createElement("a");
|
||||
caption.className = "embed-music-caption";
|
||||
caption.href = url;
|
||||
caption.target = "_blank";
|
||||
caption.rel = "noopener noreferrer";
|
||||
caption.textContent = EMBED_SPEC.musicCaption;
|
||||
box.appendChild(wrap);
|
||||
box.appendChild(caption);
|
||||
return box;
|
||||
}
|
||||
|
||||
urlToEmbed(url, budget = { music: 0 }) {
|
||||
const youtube = this.youtubeEmbed(url, budget);
|
||||
if (youtube) return youtube;
|
||||
|
||||
if (this.imageExtRe.test(url)) {
|
||||
const img = document.createElement("img");
|
||||
|
||||
@@ -0,0 +1,88 @@
|
||||
// retoor <retoor@molodetz.nl>
|
||||
|
||||
import { EMBED_SPEC } from "./embed-spec.js";
|
||||
|
||||
const VIDEO_ID_RE = new RegExp("^" + EMBED_SPEC.videoIdPattern + "$");
|
||||
const LIST_ID_RE = new RegExp("^" + EMBED_SPEC.listIdPattern + "$");
|
||||
const START_RE = new RegExp("^" + EMBED_SPEC.startPattern + "$");
|
||||
|
||||
export class EmbedParser {
|
||||
static parse(url) {
|
||||
let parts;
|
||||
try {
|
||||
parts = new URL(url);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
if (parts.protocol !== "http:" && parts.protocol !== "https:") return null;
|
||||
const host = parts.hostname.toLowerCase();
|
||||
const query = parts.searchParams;
|
||||
const segments = parts.pathname.split("/").filter(Boolean);
|
||||
const start = EmbedParser.start(query);
|
||||
if (EMBED_SPEC.shortHosts.includes(host)) {
|
||||
if (segments.length !== 1 || !VIDEO_ID_RE.test(segments[0])) return null;
|
||||
return { kind: "video", videoId: segments[0], listId: "", start, music: false };
|
||||
}
|
||||
const music = EMBED_SPEC.musicHosts.includes(host);
|
||||
if (!music && !EMBED_SPEC.youtubeHosts.includes(host)) return null;
|
||||
const listId = EmbedParser.listId(query);
|
||||
if (segments.length === 1 && segments[0] === "watch") {
|
||||
return EmbedParser.fromWatch(query, listId, start, music);
|
||||
}
|
||||
if (segments.length === 1 && segments[0] === "playlist") {
|
||||
return EmbedParser.playlist(listId, music);
|
||||
}
|
||||
if (music) return null;
|
||||
if (segments.length === 1 && segments[0] === "embed" && query.get("listType") === "playlist") {
|
||||
return EmbedParser.playlist(listId, false);
|
||||
}
|
||||
if (segments.length === 2 && EMBED_SPEC.pathVideoPrefixes.includes(segments[0])) {
|
||||
return EmbedParser.fromPath(segments, listId, start);
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
static src(embed) {
|
||||
if (embed.kind === "playlist") {
|
||||
const playlist = new URLSearchParams({ list: embed.listId });
|
||||
return EMBED_SPEC.embedBase + "/" + EMBED_SPEC.playlistSlug + "?" + playlist.toString();
|
||||
}
|
||||
const params = new URLSearchParams();
|
||||
if (embed.listId) params.set("list", embed.listId);
|
||||
if (embed.start) params.set("start", String(embed.start));
|
||||
const suffix = params.toString() ? "?" + params.toString() : "";
|
||||
return EMBED_SPEC.embedBase + "/" + embed.videoId + suffix;
|
||||
}
|
||||
|
||||
static listId(query) {
|
||||
const value = query.get("list") || "";
|
||||
return LIST_ID_RE.test(value) ? value : "";
|
||||
}
|
||||
|
||||
static start(query) {
|
||||
const raw = query.get("start") || query.get("t") || "";
|
||||
if (!START_RE.test(raw)) return 0;
|
||||
const seconds = parseInt(raw.replace(/s$/, ""), 10);
|
||||
return seconds <= EMBED_SPEC.maxStartSeconds ? seconds : 0;
|
||||
}
|
||||
|
||||
static playlist(listId, music) {
|
||||
if (!listId) return null;
|
||||
return { kind: "playlist", videoId: "", listId, start: 0, music };
|
||||
}
|
||||
|
||||
static fromWatch(query, listId, start, music) {
|
||||
const videoId = query.get("v") || "";
|
||||
if (VIDEO_ID_RE.test(videoId)) {
|
||||
return { kind: "video", videoId, listId, start, music };
|
||||
}
|
||||
return EmbedParser.playlist(listId, music);
|
||||
}
|
||||
|
||||
static fromPath(segments, listId, start) {
|
||||
const slug = segments[1];
|
||||
if (slug === EMBED_SPEC.playlistSlug) return EmbedParser.playlist(listId, false);
|
||||
if (!VIDEO_ID_RE.test(slug)) return null;
|
||||
return { kind: "video", videoId: slug, listId, start, music: false };
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
// retoor <retoor@molodetz.nl>
|
||||
|
||||
export const EMBED_SPEC = {
|
||||
"allow": "accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture",
|
||||
"embedBase": "https://www.youtube.com/embed",
|
||||
"listIdPattern": "[A-Za-z0-9_-]{13,64}",
|
||||
"maxMusicEmbeds": 3,
|
||||
"maxStartSeconds": 86400,
|
||||
"musicCaption": "Open in YouTube Music",
|
||||
"musicHosts": [
|
||||
"music.youtube.com"
|
||||
],
|
||||
"pathVideoPrefixes": [
|
||||
"embed",
|
||||
"v",
|
||||
"shorts"
|
||||
],
|
||||
"playlistSlug": "videoseries",
|
||||
"shortHosts": [
|
||||
"www.youtu.be",
|
||||
"youtu.be"
|
||||
],
|
||||
"startPattern": "[0-9]{1,6}s?",
|
||||
"videoIdPattern": "[A-Za-z0-9_-]{11}",
|
||||
"youtubeHosts": [
|
||||
"m.youtube.com",
|
||||
"www.youtube-nocookie.com",
|
||||
"www.youtube.com",
|
||||
"youtube-nocookie.com",
|
||||
"youtube.com"
|
||||
]
|
||||
};
|
||||
@@ -50,7 +50,7 @@ Every component **renders into the light DOM (no shadow root)** so the global CS
|
||||
|
||||
**The component-vs-module rule.** Self-contained presentational widgets become components. Partial-bound controllers that enhance server-rendered markup (votes, reactions, comments) and pure utilities stay plain modules, because they augment existing HTML rather than render standalone UI.
|
||||
|
||||
**Content and titles render on the backend.** For SEO, all content and titles that exist at request time are rendered server-side by the `render_content` and `render_title` Jinja globals (`devplacepy/rendering.py`, built on mistune with full feature parity to the client renderer: emoji shortcodes, markdown, sanitisation, and YouTube/video/audio/image embeds, mentions, and autolinks). The `dp-content`/`dp-title` client components and the `data-render` attribute are now reserved for genuinely live content that is generated in the browser after load (chat answers, newly posted comments, live direct messages, the planning report). Server-rendered code blocks are still syntax-highlighted client-side, and content images are wired to the lightbox via a delegated listener.
|
||||
**Content and titles render on the backend.** For SEO, all content and titles that exist at request time are rendered server-side by the `render_content` and `render_title` Jinja globals (`devplacepy/rendering.py`, built on mistune with full feature parity to the client renderer: emoji shortcodes, markdown, sanitisation, and YouTube/YouTube Music/video/audio/image embeds, mentions, and autolinks). The `dp-content`/`dp-title` client components and the `data-render` attribute are now reserved for genuinely live content that is generated in the browser after load (chat answers, newly posted comments, live direct messages, the planning report). Server-rendered code blocks are still syntax-highlighted client-side, and content images are wired to the lightbox via a delegated listener.
|
||||
|
||||
## Shared utilities
|
||||
|
||||
@@ -72,7 +72,7 @@ A few plain modules own the cross-cutting patterns. Reach for these instead of h
|
||||
3. **`DOMPurify.sanitize`**
|
||||
4. `highlight.js` on `<pre><code>`
|
||||
5. image URL -> `<img>`
|
||||
6. YouTube URL -> trusted iframe
|
||||
6. YouTube / YouTube Music URL -> trusted iframe (`EmbedParser.js`, constants from the generated `embed-spec.js`; music adds an "Open in YouTube Music" caption and is capped at 3 per item)
|
||||
7. autolink remaining URLs
|
||||
|
||||
A `NodeIterator` skips `CODE`, `PRE`, `SCRIPT`, and `STYLE` so URLs inside code are never rewritten. Sanitization runs on the raw `marked` output before the trusted YouTube embeds are injected, so user content cannot inject script, event-handler, or iframe payloads while our own embeds survive.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
# dp-content
|
||||
|
||||
Renders user-supplied text as safe rich content: emoji shortcodes, GitHub-flavored markdown,
|
||||
`DOMPurify` sanitisation, then media embeds (images, video, YouTube), mentions, and
|
||||
`DOMPurify` sanitisation, then media embeds (images, video, YouTube, YouTube Music), mentions, and
|
||||
autolinks. It exposes the shared `contentRenderer` engine (`static/js/ContentRenderer.js`),
|
||||
the same pipeline used by the site's `data-render` attribute, as a custom element.
|
||||
|
||||
|
||||
@@ -33,6 +33,7 @@ never used to build an advertising profile.
|
||||
| Recipient | What is sent | When |
|
||||
|---|---|---|
|
||||
| {{ ai_provider_name() }} | The text you asked an assistant feature to process | **Only after you grant `ai_third_party` consent.** Default: not granted. |
|
||||
| YouTube (Google) | Your browser contacts YouTube directly when a post, comment or message contains a YouTube or YouTube Music link, which DevPlace shows as an inline player. YouTube then sees your IP address and browser details, as for any embedded video. DevPlace itself sends YouTube nothing | Only when you view content that contains such a link |
|
||||
| Push services (Apple, browser push endpoints) | An encrypted notification payload to a device you registered | Only when you register a device and enable a push channel |
|
||||
| Telegram | Messages in the Telegram conversation you paired | Only when you pair Telegram yourself |
|
||||
| Software another member runs in a container | Your DevPlace API key and user id, injected as `DEVPLACE_API_KEY` and `DEVPLACE_USER_UID` | **Only after you grant `container_credentials` consent**, and only when the container is run by someone else. Your own containers use your own credentials with no separate consent. |
|
||||
|
||||
@@ -184,6 +184,7 @@ Every state-changing action in DevPlace records one append-only row through `dev
|
||||
| `cli.devii.task.disable` | `cli/devii.py` |
|
||||
| `cli.devii.task.prune` | `cli/devii.py` |
|
||||
| `cli.emoji.sync` | `cli/migrate.py` |
|
||||
| `cli.embeds.sync` | `cli/migrate.py` |
|
||||
| `cli.forks.clear` | `cli/jobs.py` |
|
||||
| `cli.forks.prune` | `cli/jobs.py` |
|
||||
| `cli.game.era.end` | `cli/game.py` |
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
[project]
|
||||
name = "devplacepy"
|
||||
version = "1.1.11"
|
||||
version = "1.1.12"
|
||||
description = "DevPlace - The Developer Social Network"
|
||||
requires-python = ">=3.12"
|
||||
dependencies = [
|
||||
|
||||
@@ -222,6 +222,32 @@ def test_issue_detail_and_listing_render_title_backend():
|
||||
_gitea_off()
|
||||
|
||||
|
||||
def test_post_content_renders_music_embeds_over_http():
|
||||
owner = _owner()
|
||||
uid = str(uuid4())
|
||||
slug = make_combined_slug("Music Render Check", uid)
|
||||
album = "OLAK5uy_nZZjkBu_E4olFSb5Ey-fQ-4a0ZCqJICdQ"
|
||||
body = (
|
||||
"track https://music.youtube.com/watch?v=4D7u5KF7SP8\n\n"
|
||||
f"album https://music.youtube.com/playlist?list={album}\n\n"
|
||||
"browse https://music.youtube.com/browse/MPREb_abcdefghijk"
|
||||
)
|
||||
get_table("posts").insert(
|
||||
{
|
||||
"deleted_at": None, "deleted_by": None, "uid": uid, "user_uid": owner,
|
||||
"slug": slug, "title": "Music Render Check", "content": body,
|
||||
"topic": "general", "project_uid": None, "image": None, "stars": 0,
|
||||
"created_at": datetime.now(timezone.utc).isoformat(),
|
||||
}
|
||||
)
|
||||
html = _client().get(f"/posts/{slug}").text
|
||||
assert html.count('class="embed-music"') == 2
|
||||
assert "youtube.com/embed/4D7u5KF7SP8" in html
|
||||
assert f"youtube.com/embed/videoseries?list={album}" in html
|
||||
assert 'href="https://music.youtube.com/browse/MPREb_abcdefghijk"' in html
|
||||
assert "Open in YouTube Music" in html
|
||||
|
||||
|
||||
def test_post_content_rendered_backend_with_media():
|
||||
owner = _owner()
|
||||
from uuid import uuid4
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
# retoor <retoor@molodetz.nl>
|
||||
|
||||
from tests.conftest import BASE_URL
|
||||
from tests.unit.embeds import EMBED_CASES, TRACK, VIDEO
|
||||
|
||||
ROCKET = "\U0001F680"
|
||||
FIRE = "\U0001F525"
|
||||
@@ -44,6 +45,58 @@ def test_content_youtube_embedded_server_side(alice):
|
||||
assert "youtube.com/embed/dQw4w9WgXcQ" in (iframe.get_attribute("src") or "")
|
||||
|
||||
|
||||
def test_content_music_embedded_server_side(alice):
|
||||
page, _ = alice
|
||||
_create_post(page, f"hear https://music.youtube.com/watch?v={TRACK} now", "Music host post")
|
||||
iframe = page.locator(".post-detail-content .embed-music .embed-youtube iframe").first
|
||||
iframe.wait_for(state="attached")
|
||||
assert f"youtube.com/embed/{TRACK}" in (iframe.get_attribute("src") or "")
|
||||
caption = page.locator(".post-detail-content .embed-music-caption").first
|
||||
caption.wait_for(state="visible")
|
||||
assert caption.text_content().strip() == "Open in YouTube Music"
|
||||
|
||||
|
||||
def test_client_embed_parser_matches_server_for_every_case(alice):
|
||||
page, _ = alice
|
||||
page.goto(f"{BASE_URL}/feed", wait_until="domcontentloaded")
|
||||
urls = [case[0] for case in EMBED_CASES]
|
||||
client = page.evaluate(
|
||||
"""async (urls) => {
|
||||
const mod = await import('/static/js/EmbedParser.js');
|
||||
return urls.map((u) => {
|
||||
const e = mod.EmbedParser.parse(u);
|
||||
return e ? [mod.EmbedParser.src(e), e.music] : null;
|
||||
});
|
||||
}""",
|
||||
urls,
|
||||
)
|
||||
for (url, expected, music), got in zip(EMBED_CASES, client):
|
||||
if expected is None:
|
||||
assert got is None, url
|
||||
else:
|
||||
assert got == [expected, music], url
|
||||
|
||||
|
||||
def test_client_renderer_embeds_music_and_caps_it(alice):
|
||||
page, _ = alice
|
||||
page.goto(f"{BASE_URL}/feed", wait_until="domcontentloaded")
|
||||
urls = " ".join(f"https://music.youtube.com/watch?v={VIDEO[:-1]}{c}" for c in "abcde")
|
||||
result = page.evaluate(
|
||||
"""async (text) => {
|
||||
const mod = await import('/static/js/ContentRenderer.js');
|
||||
const box = document.createElement('div');
|
||||
box.innerHTML = mod.contentRenderer.processMedia(text);
|
||||
return {
|
||||
frames: box.querySelectorAll('.embed-music iframe').length,
|
||||
captions: box.querySelectorAll('.embed-music-caption').length,
|
||||
links: box.querySelectorAll('a:not(.embed-music-caption)').length,
|
||||
};
|
||||
}""",
|
||||
urls,
|
||||
)
|
||||
assert result == {"frames": 3, "captions": 3, "links": 2}
|
||||
|
||||
|
||||
def test_content_code_block_has_copy_button(alice):
|
||||
page, _ = alice
|
||||
_create_post(page, "snippet\n\n```python\nprint('hi')\n```", "Code copy host post")
|
||||
|
||||
@@ -0,0 +1,175 @@
|
||||
# retoor <retoor@molodetz.nl>
|
||||
|
||||
import itertools
|
||||
import random
|
||||
import re
|
||||
|
||||
from devplacepy.embeds import (
|
||||
EMBED_JS_PATH,
|
||||
MAX_MUSIC_EMBEDS,
|
||||
MAX_START_SECONDS,
|
||||
EmbedParser,
|
||||
)
|
||||
from devplacepy.rendering import render_content
|
||||
|
||||
VIDEO = "dQw4w9WgXcQ"
|
||||
TRACK = "4D7u5KF7SP8"
|
||||
ALBUM = "OLAK5uy_nZZjkBu_E4olFSb5Ey-fQ-4a0ZCqJICdQ"
|
||||
PLAIN = "PLFgquLnL59alCl_2TQvOiD5Vgm1hCaGSI"
|
||||
BASE = "https://www.youtube.com/embed"
|
||||
|
||||
EMBED_CASES = [
|
||||
(f"https://music.youtube.com/watch?v={TRACK}", f"{BASE}/{TRACK}", True),
|
||||
(f"https://music.youtube.com/watch?v={TRACK}&list={ALBUM}", f"{BASE}/{TRACK}?list={ALBUM}", True),
|
||||
(f"https://music.youtube.com/watch?list={ALBUM}&si=abc&v={TRACK}", f"{BASE}/{TRACK}?list={ALBUM}", True),
|
||||
(f"https://music.youtube.com/watch?v=bad&list={ALBUM}", f"{BASE}/videoseries?list={ALBUM}", True),
|
||||
(f"https://music.youtube.com/playlist?list={ALBUM}", f"{BASE}/videoseries?list={ALBUM}", True),
|
||||
(f"https://music.youtube.com/watch?v={TRACK}&list=RDAMVM{TRACK}", f"{BASE}/{TRACK}?list=RDAMVM{TRACK}", True),
|
||||
(f"https://music.youtube.com/watch?v={TRACK}&t=45", f"{BASE}/{TRACK}?start=45", True),
|
||||
(f"https://www.youtube.com/playlist?list={PLAIN}", f"{BASE}/videoseries?list={PLAIN}", False),
|
||||
(f"https://www.youtube.com/watch?v={VIDEO}", f"{BASE}/{VIDEO}", False),
|
||||
(f"https://www.youtube.com/watch?feature=share&v={VIDEO}", f"{BASE}/{VIDEO}", False),
|
||||
(f"https://m.youtube.com/watch?v={VIDEO}&t=30s", f"{BASE}/{VIDEO}?start=30", False),
|
||||
(f"https://youtube.com/embed/{VIDEO}", f"{BASE}/{VIDEO}", False),
|
||||
(f"https://www.youtube-nocookie.com/embed/{VIDEO}", f"{BASE}/{VIDEO}", False),
|
||||
(f"https://www.youtube.com/v/{VIDEO}", f"{BASE}/{VIDEO}", False),
|
||||
(f"https://www.youtube.com/shorts/{VIDEO}", f"{BASE}/{VIDEO}", False),
|
||||
(f"https://youtu.be/{VIDEO}", f"{BASE}/{VIDEO}", False),
|
||||
(f"https://youtu.be/{VIDEO}?t=30", f"{BASE}/{VIDEO}?start=30", False),
|
||||
(f"http://youtu.be/{VIDEO}", f"{BASE}/{VIDEO}", False),
|
||||
(f"https://www.youtube.com/embed/videoseries?list={PLAIN}", f"{BASE}/videoseries?list={PLAIN}", False),
|
||||
(f"https://www.youtube.com/embed?listType=playlist&list={PLAIN}", f"{BASE}/videoseries?list={PLAIN}", False),
|
||||
(f"https://www.youtube.com/watch?v={VIDEO}&list=WL", f"{BASE}/{VIDEO}", False),
|
||||
(f"https://www.youtube.com/watch?v={VIDEO}&list=bad%22%3E%3Cscript%3E", f"{BASE}/{VIDEO}", False),
|
||||
(f"https://www.youtube.com/watch?v={VIDEO}&t={MAX_START_SECONDS + 1}", f"{BASE}/{VIDEO}", False),
|
||||
(f"https://www.youtube.com/watch?v={VIDEO}&t=1m30s", f"{BASE}/{VIDEO}", False),
|
||||
("https://music.youtube.com/browse/MPREb_abcdefghijk", None, False),
|
||||
("https://music.youtube.com/channel/UCabcdefghijklmnopqrstuv", None, False),
|
||||
("https://music.youtube.com/search?q=daft+punk", None, False),
|
||||
("https://music.youtube.com/library", None, False),
|
||||
("https://music.youtube.com/playlist", None, False),
|
||||
(f"https://music.youtube.com/embed/{TRACK}", None, False),
|
||||
(f"https://evilyoutube.com/watch?v={VIDEO}", None, False),
|
||||
(f"https://youtube.com.evil.example/watch?v={VIDEO}", None, False),
|
||||
(f"https://evil.example/https://www.youtube.com/watch?v={VIDEO}", None, False),
|
||||
(f"https://notyoutu.be/{VIDEO}", None, False),
|
||||
(f"https://youtu.be/{VIDEO}/extra", None, False),
|
||||
("https://www.youtube.com/watch?v=short", None, False),
|
||||
("https://www.youtube.com/watch", None, False),
|
||||
("https://www.youtube.com/embed/videoseries", None, False),
|
||||
(f"javascript://youtube.com/watch?v={VIDEO}", None, False),
|
||||
(f"ftp://youtube.com/watch?v={VIDEO}", None, False),
|
||||
("not a url", None, False),
|
||||
("", None, False),
|
||||
]
|
||||
|
||||
|
||||
def _iframe_srcs(rendered: str) -> list[str]:
|
||||
return re.findall(r'<iframe src="([^"]*)"', str(rendered))
|
||||
|
||||
|
||||
def test_parser_cases_table():
|
||||
for url, expected, music in EMBED_CASES:
|
||||
embed = EmbedParser.parse(url)
|
||||
if expected is None:
|
||||
assert embed is None, url
|
||||
continue
|
||||
assert embed is not None, url
|
||||
assert EmbedParser.src(embed) == expected, url
|
||||
assert embed.music is music, url
|
||||
|
||||
|
||||
def test_render_matches_parser_for_every_case():
|
||||
for url, expected, _ in EMBED_CASES:
|
||||
srcs = _iframe_srcs(render_content(url))
|
||||
if expected is None:
|
||||
assert srcs == [], url
|
||||
else:
|
||||
assert srcs == [expected.replace("&", "&")], url
|
||||
|
||||
|
||||
def test_music_embed_markup_and_caption():
|
||||
out = str(render_content(f"https://music.youtube.com/watch?v={TRACK}"))
|
||||
assert 'class="embed-music"' in out
|
||||
assert 'class="embed-youtube"' in out
|
||||
assert 'loading="lazy"' in out
|
||||
assert 'class="embed-music-caption"' in out
|
||||
assert "Open in YouTube Music" in out
|
||||
assert f'href="https://music.youtube.com/watch?v={TRACK}"' in out
|
||||
assert 'rel="noopener noreferrer"' in out
|
||||
|
||||
|
||||
def test_plain_youtube_keeps_old_markup():
|
||||
out = str(render_content(f"https://youtu.be/{VIDEO}"))
|
||||
assert 'class="embed-youtube"' in out
|
||||
assert "embed-music" not in out
|
||||
assert "loading=" not in out
|
||||
|
||||
|
||||
def test_music_embed_cap_turns_overflow_into_links():
|
||||
urls = [f"https://music.youtube.com/watch?v={TRACK[:-1]}{c}" for c in "abcde"]
|
||||
out = str(render_content("\n\n".join(urls)))
|
||||
assert out.count("<iframe") == MAX_MUSIC_EMBEDS
|
||||
assert out.count('class="embed-music"') == MAX_MUSIC_EMBEDS
|
||||
for url in urls[MAX_MUSIC_EMBEDS:]:
|
||||
assert f'<a href="{url}"' in out
|
||||
|
||||
|
||||
def test_music_cap_does_not_limit_plain_youtube():
|
||||
urls = [f"https://youtu.be/{VIDEO[:-1]}{c}" for c in "abcdefg"]
|
||||
out = str(render_content("\n\n".join(urls)))
|
||||
assert out.count("<iframe") == 7
|
||||
|
||||
|
||||
def test_music_cap_is_per_content_item():
|
||||
text = f"https://music.youtube.com/watch?v={TRACK}"
|
||||
for _ in range(3):
|
||||
assert str(render_content(text)).count("<iframe") == 1
|
||||
|
||||
|
||||
def test_urls_inside_code_and_links_are_not_embedded():
|
||||
url = f"https://music.youtube.com/watch?v={TRACK}"
|
||||
assert "<iframe" not in str(render_content(f"`{url}`"))
|
||||
assert "<iframe" not in str(render_content(f"[x]({url})"))
|
||||
assert "<iframe" not in str(render_content(f"```\n{url}\n```"))
|
||||
|
||||
|
||||
def test_trailing_punctuation_does_not_break_music_url():
|
||||
out = str(render_content(f"listen: https://music.youtube.com/watch?v={TRACK}."))
|
||||
assert f'src="{BASE}/{TRACK}"' in out
|
||||
assert out.count("<iframe") == 1
|
||||
|
||||
|
||||
def test_only_validated_characters_reach_the_iframe_src():
|
||||
rng = random.Random(7)
|
||||
alphabet = "abcXYZ019_-\"'<>&%/ ;()=:."
|
||||
for _ in range(2000):
|
||||
junk = "".join(rng.choice(alphabet) for _ in range(rng.randint(1, 40)))
|
||||
for url in (
|
||||
f"https://music.youtube.com/watch?v={junk}&list={junk}",
|
||||
f"https://music.youtube.com/playlist?list={junk}",
|
||||
f"https://youtu.be/{junk}",
|
||||
):
|
||||
embed = EmbedParser.parse(url)
|
||||
if embed is None:
|
||||
continue
|
||||
assert re.fullmatch(r"https://www\.youtube\.com/embed/[A-Za-z0-9_\-/?=&]+", EmbedParser.src(embed)), url
|
||||
|
||||
|
||||
def test_start_parsing_is_bounded_across_the_domain():
|
||||
for seconds, suffix in itertools.product(range(0, MAX_START_SECONDS + 5, 997), ("", "s")):
|
||||
embed = EmbedParser.parse(f"https://youtu.be/{VIDEO}?t={seconds}{suffix}")
|
||||
assert embed is not None
|
||||
assert 0 <= embed.start <= MAX_START_SECONDS
|
||||
assert embed.start in (0, seconds)
|
||||
|
||||
|
||||
def test_generated_js_spec_matches_the_python_spec():
|
||||
expected_path = EMBED_JS_PATH
|
||||
current = expected_path.read_text(encoding="utf-8")
|
||||
scratch = expected_path.with_name("embed-spec.check.js")
|
||||
try:
|
||||
EmbedParser.write_module(scratch)
|
||||
assert scratch.read_text(encoding="utf-8") == current
|
||||
finally:
|
||||
scratch.unlink(missing_ok=True)
|
||||
@@ -1,5 +1,6 @@
|
||||
# retoor <retoor@molodetz.nl>
|
||||
|
||||
import re
|
||||
from html.parser import HTMLParser
|
||||
|
||||
from devplacepy.rendering import (
|
||||
@@ -321,6 +322,26 @@ def test_xss_legitimate_youtube_embed_survives_audit():
|
||||
assert_no_executable_html(out)
|
||||
|
||||
|
||||
def test_xss_legitimate_music_embed_survives_audit():
|
||||
out = str(render_content("hear https://music.youtube.com/watch?v=4D7u5KF7SP8 now"))
|
||||
assert "youtube.com/embed/4D7u5KF7SP8" in out
|
||||
assert "embed-music-caption" in out
|
||||
assert_no_executable_html(out)
|
||||
|
||||
|
||||
def test_xss_hostile_music_urls_never_reach_an_iframe():
|
||||
hostile = (
|
||||
"https://music.youtube.com/watch?v=4D7u5KF7SP8&list=x\"onload=alert(1)",
|
||||
"https://music.youtube.com/playlist?list=\"><script>alert(1)</script>",
|
||||
"https://music.youtube.com/watch?v=\"onload=alert(1)//",
|
||||
)
|
||||
for url in hostile:
|
||||
out = str(render_content(url))
|
||||
for src in re.findall(r'<iframe src="([^"]*)"', out):
|
||||
assert re.fullmatch(r"https://www\.youtube\.com/embed/[A-Za-z0-9_\-/?=&;a-z]+", src), src
|
||||
assert_no_executable_html(out)
|
||||
|
||||
|
||||
def test_xss_legitimate_image_embed_survives_audit():
|
||||
out = str(render_content("pic https://example.com/a.png end"))
|
||||
assert "<img" in out
|
||||
|
||||
Reference in New Issue
Block a user