retoor/ad
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Molodetz blogrol volgens DPP-template

Browse Source
This commit is contained in:
retoor
2026-10-05 09:36:20 +02:00
commit f290a4c766
277 changed files with 15758 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.coveragerc
+12
View File
@@ -0,0 +1,12 @@
# retoor <retoor@molodetz.nl>
[run]
source = molodetz
parallel = False
sigterm = True
omit =
tests/*
sitecustomize.py
[report]
show_missing = True
skip_empty = True
.dockerignore
+6
View File
@@ -0,0 +1,6 @@
# retoor <retoor@molodetz.nl>
.venv
data
out
.env
.git
.editorconfig
+19
View File
@@ -0,0 +1,19 @@
# retoor <retoor@molodetz.nl>
root = true
[*]
indent_style = space
indent_size = 4
charset = utf-8
end_of_line = lf
insert_final_newline = true
trim_trailing_whitespace = true
[*.{js,css,html}]
indent_size = 2
[*.md]
trim_trailing_whitespace = false
[Makefile]
indent_style = tab
.env.example
+23
View File
@@ -0,0 +1,23 @@
# retoor <retoor@molodetz.nl>
# Copy to .env (chmod 600). Never commit .env.
SECRET_KEY=change-me-to-a-long-random-string
MOLODETZ_ADMIN_USERNAME=retoor
MOLODETZ_ADMIN_EMAIL=retoor@molodetz.nl
MOLODETZ_ADMIN_PASSWORD=
MOLODETZ_PORT=8088
MOLODETZ_TEST_PORT=8099
MOLODETZ_SITE_URL=
MOLODETZ_DATA_DIR=
MOLODETZ_DISABLE_SERVICES=
MOLODETZ_DISABLE_RATE_LIMIT=
MOLODETZ_RATE_LIMIT=60
MOLODETZ_TEMPLATE_AUTO_RELOAD=1
MOLODETZ_STATIC_VERSION=
MOLODETZ_LOG_LEVEL=INFO
MOLODETZ_OUTBOUND_PROXY_URL=
MOLODETZ_BACKUP_OFFLOAD_REMOTE=
MOLODETZ_SITEMAP_TTL=3600
MOLODETZ_LANDING_TTL=30
MOLODETZ_UNREAD_TTL=10
PLAYWRIGHT_HEADLESS=1
PLAYWRIGHT_SLOW_MO=0
.githooks/pre-commit
Executable
+21
View File
@@ -0,0 +1,21 @@
#!/usr/bin/env bash
# retoor <retoor@molodetz.nl>
set -euo pipefail
if git rev-parse -q --verify MERGE_HEAD >/dev/null; then
exit 0
fi
if git diff --cached -U0 -- pyproject.toml | grep -q '^+version = '; then
exit 0
fi
python3 - <<'PY'
import re
from pathlib import Path
path = Path("pyproject.toml")
text = path.read_text(encoding="utf-8")
match = re.search(r'^version = "(\d+)\.(\d+)\.(\d+)"', text, re.M)
if match:
major, minor, patch = map(int, match.groups())
text = text.replace(match.group(0), f'version = "{major}.{minor}.{patch + 1}"', 1)
path.write_text(text, encoding="utf-8")
PY
git add pyproject.toml
.gitignore
+11
View File
@@ -0,0 +1,11 @@
# retoor <retoor@molodetz.nl>
.venv/
data/
out/
.env
__pycache__/
*.pyc
*.egg-info/
.pytest_cache/
.coverage
.coverage.*
ARCHITECTURE.md
+27
View File
@@ -0,0 +1,27 @@
<!-- retoor <retoor@molodetz.nl> -->
# Architectuur
## Lagen
- `molodetz/main.py`: FastAPI-object (`/swagger`, `/openapi.json`), lifespan (init-lock, `init_db`, bootstrap, services met service-lock), statische mounts (uploads, versioned `/static/v<STATIC_VERSION>/`, fallback), middleware in templatevolgorde, exception handlers, routers.
- `molodetz/config.py`: enige plek voor `load_dotenv`, paden, env-bindingen met prefix `MOLODETZ_`, `APP_VERSION` via `tomllib`, `BOOT_ID`, `STATIC_VERSION`.
- `molodetz/database/`: synchrone `dataset`-handle (SQLAlchemy NullPool, WAL-PRAGMA's), per-tabel modules, soft delete met registries, `init_db` met indexen en `ANALYZE`. Geen async driver; routes lopen in de threadpool waar nodig.
- `molodetz/routers/`: een module per oppervlak. Elke HTML-route geeft JSON bij `Accept: application/json` via `responses.respond` en een Pydantic-uitvoerschema uit `schemas/`.
- `molodetz/templating.py`: de enige `Jinja2Templates`. Globals `static_url`, `local_dt`, `avatar_url`, `render_content` en meer.
- `molodetz/rendering.py`: server-side markdown (mistune, escape aan), emoji, URL-allowlist, streepjesnormalisatie. Gepubliceerde inhoud wordt altijd server-side gerenderd.
- `molodetz/stealth.py`: enige fabriek voor uitgaande HTTP-clients (httpx met curl_cffi-transport, `net_guard` SSRF-bescherming). `link_check.py` gebruikt hem voor de repo-linkcontrole bij aanmeldingen.
- `molodetz/services/`: achtergrondservices (presence, housekeeping, backup) onder een `ServiceManager`, zichtbaar en bestuurbaar via `/admin/services`.
- `molodetz/gallery.py`: flyers en memes uit `static/media/`, Pillow-thumbnails (webp) als blobs onder `data/uploads`, perceptuele hash (imagehash) tegen duplicaten.
- `molodetz/cli/`: argparse-CLI `molodetz`.
- `molodetz/static/js/`: ES6-modules zonder bundler. `Application.js` laadt `components/index.js` (custom elements in light DOM) en zet singletons op `window.app`. Vendor: marked, DOMPurify, highlight.js als ES-modules in `static/vendor/`.
- `molodetz/static/css/`: handgeschreven, tokens in `variables.css`, breakpoints exact 360/480/768/1024.
## Auth
Sessiecookie (HttpOnly, SameSite=Lax, Secure achter TLS), rijen in `sessions`. Daarnaast `X-API-KEY`, `Authorization: Bearer <api key>` en Basic voor scripts. Wachtwoorden met passlib PBKDF2-SHA256. Geen JWT, geen OAuth. De eerste gebruiker is de beheerder uit `.env`; er is geen publieke registratie.
## Datastromen
- Schrijven: beheerder -> `/admin/posts` -> `content.create_post` / `edit_post` -> SQLite -> server-render bij lezen (lru-cache op tekst).
- Aanmelden: bezoeker -> `POST /binnen` (honeypot, rate limit) -> `join_requests` -> melding aan beheerders -> `/admin/joins`.
- Galerij: start -> `sync_gallery` -> `media`-tabel + thumbnails -> `/flyers`, `/memes` met lightbox.
Makefile
+124
View File
@@ -0,0 +1,124 @@
# retoor <retoor@molodetz.nl>
SHELL := /bin/bash
export PYTHONDONTWRITEBYTECODE := 1
PY := .venv/bin/python
PIP := .venv/bin/pip
STAMP := .venv/.stamp
PKG := molodetz
HOST ?= 127.0.0.1
PORT ?= 8088
WORKERS ?= 2
LOCUST_PORT ?= 8097
LOCUST_DATA := /tmp/molodetz-locust-data
PYTEST := $(PY) -m pytest
.PHONY: install dev prod test test-headed test-unit test-api test-e2e test-fast test-failed \
test-first-failure test-slowest test-cache-clean check-ui preflight coverage coverage-headed \
coverage-html locust locust-headless prune prune-dry-run tree tree-loc zip delete-pyc clean
$(STAMP): pyproject.toml
python3 -m venv .venv
$(PIP) install --upgrade pip
$(PIP) install -e ".[dev]"
$(PY) -m playwright install chromium
touch $(STAMP)
install: $(STAMP)
git config core.hooksPath .githooks || true
dev: $(STAMP)
$(PY) -m uvicorn $(PKG).main:app --host $(HOST) --port $(PORT) --reload --reload-dir $(PKG) --backlog 4096
prod: $(STAMP)
MOLODETZ_STATIC_VERSION=$$(date +%s) MOLODETZ_TEMPLATE_AUTO_RELOAD=0 $(PY) -m uvicorn $(PKG).main:app --host $(HOST) --port $(PORT) \
--workers $(WORKERS) --backlog 8192 --proxy-headers --forwarded-allow-ips='*'
test: $(STAMP)
$(PYTEST)
test-headed: $(STAMP)
PLAYWRIGHT_HEADLESS=0 PLAYWRIGHT_SLOW_MO=150 $(PYTEST)
test-unit: $(STAMP)
$(PYTEST) tests/unit
test-api: $(STAMP)
$(PYTEST) tests/api
test-e2e: $(STAMP)
$(PYTEST) tests/e2e
test-fast: $(STAMP)
$(PYTEST) tests/unit tests/api
test-failed: $(STAMP)
$(PYTEST) --lf
test-first-failure: $(STAMP)
$(PYTEST) -x
test-slowest: $(STAMP)
$(PYTEST) --durations=40
test-cache-clean:
rm -rf .pytest_cache
check-ui: $(STAMP)
$(PY) -m $(PKG).gates
preflight: $(STAMP)
$(PY) -c "import $(PKG).main; print('import ok')"
$(PY) -m pyflakes $(PKG) tests
for f in $$(find $(PKG)/static/js -name '*.js'); do node --check "$$f" 2>/dev/null || node --input-type=module --check < "$$f" || exit 1; done
$(PY) -m $(PKG).cli system sql-lint
$(MAKE) check-ui
coverage: $(STAMP)
rm -f .coverage .coverage.*
COVERAGE_PROCESS_START=$(CURDIR)/.coveragerc PYTHONPATH=$(CURDIR) $(PY) -m coverage run -m pytest
$(PY) -m coverage combine || true
$(PY) -m coverage report
coverage-headed: $(STAMP)
PLAYWRIGHT_HEADLESS=0 $(MAKE) coverage
coverage-html: coverage
$(PY) -m coverage html -d out/coverage
locust: $(STAMP)
$(MAKE) locust-run LOCUST_ARGS=""
locust-headless: $(STAMP)
$(MAKE) locust-run LOCUST_ARGS="--headless -u 20 -r 5 -t 120s --html out/locust.html"
locust-run:
rm -rf $(LOCUST_DATA); mkdir -p $(LOCUST_DATA) out
-fuser -k $(LOCUST_PORT)/tcp 2>/dev/null
MOLODETZ_DATA_DIR=$(LOCUST_DATA) MOLODETZ_ADMIN_PASSWORD=locust-$$RANDOM$$RANDOM MOLODETZ_DISABLE_RATE_LIMIT=1 \
$(PY) -m uvicorn $(PKG).main:app --host 127.0.0.1 --port $(LOCUST_PORT) --workers 2 & echo $$! > $(LOCUST_DATA)/server.pid
for i in $$(seq 1 60); do curl -fs http://127.0.0.1:$(LOCUST_PORT)/health >/dev/null && break; sleep 0.5; done
-$(PY) -m locust -f locustfile.py --host http://127.0.0.1:$(LOCUST_PORT) $(LOCUST_ARGS)
-kill $$(cat $(LOCUST_DATA)/server.pid)
rm -rf $(LOCUST_DATA)
prune: $(STAMP)
bin/maintenance.sh
prune-dry-run: $(STAMP)
bin/maintenance.sh --dry-run
tree:
find . -path ./.venv -prune -o -path ./.git -prune -o -path ./data -prune -o -type f -print | sort
tree-loc:
find $(PKG) tests -name '*.py' -o -name '*.js' -not -path '*/vendor/*' -not -path '*/generated/*' | xargs wc -l | sort -n
zip:
mkdir -p out && git archive --format=zip -o out/$(PKG).zip HEAD
delete-pyc:
find . -path ./.venv -prune -o \( -name '__pycache__' -o -name '*.pyc' \) -exec rm -rf {} +
clean: delete-pyc
rm -rf .venv build dist *.egg-info .pytest_cache .coverage .coverage.* out/coverage
README.md
+42
View File
@@ -0,0 +1,42 @@
<!-- retoor <retoor@molodetz.nl> -->
# Molodetz
Blogrol van de Molodetz-community. Rustig, serif, licht provocerend. Berichten zijn markdown in SQLite, geschreven door retoor via een beheeromgeving. Flyers en memes zijn galerijen. Binnen is een echt aanmeldformulier ("Ik schrijf mee").
Gebouwd volgens de bevroren DPP-template (`/workspace/dpp/dptemplate.md`). Zie `SELECTION.md` voor de modulekeuze en `COMPLIANCE.md` voor de toetsing.
## Starten
```
make install # venv in .venv, pip install -e ".[dev]", Chromium, git hooks
cp .env.example .env # alleen als er nog geen .env is; chmod 600 .env
make dev # http://127.0.0.1:8088, reload
make prod # workers, gepinde static-versie
```
Zonder make:
```
.venv/bin/python -m uvicorn molodetz.main:app --host 127.0.0.1 --port 8088
```
De eerste start maakt de beheerder aan uit `MOLODETZ_ADMIN_USERNAME` / `MOLODETZ_ADMIN_EMAIL` / `MOLODETZ_ADMIN_PASSWORD` in `.env`, zaait de placeholderberichten eenmalig en synchroniseert de galerij. Inloggen via `/auth/login`, beheer via `/admin`.
## Testen
```
make test # unit + api + e2e (Playwright, headless)
make test-headed # zichtbaar met slow motion
make check-ui # zes statische poorten + em-dash + json_error-volgorde
make preflight # import, pyflakes, JS-syntax, sqlglot-lint, check-ui
make coverage
make locust-headless
```
## CLI
`molodetz --help`: rollen, API-sleutels, posts list/import (.md en .pdf via pypdf), aanmeldingen, back-ups, prunes, sql-lint, emoji regenerate.
## Data
Alles onder `data/` (database `molodetz.db`, uploads, back-ups, sleutels, locks). `.env` en `data/` staan in `.gitignore`.
SELECTION.md
+42
View File
@@ -0,0 +1,42 @@
<!-- retoor <retoor@molodetz.nl> -->
# Feature selection: Molodetz
Package: `molodetz`
CLI: `molodetz`
Env prefix: `MOLODETZ`
Template: `/workspace/dpp/dptemplate.md` (frozen, section 0.1)
Calm elite serif community blog roll for retoor. Dutch. Posts,
flyers, memes, people, join intent. Admin authoring via sessions.
This file is the only place this project narrows the template
(section 0.3). Narrowing means omitting unselected modules,
never altering rules.
| Section | Class | Selected | Date | Reason |
|---------|-------|----------|------|--------|
| 10 Search optimization | CONVENTION | yes | 2026-10-05 | Public blog; robots + sitemap + base SEO (section 10) |
| 12 Files and blobs | CONVENTION | yes | 2026-10-05 | Gallery images under static; uploads path ready (section 12) |
| 13 Outbound HTTP | CONVENTION | yes | 2026-10-05 | Stealth factory shipped; no live outbound at launch (section 13) |
| 14.3 AI text services | OPTIONAL | no | 2026-10-05 | OPTIONAL omitted (section 0.2 minimal-implementation) |
| 14.4 Async job services | CONVENTION | no | 2026-10-05 | No durable job kinds yet (section 0.2) |
| 15 AI gateway | OPTIONAL | no | 2026-10-05 | OPTIONAL omitted (section 0.2) |
| 16 Assistant | OPTIONAL | no | 2026-10-05 | OPTIONAL omitted (section 0.2) |
| 17 Realtime | MIXED | yes | 2026-10-05 | MIXED: presence CORE only; messaging/pubsub deferred |
| 18 Moderation and safety | OPTIONAL | no | 2026-10-05 | OPTIONAL omitted; join form is owner-private ledger |
| 20.2 Points, levels, badges | OPTIONAL | no | 2026-10-05 | OPTIONAL omitted (section 0.2) |
| 22 Push providers | OPTIONAL | no | 2026-10-05 | OPTIONAL omitted (section 0.2) |
| 24.1 Containers | OPTIONAL | no | 2026-10-05 | OPTIONAL omitted (section 0.2) |
| 24.2 Content import | OPTIONAL | no | 2026-10-05 | OPTIONAL omitted (section 0.2) |
| 24.3 Synthetic-user fleet | OPTIONAL | no | 2026-10-05 | OPTIONAL omitted (section 0.2) |
| 24.4 Issue tracker | OPTIONAL | no | 2026-10-05 | OPTIONAL omitted (section 0.2) |
| 24.5 Compatibility shim | OPTIONAL | no | 2026-10-05 | OPTIONAL omitted (section 0.2) |
| 24.6 Read-only data API | OPTIONAL | no | 2026-10-05 | OPTIONAL omitted (section 0.2) |
| 24.7 Legacy RPC bridge | OPTIONAL | no | 2026-10-05 | OPTIONAL omitted (section 0.2) |
| 24.8 Play modules | OPTIONAL | no | 2026-10-05 | OPTIONAL omitted (section 0.2) |
| 24.9 Diagnostics and research | OPTIONAL | no | 2026-10-05 | OPTIONAL omitted (section 0.2) |
| 24.10 Onboarding prompt | OPTIONAL | no | 2026-10-05 | OPTIONAL omitted (section 0.2) |
| 28.2 Docker | OPTIONAL | no | 2026-10-05 | Bare metal only for this deploy (section 28.1) |
| 31 Correctness verification | CONVENTION | no | 2026-10-05 | No spendable resource / multi-path race features yet |
Optional libraries (chromadb, weasyprint, cairosvg, faker): not selected.
bin/maintenance.sh
Executable
+16
View File
@@ -0,0 +1,16 @@
#!/usr/bin/env bash
# retoor <retoor@molodetz.nl>
set -euo pipefail
cd "$(dirname "$0")/.."
DRY=""
if [[ "${1:-}" == "--dry-run" ]]; then
DRY="--dry-run"
fi
PY=.venv/bin/python
DATA="${MOLODETZ_DATA_DIR:-data}"
echo "Schijfgebruik voor:"; du -sh "$DATA" 2>/dev/null || true
$PY -m molodetz.cli sessions prune $DRY
$PY -m molodetz.cli audit prune $DRY
$PY -m molodetz.cli backups prune $DRY
$PY -m molodetz.cli system prune $DRY
echo "Schijfgebruik na:"; du -sh "$DATA" 2>/dev/null || true
events.md
+40
View File
@@ -0,0 +1,40 @@
<!-- retoor <retoor@molodetz.nl> -->
# Audit event catalogue
Every audited action goes through `molodetz/utils/audit.py` (`record` for requests, `record_system` for services, CLI and bootstrap). Rows land in `audit_log` with typed links in `audit_links`. Category comes from the key prefix (`CATEGORY_BY_PREFIX`).
| Key | Category | Emitted by | Result values |
|---|---|---|---|
| `auth.login` | auth | `routers/auth.py` login | ok |
| `auth.login.failed` | auth | `routers/auth.py` login | denied |
| `auth.logout` | auth | `routers/auth.py` logout | ok |
| `auth.terms.accept` | auth | `routers/voorwaarden.py` | ok |
| `admin.bootstrap` | admin | `bootstrap.py` first admin from env | ok |
| `admin.settings.save` | admin | `routers/admin/settings.py` | ok |
| `admin.user.role` | admin | `routers/admin/users.py`, CLI `role set` | ok, denied |
| `admin.trash.restore` | admin | `routers/admin/trash.py` | ok |
| `admin.trash.purge` | admin | `routers/admin/trash.py` | ok |
| `post.create` | content | `content.create_post` | ok |
| `post.edit` | content | `content.edit_post` | ok |
| `post.edit.denied` | content | `routers/admin/posts.py` | denied |
| `post.delete` | content | `content.delete_post` (soft delete) | ok |
| `post.delete.denied` | content | `routers/admin/posts.py` | denied |
| `post.seed` | content | `bootstrap.py` placeholder seed | ok |
| `join.request.create` | community | `routers/binnen.py` | ok |
| `join.request.honeypot` | community | `routers/binnen.py` honeypot hit | denied |
| `join.request.status` | community | `routers/admin/joins.py` | ok |
| `join.request.check` | community | `routers/admin/joins.py` repo-link check | ok, error |
| `security.authorization.denied` | security | `utils/auth.py` guards | denied |
| `security.ratelimit.exceeded` | security | `main.py` rate-limit middleware | denied |
| `security.crawler.blocked` | security | `main.py` crawler guard | denied |
| `security.maintenance.blocked` | security | `main.py` maintenance gate | denied |
| `security.suspension.refused` | security | `utils/auth.py refuse_suspended` | denied |
| `service.command.start` / `.stop` / `.run` / `.clear` | service | `routers/admin/services.py` | ok |
| `service.config.save` | service | `routers/admin/services.py` | ok |
| `backup.enqueue` | service | `routers/admin/backups.py` | ok |
| `backup.download` | service | `routers/admin/backups.py` (primary admin only) | ok, denied |
| `backup.finished` | service | `services/backup.py` | ok |
| `backup.failed` | service | `services/backup.py` | error |
| `profile.apikey.regenerate` | account | `routers/profile.py`, CLI `apikey reset` | ok |
| `profile.avatar.regenerate` | account | `routers/profile.py` | ok |
| `notification.create` | system | `utils/notifications.py` funnel | ok |
hooks/prod_guard.py
+22
View File
@@ -0,0 +1,22 @@
# retoor <retoor@molodetz.nl>
import json
import re
import sys
PROTECTED = re.compile(r"(\bdata/|molodetz\.db|\.env\b|rm\s+-rf\s+/|DROP\s+TABLE|git\s+push\s+--force)", re.IGNORECASE)
def main():
try:
event = json.load(sys.stdin)
except json.JSONDecodeError:
return 0
command = json.dumps(event.get("tool_input", event))
if PROTECTED.search(command):
print("prod_guard: command touches production data or secrets; confirm manually.", file=sys.stderr)
return 2
return 0
if __name__ == "__main__":
sys.exit(main())
locustfile.py
+39
View File
@@ -0,0 +1,39 @@
# retoor <retoor@molodetz.nl>
import random
from locust import HttpUser, between, task
PAGES = ["/", "/rol", "/standaard", "/flyers", "/memes", "/mensen", "/binnen", "/docs"]
class Reader(HttpUser):
wait_time = between(0.5, 2.0)
def on_start(self):
response = self.client.get("/rol", headers={"Accept": "application/json"}, name="/rol [json]")
self.slugs = [post["slug"] for post in response.json().get("posts", [])] if response.ok else []
@task(6)
def home(self):
self.client.get("/")
@task(4)
def listing(self):
self.client.get(random.choice(PAGES), name="[page]")
@task(3)
def post(self):
if self.slugs:
self.client.get(f"/posts/{random.choice(self.slugs)}", name="/posts/[slug]")
@task(1)
def json_face(self):
self.client.get("/standaard", headers={"Accept": "application/json"}, name="/standaard [json]")
@task(1)
def join_form(self):
self.client.post(
"/binnen",
data={"name": f"Load {random.randint(1, 10**6)}", "contact": "load@example.invalid", "repo_url": "", "message": "", "website": ""},
name="/binnen [post]",
)
molodetz/__init__.py
+1
View File
@@ -0,0 +1 @@
# retoor <retoor@molodetz.nl>
molodetz/assets.py
+8
View File
@@ -0,0 +1,8 @@
# retoor <retoor@molodetz.nl>
from molodetz import config
def static_url(path):
if not path.startswith("/static/") or path.startswith("/static/uploads/"):
return path
return f"/static/v{config.STATIC_VERSION}/{path[len('/static/'):]}"
molodetz/attachments.py
+92
View File
@@ -0,0 +1,92 @@
# retoor <retoor@molodetz.nl>
from pathlib import Path
from molodetz import config
from molodetz.database import db, shard_path
IMAGE_EXTENSIONS = frozenset({"jpg", "jpeg", "png", "gif", "webp", "avif"})
DOCUMENT_EXTENSIONS = frozenset({"pdf", "md", "txt"})
INLINE_EXTENSIONS = IMAGE_EXTENSIONS | frozenset({"mp4", "webm", "mp3", "ogg", "wav", "svg"})
def allowed_extensions():
return IMAGE_EXTENSIONS | DOCUMENT_EXTENSIONS
def extension_of(name):
return Path(name).suffix.lower().lstrip(".")
def is_image(name):
return extension_of(name) in IMAGE_EXTENSIONS
def format_size(size):
size = float(size or 0)
for unit in ("B", "KB", "MB", "GB"):
if size < 1024 or unit == "GB":
return f"{size:.0f} {unit}" if unit == "B" else f"{size:.1f} {unit}"
size /= 1024
return f"{size:.1f} GB"
def icon_for(name):
ext = extension_of(name)
if ext in IMAGE_EXTENSIONS:
return "image"
if ext == "pdf":
return "pdf"
return "file"
def blob_location(uid, extension):
directory = f"attachments/{shard_path(uid)}"
stored_name = f"{uid.replace('-', '')}.{extension}"
return directory, stored_name
def blob_path(directory, stored_name):
root = config.DATA_PATHS["uploads"].resolve()
path = (root / directory / stored_name).resolve()
if root not in path.parents:
raise ValueError("blob path escapes the uploads root")
return path
def blob_url(directory, stored_name):
return f"/static/uploads/{directory}/{stored_name}"
def write_blob(uid, extension, data):
if extension not in allowed_extensions():
raise ValueError(f"extension {extension} not allowed")
directory, stored_name = blob_location(uid, extension)
path = blob_path(directory, stored_name)
path.parent.mkdir(parents=True, exist_ok=True)
path.write_bytes(data)
return directory, stored_name
def unlink_blob(directory, stored_name):
try:
blob_path(directory, stored_name).unlink(missing_ok=True)
return True
except (OSError, ValueError):
return False
def referenced_blob_paths():
paths = set()
if "media_items" in db.tables:
for row in db["media_items"].find(deleted_at=None):
if row.get("thumb_directory") and row.get("thumb_name"):
paths.add(blob_path(row["thumb_directory"], row["thumb_name"]))
return paths
def orphan_blobs():
root = config.DATA_PATHS["attachments"]
if not root.exists():
return []
referenced = referenced_blob_paths()
return [path for path in root.rglob("*") if path.is_file() and path.resolve() not in referenced]
molodetz/avatar.py
+42
View File
@@ -0,0 +1,42 @@
# retoor <retoor@molodetz.nl>
import hashlib
from functools import lru_cache
from html import escape
PALETTE = (
("#1b1a17", "#e2a100"),
("#141414", "#c9c2b2"),
("#1d1b16", "#b98a2e"),
("#121417", "#8fa3a8"),
("#191512", "#d9b779"),
)
@lru_cache(maxsize=4096)
def render_avatar_svg(seed, size=64):
size = max(16, min(int(size), 512))
try:
digest = hashlib.sha256(seed.encode("utf-8")).digest()
background, ink = PALETTE[digest[0] % len(PALETTE)]
cells = []
unit = size / 5
for row in range(5):
for col in range(3):
if digest[1 + row * 3 + col] % 2 == 0:
for mirrored in {col, 4 - col}:
cells.append(
f'<rect x="{mirrored * unit:.2f}" y="{row * unit:.2f}" width="{unit:.2f}" height="{unit:.2f}" fill="{ink}"/>'
)
body = "".join(cells)
except Exception:
background, ink = PALETTE[0]
letter = escape((seed or "?")[:1].upper())
body = f'<text x="50%" y="58%" text-anchor="middle" font-family="Georgia, serif" font-size="{size // 2}" fill="{ink}">{letter}</text>'
return (
f'<svg xmlns="http://www.w3.org/2000/svg" width="{size}" height="{size}" viewBox="0 0 {size} {size}">'
f'<rect width="{size}" height="{size}" fill="{background}"/>{body}</svg>'
)
def avatar_url(seed, size=64):
return f"/avatar/svg/{seed}?size={size}"
molodetz/bootstrap.py
+34
View File
@@ -0,0 +1,34 @@
# retoor <retoor@molodetz.nl>
import logging
from molodetz import config
from molodetz.content import seed_placeholder_posts
from molodetz.database import count_users, create_user, get_setting, get_user_by_username, set_setting
from molodetz.gallery import sync_gallery
from molodetz.utils.audit import record_system
from molodetz.utils.auth import hash_password
logger = logging.getLogger(__name__)
ADMIN_BIO = "Begint de rol. Schrijft over het werk, het afmaken en het oordeel."
def bootstrap_admin():
if count_users() > 0:
return get_user_by_username(config.ADMIN_USERNAME)
if not config.ADMIN_PASSWORD:
logger.warning("no users and MOLODETZ_ADMIN_PASSWORD unset; skipping admin bootstrap")
return None
user = create_user(config.ADMIN_USERNAME, config.ADMIN_EMAIL, hash_password(config.ADMIN_PASSWORD), bio=ADMIN_BIO)
record_system("admin.bootstrap", payload={"username": user["username"]}, targets=[("user", user["uid"])])
return user
def bootstrap_content():
admin = bootstrap_admin()
if admin and get_setting("seed_posts_done", "0") != "1":
created = seed_placeholder_posts(admin)
set_setting("seed_posts_done", "1")
record_system("post.seed", payload={"count": len(created)})
synced = sync_gallery()
logger.info("gallery synced: %d items", synced)
molodetz/cache.py
+54
View File
@@ -0,0 +1,54 @@
# retoor <retoor@molodetz.nl>
import threading
import time
class TTLCache:
def __init__(self, ttl=60, max_size=1000):
self.ttl = ttl
self.max_size = max_size
self._data = {}
self._lock = threading.Lock()
def get(self, key, default=None):
if self.ttl <= 0:
return default
with self._lock:
item = self._data.get(key)
if item is None:
return default
expires, value = item
if expires < time.monotonic():
self._data.pop(key, None)
return default
return value
def set(self, key, value):
if self.ttl <= 0:
return
with self._lock:
if len(self._data) >= self.max_size:
self._evict()
self._data[key] = (time.monotonic() + self.ttl, value)
def pop(self, key):
with self._lock:
self._data.pop(key, None)
def clear(self):
with self._lock:
self._data.clear()
def __contains__(self, key):
return self.get(key, _MISSING) is not _MISSING
def _evict(self):
now = time.monotonic()
expired = [k for k, (exp, _) in self._data.items() if exp < now]
for key in expired:
self._data.pop(key, None)
while len(self._data) >= self.max_size:
self._data.pop(next(iter(self._data)))
_MISSING = object()
molodetz/cli/__init__.py
+86
View File
@@ -0,0 +1,86 @@
# retoor <retoor@molodetz.nl>
import argparse
import sys
from molodetz.cli import commands
def build_parser():
parser = argparse.ArgumentParser(prog="molodetz", description="Beheer-CLI voor Molodetz.")
groups = parser.add_subparsers(dest="group", required=True)
role = groups.add_parser("role", help="Rollen lezen en zetten").add_subparsers(dest="action", required=True)
role_get = role.add_parser("get")
role_get.add_argument("username")
role_get.set_defaults(handler=commands.role_get)
role_set = role.add_parser("set")
role_set.add_argument("username")
role_set.add_argument("role", choices=["Member", "Admin"])
role_set.set_defaults(handler=commands.role_set)
apikey = groups.add_parser("apikey", help="API-sleutels").add_subparsers(dest="action", required=True)
key_get = apikey.add_parser("get")
key_get.add_argument("username")
key_get.set_defaults(handler=commands.apikey_get)
key_reset = apikey.add_parser("reset")
key_reset.add_argument("username")
key_reset.set_defaults(handler=commands.apikey_reset)
apikey.add_parser("backfill").set_defaults(handler=commands.apikey_backfill)
posts = groups.add_parser("posts", help="Berichten").add_subparsers(dest="action", required=True)
posts.add_parser("list").set_defaults(handler=commands.posts_list)
posts_import = posts.add_parser("import", help="Importeer .md of .pdf als concept")
posts_import.add_argument("path")
posts_import.add_argument("--topic", default="rol", choices=["rol", "standaard"])
posts_import.add_argument("--title")
posts_import.add_argument("--publish", action="store_true")
posts_import.set_defaults(handler=commands.posts_import)
joins = groups.add_parser("joins", help="Aanmeldingen").add_subparsers(dest="action", required=True)
joins.add_parser("list").set_defaults(handler=commands.joins_list)
backups = groups.add_parser("backups", help="Back-ups").add_subparsers(dest="action", required=True)
backups.add_parser("list").set_defaults(handler=commands.backups_list)
backup_run = backups.add_parser("run")
backup_run.add_argument("--target", default="database", choices=["database", "uploads", "keys", "full"])
backup_run.add_argument("--codec", default="zstd", choices=["zstd", "brotli"])
backup_run.set_defaults(handler=commands.backups_run)
backup_prune = backups.add_parser("prune")
backup_prune.add_argument("--keep", type=int, default=14)
backup_prune.add_argument("--dry-run", action="store_true")
backup_prune.set_defaults(handler=commands.backups_prune)
backup_clear = backups.add_parser("clear")
backup_clear.add_argument("--dry-run", action="store_true")
backup_clear.set_defaults(handler=commands.backups_clear)
sessions = groups.add_parser("sessions", help="Sessies").add_subparsers(dest="action", required=True)
sessions_prune = sessions.add_parser("prune")
sessions_prune.add_argument("--dry-run", action="store_true")
sessions_prune.set_defaults(handler=commands.sessions_prune)
audit = groups.add_parser("audit", help="Auditlog").add_subparsers(dest="action", required=True)
audit_prune = audit.add_parser("prune")
audit_prune.add_argument("--days", type=int, default=365)
audit_prune.add_argument("--dry-run", action="store_true")
audit_prune.set_defaults(handler=commands.audit_prune)
system = groups.add_parser("system", help="Systeemonderhoud").add_subparsers(dest="action", required=True)
system_prune = system.add_parser("prune", help="Verwijder wees-blobs")
system_prune.add_argument("--dry-run", action="store_true")
system_prune.set_defaults(handler=commands.system_prune)
system.add_parser("sql-lint", help="Parse alle SQL-literals met sqlglot").set_defaults(handler=commands.system_sql_lint)
emoji = groups.add_parser("emoji", help="Emoji-kaart").add_subparsers(dest="action", required=True)
emoji.add_parser("regenerate").set_defaults(handler=commands.emoji_regenerate)
return parser
def main(argv=None):
parser = build_parser()
args = parser.parse_args(argv)
commands.prepare()
return args.handler(args) or 0
if __name__ == "__main__":
sys.exit(main())
molodetz/cli/__main__.py
+6
View File
@@ -0,0 +1,6 @@
# retoor <retoor@molodetz.nl>
import sys
from molodetz.cli import main
sys.exit(main())
molodetz/cli/commands.py
+154
View File
@@ -0,0 +1,154 @@
# retoor <retoor@molodetz.nl>
from datetime import datetime, timedelta, timezone
from pathlib import Path
from pypdf import PdfReader
from molodetz import config
from molodetz.attachments import orphan_blobs
from molodetz.bootstrap import bootstrap_admin
from molodetz.content import create_post
from molodetz.database import (
all_posts,
enqueue_backup,
generate_uid,
get_user_by_username,
init_db,
list_backups,
list_join_requests,
list_users,
prune_audit,
prune_expired_sessions,
set_role,
update_user,
)
from molodetz.database.sql_lint import lint
from molodetz.emoji_builder import write_js_module
from molodetz.services.backup import clear_backups, process_backup, prune_backups
from molodetz.utils.audit import record_system
def prepare():
config.ensure_data_dirs()
init_db()
def _user(username):
user = get_user_by_username(username)
if user is None:
raise SystemExit(f"Onbekende gebruiker: {username}")
return user
def role_get(args):
print(_user(args.username).get("role"))
def role_set(args):
user = _user(args.username)
set_role(user["uid"], args.role)
record_system("admin.user.role", actor_kind="cli", origin="cli", payload={"role": args.role}, targets=[("user", user["uid"])])
print(f"{user['username']} -> {args.role}")
def apikey_get(args):
print(_user(args.username).get("api_key"))
def apikey_reset(args):
user = _user(args.username)
key = generate_uid()
update_user(user["uid"], api_key=key)
record_system("profile.apikey.regenerate", actor_kind="cli", origin="cli", targets=[("user", user["uid"])])
print(key)
def apikey_backfill(args):
changed = 0
for user in list_users(limit=100000):
if not user.get("api_key"):
update_user(user["uid"], api_key=generate_uid())
changed += 1
print(f"{changed} sleutels aangevuld")
def posts_list(args):
for post in all_posts():
flag = " [placeholder]" if post.get("is_placeholder") else ""
print(f"{post['uid']} {post['status']:<9} {post['topic']:<9} {post['title']}{flag}")
def read_source(path):
path = Path(path)
if path.suffix.lower() == ".pdf":
reader = PdfReader(str(path))
return "\n\n".join((page.extract_text() or "").strip() for page in reader.pages).strip()
return path.read_text(encoding="utf-8")
def posts_import(args):
path = Path(args.path)
if not path.is_file():
raise SystemExit(f"Bestand niet gevonden: {path}")
body = read_source(path)
title = args.title or path.stem.replace("-", " ").replace("_", " ").strip().capitalize()
author = bootstrap_admin() or _user(config.ADMIN_USERNAME)
post = create_post(author, title, body, args.topic, "published" if args.publish else "draft")
print(f"Geimporteerd: {post['uid']} {post['title']}")
def joins_list(args):
for row in list_join_requests(limit=1000):
print(f"{row['created_at']} {row['status']:<9} {row['name']} {row['contact']} {row.get('repo_url') or '-'}")
def backups_list(args):
for row in list_backups(limit=1000):
print(f"{row['uid']} {row['status']:<8} {row['target']:<9} {row.get('stored_name') or '-'} {row.get('size_bytes') or 0}")
def backups_run(args):
row = enqueue_backup(args.target, None)
ok = process_backup(row, args.codec)
print("klaar" if ok else "mislukt")
return 0 if ok else 1
def backups_prune(args):
count = prune_backups(args.keep, dry_run=args.dry_run)
print(f"{count} archieven {'zouden worden verwijderd' if args.dry_run else 'verwijderd'}")
def backups_clear(args):
count = clear_backups(dry_run=args.dry_run)
print(f"{count} archieven {'zouden worden verwijderd' if args.dry_run else 'verwijderd'}")
def sessions_prune(args):
print(f"{prune_expired_sessions(dry_run=args.dry_run)} verlopen sessies")
def audit_prune(args):
cutoff = (datetime.now(timezone.utc) - timedelta(days=args.days)).isoformat()
print(f"{prune_audit(cutoff, dry_run=args.dry_run)} auditregels ouder dan {args.days} dagen")
def system_prune(args):
orphans = orphan_blobs()
for path in orphans:
print(("zou verwijderen: " if args.dry_run else "verwijderd: ") + str(path))
if not args.dry_run:
path.unlink(missing_ok=True)
print(f"{len(orphans)} wees-blobs")
def system_sql_lint(args):
problems = lint()
for problem in problems:
print(problem)
print(f"{len(problems)} problemen")
return 1 if problems else 0
def emoji_regenerate(args):
print(write_js_module())
molodetz/config.py
+89
View File
@@ -0,0 +1,89 @@
# retoor <retoor@molodetz.nl>
import os
import time
import tomllib
from pathlib import Path
from dotenv import load_dotenv
ROOT = Path(__file__).resolve().parents[1]
PKG_ROOT = Path(__file__).resolve().parent
load_dotenv(ROOT / ".env")
with (ROOT / "pyproject.toml").open("rb") as handle:
APP_VERSION = tomllib.load(handle)["project"]["version"]
APP_TITLE = "Molodetz"
APP_PORT = 8088
DEFAULT_TEST_PORT = 8099
def _env(name, default=""):
return os.environ.get(f"MOLODETZ_{name}", default)
def _env_int(name, default):
try:
return int(_env(name, str(default)))
except ValueError:
return default
DATA_DIR = Path(_env("DATA_DIR", str(ROOT / "data"))).resolve()
DATABASE_URL = _env("DATABASE_URL", f"sqlite:///{DATA_DIR / 'molodetz.db'}")
PORT = _env_int("PORT", APP_PORT)
TEST_PORT = _env_int("TEST_PORT", DEFAULT_TEST_PORT)
SECRET_KEY = os.environ.get("SECRET_KEY", "molodetz-development-secret-change-in-production")
SITE_URL = _env("SITE_URL", "").rstrip("/")
INTERNAL_BASE_URL = _env("INTERNAL_BASE_URL", f"http://localhost:{PORT}").rstrip("/")
DISABLE_SERVICES = _env("DISABLE_SERVICES") == "1"
DISABLE_RATE_LIMIT = _env("DISABLE_RATE_LIMIT") == "1"
RATE_LIMIT = _env_int("RATE_LIMIT", 60)
WEB_WORKERS = max(1, _env_int("WEB_WORKERS", os.cpu_count() or 1))
TEMPLATE_AUTO_RELOAD = _env("TEMPLATE_AUTO_RELOAD", "1") == "1"
BOOT_ID = _env("STATIC_VERSION") or str(int(time.time()))
STATIC_VERSION = f"{APP_VERSION}-{BOOT_ID}"
LOG_LEVEL = _env("LOG_LEVEL", "INFO")
PRESENCE_TIMEOUT_SECONDS = _env_int("PRESENCE_TIMEOUT_SECONDS", 60)
PRESENCE_ONLINE_LIMIT = _env_int("PRESENCE_ONLINE_LIMIT", 30)
PRESENCE_TRACK_LIMIT = _env_int("PRESENCE_TRACK_LIMIT", 500)
PRESENCE_ONLINE_MARGIN_SECONDS = _env_int("PRESENCE_ONLINE_MARGIN_SECONDS", 20)
OUTBOUND_PROXY_URL = _env("OUTBOUND_PROXY_URL", "")
RCLONE_BIN = _env("RCLONE_BIN", "rclone")
RCLONE_CONFIG = _env("RCLONE_CONFIG", str(Path.home() / ".config/rclone/rclone.conf"))
BACKUP_OFFLOAD_REMOTE = _env("BACKUP_OFFLOAD_REMOTE", "")
SITEMAP_TTL = _env_int("SITEMAP_TTL", 3600)
LANDING_TTL = _env_int("LANDING_TTL", 30)
UNREAD_TTL = _env_int("UNREAD_TTL", 10)
PLAYWRIGHT_HEADLESS = os.environ.get("PLAYWRIGHT_HEADLESS", "1") == "1"
PLAYWRIGHT_SLOW_MO = int(os.environ.get("PLAYWRIGHT_SLOW_MO", "0") or 0)
ADMIN_USERNAME = _env("ADMIN_USERNAME", "retoor")
ADMIN_EMAIL = _env("ADMIN_EMAIL", "retoor@molodetz.nl")
ADMIN_PASSWORD = _env("ADMIN_PASSWORD", "")
STATIC_DIR = PKG_ROOT / "static"
TEMPLATES_DIR = PKG_ROOT / "templates"
MEDIA_SOURCE_DIR = STATIC_DIR / "media"
DATA_PATHS: dict[str, Path] = {
"data": DATA_DIR,
"uploads": DATA_DIR / "uploads",
"attachments": DATA_DIR / "uploads" / "attachments",
"backups": DATA_DIR / "backups",
"backup_staging": DATA_DIR / "backups" / "staging",
"seo_reports": DATA_DIR / "seo_reports",
"keys": DATA_DIR / "keys",
"locks": DATA_DIR / "locks",
}
INIT_LOCK_FILE = DATA_PATHS["locks"] / "init.lock"
SERVICE_LOCK_FILE = DATA_PATHS["locks"] / "service.lock"
WEB_LOCK_FILE = DATA_PATHS["locks"] / "web.lock"
NOTIFICATION_KEY_FILE = DATA_PATHS["keys"] / "notification.key"
def ensure_data_dirs():
for path in DATA_PATHS.values():
path.mkdir(parents=True, exist_ok=True)
molodetz/constants.py
+93
View File
@@ -0,0 +1,93 @@
# retoor <retoor@molodetz.nl>
ROLE_ADMIN = "Admin"
ROLE_MEMBER = "Member"
ROLES = (ROLE_ADMIN, ROLE_MEMBER)
TOPICS = {
"rol": "Rol",
"standaard": "Standaard",
}
POST_STATUSES = ("draft", "published")
MEDIA_KINDS = {
"flyer": "Flyers",
"meme": "Memes",
}
JOIN_STATUSES = ("open", "contacted", "accepted", "declined")
SOFT_DELETE_TABLES = (
"users",
"sessions",
"posts",
"join_requests",
"notifications",
"media_items",
)
UNREPORTABLE_TABLES = {
"users": "Only the operator holds accounts; no public user content.",
"sessions": "Private credential rows.",
"posts": "Authored by administrators only; moderation module not selected.",
"join_requests": "Owner-private ledger visible to administrators only.",
"notifications": "Owner-private inbox rows.",
"media_items": "Operator-curated gallery assets.",
}
REPORTABLE_TARGETS: frozenset[str] = frozenset()
NOTIFICATION_TYPES = {
"join.request": "Nieuwe aanmelding via Binnen",
"post.published": "Bericht gepubliceerd",
"backup.finished": "Back-up afgerond",
"backup.failed": "Back-up mislukt",
"auth.apikey": "API-sleutel vernieuwd",
"system.maintenance": "Onderhoudsmodus gewijzigd",
}
CRAWLER_PRIVATE_PREFIXES = ("/notifications", "/admin", "/profile")
CRAWLER_MARKERS = (
"bot",
"crawler",
"spider",
"slurp",
"crawl",
"facebookexternalhit",
"bingpreview",
)
TERMS_EXEMPT_PREFIXES = (
"/static",
"/avatar",
"/auth",
"/docs",
"/voorwaarden",
"/privacy",
)
MAINTENANCE_ALLOWED_PREFIXES = ("/static", "/avatar", "/auth", "/admin")
RATE_LIMIT_EXEMPT_PREFIXES: tuple[str, ...] = ()
GZIP_SKIP_PREFIXES: tuple[str, ...] = ()
SETTINGS_DEFAULTS = {
"site_title": "Molodetz",
"site_tagline": "De kring, niet de reclame.",
"maintenance_mode": "0",
"maintenance_message": "Molodetz is even dicht voor onderhoud.",
"rate_limit_per_minute": "60",
"rate_limit_window_seconds": "60",
"session_max_age_days": "7",
"session_remember_days": "30",
"terms_version": "1",
"friendly_404": "1",
"audit_retention_days": "365",
"outbound_proxy_url": "",
}
CLIENT_RENDER_TEMPLATES = frozenset({"admin/post_form.html"})
HTTP_CLIENT_ALLOWLIST = frozenset({"molodetz/stealth.py"})
molodetz/content.py
+72
View File
@@ -0,0 +1,72 @@
# retoor <retoor@molodetz.nl>
from molodetz.database import (
insert_post,
soft_delete,
update_post_row,
now_iso,
db,
)
from molodetz.utils.audit import record
from molodetz.utils.notifications import create_notification
PLACEHOLDER_NOTE = (
"\n\n> Placeholder. Dit is overgenomen uit de oude Molodetz-pagina. "
"retoor vervangt dit later door zijn eigen verhaal."
)
def create_post(user, title, body, topic, status="draft", is_placeholder=False, request=None):
if is_placeholder and PLACEHOLDER_NOTE.strip() not in body:
body = body.rstrip() + PLACEHOLDER_NOTE
row = insert_post(user["uid"], title, body, topic, status=status, is_placeholder=is_placeholder)
record(request, "post.create", payload={"uid": row["uid"], "status": status}, targets=[("post", row["uid"])], user=user)
if status == "published":
create_notification(user["uid"], "post.published", f"Gepubliceerd: {title}", f"/posts/{row['slug']}")
return row
def edit_post(post, user, title, body, topic, status, is_placeholder=False, request=None):
fields = {
"title": title,
"body": body,
"topic": topic,
"status": status,
"is_placeholder": bool(is_placeholder),
"updated_at": now_iso(),
}
if status == "published" and not post.get("published_at"):
fields["published_at"] = now_iso()
update_post_row(post["uid"], **fields)
record(request, "post.edit", payload={"uid": post["uid"]}, targets=[("post", post["uid"])], user=user)
return {**post, **fields}
def delete_post(post, actor, request=None):
stamp = now_iso()
soft_delete("posts", actor["uid"], stamp=stamp, uid=post["uid"])
record(
request,
"post.delete",
payload={"uid": post["uid"], "stamp": stamp},
targets=[("post", post["uid"])],
user=actor,
)
return stamp
def seed_placeholder_posts(admin):
seeds = [
("standaard", "Eenendertig keuzes. Jij hebt er nul.", "Python 3.12. FastAPI. Jinja2. SQLite. Geen JWT, want dat is een hobby. Geen JavaScript-framework, want dat is een catalogus. Het document zegt 107 keer MUST. Wie daar \"het hangt ervan af\" van maakt, heeft geen smaak. Die heeft een roadmap van andermans tutorials."),
("rol", "Een zes is geen ambitie.", "Wie tevreden is met voldoende, bouwt voldoende. Molodetz begint waar dat ophoudt."),
("rol", "Af is een eigenschap.", "Een project dat open blijft, is een mening. Een project dat af is, is een feit."),
("rol", "De assistent is geen senior.", "AI versnelt het werk. Het vervangt het oordeel niet. Wie dat door elkaar haalt, levert een chatlog in."),
("rol", "Geen cursus.", "Kennis gaat hier van mensen naar mensen. Niet via een ebook dat belooft dat je in een weekend iemand anders wordt."),
]
created = []
for topic, title, body in seeds:
existing = db["posts"].find_one(title=title, deleted_at=None) if "posts" in db.tables else None
if existing:
continue
created.append(create_post(admin, title, body, topic, status="published", is_placeholder=topic == "rol"))
return created
molodetz/curl_transport.py
+61
View File
@@ -0,0 +1,61 @@
# retoor <retoor@molodetz.nl>
import httpx
try:
from curl_cffi import CurlHttpVersion
from curl_cffi.requests import AsyncSession
CURL_AVAILABLE = True
except ImportError:
CURL_AVAILABLE = False
IMPERSONATE_PROFILE = "chrome"
class CurlImpersonationTransport(httpx.AsyncBaseTransport):
def __init__(self, proxy=None, timeout=30.0, verify=True):
self.proxy = proxy or None
self.timeout = timeout
self.verify = verify
self._session = None
def _session_for(self):
if self._session is None:
self._session = AsyncSession(impersonate=IMPERSONATE_PROFILE, verify=self.verify)
return self._session
async def handle_async_request(self, request):
url = str(request.url)
http_version = CurlHttpVersion.V1_1 if request.url.scheme == "http" else None
headers = {
key.decode("latin-1"): value.decode("latin-1")
for key, value in request.headers.raw
if key.lower() not in (b"host", b"content-length", b"user-agent", b"accept-encoding", b"connection")
}
body = await request.aread()
kwargs = {
"headers": headers,
"data": body or None,
"timeout": self.timeout,
"allow_redirects": False,
"proxy": self.proxy,
}
if http_version is not None:
kwargs["http_version"] = http_version
response = await self._session_for().request(request.method, url, **kwargs)
response_headers = [
(key, value)
for key, value in response.headers.multi_items()
if key.lower() not in ("content-encoding", "content-length", "transfer-encoding")
]
return httpx.Response(
status_code=response.status_code,
headers=response_headers,
content=response.content,
request=request,
)
async def aclose(self):
if self._session is not None:
await self._session.close()
self._session = None
molodetz/database/__init__.py
+169
View File
@@ -0,0 +1,169 @@
# retoor <retoor@molodetz.nl>
from molodetz.database.audit import count_audit, insert_audit, list_audit, prune_audit
from molodetz.database.backups import (
delete_backup_rows,
enqueue_backup,
get_backup,
list_backups,
pending_backups,
update_backup,
)
from molodetz.database.core import (
db,
generate_uid,
make_combined_slug,
now_iso,
parse_iso,
refresh_snapshot,
resolve_by_slug,
shard_path,
table_exists,
)
from molodetz.database.init_db import init_db
from molodetz.database.joins import (
count_open_join_requests,
get_join_request,
insert_join_request,
list_join_requests,
update_join_request,
)
from molodetz.database.media import count_media, media_by_kind, upsert_media
from molodetz.database.notifications import insert_notification, list_notifications, mark_all_read, unread_count
from molodetz.database.pagination import build_pagination, paginate
from molodetz.database.posts import (
all_posts,
count_published,
get_post,
get_post_by_slug,
insert_post,
post_counts_by_author,
published_posts,
update_post_row,
)
from molodetz.database.services_state import ensure_service_state, get_service_state, write_service_state
from molodetz.database.sessions import (
create_session,
end_all_sessions_for,
end_session,
get_session,
hard_delete_session,
prune_expired_sessions,
)
from molodetz.database.settings import (
all_settings,
bump_cache_version,
get_int_setting,
get_setting,
set_setting,
sync_local_cache,
)
from molodetz.database.soft_delete import (
count_deleted,
list_deleted,
purge,
purge_event,
restore,
restore_event,
soft_delete,
soft_delete_in,
)
from molodetz.database.users import (
admin_uids_ordered,
count_users,
create_user,
get_user,
get_user_by_api_key,
get_user_by_email,
get_user_by_username,
invalidate_admins_cache,
list_users,
public_people,
set_role,
update_user,
users_by_uids,
)
from molodetz.database.visits import daily_totals, flush_visits, top_paths
__all__ = [
"all_posts",
"all_settings",
"admin_uids_ordered",
"build_pagination",
"bump_cache_version",
"count_audit",
"count_deleted",
"count_media",
"count_open_join_requests",
"count_published",
"count_users",
"create_session",
"create_user",
"daily_totals",
"db",
"delete_backup_rows",
"end_all_sessions_for",
"end_session",
"enqueue_backup",
"ensure_service_state",
"flush_visits",
"generate_uid",
"get_backup",
"get_int_setting",
"get_join_request",
"get_post",
"get_post_by_slug",
"get_service_state",
"get_session",
"get_setting",
"get_user",
"get_user_by_api_key",
"get_user_by_email",
"get_user_by_username",
"hard_delete_session",
"init_db",
"insert_audit",
"insert_join_request",
"insert_notification",
"insert_post",
"invalidate_admins_cache",
"list_audit",
"list_backups",
"list_deleted",
"list_join_requests",
"list_notifications",
"list_users",
"make_combined_slug",
"mark_all_read",
"media_by_kind",
"now_iso",
"paginate",
"parse_iso",
"pending_backups",
"post_counts_by_author",
"prune_audit",
"prune_expired_sessions",
"public_people",
"published_posts",
"purge",
"purge_event",
"refresh_snapshot",
"resolve_by_slug",
"restore",
"restore_event",
"set_role",
"set_setting",
"shard_path",
"soft_delete",
"soft_delete_in",
"sync_local_cache",
"table_exists",
"top_paths",
"unread_count",
"update_backup",
"update_join_request",
"update_post_row",
"update_user",
"upsert_media",
"users_by_uids",
"write_service_state",
]
molodetz/database/audit.py
+37
View File
@@ -0,0 +1,37 @@
# retoor <retoor@molodetz.nl>
from molodetz.database.core import db
def insert_audit(row, links):
db["audit_log"].insert(row)
for link in links:
db["audit_links"].insert(link)
def list_audit(limit=200, category=None):
if "audit_log" not in db.tables:
return []
criteria = {}
if category:
criteria["category"] = category
return list(db["audit_log"].find(order_by=["-created_at"], _limit=limit, **criteria))
def count_audit():
if "audit_log" not in db.tables:
return 0
return db["audit_log"].count()
def prune_audit(before_iso, dry_run=False):
if "audit_log" not in db.tables:
return 0
rows = list(db.query("SELECT uid FROM audit_log WHERE created_at < :before", before=before_iso))
if rows and not dry_run:
with db:
db.query(
"DELETE FROM audit_links WHERE audit_uid IN (SELECT uid FROM audit_log WHERE created_at < :before)",
before=before_iso,
)
db.query("DELETE FROM audit_log WHERE created_at < :before", before=before_iso)
return len(rows)
molodetz/database/backups.py
+47
View File
@@ -0,0 +1,47 @@
# retoor <retoor@molodetz.nl>
from molodetz.database.core import db, generate_uid, now_iso
def enqueue_backup(target, requested_by):
row = {
"uid": generate_uid(),
"target": target,
"status": "pending",
"requested_by": requested_by,
"directory": None,
"stored_name": None,
"size_bytes": None,
"error": None,
"created_at": now_iso(),
"finished_at": None,
}
db["backups"].insert(row)
return row
def pending_backups():
if "backups" not in db.tables:
return []
return list(db["backups"].find(status="pending", order_by=["uid"]))
def list_backups(limit=100):
if "backups" not in db.tables:
return []
return list(db["backups"].find(order_by=["-created_at"], _limit=limit))
def get_backup(uid):
if "backups" not in db.tables:
return None
return db["backups"].find_one(uid=uid)
def update_backup(uid, **fields):
fields["uid"] = uid
db["backups"].update(fields, ["uid"])
def delete_backup_rows(uids):
for uid in uids:
db["backups"].delete(uid=uid)
molodetz/database/core.py
+116
View File
@@ -0,0 +1,116 @@
# retoor <retoor@molodetz.nl>
import logging
import re
from datetime import datetime, timezone
import dataset
import sqlalchemy
import uuid_utils
from sqlalchemy.pool import NullPool
from molodetz import config
logger = logging.getLogger(__name__)
ON_CONNECT_STATEMENTS = [
"PRAGMA journal_mode=WAL",
"PRAGMA synchronous=NORMAL",
"PRAGMA busy_timeout=30000",
"PRAGMA cache_size=-8000",
"PRAGMA temp_store=MEMORY",
"PRAGMA mmap_size=268435456",
]
config.ensure_data_dirs()
db = dataset.connect(
config.DATABASE_URL,
engine_kwargs={
"connect_args": {"timeout": 30, "check_same_thread": False},
"poolclass": NullPool,
},
on_connect_statements=ON_CONNECT_STATEMENTS,
)
def generate_uid():
return str(uuid_utils.uuid7())
def now_iso():
return datetime.now(timezone.utc).isoformat()
def parse_iso(value):
if not value:
return None
try:
parsed = datetime.fromisoformat(str(value))
except ValueError:
return None
if parsed.tzinfo is None:
parsed = parsed.replace(tzinfo=timezone.utc)
return parsed
def refresh_snapshot():
try:
if getattr(db.local, "tx", None):
return
conn = db.executable
if conn.in_transaction():
conn.commit()
except Exception as exc:
logger.warning("refresh_snapshot failed: %s", exc)
def table_exists(name):
return name in db.tables
def in_clause(values, prefix="v"):
params = {f"{prefix}{index}": value for index, value in enumerate(values)}
placeholders = ", ".join(f":{key}" for key in params)
return placeholders, params
def atomic_update(sql, **params):
with db:
result = db.executable.execute(sqlalchemy.text(sql), params)
return result.rowcount
def run_write(sql, **params):
with db:
db.query(sql, **params)
def slugify(text):
lowered = (text or "").lower()
replaced = re.sub(r"[^a-z0-9-]", "-", lowered)
return re.sub(r"-+", "-", replaced).strip("-")
def make_combined_slug(text, uid):
tail = uid.replace("-", "")[-12:]
slug = slugify(text)
return f"{slug}-{tail}" if slug else tail
def shard_path(uid):
tail = uid.replace("-", "")
return f"{tail[-2:]}/{tail[-4:-2]}"
def resolve_by_slug(table_name, slug, include_deleted=False):
if table_name not in db.tables:
return None
table = db[table_name]
for key in ("slug", "uid"):
criteria = {key: slug}
if not include_deleted:
criteria["deleted_at"] = None
row = table.find_one(**criteria)
if row:
return row
return None
molodetz/database/init_db.py
+233
View File
@@ -0,0 +1,233 @@
# retoor <retoor@molodetz.nl>
import logging
from molodetz import config
from molodetz.constants import SETTINGS_DEFAULTS
from molodetz.database.core import db, generate_uid, now_iso
logger = logging.getLogger(__name__)
TABLE_EXAMPLES = {
"users": {
"uid": "x",
"username": "x",
"email": "x@x.nl",
"password_hash": "x",
"role": "Member",
"api_key": "x",
"bio": "",
"avatar_seed": None,
"is_active": True,
"last_seen_at": None,
"suspended_until": None,
"suspend_reason": None,
"terms_version": "1",
"created_at": now_iso(),
"updated_at": now_iso(),
"deleted_at": None,
"deleted_by": None,
},
"sessions": {
"uid": "x",
"token": "x",
"user_uid": "x",
"created_at": now_iso(),
"expires_at": now_iso(),
"deleted_at": None,
"deleted_by": None,
},
"posts": {
"uid": "x",
"slug": "x",
"title": "x",
"body": "x",
"topic": "rol",
"status": "draft",
"is_placeholder": False,
"user_uid": "x",
"published_at": None,
"created_at": now_iso(),
"updated_at": now_iso(),
"deleted_at": None,
"deleted_by": None,
},
"join_requests": {
"uid": "x",
"name": "x",
"contact": "x",
"repo_url": None,
"message": "",
"status": "open",
"ip_hash": None,
"link_title": None,
"link_checked_at": None,
"link_status": None,
"created_at": now_iso(),
"updated_at": now_iso(),
"deleted_at": None,
"deleted_by": None,
},
"media_items": {
"uid": "x",
"filename": "x.jpg",
"kind": "flyer",
"caption": "",
"position": 0,
"width": 0,
"height": 0,
"phash": None,
"thumb_directory": None,
"thumb_name": None,
"created_at": now_iso(),
"updated_at": now_iso(),
"deleted_at": None,
"deleted_by": None,
},
"notifications": {
"uid": "x",
"user_uid": "x",
"kind": "system.maintenance",
"message": "x",
"url": "/",
"read_at": None,
"created_at": now_iso(),
"deleted_at": None,
"deleted_by": None,
},
"site_settings": {"uid": "setting_x", "key": "x", "value": "x", "updated_at": now_iso()},
"cache_state": {"name": "settings", "version": 0, "updated_at": now_iso()},
"service_state": {
"name": "backup",
"status": "stopped",
"last_run": None,
"next_run": None,
"started_at": None,
"heartbeat": None,
"logs": "[]",
"metrics": "{}",
"updated_at": now_iso(),
},
"audit_log": {
"uid": "x",
"event_key": "auth.login",
"category": "auth",
"actor_kind": "user",
"actor_uid": None,
"origin": "web",
"result": "ok",
"ip": None,
"user_agent": None,
"message": None,
"payload": "{}",
"created_at": now_iso(),
},
"audit_links": {"audit_uid": "x", "target_type": "user", "target_uid": "x"},
"visit_stats": {"day": "2026-01-01", "path": "/", "status": 200, "hits": 0},
"backups": {
"uid": "x",
"target": "database",
"status": "pending",
"requested_by": "x",
"directory": None,
"stored_name": None,
"size_bytes": None,
"error": None,
"created_at": now_iso(),
"finished_at": None,
},
}
def _ensure_table(name, example):
table = db.get_table(name)
for column, value in example.items():
table.create_column_by_example(column, value)
return table
def _index(sql):
try:
with db:
db.query(sql)
except Exception as exc:
logger.warning("index ensure failed: %s (%s)", sql, exc)
def _seed_settings():
if "site_settings" not in db.tables:
return
for key, value in SETTINGS_DEFAULTS.items():
uid = f"default_{key}"
if db["site_settings"].find_one(uid=uid) is None:
db["site_settings"].insert({"uid": uid, "key": key, "value": value, "updated_at": now_iso()})
def _seed_notification_key():
path = config.NOTIFICATION_KEY_FILE
if not path.exists():
path.write_text(generate_uid(), encoding="utf-8")
path.chmod(0o600)
def _backfill_api_keys():
if "users" not in db.tables:
return
with db:
db.query("UPDATE users SET api_key = :key WHERE api_key IS NULL OR api_key = ''", key=generate_uid())
def init_db():
for name, example in TABLE_EXAMPLES.items():
_ensure_table(name, example)
_index("CREATE UNIQUE INDEX IF NOT EXISTS idx_users_uid ON users(uid)")
_index("CREATE UNIQUE INDEX IF NOT EXISTS idx_users_username ON users(username)")
_index("CREATE UNIQUE INDEX IF NOT EXISTS idx_users_email ON users(email)")
_index("CREATE UNIQUE INDEX IF NOT EXISTS idx_users_api_key ON users(api_key)")
_index("CREATE INDEX IF NOT EXISTS idx_users_created_at ON users(created_at)")
_index("CREATE INDEX IF NOT EXISTS idx_users_trash ON users(deleted_at) WHERE deleted_at IS NOT NULL")
_index("CREATE UNIQUE INDEX IF NOT EXISTS idx_sessions_uid ON sessions(uid)")
_index("CREATE UNIQUE INDEX IF NOT EXISTS idx_sessions_token ON sessions(token)")
_index("CREATE INDEX IF NOT EXISTS idx_sessions_user ON sessions(user_uid, created_at)")
_index("CREATE INDEX IF NOT EXISTS idx_sessions_trash ON sessions(deleted_at) WHERE deleted_at IS NOT NULL")
_index("CREATE UNIQUE INDEX IF NOT EXISTS idx_posts_uid ON posts(uid)")
_index("CREATE UNIQUE INDEX IF NOT EXISTS idx_posts_slug ON posts(slug)")
_index(
"CREATE INDEX IF NOT EXISTS idx_posts_published ON posts(topic, published_at) "
"WHERE deleted_at IS NULL AND status = 'published'"
)
_index("CREATE INDEX IF NOT EXISTS idx_posts_author ON posts(user_uid, created_at)")
_index("CREATE INDEX IF NOT EXISTS idx_posts_trash ON posts(deleted_at) WHERE deleted_at IS NOT NULL")
_index("CREATE UNIQUE INDEX IF NOT EXISTS idx_join_requests_uid ON join_requests(uid)")
_index("CREATE INDEX IF NOT EXISTS idx_join_requests_status ON join_requests(status, created_at)")
_index("CREATE INDEX IF NOT EXISTS idx_join_requests_trash ON join_requests(deleted_at) WHERE deleted_at IS NOT NULL")
_index("CREATE UNIQUE INDEX IF NOT EXISTS idx_media_items_uid ON media_items(uid)")
_index("CREATE UNIQUE INDEX IF NOT EXISTS idx_media_items_filename ON media_items(filename)")
_index("CREATE INDEX IF NOT EXISTS idx_media_items_kind ON media_items(kind, position)")
_index("CREATE UNIQUE INDEX IF NOT EXISTS idx_notifications_uid ON notifications(uid)")
_index("CREATE INDEX IF NOT EXISTS idx_notifications_user ON notifications(user_uid, created_at)")
_index("CREATE UNIQUE INDEX IF NOT EXISTS idx_site_settings_uid ON site_settings(uid)")
_index("CREATE UNIQUE INDEX IF NOT EXISTS idx_site_settings_key ON site_settings(key)")
_index("CREATE UNIQUE INDEX IF NOT EXISTS idx_cache_state_name ON cache_state(name)")
_index("CREATE UNIQUE INDEX IF NOT EXISTS idx_service_state_name ON service_state(name)")
_index("CREATE UNIQUE INDEX IF NOT EXISTS idx_audit_log_uid ON audit_log(uid)")
_index("CREATE INDEX IF NOT EXISTS idx_audit_log_created ON audit_log(created_at)")
_index("CREATE INDEX IF NOT EXISTS idx_audit_links_audit ON audit_links(audit_uid)")
_index("CREATE UNIQUE INDEX IF NOT EXISTS idx_visit_stats_day_path ON visit_stats(day, path, status)")
_index("CREATE UNIQUE INDEX IF NOT EXISTS idx_backups_uid ON backups(uid)")
_seed_settings()
_seed_notification_key()
_backfill_api_keys()
with db:
has_stats = next(iter(db.query("SELECT name FROM sqlite_master WHERE type='table' AND name='sqlite_stat1'")), None)
if has_stats is None:
db.query("ANALYZE")
db.query("PRAGMA optimize")
molodetz/database/joins.py
+47
View File
@@ -0,0 +1,47 @@
# retoor <retoor@molodetz.nl>
from molodetz.database.core import db, generate_uid, now_iso
def insert_join_request(name, contact, repo_url, message, ip_hash):
row = {
"uid": generate_uid(),
"name": name,
"contact": contact,
"repo_url": repo_url or None,
"message": message or "",
"status": "open",
"ip_hash": ip_hash,
"link_title": None,
"link_checked_at": None,
"link_status": None,
"created_at": now_iso(),
"updated_at": now_iso(),
"deleted_at": None,
"deleted_by": None,
}
db["join_requests"].insert(row)
return row
def list_join_requests(limit=500):
if "join_requests" not in db.tables:
return []
return list(db["join_requests"].find(deleted_at=None, order_by=["-created_at"], _limit=limit))
def get_join_request(uid):
if "join_requests" not in db.tables:
return None
return db["join_requests"].find_one(uid=uid, deleted_at=None)
def update_join_request(uid, **fields):
fields["uid"] = uid
fields["updated_at"] = now_iso()
db["join_requests"].update(fields, ["uid"])
def count_open_join_requests():
if "join_requests" not in db.tables:
return 0
return db["join_requests"].count(deleted_at=None, status="open")
molodetz/database/media.py
+45
View File
@@ -0,0 +1,45 @@
# retoor <retoor@molodetz.nl>
from molodetz.database.core import db, generate_uid, now_iso
def media_by_kind(kind):
if "media_items" not in db.tables:
return []
return list(db["media_items"].find(kind=kind, deleted_at=None, order_by=["position", "id"]))
def get_media_by_file(filename):
if "media_items" not in db.tables:
return None
return db["media_items"].find_one(filename=filename)
def upsert_media(filename, kind, caption, position, width, height, phash, thumb_directory, thumb_name):
existing = get_media_by_file(filename)
fields = {
"filename": filename,
"kind": kind,
"caption": caption,
"position": position,
"width": width,
"height": height,
"phash": phash,
"thumb_directory": thumb_directory,
"thumb_name": thumb_name,
"updated_at": now_iso(),
}
if existing:
fields["id"] = existing["id"]
db["media_items"].update(fields, ["id"])
return existing["uid"]
fields.update({"uid": generate_uid(), "created_at": now_iso(), "deleted_at": None, "deleted_by": None})
db["media_items"].insert(fields)
return fields["uid"]
def count_media(kind=None):
if "media_items" not in db.tables:
return 0
if kind:
return db["media_items"].count(kind=kind, deleted_at=None)
return db["media_items"].count(deleted_at=None)
molodetz/database/notifications.py
+55
View File
@@ -0,0 +1,55 @@
# retoor <retoor@molodetz.nl>
from molodetz.cache import TTLCache
from molodetz import config
from molodetz.database.core import db, generate_uid, now_iso
_unread_cache = TTLCache(ttl=config.UNREAD_TTL, max_size=1000)
def insert_notification(user_uid, kind, message, url):
row = {
"uid": generate_uid(),
"user_uid": user_uid,
"kind": kind,
"message": message,
"url": url,
"read_at": None,
"created_at": now_iso(),
"deleted_at": None,
"deleted_by": None,
}
db["notifications"].insert(row)
_unread_cache.pop(user_uid)
return row
def list_notifications(user_uid, limit=100):
if "notifications" not in db.tables:
return []
return list(db["notifications"].find(user_uid=user_uid, deleted_at=None, order_by=["-created_at"], _limit=limit))
def unread_count(user_uid):
if not user_uid:
return 0
cached = _unread_cache.get(user_uid)
if cached is not None:
return cached
count = 0
if "notifications" in db.tables:
count = db["notifications"].count(user_uid=user_uid, deleted_at=None, read_at=None)
_unread_cache.set(user_uid, count)
return count
def mark_all_read(user_uid):
if "notifications" not in db.tables:
return
with db:
db.query(
"UPDATE notifications SET read_at = :stamp WHERE user_uid = :user_uid "
"AND read_at IS NULL AND deleted_at IS NULL",
stamp=now_iso(),
user_uid=user_uid,
)
_unread_cache.pop(user_uid)
molodetz/database/pagination.py
+36
View File
@@ -0,0 +1,36 @@
# retoor <retoor@molodetz.nl>
import math
from molodetz.database.core import db
def paginate(table_name, before=None, order="desc", cursor_field="created_at", limit=25, **filters):
if table_name not in db.tables:
return [], None
table = db[table_name]
criteria = dict(filters)
if "deleted_at" not in criteria:
criteria["deleted_at"] = None
if before:
criteria[cursor_field] = {"<": before} if order == "desc" else {">": before}
order_by = [f"-{cursor_field}"] if order == "desc" else [cursor_field]
rows = list(table.find(_limit=limit + 1, order_by=order_by, **criteria))
next_cursor = None
if len(rows) > limit:
rows = rows[:limit]
next_cursor = rows[-1][cursor_field]
return rows, next_cursor
def build_pagination(page, total, per_page):
pages = max(1, math.ceil(total / per_page)) if per_page else 1
page = min(max(1, page), pages)
return {
"page": page,
"pages": pages,
"total": total,
"per_page": per_page,
"offset": (page - 1) * per_page,
"has_prev": page > 1,
"has_next": page < pages,
}
molodetz/database/posts.py
+74
View File
@@ -0,0 +1,74 @@
# retoor <retoor@molodetz.nl>
from molodetz.database.core import db, generate_uid, in_clause, make_combined_slug, now_iso, resolve_by_slug
from molodetz.database.pagination import paginate
def insert_post(user_uid, title, body, topic, status="draft", is_placeholder=False, created_at=None):
uid = generate_uid()
stamp = created_at or now_iso()
row = {
"uid": uid,
"slug": make_combined_slug(title, uid),
"title": title,
"body": body,
"topic": topic,
"status": status,
"is_placeholder": bool(is_placeholder),
"user_uid": user_uid,
"published_at": stamp if status == "published" else None,
"created_at": stamp,
"updated_at": stamp,
"deleted_at": None,
"deleted_by": None,
}
db["posts"].insert(row)
return row
def update_post_row(uid, **fields):
fields["uid"] = uid
db["posts"].update(fields, ["uid"])
def get_post(uid):
if "posts" not in db.tables:
return None
return db["posts"].find_one(uid=uid, deleted_at=None)
def get_post_by_slug(slug):
return resolve_by_slug("posts", slug)
def published_posts(topic=None, before=None, limit=25, user_uid=None):
filters = {"status": "published"}
if topic:
filters["topic"] = topic
if user_uid:
filters["user_uid"] = user_uid
return paginate("posts", before=before, cursor_field="published_at", limit=limit, **filters)
def all_posts(limit=200):
if "posts" not in db.tables:
return []
return list(db["posts"].find(deleted_at=None, order_by=["-created_at"], _limit=limit))
def count_published():
if "posts" not in db.tables:
return 0
return db["posts"].count(deleted_at=None, status="published")
def post_counts_by_author(user_uids):
user_uids = [uid for uid in set(user_uids) if uid]
if not user_uids or "posts" not in db.tables:
return {}
placeholders, params = in_clause(user_uids)
rows = db.query(
f"SELECT user_uid, COUNT(*) AS n FROM posts WHERE user_uid IN ({placeholders}) "
"AND deleted_at IS NULL AND status = 'published' GROUP BY user_uid",
**params,
)
return {row["user_uid"]: int(row["n"]) for row in rows}
molodetz/database/services_state.py
+38
View File
@@ -0,0 +1,38 @@
# retoor <retoor@molodetz.nl>
import json
from molodetz.database.core import db, now_iso
def get_service_state(name):
if "service_state" not in db.tables:
return None
return db["service_state"].find_one(name=name)
def ensure_service_state(name):
row = get_service_state(name)
if row:
return row["id"]
return db["service_state"].insert(
{
"name": name,
"status": "stopped",
"last_run": None,
"next_run": None,
"started_at": None,
"heartbeat": None,
"logs": "[]",
"metrics": "{}",
"updated_at": now_iso(),
}
)
def write_service_state(row_id, **fields):
for key in ("logs", "metrics"):
if key in fields and not isinstance(fields[key], str):
fields[key] = json.dumps(fields[key])
fields["id"] = row_id
fields["updated_at"] = now_iso()
db["service_state"].update(fields, ["id"])
molodetz/database/sessions.py
+66
View File
@@ -0,0 +1,66 @@
# retoor <retoor@molodetz.nl>
import secrets
from datetime import datetime, timedelta, timezone
from molodetz.database.core import db, generate_uid, now_iso
def create_session(user_uid, max_age_seconds):
token = secrets.token_hex(32)
expires = datetime.now(timezone.utc) + timedelta(seconds=max_age_seconds)
db["sessions"].insert(
{
"uid": generate_uid(),
"token": token,
"user_uid": user_uid,
"created_at": now_iso(),
"expires_at": expires.isoformat(),
"deleted_at": None,
"deleted_by": None,
}
)
return token
def get_session(token):
if not token or "sessions" not in db.tables:
return None
return db["sessions"].find_one(token=token, deleted_at=None)
def hard_delete_session(session_id):
db["sessions"].delete(id=session_id)
def end_session(token, actor):
if "sessions" not in db.tables:
return
row = db["sessions"].find_one(token=token, deleted_at=None)
if row:
db["sessions"].update({"id": row["id"], "deleted_at": now_iso(), "deleted_by": actor}, ["id"])
def prune_expired_sessions(dry_run=False):
if "sessions" not in db.tables:
return 0
stamp = now_iso()
rows = list(db.query("SELECT id FROM sessions WHERE expires_at < :stamp", stamp=stamp))
if not dry_run and rows:
with db:
db.query("DELETE FROM sessions WHERE expires_at < :stamp", stamp=stamp)
return len(rows)
def end_all_sessions_for(user_uid, actor):
if "sessions" not in db.tables:
return 0
stamp = now_iso()
with db:
db.query(
"UPDATE sessions SET deleted_at = :stamp, deleted_by = :actor "
"WHERE user_uid = :user_uid AND deleted_at IS NULL",
stamp=stamp,
actor=actor,
user_uid=user_uid,
)
return 1
molodetz/database/settings.py
+89
View File
@@ -0,0 +1,89 @@
# retoor <retoor@molodetz.nl>
import logging
from molodetz.cache import TTLCache
from molodetz.constants import SETTINGS_DEFAULTS
from molodetz.database.core import db, now_iso
logger = logging.getLogger(__name__)
_settings_cache = TTLCache(ttl=60, max_size=100)
_version_cache = TTLCache(ttl=1, max_size=100)
_local_versions = {}
def read_cache_version(name):
cached = _version_cache.get(name)
if cached is not None:
return cached
try:
if "cache_state" not in db.tables:
return 0
row = db["cache_state"].find_one(name=name)
version = int(row["version"]) if row else 0
except Exception as exc:
logger.warning("cache version read failed for %s: %s", name, exc)
return 0
_version_cache.set(name, version)
return version
def sync_local_cache(name, cache):
version = read_cache_version(name)
if _local_versions.get(name) != version:
cache.clear()
_local_versions[name] = version
def bump_cache_version(name, cache=None):
try:
with db:
db.query(
"INSERT INTO cache_state (name, version, updated_at) VALUES (:name, 1, :stamp) "
"ON CONFLICT(name) DO UPDATE SET version = version + 1, updated_at = excluded.updated_at",
name=name,
stamp=now_iso(),
)
except Exception as exc:
logger.warning("cache version bump failed for %s: %s", name, exc)
_version_cache.pop(name)
if cache is not None:
cache.clear()
_local_versions.pop(name, None)
def get_setting(key, default=None):
if default is None:
default = SETTINGS_DEFAULTS.get(key, "")
sync_local_cache("settings", _settings_cache)
cached = _settings_cache.get(key)
if cached is not None:
return cached
value = default
if "site_settings" in db.tables:
row = db["site_settings"].find_one(uid=f"setting_{key}")
if row is None:
row = db["site_settings"].find_one(uid=f"default_{key}")
if row is not None and row.get("value") is not None:
value = str(row["value"])
_settings_cache.set(key, value)
return value
def get_int_setting(key, default):
try:
return int(get_setting(key, str(default)))
except (TypeError, ValueError):
return default
def set_setting(key, value):
db["site_settings"].upsert(
{"uid": f"setting_{key}", "key": key, "value": str(value), "updated_at": now_iso()},
["uid"],
)
bump_cache_version("settings", _settings_cache)
def all_settings():
return {key: get_setting(key, default) for key, default in SETTINGS_DEFAULTS.items()}
molodetz/database/soft_delete.py
+104
View File
@@ -0,0 +1,104 @@
# retoor <retoor@molodetz.nl>
from molodetz.constants import SOFT_DELETE_TABLES
from molodetz.database.core import db, in_clause, now_iso
def _check(table_name):
if table_name not in SOFT_DELETE_TABLES:
raise ValueError(f"{table_name} is not soft-deletable")
def soft_delete(table_name, actor, stamp=None, **criteria):
_check(table_name)
if table_name not in db.tables:
return 0
stamp = stamp or now_iso()
table = db[table_name]
count = 0
for row in list(table.find(deleted_at=None, **criteria)):
table.update({"id": row["id"], "deleted_at": stamp, "deleted_by": actor}, ["id"])
count += 1
return count
def soft_delete_in(table_name, column, values, actor, stamp=None):
_check(table_name)
if not values or table_name not in db.tables:
return 0
stamp = stamp or now_iso()
placeholders, params = in_clause(list(values))
with db:
db.query(
f"UPDATE {table_name} SET deleted_at = :stamp, deleted_by = :actor "
f"WHERE {column} IN ({placeholders}) AND deleted_at IS NULL",
stamp=stamp,
actor=actor,
**params,
)
return len(values)
def restore(table_name, uid):
_check(table_name)
if table_name not in db.tables:
return False
row = db[table_name].find_one(uid=uid)
if not row or row.get("deleted_at") is None:
return False
db[table_name].update({"id": row["id"], "deleted_at": None, "deleted_by": None}, ["id"])
return True
def purge(table_name, uid):
_check(table_name)
if table_name not in db.tables:
return False
return bool(db[table_name].delete(uid=uid))
def list_deleted(table_name, limit=200):
_check(table_name)
if table_name not in db.tables:
return []
return list(
db.query(
f"SELECT * FROM {table_name} WHERE deleted_at IS NOT NULL ORDER BY deleted_at DESC LIMIT :limit",
limit=limit,
)
)
def count_deleted(table_name):
_check(table_name)
if table_name not in db.tables:
return 0
row = next(iter(db.query(f"SELECT COUNT(*) AS n FROM {table_name} WHERE deleted_at IS NOT NULL")), None)
return int(row["n"]) if row else 0
def restore_event(stamp):
restored = []
tables = set(db.tables)
for table_name in SOFT_DELETE_TABLES:
if table_name not in tables:
continue
with db:
db.query(
f"UPDATE {table_name} SET deleted_at = NULL, deleted_by = NULL WHERE deleted_at = :stamp",
stamp=stamp,
)
restored.append(table_name)
return restored
def purge_event(stamp):
purged = []
tables = set(db.tables)
for table_name in SOFT_DELETE_TABLES:
if table_name not in tables:
continue
rows = list(db[table_name].find(deleted_at=stamp))
if rows:
db[table_name].delete(deleted_at=stamp)
purged.append((table_name, rows))
return purged
molodetz/database/sql_lint.py
+58
View File
@@ -0,0 +1,58 @@
# retoor <retoor@molodetz.nl>
import ast
from pathlib import Path
import sqlglot
from sqlglot.errors import ParseError
from molodetz import config
SQL_PREFIXES = ("SELECT ", "INSERT ", "UPDATE ", "DELETE ", "CREATE ", "PRAGMA ", "WITH ", "ANALYZE")
def _joined_text(node):
parts = []
for value in node.values:
if isinstance(value, ast.Constant):
parts.append(str(value.value))
else:
parts.append("lint_placeholder")
return "".join(parts)
def _literal_sql(node):
if isinstance(node, ast.JoinedStr):
raw = _joined_text(node)
elif isinstance(node, ast.Constant) and isinstance(node.value, str):
raw = node.value
else:
return None
text = " ".join(raw.split())
return text if text.upper().startswith(SQL_PREFIXES) else None
def collect_sql(root=None):
root = Path(root or config.PKG_ROOT)
found = []
for path in sorted(root.rglob("*.py")):
tree = ast.parse(path.read_text(encoding="utf-8"))
inner = {id(value) for node in ast.walk(tree) if isinstance(node, ast.JoinedStr) for value in node.values}
for node in ast.walk(tree):
if id(node) in inner:
continue
sql = _literal_sql(node)
if sql:
found.append((path, node.lineno, sql))
return found
def lint(root=None):
problems = []
for path, line, sql in collect_sql(root):
if sql.upper().startswith(("PRAGMA", "ANALYZE")):
continue
try:
sqlglot.parse_one(sql, read="sqlite")
except ParseError as exc:
problems.append(f"{path}:{line}: {str(exc).splitlines()[0]}")
return problems
molodetz/database/users.py
+120
View File
@@ -0,0 +1,120 @@
# retoor <retoor@molodetz.nl>
from molodetz.cache import TTLCache
from molodetz.constants import ROLE_ADMIN, ROLE_MEMBER
from molodetz.database.core import db, generate_uid, in_clause, now_iso
from molodetz.database.settings import bump_cache_version, get_setting, sync_local_cache
_admins_cache = TTLCache(ttl=300, max_size=4)
def users_table():
return db.get_table("users")
def get_user(uid):
if not uid or "users" not in db.tables:
return None
return db["users"].find_one(uid=uid, deleted_at=None)
def get_user_by_username(username):
if not username or "users" not in db.tables:
return None
return db["users"].find_one(username=username, deleted_at=None)
def get_user_by_email(email):
if not email or "users" not in db.tables:
return None
return db["users"].find_one(email=email.lower(), deleted_at=None)
def get_user_by_api_key(api_key):
if not api_key or "users" not in db.tables:
return None
return db["users"].find_one(api_key=api_key, deleted_at=None)
def users_by_uids(uids):
uids = [uid for uid in set(uids) if uid]
if not uids or "users" not in db.tables:
return {}
placeholders, params = in_clause(uids)
rows = db.query(
f"SELECT * FROM users WHERE uid IN ({placeholders}) AND deleted_at IS NULL",
**params,
)
return {row["uid"]: dict(row) for row in rows}
def count_users():
if "users" not in db.tables:
return 0
return db["users"].count(deleted_at=None)
def create_user(username, email, password_hash, role=None, bio="", created_at=None):
role = role or (ROLE_ADMIN if count_users() == 0 else ROLE_MEMBER)
uid = generate_uid()
row = {
"uid": uid,
"username": username,
"email": (email or "").lower(),
"password_hash": password_hash,
"role": role,
"api_key": generate_uid(),
"bio": bio,
"avatar_seed": None,
"is_active": True,
"last_seen_at": None,
"suspended_until": None,
"suspend_reason": None,
"terms_version": get_setting("terms_version", "1"),
"created_at": created_at or now_iso(),
"updated_at": now_iso(),
"deleted_at": None,
"deleted_by": None,
}
db["users"].insert(row)
invalidate_admins_cache()
return row
def update_user(uid, **fields):
fields["uid"] = uid
fields["updated_at"] = now_iso()
db["users"].update(fields, ["uid"])
def set_role(uid, role):
update_user(uid, role=role)
invalidate_admins_cache()
def invalidate_admins_cache():
bump_cache_version("admins", _admins_cache)
def admin_uids_ordered():
sync_local_cache("admins", _admins_cache)
cached = _admins_cache.get("ordered")
if cached is not None:
return cached
if "users" not in db.tables:
return []
rows = db["users"].find(role=ROLE_ADMIN, deleted_at=None, order_by=["created_at", "id"])
ordered = [row["uid"] for row in rows]
_admins_cache.set("ordered", ordered)
return ordered
def list_users(limit=200):
if "users" not in db.tables:
return []
return list(db["users"].find(deleted_at=None, order_by=["created_at"], _limit=limit))
def public_people():
if "users" not in db.tables:
return []
return list(db["users"].find(deleted_at=None, is_active=True, order_by=["created_at"]))
molodetz/database/visits.py
+39
View File
@@ -0,0 +1,39 @@
# retoor <retoor@molodetz.nl>
from molodetz.database.core import db
def flush_visits(day, counts):
if not counts:
return
with db:
for (path, status), hits in counts.items():
db.query(
"INSERT INTO visit_stats (day, path, status, hits) VALUES (:day, :path, :status, :hits) "
"ON CONFLICT(day, path, status) DO UPDATE SET hits = hits + excluded.hits",
day=day,
path=path,
status=status,
hits=hits,
)
def top_paths(limit=25):
if "visit_stats" not in db.tables:
return []
return list(
db.query(
"SELECT path, SUM(hits) AS hits FROM visit_stats GROUP BY path ORDER BY hits DESC LIMIT :limit",
limit=limit,
)
)
def daily_totals(limit=30):
if "visit_stats" not in db.tables:
return []
return list(
db.query(
"SELECT day, SUM(hits) AS hits FROM visit_stats GROUP BY day ORDER BY day DESC LIMIT :limit",
limit=limit,
)
)
molodetz/dependencies.py
+29
View File
@@ -0,0 +1,29 @@
# retoor <retoor@molodetz.nl>
import json
from fastapi import Request
from fastapi.exceptions import RequestValidationError
from pydantic import ValidationError
async def json_or_form(request: Request, model):
content_type = request.headers.get("content-type", "")
if content_type.startswith("application/json"):
try:
data = await request.json()
except json.JSONDecodeError as exc:
raise RequestValidationError([{"loc": ("body",), "msg": "Invalid JSON", "type": "json_invalid"}]) from exc
if not isinstance(data, dict):
raise RequestValidationError([{"loc": ("body",), "msg": "Expected an object", "type": "type_error"}])
else:
form = await request.form()
data = {}
for key in form.keys():
values = [value for value in form.getlist(key) if value != ""]
if not values:
continue
data[key] = values if len(values) > 1 else values[0]
try:
return model.model_validate(data)
except ValidationError as exc:
raise RequestValidationError(exc.errors()) from exc
molodetz/docs_api/__init__.py
+21
View File
@@ -0,0 +1,21 @@
# retoor <retoor@molodetz.nl>
from molodetz.docs_api import account, admin, community, content
from molodetz.docs_api.registry import endpoint, field, substitute
def ordered_groups():
return [content.GROUP, community.GROUP, account.GROUP, admin.GROUP]
def all_endpoints():
return [item for group in ordered_groups() for item in group["endpoints"]]
def group_by_slug(slug):
for group in ordered_groups():
if group["slug"] == slug:
return group
return None
__all__ = ["all_endpoints", "endpoint", "field", "group_by_slug", "ordered_groups", "substitute"]
molodetz/docs_api/account.py
+29
View File
@@ -0,0 +1,29 @@
# retoor <retoor@molodetz.nl>
from molodetz.docs_api.registry import endpoint, field
GROUP = {
"slug": "account",
"title": "Account",
"description": "Inloggen met sessies, API-sleutel, meldingen. Geen JWT, geen OAuth.",
"endpoints": [
endpoint("GET", "/auth/login", "Inlogpagina", sample={"errors": [], "next": "/admin"}),
endpoint(
"POST",
"/auth/login",
"Inloggen",
"Zet een httponly sessiecookie (7 dagen, 30 met onthouden).",
fields=[field("username", required=True, example="{{username}}"), field("password", required=True, type="password"), field("remember", type="boolean")],
sample={"ok": True, "redirect": "/admin", "data": None},
interactive=False,
),
endpoint("POST", "/auth/logout", "Uitloggen", sample={"ok": True, "redirect": "/", "data": None}, interactive=False),
endpoint("GET", "/profile/api-key", "API-sleutel bekijken", auth="member", sample={"api_key": "{{api_key}}"}),
endpoint("POST", "/profile/api-key/regenerate", "API-sleutel vernieuwen", auth="member", destructive=True, sample={"ok": True, "redirect": "/profile/api-key", "data": {"api_key": "..."}}, interactive=False),
endpoint("POST", "/profile/avatar/regenerate", "Avatar vernieuwen", auth="member", destructive=True, sample={"ok": True}, interactive=False),
endpoint("GET", "/notifications", "Meldingen", auth="member", sample={"notifications": [], "unread": 0}),
endpoint("POST", "/notifications/read", "Alles gelezen", auth="member", sample={"ok": True, "redirect": "/notifications"}),
endpoint("GET", "/voorwaarden", "Voorwaarden", sample={"title": "Voorwaarden"}),
endpoint("POST", "/voorwaarden/accept", "Voorwaarden accepteren", auth="member", sample={"ok": True, "redirect": "/"}),
endpoint("GET", "/privacy", "Privacy", sample={"title": "Privacy"}),
],
}
molodetz/docs_api/admin.py
+52
View File
@@ -0,0 +1,52 @@
# retoor <retoor@molodetz.nl>
from molodetz.docs_api.registry import endpoint, field
GROUP = {
"slug": "beheer",
"title": "Beheer",
"description": "Alleen voor beheerders. Berichten schrijven en publiceren, aanmeldingen, services, prullenbak, instellingen.",
"endpoints": [
endpoint("GET", "/admin", "Overzicht", auth="admin", sample={"post_count": 5, "open_joins": 0, "user_count": 1, "deleted_count": 0}),
endpoint("GET", "/admin/posts", "Berichten", auth="admin", sample={"posts": []}),
endpoint("GET", "/admin/posts/new", "Nieuw bericht", auth="admin", sample={}),
endpoint(
"POST",
"/admin/posts",
"Bericht aanmaken",
"Markdown-bericht aanmaken; status draft of published.",
auth="admin",
fields=[
field("title", required=True, example="Een titel"),
field("body", required=True, example="Tekst in **markdown**."),
field("topic", enum=["rol", "standaard"], example="rol"),
field("status", enum=["draft", "published"], example="draft"),
field("is_placeholder", type="boolean"),
],
sample={"ok": True, "redirect": "/admin/posts", "data": {"uid": "..."}},
probe_body={"title": "x"},
),
endpoint("GET", "/admin/posts/{uid}/edit", "Bericht bewerken", auth="admin", fields=[field("uid", required=True, location="path")], probe_path="/admin/posts/onbekend/edit"),
endpoint("POST", "/admin/posts/{uid}", "Bericht opslaan", auth="admin", fields=[field("uid", required=True, location="path")], probe_path="/admin/posts/onbekend", probe_body={"title": "x"}),
endpoint("POST", "/admin/posts/{uid}/publish", "Publiceren", auth="admin", fields=[field("uid", required=True, location="path")], probe_path="/admin/posts/onbekend/publish"),
endpoint("POST", "/admin/posts/{uid}/delete", "Bericht verwijderen", "Zachte verwijdering; herstel via prullenbak.", auth="admin", destructive=True, fields=[field("uid", required=True, location="path")], probe_path="/admin/posts/onbekend/delete"),
endpoint("GET", "/admin/joins", "Aanmeldingen", auth="admin", sample={"requests": []}),
endpoint("POST", "/admin/joins/{uid}/status", "Status aanmelding", auth="admin", fields=[field("uid", required=True, location="path"), field("status", enum=["open", "contacted", "accepted", "declined"])], probe_path="/admin/joins/onbekend/status", probe_body={"status": "open"}),
endpoint("POST", "/admin/joins/{uid}/check", "Repo-link controleren", "Haalt de titel op via de beveiligde uitgaande client.", auth="admin", fields=[field("uid", required=True, location="path")], probe_path="/admin/joins/onbekend/check"),
endpoint("GET", "/admin/services", "Services", auth="admin", sample={"services": []}),
endpoint("GET", "/admin/services/{name}", "Service", auth="admin", fields=[field("name", required=True, location="path", example="backup")], probe_path="/admin/services/backup"),
endpoint("POST", "/admin/services/{name}/command", "Service-commando", auth="admin", fields=[field("name", required=True, location="path"), field("verb", enum=["start", "stop", "run", "clear"])], probe_path="/admin/services/backup/command", probe_body={"verb": "noop"}),
endpoint("POST", "/admin/services/{name}/config", "Service-configuratie", auth="admin", fields=[field("name", required=True, location="path")], probe_path="/admin/services/backup/config", probe_body={}),
endpoint("GET", "/admin/trash", "Prullenbak", auth="admin", sample={"items": []}),
endpoint("POST", "/admin/trash/restore", "Herstellen", auth="admin", fields=[field("stamp", required=True)], probe_body={"stamp": "none"}),
endpoint("POST", "/admin/trash/purge", "Definitief verwijderen", auth="admin", destructive=True, fields=[field("stamp", required=True)], probe_body={"stamp": "none"}),
endpoint("GET", "/admin/settings", "Instellingen", auth="admin", sample={"settings": {}}),
endpoint("POST", "/admin/settings", "Instellingen opslaan", auth="admin", probe_body={}),
endpoint("GET", "/admin/users", "Accounts", auth="admin", sample={"users": []}),
endpoint("POST", "/admin/users/{uid}/role", "Rol wijzigen", auth="admin", fields=[field("uid", required=True, location="path"), field("role", enum=["Admin", "Member"])], probe_path="/admin/users/onbekend/role", probe_body={"role": "Member"}),
endpoint("GET", "/admin/audit", "Auditlog", auth="admin", sample={"rows": []}),
endpoint("GET", "/admin/backups", "Back-ups", auth="admin", sample={"backups": []}),
endpoint("POST", "/admin/backups", "Back-up inplannen", auth="admin", fields=[field("target", enum=["database", "uploads", "keys", "full"])], probe_body={"target": "nope"}),
endpoint("GET", "/admin/backups/{uid}/download", "Back-up downloaden", "Alleen de primaire beheerder.", auth="admin", negotiation="none", interactive=False, probe_path="/admin/backups/onbekend/download"),
endpoint("GET", "/admin/stats", "Statistieken", auth="admin", sample={"top_paths": [], "daily": []}),
],
}
molodetz/docs_api/community.py
+25
View File
@@ -0,0 +1,25 @@
# retoor <retoor@molodetz.nl>
from molodetz.docs_api.registry import endpoint, field
GROUP = {
"slug": "binnen",
"title": "Binnen",
"description": "Aanmelden om mee te schrijven. Opgeslagen voor beheerders, nergens publiek.",
"endpoints": [
endpoint("GET", "/binnen", "Aanmeldformulier", sample={"submitted": False}),
endpoint(
"POST",
"/binnen",
"Ik schrijf mee",
"Slaat een aanmelding op: naam, contact en optioneel een repo-link.",
fields=[
field("name", required=True, example="Ada"),
field("contact", required=True, example="ada@example.nl"),
field("repo_url", example="https://git.example.nl/ada"),
field("message", example="Ik schrijf over afmaken."),
],
sample={"ok": True, "redirect": "/binnen?ok=1", "data": {"uid": "0190..."}},
probe_body={"name": "Probe", "contact": "probe@example.nl"},
),
],
}
molodetz/docs_api/content.py
+46
View File
@@ -0,0 +1,46 @@
# retoor <retoor@molodetz.nl>
from molodetz.docs_api.registry import endpoint, field
GROUP = {
"slug": "inhoud",
"title": "Inhoud",
"description": "Publieke leesroutes: de rol, de standaard, berichten, galerijen en mensen.",
"endpoints": [
endpoint("GET", "/", "Voorpagina", "Laatste berichten en uitgelichte flyer.", sample={"posts": [], "flyer_count": 4, "meme_count": 20}),
endpoint(
"GET",
"/rol",
"De rol",
"Gepubliceerde notities in het onderwerp rol, nieuwste eerst, cursor-gepagineerd.",
fields=[field("before", description="Cursor uit next_cursor", location="query")],
sample={"title": "Rol", "topic": "rol", "posts": [], "next_cursor": None},
),
endpoint("GET", "/standaard", "De standaard", "Gepubliceerde berichten in het onderwerp standaard.", sample={"title": "Standaard", "topic": "standaard", "posts": []}),
endpoint(
"GET",
"/posts/{slug}",
"Bericht",
"Een gepubliceerd bericht via slug of uid.",
fields=[field("slug", required=True, location="path", example="een-zes-is-geen-ambitie")],
sample={"post": {"title": "Een zes is geen ambitie.", "topic": "rol"}, "author": {"username": "retoor"}},
probe_path="/posts/bestaat-niet",
),
endpoint("GET", "/flyers", "Flyers", "Galerij met flyers.", sample={"kind": "flyer", "title": "Flyers", "items": []}),
endpoint("GET", "/memes", "Memes", "Galerij met memes.", sample={"kind": "meme", "title": "Memes", "items": []}),
endpoint("GET", "/mensen", "Mensen", "Mensen op de rol.", sample={"people": [{"username": "retoor", "post_count": 5}]}),
endpoint(
"GET",
"/mensen/{username}",
"Persoon",
"Profiel met berichten.",
fields=[field("username", required=True, location="path", example="retoor")],
sample={"person": {"username": "retoor"}, "posts": []},
probe_path="/mensen/retoor",
),
endpoint("GET", "/robots.txt", "robots.txt", negotiation="none", interactive=False),
endpoint("GET", "/sitemap.xml", "Sitemap", negotiation="none", interactive=False),
endpoint("GET", "/avatar/svg/{seed}", "Avatar", "Deterministische SVG-avatar.", negotiation="none", interactive=False, probe_path="/avatar/svg/retoor"),
endpoint("GET", "/health", "Health", "Liveness en versie.", negotiation="json", sample={"status": "ok", "version": "1.0.0"}),
endpoint("GET", "/presence/roster", "Presence", "Online-set (uids).", negotiation="json", sample={"online": [], "tracked": 0}),
],
}
molodetz/docs_api/registry.py
+56
View File
@@ -0,0 +1,56 @@
# retoor <retoor@molodetz.nl>
ROLE_BY_AUTH = {"public": "guest", "member": "Member", "admin": "Admin"}
TARGET_TYPES = ("post", "join_request", "user")
def field(name, type="string", required=False, description="", enum=None, example=None, location="body"):
return {
"name": name,
"type": type,
"required": required,
"description": description,
"enum": list(enum or []),
"example": example,
"location": location,
}
def endpoint(
method,
path,
title,
description="",
auth="public",
fields=None,
sample=None,
negotiation="negotiable",
destructive=False,
interactive=True,
probe_path=None,
probe_body=None,
):
return {
"method": method,
"path": path,
"title": title,
"description": description,
"auth": auth,
"min_role": ROLE_BY_AUTH[auth],
"fields": list(fields or []),
"sample": sample,
"negotiation": negotiation,
"destructive": destructive,
"interactive": interactive,
"probe_path": probe_path or path,
"probe_body": probe_body,
}
def substitute(text, base="", username="", api_key="", app_reference="molodetz-docs-v-1-0-0"):
return (
text.replace("{{base}}", base)
.replace("{{username}}", username)
.replace("{{api_key}}", api_key)
.replace("{{app_reference}}", app_reference)
)
molodetz/docs_export.py
+37
View File
@@ -0,0 +1,37 @@
# retoor <retoor@molodetz.nl>
from html import escape
from molodetz.docs_api import ordered_groups
from molodetz.docs_prose import page_source, render_page, visible_pages
def export_markdown(viewer_is_admin):
parts = ["# Molodetz documentatie", ""]
for page in visible_pages(viewer_is_admin):
parts += [page_source(page["slug"]).strip(), ""]
for group in ordered_groups():
if group["slug"] == "beheer" and not viewer_is_admin:
continue
parts += [f"## API: {group['title']}", "", group["description"], ""]
for item in group["endpoints"]:
parts.append(f"- `{item['method']} {item['path']}` ({item['auth']}): {item['title']}")
parts.append("")
return "\n".join(parts)
def export_html(viewer_is_admin):
body = []
for page in visible_pages(viewer_is_admin):
body.append(f"<section>{render_page(page['slug'])}</section>")
for group in ordered_groups():
if group["slug"] == "beheer" and not viewer_is_admin:
continue
items = "".join(
f"<li><code>{escape(item['method'])} {escape(item['path'])}</code> ({escape(item['auth'])}): {escape(item['title'])}</li>"
for item in group["endpoints"]
)
body.append(f"<section><h2>API: {escape(group['title'])}</h2><ul>{items}</ul></section>")
return (
"<!doctype html><html lang=\"nl\"><head><meta charset=\"utf-8\"><title>Molodetz documentatie</title></head>"
f"<body><h1>Molodetz documentatie</h1>{''.join(body)}</body></html>"
)
molodetz/docs_live.py
+19
View File
@@ -0,0 +1,19 @@
# retoor <retoor@molodetz.nl>
from molodetz.services.base import manager
def live_services():
rows = []
for service in manager.services.values():
rows.append(
{
"name": service.name,
"title": service.title,
"description": service.description,
"fields": [
{"key": field.key, "type": field.type, "label": field.label, "group": field.group}
for field in service.fields()
],
}
)
return rows
molodetz/docs_prose.py
+74
View File
@@ -0,0 +1,74 @@
# retoor <retoor@molodetz.nl>
import re
from functools import lru_cache
import mistune
from mistune.renderers.html import HTMLRenderer
from molodetz import config
from molodetz.rendering import normalize_dashes
PROSE_DIR = config.TEMPLATES_DIR / "docs" / "prose"
SECTIONS = ("Begin hier", "Bouwen met de API", "Bijdragen en internals", "Beheer")
COMPONENTS = ("avatar", "code", "content", "title", "toast", "dialog", "context-menu", "upload", "lightbox", "chat")
DOCS_PAGES = [
{"slug": "welkom", "title": "Welkom", "section": "Begin hier", "file": "welkom.md"},
{"slug": "schrijven", "title": "Schrijven op de rol", "section": "Begin hier", "file": "schrijven.md"},
{"slug": "binnen", "title": "Binnenkomen", "section": "Begin hier", "file": "binnen.md"},
{"slug": "juridisch", "title": "Voorwaarden en privacy", "section": "Begin hier", "file": "juridisch.md"},
{"slug": "api", "title": "API en authenticatie", "section": "Bouwen met de API", "file": "api.md"},
{"slug": "architectuur", "title": "Architectuur", "section": "Bijdragen en internals", "file": "architectuur.md"},
{"slug": "frontend", "title": "Frontend en componenten", "section": "Bijdragen en internals", "file": "frontend.md"},
{"slug": "beheer", "title": "Beheer en services", "section": "Beheer", "file": "beheer.md", "admin": True},
{"slug": "backups", "title": "Back-ups en onderhoud", "section": "Beheer", "file": "backups.md", "admin": True},
] + [
{
"slug": f"component-{name}",
"title": f"dp-{name}",
"section": "Bijdragen en internals",
"file": f"component-{name}.md",
}
for name in COMPONENTS
]
class ProseRenderer(HTMLRenderer):
def heading(self, text, level, **attrs):
anchor = re.sub(r"[^a-z0-9-]", "-", re.sub(r"<[^>]+>", "", text).lower())
anchor = re.sub(r"-+", "-", anchor).strip("-")
return f'<h{level} id="{anchor}"><a class="permalink" href="#{anchor}">{text}</a></h{level}>\n'
_prose_markdown = mistune.create_markdown(escape=False, hard_wrap=True, renderer=ProseRenderer(escape=False), plugins=["table", "strikethrough", "url"])
def visible_pages(viewer_is_admin):
return [page for page in DOCS_PAGES if viewer_is_admin or not page.get("admin")]
def page_by_slug(slug):
for page in DOCS_PAGES:
if page["slug"] == slug:
return page
return None
@lru_cache(maxsize=64)
def page_source(slug):
page = page_by_slug(slug)
if page is None:
return ""
return normalize_dashes((PROSE_DIR / page["file"]).read_text(encoding="utf-8"))
@lru_cache(maxsize=64)
def render_page(slug):
return _prose_markdown(page_source(slug))
def sidebar(viewer_is_admin):
pages = visible_pages(viewer_is_admin)
return [(section, [page for page in pages if page["section"] == section]) for section in SECTIONS]
molodetz/docs_search.py
+73
View File
@@ -0,0 +1,73 @@
# retoor <retoor@molodetz.nl>
import math
import re
from collections import Counter
from functools import lru_cache
from bs4 import BeautifulSoup
from molodetz.docs_api import ordered_groups
from molodetz.docs_prose import DOCS_PAGES, render_page
TOKEN = re.compile(r"[a-z0-9_]+")
K1 = 1.5
B = 0.75
def tokenize(text):
return TOKEN.findall((text or "").lower())
@lru_cache(maxsize=1)
def build_index():
documents = []
for page in DOCS_PAGES:
text = BeautifulSoup(render_page(page["slug"]), "html.parser").get_text(" ")
documents.append({"title": page["title"], "url": f"/docs/{page['slug']}", "text": text, "admin": bool(page.get("admin"))})
for group in ordered_groups():
lines = [group["title"], group["description"]]
for item in group["endpoints"]:
lines.append(f"{item['method']} {item['path']} {item['title']} {item['description']}")
documents.append(
{"title": f"API: {group['title']}", "url": f"/docs/api/{group['slug']}", "text": " ".join(lines), "admin": group["slug"] == "beheer"}
)
tokens = [Counter(tokenize(doc["title"] + " " + doc["text"])) for doc in documents]
lengths = [sum(counter.values()) for counter in tokens]
average = sum(lengths) / max(1, len(lengths))
frequency = Counter()
for counter in tokens:
frequency.update(counter.keys())
return documents, tokens, lengths, average, frequency
def _snippet(text, terms, width=160):
lowered = text.lower()
position = min((lowered.find(term) for term in terms if lowered.find(term) >= 0), default=0)
start = max(0, position - width // 3)
snippet = text[start : start + width].strip()
for term in terms:
snippet = re.sub(f"({re.escape(term)})", r"[[\1]]", snippet, flags=re.IGNORECASE)
return snippet
def search(query, viewer_is_admin=False, limit=20):
terms = tokenize(query)
if not terms:
return []
documents, tokens, lengths, average, frequency = build_index()
total = len(documents)
results = []
for index, doc in enumerate(documents):
if doc["admin"] and not viewer_is_admin:
continue
score = 0.0
for term in terms:
tf = tokens[index].get(term, 0)
if not tf:
continue
idf = math.log(1 + (total - frequency[term] + 0.5) / (frequency[term] + 0.5))
score += idf * tf * (K1 + 1) / (tf + K1 * (1 - B + B * lengths[index] / average))
if score > 0:
results.append({"title": doc["title"], "url": doc["url"], "score": round(score, 3), "snippet": _snippet(doc["text"], terms)})
results.sort(key=lambda item: item["score"], reverse=True)
return results[:limit]
molodetz/emoji_builder.py
+45
View File
@@ -0,0 +1,45 @@
# retoor <retoor@molodetz.nl>
import json
from functools import lru_cache
import emoji
from molodetz import config
CUSTOM_SHORTCODES = {
"molodetz": "\U0001F58B\uFE0F",
"af": "\u2705",
}
GENERATED_JS = config.STATIC_DIR / "js" / "generated" / "EmojiMap.js"
@lru_cache(maxsize=1)
def shortcode_map():
mapping = {}
for char, data in emoji.EMOJI_DATA.items():
if data.get("status") != emoji.STATUS["fully_qualified"]:
continue
names = [data.get("en", "")] + list(data.get("alias", []))
for name in names:
key = name.strip(":")
if key and key not in mapping:
mapping[key] = char
mapping.update(CUSTOM_SHORTCODES)
return mapping
def write_js_module(path=GENERATED_JS):
path.parent.mkdir(parents=True, exist_ok=True)
payload = json.dumps(dict(sorted(shortcode_map().items())), ensure_ascii=False, separators=(",", ":"))
path.write_text(
"// retoor <retoor@molodetz.nl>\n"
f"export const EMOJI_MAP = Object.freeze({payload});\n",
encoding="utf-8",
)
return path
def is_single_emoji(value):
value = (value or "").strip()
return emoji.emoji_count(value) == 1 and emoji.purely_emoji(value)
molodetz/friendly404.py
+13
View File
@@ -0,0 +1,13 @@
# retoor <retoor@molodetz.nl>
import random
POOL = (
"Deze pagina is nooit afgemaakt. Dat zegt genoeg.",
"Hier stond iets. Of het was bijna klaar.",
"Niet gevonden. Wel een goede reden om terug te gaan naar de rol.",
"Een link die nergens heen gaat is een mening, geen feit.",
)
def friendly_message():
return random.choice(POOL)
molodetz/gallery.py
+96
View File
@@ -0,0 +1,96 @@
# retoor <retoor@molodetz.nl>
import io
import logging
import imagehash
from PIL import Image
from molodetz import config
from molodetz.attachments import blob_path, blob_url, write_blob
from molodetz.database import generate_uid, media_by_kind, upsert_media
from molodetz.database.media import get_media_by_file
logger = logging.getLogger(__name__)
THUMB_WIDTH = 720
DUPLICATE_DISTANCE = 2
GALLERY = (
("flyer-keuzes.jpg", "flyer", "Eenendertig keuzes. Jij hebt er nul."),
("flyer-chat.jpg", "flyer", "Een chat is geen portfolio."),
("flyer-bijna.jpg", "flyer", "Bijna is geen staat."),
("flyer-assistent.jpg", "flyer", "De assistent oordeelt niet."),
("meme-vloer.jpg", "meme", "Een zesje is prima. De vloer, bedoel je."),
("meme-split.jpg", "meme", "Stuur ze naar huis."),
("meme-cursus.jpg", "meme", "Geen ebook. Wel afmaken."),
("take-chatlog.jpg", "meme", "Je portfolio is een chatlog."),
("take-copilot.jpg", "meme", "Copilot is geen senior."),
("take-coping.jpg", "meme", "Een zesje is coping."),
("poster-thuis.jpg", "meme", "Vibe coders, ga naar huis."),
("poster-vloer.jpg", "meme", "Een zesje? Dat is de vloer."),
("poster-cursus.jpg", "meme", "Geen cursus. Wel afmaken."),
("poster-luiheid.jpg", "meme", "AI pakt je baan niet. Luiheid wel."),
("poster-spel.jpg", "meme", "Het spel is om. Jij bent het niet."),
("story.jpg", "meme", "Geen ebook. Geen x10. Afwerken wel."),
("meme-zes.jpg", "meme", "Een zesje. Gefeliciteerd, je bent gemiddeld."),
("meme-gesprek.jpg", "meme", "Mijn portfolio is een gesprek."),
("meme-product.jpg", "meme", "De cursus is het product."),
("meme-bijna.jpg", "meme", "Iedereen staat bij bijna."),
("meme-compiles.jpg", "meme", "It compiles. Ship it, king."),
("meme-senior.jpg", "meme", "My copilot is the senior."),
("meme-trophy.jpg", "meme", "Peak performance, 6 out of 10."),
("meme-todo.jpg", "meme", "Side projects, a love story."),
)
def _thumbnail_bytes(image):
copy = image.convert("RGB")
copy.thumbnail((THUMB_WIDTH, THUMB_WIDTH * 2))
buffer = io.BytesIO()
copy.save(buffer, format="WEBP", quality=82, method=4)
return buffer.getvalue()
def sync_gallery():
seen_hashes = []
synced = 0
for position, (filename, kind, caption) in enumerate(GALLERY):
source = config.MEDIA_SOURCE_DIR / filename
if not source.exists():
logger.warning("gallery source missing: %s", source)
continue
existing = get_media_by_file(filename)
with Image.open(source) as image:
width, height = image.size
phash = imagehash.phash(image)
if any(phash - other <= DUPLICATE_DISTANCE for other in seen_hashes):
logger.info("gallery duplicate skipped: %s", filename)
continue
seen_hashes.append(phash)
thumb_directory = existing.get("thumb_directory") if existing else None
thumb_name = existing.get("thumb_name") if existing else None
if not thumb_directory or not blob_path(thumb_directory, thumb_name).exists():
thumb_directory, thumb_name = write_blob(
existing["uid"] if existing else generate_uid(), "webp", _thumbnail_bytes(image)
)
upsert_media(filename, kind, caption, position, width, height, str(phash), thumb_directory, thumb_name)
synced += 1
return synced
def gallery_items(kind):
items = []
for row in media_by_kind(kind):
thumb = blob_url(row["thumb_directory"], row["thumb_name"]) if row.get("thumb_directory") else None
items.append(
{
"uid": row["uid"],
"filename": row["filename"],
"caption": row["caption"],
"width": row["width"],
"height": row["height"],
"source_path": f"/static/media/{row['filename']}",
"thumb_url": thumb,
}
)
return items
molodetz/gates.py
+129
View File
@@ -0,0 +1,129 @@
# retoor <retoor@molodetz.nl>
import ast
import re
import sys
from pathlib import Path
PKG_ROOT = Path(__file__).resolve().parent
REPO_ROOT = PKG_ROOT.parent
BREAKPOINTS = {360, 480, 768, 1024}
HTTP_CLIENT_ALLOWLIST = {"molodetz/stealth.py", "molodetz/curl_transport.py"}
CLIENT_RENDER_ALLOWLIST = {"molodetz/templates/admin/post_form.html"}
LOADING_BUTTON_MODULE = "molodetz/static/js/utils/LoadingButton.js"
_DASH_ENTITIES = ["&" + name + ";" for name in ("m" + "dash", "n" + "dash", "#8212", "#8211", "#x2014", "#x2013")]
DASH_PATTERN = re.compile("|".join(["\u2014", "\u2013", *_DASH_ENTITIES]), re.IGNORECASE)
DASH_SUFFIXES = {".py", ".html", ".md", ".js", ".css", ".toml", ".txt", ".sh", ".cfg", ".ini"}
DASH_EXEMPT_DIRS = ("molodetz/static/vendor", "molodetz/static/js/generated")
def _rel(path):
return path.resolve().relative_to(REPO_ROOT).as_posix()
def _files(base, pattern):
return sorted((REPO_ROOT / base).rglob(pattern))
def gate_template_environment():
offenders = set()
for path in _files("molodetz", "*.py"):
text = path.read_text(encoding="utf-8")
if re.search(r"\b(Jinja2Templates|Environment)\(", text) and _rel(path) != "molodetz/templating.py":
offenders.add(_rel(path))
return offenders
def gate_breakpoints():
found = set()
for path in _files("molodetz/static/css", "*.css"):
for match in re.finditer(r"@media[^{]*\((?:max|min)-width:\s*(\d+)px\)", path.read_text(encoding="utf-8")):
found.add(int(match.group(1)))
return found
def gate_http_clients():
found = set()
pattern = re.compile(r"\b(httpx\.(Async)?Client|AsyncSession|aiohttp\.ClientSession|urllib\.request\.urlopen)\(")
for path in _files("molodetz", "*.py"):
if pattern.search(path.read_text(encoding="utf-8")):
found.add(_rel(path))
return found
def gate_client_render_templates():
found = set()
for path in _files("molodetz/templates", "*.html"):
if "/templates/docs/" in path.as_posix():
continue
if re.search(r"<dp-(content|title)\b", path.read_text(encoding="utf-8")):
found.add(_rel(path))
return found
def gate_bare_static():
offenders = []
for path in _files("molodetz/templates", "*.html"):
for number, line in enumerate(path.read_text(encoding="utf-8").splitlines(), 1):
if re.search(r"""(href|src|srcset|content)\s*=\s*["']/static/""", line):
offenders.append(f"{_rel(path)}:{number}")
return offenders
def gate_loading_toggles():
offenders = []
for path in _files("molodetz/static/js", "*.js"):
if _rel(path) == LOADING_BUTTON_MODULE:
continue
if "is-loading" in path.read_text(encoding="utf-8"):
offenders.append(_rel(path))
return offenders
def gate_em_dashes():
offenders = []
for path in sorted(REPO_ROOT.rglob("*")):
if not path.is_file() or path.suffix not in DASH_SUFFIXES:
continue
rel = _rel(path)
if rel.startswith((".venv", "data", "out", ".git")) or rel.startswith(DASH_EXEMPT_DIRS):
continue
if DASH_PATTERN.search(path.read_text(encoding="utf-8", errors="ignore")):
offenders.append(rel)
return offenders
def gate_json_error_order():
offenders = []
for path in _files("molodetz", "*.py"):
tree = ast.parse(path.read_text(encoding="utf-8"))
for node in ast.walk(tree):
if isinstance(node, ast.Call) and getattr(node.func, "id", getattr(node.func, "attr", None)) == "json_error":
first = node.args[0] if node.args else None
is_status = isinstance(first, ast.Constant) and isinstance(first.value, int)
is_name = isinstance(first, (ast.Name, ast.Attribute)) and "status" in ast.unparse(first).lower()
if not (is_status or is_name):
offenders.append(f"{_rel(path)}:{node.lineno}")
return offenders
def run_all():
results = {
"1 template environment only in templating.py": (not gate_template_environment(), gate_template_environment()),
"2 breakpoints equal 360 480 768 1024": (gate_breakpoints() == BREAKPOINTS, sorted(gate_breakpoints())),
"3 http client files equal allowlist": (gate_http_clients() == HTTP_CLIENT_ALLOWLIST, sorted(gate_http_clients())),
"4 client-render templates equal allowlist": (gate_client_render_templates() == CLIENT_RENDER_ALLOWLIST, sorted(gate_client_render_templates())),
"5 no bare /static/ in templates": (not gate_bare_static(), gate_bare_static()),
"6 no is-loading outside LoadingButton": (not gate_loading_toggles(), gate_loading_toggles()),
"em-dash scan": (not gate_em_dashes(), gate_em_dashes()),
"json_error argument order": (not gate_json_error_order(), gate_json_error_order()),
}
failed = 0
for name, (ok, detail) in results.items():
print(f"{'PASS' if ok else 'FAIL'} {name} {detail if not ok else ''}".rstrip())
failed += 0 if ok else 1
return failed
if __name__ == "__main__":
sys.exit(1 if run_all() else 0)
molodetz/html_reformat.py
+12
View File
@@ -0,0 +1,12 @@
# retoor <retoor@molodetz.nl>
import re
BLANK_LINES = re.compile(r"\n[ \t]*\n(?:[ \t]*\n)+")
TRAILING = re.compile(r"[ \t]+\n")
def reformat_html(text):
if "<pre" in text or "<textarea" in text:
return TRAILING.sub("\n", text)
text = TRAILING.sub("\n", text)
return BLANK_LINES.sub("\n\n", text)
molodetz/link_check.py
+53
View File
@@ -0,0 +1,53 @@
# retoor <retoor@molodetz.nl>
import logging
import defusedxml.ElementTree as SafeET
import httpx
from bs4 import BeautifulSoup
from molodetz.database import now_iso, update_join_request
from molodetz.net_guard import UnsafeURL, guard_public_url
from molodetz.stealth import detect_consent_gate, guarded_async_client
logger = logging.getLogger(__name__)
MAX_BYTES = 512 * 1024
def extract_title(content_type, body):
if "xml" in content_type and "html" not in content_type:
root = SafeET.fromstring(body)
for element in root.iter():
if element.tag.split("}")[-1] == "title" and element.text:
return element.text.strip()[:200]
return None
soup = BeautifulSoup(body, "lxml")
if soup.title and soup.title.string:
return soup.title.string.strip()[:200]
heading = soup.find("h1")
return heading.get_text(strip=True)[:200] if heading else None
async def check_repo_link(row):
url = row.get("repo_url")
if not url:
return {"status": "none"}
try:
await guard_public_url(url)
async with guarded_async_client(timeout=10.0) as client:
response = await client.get(url)
body = response.content[:MAX_BYTES]
redirect = detect_consent_gate(body.decode("utf-8", "ignore"))
if redirect:
response = await client.get(redirect)
body = response.content[:MAX_BYTES]
title = extract_title(response.headers.get("content-type", ""), body) if response.status_code < 400 else None
status = f"http {response.status_code}"
except UnsafeURL as exc:
title, status = None, f"geweigerd: {exc}"
except (httpx.HTTPError, OSError, SafeET.ParseError) as exc:
title, status = None, f"fout: {type(exc).__name__}"
except Exception as exc:
logger.warning("repo link check failed: %s", exc)
title, status = None, f"fout: {type(exc).__name__}"
update_join_request(row["uid"], link_title=title, link_status=status, link_checked_at=now_iso())
return {"status": status, "title": title}
molodetz/main.py
+440
View File
@@ -0,0 +1,440 @@
# retoor <retoor@molodetz.nl>
import fcntl
import logging
import math
import time
import traceback
from collections import defaultdict
from contextlib import asynccontextmanager
from datetime import datetime, timezone
from fastapi import FastAPI, HTTPException, Request
from fastapi.exceptions import RequestValidationError
from fastapi.responses import JSONResponse, PlainTextResponse, RedirectResponse, Response
from fastapi.staticfiles import StaticFiles
from starlette.middleware.gzip import GZipMiddleware
from molodetz import config
from molodetz.attachments import INLINE_EXTENSIONS, extension_of
from molodetz.constants import (
CRAWLER_MARKERS,
CRAWLER_PRIVATE_PREFIXES,
GZIP_SKIP_PREFIXES,
MAINTENANCE_ALLOWED_PREFIXES,
RATE_LIMIT_EXEMPT_PREFIXES,
TERMS_EXEMPT_PREFIXES,
)
from molodetz.database import flush_visits, get_int_setting, get_setting, init_db, refresh_snapshot
from molodetz.emoji_builder import GENERATED_JS, write_js_module
from molodetz.friendly404 import friendly_message
from molodetz.html_reformat import reformat_html
from molodetz.responses import json_error, wants_json
from molodetz.routers import (
auth,
avatar,
binnen,
docs,
flyers,
health,
home,
memes,
mensen,
notifications,
posts,
presence,
privacy,
profile,
rol,
seo,
standaard,
voorwaarden,
)
from molodetz.routers import admin
from molodetz.bootstrap import bootstrap_content
from molodetz.services.base import manager
from molodetz.services.queue import background
from molodetz.services.registry import build_services
from molodetz.templating import templates
from molodetz.utils.audit import record
from molodetz.utils.auth import GuardRedirect, get_current_user, is_admin
from molodetz.utils.presence import touch
from molodetz.utils.request_meta import client_ip, safe_next
logging.basicConfig(level=config.LOG_LEVEL, format="%(asctime)s %(levelname)s %(name)s: %(message)s")
logger = logging.getLogger("molodetz")
_visit_counts = defaultdict(int)
_visit_state = {"last_flush": time.monotonic()}
_lock_handles = {}
def _take_lock(path, blocking):
handle = open(path, "a+")
flags = fcntl.LOCK_EX if blocking else fcntl.LOCK_EX | fcntl.LOCK_NB
try:
fcntl.flock(handle, flags)
except BlockingIOError:
handle.close()
return None
return handle
def _flush_visit_counts():
if not _visit_counts:
return
day = datetime.now(timezone.utc).date().isoformat()
snapshot = dict(_visit_counts)
_visit_counts.clear()
_visit_state["last_flush"] = time.monotonic()
background.submit(flush_visits, day, snapshot)
@asynccontextmanager
async def lifespan(app):
config.ensure_data_dirs()
init_handle = _take_lock(config.INIT_LOCK_FILE, blocking=True)
try:
init_db()
bootstrap_content()
if not GENERATED_JS.exists():
write_js_module()
finally:
fcntl.flock(init_handle, fcntl.LOCK_UN)
init_handle.close()
for service in build_services():
manager.register(service)
if not config.DISABLE_SERVICES:
background.start()
service_handle = _take_lock(config.SERVICE_LOCK_FILE, blocking=False)
if service_handle is not None:
_lock_handles["service"] = service_handle
manager.supervise()
logger.info("service lock acquired; supervising %d services", len(manager.services))
else:
logger.info("service lock held elsewhere; skipping supervision")
yield
if _visit_counts:
_flush_visit_counts()
await manager.shutdown_all()
await background.stop()
for handle in _lock_handles.values():
handle.close()
_lock_handles.clear()
app = FastAPI(
title=config.APP_TITLE,
docs_url="/swagger",
redoc_url=None,
swagger_ui_oauth2_redirect_url=None,
openapi_url="/openapi.json",
lifespan=lifespan,
)
app.include_router(auth.router, prefix="/auth")
app.include_router(admin.router, prefix="/admin")
app.include_router(avatar.router, prefix="/avatar")
app.include_router(profile.router, prefix="/profile")
app.include_router(notifications.router, prefix="/notifications")
app.include_router(presence.router, prefix="/presence")
app.include_router(posts.router, prefix="/posts")
app.include_router(rol.router, prefix="/rol")
app.include_router(standaard.router, prefix="/standaard")
app.include_router(flyers.router, prefix="/flyers")
app.include_router(memes.router, prefix="/memes")
app.include_router(mensen.router, prefix="/mensen")
app.include_router(binnen.router, prefix="/binnen")
app.include_router(voorwaarden.router, prefix="/voorwaarden")
app.include_router(privacy.router, prefix="/privacy")
app.include_router(health.router, prefix="/health")
app.include_router(seo.router)
app.include_router(docs.router)
app.include_router(home.router)
class UploadsStatic(StaticFiles):
async def get_response(self, path, scope):
response = await super().get_response(path, scope)
disposition = "inline" if extension_of(path) in INLINE_EXTENSIONS else "attachment"
response.headers["Content-Disposition"] = disposition
response.headers["Cache-Control"] = "public, max-age=604800"
return response
class VersionedStatic(StaticFiles):
async def get_response(self, path, scope):
response = await super().get_response(path, scope)
if path.endswith("sw.js"):
response.headers["Cache-Control"] = "no-cache"
else:
response.headers["Cache-Control"] = "public, max-age=31536000, immutable"
return response
class FallbackStatic(StaticFiles):
async def get_response(self, path, scope):
response = await super().get_response(path, scope)
response.headers["Cache-Control"] = "no-cache" if path.endswith("sw.js") else "public, max-age=3600"
return response
app.mount("/static/uploads", UploadsStatic(directory=str(config.DATA_PATHS["uploads"]), check_dir=False), name="uploads")
app.mount(f"/static/v{config.STATIC_VERSION}", VersionedStatic(directory=str(config.STATIC_DIR)), name="static_versioned")
app.mount("/static", FallbackStatic(directory=str(config.STATIC_DIR)), name="static")
def _is_mutating(request):
return request.method in ("POST", "PUT", "PATCH", "DELETE")
@app.middleware("http")
async def snapshot_middleware(request: Request, call_next):
path = request.url.path
if not path.startswith(("/static", "/avatar")):
refresh_snapshot()
return await call_next(request)
@app.middleware("http")
async def security_headers_middleware(request: Request, call_next):
response = await call_next(request)
response.headers.setdefault("X-Robots-Tag", "index, follow")
response.headers["X-Content-Type-Options"] = "nosniff"
response.headers["Strict-Transport-Security"] = "max-age=31536000; includeSubDomains"
response.headers["Referrer-Policy"] = "strict-origin-when-cross-origin"
response.headers["Content-Security-Policy"] = (
"object-src 'none'; base-uri 'self'; frame-ancestors 'self'; form-action 'self'"
)
if request.url.path.startswith("/admin"):
response.headers["Cache-Control"] = "no-store"
response.headers["Pragma"] = "no-cache"
response.headers["Expires"] = "0"
return response
_rate_buckets = defaultdict(list)
_rate_state = {"limit": None, "window": None, "loaded": 0.0, "swept": time.monotonic()}
def _rate_settings():
now = time.monotonic()
if now - _rate_state["loaded"] > 2:
_rate_state["limit"] = max(1, get_int_setting("rate_limit_per_minute", config.RATE_LIMIT))
_rate_state["window"] = max(1, get_int_setting("rate_limit_window_seconds", 60))
_rate_state["loaded"] = now
per_worker = max(1, math.ceil(_rate_state["limit"] / max(1, config.WEB_WORKERS)))
return per_worker, _rate_state["window"]
@app.middleware("http")
async def rate_limit_middleware(request: Request, call_next):
if config.DISABLE_RATE_LIMIT or not _is_mutating(request):
return await call_next(request)
if RATE_LIMIT_EXEMPT_PREFIXES and request.url.path.startswith(RATE_LIMIT_EXEMPT_PREFIXES):
return await call_next(request)
limit, window = _rate_settings()
now = time.monotonic()
if now - _rate_state["swept"] > 60:
for key in list(_rate_buckets):
_rate_buckets[key] = [stamp for stamp in _rate_buckets[key] if now - stamp < window]
if not _rate_buckets[key]:
del _rate_buckets[key]
_rate_state["swept"] = now
key = client_ip(request)
bucket = [stamp for stamp in _rate_buckets[key] if now - stamp < window]
if len(bucket) >= max(1, limit):
_rate_buckets[key] = bucket
retry = max(1, int(window - (now - bucket[0])))
record(request, "security.ratelimit.exceeded", result="denied", payload={"ip": key})
if wants_json(request):
response = json_error(429, "Te veel verzoeken", retry_after=retry)
else:
response = PlainTextResponse("Te veel verzoeken. Probeer het zo opnieuw.", status_code=429)
response.headers["Retry-After"] = str(retry)
return response
bucket.append(now)
_rate_buckets[key] = bucket
return await call_next(request)
@app.middleware("http")
async def maintenance_middleware(request: Request, call_next):
if get_setting("maintenance_mode", "0") != "1":
return await call_next(request)
if request.url.path.startswith(MAINTENANCE_ALLOWED_PREFIXES) or is_admin(get_current_user(request)):
return await call_next(request)
message = get_setting("maintenance_message", "Onderhoud")
record(request, "security.maintenance.blocked", result="denied")
if wants_json(request):
return json_error(503, message)
return templates.TemplateResponse(
request,
"error.html",
{"status_code": 503, "message": message, "seo": {"title": "Onderhoud", "robots": "noindex,nofollow"}},
status_code=503,
)
@app.middleware("http")
async def terms_middleware(request: Request, call_next):
if not _is_mutating(request) or request.url.path.startswith(TERMS_EXEMPT_PREFIXES):
return await call_next(request)
user = get_current_user(request)
version = get_setting("terms_version", "1")
if user is not None and str(user.get("terms_version") or "") != str(version):
target = f"/voorwaarden?next={request.url.path}"
if wants_json(request):
return json_error(403, "Accepteer eerst de voorwaarden", code="terms_required", redirect=target, terms_version=version)
return RedirectResponse(target, status_code=303)
return await call_next(request)
@app.middleware("http")
async def crawler_block_middleware(request: Request, call_next):
if request.method == "GET" and request.url.path.startswith(CRAWLER_PRIVATE_PREFIXES):
agent = (request.headers.get("user-agent") or "").lower()
if any(marker in agent for marker in CRAWLER_MARKERS):
record(request, "security.crawler.blocked", result="denied", payload={"agent": agent[:120]})
if wants_json(request):
return json_error(403, "Crawlers are not allowed here")
return PlainTextResponse("Forbidden", status_code=403)
return await call_next(request)
@app.middleware("http")
async def presence_middleware(request: Request, call_next):
if not request.url.path.startswith(("/static", "/avatar")):
user = get_current_user(request)
if user:
touch(user["uid"])
return await call_next(request)
@app.middleware("http")
async def visits_middleware(request: Request, call_next):
response = await call_next(request)
path = request.url.path
if not path.startswith(("/static", "/avatar")):
_visit_counts[(path[:200], response.status_code)] += 1
if time.monotonic() - _visit_state["last_flush"] > 30:
_flush_visit_counts()
return response
@app.middleware("http")
async def html_reformat_middleware(request: Request, call_next):
response = await call_next(request)
content_type = response.headers.get("content-type", "")
if not content_type.startswith("text/html") or response.headers.get("content-encoding"):
return response
body = b""
async for chunk in response.body_iterator:
body += chunk
try:
body = reformat_html(body.decode("utf-8")).encode("utf-8")
except Exception:
logger.warning("html reformat failed; serving original body")
headers = dict(response.headers)
headers["content-length"] = str(len(body))
return Response(content=body, status_code=response.status_code, headers=headers, media_type=None)
@app.middleware("http")
async def timing_middleware(request: Request, call_next):
request.state.request_start = time.perf_counter()
response = await call_next(request)
response.headers["X-Response-Time"] = f"{(time.perf_counter() - request.state.request_start) * 1000:.1f}ms"
return response
class SelectiveGZipMiddleware(GZipMiddleware):
async def __call__(self, scope, receive, send):
if scope["type"] == "http" and GZIP_SKIP_PREFIXES and scope["path"].startswith(GZIP_SKIP_PREFIXES):
await self.app(scope, receive, send)
return
await super().__call__(scope, receive, send)
app.add_middleware(SelectiveGZipMiddleware, minimum_size=512, compresslevel=5)
def _error_page(request, status_code, message, headers=None):
seo = {"title": f"{status_code}", "robots": "noindex,nofollow", "description": message}
return templates.TemplateResponse(
request,
"error.html",
{"status_code": status_code, "message": message, "seo": seo},
status_code=status_code,
headers=headers,
)
@app.exception_handler(GuardRedirect)
async def guard_redirect_handler(request: Request, exc: GuardRedirect):
return RedirectResponse(exc.url, status_code=exc.status_code)
@app.exception_handler(HTTPException)
async def http_exception_handler(request: Request, exc: HTTPException):
if wants_json(request) or request.headers.get("x-api-key") or request.headers.get("authorization"):
return json_error(exc.status_code, str(exc.detail))
if exc.status_code == 404:
message = friendly_message() if get_setting("friendly_404", "1") == "1" else "Niet gevonden."
return _error_page(request, 404, message)
return _error_page(request, exc.status_code, str(exc.detail), headers=getattr(exc, "headers", None))
@app.exception_handler(404)
async def not_found_handler(request: Request, exc):
return await http_exception_handler(request, HTTPException(404, "Niet gevonden"))
@app.exception_handler(500)
async def server_error_handler(request: Request, exc):
logger.error("unhandled error: %s", "".join(traceback.format_exception(exc)))
if wants_json(request):
return json_error(500, "Interne fout")
return _error_page(request, 500, "Er ging iets mis aan onze kant.")
AUTH_TEMPLATES = {"/auth/login": "auth/login.html"}
def _friendly_messages(errors):
fields, messages = [], []
for error in errors:
loc = [str(part) for part in error.get("loc", ()) if part not in ("body", "query")]
field = loc[-1] if loc else "body"
message = str(error.get("msg", "Ongeldige invoer")).replace("Value error, ", "")
if field == "password" and "at least" in message:
message = "Wachtwoord moet 6 tot 128 tekens zijn."
fields.append(field)
messages.append(f"{field}: {message}" if field != "body" else message)
return fields, messages
@app.exception_handler(RequestValidationError)
async def validation_handler(request: Request, exc: RequestValidationError):
fields, messages = _friendly_messages(exc.errors())
if wants_json(request):
return JSONResponse({"error": "validation", "fields": fields, "messages": messages}, status_code=422)
template = AUTH_TEMPLATES.get(request.url.path)
if template:
form = await request.form()
echoed = {key: value for key, value in form.items() if isinstance(value, str) and key != "password"}
return templates.TemplateResponse(
request,
template,
{"errors": messages, "form": echoed, "next": echoed.get("next"), "seo": {"title": "Inloggen", "robots": "noindex,nofollow"}},
status_code=400,
)
if request.url.path == "/binnen":
form = await request.form()
echoed = {key: value for key, value in form.items() if isinstance(value, str)}
return templates.TemplateResponse(
request,
"binnen.html",
{"errors": messages, "form": echoed, "submitted": False, "seo": {"title": "Binnen", "robots": "index,follow"}},
status_code=400,
)
return RedirectResponse(safe_next(request.headers.get("referer", "").replace(str(request.base_url).rstrip("/"), ""), "/"), status_code=303)
molodetz/models.py
+131
View File
@@ -0,0 +1,131 @@
# retoor <retoor@molodetz.nl>
import re
from datetime import datetime
from pydantic import BaseModel, ConfigDict, Field, field_validator
from molodetz.constants import JOIN_STATUSES, POST_STATUSES, ROLES, TOPICS
URL_HOST_PATTERN = re.compile(r"^https?://[^\s/]+\.[^\s/]+")
PROSE_CAP = 125000
def normalize_website(value):
value = (value or "").strip()
if not value:
return None
if not value.lower().startswith(("http://", "https://")):
value = "https://" + value
if not URL_HOST_PATTERN.match(value):
raise ValueError("Geef een geldige link op, bijvoorbeeld https://git.example.nl/naam")
return value[:500]
def normalize_european_date(value):
value = (value or "").strip()
if not value:
return None
for fmt in ("%d/%m/%Y", "%Y-%m-%d"):
try:
return datetime.strptime(value, fmt).date().isoformat()
except ValueError:
continue
raise ValueError("Gebruik DD/MM/JJJJ")
class FormModel(BaseModel):
model_config = ConfigDict(str_strip_whitespace=True, extra="ignore")
class LoginForm(FormModel):
username: str = Field(min_length=3, max_length=255)
password: str = Field(min_length=6, max_length=128)
remember: bool = False
next: str | None = None
class PostForm(FormModel):
title: str = Field(min_length=3, max_length=200)
body: str = Field(min_length=10, max_length=PROSE_CAP)
topic: str = "rol"
status: str = "draft"
is_placeholder: bool = False
@field_validator("topic")
@classmethod
def check_topic(cls, value):
if value not in TOPICS:
raise ValueError("Onbekend onderwerp")
return value
@field_validator("status")
@classmethod
def check_status(cls, value):
if value not in POST_STATUSES:
raise ValueError("Onbekende status")
return value
class JoinForm(FormModel):
name: str = Field(min_length=2, max_length=80)
contact: str = Field(min_length=3, max_length=255)
repo_url: str | None = Field(default=None, max_length=500)
message: str | None = Field(default=None, max_length=2000)
website: str | None = Field(default=None, max_length=200)
@field_validator("repo_url")
@classmethod
def check_repo(cls, value):
return normalize_website(value)
class JoinStatusForm(FormModel):
status: str
@field_validator("status")
@classmethod
def check_status(cls, value):
if value not in JOIN_STATUSES:
raise ValueError("Onbekende status")
return value
class RoleForm(FormModel):
role: str
@field_validator("role")
@classmethod
def check_role(cls, value):
if value not in ROLES:
raise ValueError("Onbekende rol")
return value
class SettingsForm(FormModel):
site_title: str | None = Field(default=None, max_length=80)
site_tagline: str | None = Field(default=None, max_length=200)
maintenance_mode: str | None = None
maintenance_message: str | None = Field(default=None, max_length=500)
rate_limit_per_minute: str | None = None
rate_limit_window_seconds: str | None = None
session_max_age_days: str | None = None
session_remember_days: str | None = None
terms_version: str | None = None
friendly_404: str | None = None
audit_retention_days: str | None = None
outbound_proxy_url: str | None = Field(default=None, max_length=500)
class ServiceConfigForm(FormModel):
model_config = ConfigDict(extra="allow", str_strip_whitespace=True)
class BackupForm(FormModel):
target: str = "database"
@field_validator("target")
@classmethod
def check_target(cls, value):
if value not in ("database", "uploads", "keys", "full"):
raise ValueError("Onbekend doel")
return value
molodetz/net_guard.py
+68
View File
@@ -0,0 +1,68 @@
# retoor <retoor@molodetz.nl>
import asyncio
import ipaddress
import socket
from urllib.parse import urlsplit
import httpx
NAT64_PREFIX = ipaddress.ip_network("64:ff9b::/96")
class UnsafeURL(ValueError):
pass
def _is_private(address):
ip = ipaddress.ip_address(address)
if isinstance(ip, ipaddress.IPv6Address):
if ip.ipv4_mapped is not None:
ip = ip.ipv4_mapped
elif ip in NAT64_PREFIX:
ip = ipaddress.IPv4Address(int(ip) & 0xFFFFFFFF)
return (
ip.is_private
or ip.is_loopback
or ip.is_link_local
or ip.is_reserved
or ip.is_multicast
or ip.is_unspecified
)
def _resolve(host, port):
return {info[4][0] for info in socket.getaddrinfo(host, port, proto=socket.IPPROTO_TCP)}
async def guard_public_host(host, port=443):
if not host:
raise UnsafeURL("missing host")
try:
addresses = await asyncio.to_thread(_resolve, host, port)
except socket.gaierror as exc:
raise UnsafeURL(f"cannot resolve {host}") from exc
for address in addresses:
if _is_private(address):
raise UnsafeURL(f"{host} resolves to a non-public address")
return addresses
async def guard_public_url(url):
parts = urlsplit(url)
if parts.scheme not in ("http", "https"):
raise UnsafeURL("scheme must be http or https")
port = parts.port or (443 if parts.scheme == "https" else 80)
await guard_public_host(parts.hostname, port)
return url
class GuardedTransport(httpx.AsyncBaseTransport):
def __init__(self, inner):
self.inner = inner
async def handle_async_request(self, request):
await guard_public_url(str(request.url))
return await self.inner.handle_async_request(request)
async def aclose(self):
await self.inner.aclose()
molodetz/rendering.py
+167
View File
@@ -0,0 +1,167 @@
# retoor <retoor@molodetz.nl>
import re
from functools import lru_cache
import mistune
from bs4 import BeautifulSoup, NavigableString
from mistune.renderers.html import HTMLRenderer
from molodetz.emoji_builder import shortcode_map
_DASH_ENTITIES = ["&" + name + ";" for name in ("m" + "dash", "n" + "dash", "#8212", "#8211", "#x2014", "#x2013")]
DASH_PATTERN = re.compile("|".join(["\u2014", "\u2013", *_DASH_ENTITIES]), re.IGNORECASE)
SHORTCODE_PATTERN = re.compile(r":([a-z0-9_+\-]+):")
URL_PATTERN = re.compile(r"(?<![\"'=])(https?://[^\s<>\"']+)")
MENTION_PATTERN = re.compile(r"(?<![\w/])@([A-Za-z0-9_-]{3,32})")
IMAGE_EXTENSIONS = (".png", ".jpg", ".jpeg", ".gif", ".webp", ".avif")
VIDEO_EXTENSIONS = (".mp4", ".webm")
AUDIO_EXTENSIONS = (".mp3", ".ogg", ".wav")
ALLOWED_SCHEMES = ("http:", "https:", "mailto:", "tel:")
SKIP_PARENTS = {"a", "code", "pre"}
def normalize_dashes(text):
return DASH_PATTERN.sub("-", text or "")
def replace_shortcodes(text):
mapping = shortcode_map()
return SHORTCODE_PATTERN.sub(lambda m: mapping.get(m.group(1), m.group(0)), text)
def is_allowed_url(url):
lowered = (url or "").strip().lower()
if lowered.startswith(("/", "#")) and not lowered.startswith("//"):
return True
return lowered.startswith(ALLOWED_SCHEMES)
class ContentRenderer(HTMLRenderer):
def safe_url(self, url):
if not is_allowed_url(url):
return "#"
return super().safe_url(url)
def link(self, text, url, title=None):
href = self.safe_url(url)
extra = ' rel="nofollow noopener" target="_blank"' if href.startswith("http") else ""
title_attr = f' title="{mistune.escape(title)}"' if title else ""
return f'<a href="{href}"{title_attr}{extra}>{text}</a>'
def image(self, text, url, title=None):
src = self.safe_url(url)
alt = mistune.escape(text or "")
return f'<img src="{src}" alt="{alt}" loading="lazy" data-lightbox>'
_markdown = mistune.create_markdown(
escape=True,
hard_wrap=True,
renderer=ContentRenderer(escape=True),
plugins=["strikethrough", "table", "url"],
)
_inline_markdown = mistune.create_markdown(escape=True, renderer=ContentRenderer(escape=True), plugins=["strikethrough"])
def _embed_for(url):
path = url.lower().split("?", 1)[0]
if path.endswith(IMAGE_EXTENSIONS):
return f'<img src="{url}" alt="" loading="lazy" data-lightbox>'
if path.endswith(VIDEO_EXTENSIONS):
return f'<video src="{url}" controls preload="metadata"></video>'
if path.endswith(AUDIO_EXTENSIONS):
return f'<audio src="{url}" controls preload="none"></audio>'
return f'<a href="{url}" rel="nofollow noopener" target="_blank">{url}</a>'
def _media_pass(html):
soup = BeautifulSoup(html, "html.parser")
for anchor in list(soup.find_all("a")):
href = anchor.get("href", "")
if anchor.get_text() == href and any(parent.name in ("code", "pre") for parent in anchor.parents) is False:
embed = _embed_for(href)
if not embed.startswith("<a "):
anchor.replace_with(BeautifulSoup(embed, "html.parser"))
for node in list(soup.find_all(string=True)):
if not isinstance(node, NavigableString) or not node.strip():
continue
if any(parent.name in SKIP_PARENTS for parent in node.parents):
continue
text = str(node)
if not MENTION_PATTERN.search(text):
continue
escaped = mistune.escape(text)
replaced = MENTION_PATTERN.sub(lambda m: f'<a class="mention" href="/mensen/{m.group(1)}">@{m.group(1)}</a>', escaped)
fragment = BeautifulSoup(replaced, "html.parser")
node.replace_with(*list(fragment.contents))
return str(soup)
@lru_cache(maxsize=2048)
def render_content(text):
prepared = replace_shortcodes(normalize_dashes(text))
html = _markdown(prepared)
return _media_pass(html)
@lru_cache(maxsize=2048)
def render_title(text):
prepared = replace_shortcodes(normalize_dashes(text))
html = _inline_markdown(prepared).strip()
if html.startswith("<p>") and html.endswith("</p>"):
html = html[3:-4]
soup = BeautifulSoup(html, "html.parser")
for tag in soup.find_all(True):
if tag.name not in ("em", "strong", "code", "del"):
tag.unwrap()
return str(soup)
def plain_text(text):
html = render_content(text or "")
return BeautifulSoup(html, "lxml").get_text(" ", strip=True)
def safe_truncate(text, limit=180):
text = re.sub(r"\s+", " ", text or "").strip()
if len(text) <= limit:
return text
tokens = text[:limit].split(" ")
if len(tokens) > 1:
tokens = tokens[:-1]
return " ".join(tokens).rstrip(",.;:") + "..."
def content_preview(text, limit=180):
return safe_truncate(plain_text(text), limit)
def plain_title(text):
return BeautifulSoup(render_title(text or ""), "html.parser").get_text()
def markdown_structure_signature(text):
text = text or ""
return {
"code_fences": len(re.findall(r"^```", text, re.MULTILINE)) // 2,
"list_items": len(re.findall(r"^\s*(?:[-*+]|\d+\.)\s+", text, re.MULTILINE)),
"headers": len(re.findall(r"^#{1,6}\s", text, re.MULTILINE)),
"links": len(re.findall(r"\[[^\]]+\]\([^)]+\)", text)),
}
def extract_preview_urls(text, self_host=""):
urls = []
for match in URL_PATTERN.finditer(text or ""):
url = match.group(1)
lowered = url.lower().split("?", 1)[0]
if lowered.endswith(IMAGE_EXTENSIONS + VIDEO_EXTENSIONS + AUDIO_EXTENSIONS):
continue
if self_host and self_host in url:
continue
if url not in urls:
urls.append(url)
if len(urls) == 4:
break
return urls
molodetz/responses.py
+40
View File
@@ -0,0 +1,40 @@
# retoor <retoor@molodetz.nl>
import logging
from fastapi.responses import JSONResponse, RedirectResponse
logger = logging.getLogger(__name__)
def wants_json(request):
content_type = request.headers.get("content-type", "")
if content_type.startswith("application/json"):
return True
accept = request.headers.get("accept", "")
return "application/json" in accept and "text/html" not in accept
def json_error(status_code, message, **extra):
body = {"status": status_code, "message": message}
body.update(extra)
return JSONResponse({"error": body}, status_code=status_code)
def respond(request, template, ctx, *, model, status_code=200):
if wants_json(request):
try:
payload = model.model_validate(ctx).model_dump(mode="json")
except Exception:
logger.exception("schema validation failed for %s", model.__name__)
return json_error(500, "Response serialization failed")
return JSONResponse(payload, status_code=status_code)
from molodetz.templating import templates
context = dict(ctx)
return templates.TemplateResponse(request, template, context, status_code=status_code)
def action_result(request, redirect_url, *, data=None, status_code=302):
if wants_json(request):
return JSONResponse({"ok": True, "redirect": redirect_url, "data": data})
return RedirectResponse(redirect_url, status_code=status_code)
molodetz/routers/__init__.py
+1
View File
@@ -0,0 +1 @@
# retoor <retoor@molodetz.nl>
molodetz/routers/_gallery.py
+24
View File
@@ -0,0 +1,24 @@
# retoor <retoor@molodetz.nl>
from molodetz.constants import MEDIA_KINDS
from molodetz.gallery import gallery_items
from molodetz.responses import respond
from molodetz.schemas.gallery import GalleryOut
from molodetz.seo import base_seo_context
from molodetz.seo_meta_text import description_for
from molodetz.templating import static_url
def gallery_page(request, kind, slug):
title = MEDIA_KINDS[kind]
items = gallery_items(kind)
for item in items:
item["url"] = static_url(item["source_path"])
first = items[0]["url"] if items else None
seo = base_seo_context(
request,
title,
description_for(slug),
og_image=first,
breadcrumbs=[("Molodetz", "/"), (title, f"/{slug}")],
)
return respond(request, "gallery.html", {"kind": kind, "title": title, "items": items, "seo": seo}, model=GalleryOut)
molodetz/routers/_listing.py
+25
View File
@@ -0,0 +1,25 @@
# retoor <retoor@molodetz.nl>
from molodetz.constants import TOPICS
from molodetz.database import published_posts
from molodetz.responses import respond
from molodetz.schemas.posts import PostListOut
from molodetz.seo import base_seo_context, website_schema
from molodetz.seo_meta_text import description_for
def topic_listing(request, topic):
before = request.query_params.get("before")
posts, next_cursor = published_posts(topic=topic, before=before, limit=25)
title = TOPICS[topic]
next_url = f"{request.url.path}?before={next_cursor}" if next_cursor else None
seo = base_seo_context(
request,
title,
description_for(topic),
breadcrumbs=[("Molodetz", "/"), (title, f"/{topic}")],
schemas=[website_schema(request)],
next_url=next_url,
robots="noindex,follow" if before else "index,follow",
)
ctx = {"title": title, "topic": topic, "posts": posts, "next_cursor": next_cursor, "next_url": next_url, "seo": seo}
return respond(request, "listing.html", ctx, model=PostListOut)
molodetz/routers/admin/__init__.py
+13
View File
@@ -0,0 +1,13 @@
# retoor <retoor@molodetz.nl>
from molodetz.routers.admin import audit, backups, joins, posts, services, settings, stats, trash, users
from molodetz.routers.admin.index import router
router.include_router(posts.router)
router.include_router(joins.router)
router.include_router(services.router)
router.include_router(trash.router)
router.include_router(settings.router)
router.include_router(users.router)
router.include_router(audit.router)
router.include_router(backups.router)
router.include_router(stats.router)
molodetz/routers/admin/_shared.py
+6
View File
@@ -0,0 +1,6 @@
# retoor <retoor@molodetz.nl>
from molodetz.seo import base_seo_context
def admin_seo(request, title):
return base_seo_context(request, title, robots="noindex,nofollow")
molodetz/routers/admin/audit.py
+17
View File
@@ -0,0 +1,17 @@
# retoor <retoor@molodetz.nl>
from fastapi import APIRouter, Request
from molodetz.database import list_audit
from molodetz.responses import respond
from molodetz.routers.admin._shared import admin_seo
from molodetz.schemas.admin import AuditOut
from molodetz.utils.auth import require_admin
router = APIRouter()
@router.get("/audit")
async def audit_index(request: Request):
require_admin(request)
category = request.query_params.get("category") or None
return respond(request, "admin/audit.html", {"rows": list_audit(200, category), "category": category, "seo": admin_seo(request, "Auditlog")}, model=AuditOut)
molodetz/routers/admin/backups.py
+60
View File
@@ -0,0 +1,60 @@
# retoor <retoor@molodetz.nl>
import aiofiles
from fastapi import APIRouter, HTTPException, Request
from fastapi.responses import StreamingResponse
from molodetz.attachments import format_size
from molodetz.database import enqueue_backup, get_backup, list_backups
from molodetz.dependencies import json_or_form
from molodetz.models import BackupForm
from molodetz.responses import action_result, respond
from molodetz.routers.admin._shared import admin_seo
from molodetz.schemas.admin import BackupsOut
from molodetz.services.backup import backup_path
from molodetz.utils.audit import record
from molodetz.utils.auth import is_primary_admin, not_found, require_admin
router = APIRouter()
@router.get("/backups")
async def backups_index(request: Request):
user = require_admin(request)
rows = [{**row, "size_label": format_size(row.get("size_bytes"))} for row in list_backups()]
ctx = {"backups": rows, "can_download": is_primary_admin(user), "seo": admin_seo(request, "Back-ups")}
return respond(request, "admin/backups.html", ctx, model=BackupsOut)
@router.post("/backups")
async def backups_enqueue(request: Request):
user = require_admin(request)
data = await json_or_form(request, BackupForm)
row = enqueue_backup(data.target, user["uid"])
record(request, "backup.enqueue", payload={"target": data.target}, targets=[("backup", row["uid"])], user=user)
return action_result(request, "/admin/backups", data={"uid": row["uid"]})
@router.get("/backups/{uid}/download")
async def backups_download(request: Request, uid: str):
user = require_admin(request)
if not is_primary_admin(user):
record(request, "backup.download", result="denied", targets=[("backup", uid)], user=user)
raise HTTPException(403, "Alleen de primaire beheerder")
row = get_backup(uid)
if row is None or row.get("status") != "done":
not_found("Back-up niet gevonden")
path = backup_path(row)
if not path.exists():
not_found("Archief ontbreekt")
record(request, "backup.download", targets=[("backup", uid)], user=user)
return StreamingResponse(
_stream_file(path),
media_type="application/octet-stream",
headers={"Content-Disposition": f'attachment; filename="{row["stored_name"]}"', "Content-Length": str(path.stat().st_size)},
)
async def _stream_file(path, chunk_size=1024 * 256):
async with aiofiles.open(path, "rb") as handle:
while chunk := await handle.read(chunk_size):
yield chunk
molodetz/routers/admin/index.py
+24
View File
@@ -0,0 +1,24 @@
# retoor <retoor@molodetz.nl>
from fastapi import APIRouter, Request
from molodetz.constants import SOFT_DELETE_TABLES
from molodetz.database import all_posts, count_deleted, count_open_join_requests, count_users
from molodetz.responses import respond
from molodetz.routers.admin._shared import admin_seo
from molodetz.schemas.admin import AdminIndexOut
from molodetz.utils.auth import require_admin
router = APIRouter()
@router.get("")
async def admin_index(request: Request):
require_admin(request)
ctx = {
"post_count": len(all_posts()),
"open_joins": count_open_join_requests(),
"user_count": count_users(),
"deleted_count": sum(count_deleted(name) for name in SOFT_DELETE_TABLES),
"seo": admin_seo(request, "Beheer"),
}
return respond(request, "admin/index.html", ctx, model=AdminIndexOut)
molodetz/routers/admin/joins.py
+43
View File
@@ -0,0 +1,43 @@
# retoor <retoor@molodetz.nl>
from fastapi import APIRouter, Request
from molodetz.database import get_join_request, list_join_requests, update_join_request
from molodetz.dependencies import json_or_form
from molodetz.link_check import check_repo_link
from molodetz.models import JoinStatusForm
from molodetz.responses import action_result, respond
from molodetz.routers.admin._shared import admin_seo
from molodetz.schemas.joins import JoinListOut
from molodetz.utils.audit import record
from molodetz.utils.auth import not_found, require_admin
router = APIRouter()
@router.get("/joins")
async def joins_index(request: Request):
require_admin(request)
return respond(request, "admin/joins.html", {"requests": list_join_requests(), "seo": admin_seo(request, "Aanmeldingen")}, model=JoinListOut)
@router.post("/joins/{uid}/status")
async def joins_status(request: Request, uid: str):
user = require_admin(request)
row = get_join_request(uid)
if row is None:
not_found("Aanmelding niet gevonden")
data = await json_or_form(request, JoinStatusForm)
update_join_request(uid, status=data.status)
record(request, "join.request.status", payload={"status": data.status}, targets=[("join_request", uid)], user=user)
return action_result(request, "/admin/joins", data={"uid": uid, "status": data.status})
@router.post("/joins/{uid}/check")
async def joins_check(request: Request, uid: str):
user = require_admin(request)
row = get_join_request(uid)
if row is None:
not_found("Aanmelding niet gevonden")
result = await check_repo_link(row)
record(request, "join.request.check", payload=result, targets=[("join_request", uid)], user=user)
return action_result(request, "/admin/joins", data=result)
molodetz/routers/admin/posts.py
+88
View File
@@ -0,0 +1,88 @@
# retoor <retoor@molodetz.nl>
from fastapi import APIRouter, HTTPException, Request
from molodetz.constants import TOPICS
from molodetz.content import create_post, delete_post, edit_post
from molodetz.database import all_posts, get_post
from molodetz.dependencies import json_or_form
from molodetz.models import PostForm
from molodetz.responses import action_result, respond
from molodetz.routers.admin._shared import admin_seo
from molodetz.schemas.admin import AdminPostsOut
from molodetz.schemas.posts import PostDetailOut
from molodetz.utils.audit import record
from molodetz.utils.auth import is_admin, is_owner, not_found, require_admin
router = APIRouter()
@router.get("/posts")
async def posts_index(request: Request):
require_admin(request)
return respond(request, "admin/posts.html", {"posts": all_posts(), "seo": admin_seo(request, "Berichten")}, model=AdminPostsOut)
@router.get("/posts/new")
async def posts_new(request: Request):
require_admin(request)
ctx = {"post": None, "topics": TOPICS, "errors": [], "seo": admin_seo(request, "Nieuw bericht")}
return respond(request, "admin/post_form.html", ctx, model=PostDetailOut)
@router.post("/posts")
async def posts_create(request: Request):
user = require_admin(request)
data = await json_or_form(request, PostForm)
row = create_post(user, data.title, data.body, data.topic, data.status, data.is_placeholder, request=request)
return action_result(request, "/admin/posts", data={"uid": row["uid"], "slug": row["slug"]})
@router.get("/posts/{uid}/edit")
async def posts_edit(request: Request, uid: str):
require_admin(request)
post = get_post(uid)
if post is None:
not_found("Bericht niet gevonden")
ctx = {"post": post, "topics": TOPICS, "errors": [], "seo": admin_seo(request, "Bericht bewerken")}
return respond(request, "admin/post_form.html", ctx, model=PostDetailOut)
def _owned_post(request, user, uid):
post = get_post(uid)
if post is None:
not_found("Bericht niet gevonden")
if not is_owner(post, user):
record(request, "post.edit.denied", result="denied", targets=[("post", uid)], user=user)
raise HTTPException(403, "Alleen de auteur mag dit bericht bewerken")
return post
@router.post("/posts/{uid}")
async def posts_update(request: Request, uid: str):
user = require_admin(request)
post = _owned_post(request, user, uid)
data = await json_or_form(request, PostForm)
edit_post(post, user, data.title, data.body, data.topic, data.status, data.is_placeholder, request=request)
return action_result(request, "/admin/posts", data={"uid": uid})
@router.post("/posts/{uid}/publish")
async def posts_publish(request: Request, uid: str):
user = require_admin(request)
post = _owned_post(request, user, uid)
status = "draft" if post["status"] == "published" else "published"
edit_post(post, user, post["title"], post["body"], post["topic"], status, post.get("is_placeholder"), request=request)
return action_result(request, "/admin/posts", data={"uid": uid, "status": status})
@router.post("/posts/{uid}/delete")
async def posts_delete(request: Request, uid: str):
user = require_admin(request)
post = get_post(uid)
if post is None:
not_found("Bericht niet gevonden")
if not (is_owner(post, user) or is_admin(user)):
record(request, "post.delete.denied", result="denied", targets=[("post", uid)], user=user)
raise HTTPException(403, "Niet toegestaan")
stamp = delete_post(post, user, request=request)
return action_result(request, "/admin/posts", data={"uid": uid, "stamp": stamp})
molodetz/routers/admin/services.py
+69
View File
@@ -0,0 +1,69 @@
# retoor <retoor@molodetz.nl>
from fastapi import APIRouter, HTTPException, Request
from molodetz.responses import action_result, respond
from molodetz.routers.admin._shared import admin_seo
from molodetz.schemas.admin import ServiceDetailOut, ServicesOut
from molodetz.services.base import manager
from molodetz.utils.audit import record
from molodetz.utils.auth import not_found, require_admin
router = APIRouter()
VERBS = ("start", "stop", "run", "clear")
def _service(name):
service = manager.get(name)
if service is None:
not_found("Onbekende service")
return service
@router.get("/services")
async def services_index(request: Request):
require_admin(request)
return respond(request, "admin/services.html", {"services": manager.describe_all(), "seo": admin_seo(request, "Services")}, model=ServicesOut)
@router.get("/services/{name}")
async def services_detail(request: Request, name: str):
require_admin(request)
service = _service(name)
ctx = {"service": manager.describe(service), "seo": admin_seo(request, service.title)}
return respond(request, "admin/service.html", ctx, model=ServiceDetailOut)
async def _payload(request):
if request.headers.get("content-type", "").startswith("application/json"):
data = await request.json()
return data if isinstance(data, dict) else {}
form = await request.form()
return {key: value for key, value in form.items()}
@router.post("/services/{name}/command")
async def services_command(request: Request, name: str):
user = require_admin(request)
service = _service(name)
verb = (await _payload(request)).get("verb", "")
if verb not in VERBS:
raise HTTPException(400, "Onbekend commando")
if verb in ("start", "stop"):
manager.set_enabled(service.name, verb == "start")
else:
manager.send_command(service.name, verb)
record(request, f"service.command.{verb}", targets=[("service", service.name)], user=user)
return action_result(request, f"/admin/services/{service.name}", data={"verb": verb})
@router.post("/services/{name}/config")
async def services_config(request: Request, name: str):
user = require_admin(request)
service = _service(name)
values = await _payload(request)
try:
updates = manager.save_config(service.name, values)
except ValueError as exc:
raise HTTPException(400, str(exc)) from exc
record(request, "service.config.save", payload={"keys": sorted(updates)}, targets=[("service", service.name)], user=user)
return action_result(request, f"/admin/services/{service.name}", data={"saved": sorted(updates)})
molodetz/routers/admin/settings.py
+43
View File
@@ -0,0 +1,43 @@
# retoor <retoor@molodetz.nl>
from fastapi import APIRouter, Request
from molodetz.database import all_settings, get_setting, set_setting
from molodetz.dependencies import json_or_form
from molodetz.models import SettingsForm
from molodetz.responses import action_result, respond
from molodetz.routers.admin._shared import admin_seo
from molodetz.schemas.admin import SettingsOut
from molodetz.utils.audit import record
from molodetz.utils.auth import require_admin
router = APIRouter()
NUMERIC = ("rate_limit_per_minute", "rate_limit_window_seconds", "session_max_age_days", "session_remember_days", "audit_retention_days")
BOOLEAN = ("maintenance_mode", "friendly_404")
@router.get("/settings")
async def settings_index(request: Request):
require_admin(request)
return respond(request, "admin/settings.html", {"settings": all_settings(), "seo": admin_seo(request, "Instellingen")}, model=SettingsOut)
@router.post("/settings")
async def settings_save(request: Request):
user = require_admin(request)
data = await json_or_form(request, SettingsForm)
changed = []
for key, value in data.model_dump(exclude_none=True).items():
if value == "":
continue
if key in NUMERIC:
try:
value = str(max(1, int(value)))
except ValueError:
continue
if key in BOOLEAN and value not in ("0", "1"):
continue
if get_setting(key) != value:
set_setting(key, value)
changed.append(key)
record(request, "admin.settings.save", payload={"keys": changed}, user=user)
return action_result(request, "/admin/settings", data={"changed": changed})
molodetz/routers/admin/stats.py
+17
View File
@@ -0,0 +1,17 @@
# retoor <retoor@molodetz.nl>
from fastapi import APIRouter, Request
from molodetz.database import daily_totals, top_paths
from molodetz.responses import respond
from molodetz.routers.admin._shared import admin_seo
from molodetz.schemas.admin import StatsOut
from molodetz.utils.auth import require_admin
router = APIRouter()
@router.get("/stats")
async def stats_index(request: Request):
require_admin(request)
ctx = {"top_paths": [dict(row) for row in top_paths()], "daily": [dict(row) for row in daily_totals()], "seo": admin_seo(request, "Statistieken")}
return respond(request, "admin/stats.html", ctx, model=StatsOut)
molodetz/routers/admin/trash.py
+62
View File
@@ -0,0 +1,62 @@
# retoor <retoor@molodetz.nl>
from fastapi import APIRouter, HTTPException, Request
from molodetz.constants import SOFT_DELETE_TABLES
from molodetz.database import list_deleted, purge_event, restore_event
from molodetz.responses import action_result, respond
from molodetz.routers.admin._shared import admin_seo
from molodetz.schemas.admin import TrashOut
from molodetz.utils.audit import record
from molodetz.utils.auth import require_admin
router = APIRouter()
LABEL_FIELDS = ("title", "name", "username", "message", "filename", "token")
def _label(row):
for key in LABEL_FIELDS:
if row.get(key):
return str(row[key])[:24] if key == "token" else str(row[key])[:120]
return row.get("uid", "")
@router.get("/trash")
async def trash_index(request: Request):
require_admin(request)
items = []
for table in SOFT_DELETE_TABLES:
if table == "sessions":
continue
for row in list_deleted(table):
items.append({"table": table, "uid": row.get("uid"), "label": _label(row), "deleted_at": row.get("deleted_at"), "deleted_by": row.get("deleted_by")})
items.sort(key=lambda item: item["deleted_at"] or "", reverse=True)
return respond(request, "admin/trash.html", {"items": items, "seo": admin_seo(request, "Prullenbak")}, model=TrashOut)
async def _stamp(request):
if request.headers.get("content-type", "").startswith("application/json"):
data = await request.json()
stamp = data.get("stamp") if isinstance(data, dict) else None
else:
stamp = (await request.form()).get("stamp")
if not stamp:
raise HTTPException(400, "stamp ontbreekt")
return str(stamp)
@router.post("/trash/restore")
async def trash_restore(request: Request):
user = require_admin(request)
stamp = await _stamp(request)
tables = restore_event(stamp)
record(request, "admin.trash.restore", payload={"stamp": stamp}, user=user)
return action_result(request, "/admin/trash", data={"tables": tables})
@router.post("/trash/purge")
async def trash_purge(request: Request):
user = require_admin(request)
stamp = await _stamp(request)
purged = purge_event(stamp)
record(request, "admin.trash.purge", payload={"stamp": stamp, "tables": [name for name, _ in purged]}, user=user)
return action_result(request, "/admin/trash", data={"purged": [name for name, _ in purged]})
molodetz/routers/admin/users.py
+35
View File
@@ -0,0 +1,35 @@
# retoor <retoor@molodetz.nl>
from fastapi import APIRouter, HTTPException, Request
from molodetz.database import get_user, list_users, set_role
from molodetz.dependencies import json_or_form
from molodetz.models import RoleForm
from molodetz.responses import action_result, respond
from molodetz.routers.admin._shared import admin_seo
from molodetz.schemas.admin import AdminUsersOut
from molodetz.utils.audit import record
from molodetz.utils.auth import invalidate_user_cache, is_senior_admin, not_found, require_admin
router = APIRouter()
@router.get("/users")
async def users_index(request: Request):
require_admin(request)
return respond(request, "admin/users.html", {"users": list_users(), "seo": admin_seo(request, "Accounts")}, model=AdminUsersOut)
@router.post("/users/{uid}/role")
async def users_role(request: Request, uid: str):
actor = require_admin(request)
target = get_user(uid)
if target is None:
not_found("Account niet gevonden")
data = await json_or_form(request, RoleForm)
if is_senior_admin(actor, target) or (target["uid"] == actor["uid"] and data.role != "Admin"):
record(request, "admin.user.role", result="denied", targets=[("user", uid)], user=actor)
raise HTTPException(403, "Je kunt een senior beheerder niet beheren")
set_role(uid, data.role)
invalidate_user_cache(propagate=True)
record(request, "admin.user.role", payload={"role": data.role}, targets=[("user", uid)], user=actor)
return action_result(request, "/admin/users", data={"uid": uid, "role": data.role})
molodetz/routers/auth.py
+70
View File
@@ -0,0 +1,70 @@
# retoor <retoor@molodetz.nl>
from fastapi import APIRouter, Request
from molodetz.database import create_session, end_session, get_int_setting, get_user_by_email, get_user_by_username
from molodetz.dependencies import json_or_form
from molodetz.models import LoginForm
from molodetz.responses import action_result, json_error, respond, wants_json
from molodetz.schemas.account import LoginPageOut
from molodetz.seo import base_seo_context
from molodetz.templating import templates
from molodetz.utils.audit import record
from molodetz.utils.auth import get_current_user, invalidate_user_cache, is_account_active, verify_password_async
from molodetz.utils.request_meta import is_secure_request, safe_next
router = APIRouter()
def _seo(request):
return base_seo_context(request, "Inloggen", robots="noindex,nofollow")
@router.get("/login")
async def login_page(request: Request):
next_url = safe_next(request.query_params.get("next"), "/admin")
return respond(request, "auth/login.html", {"errors": [], "form": {}, "next": next_url, "seo": _seo(request)}, model=LoginPageOut)
@router.post("/login")
async def login_submit(request: Request):
data = await json_or_form(request, LoginForm)
user = get_user_by_username(data.username) or get_user_by_email(data.username.lower())
valid = user is not None and is_account_active(user) and await verify_password_async(data.password, user.get("password_hash"))
if not valid:
record(request, "auth.login.failed", result="denied", payload={"username": data.username[:64]})
ctx = {
"errors": ["Onbekende combinatie van naam en wachtwoord."],
"form": {"username": data.username},
"next": data.next,
"seo": _seo(request),
}
if wants_json(request):
return json_error(401, "Invalid credentials")
return templates.TemplateResponse(request, "auth/login.html", ctx, status_code=400)
days = get_int_setting("session_remember_days", 30) if data.remember else get_int_setting("session_max_age_days", 7)
max_age = max(1, days) * 86400
token = create_session(user["uid"], max_age)
record(request, "auth.login", targets=[("user", user["uid"])], user=user)
response = action_result(request, safe_next(data.next, "/admin"))
response.set_cookie(
key="session",
value=token,
max_age=max_age,
httponly=True,
samesite="lax",
secure=is_secure_request(request),
)
return response
@router.post("/logout")
async def logout(request: Request):
user = get_current_user(request)
token = request.cookies.get("session")
if token:
end_session(token, user["uid"] if user else "guest")
invalidate_user_cache(propagate=True)
record(request, "auth.logout", user=user)
response = action_result(request, "/")
response.delete_cookie("session")
return response
molodetz/routers/avatar.py
+13
View File
@@ -0,0 +1,13 @@
# retoor <retoor@molodetz.nl>
from fastapi import APIRouter
from fastapi.responses import Response
from molodetz.avatar import render_avatar_svg
router = APIRouter()
@router.get("/svg/{seed}")
async def avatar_svg(seed: str, size: int = 64):
svg = render_avatar_svg(seed[:128], size)
return Response(svg, media_type="image/svg+xml", headers={"Cache-Control": "public, max-age=86400"})
molodetz/routers/binnen.py
+42
View File
@@ -0,0 +1,42 @@
# retoor <retoor@molodetz.nl>
import hashlib
from fastapi import APIRouter, Request
from molodetz import config
from molodetz.database import admin_uids_ordered, insert_join_request
from molodetz.dependencies import json_or_form
from molodetz.models import JoinForm
from molodetz.responses import action_result, respond
from molodetz.schemas.joins import JoinPageOut
from molodetz.seo import base_seo_context
from molodetz.seo_meta_text import description_for
from molodetz.utils.audit import record
from molodetz.utils.notifications import create_notification
from molodetz.utils.request_meta import client_ip
router = APIRouter()
def _seo(request):
return base_seo_context(request, "Binnen", description_for("binnen"), breadcrumbs=[("Molodetz", "/"), ("Binnen", "/binnen")])
@router.get("")
async def join_page(request: Request):
submitted = request.query_params.get("ok") == "1"
return respond(request, "binnen.html", {"submitted": submitted, "errors": [], "form": {}, "seo": _seo(request)}, model=JoinPageOut)
@router.post("")
async def join_submit(request: Request):
data = await json_or_form(request, JoinForm)
if data.website:
record(request, "join.request.honeypot", result="denied")
return action_result(request, "/binnen?ok=1")
ip_hash = hashlib.sha256(f"{config.SECRET_KEY}:{client_ip(request)}".encode()).hexdigest()[:24]
row = insert_join_request(data.name, data.contact, data.repo_url, data.message, ip_hash)
record(request, "join.request.create", payload={"uid": row["uid"]}, targets=[("join_request", row["uid"])])
for uid in admin_uids_ordered():
create_notification(uid, "join.request", f"{data.name} wil meeschrijven", "/admin/joins")
return action_result(request, "/binnen?ok=1", data={"uid": row["uid"]})
molodetz/routers/docs.py
+76
View File
@@ -0,0 +1,76 @@
# retoor <retoor@molodetz.nl>
from fastapi import APIRouter, Request
from fastapi.responses import HTMLResponse, PlainTextResponse
from molodetz.docs_api import group_by_slug, ordered_groups, substitute
from molodetz.docs_export import export_html, export_markdown
from molodetz.docs_live import live_services
from molodetz.docs_prose import page_by_slug, render_page, sidebar
from molodetz.docs_search import search
from molodetz.seo import base_seo_context, base_url
from molodetz.templating import templates
from molodetz.utils.auth import get_current_user, is_admin, not_found
router = APIRouter()
def _ctx(request, title, **extra):
viewer = get_current_user(request)
viewer_is_admin = is_admin(viewer)
groups = [group for group in ordered_groups() if viewer_is_admin or group["slug"] != "beheer"]
seo = base_seo_context(request, title, breadcrumbs=[("Molodetz", "/"), ("Docs", "/docs")])
return {"sidebar": sidebar(viewer_is_admin), "api_groups": groups, "viewer_is_admin": viewer_is_admin, "seo": seo, "doc_title": title, **extra}
@router.get("/docs")
async def docs_index(request: Request):
return templates.TemplateResponse(request, "docs/index.html", _ctx(request, "Documentatie", body=render_page("welkom")))
@router.get("/docs/search")
async def docs_search(request: Request):
query = request.query_params.get("q", "")[:200]
viewer_is_admin = is_admin(get_current_user(request))
results = search(query, viewer_is_admin)
return templates.TemplateResponse(request, "docs/search.html", _ctx(request, "Zoeken", query=query, results=results))
@router.get("/docs/download.md")
async def docs_download_md(request: Request):
viewer_is_admin = is_admin(get_current_user(request))
return PlainTextResponse(export_markdown(viewer_is_admin), media_type="text/markdown; charset=utf-8")
@router.get("/docs/download.html")
async def docs_download_html(request: Request):
viewer_is_admin = is_admin(get_current_user(request))
return HTMLResponse(export_html(viewer_is_admin))
@router.get("/docs/api/{group_slug}")
async def docs_api_group(request: Request, group_slug: str):
group = group_by_slug(group_slug)
viewer = get_current_user(request)
if group is None or (group["slug"] == "beheer" and not is_admin(viewer)):
not_found("Onbekende API-groep")
base = base_url(request)
username = viewer["username"] if viewer else "retoor"
api_key = viewer["api_key"] if viewer else "JOUW_API_SLEUTEL"
endpoints = []
for item in group["endpoints"]:
example = substitute(f"curl -H 'Accept: application/json' -H 'X-API-KEY: {{{{api_key}}}}' {{{{base}}}}{item['path']}", base, username, api_key)
endpoints.append({**item, "curl": example})
return templates.TemplateResponse(
request,
"docs/api.html",
_ctx(request, f"API: {group['title']}", group=group, endpoints=endpoints, viewer=viewer, services=live_services() if group["slug"] == "beheer" else []),
)
@router.get("/docs/{slug}")
async def docs_page(request: Request, slug: str):
page = page_by_slug(slug)
viewer_is_admin = is_admin(get_current_user(request))
if page is None or (page.get("admin") and not viewer_is_admin):
not_found("Pagina niet gevonden")
return templates.TemplateResponse(request, "docs/page.html", _ctx(request, page["title"], body=render_page(slug), page=page))
molodetz/routers/flyers.py
+11
View File
@@ -0,0 +1,11 @@
# retoor <retoor@molodetz.nl>
from fastapi import APIRouter, Request
from molodetz.routers._gallery import gallery_page
router = APIRouter()
@router.get("")
async def flyers(request: Request):
return gallery_page(request, "flyer", "flyers")
molodetz/routers/health.py
+12
View File
@@ -0,0 +1,12 @@
# retoor <retoor@molodetz.nl>
from fastapi import APIRouter
from fastapi.responses import JSONResponse
from molodetz import config
router = APIRouter()
@router.get("")
async def health():
return JSONResponse({"status": "ok", "version": config.APP_VERSION})
molodetz/routers/home.py
+42
View File
@@ -0,0 +1,42 @@
# retoor <retoor@molodetz.nl>
from fastapi import APIRouter, Request
from molodetz import config
from molodetz.cache import TTLCache
from molodetz.database import count_media, published_posts
from molodetz.gallery import gallery_items
from molodetz.responses import respond
from molodetz.schemas.posts import HomeOut
from molodetz.seo import base_seo_context, organization_schema, website_schema
from molodetz.seo_meta_text import description_for
router = APIRouter()
_landing_cache = TTLCache(ttl=config.LANDING_TTL, max_size=4)
def _landing_blocks():
cached = _landing_cache.get("blocks")
if cached is not None:
return cached
posts, _ = published_posts(limit=8)
flyers = gallery_items("flyer")
blocks = {
"posts": posts,
"featured_flyer": flyers[0] if flyers else None,
"flyer_count": count_media("flyer"),
"meme_count": count_media("meme"),
}
_landing_cache.set("blocks", blocks)
return blocks
@router.get("/")
async def index(request: Request):
blocks = _landing_blocks()
seo = base_seo_context(
request,
config.APP_TITLE,
description_for("home"),
schemas=[website_schema(request), organization_schema(request)],
)
return respond(request, "home.html", {**blocks, "seo": seo}, model=HomeOut)
molodetz/routers/memes.py
+11
View File
@@ -0,0 +1,11 @@
# retoor <retoor@molodetz.nl>
from fastapi import APIRouter, Request
from molodetz.routers._gallery import gallery_page
router = APIRouter()
@router.get("")
async def memes(request: Request):
return gallery_page(request, "meme", "memes")
molodetz/routers/mensen.py
+42
View File
@@ -0,0 +1,42 @@
# retoor <retoor@molodetz.nl>
from fastapi import APIRouter, Request
from molodetz.database import get_user_by_username, post_counts_by_author, public_people, published_posts
from molodetz.responses import respond
from molodetz.schemas.people import PeopleOut, ProfileOut
from molodetz.seo import base_seo_context, profile_schema
from molodetz.seo_meta_text import description_for
from molodetz.utils.auth import not_found
router = APIRouter()
def _with_counts(people):
counts = post_counts_by_author([person["uid"] for person in people])
return [{**person, "post_count": counts.get(person["uid"], 0)} for person in people]
@router.get("")
async def people(request: Request):
persons = _with_counts(public_people())
seo = base_seo_context(request, "Mensen", description_for("mensen"), breadcrumbs=[("Molodetz", "/"), ("Mensen", "/mensen")])
return respond(request, "mensen.html", {"people": persons, "seo": seo}, model=PeopleOut)
@router.get("/{username}")
async def person(request: Request, username: str):
row = get_user_by_username(username)
if row is None:
not_found("Persoon niet gevonden")
person_row = _with_counts([row])[0]
own, _ = published_posts(limit=50, user_uid=row["uid"])
seo = base_seo_context(
request,
row["username"],
row.get("bio") or description_for("mensen"),
og_type="profile",
robots="index,follow" if person_row["post_count"] > 0 else "noindex,follow",
breadcrumbs=[("Molodetz", "/"), ("Mensen", "/mensen"), (row["username"], f"/mensen/{row['username']}")],
schemas=[profile_schema(request, row)],
)
return respond(request, "person.html", {"person": person_row, "posts": own, "seo": seo}, model=ProfileOut)
molodetz/routers/notifications.py
+26
View File
@@ -0,0 +1,26 @@
# retoor <retoor@molodetz.nl>
from fastapi import APIRouter, Request
from molodetz.database import list_notifications, mark_all_read, unread_count
from molodetz.responses import action_result, respond
from molodetz.schemas.account import NotificationsOut
from molodetz.seo import base_seo_context
from molodetz.utils.auth import require_user
router = APIRouter()
@router.get("")
async def notifications_page(request: Request):
user = require_user(request)
rows = list_notifications(user["uid"])
seo = base_seo_context(request, "Meldingen", robots="noindex,nofollow")
ctx = {"notifications": rows, "unread": unread_count(user["uid"]), "seo": seo}
return respond(request, "notifications.html", ctx, model=NotificationsOut)
@router.post("/read")
async def notifications_read(request: Request):
user = require_user(request)
mark_all_read(user["uid"])
return action_result(request, "/notifications")
molodetz/routers/posts.py
+36
View File
@@ -0,0 +1,36 @@
# retoor <retoor@molodetz.nl>
from fastapi import APIRouter, Request
from fastapi.responses import RedirectResponse
from molodetz.constants import TOPICS
from molodetz.database import get_post_by_slug, get_user
from molodetz.rendering import content_preview, plain_title
from molodetz.responses import respond
from molodetz.schemas.posts import PostDetailOut
from molodetz.seo import article_schema, base_seo_context
from molodetz.utils.auth import get_current_user, is_admin, not_found
router = APIRouter()
@router.get("/{slug}")
async def post_detail(request: Request, slug: str):
post = get_post_by_slug(slug)
viewer = get_current_user(request)
if post is None or (post["status"] != "published" and not is_admin(viewer)):
not_found("Bericht niet gevonden")
if slug != post["slug"]:
return RedirectResponse(f"/posts/{post['slug']}", status_code=301)
author = get_user(post["user_uid"])
topic_title = TOPICS.get(post["topic"], post["topic"])
seo = base_seo_context(
request,
plain_title(post["title"]),
content_preview(post["body"], 160),
robots="index,follow" if post["status"] == "published" else "noindex,nofollow",
og_type="article",
breadcrumbs=[("Molodetz", "/"), (topic_title, f"/{post['topic']}"), (post["title"], f"/posts/{post['slug']}")],
schemas=[article_schema(request, post, author)],
)
ctx = {"post": post, "author": author, "topic_title": topic_title, "seo": seo}
return respond(request, "post.html", ctx, model=PostDetailOut)
molodetz/routers/presence.py
+14
View File
@@ -0,0 +1,14 @@
# retoor <retoor@molodetz.nl>
from fastapi import APIRouter, Request
from fastapi.responses import JSONResponse
from molodetz import config
from molodetz.utils.presence import published_roster
router = APIRouter()
@router.get("/roster")
async def roster(request: Request):
tracked = published_roster()
return JSONResponse({"online": tracked[: config.PRESENCE_ONLINE_LIMIT], "tracked": len(tracked)})
molodetz/routers/privacy.py
+14
View File
@@ -0,0 +1,14 @@
# retoor <retoor@molodetz.nl>
from fastapi import APIRouter, Request
from molodetz.responses import respond
from molodetz.schemas.account import PageOut
from molodetz.seo import base_seo_context
router = APIRouter()
@router.get("")
async def privacy(request: Request):
seo = base_seo_context(request, "Privacy", breadcrumbs=[("Molodetz", "/"), ("Privacy", "/privacy")])
return respond(request, "privacy.html", {"title": "Privacy", "seo": seo}, model=PageOut)
molodetz/routers/profile.py
+39
View File
@@ -0,0 +1,39 @@
# retoor <retoor@molodetz.nl>
from fastapi import APIRouter, Request
from molodetz.database import generate_uid, update_user
from molodetz.responses import action_result, respond
from molodetz.schemas.account import ApiKeyOut
from molodetz.seo import base_seo_context
from molodetz.utils.audit import record
from molodetz.utils.auth import invalidate_user_cache, require_user
from molodetz.utils.notifications import create_notification
router = APIRouter()
@router.get("/api-key")
async def api_key_page(request: Request):
user = require_user(request)
seo = base_seo_context(request, "API-sleutel", robots="noindex,nofollow")
return respond(request, "profile/api_key.html", {"api_key": user["api_key"], "seo": seo}, model=ApiKeyOut)
@router.post("/api-key/regenerate")
async def api_key_regenerate(request: Request):
user = require_user(request)
key = generate_uid()
update_user(user["uid"], api_key=key)
invalidate_user_cache(propagate=True)
record(request, "profile.apikey.regenerate", targets=[("user", user["uid"])], user=user)
create_notification(user["uid"], "auth.apikey", "Je API-sleutel is vernieuwd.", "/profile/api-key")
return action_result(request, "/profile/api-key", data={"api_key": key})
@router.post("/avatar/regenerate")
async def avatar_regenerate(request: Request):
user = require_user(request)
update_user(user["uid"], avatar_seed=generate_uid())
invalidate_user_cache(propagate=True)
record(request, "profile.avatar.regenerate", targets=[("user", user["uid"])], user=user)
return action_result(request, f"/mensen/{user['username']}")
molodetz/routers/rol.py
+11
View File
@@ -0,0 +1,11 @@
# retoor <retoor@molodetz.nl>
from fastapi import APIRouter, Request
from molodetz.routers._listing import topic_listing
router = APIRouter()
@router.get("")
async def rol(request: Request):
return topic_listing(request, "rol")
molodetz/routers/seo.py
+71
View File
@@ -0,0 +1,71 @@
# retoor <retoor@molodetz.nl>
import logging
from xml.sax.saxutils import escape
import defusedxml.ElementTree as SafeET
from fastapi import APIRouter, Request
from fastapi.responses import PlainTextResponse, Response
from molodetz import config
from molodetz.cache import TTLCache
from molodetz.constants import TOPICS
from molodetz.database import db, public_people
from molodetz.docs_prose import DOCS_PAGES
from molodetz.seo import base_url
router = APIRouter()
logger = logging.getLogger(__name__)
_sitemap_cache = TTLCache(ttl=config.SITEMAP_TTL, max_size=4)
PER_TABLE_CAP = 5000
STATIC_PATHS = ("/", "/flyers", "/memes", "/mensen", "/binnen", "/voorwaarden", "/privacy", "/docs")
@router.get("/robots.txt")
async def robots(request: Request):
base = base_url(request)
body = "\n".join(
[
"User-agent: *",
"Disallow: /admin",
"Disallow: /auth",
"Disallow: /notifications",
"Disallow: /profile",
"Allow: /",
f"Sitemap: {base}/sitemap.xml",
"",
]
)
return PlainTextResponse(body)
def build_sitemap(base):
urls = [(path, None) for path in STATIC_PATHS]
urls += [(f"/{topic}", None) for topic in TOPICS]
if "posts" in db.tables:
rows = list(db["posts"].find(deleted_at=None, status="published", order_by=["-published_at"], _limit=PER_TABLE_CAP + 1))
if len(rows) > PER_TABLE_CAP:
logger.warning("sitemap truncated posts at %d", PER_TABLE_CAP)
rows = rows[:PER_TABLE_CAP]
urls += [(f"/posts/{row['slug']}", row.get("updated_at")) for row in rows]
urls += [(f"/mensen/{person['username']}", None) for person in public_people()[:PER_TABLE_CAP]]
urls += [(f"/docs/{page['slug']}", None) for page in DOCS_PAGES if not page.get("admin")]
parts = ['<?xml version="1.0" encoding="UTF-8"?>', '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">']
for path, modified in urls:
entry = f"<url><loc>{escape(base + path)}</loc>"
if modified:
entry += f"<lastmod>{escape(modified[:10])}</lastmod>"
parts.append(entry + "</url>")
parts.append("</urlset>")
xml = "\n".join(parts)
SafeET.fromstring(xml.encode("utf-8"))
return xml
@router.get("/sitemap.xml")
async def sitemap(request: Request):
base = base_url(request)
xml = _sitemap_cache.get(base)
if xml is None:
xml = build_sitemap(base)
_sitemap_cache.set(base, xml)
return Response(xml, media_type="application/xml")
molodetz/routers/standaard.py
+11
View File
@@ -0,0 +1,11 @@
# retoor <retoor@molodetz.nl>
from fastapi import APIRouter, Request
from molodetz.routers._listing import topic_listing
router = APIRouter()
@router.get("")
async def standaard(request: Request):
return topic_listing(request, "standaard")
molodetz/routers/voorwaarden.py
+30
View File
@@ -0,0 +1,30 @@
# retoor <retoor@molodetz.nl>
from fastapi import APIRouter, Request
from molodetz.database import get_setting, update_user
from molodetz.responses import action_result, respond
from molodetz.schemas.account import PageOut
from molodetz.seo import base_seo_context
from molodetz.utils.audit import record
from molodetz.utils.auth import invalidate_user_cache, require_user
from molodetz.utils.request_meta import safe_next
router = APIRouter()
@router.get("")
async def terms(request: Request):
seo = base_seo_context(request, "Voorwaarden", breadcrumbs=[("Molodetz", "/"), ("Voorwaarden", "/voorwaarden")])
ctx = {"title": "Voorwaarden", "terms_version": get_setting("terms_version", "1"), "next": safe_next(request.query_params.get("next"), "/"), "seo": seo}
return respond(request, "voorwaarden.html", ctx, model=PageOut)
@router.post("/accept")
async def accept_terms(request: Request):
user = require_user(request)
version = get_setting("terms_version", "1")
update_user(user["uid"], terms_version=version)
invalidate_user_cache(propagate=True)
record(request, "auth.terms.accept", payload={"version": version}, user=user)
form = await request.form() if not request.headers.get("content-type", "").startswith("application/json") else {}
return action_result(request, safe_next(form.get("next") if form else None, "/"))
molodetz/schemas/__init__.py
+1
View File
@@ -0,0 +1 @@
# retoor <retoor@molodetz.nl>

Some files were not shown because too many files have changed in this diff Show More