Render markdown in gist and project sidebar titles; add content visibility and GIF picker
DevPlace CI / test (push) Failing after 1h5m29s

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-05 17:59:21 +01:00
co-authored by Claude Sonnet 5.5
parent bc11475f29
commit 05d6633204
154 changed files with 6394 additions and 289 deletions
+19
View File
@@ -246,3 +246,22 @@ Operational settings - read sites and rules:
**Booleans are `<select>`, never checkboxes.** The settings save handler (`admin.py`) skips empty form values so empty fields don't clobber existing rows. An unchecked checkbox submits nothing, so it could never be turned off - `registration_open` and `maintenance_mode` use `<option value="1">`/`<option value="0">` so a value is always submitted.
## Content visibility (`database/visibility.py`, the single specification)
Posts, gists, projects and news carry one status from `VISIBILITY_VALUES`: `published` (default), `hidden`, `draft`, `offline`. The module is the specification; routers, forms, schemas, templates, Devii and the docs import it and never repeat a literal.
| Status | Listed (feed, search, topics, profiles, sitemap, counters) | Opens by link | Who |
|---|---|---|---|
| `published` | yes | yes | everyone |
| `hidden` | no | yes, `noindex,nofollow` | everyone with the link |
| `offline` | no | no (404) | owner and administrators |
| `draft` | no | no (404) | owner and administrators; accepted by every API but not offered by the UI (`VISIBILITY_CHOICES` is the three user-facing values) |
- **Column.** `VISIBILITY_COLUMNS = {posts, gists, projects: "visibility", news: "status"}`. News reuses its existing `status` column (it already held `published`/`draft`), so no news migration exists. `ensure_visibility_columns()` (called from `init_db`) adds the text column and `idx_<table>_visibility` ONLY to tables that already exist; it must never create the table (an empty `posts` table breaks every `"posts" in db.tables` guard and the first-signup schema). A NULL or unknown value reads as `published` everywhere (`normalize_visibility`, `item_visibility`), so legacy rows needed no backfill and `init_db` does no per-row work.
- **Every read goes through one of three choke points, never an inline compare.** `paginate`/`paginate_diverse` auto-append `listed_clause(table)` for any table in `VISIBILITY_COLUMNS` (`include_unlisted_of=<uid>` ORs in that owner's own rows, used for an owner's own profile/listing); `listed_clause(table, own_uid)` for `count()`/`find()` call sites; `listed_sql(table, alias)` for raw SQL (it returns `1 = 1` when the table or column does not exist yet, so a fresh database never errors). The predicates `is_listed`/`is_open` work on a row dict. Single-item access is `content.can_view_item(table, item, user)` (open, or owner, or admin) and `content.can_view_target(target_type, uid, user)` for polymorphic targets; `load_detail` applies the first, so every detail page 404s without per-route code. `can_view_project` composes visibility first, then the `is_private` rule.
- **Writes.** `content.apply_visibility` is the one writer (column update, cache invalidation, `<kind>.visibility.set` audit with old/new value). `edit_content_item` pops the optional `visibility` form field and calls `apply_visibility` when it changed; `content.set_visibility_item` exists only for the administrator news route (news has no edit form). `create_content_item` stamps the column (`_with_visibility`) and skips mention notifications for non-open items.
- **Public counters** (`get_user_post_count`, `get_site_stats`, `get_post_counts_by_user_uids`, trending topics, topic hub counts, sitemap author counts, happy-404 pool) count listed rows only, so the owner's own profile count excludes their hidden and offline posts.
- **Backwards compatibility (hard rule).** `visibility` is optional on every API: `VisibilityCreateMixin` defaults to `published`, `VisibilityEditMixin` defaults to `None` meaning keep the current value (never reset to published), NULL rows read as published, and the docs/Devii parameter descriptions say so (`tests/unit/docs_api/visibility.py`, `tests/api/content/visibility.py` legacy-client tests). Never make the field required.
- **Outputs.** `VisibleOut` (`schemas/content.py`) gives `PostOut`/`GistOut`/`ProjectOut` a `visibility` that reads NULL as `published`. Forms use `VisibilityCreateMixin` (default `published`) and `VisibilityEditMixin` (`None` keeps the current value); `VisibilityForm{value}` backs the admin news route.
- **Adding a table.** Put it in `VISIBILITY_COLUMNS` and `TARGET_TABLES`, add its form mixin and Out base, apply `listed_clause`/`listed_sql` to each new read, and extend `tests/api/content/visibility.py`. `tests/unit/database/visibility.py` asserts the spec is self-consistent.
+24
View File
@@ -7,6 +7,7 @@ from .settings import _settings_cache, get_setting, get_int_setting, set_setting
from .users import get_users_by_uids, _admins_cache, invalidate_admins_cache, get_admin_uids, set_user_timezone, set_last_seen, get_online_users, get_primary_admin_uid, is_account_active, search_users_by_username
from .relations import _relations_cache, get_user_relations, get_blocked_uids, get_muted_uids, get_silenced_uids, invalidate_user_relations
from .pagination import PAGE_SIZE, paginate, interleave_by_author, paginate_diverse, get_user_post_count, clear_user_post_count, build_pagination
from .visibility import PUBLISHED, HIDDEN, DRAFT, OFFLINE, VISIBILITY_VALUES, VISIBILITY_CHOICES, DEFAULT_VISIBILITY, LISTED_VALUES, OPEN_VALUES, VISIBILITY_COLUMNS, TARGET_TABLES, VISIBILITY_LABELS, VISIBILITY_HINTS, normalize_visibility, visibility_column, item_visibility, is_listed, is_open, listed_clause, listed_sql, ensure_visibility_columns, visibility_options, visibility_label
from .soft_delete import SOFT_DELETE_TABLES, ensure_soft_delete_columns, soft_delete, soft_delete_in, restore, purge, list_deleted, count_deleted, restore_event, purge_event
from .engagement import _comment_count_cache, get_comment_counts_by_post_uids, get_post_counts_by_user_uids, get_vote_counts, get_user_votes, get_reactions_by_targets, get_user_bookmarks, get_user_notes, get_polls_by_post_uids, get_poll_for_post
from .usage import _add_usage, _get_usage, add_correction_usage, get_correction_usage, add_modifier_usage, get_modifier_usage, NEWS_USAGE_KEY, add_news_usage, get_news_usage, ISSUE_USAGE_KEY, add_issue_usage, get_issue_usage, SEO_USAGE_KEY, add_seo_usage, get_seo_usage, AWARD_USAGE_KEY, add_award_usage, get_award_usage
@@ -143,6 +144,29 @@ __all__ = [
"clear_user_post_count",
"build_pagination",
"SOFT_DELETE_TABLES",
"PUBLISHED",
"HIDDEN",
"DRAFT",
"OFFLINE",
"VISIBILITY_VALUES",
"VISIBILITY_CHOICES",
"DEFAULT_VISIBILITY",
"LISTED_VALUES",
"OPEN_VALUES",
"VISIBILITY_COLUMNS",
"TARGET_TABLES",
"VISIBILITY_LABELS",
"VISIBILITY_HINTS",
"normalize_visibility",
"visibility_column",
"item_visibility",
"is_listed",
"is_open",
"listed_clause",
"listed_sql",
"ensure_visibility_columns",
"visibility_options",
"visibility_label",
"ensure_soft_delete_columns",
"soft_delete",
"soft_delete_in",
+2 -1
View File
@@ -1,6 +1,7 @@
# retoor <retoor@molodetz.nl>
from .core import TTLCache, datetime, db, timedelta, timezone
from .visibility import listed_sql
def record_activity(user_uid: str, action: str) -> int:
@@ -79,7 +80,7 @@ def get_activity_calendar(user_uid: str) -> dict:
if sources:
cutoff = (datetime.now(timezone.utc) - timedelta(days=364)).date().isoformat()
union = " UNION ALL ".join(
f"SELECT created_at FROM {table} WHERE user_uid = :u AND deleted_at IS NULL"
f"SELECT created_at FROM {table} WHERE user_uid = :u AND deleted_at IS NULL AND {listed_sql(table)}"
for table in sources
)
rows = db.query(
+2
View File
@@ -7,6 +7,7 @@ from collections import Counter
from devplacepy.cache import TTLCache
from .core import db, get_table, or_
from .visibility import listed_sql
_daily_topic_cache = TTLCache(ttl=60, max_size=1)
_trending_cache = TTLCache(ttl=15, max_size=1)
@@ -284,6 +285,7 @@ def get_trending_topics(limit: int = 6) -> list[dict]:
return []
rows = db.query(
"SELECT topic FROM posts WHERE deleted_at IS NULL "
f"AND {listed_sql('posts')} "
"AND topic IS NOT NULL AND topic != '' "
"ORDER BY created_at DESC LIMIT 200"
)
+2 -1
View File
@@ -1,6 +1,7 @@
# retoor <retoor@molodetz.nl>
from .core import TTLCache, _in_clause, db, defaultdict
from .visibility import listed_sql
_comment_count_cache = TTLCache(ttl=15, max_size=10000)
@@ -36,7 +37,7 @@ def get_post_counts_by_user_uids(user_uids):
return {}
placeholders, params = _in_clause(user_uids)
rows = db.query(
f"SELECT user_uid, COUNT(*) as c FROM posts WHERE user_uid IN ({placeholders}) AND deleted_at IS NULL GROUP BY user_uid",
f"SELECT user_uid, COUNT(*) as c FROM posts WHERE user_uid IN ({placeholders}) AND deleted_at IS NULL AND {listed_sql('posts')} GROUP BY user_uid",
**params,
)
return {r["user_uid"]: r["c"] for r in rows}
+10 -1
View File
@@ -3,6 +3,7 @@
from devplacepy.cache import TTLCache
from .core import db, get_table
from .relations import get_blocked_uids
from .visibility import listed_clause
PAGE_SIZE = 25
@@ -19,10 +20,14 @@ def paginate(
cursor_field="created_at",
viewer_uid=None,
limit=PAGE_SIZE,
include_unlisted_of=None,
**filters,
):
order = order or ["-" + cursor_field]
clauses = list(clauses)
listed = listed_clause(table, own_uid=include_unlisted_of)
if listed is not None:
clauses.append(listed)
if table.has_column("deleted_at") and "deleted_at" not in filters:
clauses.append(table.table.columns.deleted_at.is_(None))
if viewer_uid and table.has_column("user_uid"):
@@ -66,6 +71,7 @@ def paginate_diverse(
uid_key="user_uid",
viewer_uid=None,
limit=PAGE_SIZE,
include_unlisted_of=None,
**filters,
):
rows, next_cursor = paginate(
@@ -76,6 +82,7 @@ def paginate_diverse(
cursor_field=cursor_field,
viewer_uid=viewer_uid,
limit=limit,
include_unlisted_of=include_unlisted_of,
**filters,
)
return interleave_by_author(rows, uid_key=uid_key), next_cursor
@@ -91,7 +98,9 @@ def get_user_post_count(user_uid: str) -> int:
return cached
if "posts" not in db.tables:
return 0
count = get_table("posts").count(user_uid=user_uid, deleted_at=None)
listed = listed_clause(get_table("posts"))
clauses = [listed] if listed is not None else []
count = get_table("posts").count(*clauses, user_uid=user_uid, deleted_at=None)
_user_post_count_cache.set(user_uid, count)
return count
+5
View File
@@ -4,6 +4,7 @@ from devplacepy.services.nodes.spec import SPEC as NODES_SPEC
from .core import AQUALITY_NEWS_GRADING_MODEL, AQUALITY_NEWS_GRADING_URL, DEFAULT_CORRECTION_PROMPT, DEFAULT_MODIFIER_PROMPT, INTERNAL_GATEWAY_URL, NEWS_GRADING_MODEL_DEFAULT, _drop_index, _index, _uid_index, db, defaultdict, get_table, logger
from .settings import get_setting, set_setting
from .soft_delete import SOFT_DELETE_TABLES, ensure_soft_delete_columns
from .visibility import ensure_visibility_columns
from .ranking import _authors_cache
@@ -181,6 +182,8 @@ def init_db():
("created_at", ""),
("updated_at", ""),
("client_id", ""),
("deleted_at", ""),
("deleted_by", ""),
):
if not messages.has_column(column):
messages.create_column_by_example(column, example)
@@ -1899,6 +1902,8 @@ def init_db():
for table in SOFT_DELETE_TABLES:
ensure_soft_delete_columns(table)
ensure_visibility_columns()
from .sync import ensure_export_indexes
ensure_export_indexes()
+1
View File
@@ -17,6 +17,7 @@ SOFT_DELETE_TABLES = [
"reactions",
"bookmarks",
"notes",
"messages",
"skills",
"communities",
"node_devices",
+13 -11
View File
@@ -2,6 +2,7 @@
from .core import TTLCache, datetime, db, timedelta, timezone
from .ranking import get_top_authors
from .visibility import listed_clause
_stats_cache = TTLCache(ttl=30, max_size=50)
@@ -16,19 +17,20 @@ def get_site_stats() -> dict:
.replace(hour=0, minute=0, second=0, microsecond=0)
.isoformat()
)
def listed_count(name, **filters):
if name not in db.tables:
return 0
listed = listed_clause(db[name])
clauses = [listed] if listed is not None else []
return db[name].count(*clauses, **filters)
stats = {
"total_members": db["users"].count() if "users" in db.tables else 0,
"posts_today": db["posts"].count(
created_at={">=": today_start}, deleted_at=None
)
if "posts" in db.tables
else 0,
"total_projects": db["projects"].count(deleted_at=None)
if "projects" in db.tables
else 0,
"total_gists": db["gists"].count(deleted_at=None)
if "gists" in db.tables
else 0,
"posts_today": listed_count(
"posts", created_at={">=": today_start}, deleted_at=None
),
"total_projects": listed_count("projects", deleted_at=None),
"total_gists": listed_count("gists", deleted_at=None),
}
_stats_cache.set("site", stats)
return stats
+112
View File
@@ -0,0 +1,112 @@
# retoor <retoor@molodetz.nl>
from sqlalchemy import or_
from .core import db, get_table, _index
PUBLISHED = "published"
HIDDEN = "hidden"
DRAFT = "draft"
OFFLINE = "offline"
VISIBILITY_VALUES = (PUBLISHED, HIDDEN, DRAFT, OFFLINE)
VISIBILITY_CHOICES = (PUBLISHED, HIDDEN, OFFLINE)
DEFAULT_VISIBILITY = PUBLISHED
LISTED_VALUES = (PUBLISHED,)
OPEN_VALUES = (PUBLISHED, HIDDEN)
VISIBILITY_COLUMNS = {
"posts": "visibility",
"gists": "visibility",
"projects": "visibility",
"news": "status",
}
TARGET_TABLES = {
"post": "posts",
"gist": "gists",
"project": "projects",
"news": "news",
}
VISIBILITY_LABELS = {
PUBLISHED: "Published",
HIDDEN: "Hidden",
DRAFT: "Draft",
OFFLINE: "Offline",
}
VISIBILITY_HINTS = {
PUBLISHED: "Listed everywhere and open to everyone.",
HIDDEN: "Not listed or searchable; anyone with the link can still open it.",
DRAFT: "Only you can see it. Not published yet.",
OFFLINE: "Taken offline for now. Only you can see it; publish it again at any time.",
}
def normalize_visibility(value, fallback: str = DEFAULT_VISIBILITY) -> str:
text = str(value or "").strip().lower()
return text if text in VISIBILITY_VALUES else fallback
def visibility_column(table_name: str) -> str | None:
return VISIBILITY_COLUMNS.get(table_name)
def item_visibility(table_name: str, item: dict | None) -> str:
column = visibility_column(table_name)
if not column or not item:
return DEFAULT_VISIBILITY
return normalize_visibility(item.get(column))
def is_listed(table_name: str, item: dict | None) -> bool:
return item_visibility(table_name, item) in LISTED_VALUES
def is_open(table_name: str, item: dict | None) -> bool:
return item_visibility(table_name, item) in OPEN_VALUES
def listed_clause(table, own_uid: str | None = None):
name = getattr(table, "name", None) or getattr(table.table, "name", None)
column_name = visibility_column(name)
if not column_name or not table.exists:
return None
columns = table.table.columns
if column_name not in columns:
return None
column = columns[column_name]
clause = or_(column.is_(None), column.in_(LISTED_VALUES))
if own_uid and "user_uid" in columns:
clause = or_(clause, columns.user_uid == own_uid)
return clause
def listed_sql(table_name: str, alias: str = "") -> str:
column = visibility_column(table_name)
if not column or table_name not in db.tables:
return "1 = 1"
if not get_table(table_name).has_column(column):
return "1 = 1"
prefix = f"{alias}." if alias else ""
return f"COALESCE({prefix}{column}, '{DEFAULT_VISIBILITY}') = '{PUBLISHED}'"
def ensure_visibility_columns() -> None:
for table_name, column in VISIBILITY_COLUMNS.items():
if table_name == "news" or table_name not in db.tables:
continue
table = get_table(table_name)
if not table.has_column(column):
table.create_column_by_example(column, "")
_index(db, table_name, f"idx_{table_name}_visibility", [column])
def visibility_options(values=VISIBILITY_CHOICES) -> list[dict]:
return [
{"value": v, "label": VISIBILITY_LABELS[v], "hint": VISIBILITY_HINTS[v]}
for v in values
]
def visibility_label(value) -> str:
return VISIBILITY_LABELS[normalize_visibility(value)]