Render markdown in gist and project sidebar titles; add content visibility and GIF picker
DevPlace CI / test (push) Failing after 1h5m29s
DevPlace CI / test (push) Failing after 1h5m29s
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -246,3 +246,22 @@ Operational settings - read sites and rules:
|
||||
**Booleans are `<select>`, never checkboxes.** The settings save handler (`admin.py`) skips empty form values so empty fields don't clobber existing rows. An unchecked checkbox submits nothing, so it could never be turned off - `registration_open` and `maintenance_mode` use `<option value="1">`/`<option value="0">` so a value is always submitted.
|
||||
|
||||
|
||||
|
||||
## Content visibility (`database/visibility.py`, the single specification)
|
||||
|
||||
Posts, gists, projects and news carry one status from `VISIBILITY_VALUES`: `published` (default), `hidden`, `draft`, `offline`. The module is the specification; routers, forms, schemas, templates, Devii and the docs import it and never repeat a literal.
|
||||
|
||||
| Status | Listed (feed, search, topics, profiles, sitemap, counters) | Opens by link | Who |
|
||||
|---|---|---|---|
|
||||
| `published` | yes | yes | everyone |
|
||||
| `hidden` | no | yes, `noindex,nofollow` | everyone with the link |
|
||||
| `offline` | no | no (404) | owner and administrators |
|
||||
| `draft` | no | no (404) | owner and administrators; accepted by every API but not offered by the UI (`VISIBILITY_CHOICES` is the three user-facing values) |
|
||||
|
||||
- **Column.** `VISIBILITY_COLUMNS = {posts, gists, projects: "visibility", news: "status"}`. News reuses its existing `status` column (it already held `published`/`draft`), so no news migration exists. `ensure_visibility_columns()` (called from `init_db`) adds the text column and `idx_<table>_visibility` ONLY to tables that already exist; it must never create the table (an empty `posts` table breaks every `"posts" in db.tables` guard and the first-signup schema). A NULL or unknown value reads as `published` everywhere (`normalize_visibility`, `item_visibility`), so legacy rows needed no backfill and `init_db` does no per-row work.
|
||||
- **Every read goes through one of three choke points, never an inline compare.** `paginate`/`paginate_diverse` auto-append `listed_clause(table)` for any table in `VISIBILITY_COLUMNS` (`include_unlisted_of=<uid>` ORs in that owner's own rows, used for an owner's own profile/listing); `listed_clause(table, own_uid)` for `count()`/`find()` call sites; `listed_sql(table, alias)` for raw SQL (it returns `1 = 1` when the table or column does not exist yet, so a fresh database never errors). The predicates `is_listed`/`is_open` work on a row dict. Single-item access is `content.can_view_item(table, item, user)` (open, or owner, or admin) and `content.can_view_target(target_type, uid, user)` for polymorphic targets; `load_detail` applies the first, so every detail page 404s without per-route code. `can_view_project` composes visibility first, then the `is_private` rule.
|
||||
- **Writes.** `content.apply_visibility` is the one writer (column update, cache invalidation, `<kind>.visibility.set` audit with old/new value). `edit_content_item` pops the optional `visibility` form field and calls `apply_visibility` when it changed; `content.set_visibility_item` exists only for the administrator news route (news has no edit form). `create_content_item` stamps the column (`_with_visibility`) and skips mention notifications for non-open items.
|
||||
- **Public counters** (`get_user_post_count`, `get_site_stats`, `get_post_counts_by_user_uids`, trending topics, topic hub counts, sitemap author counts, happy-404 pool) count listed rows only, so the owner's own profile count excludes their hidden and offline posts.
|
||||
- **Backwards compatibility (hard rule).** `visibility` is optional on every API: `VisibilityCreateMixin` defaults to `published`, `VisibilityEditMixin` defaults to `None` meaning keep the current value (never reset to published), NULL rows read as published, and the docs/Devii parameter descriptions say so (`tests/unit/docs_api/visibility.py`, `tests/api/content/visibility.py` legacy-client tests). Never make the field required.
|
||||
- **Outputs.** `VisibleOut` (`schemas/content.py`) gives `PostOut`/`GistOut`/`ProjectOut` a `visibility` that reads NULL as `published`. Forms use `VisibilityCreateMixin` (default `published`) and `VisibilityEditMixin` (`None` keeps the current value); `VisibilityForm{value}` backs the admin news route.
|
||||
- **Adding a table.** Put it in `VISIBILITY_COLUMNS` and `TARGET_TABLES`, add its form mixin and Out base, apply `listed_clause`/`listed_sql` to each new read, and extend `tests/api/content/visibility.py`. `tests/unit/database/visibility.py` asserts the spec is self-consistent.
|
||||
|
||||
@@ -7,6 +7,7 @@ from .settings import _settings_cache, get_setting, get_int_setting, set_setting
|
||||
from .users import get_users_by_uids, _admins_cache, invalidate_admins_cache, get_admin_uids, set_user_timezone, set_last_seen, get_online_users, get_primary_admin_uid, is_account_active, search_users_by_username
|
||||
from .relations import _relations_cache, get_user_relations, get_blocked_uids, get_muted_uids, get_silenced_uids, invalidate_user_relations
|
||||
from .pagination import PAGE_SIZE, paginate, interleave_by_author, paginate_diverse, get_user_post_count, clear_user_post_count, build_pagination
|
||||
from .visibility import PUBLISHED, HIDDEN, DRAFT, OFFLINE, VISIBILITY_VALUES, VISIBILITY_CHOICES, DEFAULT_VISIBILITY, LISTED_VALUES, OPEN_VALUES, VISIBILITY_COLUMNS, TARGET_TABLES, VISIBILITY_LABELS, VISIBILITY_HINTS, normalize_visibility, visibility_column, item_visibility, is_listed, is_open, listed_clause, listed_sql, ensure_visibility_columns, visibility_options, visibility_label
|
||||
from .soft_delete import SOFT_DELETE_TABLES, ensure_soft_delete_columns, soft_delete, soft_delete_in, restore, purge, list_deleted, count_deleted, restore_event, purge_event
|
||||
from .engagement import _comment_count_cache, get_comment_counts_by_post_uids, get_post_counts_by_user_uids, get_vote_counts, get_user_votes, get_reactions_by_targets, get_user_bookmarks, get_user_notes, get_polls_by_post_uids, get_poll_for_post
|
||||
from .usage import _add_usage, _get_usage, add_correction_usage, get_correction_usage, add_modifier_usage, get_modifier_usage, NEWS_USAGE_KEY, add_news_usage, get_news_usage, ISSUE_USAGE_KEY, add_issue_usage, get_issue_usage, SEO_USAGE_KEY, add_seo_usage, get_seo_usage, AWARD_USAGE_KEY, add_award_usage, get_award_usage
|
||||
@@ -143,6 +144,29 @@ __all__ = [
|
||||
"clear_user_post_count",
|
||||
"build_pagination",
|
||||
"SOFT_DELETE_TABLES",
|
||||
"PUBLISHED",
|
||||
"HIDDEN",
|
||||
"DRAFT",
|
||||
"OFFLINE",
|
||||
"VISIBILITY_VALUES",
|
||||
"VISIBILITY_CHOICES",
|
||||
"DEFAULT_VISIBILITY",
|
||||
"LISTED_VALUES",
|
||||
"OPEN_VALUES",
|
||||
"VISIBILITY_COLUMNS",
|
||||
"TARGET_TABLES",
|
||||
"VISIBILITY_LABELS",
|
||||
"VISIBILITY_HINTS",
|
||||
"normalize_visibility",
|
||||
"visibility_column",
|
||||
"item_visibility",
|
||||
"is_listed",
|
||||
"is_open",
|
||||
"listed_clause",
|
||||
"listed_sql",
|
||||
"ensure_visibility_columns",
|
||||
"visibility_options",
|
||||
"visibility_label",
|
||||
"ensure_soft_delete_columns",
|
||||
"soft_delete",
|
||||
"soft_delete_in",
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
# retoor <retoor@molodetz.nl>
|
||||
|
||||
from .core import TTLCache, datetime, db, timedelta, timezone
|
||||
from .visibility import listed_sql
|
||||
|
||||
|
||||
def record_activity(user_uid: str, action: str) -> int:
|
||||
@@ -79,7 +80,7 @@ def get_activity_calendar(user_uid: str) -> dict:
|
||||
if sources:
|
||||
cutoff = (datetime.now(timezone.utc) - timedelta(days=364)).date().isoformat()
|
||||
union = " UNION ALL ".join(
|
||||
f"SELECT created_at FROM {table} WHERE user_uid = :u AND deleted_at IS NULL"
|
||||
f"SELECT created_at FROM {table} WHERE user_uid = :u AND deleted_at IS NULL AND {listed_sql(table)}"
|
||||
for table in sources
|
||||
)
|
||||
rows = db.query(
|
||||
|
||||
@@ -7,6 +7,7 @@ from collections import Counter
|
||||
from devplacepy.cache import TTLCache
|
||||
|
||||
from .core import db, get_table, or_
|
||||
from .visibility import listed_sql
|
||||
|
||||
_daily_topic_cache = TTLCache(ttl=60, max_size=1)
|
||||
_trending_cache = TTLCache(ttl=15, max_size=1)
|
||||
@@ -284,6 +285,7 @@ def get_trending_topics(limit: int = 6) -> list[dict]:
|
||||
return []
|
||||
rows = db.query(
|
||||
"SELECT topic FROM posts WHERE deleted_at IS NULL "
|
||||
f"AND {listed_sql('posts')} "
|
||||
"AND topic IS NOT NULL AND topic != '' "
|
||||
"ORDER BY created_at DESC LIMIT 200"
|
||||
)
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
# retoor <retoor@molodetz.nl>
|
||||
|
||||
from .core import TTLCache, _in_clause, db, defaultdict
|
||||
from .visibility import listed_sql
|
||||
|
||||
|
||||
_comment_count_cache = TTLCache(ttl=15, max_size=10000)
|
||||
@@ -36,7 +37,7 @@ def get_post_counts_by_user_uids(user_uids):
|
||||
return {}
|
||||
placeholders, params = _in_clause(user_uids)
|
||||
rows = db.query(
|
||||
f"SELECT user_uid, COUNT(*) as c FROM posts WHERE user_uid IN ({placeholders}) AND deleted_at IS NULL GROUP BY user_uid",
|
||||
f"SELECT user_uid, COUNT(*) as c FROM posts WHERE user_uid IN ({placeholders}) AND deleted_at IS NULL AND {listed_sql('posts')} GROUP BY user_uid",
|
||||
**params,
|
||||
)
|
||||
return {r["user_uid"]: r["c"] for r in rows}
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
from devplacepy.cache import TTLCache
|
||||
from .core import db, get_table
|
||||
from .relations import get_blocked_uids
|
||||
from .visibility import listed_clause
|
||||
|
||||
|
||||
PAGE_SIZE = 25
|
||||
@@ -19,10 +20,14 @@ def paginate(
|
||||
cursor_field="created_at",
|
||||
viewer_uid=None,
|
||||
limit=PAGE_SIZE,
|
||||
include_unlisted_of=None,
|
||||
**filters,
|
||||
):
|
||||
order = order or ["-" + cursor_field]
|
||||
clauses = list(clauses)
|
||||
listed = listed_clause(table, own_uid=include_unlisted_of)
|
||||
if listed is not None:
|
||||
clauses.append(listed)
|
||||
if table.has_column("deleted_at") and "deleted_at" not in filters:
|
||||
clauses.append(table.table.columns.deleted_at.is_(None))
|
||||
if viewer_uid and table.has_column("user_uid"):
|
||||
@@ -66,6 +71,7 @@ def paginate_diverse(
|
||||
uid_key="user_uid",
|
||||
viewer_uid=None,
|
||||
limit=PAGE_SIZE,
|
||||
include_unlisted_of=None,
|
||||
**filters,
|
||||
):
|
||||
rows, next_cursor = paginate(
|
||||
@@ -76,6 +82,7 @@ def paginate_diverse(
|
||||
cursor_field=cursor_field,
|
||||
viewer_uid=viewer_uid,
|
||||
limit=limit,
|
||||
include_unlisted_of=include_unlisted_of,
|
||||
**filters,
|
||||
)
|
||||
return interleave_by_author(rows, uid_key=uid_key), next_cursor
|
||||
@@ -91,7 +98,9 @@ def get_user_post_count(user_uid: str) -> int:
|
||||
return cached
|
||||
if "posts" not in db.tables:
|
||||
return 0
|
||||
count = get_table("posts").count(user_uid=user_uid, deleted_at=None)
|
||||
listed = listed_clause(get_table("posts"))
|
||||
clauses = [listed] if listed is not None else []
|
||||
count = get_table("posts").count(*clauses, user_uid=user_uid, deleted_at=None)
|
||||
_user_post_count_cache.set(user_uid, count)
|
||||
return count
|
||||
|
||||
|
||||
@@ -4,6 +4,7 @@ from devplacepy.services.nodes.spec import SPEC as NODES_SPEC
|
||||
from .core import AQUALITY_NEWS_GRADING_MODEL, AQUALITY_NEWS_GRADING_URL, DEFAULT_CORRECTION_PROMPT, DEFAULT_MODIFIER_PROMPT, INTERNAL_GATEWAY_URL, NEWS_GRADING_MODEL_DEFAULT, _drop_index, _index, _uid_index, db, defaultdict, get_table, logger
|
||||
from .settings import get_setting, set_setting
|
||||
from .soft_delete import SOFT_DELETE_TABLES, ensure_soft_delete_columns
|
||||
from .visibility import ensure_visibility_columns
|
||||
from .ranking import _authors_cache
|
||||
|
||||
|
||||
@@ -181,6 +182,8 @@ def init_db():
|
||||
("created_at", ""),
|
||||
("updated_at", ""),
|
||||
("client_id", ""),
|
||||
("deleted_at", ""),
|
||||
("deleted_by", ""),
|
||||
):
|
||||
if not messages.has_column(column):
|
||||
messages.create_column_by_example(column, example)
|
||||
@@ -1899,6 +1902,8 @@ def init_db():
|
||||
for table in SOFT_DELETE_TABLES:
|
||||
ensure_soft_delete_columns(table)
|
||||
|
||||
ensure_visibility_columns()
|
||||
|
||||
from .sync import ensure_export_indexes
|
||||
|
||||
ensure_export_indexes()
|
||||
|
||||
@@ -17,6 +17,7 @@ SOFT_DELETE_TABLES = [
|
||||
"reactions",
|
||||
"bookmarks",
|
||||
"notes",
|
||||
"messages",
|
||||
"skills",
|
||||
"communities",
|
||||
"node_devices",
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
|
||||
from .core import TTLCache, datetime, db, timedelta, timezone
|
||||
from .ranking import get_top_authors
|
||||
from .visibility import listed_clause
|
||||
|
||||
|
||||
_stats_cache = TTLCache(ttl=30, max_size=50)
|
||||
@@ -16,19 +17,20 @@ def get_site_stats() -> dict:
|
||||
.replace(hour=0, minute=0, second=0, microsecond=0)
|
||||
.isoformat()
|
||||
)
|
||||
def listed_count(name, **filters):
|
||||
if name not in db.tables:
|
||||
return 0
|
||||
listed = listed_clause(db[name])
|
||||
clauses = [listed] if listed is not None else []
|
||||
return db[name].count(*clauses, **filters)
|
||||
|
||||
stats = {
|
||||
"total_members": db["users"].count() if "users" in db.tables else 0,
|
||||
"posts_today": db["posts"].count(
|
||||
created_at={">=": today_start}, deleted_at=None
|
||||
)
|
||||
if "posts" in db.tables
|
||||
else 0,
|
||||
"total_projects": db["projects"].count(deleted_at=None)
|
||||
if "projects" in db.tables
|
||||
else 0,
|
||||
"total_gists": db["gists"].count(deleted_at=None)
|
||||
if "gists" in db.tables
|
||||
else 0,
|
||||
"posts_today": listed_count(
|
||||
"posts", created_at={">=": today_start}, deleted_at=None
|
||||
),
|
||||
"total_projects": listed_count("projects", deleted_at=None),
|
||||
"total_gists": listed_count("gists", deleted_at=None),
|
||||
}
|
||||
_stats_cache.set("site", stats)
|
||||
return stats
|
||||
|
||||
@@ -0,0 +1,112 @@
|
||||
# retoor <retoor@molodetz.nl>
|
||||
|
||||
from sqlalchemy import or_
|
||||
from .core import db, get_table, _index
|
||||
|
||||
PUBLISHED = "published"
|
||||
HIDDEN = "hidden"
|
||||
DRAFT = "draft"
|
||||
OFFLINE = "offline"
|
||||
|
||||
VISIBILITY_VALUES = (PUBLISHED, HIDDEN, DRAFT, OFFLINE)
|
||||
VISIBILITY_CHOICES = (PUBLISHED, HIDDEN, OFFLINE)
|
||||
DEFAULT_VISIBILITY = PUBLISHED
|
||||
LISTED_VALUES = (PUBLISHED,)
|
||||
OPEN_VALUES = (PUBLISHED, HIDDEN)
|
||||
|
||||
VISIBILITY_COLUMNS = {
|
||||
"posts": "visibility",
|
||||
"gists": "visibility",
|
||||
"projects": "visibility",
|
||||
"news": "status",
|
||||
}
|
||||
|
||||
TARGET_TABLES = {
|
||||
"post": "posts",
|
||||
"gist": "gists",
|
||||
"project": "projects",
|
||||
"news": "news",
|
||||
}
|
||||
|
||||
VISIBILITY_LABELS = {
|
||||
PUBLISHED: "Published",
|
||||
HIDDEN: "Hidden",
|
||||
DRAFT: "Draft",
|
||||
OFFLINE: "Offline",
|
||||
}
|
||||
|
||||
VISIBILITY_HINTS = {
|
||||
PUBLISHED: "Listed everywhere and open to everyone.",
|
||||
HIDDEN: "Not listed or searchable; anyone with the link can still open it.",
|
||||
DRAFT: "Only you can see it. Not published yet.",
|
||||
OFFLINE: "Taken offline for now. Only you can see it; publish it again at any time.",
|
||||
}
|
||||
|
||||
def normalize_visibility(value, fallback: str = DEFAULT_VISIBILITY) -> str:
|
||||
text = str(value or "").strip().lower()
|
||||
return text if text in VISIBILITY_VALUES else fallback
|
||||
|
||||
|
||||
def visibility_column(table_name: str) -> str | None:
|
||||
return VISIBILITY_COLUMNS.get(table_name)
|
||||
|
||||
|
||||
def item_visibility(table_name: str, item: dict | None) -> str:
|
||||
column = visibility_column(table_name)
|
||||
if not column or not item:
|
||||
return DEFAULT_VISIBILITY
|
||||
return normalize_visibility(item.get(column))
|
||||
|
||||
|
||||
def is_listed(table_name: str, item: dict | None) -> bool:
|
||||
return item_visibility(table_name, item) in LISTED_VALUES
|
||||
|
||||
|
||||
def is_open(table_name: str, item: dict | None) -> bool:
|
||||
return item_visibility(table_name, item) in OPEN_VALUES
|
||||
|
||||
|
||||
def listed_clause(table, own_uid: str | None = None):
|
||||
name = getattr(table, "name", None) or getattr(table.table, "name", None)
|
||||
column_name = visibility_column(name)
|
||||
if not column_name or not table.exists:
|
||||
return None
|
||||
columns = table.table.columns
|
||||
if column_name not in columns:
|
||||
return None
|
||||
column = columns[column_name]
|
||||
clause = or_(column.is_(None), column.in_(LISTED_VALUES))
|
||||
if own_uid and "user_uid" in columns:
|
||||
clause = or_(clause, columns.user_uid == own_uid)
|
||||
return clause
|
||||
|
||||
|
||||
def listed_sql(table_name: str, alias: str = "") -> str:
|
||||
column = visibility_column(table_name)
|
||||
if not column or table_name not in db.tables:
|
||||
return "1 = 1"
|
||||
if not get_table(table_name).has_column(column):
|
||||
return "1 = 1"
|
||||
prefix = f"{alias}." if alias else ""
|
||||
return f"COALESCE({prefix}{column}, '{DEFAULT_VISIBILITY}') = '{PUBLISHED}'"
|
||||
|
||||
|
||||
def ensure_visibility_columns() -> None:
|
||||
for table_name, column in VISIBILITY_COLUMNS.items():
|
||||
if table_name == "news" or table_name not in db.tables:
|
||||
continue
|
||||
table = get_table(table_name)
|
||||
if not table.has_column(column):
|
||||
table.create_column_by_example(column, "")
|
||||
_index(db, table_name, f"idx_{table_name}_visibility", [column])
|
||||
|
||||
|
||||
def visibility_options(values=VISIBILITY_CHOICES) -> list[dict]:
|
||||
return [
|
||||
{"value": v, "label": VISIBILITY_LABELS[v], "hint": VISIBILITY_HINTS[v]}
|
||||
for v in values
|
||||
]
|
||||
|
||||
|
||||
def visibility_label(value) -> str:
|
||||
return VISIBILITY_LABELS[normalize_visibility(value)]
|
||||
Reference in New Issue
Block a user