Add personal notes, DeepSearch history, backup offload, and gateway auth throttling
DevPlace CI / test (push) Failing after 28m20s
DevPlace CI / test (push) Failing after 28m20s
Also streamline the top navigation: drop the Tools dropdown, make Quizzes and Battles icon-only entries, and remove the Workspace, Containers and Editor entry points from the project detail page.
This commit is contained in:
+17
-2
@@ -19,6 +19,7 @@ from devplacepy.database import (
|
||||
get_user_votes,
|
||||
get_reactions_by_targets,
|
||||
get_user_bookmarks,
|
||||
get_user_notes,
|
||||
get_blocked_uids,
|
||||
get_poll_for_post,
|
||||
update_target_stars,
|
||||
@@ -70,6 +71,7 @@ CREATE_METADATA_KEYS = ("project_type", "is_private", "language", "topic", "stat
|
||||
|
||||
BOOKMARKABLE_TYPES = {"post", "gist", "project", "news", "quiz"}
|
||||
REACTABLE_TYPES = {"post", "comment", "gist", "project", "quiz"}
|
||||
NOTABLE_TYPES = {"post", "gist", "project", "news"}
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -325,10 +327,16 @@ def apply_vote(request, user: dict, target_type: str, target_uid: str, value: in
|
||||
existing = votes.find_one(
|
||||
user_uid=user["uid"], target_uid=target_uid, target_type=target_type
|
||||
)
|
||||
old_value = int(existing["value"]) if existing else 0
|
||||
old_value = int(existing["value"]) if existing and not existing.get("deleted_at") else 0
|
||||
did_upvote = False
|
||||
new_value = value
|
||||
if existing:
|
||||
if value == 0:
|
||||
if existing and not existing.get("deleted_at"):
|
||||
votes.update(
|
||||
{"id": existing["id"], "deleted_at": _now_iso(), "deleted_by": user["uid"]},
|
||||
["id"],
|
||||
)
|
||||
elif existing:
|
||||
if existing.get("deleted_at"):
|
||||
votes.update(
|
||||
{
|
||||
@@ -648,6 +656,7 @@ def detail_context(
|
||||
"attachments": detail["attachments"],
|
||||
"reactions": detail.get("reactions", {"counts": {}, "mine": []}),
|
||||
"bookmarked": detail.get("bookmarked", False),
|
||||
"note_content": detail.get("note_content"),
|
||||
"poll": detail.get("poll"),
|
||||
"war": detail.get("war"),
|
||||
"project_link": detail.get("project_link"),
|
||||
@@ -828,6 +837,11 @@ def load_detail(
|
||||
and target_type in BOOKMARKABLE_TYPES
|
||||
and item["uid"] in get_user_bookmarks(user["uid"], target_type, [item["uid"]])
|
||||
)
|
||||
note_content = (
|
||||
get_user_notes(user["uid"], target_type, [item["uid"]]).get(item["uid"])
|
||||
if user and target_type in NOTABLE_TYPES
|
||||
else None
|
||||
)
|
||||
return {
|
||||
"item": item,
|
||||
"author": author,
|
||||
@@ -841,6 +855,7 @@ def load_detail(
|
||||
"time_ago": time_ago(item["created_at"]),
|
||||
"reactions": reactions,
|
||||
"bookmarked": bookmarked,
|
||||
"note_content": note_content,
|
||||
"poll": get_poll_for_post(item["uid"], user) if target_type == "post" else None,
|
||||
"war": war_store.get_war_serialized_for_post(item["uid"], user)
|
||||
if target_type == "post"
|
||||
|
||||
@@ -124,7 +124,7 @@ Profile with the real database before and after any change here (`cProfile` arou
|
||||
|
||||
Attachments (see "Profile media gallery and soft-deleted attachments" below) are one instance of a platform-wide model: **every removal is a soft delete; only garbage collection is a hard delete.** Every removable row carries two columns, `deleted_at` (ISO timestamp) and `deleted_by` (actor uid, or `system`). A live row has `deleted_at = NULL`, and every list/count read filters `deleted_at IS NULL`.
|
||||
|
||||
- **Tables (`database.SOFT_DELETE_TABLES`):** posts, comments, gists, projects, news, news_images, project_files, attachments, votes, reactions, bookmarks, follows, poll_votes, polls, poll_options, sessions, instances, instance_schedules, devii_conversations, devii_tasks, devii_lessons, devii_virtual_tools, user_customizations, project_forks. `init_db` loops `ensure_soft_delete_columns(t)` for each (adds both columns + `idx_<t>_deleted` / the partial trash index described above). The Devii task/lesson/virtual-tool stores run on their own DB handles, so their `_ensure_indexes` adds the columns there too.
|
||||
- **Tables (`database.SOFT_DELETE_TABLES`):** posts, comments, gists, projects, news, news_images, project_files, attachments, votes, reactions, bookmarks, notes, follows, poll_votes, polls, poll_options, sessions, instances, instance_schedules, devii_conversations, devii_tasks, devii_lessons, devii_virtual_tools, user_customizations, project_forks. `init_db` loops `ensure_soft_delete_columns(t)` for each (adds both columns + `idx_<t>_deleted` / the partial trash index described above). The Devii task/lesson/virtual-tool stores run on their own DB handles, so their `_ensure_indexes` adds the columns there too.
|
||||
- **Every INSERT into a soft-deletable table MUST write `deleted_at: None, deleted_by: None`.** `dataset.find(deleted_at=None)` on a table that lacks the column matches NOTHING (a false predicate), silently hiding all rows - the born-live insert is what creates the column. Add the pair to any new insert.
|
||||
- **Central helpers (`database/`):** `soft_delete(table, deleted_by, *, stamp=None, **criteria)` (equality), `soft_delete_in(table, column, uids, deleted_by, *, stamp=None, **extra)` (IN-clause cascade), `restore(table, **criteria)`, `purge(table, **criteria)` (real delete), `list_deleted(table, page)` / `count_deleted(table)` (trash listings), and the event helpers `restore_event(stamp)` / `purge_event(stamp)` that act across ALL tables sharing one `deleted_at` stamp.
|
||||
- **Two generic chokepoints are conditionally filtered:** `resolve_by_slug(table, slug, include_deleted=False)` (detail-page lookups; restore passes `include_deleted=True`) and `paginate(table, ...)` (auto-appends `deleted_at IS NULL` when the table has the column and the caller did not pass `deleted_at`). `seo._collect` does the same for the sitemap. Read filters were threaded through every batch helper, analytics/activity/leaderboard UNION, feed/profile/listing route, and store; never re-introduce an unfiltered read of a soft-deletable table.
|
||||
|
||||
@@ -8,7 +8,7 @@ from .users import get_users_by_uids, _admins_cache, invalidate_admins_cache, ge
|
||||
from .relations import _relations_cache, get_user_relations, get_blocked_uids, get_muted_uids, get_silenced_uids, invalidate_user_relations
|
||||
from .pagination import PAGE_SIZE, paginate, interleave_by_author, paginate_diverse, get_user_post_count, clear_user_post_count, build_pagination
|
||||
from .soft_delete import SOFT_DELETE_TABLES, ensure_soft_delete_columns, soft_delete, soft_delete_in, restore, purge, list_deleted, count_deleted, restore_event, purge_event
|
||||
from .engagement import _comment_count_cache, get_comment_counts_by_post_uids, get_post_counts_by_user_uids, get_vote_counts, get_user_votes, get_reactions_by_targets, get_user_bookmarks, get_polls_by_post_uids, get_poll_for_post
|
||||
from .engagement import _comment_count_cache, get_comment_counts_by_post_uids, get_post_counts_by_user_uids, get_vote_counts, get_user_votes, get_reactions_by_targets, get_user_bookmarks, get_user_notes, get_polls_by_post_uids, get_poll_for_post
|
||||
from .usage import _add_usage, _get_usage, add_correction_usage, get_correction_usage, add_modifier_usage, get_modifier_usage, NEWS_USAGE_KEY, add_news_usage, get_news_usage, ISSUE_USAGE_KEY, add_issue_usage, get_issue_usage, SEO_USAGE_KEY, add_seo_usage, get_seo_usage, AWARD_USAGE_KEY, add_award_usage, get_award_usage
|
||||
from .awards import (
|
||||
AWARDS_PER_PAGE,
|
||||
@@ -34,7 +34,7 @@ from .email import EMAIL_ACCOUNT_DEFAULTS, list_email_accounts, get_email_accoun
|
||||
from .notifications import NOTIFICATION_TYPES, NOTIFICATION_CHANNELS, _NOTIFICATION_CHANNEL_COLUMNS, _NOTIFICATION_CHANNEL_DEFAULTS, _NOTIFICATION_TYPE_KEYS, _notification_prefs_cache, _notification_default, get_notification_default, set_notification_default, _notification_overrides, notification_enabled, get_notification_prefs, set_notification_pref, reset_notification_prefs, mark_notifications_read_by_target
|
||||
from .forks import record_fork, get_fork_parent, count_forks, soft_delete_fork_relations, delete_fork_relations
|
||||
from .follows import get_follow_counts, get_follow_list, get_following_among
|
||||
from .deepsearch import _ds_now, create_deepsearch_session, update_deepsearch_session, get_deepsearch_session, add_deepsearch_message, get_deepsearch_messages, get_cached_deepsearch_url, upsert_deepsearch_url_cache
|
||||
from .deepsearch import _ds_now, create_deepsearch_session, update_deepsearch_session, get_deepsearch_session, list_deepsearch_sessions, add_deepsearch_message, get_deepsearch_messages, get_cached_deepsearch_url, upsert_deepsearch_url_cache
|
||||
from .ranking import VOTABLE_TARGETS, STAR_TARGETS, _authors_cache, _ranked_authors, _rank_map, get_top_authors, get_leaderboard, get_user_rank, get_user_stars, clear_user_stars, update_target_stars, soft_delete_engagement, delete_engagement, get_target_owner_uid
|
||||
from .moderation import (
|
||||
ACTIONS_TABLE,
|
||||
@@ -157,6 +157,7 @@ __all__ = [
|
||||
"get_user_votes",
|
||||
"get_reactions_by_targets",
|
||||
"get_user_bookmarks",
|
||||
"get_user_notes",
|
||||
"get_polls_by_post_uids",
|
||||
"get_poll_for_post",
|
||||
"_add_usage",
|
||||
@@ -236,6 +237,7 @@ __all__ = [
|
||||
"create_deepsearch_session",
|
||||
"update_deepsearch_session",
|
||||
"get_deepsearch_session",
|
||||
"list_deepsearch_sessions",
|
||||
"add_deepsearch_message",
|
||||
"get_deepsearch_messages",
|
||||
"get_cached_deepsearch_url",
|
||||
|
||||
@@ -54,6 +54,20 @@ def get_deepsearch_session(uid: str) -> dict | None:
|
||||
return get_table("deepsearch_sessions").find_one(uid=uid, deleted_at=None)
|
||||
|
||||
|
||||
def list_deepsearch_sessions(owner_kind: str, owner_id: str, limit: int = 50) -> list[dict]:
|
||||
if "deepsearch_sessions" not in db.tables:
|
||||
return []
|
||||
return list(
|
||||
get_table("deepsearch_sessions").find(
|
||||
owner_kind=owner_kind,
|
||||
owner_id=owner_id,
|
||||
deleted_at=None,
|
||||
order_by=["-created_at"],
|
||||
_limit=limit,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def add_deepsearch_message(
|
||||
uid: str, session_uid: str, role: str, content: str, citations: str = ""
|
||||
) -> None:
|
||||
|
||||
@@ -116,6 +116,19 @@ def get_user_bookmarks(user_uid, target_type, target_uids):
|
||||
return {row["target_uid"] for row in rows}
|
||||
|
||||
|
||||
def get_user_notes(user_uid, target_type, target_uids):
|
||||
if not user_uid or not target_uids or "notes" not in db.tables:
|
||||
return {}
|
||||
placeholders, params = _in_clause(target_uids)
|
||||
params["u"] = user_uid
|
||||
params["tt"] = target_type
|
||||
rows = db.query(
|
||||
f"SELECT target_uid, content FROM notes WHERE user_uid=:u AND target_type=:tt AND target_uid IN ({placeholders}) AND deleted_at IS NULL",
|
||||
**params,
|
||||
)
|
||||
return {row["target_uid"]: row["content"] for row in rows}
|
||||
|
||||
|
||||
def get_polls_by_post_uids(post_uids, user=None):
|
||||
if not post_uids or "polls" not in db.tables:
|
||||
return {}
|
||||
|
||||
@@ -47,6 +47,7 @@ UNREPORTABLE_TABLES: dict[str, str] = {
|
||||
"votes": "engagement counters, carry no authored content",
|
||||
"reactions": "engagement counters, carry no authored content",
|
||||
"bookmarks": "private to the owner",
|
||||
"notes": "private to the owner",
|
||||
"follows": "relationship rows, carry no authored content",
|
||||
"poll_votes": "private ballots",
|
||||
"opinion_war_fighters": "membership and damage counters, carry no authored content",
|
||||
|
||||
@@ -24,6 +24,7 @@ NOTIFICATION_TYPES = [
|
||||
{"key": "workspace", "label": "Dev workspaces", "description": "Idle, quota, retention and moderation notices for your dev workspaces"},
|
||||
{"key": "moderation", "label": "Moderation", "description": "Acknowledgements for reports you file and decisions taken on your content"},
|
||||
{"key": "system", "label": "System alerts", "description": "Platform infrastructure alerts (e.g. the AI gateway going down)"},
|
||||
{"key": "ai_quota_warning", "label": "AI quota warnings", "description": "You're approaching your daily Devii AI usage limit"},
|
||||
]
|
||||
|
||||
|
||||
|
||||
@@ -179,6 +179,7 @@ def init_db():
|
||||
("read", False),
|
||||
("created_at", ""),
|
||||
("updated_at", ""),
|
||||
("client_id", ""),
|
||||
):
|
||||
if not messages.has_column(column):
|
||||
messages.create_column_by_example(column, example)
|
||||
@@ -194,6 +195,7 @@ def init_db():
|
||||
["receiver_uid", "sender_uid"],
|
||||
)
|
||||
_index(db, "messages", "idx_messages_updated_at", ["updated_at"])
|
||||
_index(db, "messages", "idx_messages_dedupe", ["sender_uid", "client_id"])
|
||||
_index(db, "notifications", "idx_notifications_user", ["user_uid"])
|
||||
_index(db, "notifications", "idx_notifications_user_read", ["user_uid", "read"])
|
||||
push_registration = get_table("push_registration")
|
||||
@@ -372,6 +374,8 @@ def init_db():
|
||||
)
|
||||
_index(db, "bookmarks", "idx_bookmarks_user", ["user_uid"])
|
||||
_index(db, "bookmarks", "idx_bookmarks_target", ["target_type", "target_uid"])
|
||||
_index(db, "notes", "idx_notes_user", ["user_uid"])
|
||||
_index(db, "notes", "idx_notes_target", ["target_type", "target_uid"])
|
||||
_index(db, "polls", "idx_polls_post", ["post_uid"])
|
||||
_index(db, "poll_options", "idx_poll_options_poll", ["poll_uid"])
|
||||
_index(db, "poll_votes", "idx_poll_votes_poll", ["poll_uid"])
|
||||
@@ -1248,6 +1252,12 @@ def init_db():
|
||||
_index(db, "deepsearch_sessions", "idx_deepsearch_sessions_owner", ["owner_kind", "owner_id"])
|
||||
_index(db, "deepsearch_sessions", "idx_deepsearch_sessions_status", ["status"])
|
||||
_index(db, "deepsearch_sessions", "idx_deepsearch_sessions_created", ["created_at"])
|
||||
_index(
|
||||
db,
|
||||
"deepsearch_sessions",
|
||||
"idx_deepsearch_sessions_owner_created",
|
||||
["owner_kind", "owner_id", "created_at"],
|
||||
)
|
||||
|
||||
deepsearch_messages = get_table("deepsearch_messages")
|
||||
for column, example in (
|
||||
@@ -2062,6 +2072,7 @@ def init_db():
|
||||
_index(db, "reactions", "idx_reactions_created_at", ["created_at"])
|
||||
_index(db, "votes", "idx_votes_created_at", ["created_at"])
|
||||
_index(db, "bookmarks", "idx_bookmarks_created_at", ["created_at"])
|
||||
_index(db, "notes", "idx_notes_created_at", ["created_at"])
|
||||
_index(db, "badges", "idx_badges_created_at", ["created_at"])
|
||||
_index(db, "audit_log", "idx_audit_result_created", ["result", "created_at"])
|
||||
_index(db, "jobs", "idx_jobs_created_at", ["created_at"])
|
||||
@@ -2281,6 +2292,10 @@ def backfill_api_keys() -> int:
|
||||
users.create_column_by_example("suspension_reason", "")
|
||||
if not users.has_column("deletion_requested_at"):
|
||||
users.create_column_by_example("deletion_requested_at", "")
|
||||
if not users.has_column("active_conversation_uid"):
|
||||
users.create_column_by_example("active_conversation_uid", "")
|
||||
if not users.has_column("active_conversation_at"):
|
||||
users.create_column_by_example("active_conversation_at", "")
|
||||
with db:
|
||||
db.query(
|
||||
"UPDATE users SET ai_modifier_enabled = 1 WHERE ai_modifier_enabled IS NULL"
|
||||
|
||||
@@ -16,6 +16,7 @@ SOFT_DELETE_TABLES = [
|
||||
"votes",
|
||||
"reactions",
|
||||
"bookmarks",
|
||||
"notes",
|
||||
"follows",
|
||||
"poll_votes",
|
||||
"polls",
|
||||
|
||||
@@ -118,15 +118,18 @@ def search_users_by_username(q, *, exclude_uid=None, limit=10):
|
||||
return []
|
||||
if exclude_uid is not None:
|
||||
rows = db.query(
|
||||
"SELECT uid, username FROM users WHERE username LIKE :q AND uid != :me LIMIT :limit",
|
||||
"SELECT uid, username, avatar_seed FROM users WHERE username LIKE :q AND uid != :me LIMIT :limit",
|
||||
q=f"%{q}%",
|
||||
me=exclude_uid,
|
||||
limit=limit,
|
||||
)
|
||||
else:
|
||||
rows = db.query(
|
||||
"SELECT uid, username FROM users WHERE username LIKE :q LIMIT :limit",
|
||||
"SELECT uid, username, avatar_seed FROM users WHERE username LIKE :q LIMIT :limit",
|
||||
q=f"%{q}%",
|
||||
limit=limit,
|
||||
)
|
||||
return [{"uid": r["uid"], "username": r["username"]} for r in rows]
|
||||
return [
|
||||
{"uid": r["uid"], "username": r["username"], "avatar_seed": r["avatar_seed"]}
|
||||
for r in rows
|
||||
]
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
VOTE_TARGETS = ["post", "comment", "gist", "project", "quiz"]
|
||||
REACTION_TARGETS = ["post", "comment", "gist", "project", "quiz"]
|
||||
BOOKMARK_TARGETS = ["post", "gist", "project", "news", "quiz"]
|
||||
NOTE_TARGETS = ["post", "gist", "project", "news"]
|
||||
COMMENT_TARGETS = ["post", "project", "news", "issue", "gist", "quiz"]
|
||||
PROJECT_TYPES = ["game", "game_asset", "software", "mobile_app", "website"]
|
||||
GIST_LANGUAGES = [
|
||||
@@ -38,6 +39,7 @@ def field(
|
||||
example="",
|
||||
description="",
|
||||
options=None,
|
||||
nullable=False,
|
||||
):
|
||||
spec = {
|
||||
"name": name,
|
||||
@@ -46,6 +48,7 @@ def field(
|
||||
"required": required,
|
||||
"example": example,
|
||||
"description": description,
|
||||
"nullable": nullable,
|
||||
}
|
||||
if options:
|
||||
spec["options"] = list(options)
|
||||
|
||||
@@ -62,6 +62,48 @@ four ways to sign requests.
|
||||
"Search post title, content, and author username.",
|
||||
),
|
||||
field("before", "query", "string", False, "", "Pagination cursor."),
|
||||
field(
|
||||
"items[].poll",
|
||||
"response",
|
||||
"object",
|
||||
description="Present only when the post has a poll attached.",
|
||||
nullable=True,
|
||||
),
|
||||
field(
|
||||
"items[].war",
|
||||
"response",
|
||||
"object",
|
||||
description="Present only when an Opinion War is running on the post.",
|
||||
nullable=True,
|
||||
),
|
||||
field(
|
||||
"items[].project_link",
|
||||
"response",
|
||||
"object",
|
||||
description="Present only when the post is attached to a project.",
|
||||
nullable=True,
|
||||
),
|
||||
field(
|
||||
"items[].maturity",
|
||||
"response",
|
||||
"string",
|
||||
description="Content maturity label; absent for unrated posts.",
|
||||
nullable=True,
|
||||
),
|
||||
field(
|
||||
"items[].my_vote",
|
||||
"response",
|
||||
"integer",
|
||||
description="Viewer's own vote on the post, defaults to 0 (never null).",
|
||||
nullable=False,
|
||||
),
|
||||
field(
|
||||
"items[].comment_count",
|
||||
"response",
|
||||
"integer",
|
||||
description="Total comment count, defaults to 0 (never null).",
|
||||
nullable=False,
|
||||
),
|
||||
],
|
||||
),
|
||||
endpoint(
|
||||
@@ -179,7 +221,49 @@ four ways to sign requests.
|
||||
True,
|
||||
"POST_SLUG",
|
||||
"Slug or UID of the post.",
|
||||
)
|
||||
),
|
||||
field(
|
||||
"post.title",
|
||||
"response",
|
||||
"string",
|
||||
description="Optional title, blank when the author posted without one.",
|
||||
nullable=True,
|
||||
),
|
||||
field(
|
||||
"post.image",
|
||||
"response",
|
||||
"string",
|
||||
description="Cover image URL; null unless one was attached.",
|
||||
nullable=True,
|
||||
),
|
||||
field(
|
||||
"post.project_uid",
|
||||
"response",
|
||||
"string",
|
||||
description="Attached project uid; null unless the post is linked to a project.",
|
||||
nullable=True,
|
||||
),
|
||||
field(
|
||||
"post.updated_at",
|
||||
"response",
|
||||
"string",
|
||||
description="ISO timestamp of the last edit; null until the post is edited for the first time.",
|
||||
nullable=True,
|
||||
),
|
||||
field(
|
||||
"post.content",
|
||||
"response",
|
||||
"string",
|
||||
description="Post body, always present (minimum 10 characters at creation).",
|
||||
nullable=False,
|
||||
),
|
||||
field(
|
||||
"post.slug",
|
||||
"response",
|
||||
"string",
|
||||
description="Always generated at creation, never null.",
|
||||
nullable=False,
|
||||
),
|
||||
],
|
||||
),
|
||||
endpoint(
|
||||
|
||||
@@ -130,9 +130,17 @@ X-App-Reference: devplace-devii-v-1-0-0
|
||||
Administrators enable and configure this gateway under [Background Services](/docs/services.html)
|
||||
(the `openai` service).
|
||||
|
||||
The gateway is exempt from rate limiting, but every other endpoint follows the shared
|
||||
The gateway is exempt from general rate limiting, but every other endpoint follows the shared
|
||||
[Conventions & Errors](/docs/conventions.html); see [Authentication](/docs/authentication.html)
|
||||
for signing DevPlace's own requests.
|
||||
|
||||
A dedicated failed-authentication throttle protects the gateway from unauthenticated probing:
|
||||
an IP that repeatedly presents no credentials at all and fails authentication is answered `429`
|
||||
(with a `Retry-After` header) once it crosses a configurable threshold within a rolling window.
|
||||
This never affects a request that presents a valid key or session, even from an IP that has
|
||||
recently failed - a legitimate caller sharing a network address with a prior bad actor is never
|
||||
locked out. A `429` returned to a properly-authenticated call is a separate daily quota limit,
|
||||
not this throttle.
|
||||
""",
|
||||
"endpoints": [
|
||||
endpoint(
|
||||
|
||||
@@ -34,7 +34,13 @@ four ways to sign requests.
|
||||
)
|
||||
],
|
||||
sample_response={
|
||||
"results": [{"uid": "8f14e45f-...", "username": "alice_test"}]
|
||||
"results": [
|
||||
{
|
||||
"uid": "8f14e45f-...",
|
||||
"username": "alice_test",
|
||||
"avatar_seed": None,
|
||||
}
|
||||
]
|
||||
},
|
||||
),
|
||||
endpoint(
|
||||
@@ -54,7 +60,13 @@ four ways to sign requests.
|
||||
)
|
||||
],
|
||||
sample_response={
|
||||
"results": [{"uid": "0cc175b9-...", "username": "bob_test"}]
|
||||
"results": [
|
||||
{
|
||||
"uid": "0cc175b9-...",
|
||||
"username": "bob_test",
|
||||
"avatar_seed": None,
|
||||
}
|
||||
]
|
||||
},
|
||||
),
|
||||
],
|
||||
|
||||
@@ -65,6 +65,55 @@ four ways to sign requests.
|
||||
"Profile tab.",
|
||||
["posts", "activity", "followers", "following", "media", "awards"],
|
||||
),
|
||||
field(
|
||||
"api_key",
|
||||
"response",
|
||||
"string",
|
||||
description="The profile's API key; null unless the viewer is the owner or an admin (see can_view_api_key).",
|
||||
nullable=True,
|
||||
),
|
||||
field(
|
||||
"rank",
|
||||
"response",
|
||||
"integer",
|
||||
description="Leaderboard rank; null for a user with no stars yet.",
|
||||
nullable=True,
|
||||
),
|
||||
field(
|
||||
"follow_pagination",
|
||||
"response",
|
||||
"object",
|
||||
description="Pagination metadata for the followers/following tabs; null on every other tab.",
|
||||
nullable=True,
|
||||
),
|
||||
field(
|
||||
"profile_user.avatar_seed",
|
||||
"response",
|
||||
"string",
|
||||
description="Nullable users.avatar_seed override; null falls back to the username as the avatar seed.",
|
||||
nullable=True,
|
||||
),
|
||||
field(
|
||||
"profile_user.bio",
|
||||
"response",
|
||||
"string",
|
||||
description="Optional profile bio; null when never set.",
|
||||
nullable=True,
|
||||
),
|
||||
field(
|
||||
"is_following",
|
||||
"response",
|
||||
"boolean",
|
||||
description="Always a boolean, defaults to false, never null.",
|
||||
nullable=False,
|
||||
),
|
||||
field(
|
||||
"profile_user.username",
|
||||
"response",
|
||||
"string",
|
||||
description="Always present, never null.",
|
||||
nullable=False,
|
||||
),
|
||||
],
|
||||
),
|
||||
endpoint(
|
||||
|
||||
@@ -20,9 +20,9 @@ its build. A registration body without a `provider` field is a `webpush` registr
|
||||
existing clients need no change. An APNs body may include a stable `client_id` so a later
|
||||
token rotation updates the same device instead of inserting another row.
|
||||
|
||||
There is no server-side unsubscribe endpoint: unsubscription is handled entirely in the
|
||||
browser by calling `PushManager.unsubscribe()` on the subscription. The server stops delivering
|
||||
to a subscription once its push endpoint reports it as gone. These mirror the in-app
|
||||
`DELETE /push.json` removes a single registration by the same identity used to create it
|
||||
(`endpoint` for webpush, `token` or `client_id` for apns). The server also stops delivering to a
|
||||
subscription once its push endpoint reports it as gone. These mirror the in-app
|
||||
[Notifications](/docs/notifications.html) feed.
|
||||
|
||||
Every endpoint follows the shared [Conventions & Errors](/docs/conventions.html) (auth, content
|
||||
@@ -104,5 +104,56 @@ four ways to sign requests.
|
||||
],
|
||||
sample_response={"registered": True, "delivered": True},
|
||||
),
|
||||
endpoint(
|
||||
id="push-unregister",
|
||||
method="DELETE",
|
||||
path="/push.json",
|
||||
title="Unregister a subscription",
|
||||
summary="Remove one push registration by its identity.",
|
||||
auth="user",
|
||||
encoding="json",
|
||||
interactive=False,
|
||||
params=[
|
||||
field(
|
||||
"provider",
|
||||
"json",
|
||||
"string",
|
||||
False,
|
||||
"webpush",
|
||||
"Provider the registration belongs to. Omit for webpush.",
|
||||
),
|
||||
field(
|
||||
"endpoint",
|
||||
"json",
|
||||
"string",
|
||||
False,
|
||||
"https://fcm.googleapis.com/...",
|
||||
"Subscription endpoint URL. Identifies a webpush registration.",
|
||||
),
|
||||
field(
|
||||
"token",
|
||||
"json",
|
||||
"string",
|
||||
False,
|
||||
"a1b2c3...",
|
||||
"Hexadecimal device token. Identifies an apns registration.",
|
||||
),
|
||||
field(
|
||||
"client_id",
|
||||
"json",
|
||||
"string",
|
||||
False,
|
||||
"vendor-uuid",
|
||||
"Stable per-device id. Identifies an apns registration when set.",
|
||||
),
|
||||
],
|
||||
notes=[
|
||||
"Exactly one identity field is required: `endpoint` for webpush, `token` or `client_id` for apns.",
|
||||
"Matches the same identity priority as registration: `client_id`, then `token`, then `endpoint`.",
|
||||
"Idempotent: unregistering an unknown or already-removed identity still returns 200 with `unregistered: false`.",
|
||||
"Call this before logging out to stop delivery to the device that is logging out - the server has no way to know a browser tab or native app closed on its own.",
|
||||
],
|
||||
sample_response={"unregistered": True},
|
||||
),
|
||||
],
|
||||
}
|
||||
|
||||
@@ -1,18 +1,29 @@
|
||||
# retoor <retoor@molodetz.nl>
|
||||
|
||||
from .._shared import BOOKMARK_TARGETS, REACTION_TARGETS, VOTE_TARGETS, endpoint, field
|
||||
from .._shared import (
|
||||
BOOKMARK_TARGETS,
|
||||
NOTE_TARGETS,
|
||||
REACTION_TARGETS,
|
||||
VOTE_TARGETS,
|
||||
endpoint,
|
||||
field,
|
||||
)
|
||||
from devplacepy.constants import REACTION_EMOJI
|
||||
|
||||
GROUP = {
|
||||
"slug": "social-actions",
|
||||
"title": "Votes, Reactions, Bookmarks & Polls",
|
||||
"title": "Votes, Reactions, Bookmarks, Notes & Polls",
|
||||
"intro": """
|
||||
# Votes, Reactions, Bookmarks & Polls
|
||||
# Votes, Reactions, Bookmarks, Notes & Polls
|
||||
|
||||
Lightweight engagement actions. The POST endpoints here are **toggles** - sending the same
|
||||
action again removes it. They return JSON when called with `X-Requested-With: fetch` (sent
|
||||
automatically by the panels below); the [Conventions & Errors](/docs/conventions.html) page
|
||||
explains that header rule and the response envelope.
|
||||
Lightweight engagement actions. The vote/reaction/bookmark POST endpoints here are **toggles** -
|
||||
sending the same action again removes it. They return JSON when called with
|
||||
`X-Requested-With: fetch` (sent automatically by the panels below); the
|
||||
[Conventions & Errors](/docs/conventions.html) page explains that header rule and the response
|
||||
envelope.
|
||||
|
||||
Personal notes are private, per-user annotations attached to a piece of content - only you can
|
||||
ever see your own notes.
|
||||
|
||||
Every endpoint follows the shared [Conventions & Errors](/docs/conventions.html) (auth, content
|
||||
negotiation, pagination, status codes); see [Authentication](/docs/authentication.html) for the
|
||||
@@ -24,7 +35,7 @@ four ways to sign requests.
|
||||
method="POST",
|
||||
path="/votes/{target_type}/{target_uid}",
|
||||
title="Cast or toggle a vote",
|
||||
summary="Upvote or downvote a target. Re-sending the same value removes the vote.",
|
||||
summary="Upvote or downvote a target. Re-sending the same value, or sending 0, removes the vote.",
|
||||
auth="user",
|
||||
ajax=True,
|
||||
encoding="form",
|
||||
@@ -52,8 +63,8 @@ four ways to sign requests.
|
||||
"enum",
|
||||
True,
|
||||
"1",
|
||||
"1 to upvote, -1 to downvote.",
|
||||
["1", "-1"],
|
||||
"1 to upvote, -1 to downvote, 0 to retract your existing vote.",
|
||||
["1", "-1", "0"],
|
||||
),
|
||||
],
|
||||
sample_response={"net": 3, "up": 4, "down": 1, "value": 1},
|
||||
@@ -151,6 +162,97 @@ four ways to sign requests.
|
||||
"Bookmarks target posts, projects, gists, and news; see [Posts, Comments, Projects, Gists & News](/docs/content.html)."
|
||||
],
|
||||
),
|
||||
endpoint(
|
||||
id="notes-set",
|
||||
method="POST",
|
||||
path="/notes/{target_type}/{target_uid}",
|
||||
title="Add or update a personal note",
|
||||
summary="Save a private note on a target. Only you can ever see it.",
|
||||
auth="user",
|
||||
ajax=True,
|
||||
encoding="form",
|
||||
params=[
|
||||
field(
|
||||
"target_type",
|
||||
"path",
|
||||
"enum",
|
||||
True,
|
||||
"post",
|
||||
"Type of content to annotate.",
|
||||
NOTE_TARGETS,
|
||||
),
|
||||
field(
|
||||
"target_uid",
|
||||
"path",
|
||||
"string",
|
||||
True,
|
||||
"POST_UID",
|
||||
"UID of the target.",
|
||||
),
|
||||
field(
|
||||
"content",
|
||||
"form",
|
||||
"string",
|
||||
True,
|
||||
"Remember to check this later.",
|
||||
"Note body, up to 4000 characters. Sending again on the same target replaces the note.",
|
||||
),
|
||||
],
|
||||
sample_response={"uid": "NOTE_UID", "content": "Remember to check this later."},
|
||||
),
|
||||
endpoint(
|
||||
id="notes-delete",
|
||||
method="POST",
|
||||
path="/notes/{target_type}/{target_uid}/delete",
|
||||
title="Delete a personal note",
|
||||
summary="Remove your private note from a target.",
|
||||
auth="user",
|
||||
ajax=True,
|
||||
encoding="none",
|
||||
params=[
|
||||
field(
|
||||
"target_type",
|
||||
"path",
|
||||
"enum",
|
||||
True,
|
||||
"post",
|
||||
"Type of content the note is on.",
|
||||
NOTE_TARGETS,
|
||||
),
|
||||
field(
|
||||
"target_uid",
|
||||
"path",
|
||||
"string",
|
||||
True,
|
||||
"POST_UID",
|
||||
"UID of the target.",
|
||||
),
|
||||
],
|
||||
sample_response={"deleted": True},
|
||||
),
|
||||
endpoint(
|
||||
id="notes-saved",
|
||||
method="GET",
|
||||
path="/notes/saved",
|
||||
title="View your personal notes",
|
||||
summary="Render your saved notes. Returns an HTML page.",
|
||||
auth="user",
|
||||
interactive=True,
|
||||
params=[
|
||||
field(
|
||||
"before",
|
||||
"query",
|
||||
"string",
|
||||
False,
|
||||
"",
|
||||
"Pagination cursor (created_at of the last item).",
|
||||
)
|
||||
],
|
||||
notes=[
|
||||
"Notes target posts, projects, gists, and news; see [Posts, Comments, Projects, Gists & News](/docs/content.html).",
|
||||
"Nobody else can ever see your notes, not even the author of the content you annotated.",
|
||||
],
|
||||
),
|
||||
endpoint(
|
||||
id="polls-vote",
|
||||
method="POST",
|
||||
|
||||
@@ -233,6 +233,36 @@ status and report.
|
||||
"export_pdf_url": "/tools/deepsearch/DEEPSEARCH_JOB_UID/export.pdf",
|
||||
},
|
||||
),
|
||||
endpoint(
|
||||
id="tools-deepsearch-history",
|
||||
method="GET",
|
||||
path="/tools/deepsearch/history",
|
||||
title="My DeepSearch history",
|
||||
summary="List the caller's own past DeepSearch runs, newest first, with a link back to each report/chat. Member history is account-bound; guest history is scoped to the visitor's address. Negotiates HTML or JSON.",
|
||||
auth="public",
|
||||
params=[
|
||||
field("limit", "query", "integer", False, "20", "Maximum sessions to return (1-100)."),
|
||||
],
|
||||
sample_response={
|
||||
"sessions": [
|
||||
{
|
||||
"uid": "DEEPSEARCH_JOB_UID",
|
||||
"query": "history of the transistor",
|
||||
"status": "done",
|
||||
"score": 78,
|
||||
"confidence": 0.72,
|
||||
"page_count": 11,
|
||||
"chunk_count": 240,
|
||||
"summary": "The transistor was invented at Bell Labs in 1947...",
|
||||
"reopen_url": "/tools/deepsearch/DEEPSEARCH_JOB_UID/session",
|
||||
"chat_available": True,
|
||||
"available": True,
|
||||
"created_at": "2026-06-14T10:00:00+00:00",
|
||||
"completed_at": "2026-06-14T10:01:40+00:00",
|
||||
}
|
||||
]
|
||||
},
|
||||
),
|
||||
endpoint(
|
||||
id="tools-isslop-run",
|
||||
method="POST",
|
||||
|
||||
@@ -37,6 +37,7 @@ _PAGE_RESPONSES = {
|
||||
"issues-detail": schemas.IssueDetailOut,
|
||||
"issues-attachments-list": schemas.IssueAttachmentsOut,
|
||||
"bookmarks-saved": schemas.SavedOut,
|
||||
"notes-saved": schemas.NotesOut,
|
||||
"admin-users": schemas.AdminUsersOut,
|
||||
"admin-news-list": schemas.AdminNewsOut,
|
||||
"admin-settings-get": schemas.AdminSettingsOut,
|
||||
|
||||
@@ -31,8 +31,8 @@ def _params_table(params: list) -> str:
|
||||
if not params:
|
||||
return ""
|
||||
rows = [
|
||||
"| Name | In | Type | Required | Description |",
|
||||
"|------|----|------|----------|-------------|",
|
||||
"| Name | In | Type | Required | Nullable | Description |",
|
||||
"|------|----|------|----------|----------|-------------|",
|
||||
]
|
||||
for p in params:
|
||||
desc = (p.get("description", "") or "").replace("|", "\\|")
|
||||
@@ -43,7 +43,8 @@ def _params_table(params: list) -> str:
|
||||
desc = f"{desc} Allowed: {allowed}.".strip()
|
||||
rows.append(
|
||||
f"| `{p['name']}` | {p['location']} | {p['type']} | "
|
||||
f"{'yes' if p['required'] else 'no'} | {desc} |"
|
||||
f"{'yes' if p['required'] else 'no'} | "
|
||||
f"{'yes' if p.get('nullable') else 'no'} | {desc} |"
|
||||
)
|
||||
return "\n".join(rows)
|
||||
|
||||
|
||||
@@ -78,6 +78,7 @@ from devplacepy.routers import (
|
||||
reactions,
|
||||
reports,
|
||||
bookmarks,
|
||||
notes,
|
||||
polls,
|
||||
docs,
|
||||
openai_gateway,
|
||||
@@ -493,6 +494,7 @@ app.include_router(votes.router, prefix="/votes")
|
||||
app.include_router(reactions.router, prefix="/reactions")
|
||||
app.include_router(reports.router, prefix="/reports")
|
||||
app.include_router(bookmarks.router, prefix="/bookmarks")
|
||||
app.include_router(notes.router, prefix="/notes")
|
||||
app.include_router(polls.router, prefix="/polls")
|
||||
app.include_router(avatar.router, prefix="/avatar")
|
||||
app.include_router(awards.router, prefix="/awards")
|
||||
|
||||
@@ -258,6 +258,10 @@ class CommentEditForm(BaseModel):
|
||||
content: str = Field(min_length=3, max_length=125000)
|
||||
|
||||
|
||||
class NoteForm(BaseModel):
|
||||
content: str = Field(min_length=1, max_length=4000)
|
||||
|
||||
|
||||
class ProjectForm(BaseModel):
|
||||
title: str = Field(min_length=1, max_length=200)
|
||||
description: str = Field(min_length=1, max_length=5000)
|
||||
@@ -547,8 +551,8 @@ class VoteForm(BaseModel):
|
||||
@field_validator("value")
|
||||
@classmethod
|
||||
def valid_value(cls, value):
|
||||
if value not in (1, -1):
|
||||
raise ValueError("value must be 1 or -1")
|
||||
if value not in (1, -1, 0):
|
||||
raise ValueError("value must be 1, -1 or 0")
|
||||
return value
|
||||
|
||||
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
import logging
|
||||
import os
|
||||
import shutil
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
@@ -18,6 +19,8 @@ MAX_FILES_PER_PROJECT = 5000
|
||||
MAX_PATH_LENGTH = 1024
|
||||
MAX_SEGMENT_LENGTH = 255
|
||||
MAX_DEPTH = 32
|
||||
_MISSING_BLOB_LOG_SECONDS = 60
|
||||
_missing_blob_logged: dict[str, tuple[int, float]] = {}
|
||||
|
||||
TEXT_EXTENSIONS = {
|
||||
".txt",
|
||||
@@ -597,6 +600,7 @@ def sync_dir_bidirectional(project_uid: str, workspace, user: dict) -> dict:
|
||||
manifest = _load_sync_manifest(project_uid)
|
||||
new_manifest: dict = {}
|
||||
counts = dict(empty)
|
||||
missing_blobs = 0
|
||||
|
||||
for path in set(manifest) | set(db_files) | set(fs_files):
|
||||
row = db_files.get(path)
|
||||
@@ -621,6 +625,8 @@ def sync_dir_bidirectional(project_uid: str, workspace, user: dict) -> dict:
|
||||
if recorded:
|
||||
counts["exported"] += 1
|
||||
new_manifest[path] = recorded
|
||||
elif row.get("is_binary"):
|
||||
missing_blobs += 1
|
||||
else:
|
||||
recorded = _record_import(project_uid, user, path, fs_full, fs_epoch)
|
||||
if recorded:
|
||||
@@ -639,6 +645,8 @@ def sync_dir_bidirectional(project_uid: str, workspace, user: dict) -> dict:
|
||||
if recorded:
|
||||
counts["exported"] += 1
|
||||
new_manifest[path] = recorded
|
||||
elif row.get("is_binary"):
|
||||
missing_blobs += 1
|
||||
else:
|
||||
_delete_db_row_for_sync(row, user["uid"])
|
||||
counts["deleted_in_project"] += 1
|
||||
@@ -670,6 +678,7 @@ def sync_dir_bidirectional(project_uid: str, workspace, user: dict) -> dict:
|
||||
continue
|
||||
|
||||
_save_sync_manifest(project_uid, manifest, new_manifest)
|
||||
_warn_missing_blobs(project_uid, missing_blobs, "while syncing")
|
||||
return counts
|
||||
|
||||
|
||||
@@ -692,6 +701,34 @@ def _record_import(project_uid: str, user: dict, path: str, fs_full: Path, fs_ep
|
||||
return {"db_epoch": _epoch_of(imported.get("updated_at")), "fs_epoch": fs_epoch}
|
||||
|
||||
|
||||
def _copy_blob(src: Path, target: Path) -> bool:
|
||||
try:
|
||||
shutil.copyfile(src, target)
|
||||
return True
|
||||
except (FileNotFoundError, OSError):
|
||||
return False
|
||||
|
||||
|
||||
def _warn_missing_blobs(project_uid: str, missing: int, action: str) -> None:
|
||||
if missing <= 0:
|
||||
return
|
||||
now = time.monotonic()
|
||||
previous = _missing_blob_logged.get(project_uid)
|
||||
if (
|
||||
previous is not None
|
||||
and previous[0] == missing
|
||||
and (now - previous[1]) < _MISSING_BLOB_LOG_SECONDS
|
||||
):
|
||||
return
|
||||
_missing_blob_logged[project_uid] = (missing, now)
|
||||
logger.warning(
|
||||
"Skipped %s missing blob file(s) %s project %s",
|
||||
missing,
|
||||
action,
|
||||
project_uid,
|
||||
)
|
||||
|
||||
|
||||
def _export_node(row: dict, dest: Path):
|
||||
target = (dest / row["path"]).resolve()
|
||||
if target != dest and not target.is_relative_to(dest):
|
||||
@@ -701,10 +738,7 @@ def _export_node(row: dict, dest: Path):
|
||||
target.unlink()
|
||||
if row.get("is_binary") and row.get("stored_name") and row.get("directory"):
|
||||
src = PROJECT_FILES_DIR / row["directory"] / row["stored_name"]
|
||||
try:
|
||||
shutil.copyfile(src, target)
|
||||
except (FileNotFoundError, OSError):
|
||||
logger.warning("Blob file missing during export: %s", src)
|
||||
if not _copy_blob(src, target):
|
||||
return None
|
||||
else:
|
||||
target.write_text(row.get("content") or "", encoding="utf-8")
|
||||
@@ -853,6 +887,7 @@ def export_to_dir(project_uid: str, subpath: str, dest_dir) -> int:
|
||||
rows = list(_table().find(project_uid=project_uid, deleted_at=None))
|
||||
strip = ""
|
||||
written = 0
|
||||
missing = 0
|
||||
for row in sorted(rows, key=lambda r: r["path"]):
|
||||
relative = row["path"][len(strip) :].lstrip("/") if strip else row["path"]
|
||||
target = (dest / relative).resolve()
|
||||
@@ -866,14 +901,13 @@ def export_to_dir(project_uid: str, subpath: str, dest_dir) -> int:
|
||||
target.unlink()
|
||||
if row.get("is_binary") and row.get("stored_name") and row.get("directory"):
|
||||
src = PROJECT_FILES_DIR / row["directory"] / row["stored_name"]
|
||||
try:
|
||||
shutil.copyfile(src, target)
|
||||
except (FileNotFoundError, OSError):
|
||||
logger.warning("Blob file missing: %s", src)
|
||||
if not _copy_blob(src, target):
|
||||
missing += 1
|
||||
continue
|
||||
else:
|
||||
target.write_text(row.get("content") or "", encoding="utf-8")
|
||||
written += 1
|
||||
_warn_missing_blobs(project_uid, missing, "during export of")
|
||||
return written
|
||||
|
||||
|
||||
|
||||
@@ -2,7 +2,7 @@ This file documents `devplacepy/push/` - push notification delivery and its prov
|
||||
|
||||
## What this package is
|
||||
|
||||
One delivery library behind one public surface. `devplacepy.push` exports `notify_user`, `notify_registration`, `register`, `ensure_certificates`, `public_key_standard_b64` and the Web Push crypto helpers; every caller in the codebase (`main.py`, `utils/notifications.py`, `routers/push.py`) imports only those names. Everything else is internal to the package.
|
||||
One delivery library behind one public surface. `devplacepy.push` exports `notify_user`, `notify_registration`, `register`, `unregister`, `ensure_certificates`, `public_key_standard_b64` and the Web Push crypto helpers; every caller in the codebase (`main.py`, `utils/notifications.py`, `routers/push.py`) imports only those names. Everything else is internal to the package.
|
||||
|
||||
| Module | Role |
|
||||
|---|---|
|
||||
@@ -57,6 +57,14 @@ The default `delivery_client` is `stealth_async_client`. Override it only when t
|
||||
|
||||
A match **updates** the row (token, client_id, environment) and clears `deleted_at` if it was dead. Token rotation with the same `client_id` therefore replaces the token on one row. Sibling live rows that share the new token, the previous token, or the same `client_id` are marked dead so one device cannot accumulate duplicates. A body without `client_id` still works: the same token revives, a new token inserts. `push.update` is a real update, not a no-op of an identical POST.
|
||||
|
||||
## Unregistering
|
||||
|
||||
`DELETE /push.json` (`routers/push.py` `push_unregister`) removes exactly the one registration named by the caller - never every registration for the user, mirroring how `POST /push.json` creates or updates exactly one row. `store.unregister(user_uid, provider, fields)` looks the row up with the **same identity priority as `register`**: `client_id`, then `token`, then `endpoint`. A match stamps `deleted_at` (the same bespoke soft-delete `mark_dead` already uses for a dead token - `push_registration` has no `deleted_by` and is not restorable from Trash by design, see "Invariants" above), so a later re-`register` with the same identity revives the row exactly like a dead token does, rather than accumulating a duplicate. No match (unknown identity, already-removed, or wrong owner) is a no-op that still returns `200 {"unregistered": false}` - the call is idempotent so a client can fire it speculatively at logout without first checking whether a subscription exists. The route validates that at least one identity field is a non-empty string before calling the store, refusing `400` on a body carrying none, exactly like `POST /push.json` refuses an unparseable registration body.
|
||||
|
||||
**Client wiring.** `static/js/PushManager.js` intercepts every `a[href="/auth/logout"]` click, resolves the current `PushManager.getSubscription()` (only ever a webpush one - browser JS has no access to a native APNs device token), calls `DELETE /push.json` with `{endpoint}`, then `subscription.unsubscribe()` client-side, then navigates to the logout link's `href`. This closes the webpush half of the "token stays registered after logout" gap. **A native (iOS/Android) client integrating APNs must call `DELETE /push.json` with `{provider: "apns", token}` (or `client_id`) itself before or alongside its own logout** - there is no way for server-side `GET /auth/logout` (a plain, bodyless navigation) or this web frontend to know about, let alone unregister, a native app's device token.
|
||||
|
||||
**No Devii action.** Unlike most mutating endpoints, `push.json` (register OR unregister) has no Devii catalog entry - a push identity (a webpush subscription endpoint/keys or an APNs device token) is private per-device browser/OS state that Devii cannot obtain, generate, or usefully ask the user to paste into a chat turn. This mirrors `POST /push.json` already having no Devii action for the identical reason.
|
||||
|
||||
## APNs specifics
|
||||
|
||||
- **Persistent HTTP/2 connection, not one per notification.** `apns.cached_client(timeout)` lazily creates ONE module-level `httpx.AsyncClient(http2=True, ...)` and reuses it across every `notify_user`/`notify_registration` call for the lifetime of the process, following Apple's explicit guidance to keep the connection open rather than repeatedly opening/closing (`sending-notification-requests-to-apns`). `ApnsProvider.closes_delivery_client()` returns `False` so `delivery.py` never closes it after a batch (Web Push still opens/closes per call via `stealth_async_client`, `closes_delivery_client()` defaulting `True` on the base class). Closed once, gracefully, in `main.py`'s shutdown via `push.shutdown_providers()` -> `ApnsProvider.aclose()` -> `apns.close_client()`, mirroring the identical `services/containers/forward.py` `client()`/`close_client()` pattern. Not stealth, not curl_cffi Chrome impersonation, not the outbound proxy - Apple's provider API is a first-party HTTP/2 service; browser PRIORITY frames, HPACK indexing of `:path`/`authorization`, extra `sec-ch-ua` headers, and a scraping proxy all violate that contract. This is the second documented exception to the stealth-only outbound rule (the other is container reverse-proxy forwarding). If the admin-configured delivery timeout changes, the cached client is rebuilt with the new timeout on next use and the old one is left for GC (not explicitly closed) - a deliberate, rare-path simplification.
|
||||
|
||||
@@ -13,7 +13,7 @@ from devplacepy.push.providers.webpush import (
|
||||
hkdf,
|
||||
public_key_standard_b64,
|
||||
)
|
||||
from devplacepy.push.store import register
|
||||
from devplacepy.push.store import register, unregister
|
||||
|
||||
__all__ = [
|
||||
"browser_base64",
|
||||
@@ -29,4 +29,5 @@ __all__ = [
|
||||
"public_key_standard_b64",
|
||||
"register",
|
||||
"shutdown_providers",
|
||||
"unregister",
|
||||
]
|
||||
|
||||
@@ -179,6 +179,30 @@ def register(
|
||||
return RegistrationWrite(record, True, False)
|
||||
|
||||
|
||||
def unregister(user_uid: str, provider: str, fields: dict[str, Any]) -> bool:
|
||||
fields = _filled(fields)
|
||||
client_id = fields.get("client_id")
|
||||
token = fields.get("token")
|
||||
endpoint = fields.get("endpoint")
|
||||
|
||||
row = None
|
||||
if client_id:
|
||||
row = _lookup(user_uid, provider, client_id=client_id)
|
||||
if row is None and token:
|
||||
row = _lookup(user_uid, provider, token=token)
|
||||
if row is None and endpoint:
|
||||
row = _lookup(user_uid, provider, endpoint=endpoint)
|
||||
if row is None or row.get("deleted_at"):
|
||||
return False
|
||||
|
||||
table().update(
|
||||
{"id": row["id"], "deleted_at": datetime.now(timezone.utc).isoformat()},
|
||||
["id"],
|
||||
)
|
||||
logger.info("Unregistered %s push subscription for user %s", provider, user_uid)
|
||||
return True
|
||||
|
||||
|
||||
def mark_dead(registration_id: int, dead_before: str | None = None) -> None:
|
||||
if dead_before is not None:
|
||||
row = table().find_one(id=registration_id)
|
||||
|
||||
+76
-1
@@ -70,6 +70,26 @@ _TOKEN_RE = re.compile(
|
||||
_SCHEME_RE = re.compile(r"^([a-z][a-z0-9+.\-]*):", re.I)
|
||||
_ALLOWED_SCHEMES = {"http", "https", "mailto", "tel"}
|
||||
|
||||
_TRAILING_PUNCT = ".,;:!?"
|
||||
_TRAILING_CLOSERS = {")": "(", "]": "[", "}": "{"}
|
||||
|
||||
|
||||
def _split_trailing_punct(url: str) -> tuple[str, str]:
|
||||
trailing: list[str] = []
|
||||
while url:
|
||||
char = url[-1]
|
||||
if char in _TRAILING_PUNCT:
|
||||
trailing.append(char)
|
||||
url = url[:-1]
|
||||
continue
|
||||
opener = _TRAILING_CLOSERS.get(char)
|
||||
if opener is not None and url.count(char) > url.count(opener):
|
||||
trailing.append(char)
|
||||
url = url[:-1]
|
||||
continue
|
||||
break
|
||||
return url, "".join(reversed(trailing))
|
||||
|
||||
_YOUTUBE_ALLOW = (
|
||||
"accelerometer; autoplay; clipboard-write; encrypted-media; "
|
||||
"gyroscope; picture-in-picture"
|
||||
@@ -112,6 +132,40 @@ _content_markdown = mistune.create_markdown(
|
||||
plugins=["strikethrough", "table"],
|
||||
)
|
||||
|
||||
_structure_markdown = mistune.create_markdown(
|
||||
renderer=None,
|
||||
plugins=["strikethrough", "table"],
|
||||
)
|
||||
|
||||
_STRUCTURE_TOKEN_KEYS = {
|
||||
"block_code": "code_fences",
|
||||
"list_item": "list_items",
|
||||
"heading": "headers",
|
||||
"link": "links",
|
||||
}
|
||||
|
||||
|
||||
def _count_structure_tokens(tokens, signature: dict[str, int]) -> None:
|
||||
for token in tokens or []:
|
||||
key = _STRUCTURE_TOKEN_KEYS.get(token.get("type"))
|
||||
if key:
|
||||
signature[key] += 1
|
||||
children = token.get("children")
|
||||
if children:
|
||||
_count_structure_tokens(children, signature)
|
||||
|
||||
|
||||
def markdown_structure_signature(text: str) -> dict[str, int]:
|
||||
signature = {key: 0 for key in _STRUCTURE_TOKEN_KEYS.values()}
|
||||
if not text or not text.strip():
|
||||
return signature
|
||||
try:
|
||||
tokens = _structure_markdown(text)
|
||||
except (TypeError, ValueError):
|
||||
return signature
|
||||
_count_structure_tokens(tokens, signature)
|
||||
return signature
|
||||
|
||||
|
||||
def _normalize_dashes(text: str) -> str:
|
||||
text = text.replace("\u2014", "-")
|
||||
@@ -183,7 +237,10 @@ def _transform_text(text: str) -> str:
|
||||
if match.start() > pos:
|
||||
out.append(html.escape(_mask_emails(text[pos:match.start()])))
|
||||
if match.group("url"):
|
||||
out.append(_embed_url(match.group("url")))
|
||||
url, trailing = _split_trailing_punct(match.group("url"))
|
||||
out.append(_embed_url(url))
|
||||
if trailing:
|
||||
out.append(html.escape(trailing))
|
||||
else:
|
||||
user = match.group("mention")
|
||||
out.append(
|
||||
@@ -335,3 +392,21 @@ def content_preview(text, length: int = 60) -> str:
|
||||
if len(plain) <= length:
|
||||
return plain
|
||||
return plain[:length].rstrip() + "..."
|
||||
|
||||
|
||||
def safe_truncate(text, length: int = 300) -> str:
|
||||
if not text:
|
||||
return ""
|
||||
text_str = str(text)
|
||||
if len(text_str) <= length:
|
||||
return text_str
|
||||
cutoff = length
|
||||
for match in _TOKEN_RE.finditer(text_str):
|
||||
if match.start() < cutoff < match.end():
|
||||
cutoff = match.start()
|
||||
break
|
||||
if cutoff > 0 and not text_str[cutoff - 1].isspace() and not text_str[cutoff].isspace():
|
||||
boundary = text_str.rfind(" ", 0, cutoff)
|
||||
if boundary > 0:
|
||||
cutoff = boundary
|
||||
return text_str[:cutoff].rstrip()
|
||||
|
||||
@@ -21,6 +21,7 @@ Prefixes are wired in `main.py`:
|
||||
| `/votes` | votes.py |
|
||||
| `/reactions` | reactions.py - emoji reaction toggle on any target: `POST /reactions/{target_type}/{target_uid}` (optimistic `ReactionBar`) |
|
||||
| `/bookmarks` | bookmarks.py - bookmark/favorite toggle: `GET /bookmarks/saved` (the viewer's saved list) and `POST /bookmarks/{target_type}/{target_uid}` (toggle a bookmark) |
|
||||
| `/notes` | notes.py - private per-user annotation on a target: `GET /notes/saved` (the viewer's personal notes list), `POST /notes/{target_type}/{target_uid}` (add/replace the note), `POST /notes/{target_type}/{target_uid}/delete` (remove it) |
|
||||
| `/polls` | polls.py - poll voting: `POST /polls/{poll_uid}/vote` |
|
||||
| `/avatar` | avatar.py |
|
||||
| `/follow` | follow.py |
|
||||
@@ -39,8 +40,8 @@ Prefixes are wired in `main.py`:
|
||||
| `/devii` | devii.py - WebSocket terminal (`/devii/ws`), page, `/devii/usage`, `/devii/session` |
|
||||
| `/zips` | zips.py - generic zip job status (`/zips/{uid}`) and archive download (`/zips/{uid}/download`); enqueued from `/projects/{slug}/zip` and `/projects/{slug}/files/zip` |
|
||||
| `/forks` | forks.py - fork job status (`/forks/{uid}`); enqueued from `/projects/{slug}/fork`. When done its `project_url` points at the new forked project |
|
||||
| `/tools` | tools/ package - public developer tools surface. `index.py` (`/tools` landing) plus `seo.py` (**SEO Diagnostics**): `GET /tools/seo` page, `POST /tools/seo/run` (enqueue `seo` job, per-owner one-active-job cap), `GET /tools/seo/{uid}` (`SeoJobOut`), `GET /tools/seo/{uid}/report` (HTML+JSON `SeoReportOut`), `WS /tools/seo/{uid}/ws` (live progress, lock-owner gated, close `4013` retry), `GET /tools/seo/{uid}/screenshot/{n}`. Also `deepsearch.py` (**DeepSearch**): `GET /tools/deepsearch` page, `POST /tools/deepsearch/run` (enqueue `deepsearch` job, per-owner one-active-job cap; resolves the user `api_key` into the payload, guests use the internal key), `GET /tools/deepsearch/{uid}` (`DeepsearchJobOut`), `GET /tools/deepsearch/{uid}/session` (HTML+JSON `DeepsearchSessionOut`), `WS /tools/deepsearch/{uid}/ws` (live progress, lock-owner gated, `4013` retry), `WS /tools/deepsearch/{uid}/chat` (grounded RAG chat over the session collection), `POST /tools/deepsearch/{uid}/{pause|resume|cancel}`, `GET /tools/deepsearch/{uid}/export.{md,json,pdf}`. The shared owner helper is `routers/tools/_shared.py` `owner_for`. Also `isslop.py` (**AI Usage Analyzer**): `GET /tools/isslop` page, `POST /tools/isslop/run` (enqueue `isslop` job, per-owner one-active-job cap; owner = user uid or the shared `DEVII_GUEST_COOKIE` guest identity, minted when absent), `GET /tools/isslop/list` (owner history; a signed-in request first claims any guest-cookie analyses via `store.claim_guest_analyses` - a move, never a copy), `GET /tools/isslop/{uid}` (`IsslopAnalysisOut`), `GET /tools/isslop/{uid}/events` (persisted ordered event trail, `?after=SEQ` incremental poll; live frames also publish on pub/sub `public.isslop.{uid}` - the DB trail is the source of truth, pub/sub the fast path), `GET /tools/isslop/{uid}/report` (HTML+JSON `IsslopReportOut`; live `<dp-isslop-run>` while running, server-rendered report via `render_content` when done), `GET /tools/isslop/{uid}/report.md`, `GET /tools/isslop/{uid}/badge.svg` (embeddable SVG authenticity badge). Analyses/reports/badges are permanent public capability URLs (`IsslopService.cleanup` never deletes them; only the job row is swept). Surfaced by a collapsible **Tools** header dropdown (`base.html`, visible to all) |
|
||||
| `/projects/{slug}/containers` | projects/containers/ subpackage - admin per-project container manager (`instances.py` for creation/lifecycle/exec/logs/metrics/sync plus the exec websocket, `schedules.py` for cron/interval/once schedules, shared helpers in `_shared.py`). Every instance runs the shared `ppy` image. Discoverable from the project detail page (admin-only **Containers** button) and from the admin index |
|
||||
| `/tools` | tools/ package - public developer tools surface. `index.py` (`/tools` landing) plus `seo.py` (**SEO Diagnostics**): `GET /tools/seo` page, `POST /tools/seo/run` (enqueue `seo` job, per-owner one-active-job cap), `GET /tools/seo/{uid}` (`SeoJobOut`), `GET /tools/seo/{uid}/report` (HTML+JSON `SeoReportOut`), `WS /tools/seo/{uid}/ws` (live progress, lock-owner gated, close `4013` retry), `GET /tools/seo/{uid}/screenshot/{n}`. Also `deepsearch.py` (**DeepSearch**): `GET /tools/deepsearch` page, `POST /tools/deepsearch/run` (enqueue `deepsearch` job, per-owner one-active-job cap; resolves the user `api_key` into the payload, guests use the internal key), `GET /tools/deepsearch/history` (HTML+JSON `DeepsearchHistoryOut`, owner's own past sessions newest-first with a `reopen_url` back into the session/chat routes - `database.list_deepsearch_sessions`, the `deepsearch_sessions` table already persists past runs independently of job retention), `GET /tools/deepsearch/{uid}` (`DeepsearchJobOut`), `GET /tools/deepsearch/{uid}/session` (HTML+JSON `DeepsearchSessionOut`), `WS /tools/deepsearch/{uid}/ws` (live progress, lock-owner gated, `4013` retry), `WS /tools/deepsearch/{uid}/chat` (grounded RAG chat over the session collection, reconnectable on any completed session within its job's retention window), `POST /tools/deepsearch/{uid}/{pause|resume|cancel}`, `GET /tools/deepsearch/{uid}/export.{md,json,pdf}`. The shared owner helper is `routers/tools/_shared.py` `owner_for`. Also `isslop.py` (**AI Usage Analyzer**): `GET /tools/isslop` page, `POST /tools/isslop/run` (enqueue `isslop` job, per-owner one-active-job cap; owner = user uid or the shared `DEVII_GUEST_COOKIE` guest identity, minted when absent), `GET /tools/isslop/list` (owner history; a signed-in request first claims any guest-cookie analyses via `store.claim_guest_analyses` - a move, never a copy), `GET /tools/isslop/{uid}` (`IsslopAnalysisOut`), `GET /tools/isslop/{uid}/events` (persisted ordered event trail, `?after=SEQ` incremental poll; live frames also publish on pub/sub `public.isslop.{uid}` - the DB trail is the source of truth, pub/sub the fast path), `GET /tools/isslop/{uid}/report` (HTML+JSON `IsslopReportOut`; live `<dp-isslop-run>` while running, server-rendered report via `render_content` when done), `GET /tools/isslop/{uid}/report.md`, `GET /tools/isslop/{uid}/badge.svg` (embeddable SVG authenticity badge). Analyses/reports/badges are permanent public capability URLs (`IsslopService.cleanup` never deletes them; only the job row is swept). Reachable by direct URL only; the former **Tools** header dropdown in `base.html` was removed |
|
||||
| `/projects/{slug}/containers` | projects/containers/ subpackage - admin per-project container manager (`instances.py` for creation/lifecycle/exec/logs/metrics/sync plus the exec websocket, `schedules.py` for cron/interval/once schedules, shared helpers in `_shared.py`). Every instance runs the shared `ppy` image. Reachable by direct URL and from the admin index; the former project detail page **Containers** button was removed |
|
||||
| `/admin/containers` | admin/containers.py - admin **Containers** manager: `/admin/containers` lists every instance across all projects with inline actions (start/stop/restart/terminal/edit/delete) plus a create modal (project search-select, run-as user search-select, boot language + source editor, restart policy, start-on-boot, env/ports/limits/ingress); `/admin/containers/{uid}` is the per-instance detail page (lifecycle, live logs/metrics, PTY terminal, schedules, ingress, sync, status history) and `/admin/containers/{uid}/edit` edits run-as user, boot language/script/command, restart policy, start-on-boot, and limits. `POST /admin/containers/create`, `/{uid}/edit`, `/{uid}/{start,stop,restart,pause,resume}`, `/{uid}/sync`, `/{uid}/delete` call `api.*` directly under `require_admin` (no docker/exec backend duplicated); `GET /admin/containers/projects/search` and `/users/search` back the create/edit search-selects. The lifecycle and detail views stay layered over the per-project `/projects/{slug}/containers/instances/{uid}/...` endpoints (the instance carries its `project_uid`) |
|
||||
| `/p/{slug}` | proxy.py - public ingress reverse proxy (HTTP + WebSocket) to a running container instance's published host port, opt-in per instance via `ingress_slug` |
|
||||
| `/xmlrpc` | xmlrpc.py - reverse-proxies XML-RPC calls to the forking XML-RPC bridge (`services/xmlrpc/`, supervised by `XmlrpcService` on loopback `config.XMLRPC_PORT`), which generates one XML-RPC method per documented REST endpoint from `docs_api.API_GROUPS` (`posts.create`, `feed.list`, ...). One struct of named params per call; auth via in-band `api_key`, `X-API-KEY`/`Bearer` header, or `http://user:pass@host/xmlrpc` Basic. Full introspection + `system.multicall`; REST errors become XML-RPC faults. Exempt from the rate limiter (enforced on the forwarded internal hop). See `devplacepy/services/xmlrpc/CLAUDE.md` |
|
||||
@@ -49,7 +50,7 @@ Prefixes are wired in `main.py`:
|
||||
| `/quizzes` | quizzes/ package - **Quizzes**: `index.py` (hub, create, import, detail, export, edit, delete, publish, per-quiz leaderboard, cross-quiz scoreboard), `questions.py` (builder page + question CRUD + reorder), `attempts.py` (start/resume, play, answer, finish, results). The hub `GET /quizzes` is the approved three-column `.feed-layout` reused verbatim from `/feed`: filters + search left (`?filter=all|todo|done|mine|drafts`, `?search=`, `?page=`), the quiz list centre with the **New quiz** and **Create quiz with Devii** actions, and the cross-quiz **scoreboard** rail right (score per user, best attempt per quiz, 15s display cache). Publishing is terminal - every write on a published quiz is a 400 and there is no unpublish route. See `devplacepy/services/quiz/CLAUDE.md` |
|
||||
| `/battles` | battles.py - **Opinion Wars**, week-long two-faction battles attached to posts: `GET ""` (listing, quizzes-style filters active/ended/mine + search), `GET /{uid}` (JSON-only full state, runs lazy resolution), `GET /{uid}/events?after=` (durable event trail replay), `POST /{uid}/join` (join or switch faction), `POST /{uid}/fight` (spend 25 Code Farm coins, deal level-weighted damage, 24h cooldown). The battle card renders on the parent post (no battle detail HTML page). See `devplacepy/services/opinionwar/CLAUDE.md` |
|
||||
| `/game` | game/ package - the **Code Farm** idle game (`index.py` + `farm.py`): `GET /game` (page), `GET /game/state`, `GET /game/leaderboard?board=`, `POST /game/{plant,harvest,buy-plot,upgrade,fertilize,daily,grant,perk,prestige,legacy,mastery,quests/claim}`, `POST /game/{defense/upgrade,defense/downgrade,infrastructure/buy,cosmetics/buy,cosmetics/equip}`, plus social `GET /game/farm/{username}` and `POST /game/farm/{username}/{water,steal}`. See `devplacepy/services/game/CLAUDE.md` |
|
||||
| (none) | push.py - push + PWA: `GET /push.json` (VAPID public key + the providers accepting registrations; `apns` includes `environment` when active), `POST /push.json` (register with any active provider; a body without `provider` is a `webpush` body; an APNs body is `token` plus optional `client_id` for device-stable upsert; created/revived rows are probed and the JSON may include `delivered`/`error`; an unknown, disabled or unconfigured provider is a 400), `GET /service-worker.js`, `GET /manifest.json`. Provider protocol and delivery live in `devplacepy/push/` - see `devplacepy/push/CLAUDE.md` |
|
||||
| (none) | push.py - push + PWA: `GET /push.json` (VAPID public key + the providers accepting registrations; `apns` includes `environment` when active), `POST /push.json` (register with any active provider; a body without `provider` is a `webpush` body; an APNs body is `token` plus optional `client_id` for device-stable upsert; created/revived rows are probed and the JSON may include `delivered`/`error`; an unknown, disabled or unconfigured provider is a 400), `DELETE /push.json` (unregister exactly the one registration named by `endpoint`/`token`/`client_id`, same identity priority as registration; idempotent, always 200 `{unregistered}`; wired into `PushManager.js`'s logout-link interceptor so a webpush subscription is dropped before the browser navigates to `/auth/logout`), `GET /service-worker.js`, `GET /manifest.json`. Provider protocol and delivery live in `devplacepy/push/` - see `devplacepy/push/CLAUDE.md` |
|
||||
| (none) | docs.py (docs/ package) - the documentation site (prose pages + API reference). See `routers/docs/CLAUDE.md` for the deep detail on this tree (`DOCS_PAGES`, audience tiers, prose rendering pipeline) |
|
||||
| `/pubsub` | pubsub.py - **database-free** publish/subscribe bus. `WS /pubsub/ws` (subscribe/unsubscribe/publish frames, `foo.*` wildcards), `POST /pubsub/publish` + `GET /pubsub/topics` (admin/internal). Lock-owner-gated WS (`4013` retry) so subscribers converge on one worker; topic authz in `services/pubsub/policy.py` (`user.{uid}.*` private, `public.*` shared, admin/internal anywhere, guests opt-in). In-process `services.pubsub.publish(topic, data)` for backends; frontend `app.pubsub`. See `devplacepy/services/pubsub/CLAUDE.md` |
|
||||
| (none) | seo.py - `/robots.txt`, `/sitemap.xml` |
|
||||
@@ -92,6 +93,10 @@ The `comments` table uses `(target_type, target_uid)` so the same `_comment_sect
|
||||
|
||||
Every post card on the feed has an inline comment form (`.feed-comment-form`) beneath the post actions. It posts to `/comments/create` like the detail page comment form. Tests must scope Post button clicks to `#create-post-modal button.btn-primary:has-text('Post')` to avoid matching the inline comment's Post button.
|
||||
|
||||
### Comment permalinks
|
||||
|
||||
Every comment's action bar (`_comment.html`) includes a "Copy link" button built on the existing `id="comment-{uid}"` anchor (already used by notifications, the profile Activity tab, and `NotificationManager.js`'s scroll-to-highlight) - no new route or URL scheme. It is a plain `data-share="#comment-{{ item.comment['uid'] }}"` button: `DomUtils.initShareButtons` resolves the relative hash against `window.location.href` at click time (so it always copies the exact page the viewer is on, canonical or not), copies it via `navigator.clipboard.writeText`, and flashes "Copied!" on the button (`Toast.flash`) - the same mechanism the gist detail page's Share button already uses.
|
||||
|
||||
### Comment editing
|
||||
|
||||
A comment's owner (only the owner, never an admin) sees an inline "Edit" button (`data-action='edit'`) in `_comment.html`. `CommentManager.toggleEditForm` swaps the `.comment-text` for a textarea seeded from its `data-raw` attribute (the raw markdown, since `contentRenderer.applyTo` overwrites `textContent` on first render), posts via `Http.send` to `POST /comments/edit/{comment_uid}`, then re-renders the new body in place with `contentRenderer.applyTo`. The route (`content.edit_comment_record`) is `is_owner`-only, writes `content` + `updated_at`, records the `comment.edit` audit event, and branches on `wants_json`: JSON clients get `CommentEditOut{uid, content, url, updated_at}`, the no-JS form falls back to a redirect to the comment anchor. Edits are NOT soft-delete related (the body is overwritten in place). Devii tool: `edit_comment` (owner-only, no confirm). Scope test Edit clicks to `.comment-action-btn:has-text('Edit')`.
|
||||
@@ -147,8 +152,9 @@ A dedicated `/gists` page for sharing code snippets. Uses `gists` table (auto-cr
|
||||
|
||||
- Source code rendered in `<pre><code class="language-xxx">` block on detail page
|
||||
- Syntax highlighting handled by existing `highlight.js` loaded globally in `base.html`
|
||||
- Copy button uses `navigator.clipboard.writeText()`
|
||||
- Copy button uses `navigator.clipboard.writeText()` via the shared `data-copy` handler (`DomUtils.initClipboardCopy`)
|
||||
- Cards in listing show language badge, title, truncated description, author, star count
|
||||
- **Raw/rendered toggle for `language == "markdown"`:** unlike `"markdown_rendered"` (always rendered) and every other language (always raw), the plain `"markdown"` gist dual-renders server-side in `gist_detail.html` - the existing raw `<pre>` block plus a `render_content(gist['source_code'], ...)` block, the second one starting `hidden`. A `View rendered`/`View raw` button next to Copy uses the generic `data-view-toggle`/`data-view-toggle-alt` (+ the two `-label`/`-label-alt` pairs) attribute pair (`DomUtils.initViewToggles`, see `static/js/CLAUDE.md`) to swap the `hidden` class between the two blocks and its own label - no fetch, no new endpoint.
|
||||
|
||||
### Sitemap
|
||||
|
||||
@@ -317,6 +323,11 @@ All three columns are populated by `routers/posts.py` `post_page_context()`, the
|
||||
- `POST /bookmarks/{target_type}/{target_uid}` toggles a `bookmarks` row; `GET /bookmarks/saved` renders the personal list (`saved.html`). Target types: `post`, `gist`, `project`, `news`.
|
||||
- `_bookmark_button.html` takes `_type`, `_uid`, `_bookmarked`; `BookmarkManager.js` swaps the label/`bookmarked` class from the JSON `{saved}`. Batch state via `get_user_bookmarks(user_uid, target_type, uids)`.
|
||||
|
||||
### Personal notes
|
||||
- A `notes` row is a private, per-user text annotation on a target (`post`, `gist`, `project`, `news` - the four detail pages, unlike bookmarks' listing-card coverage, since a note is read/written on the full content view, not skimmed from a card). `POST /notes/{target_type}/{target_uid}` (`NoteForm{content}`, max 4000 chars) creates or replaces the caller's own note on that target (revives a soft-deleted row rather than duplicating it, exactly like bookmarks); `POST /notes/{target_type}/{target_uid}/delete` soft-deletes it; `GET /notes/saved` renders the personal notes list (`notes.html`), mirroring `saved.html` but including each note's body.
|
||||
- There is no read/edit access for anyone but the author - the route only ever looks up `user_uid=user["uid"]`, so there is no "someone else's note" to view or moderate. `database/moderation.py` lists `notes` in `UNREPORTABLE_TABLES` ("private to the owner") for that reason.
|
||||
- `_note_button.html` takes `_type`, `_uid`, `_note` (the current content or `None`) and renders a button that opens a small inline textarea editor (not a modal - the body is short and the surrounding action bar has no room for a full dialog); `NoteManager.js` (`app.notes`) wires open/cancel/save/delete and swaps the button label/`has-note` class from the JSON `{content}` / `{deleted}` response, extending the shared `OptimisticAction` base like the other engagement controllers. Batch/single state via `get_user_notes(user_uid, target_type, uids)` (`database/engagement.py`), wired into `content.load_detail`/`detail_context` (post/gist/project) and `routers/news.py`'s detail route as `note_content` on the page context and the matching `*DetailOut` schema.
|
||||
|
||||
### Polls
|
||||
- A poll rides on a post (one `polls` row keyed by `post_uid`, options in `poll_options`, one-per-user votes in `poll_votes`). Created in `posts.py:create_poll` when `poll_question` plus >= 2 non-empty `poll_options` are submitted (capped at 6). Both `create_post` and `edit_post` accept the poll fields; `edit_post` only attaches a poll when the post has **none** yet (it never replaces an existing poll). The builders live in the create-post modal (`feed.html`) and the edit-post modal (`post.html`, rendered only when the post has no poll) using `data-poll-toggle` / `data-poll-add-option`.
|
||||
- `poll_options` accepts either repeated form fields (the web builders, which preserve commas inside an option label) **or** a single newline- or comma-separated string (the API/Devii path). `models.py:normalize_poll_options` splits a lone delimited element - applied as a `mode="before"` validator on `PostForm`/`PostEditForm` - so the documented "one per line or comma separated" agent format actually produces a multi-option poll instead of a single dropped option.
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
# retoor <retoor@molodetz.nl>
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from typing import Annotated
|
||||
@@ -61,11 +62,12 @@ def _targets() -> list[dict]:
|
||||
for key, meta in store.BACKUP_TARGETS.items()
|
||||
]
|
||||
|
||||
def _dashboard(can_download: bool) -> dict:
|
||||
async def _dashboard(can_download: bool) -> dict:
|
||||
backups = [_backup_payload(row, can_download) for row in store.list_backups()]
|
||||
schedules = store.list_schedules()
|
||||
storage = await asyncio.to_thread(store.compute_storage_stats)
|
||||
return {
|
||||
"storage": store.compute_storage_stats(),
|
||||
"storage": storage,
|
||||
"backups": backups,
|
||||
"schedules": schedules,
|
||||
"targets": _targets(),
|
||||
@@ -77,7 +79,7 @@ def _dashboard(can_download: bool) -> dict:
|
||||
@router.get("/backups", response_class=HTMLResponse)
|
||||
async def admin_backups(request: Request):
|
||||
admin = require_admin(request)
|
||||
data = _dashboard(is_primary_admin(admin))
|
||||
data = await _dashboard(is_primary_admin(admin))
|
||||
base = site_url(request)
|
||||
seo_ctx = base_seo_context(
|
||||
request,
|
||||
@@ -107,7 +109,7 @@ async def admin_backups(request: Request):
|
||||
@router.get("/backups/data")
|
||||
async def admin_backups_data(request: Request):
|
||||
admin = require_admin(request)
|
||||
data = _dashboard(is_primary_admin(admin))
|
||||
data = await _dashboard(is_primary_admin(admin))
|
||||
return JSONResponse(BackupDashboardOut.model_validate(data).model_dump(mode="json"))
|
||||
|
||||
@router.post("/backups/run")
|
||||
|
||||
@@ -286,6 +286,7 @@ async def devii_ws(websocket: WebSocket):
|
||||
}
|
||||
)
|
||||
continue
|
||||
svc.maybe_warn_quota_threshold(owner_kind, owner_id, owner_is_admin)
|
||||
session.spawn_turn(text)
|
||||
elif kind == "reset":
|
||||
await session.reset()
|
||||
|
||||
@@ -39,6 +39,7 @@ from devplacepy.services.messaging import (
|
||||
persist_message,
|
||||
redeem_ticket,
|
||||
stamp_content_revision,
|
||||
touch_active_conversation,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -276,6 +277,7 @@ async def send_message(request: Request, data: Annotated[MessageForm, Depends(js
|
||||
data.attachment_uids,
|
||||
request=request,
|
||||
origin="web",
|
||||
client_id=data.client_id,
|
||||
)
|
||||
if message is None:
|
||||
return action_result(request, "/messages")
|
||||
@@ -457,6 +459,7 @@ async def messages_ws(websocket: WebSocket):
|
||||
attachment_uids,
|
||||
request=websocket,
|
||||
origin="websocket",
|
||||
client_id=client_id,
|
||||
)
|
||||
except ContentRefused as exc:
|
||||
await websocket.send_json(
|
||||
@@ -483,6 +486,10 @@ async def messages_ws(websocket: WebSocket):
|
||||
await message_hub.send_to_user(
|
||||
receiver_uid, {"type": "typing", "from_uid": user_uid}
|
||||
)
|
||||
elif kind == "active":
|
||||
with_uid = str(data.get("with_uid", "")).strip()
|
||||
if with_uid:
|
||||
touch_active_conversation(user_uid, with_uid)
|
||||
elif kind == "read":
|
||||
with_uid = str(data.get("with_uid", "")).strip()
|
||||
if with_uid:
|
||||
|
||||
@@ -13,6 +13,7 @@ from devplacepy.database import (
|
||||
get_news_images_by_uids,
|
||||
get_recent_comments_by_target_uids,
|
||||
get_user_bookmarks,
|
||||
get_user_notes,
|
||||
paginate,
|
||||
resolve_object_url,
|
||||
mark_notifications_read_by_target,
|
||||
@@ -125,6 +126,11 @@ async def news_detail_page(request: Request, news_slug: str):
|
||||
bookmarked = bool(user) and article["uid"] in get_user_bookmarks(
|
||||
user["uid"], "news", [article["uid"]]
|
||||
)
|
||||
note_content = (
|
||||
get_user_notes(user["uid"], "news", [article["uid"]]).get(article["uid"])
|
||||
if user
|
||||
else None
|
||||
)
|
||||
|
||||
base = site_url(request)
|
||||
page_url = f"{base}/news/{canonical_slug}"
|
||||
@@ -157,6 +163,7 @@ async def news_detail_page(request: Request, news_slug: str):
|
||||
"time_ago": time_ago(article["synced_at"]),
|
||||
"comments": comments,
|
||||
"bookmarked": bookmarked,
|
||||
"note_content": note_content,
|
||||
"maturity": get_maturity("news", article["uid"])["level"],
|
||||
},
|
||||
model=NewsDetailOut,
|
||||
|
||||
@@ -0,0 +1,188 @@
|
||||
# retoor <retoor@molodetz.nl>
|
||||
import logging
|
||||
from datetime import datetime, timezone
|
||||
from typing import Annotated
|
||||
from fastapi import APIRouter, Form, Request
|
||||
from fastapi.responses import RedirectResponse, JSONResponse, HTMLResponse
|
||||
from devplacepy.database import get_table, db, paginate, resolve_object_url, _now_iso
|
||||
from devplacepy.models import NoteForm
|
||||
from devplacepy.utils import generate_uid, require_user, time_ago, redirect_back
|
||||
from devplacepy.seo import base_seo_context
|
||||
from devplacepy.responses import respond
|
||||
from devplacepy.schemas import NotesOut
|
||||
from devplacepy.services.audit import record as audit
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter()
|
||||
|
||||
NOTABLE: set[str] = {"post", "gist", "project", "news"}
|
||||
|
||||
TABLE_BY_TYPE: dict[str, str] = {
|
||||
"post": "posts",
|
||||
"gist": "gists",
|
||||
"project": "projects",
|
||||
"news": "news",
|
||||
}
|
||||
|
||||
LABEL_BY_TYPE: dict[str, str] = {
|
||||
"post": "Post",
|
||||
"gist": "Gist",
|
||||
"project": "Project",
|
||||
"news": "Article",
|
||||
}
|
||||
|
||||
|
||||
@router.get("/saved", response_class=HTMLResponse)
|
||||
async def notes_page(request: Request, before: str = None):
|
||||
user = require_user(request)
|
||||
notes = get_table("notes")
|
||||
rows, next_cursor = paginate(notes, before=before, user_uid=user["uid"])
|
||||
|
||||
uids_by_type: dict[str, list] = {}
|
||||
for row in rows:
|
||||
uids_by_type.setdefault(row["target_type"], []).append(row["target_uid"])
|
||||
|
||||
resolved: dict[tuple, dict] = {}
|
||||
for target_type, uids in uids_by_type.items():
|
||||
table_name = TABLE_BY_TYPE.get(target_type)
|
||||
if not table_name or table_name not in db.tables:
|
||||
continue
|
||||
table = get_table(table_name)
|
||||
clauses = [table.table.columns.uid.in_(uids)]
|
||||
if table.has_column("deleted_at"):
|
||||
clauses.append(table.table.columns.deleted_at.is_(None))
|
||||
for obj in table.find(*clauses):
|
||||
resolved[(target_type, obj["uid"])] = obj
|
||||
|
||||
items = []
|
||||
for row in rows:
|
||||
obj = resolved.get((row["target_type"], row["target_uid"]))
|
||||
if not obj:
|
||||
continue
|
||||
title = obj.get("title") or (obj.get("content", "") or "")[:80] or "Untitled"
|
||||
items.append(
|
||||
{
|
||||
"target_type": row["target_type"],
|
||||
"type_label": LABEL_BY_TYPE.get(
|
||||
row["target_type"], row["target_type"].title()
|
||||
),
|
||||
"title": title,
|
||||
"url": resolve_object_url(row["target_type"], row["target_uid"]),
|
||||
"content": row["content"],
|
||||
"time_ago": time_ago(row.get("updated_at") or row["created_at"]),
|
||||
"updated_at": row.get("updated_at"),
|
||||
}
|
||||
)
|
||||
|
||||
seo_ctx = base_seo_context(
|
||||
request,
|
||||
title="Notes",
|
||||
description="Your personal notes on DevPlace.",
|
||||
robots="noindex,nofollow",
|
||||
)
|
||||
return respond(
|
||||
request,
|
||||
"notes.html",
|
||||
{
|
||||
**seo_ctx,
|
||||
"request": request,
|
||||
"user": user,
|
||||
"items": items,
|
||||
"next_cursor": next_cursor,
|
||||
},
|
||||
model=NotesOut,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/{target_type}/{target_uid}")
|
||||
async def set_note(
|
||||
request: Request,
|
||||
target_type: str,
|
||||
target_uid: str,
|
||||
data: Annotated[NoteForm, Form()],
|
||||
):
|
||||
user = require_user(request)
|
||||
if target_type not in NOTABLE:
|
||||
return JSONResponse({"error": "Invalid target"}, status_code=400)
|
||||
|
||||
notes = get_table("notes")
|
||||
existing = notes.find_one(
|
||||
user_uid=user["uid"], target_uid=target_uid, target_type=target_type
|
||||
)
|
||||
now = datetime.now(timezone.utc).isoformat()
|
||||
content = data.content.strip()
|
||||
if existing:
|
||||
uid = existing["uid"]
|
||||
notes.update(
|
||||
{
|
||||
"id": existing["id"],
|
||||
"content": content,
|
||||
"updated_at": now,
|
||||
"deleted_at": None,
|
||||
"deleted_by": None,
|
||||
},
|
||||
["id"],
|
||||
)
|
||||
else:
|
||||
uid = generate_uid()
|
||||
notes.insert(
|
||||
{
|
||||
"uid": uid,
|
||||
"user_uid": user["uid"],
|
||||
"target_uid": target_uid,
|
||||
"target_type": target_type,
|
||||
"content": content,
|
||||
"created_at": now,
|
||||
"updated_at": now,
|
||||
"deleted_at": None,
|
||||
"deleted_by": None,
|
||||
}
|
||||
)
|
||||
|
||||
audit.record(
|
||||
request,
|
||||
"note.set",
|
||||
user=user,
|
||||
target_type=target_type,
|
||||
target_uid=target_uid,
|
||||
summary=f"{user['username']} saved a note on {target_type} {target_uid}",
|
||||
links=[audit.target(target_type, target_uid)],
|
||||
)
|
||||
|
||||
if request.headers.get("x-requested-with") == "fetch":
|
||||
return JSONResponse({"uid": uid, "content": content})
|
||||
return RedirectResponse(url=redirect_back(request), status_code=302)
|
||||
|
||||
|
||||
@router.post("/{target_type}/{target_uid}/delete")
|
||||
async def delete_note(request: Request, target_type: str, target_uid: str):
|
||||
user = require_user(request)
|
||||
if target_type not in NOTABLE:
|
||||
return JSONResponse({"error": "Invalid target"}, status_code=400)
|
||||
|
||||
notes = get_table("notes")
|
||||
existing = notes.find_one(
|
||||
user_uid=user["uid"], target_uid=target_uid, target_type=target_type
|
||||
)
|
||||
if existing and not existing.get("deleted_at"):
|
||||
notes.update(
|
||||
{
|
||||
"id": existing["id"],
|
||||
"deleted_at": _now_iso(),
|
||||
"deleted_by": user["uid"],
|
||||
},
|
||||
["id"],
|
||||
)
|
||||
audit.record(
|
||||
request,
|
||||
"note.delete",
|
||||
user=user,
|
||||
target_type=target_type,
|
||||
target_uid=target_uid,
|
||||
summary=f"{user['username']} deleted a note on {target_type} {target_uid}",
|
||||
links=[audit.target(target_type, target_uid)],
|
||||
)
|
||||
|
||||
if request.headers.get("x-requested-with") == "fetch":
|
||||
return JSONResponse({"deleted": True})
|
||||
return RedirectResponse(url=redirect_back(request), status_code=302)
|
||||
@@ -11,7 +11,7 @@ Each project card links to `/projects/{project_uid}` showing full project detail
|
||||
|
||||
**Project overview page.** The detail page is a dedicated project showcase: one encompassing dark card (`.project-shell`, the site `--bg-card` surface with clipped corners) wraps the hero, the section tab bar and the two-column body, and every inner panel (tab bar, sidebar cards, devlog post cards, empty state, comments section) sits one elevation lighter on `--bg-secondary`. The hero's cover banner is the attachment referenced by `projects.cover_attachment_uid`, falling back to the first image attachment (brand-gradient band when neither exists); the title block, type/platform chips and author row render OVERLAID on the banner behind a bottom scrim (dark text-shadow for readability) beside the optional `projects.logo_attachment_uid` tile, with an owner-set **Visit Website** CTA (`projects.website_url`). Cover and logo ride the ONE existing upload pipeline: `dp-upload` widgets (`name="cover_attachment_uid"`/`"logo_attachment_uid"`, `max-files="1"`) in the create/edit modals upload to `/uploads/upload`, the route validates each uid via `database.get_user_attachment` (must exist, belong to the actor, be an image - `_hero_attachment_uid`) and links it to the project through `attachments.link_attachments`; an empty value on edit keeps the current image (no removal control). `website_url`/`repo_url` are normalized by `models.normalize_website_url` (scheme-less input gets `https://`, non-http(s) rejected) and render with `rel="noopener nofollow"`. Below the hero an anchor **section tab bar** (`.project-tabs`, underline style, Overview `.active`) links `#about` / `#devlog` / `#screenshots` (only when gallery images exist) / `#comments` / the Files page - server-rendered anchors, no JS tab state. The main column holds **About** (description + non-image attachments), the **Devlog** (every post whose `project_uid` points at the project via `_post_card.html` - the template loads `feed.css` for the card styles alongside `post.css`, the same rule as `news.html`) with `devlog_count` (`content.count_project_devlog`) and an owner **Post update** button (`.project-devlog-post-btn`) opening the shared composer preset to `topic=devlog` + this project (the form lives ONCE in `templates/_post_composer_form.html`, locals `_composer_topic`/`_composer_project`, included by `feed.html` and `project_detail.html` - never fork a second copy), a **Screenshots** gallery (image attachments minus the cover/logo, thumbnails, `data-lightbox`, capped at 12 rendered), and the comment thread; the sidebar holds Links (website/repository/files/fork source), the Stats card (5 `.project-stat` entries + a last-update line) and the Author card. Owners add gallery images via the More-menu **Add screenshots** modal: `_attachment_form.html` uploads, then `POST /projects/{slug}/screenshots` (`ProjectScreenshotsForm`, owner-only, audit `project.screenshots.add`) links the uids through the same `link_attachments` choke point; Devii action `project_add_screenshots`, docs id `projects-screenshots`. `comment_count`/`devlog_count` ride `ProjectDetailOut`; the new project fields ride `ProjectOut`; the page og:image prefers the cover attachment. **Locator discipline:** the page has several `Files` anchors (action row, tab bar, sidebar) and, for owners, a second hidden `textarea[name='content']`/Post button inside the composer modal - tests MUST scope (`.project-detail-actions a:has-text('Files')`, `.comment-form textarea[name='content']`).
|
||||
|
||||
**Action row overflow.** The detail page has more actions than fit one line, so `project_detail.html` keeps the engagement actions inline (Files, Share, star vote, bookmark, reactions) and collapses the rest behind a single **More** button (`.project-actions-more`) that opens the shared `app.contextMenu`. The secondary actions (Workspace, Containers, Download zip, Fork, the owner Edit/Private/Read-only/Delete controls) live as real elements inside a hidden `.project-actions-overflow` container, each tagged `data-menu-action` plus `data-menu-icon`/`data-menu-label`. `static/js/ProjectActionsMenu.js` builds the menu from those elements and each item's `onSelect` simply `.click()`s the real element, so all existing wiring is reused unchanged - `app.zipDownloader` (`data-zip-download`), `app.projectForker` (`data-fork-project`), `data-share`, the `data-modal` Edit trigger, and the delegated `data-confirm`/`data-confirm-danger` dialog on the owner forms. The open handler must `stopPropagation()` because `app.contextMenu`'s document-level close listener would otherwise dismiss it on the same click (every other caller opens it from a right-click `attach`, not a left-click). Reuse this pattern - a `More` trigger over `[data-menu-action]` real elements - for any future action row that overflows; do not duplicate controller logic into menu callbacks.
|
||||
**Action row overflow.** The detail page has more actions than fit one line, so `project_detail.html` keeps the engagement actions inline (Files, Share, star vote, bookmark, reactions) and collapses the rest behind a single **More** button (`.project-actions-more`) that opens the shared `app.contextMenu`. The secondary actions (Download zip, Fork, the owner Edit/Private/Read-only/Delete controls) live as real elements inside a hidden `.project-actions-overflow` container, each tagged `data-menu-action` plus `data-menu-icon`/`data-menu-label`. `static/js/ProjectActionsMenu.js` builds the menu from those elements and each item's `onSelect` simply `.click()`s the real element, so all existing wiring is reused unchanged - `app.zipDownloader` (`data-zip-download`), `app.projectForker` (`data-fork-project`), `data-share`, the `data-modal` Edit trigger, and the delegated `data-confirm`/`data-confirm-danger` dialog on the owner forms. The open handler must `stopPropagation()` because `app.contextMenu`'s document-level close listener would otherwise dismiss it on the same click (every other caller opens it from a right-click `attach`, not a left-click). Reuse this pattern - a `More` trigger over `[data-menu-action]` real elements - for any future action row that overflows; do not duplicate controller logic into menu callbacks.
|
||||
|
||||
**Owner editing.** Mirrors post editing exactly: an owner-only **Edit** menu item (`data-modal="edit-project-modal"`) opens the `modal()` macro's `edit-project-modal`, a plain `POST` form to `/projects/edit/{slug}` (route `edit_project` in `routers/projects/index.py`, body `ProjectEditForm`, owner-gated through the shared `content.edit_content_item` which returns 403 JSON / redirect for non-owners and stamps `updated_at`). The modal is the create modal pre-filled from the `project` row (title, description, type/status radios pre-checked, dates via `format_date()` back to DD/MM/YYYY). `is_private`/`read_only` are NOT edited here - they stay on their dedicated toggles. The platforms tag widget reuses the create modal's `platforms-input`/`platforms`/`platforms-tags` ids; `ProfileEditor.initPlatformTags` now **seeds existing tags** from the hidden `#platforms` value on load, so both the empty create form and the pre-filled edit form work from the same code. Devii tool `edit_project`; documented in `docs_api.py` (`projects-edit`).
|
||||
|
||||
|
||||
@@ -91,6 +91,49 @@ async def push_register(request: Request) -> JSONResponse:
|
||||
return JSONResponse(payload)
|
||||
|
||||
|
||||
@router.delete("/push.json")
|
||||
async def push_unregister(request: Request) -> JSONResponse:
|
||||
user = require_user_api(request)
|
||||
try:
|
||||
body = await request.json()
|
||||
except ValueError:
|
||||
return JSONResponse({"error": "Invalid JSON"}, status_code=400)
|
||||
|
||||
if not isinstance(body, dict):
|
||||
return JSONResponse({"error": "Invalid request"}, status_code=400)
|
||||
|
||||
provider = providers.get(body.get("provider"))
|
||||
if provider is None:
|
||||
return JSONResponse({"error": "Unknown provider"}, status_code=400)
|
||||
|
||||
identity = {key: body.get(key) for key in ("client_id", "token", "endpoint")}
|
||||
if not any(isinstance(value, str) and value.strip() for value in identity.values()):
|
||||
return JSONResponse({"error": "Invalid request"}, status_code=400)
|
||||
|
||||
removed = push.unregister(user["uid"], provider.name, identity)
|
||||
|
||||
if removed:
|
||||
endpoint = identity.get("endpoint")
|
||||
audit.record(
|
||||
request,
|
||||
"push.unsubscribe",
|
||||
user=user,
|
||||
target_type="user",
|
||||
target_uid=user["uid"],
|
||||
target_label=user.get("username"),
|
||||
metadata={
|
||||
"provider": provider.name,
|
||||
"endpoint_host": urlparse(endpoint).hostname
|
||||
if isinstance(endpoint, str) and endpoint
|
||||
else None,
|
||||
"has_client_id": bool(identity.get("client_id")),
|
||||
},
|
||||
summary=f"{user.get('username')} unregistered a push subscription",
|
||||
links=[audit.target("user", user["uid"], user.get("username"))],
|
||||
)
|
||||
return JSONResponse({"unregistered": removed})
|
||||
|
||||
|
||||
@router.get("/service-worker.js")
|
||||
async def service_worker() -> FileResponse:
|
||||
return FileResponse(
|
||||
|
||||
@@ -15,8 +15,8 @@ async def robots_txt(request: Request):
|
||||
return PlainTextResponse(
|
||||
f"""User-agent: *
|
||||
Disallow: /auth/
|
||||
Disallow: /messages/
|
||||
Disallow: /notifications/
|
||||
Disallow: /messages
|
||||
Disallow: /notifications
|
||||
Disallow: /votes/
|
||||
Disallow: /avatar/
|
||||
Disallow: /follow/
|
||||
@@ -24,6 +24,7 @@ Disallow: /admin/
|
||||
Disallow: /uploads/
|
||||
Disallow: /reports/mine
|
||||
Disallow: /profile/*/delete
|
||||
Disallow: /game
|
||||
Disallow: /*?tab=
|
||||
Disallow: /*?sort=
|
||||
Allow: /static/
|
||||
|
||||
@@ -11,7 +11,7 @@ from devplacepy import database
|
||||
from devplacepy.config import DEEPSEARCH_DIR
|
||||
from devplacepy.models import DeepsearchChatForm, DeepsearchRunForm
|
||||
from devplacepy.responses import respond
|
||||
from devplacepy.schemas import DeepsearchJobOut, DeepsearchSessionOut
|
||||
from devplacepy.schemas import DeepsearchHistoryOut, DeepsearchJobOut, DeepsearchSessionOut
|
||||
from devplacepy.seo import base_seo_context, site_url, web_application_schema, website_schema
|
||||
from devplacepy.services.deepsearch.chat import DeepsearchChat
|
||||
from devplacepy.services.deepsearch.export import to_json, to_markdown, to_pdf
|
||||
@@ -197,6 +197,52 @@ def _enqueue(uid: str, payload: dict, owner_kind: str, owner_id: str, query: str
|
||||
}
|
||||
)
|
||||
|
||||
def _history_item(row: dict) -> dict:
|
||||
uid = row.get("uid", "")
|
||||
status = row.get("status", "")
|
||||
job = queue.get_job(uid)
|
||||
return {
|
||||
"uid": uid,
|
||||
"query": row.get("query"),
|
||||
"status": status,
|
||||
"score": row.get("score"),
|
||||
"confidence": row.get("confidence"),
|
||||
"source_diversity": row.get("source_diversity"),
|
||||
"page_count": int(row.get("page_count") or 0),
|
||||
"chunk_count": int(row.get("chunk_count") or 0),
|
||||
"summary": row.get("summary") or None,
|
||||
"reopen_url": f"/tools/deepsearch/{uid}/session",
|
||||
"chat_available": status == "done" and job is not None,
|
||||
"available": job is not None,
|
||||
"created_at": row.get("created_at"),
|
||||
"completed_at": row.get("completed_at") or None,
|
||||
}
|
||||
|
||||
@router.get("/history")
|
||||
async def deepsearch_history(request: Request, limit: int = 20):
|
||||
owner_kind, owner_id = owner_for(request)
|
||||
user = get_current_user(request)
|
||||
rows = database.list_deepsearch_sessions(owner_kind, owner_id, min(max(1, limit), 100))
|
||||
sessions = [_history_item(row) for row in rows]
|
||||
seo_ctx = base_seo_context(
|
||||
request,
|
||||
title="DeepSearch History",
|
||||
description="Past DeepSearch research runs, with links back to each report and its grounded chat.",
|
||||
robots="noindex,nofollow",
|
||||
breadcrumbs=[
|
||||
{"name": "Home", "url": "/feed"},
|
||||
{"name": "Tools", "url": "/tools"},
|
||||
{"name": "DeepSearch", "url": "/tools/deepsearch"},
|
||||
{"name": "History", "url": "/tools/deepsearch/history"},
|
||||
],
|
||||
)
|
||||
return respond(
|
||||
request,
|
||||
"tools/deepsearch_history.html",
|
||||
{**seo_ctx, "request": request, "user": user, "sessions": sessions},
|
||||
model=DeepsearchHistoryOut,
|
||||
)
|
||||
|
||||
@router.get("/{uid}")
|
||||
async def deepsearch_status(request: Request, uid: str):
|
||||
job = queue.get_job(uid)
|
||||
|
||||
@@ -43,6 +43,8 @@ from devplacepy.schemas.listings import (
|
||||
NewsDetailOut,
|
||||
NewsListItemOut,
|
||||
NewsListOut,
|
||||
NoteItemOut,
|
||||
NotesOut,
|
||||
NotificationGroupOut,
|
||||
NotificationItemOut,
|
||||
NotificationsOut,
|
||||
@@ -88,6 +90,8 @@ from devplacepy.schemas.containers import (
|
||||
)
|
||||
from devplacepy.schemas.jobs import (
|
||||
DbQueryJobOut,
|
||||
DeepsearchHistoryItemOut,
|
||||
DeepsearchHistoryOut,
|
||||
DeepsearchJobOut,
|
||||
DeepsearchSessionOut,
|
||||
ForkJobOut,
|
||||
|
||||
@@ -143,6 +143,27 @@ class DeepsearchSessionOut(_Out):
|
||||
completed_at: Optional[str] = None
|
||||
|
||||
|
||||
class DeepsearchHistoryItemOut(_Out):
|
||||
uid: str = ""
|
||||
query: Optional[str] = None
|
||||
status: str = ""
|
||||
score: Optional[int] = None
|
||||
confidence: Optional[float] = None
|
||||
source_diversity: Optional[float] = None
|
||||
page_count: int = 0
|
||||
chunk_count: int = 0
|
||||
summary: Optional[str] = None
|
||||
reopen_url: Optional[str] = None
|
||||
chat_available: bool = False
|
||||
available: bool = False
|
||||
created_at: Optional[str] = None
|
||||
completed_at: Optional[str] = None
|
||||
|
||||
|
||||
class DeepsearchHistoryOut(_Out):
|
||||
sessions: list = []
|
||||
|
||||
|
||||
class DbQueryJobOut(_Out):
|
||||
uid: str = ""
|
||||
kind: str = ""
|
||||
|
||||
@@ -152,6 +152,7 @@ class PostDetailOut(_Out):
|
||||
attachments: list[AttachmentOut] = []
|
||||
reactions: ReactionsOut = ReactionsOut()
|
||||
bookmarked: bool = False
|
||||
note_content: Optional[str] = None
|
||||
poll: Optional[PollOut] = None
|
||||
war: Optional[WarOut] = None
|
||||
comment_count: Optional[int] = None
|
||||
@@ -187,6 +188,7 @@ class ProjectDetailOut(_Out):
|
||||
attachments: list[AttachmentOut] = []
|
||||
reactions: ReactionsOut = ReactionsOut()
|
||||
bookmarked: bool = False
|
||||
note_content: Optional[str] = None
|
||||
platforms: Optional[Any] = None
|
||||
is_private: bool = False
|
||||
read_only: bool = False
|
||||
@@ -227,6 +229,7 @@ class GistDetailOut(_Out):
|
||||
attachments: list[AttachmentOut] = []
|
||||
reactions: ReactionsOut = ReactionsOut()
|
||||
bookmarked: bool = False
|
||||
note_content: Optional[str] = None
|
||||
|
||||
|
||||
class NewsListOut(_Out):
|
||||
@@ -243,6 +246,7 @@ class NewsDetailOut(_Out):
|
||||
time_ago: Optional[str] = None
|
||||
comments: list[CommentItemOut] = []
|
||||
bookmarked: bool = False
|
||||
note_content: Optional[str] = None
|
||||
|
||||
|
||||
class MessagesOut(_Out):
|
||||
@@ -275,3 +279,19 @@ class SavedOut(_Out):
|
||||
items: list[SavedItemOut] = []
|
||||
next_cursor: Optional[str] = None
|
||||
|
||||
|
||||
class NoteItemOut(_Out):
|
||||
target_type: Optional[str] = None
|
||||
target_uid: Optional[str] = None
|
||||
type_label: Optional[str] = None
|
||||
title: Optional[str] = None
|
||||
url: Optional[str] = None
|
||||
content: Optional[str] = None
|
||||
time_ago: Optional[str] = None
|
||||
updated_at: Optional[str] = None
|
||||
|
||||
|
||||
class NotesOut(_Out):
|
||||
items: list[NoteItemOut] = []
|
||||
next_cursor: Optional[str] = None
|
||||
|
||||
|
||||
@@ -27,9 +27,10 @@ Generic, lightweight, **fire-and-forget** offload for non-critical side-effects
|
||||
- **Choke entrypoint.** `schedule_correction(user, table, uid, request=None)` is the only thing handlers call (every hook passes `request`). It is a no-op unless: a user dict is present, `table` is in the registry, `user["ai_correction_enabled"]` is truthy, and the user has a non-empty `api_key`. In **sync** mode (`user["ai_correction_sync"]`) it calls `_run_inline_awaited`, which - only when a `request.scope` and a running event loop exist - submits `_run_correction` to `loop.run_in_executor(AI_APPLY_EXECUTOR, ...)` (off the loop thread, on the dedicated AI pool) and stashes the future on `request.scope[PENDING_SCOPE_KEY]` for the middleware to await; if there is no request/loop it returns False and falls back to background. In **background** mode (default) it `background.submit`s `_run_correction` (per-worker queue; also runs inline under `DEVPLACE_DISABLE_SERVICES=1`). The same `_run_correction` worker function runs in all cases.
|
||||
- **The hooks (covers UI + REST + Devii + devRant in one place):** `content.create_content_item` (posts/projects/gists), `content.edit_content_item`, `content.create_comment_record`, `content.edit_comment_record`, `services/messaging/persist.persist_message` (DMs - sender is the user), `routers/profile/index.update_profile` (bio), and the two devRant direct-update edit paths (`routers/devrant/rants.edit_rant`, `routers/devrant/comments.edit_comment`). devRant create paths route through the shared content cores, so they are already hooked. Code-only and external paths (project files, Gitea) are deliberately not hooked.
|
||||
- **The gateway call is fail-soft.** `correct_text(api_key, prompt, text)` is synchronous (runs on the background worker thread), POSTs to `INTERNAL_GATEWAY_URL` with `model=correction_model()` (`get_setting("correction_model", "") or INTERNAL_MODEL` - admin-configurable at `/admin/settings`, blank falls back to the gateway default `molodetz`; the gateway URL itself is never configurable per feature, always `INTERNAL_GATEWAY_URL`) via `stealth.stealth_sync_client`, authenticated with the user's own `Bearer` api_key (per-user attribution). It returns the ORIGINAL text on any error, empty output, or suspiciously large output (`len > len(text) * MAX_GROWTH_FACTOR + 200`, rejecting hallucinated expansion). `_run_correction` reads the row back, corrects each registry field that has non-blank text, writes only changed fields via `table.update(updates, ["uid"])` without touching `updated_at` (auto-correction is not a user edit) or the slug (slugs are permanent), and `clear_user_cache(user_uid)` when `table == "users"` so the corrected bio re-caches.
|
||||
- **Structural markdown-preservation guard (issue #84).** The growth-factor check alone never caught a correction that kept a similar length while stripping/rewrapping markdown structure (a flattened list, a dropped code fence, a removed header, a stripped link) - correction is prompt-only ("preserve... markdown") with zero enforcement otherwise. `gateway_complete(..., check_structure=False)` gained an opt-in structural check, wired on ONLY from `correct_text` (`check_structure=True`), that runs right after the growth-factor check: `rendering.markdown_structure_signature(text)` parses the text through a dedicated mistune AST pipeline (`_structure_markdown = mistune.create_markdown(renderer=None, plugins=["strikethrough", "table"])`, matching `_content_markdown`'s plugin set) and counts four structural signals recursively over the token tree - `code_fences` (`block_code`), `list_items` (`list_item`, ordered+unordered together), `headers` (`heading`, all levels together), `links` (`link`). `correction.structure_diverges(original, corrected)` compares the two signatures per key with `STRUCTURE_DROP_TOLERANCE = 1`: a count that drops from >0 to exactly 0 always trips it (total loss of a structural element type is the corruption signature - a single header/link/fence is as real as five), any other decrease bigger than the 1-item tolerance also trips it (catches a 5-item list collapsed to 1, not just to 0), and any increase or a decrease of at most 1 passes (tolerates a single word fixed inside a list item, or two adjacent items reflowed into one, without ever changing the actual count of an unrelated structural family). `gateway_complete` logs and returns the ORIGINAL text on divergence, exactly like the growth-factor rejection - no partial write, no exception. `ai_modifier.py`'s `modify_text` intentionally does NOT set `check_structure` - its whole job is to genuinely restructure content per an explicit `@ai` instruction (e.g. "@ai turn this into a numbered list"), so a structural-divergence check there would reject the very thing the user asked for; distinguishing "restructuring the user asked for" from "restructuring elsewhere in the same field the instruction didn't target" is not reliably automatable from a signature diff alone, so the guard is scoped to `correct_text` only, which never intentionally restructures.
|
||||
- **Per-user usage aggregation (only on success).** `correct_text` returns `(text, usage)`; `usage` is parsed from the gateway's `X-Gateway-*` response headers (`_usage_from_headers`) whenever the upstream call returned 200 (cost was incurred, even if the corrected output was rejected), else `None` on any failure. `_usage_from_headers` captures the token and cost headers PLUS the timing headers `X-Gateway-Upstream-Latency-Ms` and `X-Gateway-Total-Latency-Ms` (as `upstream_latency_ms`/`total_latency_ms`), so each call's timing is metered. `_run_correction` accumulates the per-field `usage` into one `totals` dict and, when `totals["calls"] > 0`, makes ONE call to `database.add_correction_usage(user_uid, totals)` (a single `totals` dict, not positional args) - so a 2-field content item is a single aggregated write, and a failed/empty correction records nothing. `add_correction_usage` (and `add_modifier_usage`) delegate to the shared `database._add_usage(usage_table, user_uid, totals)`: a single atomic `INSERT ... ON CONFLICT(user_uid) DO UPDATE SET col = col + excluded.col` upsert against the `correction_usage` table (per-user running SUMS: `calls`/`prompt_tokens`/`completion_tokens`/`total_tokens`/`cost_usd`/`upstream_latency_ms`/`total_latency_ms`/`updated_at`, unique index `idx_correction_usage_user`, the two latency columns REAL default 0.0, all ensured in `init_db`). It is a derived counter table (NOT in `SOFT_DELETE_TABLES`, like `gateway_usage_ledger`) and is deliberately separate from `users` so accumulating never invalidates the auth/user cache. `database.get_correction_usage(user_uid)` (via `_get_usage`) returns the stored sums PLUS computed averages: `avg_tokens` (total_tokens/calls), `avg_upstream_latency_ms`, `avg_total_latency_ms`, `avg_tokens_per_second` (completion_tokens over total upstream seconds), and `avg_cost_usd`.
|
||||
- **Profile display:** `routers/profile/usage._correction_usage(uid, include_cost)` shapes it via the shared `_usage_view(data, include_cost)` (mirrors `_ai_quota`); `profile/index.py` builds it only for `is_owner or viewer_is_admin` and passes `include_cost=viewer_is_admin`, exposed as the `correction_usage` dict on the context and `ProfileOut`. The view surfaces the sums plus averages - `avg_tokens` (avg tokens/call), `avg_latency_ms` (avg upstream latency), `avg_total_latency_ms`, `avg_tokens_per_second` (avg speed), and `total_time_s` (total upstream seconds) - rendered as extra tiles on the card. **Financial gating:** tokens/call-count and the performance tiles show to the owner and admins; the dollar `cost_usd` and `avg_cost_usd` keys are present ONLY when `viewer_is_admin`, in BOTH the HTML card (`templates/profile.html`, `.correction-usage-*`) and the `respond(..., model=ProfileOut)` JSON (same rule as `_ai_quota`'s `spent_usd` - hiding it in the template alone would leak it to a member fetching their own profile as JSON). The card renders only when `correction_usage.calls` > 0.
|
||||
- **Import-cycle discipline.** `correction.py` imports only `stealth`, `config`, `database.get_table`/`add_correction_usage`, and `services.background.background` at module top; `clear_user_cache` is imported lazily inside `_run_correction`. Never import `content` or `utils` at module top.
|
||||
- **Import-cycle discipline.** `correction.py` imports only `stealth`, `config`, `database.get_table`/`add_correction_usage`, `rendering.markdown_structure_signature` (structural guard above - `rendering.py` has zero internal imports, so this adds no cycle risk), and `services.background.background` at module top; `clear_user_cache` is imported lazily inside `_run_correction`. Never import `content` or `utils` at module top.
|
||||
- **Settings live on `users`:** three columns `ai_correction_enabled` (0/1), `ai_correction_sync` (0/1, default 0 = background), and `ai_correction_prompt` (text, default `config.DEFAULT_CORRECTION_PROMPT`), ensured in `database.backfill_api_keys()` (the user column-ensure block run by `init_db`) and seeded born-live in `utils._create_account`. The edit route is the owner-or-admin leaf `POST /profile/{username}/ai-correction` (`routers/profile/ai_correction.py`, `AiCorrectionForm{enabled, sync, prompt}`, audit key `profile.ai_correction`). The owner-only values are exposed on the profile page context and `ProfileOut` (`ai_correction_enabled`/`ai_correction_sync`/`ai_correction_prompt`, gated by `is_owner`), the UI block lives in `profile.html` (owner-only: enable checkbox, **Apply mode** select, prompt textarea) wired by `static/js/AiCorrection.js` (`app.aiCorrection`), and Devii drives it via the owner-scoped `ai_correction_get`/`ai_correction_set` tools (`services/devii/ai_correction/`, `handler="ai_correction"`, `requires_auth=True`, not confirm-gated - it is a reversible per-user toggle; `ai_correction_set` accepts `enabled`, optional `sync`, optional `prompt`).
|
||||
|
||||
## AI modifier (`services/ai_modifier.py`, `services/ai_context.py`)
|
||||
|
||||
@@ -43,7 +43,13 @@ def modify_text(
|
||||
+ context
|
||||
)
|
||||
return gateway_complete(
|
||||
api_key, system, text, MODIFIER_TIMEOUT_SECONDS, None, model=modifier_model()
|
||||
api_key,
|
||||
system,
|
||||
text,
|
||||
MODIFIER_TIMEOUT_SECONDS,
|
||||
None,
|
||||
model=modifier_model(),
|
||||
bypass_preamble=True,
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -12,14 +12,14 @@ Admin-only, enterprise-grade backups built on the **same async-job pattern as zi
|
||||
## Storage and data model
|
||||
|
||||
- **Storage:** archives go under `config.BACKUPS_DIR` (`data/backups/`, in `DATA_PATHS`) sharded with `attachments._directory_for` on the **random uuid tail** (same load-bearing reason as zips/blobs), named `{target}-{YYYYMMDD-HHMMSS}-{tail}.tar.gz`. Staging is `config.BACKUP_STAGING_DIR` (`data/backup_staging/`), removed in `process` `finally`.
|
||||
- **Data model** (`store.py`, ensured in `database.init_db` via `backup_store.ensure_tables()`): `backups` (NOT soft-deletable - an archive is a reclaimable operational artifact, hard-deleted like zips) and `backup_schedules` (in `SOFT_DELETE_TABLES`, born-live `deleted_at:None`). `store` holds all CRUD plus `compute_storage_stats()` (du of every major data area + `shutil.disk_usage`, run in `asyncio.to_thread` from the route, 30s in-process TTL cache so the walk never blocks).
|
||||
- **Data model** (`store.py`, ensured in `database.init_db` via `backup_store.ensure_tables()`): `backups` (NOT soft-deletable - an archive is a reclaimable operational artifact, hard-deleted like zips) and `backup_schedules` (in `SOFT_DELETE_TABLES`, born-live `deleted_at:None`). `store` holds all CRUD plus two storage helpers that must never be confused: `disk_usage()` is an O(1) `shutil.disk_usage` of `DATA_DIR` (15s TTL) and is the only thing the service tick may call; `compute_storage_stats()` does a **single** `os.walk` of `DATA_DIR` (overlapping path buckets, 600s TTL, in-process lock so callers cannot stampede), and is run in `asyncio.to_thread` from the admin route and the live-view relay. **Never call `compute_storage_stats()` on the event loop.** A 30s cache with a walk that itself takes >=30s never hits, pins one core in `pathlib.rglob` / `is_symlink`, stops `accept()`, and the reverse proxy returns 502. That was a production outage.
|
||||
- **Permanent artifact:** `cleanup(job)` only removes leftover staging, NEVER the archive. Job retention prunes the `jobs` row; the archive and `backups` row persist until an admin deletes it, a schedule rotates it out (`keep_last`), or `devplace backups clear`. Deleting a backup is a HARD delete (unlink file + delete row) - correct because backups are GC artifacts, the documented exception to the soft-delete rule.
|
||||
|
||||
## Remote offload
|
||||
|
||||
`devplacepy/services/backup/offload.py` ships completed archives to a Hetzner Storage Box over WebDAV via `rclone` (`config.RCLONE_BIN`/`config.RCLONE_CONFIG_FILE`, remote name `config.BACKUP_OFFLOAD_REMOTE`, default `storagebox:devplacepy-backups`) - deliberately **not** the `/backup` davfs2 mount, whose FUSE metadata cache lives on the root filesystem and breaks exactly when disk fills (the original outage cause). `BackupService._run_offload_cycle` (throttled to `backup_offload_interval_seconds`, default 300s, via `ConfigField`s in the `Offload` group) runs each cycle after `_fire_due_schedules`:
|
||||
|
||||
1. `upload_pending` - every `done` backup with `remote_uploaded_at` unset and a live `local_path` is `rclone copyto`'d to `<remote>/<target>/<filename>`, then verified by exact byte-size match (`rclone size --json`) against `size_bytes` recorded at finalize time. Only on a verified match does `store.mark_remote_uploaded` set `remote_path`/`remote_uploaded_at`. A failed or unverified upload is silently retried next cycle - `remote_uploaded_at` is the only source of truth for "is this backup actually safe off-box."
|
||||
1. `upload_pending` - every `done` backup with `remote_uploaded_at` unset and a live `local_path` is `rclone copyto`'d to `<remote>/<target>/<filename>`, then verified by exact byte-size match (`rclone size --json`) against `size_bytes` recorded at finalize time. Only on a verified match does `store.mark_remote_uploaded` set `remote_path`/`remote_uploaded_at`. A failed or unverified upload is retried next cycle - `remote_uploaded_at` is the only source of truth for "is this backup actually safe off-box." **Auth failures short-circuit the rest of the pending list** (HTTP 401 / "didn't find section"): one stale `rclone.conf` password must not retry every local archive in the same cycle. The davfs2 mount at `/backup` is a credential oracle for the same WebDAV host; if rclone 401s while that mount still works, the obscured `pass` in `rclone.conf` is stale and must be re-obscured from `/etc/davfs2/secrets`. Do not "fix" 401 by writing through the davfs2 mount.
|
||||
2. `enforce_local_retention` (`backup_offload_keep_local`, default 1) - per target, keeps the newest N **offloaded** local copies and unlinks the rest (`store.mark_local_purged`: clears `local_path`, sets `local_purged_at`, row and `remote_path` persist). A backup with no confirmed remote copy is never touched, no matter how old.
|
||||
3. `enforce_remote_retention` (`backup_offload_keep_remote`, default 30) - per target, `rclone lsjson` the remote dir and `deletefile` anything beyond the newest N, sorted by filename (safe because the `{target}-YYYYMMDD-HHMMSS-*` name is lexicographically chronological, same property `schedule.to_iso` relies on).
|
||||
|
||||
@@ -29,6 +29,10 @@ Admin-only, enterprise-grade backups built on the **same async-job pattern as zi
|
||||
|
||||
**Operational prerequisite (production, not automatic):** the `rclone` binary is installed in the shipped Docker image, but a working WebDAV remote still needs to exist at `config.RCLONE_CONFIG_FILE` (default `$HOME/.config/rclone/rclone.conf` inside the app container, overridable via `DEVPLACE_RCLONE_CONFIG`) with a remote named to match `config.BACKUP_OFFLOAD_REMOTE`'s prefix (default `storagebox`) pointing at the Hetzner Storage Box's WebDAV endpoint and credentials - `rclone config` (interactive) or a hand-written `rclone.conf` generates it. Until that file exists, every `upload_pending` attempt fails fast (`rclone` errors "didn't find section") and is logged and retried next cycle; local retention and rotation both stay disabled the whole time (see above), so backups simply accumulate locally with no data loss, they just never leave the box. **In Docker, `HOME=/app` (the bind-mounted repo root, `docker-compose.yml`), so the default config path resolves to `<repo>/.config/rclone/rclone.conf` on the host - `.gitignore` excludes `/.config/` precisely because this file holds live remote-storage credentials; never force-add it.**
|
||||
|
||||
## Disk usage warning
|
||||
|
||||
`disk_usage()` is the only source of the used/free percentage. `BackupService._check_disk_usage()` and `collect_metrics()` call it (never `compute_storage_stats()`). `disk_warn_percent_field` (`backup_disk_warn_percent`, default 90, group "Alerts") is compared with simple hysteresis - `self.log(...)` fires once when usage reaches the threshold ("Disk usage critical: ...") and once more when it drops back below ("Disk usage back under threshold: ..."), never repeating every tick while the state is unchanged. The warning is visible in this service's own Logs tab (`/admin/services/backup`), which also gets a live "Disk usage" stat card via `collect_metrics()` (`super().collect_metrics()` from `JobService` plus the disk percentage/free space). The admin dashboard's per-directory file counts still come from `compute_storage_stats()`, off-thread. This is intentionally minimal - a log line and a stat card via the existing `BaseService` mechanisms, no new table, route, or notification channel.
|
||||
|
||||
## Schedules
|
||||
|
||||
`backup_schedules` carry `kind` (`interval`|`cron`), `every_seconds`/`cron`, `enabled`, `keep_last`, `next_run_at`, run bookkeeping. `_fire_due_schedules` (lock-owner only, so each fires once) compares `next_run_at <= to_iso(now_utc())` and enqueues a `backup` job + a `backups` record, then advances `next_run_at` via `schedule.next_run`. **Timestamp format is load-bearing:** schedule `next_run_at` uses the devii `schedule.to_iso` format (`%Y-%m-%dT%H:%M:%S`, no tz/micros) on BOTH sides of the comparison so lexicographic compare equals chronological - do not mix it with `datetime.isoformat()`.
|
||||
|
||||
@@ -30,6 +30,16 @@ def _remote_dir(target: str) -> str:
|
||||
return f"{config.BACKUP_OFFLOAD_REMOTE}/{target}"
|
||||
|
||||
|
||||
def _is_auth_error(err: str) -> bool:
|
||||
text = err.lower()
|
||||
return (
|
||||
"401" in text
|
||||
or "unauthorized" in text
|
||||
or "didn't find section" in text
|
||||
or "did not find section" in text
|
||||
)
|
||||
|
||||
|
||||
async def upload_pending(log=lambda message: None) -> int:
|
||||
uploaded = 0
|
||||
for row in store.list_pending_offload():
|
||||
@@ -38,6 +48,12 @@ async def upload_pending(log=lambda message: None) -> int:
|
||||
code, _, err = await _run_rclone("copyto", str(local_path), remote_path)
|
||||
if code != 0:
|
||||
log(f"Offload failed for {row['filename']}: {err.strip()[:300]}")
|
||||
if _is_auth_error(err):
|
||||
log(
|
||||
"Offload halted: remote storage rejected credentials; "
|
||||
"remaining uploads skipped until the next cycle"
|
||||
)
|
||||
break
|
||||
continue
|
||||
size_code, size_out, size_err = await _run_rclone("size", remote_path, "--json")
|
||||
if size_code != 0:
|
||||
|
||||
@@ -27,6 +27,7 @@ WORKER_MODULE = "devplacepy.services.jobs.backup_worker"
|
||||
DEFAULT_OFFLOAD_INTERVAL_SECONDS = 300
|
||||
DEFAULT_OFFLOAD_KEEP_LOCAL = 1
|
||||
DEFAULT_OFFLOAD_KEEP_REMOTE = 30
|
||||
DEFAULT_DISK_WARN_PERCENT = 90
|
||||
|
||||
|
||||
class BackupService(JobService):
|
||||
@@ -41,6 +42,21 @@ class BackupService(JobService):
|
||||
def __init__(self):
|
||||
super().__init__(name="backup", interval_seconds=15)
|
||||
self._last_offload_at = 0.0
|
||||
self._disk_warned = False
|
||||
self.disk_warn_percent_field = ConfigField(
|
||||
"backup_disk_warn_percent",
|
||||
"Disk usage warning threshold (%)",
|
||||
type="int",
|
||||
default=DEFAULT_DISK_WARN_PERCENT,
|
||||
minimum=50,
|
||||
maximum=99,
|
||||
help=(
|
||||
"Log a warning (visible in this service's Logs tab) once the data "
|
||||
"volume's used disk percentage reaches this threshold, and again "
|
||||
"when it drops back below it."
|
||||
),
|
||||
group="Alerts",
|
||||
)
|
||||
self.offload_enabled_field = ConfigField(
|
||||
"backup_offload_enabled",
|
||||
"Offload to remote storage",
|
||||
@@ -82,6 +98,7 @@ class BackupService(JobService):
|
||||
group="Offload",
|
||||
)
|
||||
self.config_fields += [
|
||||
self.disk_warn_percent_field,
|
||||
self.offload_enabled_field,
|
||||
self.offload_interval_field,
|
||||
self.offload_keep_local_field,
|
||||
@@ -95,6 +112,36 @@ class BackupService(JobService):
|
||||
except Exception as exc:
|
||||
self.log(f"Schedule pass failed: {exc}")
|
||||
await self._run_offload_cycle()
|
||||
try:
|
||||
self._check_disk_usage()
|
||||
except Exception as exc:
|
||||
self.log(f"Disk usage check failed: {exc}")
|
||||
|
||||
def _check_disk_usage(self) -> None:
|
||||
threshold = int(self.disk_warn_percent_field.read())
|
||||
used_percent = store.disk_usage()["used_percent"]
|
||||
if used_percent >= threshold:
|
||||
if not self._disk_warned:
|
||||
self._disk_warned = True
|
||||
self.log(
|
||||
f"Disk usage critical: {used_percent}% used on the data "
|
||||
f"volume (warning threshold {threshold}%)"
|
||||
)
|
||||
elif self._disk_warned:
|
||||
self._disk_warned = False
|
||||
self.log(f"Disk usage back under threshold: {used_percent}% used")
|
||||
|
||||
def collect_metrics(self) -> dict:
|
||||
metrics = super().collect_metrics()
|
||||
disk = store.disk_usage()
|
||||
metrics["stats"] = [
|
||||
*metrics.get("stats", []),
|
||||
{
|
||||
"label": "Disk usage",
|
||||
"value": f"{disk['used_percent']}% ({disk['free_human']} free)",
|
||||
},
|
||||
]
|
||||
return metrics
|
||||
|
||||
async def _run_offload_cycle(self) -> None:
|
||||
if not self.offload_enabled_field.read():
|
||||
|
||||
@@ -1,6 +1,9 @@
|
||||
# retoor <retoor@molodetz.nl>
|
||||
|
||||
import os
|
||||
import shutil
|
||||
import stat
|
||||
import threading
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
@@ -33,9 +36,12 @@ STATUS_RUNNING = "running"
|
||||
STATUS_DONE = "done"
|
||||
STATUS_FAILED = "failed"
|
||||
|
||||
STORAGE_CACHE_TTL_SECONDS = 30
|
||||
STORAGE_CACHE_TTL_SECONDS = 600
|
||||
DISK_CACHE_TTL_SECONDS = 15
|
||||
|
||||
_storage_cache: dict = {"at": 0.0, "data": None}
|
||||
_disk_cache: dict = {"at": 0.0, "data": None}
|
||||
_storage_lock = threading.Lock()
|
||||
|
||||
|
||||
def now_iso() -> str:
|
||||
@@ -395,20 +401,57 @@ def delete_schedule(uid: str, deleted_by: str) -> bool:
|
||||
return True
|
||||
|
||||
|
||||
def clear_storage_stats_cache() -> None:
|
||||
_storage_cache["data"] = None
|
||||
_storage_cache["at"] = 0.0
|
||||
_disk_cache["data"] = None
|
||||
_disk_cache["at"] = 0.0
|
||||
|
||||
|
||||
def disk_usage() -> dict:
|
||||
now = time.monotonic()
|
||||
cached = _disk_cache["data"]
|
||||
if cached is not None and (now - _disk_cache["at"]) < DISK_CACHE_TTL_SECONDS:
|
||||
return cached
|
||||
target = config.DATA_DIR
|
||||
probe = target if target.exists() else target.parent
|
||||
usage = shutil.disk_usage(str(probe if probe.exists() else Path("/")))
|
||||
data = {
|
||||
"total_bytes": usage.total,
|
||||
"used_bytes": usage.used,
|
||||
"free_bytes": usage.free,
|
||||
"total_human": human_bytes(usage.total),
|
||||
"used_human": human_bytes(usage.used),
|
||||
"free_human": human_bytes(usage.free),
|
||||
"used_percent": round(usage.used / usage.total * 100, 1) if usage.total else 0.0,
|
||||
}
|
||||
_disk_cache["data"] = data
|
||||
_disk_cache["at"] = now
|
||||
return data
|
||||
|
||||
|
||||
def _path_size(path: Path) -> tuple[int, int]:
|
||||
if not path.exists():
|
||||
try:
|
||||
info = os.lstat(path)
|
||||
except OSError:
|
||||
return 0, 0
|
||||
if stat.S_ISLNK(info.st_mode):
|
||||
return 0, 0
|
||||
if stat.S_ISREG(info.st_mode):
|
||||
return info.st_size, 1
|
||||
if not stat.S_ISDIR(info.st_mode):
|
||||
return 0, 0
|
||||
if path.is_file():
|
||||
return path.stat().st_size, 1
|
||||
total = 0
|
||||
files = 0
|
||||
for entry in path.rglob("*"):
|
||||
try:
|
||||
if entry.is_file() and not entry.is_symlink():
|
||||
total += entry.stat().st_size
|
||||
for root, _dirs, names in os.walk(path, followlinks=False):
|
||||
for name in names:
|
||||
try:
|
||||
info = os.lstat(os.path.join(root, name))
|
||||
except OSError:
|
||||
continue
|
||||
if stat.S_ISREG(info.st_mode):
|
||||
total += info.st_size
|
||||
files += 1
|
||||
except OSError:
|
||||
continue
|
||||
return total, files
|
||||
|
||||
|
||||
@@ -429,17 +472,57 @@ def _storage_paths() -> list[tuple[str, str, Path]]:
|
||||
]
|
||||
|
||||
|
||||
def compute_storage_stats() -> dict:
|
||||
now = time.monotonic()
|
||||
if (
|
||||
_storage_cache["data"] is not None
|
||||
and (now - _storage_cache["at"]) < STORAGE_CACHE_TTL_SECONDS
|
||||
):
|
||||
return _storage_cache["data"]
|
||||
def _inventory() -> tuple[list[dict], int, int]:
|
||||
declared = _storage_paths()
|
||||
sizes = {key: [0, 0] for key, _label, _path in declared}
|
||||
dir_prefixes: list[tuple[str, str]] = []
|
||||
file_exact: dict[str, str] = {}
|
||||
data_root = os.path.realpath(config.DATA_DIR) if config.DATA_DIR.exists() else ""
|
||||
|
||||
for key, _label, path in declared:
|
||||
try:
|
||||
info = os.lstat(path)
|
||||
except OSError:
|
||||
continue
|
||||
if stat.S_ISREG(info.st_mode):
|
||||
file_exact[os.path.realpath(path)] = key
|
||||
sizes[key] = [info.st_size, 1]
|
||||
continue
|
||||
if stat.S_ISDIR(info.st_mode):
|
||||
real = os.path.realpath(path)
|
||||
dir_prefixes.append((key, real.rstrip(os.sep) + os.sep))
|
||||
|
||||
data_size = 0
|
||||
data_files = 0
|
||||
if data_root and os.path.isdir(data_root):
|
||||
for root, _dirs, names in os.walk(data_root, followlinks=False):
|
||||
for name in names:
|
||||
full = os.path.join(root, name)
|
||||
try:
|
||||
info = os.lstat(full)
|
||||
except OSError:
|
||||
continue
|
||||
if not stat.S_ISREG(info.st_mode):
|
||||
continue
|
||||
size = info.st_size
|
||||
data_size += size
|
||||
data_files += 1
|
||||
matched = file_exact.get(full)
|
||||
if matched is not None:
|
||||
sizes[matched] = [size, 1]
|
||||
for key, prefix in dir_prefixes:
|
||||
if full.startswith(prefix):
|
||||
sizes[key][0] += size
|
||||
sizes[key][1] += 1
|
||||
else:
|
||||
for key, _label, path in declared:
|
||||
sizes[key] = list(_path_size(path))
|
||||
data_size += sizes[key][0]
|
||||
data_files += sizes[key][1]
|
||||
|
||||
paths = []
|
||||
for key, label, path in _storage_paths():
|
||||
size, files = _path_size(path)
|
||||
for key, label, path in declared:
|
||||
size, files = sizes[key]
|
||||
paths.append(
|
||||
{
|
||||
"key": key,
|
||||
@@ -451,41 +534,46 @@ def compute_storage_stats() -> dict:
|
||||
"exists": path.exists(),
|
||||
}
|
||||
)
|
||||
return paths, data_size, data_files
|
||||
|
||||
data_size, data_files = _path_size(config.DATA_DIR)
|
||||
backups_size, backups_files = _path_size(config.BACKUPS_DIR)
|
||||
backup_count = len(
|
||||
[b for b in list_backups(limit=100000) if b.get("status") == STATUS_DONE]
|
||||
)
|
||||
usage = shutil.disk_usage(str(config.DATA_DIR))
|
||||
|
||||
data = {
|
||||
"paths": paths,
|
||||
"data_dir": {
|
||||
"path": str(config.DATA_DIR),
|
||||
"size_bytes": data_size,
|
||||
"size_human": human_bytes(data_size),
|
||||
"file_count": data_files,
|
||||
},
|
||||
"backups_total": {
|
||||
"count": backup_count,
|
||||
"size_bytes": backups_size,
|
||||
"size_human": human_bytes(backups_size),
|
||||
"file_count": backups_files,
|
||||
},
|
||||
"disk": {
|
||||
"total_bytes": usage.total,
|
||||
"used_bytes": usage.used,
|
||||
"free_bytes": usage.free,
|
||||
"total_human": human_bytes(usage.total),
|
||||
"used_human": human_bytes(usage.used),
|
||||
"free_human": human_bytes(usage.free),
|
||||
"used_percent": round(usage.used / usage.total * 100, 1)
|
||||
if usage.total
|
||||
else 0.0,
|
||||
},
|
||||
"generated_at": now_iso(),
|
||||
}
|
||||
_storage_cache["data"] = data
|
||||
_storage_cache["at"] = now
|
||||
return data
|
||||
def compute_storage_stats() -> dict:
|
||||
now = time.monotonic()
|
||||
cached = _storage_cache["data"]
|
||||
if cached is not None and (now - _storage_cache["at"]) < STORAGE_CACHE_TTL_SECONDS:
|
||||
return cached
|
||||
with _storage_lock:
|
||||
now = time.monotonic()
|
||||
cached = _storage_cache["data"]
|
||||
if (
|
||||
cached is not None
|
||||
and (now - _storage_cache["at"]) < STORAGE_CACHE_TTL_SECONDS
|
||||
):
|
||||
return cached
|
||||
paths, data_size, data_files = _inventory()
|
||||
backups_entry = next((row for row in paths if row["key"] == "backups"), None)
|
||||
backups_size = backups_entry["size_bytes"] if backups_entry else 0
|
||||
backups_files = backups_entry["file_count"] if backups_entry else 0
|
||||
backup_count = len(
|
||||
[b for b in list_backups(limit=100000) if b.get("status") == STATUS_DONE]
|
||||
)
|
||||
data = {
|
||||
"paths": paths,
|
||||
"data_dir": {
|
||||
"path": str(config.DATA_DIR),
|
||||
"size_bytes": data_size,
|
||||
"size_human": human_bytes(data_size),
|
||||
"file_count": data_files,
|
||||
},
|
||||
"backups_total": {
|
||||
"count": backup_count,
|
||||
"size_bytes": backups_size,
|
||||
"size_human": human_bytes(backups_size),
|
||||
"file_count": backups_files,
|
||||
},
|
||||
"disk": disk_usage(),
|
||||
"generated_at": now_iso(),
|
||||
}
|
||||
_storage_cache["data"] = data
|
||||
_storage_cache["at"] = time.monotonic()
|
||||
return data
|
||||
|
||||
@@ -15,7 +15,7 @@ This file documents the Playwright-driven AI persona fleet. Claude Code auto-loa
|
||||
| `state.py` | `BotState` dataclass + JSON persistence (incl. the per-bot `identity` card) |
|
||||
| `llm.py` | `LLMClient` (parameterized: key/url/model/costs); content generation + quality checks + the `decide`/`generate_identity` decision engine |
|
||||
| `news_fetcher.py` | `NewsFetcher` - TTL-cached article source |
|
||||
| `browser.py` | `BotBrowser` - Playwright wrapper (human-like typing/clicking/scrolling) |
|
||||
| `browser.py` | `BotBrowser` - Playwright wrapper (human-like typing/clicking/scrolling). `fill` types character-by-character for text fields; date/time inputs (`type=date` and the other `NATIVE_FILL_TYPES`) use Playwright's native `locator.fill(value)` because Chromium date widgets ignore keystrokes and a typed `YYYY-MM-DD` never lands, which made every bot signup POST 400. |
|
||||
| `registry.py` | `ArticleRegistry` - flock-based cross-bot article dedupe |
|
||||
| `bot.py` | `DevPlaceBot` - the orchestrator (sessions, action cycle, `run_forever`) |
|
||||
| `service.py` | `BotsService(BaseService)` - fleet manager + metrics |
|
||||
|
||||
@@ -22,6 +22,13 @@ DEFAULT_USER_AGENT = (
|
||||
"(KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36"
|
||||
)
|
||||
DEFAULT_VIEWPORT = {"width": 1280, "height": 900}
|
||||
NATIVE_FILL_TYPES = frozenset(
|
||||
{"date", "datetime-local", "month", "time", "week"}
|
||||
)
|
||||
|
||||
|
||||
def uses_native_fill(input_type: str) -> bool:
|
||||
return (input_type or "").lower() in NATIVE_FILL_TYPES
|
||||
|
||||
|
||||
class BotBrowser:
|
||||
@@ -157,11 +164,15 @@ class BotBrowser:
|
||||
return False
|
||||
await el.click(timeout=3000)
|
||||
await self._idle(0.05, 0.15)
|
||||
await el.fill("")
|
||||
for ch in val:
|
||||
await self._page.keyboard.type(ch, delay=random.randint(20, 60))
|
||||
if ch == " ":
|
||||
await asyncio.sleep(random.uniform(0.02, 0.08))
|
||||
input_type = (await el.get_attribute("type") or "").lower()
|
||||
if uses_native_fill(input_type):
|
||||
await el.fill(val)
|
||||
else:
|
||||
await el.fill("")
|
||||
for ch in val:
|
||||
await self._page.keyboard.type(ch, delay=random.randint(20, 60))
|
||||
if ch == " ":
|
||||
await asyncio.sleep(random.uniform(0.02, 0.08))
|
||||
await self.capture("field input")
|
||||
return True
|
||||
except Exception as e:
|
||||
|
||||
@@ -58,7 +58,7 @@ Every exec passes `-w /app` explicitly (`DockerCliBackend.exec` and the PTY exec
|
||||
|
||||
## HTTP routing surface
|
||||
|
||||
- `/projects/{slug}/containers` - the `routers/projects/containers/` subpackage: `instances.py` (creation/lifecycle/exec/logs/metrics/sync plus the exec websocket), `schedules.py` (cron/interval/once schedules), shared helpers in `_shared.py`. Every instance runs the shared `ppy` image. Discoverable from the project detail page's admin-only **Containers** button (gated by `content.can_view_project_containers` via the `viewer_can_containers` context flag) and from the admin index.
|
||||
- `/projects/{slug}/containers` - the `routers/projects/containers/` subpackage: `instances.py` (creation/lifecycle/exec/logs/metrics/sync plus the exec websocket), `schedules.py` (cron/interval/once schedules), shared helpers in `_shared.py`. Every instance runs the shared `ppy` image. Reachable by direct URL and from the admin index; the project detail page's admin-only **Containers** button was removed (the `viewer_can_containers` context flag and `content.can_view_project_containers` gate remain).
|
||||
- `/admin/containers` - `routers/admin/containers.py`: lists every instance across all projects with inline actions (start/stop/restart/terminal/edit/delete) plus a create modal (project search-select, run-as user search-select, boot language + source editor, restart policy, start-on-boot, env/ports/limits/ingress); `/admin/containers/{uid}` is the per-instance detail page (lifecycle, live logs/metrics, PTY terminal, schedules, ingress, sync, status history) and `/admin/containers/{uid}/edit` edits run-as user, boot language/script/command, restart policy, start-on-boot, and limits.
|
||||
|
||||
**Key reuse rule:** the admin Containers section adds NO lifecycle/logs/exec endpoints of its own - the instance carries its `project_uid`, so the admin detail route resolves the project and its frontend targets the existing `/projects/{slug}/containers/instances/{uid}/...` routes. Add any new instance operation to `routers/projects/containers/instances.py` only; the admin detail page picks it up for free.
|
||||
@@ -67,7 +67,7 @@ Every exec passes `-w /app` explicitly (`DockerCliBackend.exec` and the PTY exec
|
||||
|
||||
Both are discoverable (an earlier version of the per-project page had zero links to it - fixed).
|
||||
|
||||
1. **Per-project manager**: `templates/containers.html` + `static/js/ContainerManager.js` handles instance creation through an app modal form (the `_macros.html` `modal()` macro + `ModalManager` `.visible` toggle); its instance list links out to the shared detail page. Reached from the project detail page's Containers button, and passes breadcrumbs so content clears the fixed nav.
|
||||
1. **Per-project manager**: `templates/containers.html` + `static/js/ContainerManager.js` handles instance creation through an app modal form (the `_macros.html` `modal()` macro + `ModalManager` `.visible` toggle); its instance list links out to the shared detail page. Reachable by direct URL only now (the project detail page's Containers button was removed); it passes breadcrumbs so content clears the fixed nav.
|
||||
2. **Admin Containers section**: `routers/admin/containers.py` (mounted `/admin/containers`, sidebar link in `admin_base.html`, `admin_section="containers"`). `GET /admin/containers` lists every instance via `store.all_instances()` (decorated with project title/slug from one `projects` lookup) in an `.admin-table`. `GET /admin/containers/data` is the poll JSON. `GET /admin/containers/{uid}` renders `templates/containers_instance.html` + `static/js/ContainerInstance.js` - a dedicated detail page (lifecycle, poll logs/metrics, schedules add/delete, ingress, sync, interactive exec over a PTY WebSocket gated on the lock owner).
|
||||
|
||||
All container CSS (`static/css/containers.css`) uses app design tokens (`--bg-card`, `--text-primary`, `--success`/`--danger`/`--warning`, `--radius`) and the shared `.card` recipe.
|
||||
@@ -728,19 +728,19 @@ view is the only trigger, there is no `Forward a Port` command in the palette in
|
||||
required field makes an anonymous request 422 instead of 401 and `tests/api/auth/matrix.py` fails.
|
||||
Give the field a default and validate it inside the handler after `require_user`.
|
||||
|
||||
**The editor opens through one shared partial.** The project detail page renders an inline **Editor**
|
||||
button in `.project-detail-actions` via `templates/_editor_open.html` (`target="_blank"`, plus the
|
||||
`data-editor-*` attributes `EditorLauncher` reads) straight to the code-server proxy
|
||||
`/projects/{slug}/containers/instances/{uid}/code/`, built by `_editor_url` in
|
||||
`routers/projects/index.py` and carried as `workspace_editor_url` on the context and
|
||||
**The editor opens through one shared partial.** `templates/_editor_open.html` renders the **Open
|
||||
editor** link (`target="_blank"`, plus the `data-editor-*` attributes `EditorLauncher` reads) straight
|
||||
to the code-server proxy `/projects/{slug}/containers/instances/{uid}/code/`, built by `_editor_url`
|
||||
in `routers/projects/index.py` and carried as `workspace_editor_url` on the context and
|
||||
`ProjectDetailOut`. It is emitted ONLY when the viewer passes `can_open_workspace` AND
|
||||
`provision.editor_ready(instance)` holds: the workspace exists, is not suspended, is
|
||||
`store.ST_RUNNING`, AND the editor port answers a TCP connect (`api.editor_reachable`, the same
|
||||
`tunnel_target` the proxy dials) - the states the `editor_proxy` route itself refuses (403 suspended,
|
||||
409 not running, 502 no port) plus the boot window in which the container is up but code-server is
|
||||
not yet listening, so the button can never open a dead editor. When there is no ready workspace the
|
||||
button is absent and the Workspace menu item below is the way in (create/start it there). The
|
||||
workspace page's own **Open editor** link opens in a new tab too; keep both in step.
|
||||
not yet listening, so the link can never open a dead editor. The link is rendered by the workspace
|
||||
page's **ready** phase; the project detail page's inline **Editor** button and its overflow
|
||||
**Workspace** item were removed, so the workspace page is the only surface and is reachable by direct
|
||||
URL now.
|
||||
|
||||
**The workspace page renders a PHASE, and the phase is computed once, server-side.**
|
||||
`provision.phase(instance, ready)` is a pure function of `suspended_at`, `desired_state`, `status`
|
||||
@@ -772,16 +772,14 @@ the same `{workspace, editor_url}` shape the page JSON carries. Only the start/s
|
||||
(`data-workspace-action`) go through the manager; tunnels, editor preferences and delete keep
|
||||
their native page-reloading submit, which the e2e tests assert with `wait_for_url`.
|
||||
|
||||
**Member entry point** is the project detail page's overflow menu (`project_detail.html`), gated by
|
||||
the `viewer_can_workspace` context flag (`can_open_workspace(project, user)`, set in
|
||||
`routers/projects/index.py` and declared on `ProjectDetailOut`) - exactly the pattern the admin-only
|
||||
**Containers** item uses with `viewer_can_containers`. `can_open_workspace` folds in the
|
||||
`workspace_enabled` master switch, so the item disappears for everyone while the feature is off and
|
||||
the route's own `_guard` stays the authority. **A workspace surface with no context flag is
|
||||
unreachable**: the whole feature shipped once with routes, Devii tools and docs but no link into
|
||||
`/projects/{slug}/workspace`, so it was reachable only by typing the URL - and then 404'd anyway
|
||||
because `workspace_enabled` defaults to `"0"`. Any new workspace surface needs both the flag on the
|
||||
page that links to it and the setting turned on.
|
||||
**Member entry point** was the project detail page's overflow menu (`project_detail.html`), gated by
|
||||
the `viewer_can_workspace` context flag (`can_open_workspace(project, user)`, still set in
|
||||
`routers/projects/index.py` and declared on `ProjectDetailOut`). That overflow **Workspace** item
|
||||
(and the admin-only **Containers** item, gated the same way by `viewer_can_containers`) was removed,
|
||||
so `/projects/{slug}/workspace` is reached by direct URL until the feature is fully retired.
|
||||
`can_open_workspace` folds in the `workspace_enabled` master switch, and the route's own `_guard`
|
||||
stays the authority. Re-adding any workspace link to the project page must gate it on the context
|
||||
flag again, with the setting turned on.
|
||||
|
||||
**Admin console** is `/admin/workspaces` (`routers/admin/workspaces.py`, `admin_workspaces.html`):
|
||||
list, start/stop, suspend/unsuspend with a required reason, raise/resolve/dismiss flags, per-user
|
||||
|
||||
@@ -13,7 +13,13 @@ from devplacepy.config import (
|
||||
INTERNAL_GATEWAY_URL,
|
||||
INTERNAL_MODEL,
|
||||
)
|
||||
from devplacepy.database import add_correction_usage, get_setting, get_table
|
||||
from devplacepy.database import (
|
||||
add_correction_usage,
|
||||
get_setting,
|
||||
get_table,
|
||||
internal_gateway_key,
|
||||
)
|
||||
from devplacepy.rendering import markdown_structure_signature
|
||||
from devplacepy.services.background import background
|
||||
from devplacepy.services.openai_gateway.usage import parse_usage_headers
|
||||
|
||||
@@ -31,6 +37,7 @@ CORRECTABLE_FIELDS: dict[str, tuple[str, ...]] = {
|
||||
|
||||
CORRECTION_TIMEOUT_SECONDS = 20.0
|
||||
MAX_GROWTH_FACTOR = 3
|
||||
STRUCTURE_DROP_TOLERANCE = 1
|
||||
PENDING_SCOPE_KEY = "devplace_pending_corrections"
|
||||
|
||||
AI_APPLY_EXECUTOR = ThreadPoolExecutor(max_workers=4, thread_name_prefix="ai-apply")
|
||||
@@ -73,6 +80,18 @@ def _usage_from_headers(response_headers) -> dict:
|
||||
}
|
||||
|
||||
|
||||
def structure_diverges(original: str, corrected: str) -> bool:
|
||||
baseline = markdown_structure_signature(original)
|
||||
candidate = markdown_structure_signature(corrected)
|
||||
for key, before in baseline.items():
|
||||
after = candidate.get(key, 0)
|
||||
if before > 0 and after == 0:
|
||||
return True
|
||||
if after < before - STRUCTURE_DROP_TOLERANCE:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def gateway_complete(
|
||||
api_key: str,
|
||||
system: str,
|
||||
@@ -80,6 +99,8 @@ def gateway_complete(
|
||||
timeout: float,
|
||||
max_growth_factor: int | None = None,
|
||||
model: str = INTERNAL_MODEL,
|
||||
bypass_preamble: bool = False,
|
||||
check_structure: bool = False,
|
||||
) -> tuple[str, dict | None]:
|
||||
text = text or ""
|
||||
if not text.strip():
|
||||
@@ -92,12 +113,18 @@ def gateway_complete(
|
||||
],
|
||||
"temperature": 0.1,
|
||||
}
|
||||
if bypass_preamble:
|
||||
payload["bypass_preamble"] = True
|
||||
headers = {
|
||||
"Content-Type": "application/json",
|
||||
"X-App-Reference": "devplace-correction-v-1-0-0",
|
||||
}
|
||||
if api_key:
|
||||
headers["Authorization"] = f"Bearer {api_key}"
|
||||
if bypass_preamble:
|
||||
internal_key = internal_gateway_key()
|
||||
if internal_key:
|
||||
headers["X-Gateway-Internal-Key"] = internal_key
|
||||
try:
|
||||
response = _client().post(
|
||||
INTERNAL_GATEWAY_URL, json=payload, headers=headers, timeout=timeout
|
||||
@@ -119,6 +146,9 @@ def gateway_complete(
|
||||
if max_growth_factor and len(content) > len(text) * max_growth_factor + 200:
|
||||
logger.warning("AI gateway output too large, keeping original")
|
||||
return text, usage
|
||||
if check_structure and structure_diverges(text, content):
|
||||
logger.warning("AI gateway output changed markdown structure, keeping original")
|
||||
return text, usage
|
||||
return content, usage
|
||||
|
||||
|
||||
@@ -141,6 +171,8 @@ def correct_text(api_key: str, prompt: str, text: str) -> tuple[str, dict | None
|
||||
CORRECTION_TIMEOUT_SECONDS,
|
||||
MAX_GROWTH_FACTOR,
|
||||
model=correction_model(),
|
||||
bypass_preamble=True,
|
||||
check_structure=True,
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -96,6 +96,8 @@ A task created as a reminder carries `notify=1`: when it finishes, `session._del
|
||||
|
||||
**Quotas are resettable** (clearing the owner's `devii_usage_ledger` rows): per-user via `POST /admin/users/{uid}/reset-ai-quota` (button on the admin Users page), globally via `POST /admin/ai-quota/reset-guests` and `/reset-all` (buttons on `/admin/ai-usage`), and from the CLI via `devplace devii reset-quota <username> | --guests | --all`.
|
||||
|
||||
**Proactive 80% quota warning.** `DeviiService.maybe_warn_quota_threshold(owner_kind, owner_id, is_admin)` is called right after the interactive quota gate passes (`routers/devii.py`'s `/devii/ws` handler and `services/telegram/bridge.py`'s `_run_turn`, the two turn-spawning chokepoints), so a signed-in user is warned before they hit the 100% block instead of only discovering it once blocked. It is a no-op for guests (no notifications inbox to deliver to), for administrators, and for an unlimited (`0`) cap; otherwise it fires the `ai_quota_warning` notification (via the single `create_notification` funnel, `NOTIFICATION_TYPES`, toggleable like any other type) once the owner's rolling `spent_24h` reaches 80% of their `daily_limit_for`. Dedup is a plain lookback query on the `notifications` table (`type="ai_quota_warning" AND created_at >= now-24h`) rather than a new table/column - a fresh warning can fire again only once the prior one ages out of the rolling 24h window. The message states only proximity, never a dollar figure, matching the "financial data is admin-only" rule above.
|
||||
|
||||
**Financial data is admin-only:** any monetary figure (USD cost, pricing, spend, limit) is restricted to administrators; members and guests see only the **percentage** of their 24h quota used. The owner's admin status is resolved server-side (`is_admin(user)`) and threaded WS -> `hub.get_or_create(is_admin=)` -> `DeviiSession(is_admin=)` -> `Dispatcher(is_admin=)`. The `Action` dataclass has a `requires_admin` flag (`cost_stats` is admin-only USD; the member-safe `usage_quota` tool returns **only** `{used_pct, turns_today, limit_reached}` with no money and is available to everyone). Gating is double: `Catalog.tool_schemas_for(authenticated, is_admin)` never hands an admin-only tool's schema to a non-admin, and the dispatcher independently raises `AuthRequiredError` for any `requires_admin` action a non-admin attempts. `usage_quota`'s data comes from `DeviiSession._quota_snapshot` (ledger `spent_24h`/`turns_24h` over `settings.daily_limit_usd`); for non-admin owners the system prompt also appends a hard rule forbidding any cost disclosure (defense in depth). `GET /devii/usage` mirrors this: it always returns `used_pct`/`turns_today` and adds `spent_24h`/`limit` only for admins. The standalone `devii` CLI runs with `is_admin=True` (the local operator owns the process). The catalog's cost/analytics HTTP tools `ai_usage` (GET `/admin/ai-usage/data`, USD breakdown) and `site_analytics` (GET `/admin/analytics`) are also `requires_admin=True`, so a non-admin session is never offered them and the dispatcher blocks them even if named - the platform 403 is no longer the only guard. The profile page is correctly split too (`_ai_quota(include_cost=viewer_is_admin)` emits dollars only for admins, in both its HTML and JSON forms, so a member fetching their own profile with `Accept: application/json` never sees dollars either).
|
||||
|
||||
## Aggregate analytics (no pagination)
|
||||
|
||||
@@ -22,7 +22,7 @@ ENGAGEMENT_ACTIONS: tuple[Action, ...] = (
|
||||
),
|
||||
body(
|
||||
"value",
|
||||
"Vote value: 1 to upvote, -1 to downvote (re-send to remove).",
|
||||
"Vote value: 1 to upvote, -1 to downvote (re-send to remove), 0 to explicitly retract your vote.",
|
||||
required=True,
|
||||
),
|
||||
),
|
||||
@@ -65,6 +65,45 @@ ENGAGEMENT_ACTIONS: tuple[Action, ...] = (
|
||||
),
|
||||
),
|
||||
),
|
||||
Action(
|
||||
name="list_notes",
|
||||
method="GET",
|
||||
path="/notes/saved",
|
||||
summary="List your personal notes",
|
||||
description="Notes are private annotations only you can ever see.",
|
||||
params=(query("before", "Pagination cursor."),),
|
||||
),
|
||||
Action(
|
||||
name="set_note",
|
||||
method="POST",
|
||||
path="/notes/{target_type}/{target_uid}",
|
||||
summary="Add or update a private note on a target",
|
||||
description="Overwrites any existing note on the target. Only you can ever see this note.",
|
||||
ajax=True,
|
||||
params=(
|
||||
path("target_type", TARGET_TYPE),
|
||||
path(
|
||||
"target_uid",
|
||||
"Uid of the target, copied from a listing response; do not invent it.",
|
||||
),
|
||||
body("content", "Note text, up to 4000 characters.", required=True),
|
||||
),
|
||||
),
|
||||
Action(
|
||||
name="delete_note",
|
||||
method="POST",
|
||||
path="/notes/{target_type}/{target_uid}/delete",
|
||||
summary="Delete your private note from a target",
|
||||
description="Returns {deleted: true}.",
|
||||
ajax=True,
|
||||
params=(
|
||||
path("target_type", TARGET_TYPE),
|
||||
path(
|
||||
"target_uid",
|
||||
"Uid of the target, copied from a listing response; do not invent it.",
|
||||
),
|
||||
),
|
||||
),
|
||||
Action(
|
||||
name="vote_poll",
|
||||
method="POST",
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from ..spec import Action
|
||||
from ._shared import body, path
|
||||
from ._shared import body, path, query
|
||||
|
||||
|
||||
TOOLS_ACTIONS: tuple[Action, ...] = (
|
||||
@@ -108,6 +108,19 @@ TOOLS_ACTIONS: tuple[Action, ...] = (
|
||||
params=(path("uid", "DeepSearch job uid returned by deepsearch."),),
|
||||
requires_auth=False,
|
||||
),
|
||||
Action(
|
||||
name="deepsearch_history",
|
||||
method="GET",
|
||||
path="/tools/deepsearch/history",
|
||||
summary="List the user's past DeepSearch research runs",
|
||||
description=(
|
||||
"Returns the signed-in user's (or guest's) DeepSearch history, newest first, each "
|
||||
"with its query, status, score and a reopen_url. A completed session can be reopened "
|
||||
"with deepsearch_session and its grounded chat continued exactly as when it finished."
|
||||
),
|
||||
params=(query("limit", "Maximum sessions to return (1-100, default 20)."),),
|
||||
requires_auth=False,
|
||||
),
|
||||
Action(
|
||||
name="deepsearch_pause",
|
||||
method="POST",
|
||||
|
||||
@@ -60,7 +60,7 @@ GROUP_LABELS: dict[str, str] = {
|
||||
"profile": "Profile",
|
||||
"messages": "Direct Messages",
|
||||
"notifications": "Notifications (HTTP)",
|
||||
"engagement": "Reactions, Bookmarks, Polls, Follow",
|
||||
"engagement": "Reactions, Bookmarks, Notes, Polls, Follow",
|
||||
"social": "Leaderboard and Social",
|
||||
"issues": "Issue Tracker",
|
||||
"gists": "Gists",
|
||||
|
||||
@@ -21,6 +21,9 @@ logger = logging.getLogger("devii.service")
|
||||
|
||||
INSTANCE_ORIGIN_DEFAULT = f"http://127.0.0.1:{PORT}"
|
||||
|
||||
QUOTA_WARNING_RATIO = 0.8
|
||||
QUOTA_WARNING_NOTIFICATION_TYPE = "ai_quota_warning"
|
||||
|
||||
|
||||
class DeviiService(BaseService):
|
||||
default_enabled = False
|
||||
@@ -421,6 +424,38 @@ class DeviiService(BaseService):
|
||||
limit = self.daily_limit_for(owner_kind, is_admin)
|
||||
return limit > 0 and self.spent_24h(owner_kind, owner_id) >= limit
|
||||
|
||||
def maybe_warn_quota_threshold(
|
||||
self, owner_kind: str, owner_id: str, is_admin: bool = False
|
||||
) -> None:
|
||||
if owner_kind != "user" or is_admin or not owner_id:
|
||||
return
|
||||
limit = self.daily_limit_for(owner_kind, is_admin)
|
||||
if limit <= 0:
|
||||
return
|
||||
spent = self.spent_24h(owner_kind, owner_id)
|
||||
if spent < limit * QUOTA_WARNING_RATIO:
|
||||
return
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from devplacepy.database import get_table
|
||||
from devplacepy.utils import create_notification
|
||||
|
||||
cutoff = (datetime.now(timezone.utc) - timedelta(hours=24)).isoformat()
|
||||
already_warned = get_table("notifications").find_one(
|
||||
user_uid=owner_id,
|
||||
type=QUOTA_WARNING_NOTIFICATION_TYPE,
|
||||
created_at={">=": cutoff},
|
||||
)
|
||||
if already_warned:
|
||||
return
|
||||
create_notification(
|
||||
owner_id,
|
||||
QUOTA_WARNING_NOTIFICATION_TYPE,
|
||||
"You are approaching your daily AI usage limit.",
|
||||
owner_id,
|
||||
"/devii",
|
||||
)
|
||||
|
||||
def reset_quota(self, owner_kind: str, owner_id: str) -> int:
|
||||
return self.hub().ledger.reset(owner_kind, owner_id)
|
||||
|
||||
|
||||
@@ -49,9 +49,9 @@ Forking copies a source project into a brand-new project owned by the forking us
|
||||
|
||||
## SEO Diagnostics tool - SeoService (kind `seo`, `services/jobs/seo/`, `routers/tools/`)
|
||||
|
||||
The public **Tools -> SEO Diagnostics** auditor crawls a URL or sitemap with a headless browser and runs a broad battery of SEO checks, on the **same async-job pattern as zip/fork** plus a live websocket. It is **public (guests included)**; abuse is bounded by the per-IP POST rate limit, a per-owner one-active-job cap, a page cap, and the shared SSRF guard.
|
||||
The public **SEO Diagnostics** auditor crawls a URL or sitemap with a headless browser and runs a broad battery of SEO checks, on the **same async-job pattern as zip/fork** plus a live websocket. It is **public (guests included)**; abuse is bounded by the per-IP POST rate limit, a per-owner one-active-job cap, a page cap, and the shared SSRF guard.
|
||||
|
||||
- **Surface:** a collapsible **Tools** dropdown in `base.html` (desktop center nav + a mobile section, visible to everyone) toggled by `MobileNav.initToolsDropdown`. `GET /tools` lists tools; `GET /tools/seo` is the auditor page (`static/js/SeoDiagnostics.js` -> `app.seoDiagnostics`, instantiated page-side in the template, not in `Application.js`).
|
||||
- **Surface:** reachable by direct URL (`GET /tools` lists tools; `GET /tools/seo` is the auditor page, `static/js/SeoDiagnostics.js` -> `app.seoDiagnostics`, instantiated page-side in the template, not in `Application.js`). The former topnav **Tools** dropdown was removed, so there is currently no navigation entry point.
|
||||
- **Enqueue:** `POST /tools/seo/run` (`routers/tools/seo.py`, body `SeoRunForm{url, mode: url|sitemap, max_pages 1-50}`). Owner is `("user", uid)` or `("guest", X-Real-IP)`. It rejects with `429` if the owner already has a pending/running `seo` job, then enqueues `{url, mode, max_pages, allow_private:False}` and returns `{uid, status_url, ws_url}`.
|
||||
- **`process`** writes the payload to `config.SEO_REPORTS_DIR/{uid}/payload.json`, launches `python -m devplacepy.services.jobs.seo.worker <payload_json> <output_dir>` via `create_subprocess_exec` (high `limit=` so big lines never overflow the StreamReader), reads **NDJSON frames from stdout** line by line (stage/target/progress/page/site_checks/report_ready), forwards each into the in-process **`ProgressHub`** (`services/jobs/seo/progress.py`, uid -> set of `asyncio.Queue`), and on completion loads `output_dir/report.json` as the job result. `cleanup()` clears the hub buffer and removes the report dir.
|
||||
- **Worker** (`worker.py`, subprocess): `crawler.crawl_target` resolves the target (single URL, or sitemap `<loc>` URLs capped at `max_pages`) and fetches `robots.txt`/`sitemap.xml`/`llms.txt` with `httpx`. The audited target host is guarded once with `net_guard.guard_public_url`; candidate URLs **sharing that host are pre-approved** (no redundant per-URL `getaddrinfo` - a transient DNS failure or a self-hosted server resolving its own domain must not blank the whole crawl), and only cross-host sitemap entries are re-guarded. **In sitemap mode the crawler never falls back to auditing the sitemap document itself**: if no page URLs survive it raises a clear error (a stray `or [target]` fallback previously rendered the sitemap XML as one 56k-node "page" with no title/H1). For each page it launches one Playwright navigation: a single `page.evaluate(EXTRACT_SCRIPT)` returns the whole DOM contract (title/metas/canonical/headings/images/links/jsonld/og/twitter/semantic/mixed-content/word-count), an injected `PerformanceObserver` (`add_init_script(INIT_SCRIPT)`) captures LCP/CLS, navigation timing gives TTFB/FCP/transfer/protocol, a mobile-viewport pass measures overflow/tap-targets, a screenshot is saved, and a raw `httpx` GET supplies the SSR HTML for the rendered-vs-server parity check.
|
||||
@@ -77,7 +77,7 @@ The public **Tools -> SEO Diagnostics** auditor crawls a URL or sitemap with a h
|
||||
|
||||
## DeepSearch tool - DeepsearchService (kind `deepsearch`, `services/jobs/deepsearch/`, `services/deepsearch/`, `routers/tools/deepsearch.py`)
|
||||
|
||||
The public **Tools -> DeepSearch** researcher is a multi-agent deep web researcher built on the **same async-job + ProgressHub + 4013-WS pattern as the SEO tool**, plus a per-session vector store and a grounded RAG chat. It is **public (guests included)**; abuse is bounded by the per-IP POST rate limit, a per-owner one-active-job cap, a page cap (1-30), depth cap (1-4), and the shared SSRF guard. Reuse the SEO tool as the template for any new Tools async job.
|
||||
The public **DeepSearch** researcher is a multi-agent deep web researcher built on the **same async-job + ProgressHub + 4013-WS pattern as the SEO tool**, plus a per-session vector store and a grounded RAG chat. It is **public (guests included)**; abuse is bounded by the per-IP POST rate limit, a per-owner one-active-job cap, a page cap (1-30), depth cap (1-4), and the shared SSRF guard. Reuse the SEO tool as the template for any new Tools async job.
|
||||
|
||||
- **Owner helper is shared:** `routers/tools/_shared.py` `owner_for(request)` returns `("user", uid)` or `("guest", X-Real-IP)`; both `seo.py` and `deepsearch.py` import it (do not re-inline the owner derivation).
|
||||
- **Enqueue:** `POST /tools/deepsearch/run` (body `DeepsearchRunForm{query, depth 1-4, max_pages 1-30}`). It rejects with `429` if the owner already has a pending/running `deepsearch` job. It resolves the **logged-in user's `users.api_key`** (guests use `database.internal_gateway_key()`) into the job payload for per-user embedding/LLM spend attribution, generates the uid up front, writes a `deepsearch_sessions` row (`create_deepsearch_session`), enqueues the job carrying `{query, depth, max_pages, api_key, collection}`, and returns `{uid, status_url, ws_url}`. The enqueue uses a local `_enqueue` (not `queue.enqueue`) so the session uid and the job uid match.
|
||||
@@ -92,6 +92,7 @@ The public **Tools -> DeepSearch** researcher is a multi-agent deep web research
|
||||
- **Vector store (`services/deepsearch/store.py`):** `VectorStore` wraps `chromadb.PersistentClient(path=config.DEEPSEARCH_CHROMA_DIR)`, one collection per session (`ds_<uid>`). `Chunk` is the dataclass. `hybrid_search` blends cosine vector similarity with a BM25 keyword score (weights `HYBRID_VECTOR_WEIGHT`/`HYBRID_KEYWORD_WEIGHT`) over the candidate set, with optional metadata `where` filters. `embeddings.py` `embed_texts` calls the gateway embeddings endpoint and **falls back to a deterministic local hashing vector** on any failure (so the tool degrades, never breaks).
|
||||
- **RAG chat (`services/deepsearch/chat.py` + `WS /tools/deepsearch/{uid}/chat`):** a dedicated lightweight loop (NOT the Devii hub), served **only by the service-lock owner** (closes `4013` for fast retry). Answers are grounded ONLY in the session collection via `hybrid_search`, cited inline, rendered client-side via `dp-content`. Turns persist to `deepsearch_messages` and audit `deepsearch.chat`. Frontend component `<dp-deepsearch-chat>` (`static/js/components/AppDeepsearchChat.js`) clones `AppDocsChat`'s framing but uses its own WebSocket to the chat path.
|
||||
- **Status/report/export routes:** `GET /tools/deepsearch/{uid}` (`DeepsearchJobOut`), `GET /tools/deepsearch/{uid}/session` (`respond(..., DeepsearchSessionOut)`, HTML or JSON), `GET /tools/deepsearch/{uid}/export.{md,json,pdf}` (`services/deepsearch/export.py`; PDF via weasyprint). All are **capability URLs** scoped by the unguessable uuid7. **Viewer-flag discipline:** the session schema/context use `viewer_is_admin`/`viewer_owns` (never `is_admin`/`owns`) so a `respond()` context key never shadows a Jinja global (the same class of issue as the issues `/{number}` route). `tests/api/tools/deepsearch/session.py` guards the HTML render.
|
||||
- **History and reopen (`GET /tools/deepsearch/history`).** `deepsearch_sessions` already persists every run's query/status/score/confidence/summary/timestamps independently of the `jobs` table row (unlike the disposable collection + report dir), so the only genuinely missing piece was a listing surface, not new storage. `database.list_deepsearch_sessions(owner_kind, owner_id, limit)` (index `idx_deepsearch_sessions_owner_created` on `(owner_kind, owner_id, created_at)`) backs the owner-scoped, newest-first history via the same `routers/tools/_shared.py` `owner_for(request)` every other DeepSearch route uses - no separate guest-cookie identity, matching the run/status/control routes' IP-based guest scoping. Each item carries a `reopen_url` (`/tools/deepsearch/{uid}/session`), `chat_available` (session status is `done` AND the job row still exists), and `available` (the job row has not yet been swept by the `deepsearch` job kind's 7-day retention - past that point the collection and report are gone even though the `deepsearch_sessions` row survives, so the history item shows "Expired" instead of a dead link). **Reopening a completed session's chat needed no backend change**: the `WS /tools/deepsearch/{uid}/chat` guard already accepts `job.status == queue.DONE OR session.status == "done"`, so a session reached fresh from the history list (not from the just-finished live-progress flow) renders `chat_ws_url` on `GET .../session` exactly the same way. Schema `DeepsearchHistoryOut`/`DeepsearchHistoryItemOut`; Devii tool `deepsearch_history` (public, mirrors `isslop_list`); docs `tools-deepsearch-history`.
|
||||
- **Completion race (load-bearing read-path fix).** The worker writes `report.json` to disk and `service.process` publishes the `done` frame **from inside `process()`**, but the `JobService` framework only commits `jobs.result`/`status=DONE` afterwards, in `_reap()` -> `_finish_done()` on a later tick. The frontend navigates to the session page the instant it receives `done`, so a read that keyed only off `jobs.status == DONE` returned an EMPTY report (`None` score, 0 sources) until a manual refresh. Fix: `_report_for(uid, job)` returns `job.result.report` when the job is `DONE` and non-empty, else falls back to the on-disk `report.json` (`_report_from_disk`, `DEEPSEARCH_DIR/{uid}/report.json`) - which exists before the `done` frame is ever sent - and returns `{}` only for a `FAILED` job or a genuinely still-running job with no report on disk. `_session_context` derives `done`/`status` from `bool(report)` (not raw job status), and the chat WS gate accepts `session.status == "done"` (set inside `process()` before the publish) as ready. `_export_report` reuses the same fallback. Regression: `tests/api/tools/deepsearch/session.py::test_session_reads_disk_report_before_result_commit`. **Any new read of a job result that a client reaches immediately after a `done`/`session_url` frame must use this same on-disk fallback, never bare `jobs.status`.**
|
||||
- **Clickable inline citations (`services/deepsearch/citations.py`).** The report/findings carry `[n]` markers (and the model sometimes emits `[3][9][1-2]`); the `link_citations(html, source_count)` template global (registered in `templating.py`) rewrites each `[n]` and each `[a-b]` range into `<a class="ds-cite" href="#ds-source-n">[n]</a>` anchors that jump to the numbered `<li id="ds-source-n">` in the Sources list (source numbering is page order, matching the `[n]` the summarizer was given). It splits out `<a>`/`<code>`/`<pre>` regions first so markers inside links/code are left alone, expands ranges to individual links, and drops out-of-range numbers (no broken anchors). The session template nests it over the server render: `{{ link_citations(render_content(summary), sources|length) }}` and `{{ link_citations(finding.detail|e, sources|length) }}`, plus a per-finding `.ds-finding-cites` chip row from `finding.citations`. `.ds-cite`/`.ds-sources li:target` styling lives in `deepsearch.css`. The report prompt asks for one number per bracket (never a range) so output is consistent, but the linkifier handles ranges regardless. Regression: `tests/unit/services/deepsearch/citations.py`.
|
||||
- **Tables (`deepsearch_sessions`, `deepsearch_messages` soft-deletable + in `SOFT_DELETE_TABLES`; `deepsearch_url_cache` GC-only):** columns are ensured in `init_db()` (every queried column) with indexes. Every insert writes `deleted_at:None/deleted_by:None`; every read filters `deleted_at IS NULL`.
|
||||
@@ -105,7 +106,7 @@ The public **Tools -> DeepSearch** researcher is a multi-agent deep web research
|
||||
|
||||
## AI Usage Analyzer tool - IsslopService (kind `isslop`, `services/jobs/isslop/`, `routers/tools/isslop.py`)
|
||||
|
||||
The public **Tools -> AI Usage Analyzer** classifies a git repository or website as AI slop, sophisticated AI-assisted work or genuine human work. It is built on the standard async-job pattern (a `JobService` running a subprocess worker), but its live channel is **pub/sub, not a dedicated WS route**: every worker event is published to `public.isslop.{uid}` AND persisted to `isslop_events`, and the frontend pairs the pub/sub subscription with an incremental `GET /tools/isslop/{uid}/events?after=SEQ` poll, so guests (who cannot subscribe to `public.*` unless `pubsub_allow_guests` is on) and reconnecting tabs replay from the durable trail. **Never rely on pub/sub alone for this tool: the DB event trail is the source of truth, pub/sub is the fast path.**
|
||||
The public **AI Usage Analyzer** classifies a git repository or website as AI slop, sophisticated AI-assisted work or genuine human work. It is built on the standard async-job pattern (a `JobService` running a subprocess worker), but its live channel is **pub/sub, not a dedicated WS route**: every worker event is published to `public.isslop.{uid}` AND persisted to `isslop_events`, and the frontend pairs the pub/sub subscription with an incremental `GET /tools/isslop/{uid}/events?after=SEQ` poll, so guests (who cannot subscribe to `public.*` unless `pubsub_allow_guests` is on) and reconnecting tabs replay from the durable trail. **Never rely on pub/sub alone for this tool: the DB event trail is the source of truth, pub/sub is the fast path.**
|
||||
|
||||
- **Engine layout:** `services/jobs/isslop/` holds `acquisition/` (git probe via `git ls-remote`, depth-1 clone with size preflight + live 3 GB kill guard, stealth Playwright website crawler with HTTP fallback, path-traversal-safe workspace helpers), `analysis/` (exclusion rules, stylometric metrics, language detection, per-repo baselines, `signals/` with one detector family per file, two-axis scoring), `agent/` (gateway LLM client, per-file classifier, vision reviewer, report writer with deterministic fallback), plus `pipeline.py` (the event-yielding run), `worker.py` (subprocess entry), `events.py` (frame protocol), `persistence.py` (`EventPersister` writes events/file results/image results/report and stamps the analysis row), `store.py` (all DB access), `badge.py` (SVG), `service.py` (`IsslopService`), `config.py` (all constants + `WorkerSettings`).
|
||||
- **Worker contract:** `IsslopService.process` writes the worker payload (url + admin toggles + gateway endpoint/model/key) to `config.ISSLOP_RUNS_DIR/{uid}/payload.json`, resolves the workspace under `config.ISSLOP_WORKSPACES_DIR` (`workspace_for` rejects any path escaping the root), launches `python -m devplacepy.services.jobs.isslop.worker <payload> <workspace>`, and relays each NDJSON stdout line through `EventPersister.apply` (SQLite) then `pubsub.publish`. The workspace and run dir are removed in a `finally`; the pipeline also deletes the workspace itself as its final act, so **no acquired source survives an analysis** - only the report and its evidence rows.
|
||||
|
||||
@@ -173,6 +173,7 @@ class SeoMetaService(JobService):
|
||||
source_text,
|
||||
GENERATION_TIMEOUT_SECONDS,
|
||||
model=model,
|
||||
bypass_preamble=True,
|
||||
)
|
||||
if usage:
|
||||
for key in totals:
|
||||
|
||||
@@ -113,7 +113,7 @@ async def _ai_usage(match: re.Match) -> dict:
|
||||
async def _backups(_match: re.Match) -> dict:
|
||||
from devplacepy.routers.admin.backups import _dashboard
|
||||
|
||||
return _dashboard(can_download=False)
|
||||
return await _dashboard(can_download=False)
|
||||
|
||||
|
||||
async def _workspace_detail(match: re.Match) -> Optional[dict]:
|
||||
|
||||
@@ -45,7 +45,13 @@ Because the WS accepts on every worker, a message persisted on worker A must sti
|
||||
|
||||
## DRY persist choke point
|
||||
|
||||
`services/messaging/persist.py` `persist_message(sender, receiver_uid, content, attachment_uids, *, request=None, origin)` is the ONE function that inserts the row, links attachments, fires `create_notification` + `clear_messages_cache` + `create_mention_notifications`, logs, and writes the `message.send` audit event. Both the WS `send` handler and the HTTP `POST /messages/send` handler call it, so audit/notification/mention behavior is byte-identical on both paths (the Devii `send_message` action is `handler="http"` -> `POST /messages/send`, so it also flows through here and broadcasts live). Content is capped at 2000 chars server-side; `content` itself is allowed to be empty (`MessageForm.content` is `min_length=0`) as long as at least one attachment is present - `persist_message` is the single source of truth for that rule (`if not content and not attachment_uids: return None`), so the HTTP form model deliberately does not duplicate it. Whenever `request` is not `None` it audits via `audit.record(request, ...)` - this covers BOTH the HTTP `Request` and the WS path, since the WS handler passes `request=websocket` and a `WebSocket` object is just as non-`None` as a `Request` (`audit.record` never reads HTTP-specific attributes off it beyond what's already supplied explicitly via `user=sender`). `audit.record_system(..., actor_kind="user", origin=origin)` is the fallback used ONLY when `persist_message` is called with no request/websocket context at all (e.g. a future internal/system-originated send) - same `message.send` key/category either way, no new event invented; typing/read are ephemeral and NOT audited.
|
||||
`services/messaging/persist.py` `persist_message(sender, receiver_uid, content, attachment_uids, *, request=None, origin, client_id=None)` is the ONE function that inserts the row, links attachments, fires `create_notification` + `clear_messages_cache` + `create_mention_notifications`, logs, and writes the `message.send` audit event. Both the WS `send` handler and the HTTP `POST /messages/send` handler call it, so audit/notification/mention behavior is byte-identical on both paths (the Devii `send_message` action is `handler="http"` -> `POST /messages/send`, so it also flows through here and broadcasts live). Content is capped at 2000 chars server-side; `content` itself is allowed to be empty (`MessageForm.content` is `min_length=0`) as long as at least one attachment is present - `persist_message` is the single source of truth for that rule (`if not content and not attachment_uids: return None`), so the HTTP form model deliberately does not duplicate it. Whenever `request` is not `None` it audits via `audit.record(request, ...)` - this covers BOTH the HTTP `Request` and the WS path, since the WS handler passes `request=websocket` and a `WebSocket` object is just as non-`None` as a `Request` (`audit.record` never reads HTTP-specific attributes off it beyond what's already supplied explicitly via `user=sender`). `audit.record_system(..., actor_kind="user", origin=origin)` is the fallback used ONLY when `persist_message` is called with no request/websocket context at all (e.g. a future internal/system-originated send) - same `message.send` key/category either way, no new event invented; typing/read are ephemeral and NOT audited.
|
||||
|
||||
### Send dedupe (double-submit safety)
|
||||
|
||||
`persist_message` dedupes on `(sender_uid, client_id)` **before** it ever inserts a row: `_find_recent_duplicate(sender_uid, client_id)` looks up the most recent `messages` row with the same `sender_uid` and the same client-supplied `client_id` whose `created_at` falls inside `DEDUPE_WINDOW_SECONDS` (30s). A hit short-circuits `persist_message` and returns that ALREADY-persisted row verbatim (no error, no second insert, no second AI correction/notification/audit) - both the WS `send` handler and `POST /messages/send` now pass their `client_id` straight through, so a network-level retry that resends the identical payload (same `client_id`) collapses onto the original row instead of producing a second, independently-AI-corrected message. `client_id` is attacker-controlled but the dedupe key is always scoped by the server-resolved `sender_uid`, so a client can only dedupe its own sends. The column is nullable (`messages.client_id`, ensured in `init_db()`, indexed by `idx_messages_dedupe (sender_uid, client_id)`); a request with no `client_id` (any future non-`dp-chat` caller) skips the check entirely and always inserts. This is defense-in-depth alongside the client-side send-lock below - the lock stops a human double-click from ever firing two requests, the server dedupe stops a retried *identical* request (same `client_id`) from ever becoming two rows.
|
||||
|
||||
**Client-side send-lock (`AppChat.js`).** `_initComposer`'s submit handler and the Enter-to-send path both funnel through one `form submit` listener, which now bails out while `this._sendLocked` is true. `_sendViaSocket` sets the lock (and disables the send button via the existing `_refreshSendButton()`, reusing the `.is-sending` spinner state) for every genuine new send (never for a `.failed`-bubble retry, tracked separately so a retry never blocks a fresh message) and only releases it once that specific `client_id` clears out of `_pendingSends` - on the echoed reconcile (`_reconcilePending` -> `_clearPendingSend`) or the 20s send timeout (`_failPendingSend`). Multiple sends never race the lock: `_lockingSends` is a `Set` of in-flight new-send `client_id`s, and the composer re-enables only when it is empty.
|
||||
|
||||
## AI correction and AI modifier apply to direct messages, with LIVE delivery of the final content
|
||||
|
||||
@@ -53,7 +59,16 @@ Because the WS accepts on every worker, a message persisted on worker A must sti
|
||||
|
||||
## WS protocol frames
|
||||
|
||||
Client -> server: `{type:"send", receiver_uid, content, attachment_uids?, client_id}`, `{type:"typing", receiver_uid}` (throttled client-side), `{type:"read", with_uid}`, `{type:"sync", since, with_uid?}` (replay created-or-revised rows after `since`), `{type:"ping"}` (keepalive, ignored). Server -> client: `{type:"ready", user_uid}` (sent first on connect), `{type:"message", uid, sender_uid, sender_username, sender_role, receiver_uid, content, created_at, time_ago, client_id, attachments, ai_processed, ai_pending}` (broadcast to BOTH sender and receiver sockets so multi-tab and the sender's own optimistic bubble reconcile via the echoed `client_id`; a first frame may set `ai_pending` while a sync job runs, and a later frame for the same `uid` sets `ai_processed` with the rewritten body), `{type:"typing", from_uid}` (to the receiver only), `{type:"read", by_uid}` (read-receipt to the other user), `{type:"error", client_id, text}` (sender only, on a dropped send, e.g. the recipient blocked the sender, or a screened body). There is no `presence` frame on this socket - see below.
|
||||
Client -> server: `{type:"send", receiver_uid, content, attachment_uids?, client_id}`, `{type:"typing", receiver_uid}` (throttled client-side), `{type:"read", with_uid}`, `{type:"active", with_uid}` (marks the sender as currently viewing that conversation - see "Active-conversation notification suppression" below), `{type:"sync", since, with_uid?}` (replay created-or-revised rows after `since`), `{type:"ping"}` (keepalive, ignored). Server -> client: `{type:"ready", user_uid}` (sent first on connect), `{type:"message", uid, sender_uid, sender_username, sender_role, receiver_uid, content, created_at, time_ago, client_id, attachments, ai_processed, ai_pending}` (broadcast to BOTH sender and receiver sockets so multi-tab and the sender's own optimistic bubble reconcile via the echoed `client_id`; a first frame may set `ai_pending` while a sync job runs, and a later frame for the same `uid` sets `ai_processed` with the rewritten body), `{type:"typing", from_uid}` (to the receiver only), `{type:"read", by_uid}` (read-receipt to the other user), `{type:"error", client_id, text}` (sender only, on a dropped send, e.g. the recipient blocked the sender, or a screened body). There is no `presence` frame on this socket - see below. `active` has no server -> client echo; it only writes the DB-backed marker consumed server-side by `persist_message`.
|
||||
|
||||
## Active-conversation notification suppression
|
||||
|
||||
A `message` notification (and its live toast) is suppressed when the receiver is demonstrably looking at that exact conversation right now, so an open DM thread never also produces a redundant bell/toast for the same message. The message itself still delivers live over the WS/relay path exactly as before - only the `create_notification("message", ...)` call in `persist_message` is skipped.
|
||||
|
||||
- **Marker: two columns on `users`**, `active_conversation_uid` (the uid of the conversation partner currently being viewed) and `active_conversation_at` (UTC ISO timestamp of the last time it was refreshed), ensured in `database.backfill_api_keys()` like every other non-signup `users` column. This follows the `services/presence.py` `last_seen` pattern deliberately - a WS connection is per-worker (`message_hub` has no cross-worker visibility), but the sender and receiver of a DM can be on different uvicorn workers, so the "is the receiver looking at this" check has to be readable from ANY worker. A DB column is the only cross-worker-correct medium here, exactly as presence is.
|
||||
- **Write path**: `services/messaging/active_conversation.py` `touch_active_conversation(viewer_uid, with_uid)`, called from the WS handler's new `{type:"active", with_uid}` frame. Throttled per-worker exactly like `presence.touch` (`WRITE_THROTTLE_SECONDS = 5`, an in-memory `dict[f"{viewer_uid}:{with_uid}" -> monotonic]` so a burst of focus/visibility events costs at most one write per 5s per conversation) before the `UPDATE users SET active_conversation_uid=..., active_conversation_at=...` write.
|
||||
- **Client trigger (`AppChat.js`)**: `_sendActiveMarker()` sends the frame whenever `_threadIsActive()` is true (desktop: any open thread; mobile: only when the thread pane, not the list pane, is showing). It fires wherever the existing `_markRead()` already fires (initial WS connect, opening/switching a conversation) plus two new listeners bound in `_bindActiveConversation()`: `window` `focus` and `document` `visibilitychange` (`visibilityState === "visible"`) - so returning to a backgrounded/blurred tab that still has the conversation open re-asserts the marker.
|
||||
- **Read path / freshness window**: `is_actively_viewing(viewer_uid, other_uid)` (same module) reads the receiver's user row, checks `active_conversation_uid == other_uid`, and requires `active_conversation_at` to be less than `FRESH_SECONDS = 20` old. `persist_message` calls it as `is_actively_viewing(receiver_uid, sender_uid)` right before the `create_notification` call and skips only that call on a hit; `clear_messages_cache(receiver_uid)` still runs unconditionally, and the WS/relay delivery of the message frame is completely unaffected (it is broadcast from `routers/messages.py`, not from `persist_message`). The freshness window (20s) is intentionally wider than the write throttle (5s) so it never lapses between two consecutive throttled writes, and the marker is a soft "recently confirmed as active" signal, not a hard connection state - if it goes stale (tab closed, thread switched away without a fresh trigger) the very next message just resumes notifying normally, which is the safe failure direction.
|
||||
|
||||
## Presence is NOT part of the messaging WS
|
||||
|
||||
@@ -65,7 +80,9 @@ Live/echoed message bubbles are NEVER injected as raw HTML. `AppChat._buildBubbl
|
||||
|
||||
## Frontend
|
||||
|
||||
The messaging frontend is the single self-booting custom element `<dp-chat>` (`static/js/components/AppChat.js`), which replaced the old page-controller trio `MessagesLayout.js`/`MessagesSocket.js`/`MessageSearch.js` (all deleted) and `messages.css` (superseded by `static/css/chat.css`) - see `devplacepy/static/js/CLAUDE.md` for the component roster entry. `MobileNav.js` does **not** own the mobile pane (that duplicate was removed; `dp-chat` is the only pane controller). `templates/messages.html` renders `<dp-chat mode="page" self-uid="..." with-uid="..." conversations-url="/messages/conversations" search-url="/messages/search" send-url="/messages/send" ws-url="/messages/ws" ai-indicator="true">` wrapping the SAME server-rendered `.messages-list`/`.messages-main`/`.messages-thread`/`.message-bubble` markup as before (no-JS/crawler fallback), which the component *adopts* on `connectedCallback` rather than discarding. It owns a `ChatSocket` (`static/js/chat/ChatSocket.js`, exponential backoff, `4013` fast-path, 25s ping), sends over WS with an optimistic pending bubble keyed by `client_id`, reconciles on the echoed `message` frame, applies later `ai_processed` frames in place (compare `dp-content[data-source]`, never rendered `textContent`), injects the Report control on every live incoming bubble, catches up with `{type:"sync"}` on reconnect, and switches conversations without dropping the socket (`GET /messages?with_uid=` JSON + `history.pushState`). Older history loads via `GET /messages?with_uid=&before=` when the thread is scrolled to the top. Mobile: CSS hides the inactive pane from `with-uid` / `.show-list` before JS runs (no stacked FOUC); the composer does not auto-focus on coarse pointers; keyboard inset uses `visualViewport.offsetTop + height`; Enter-to-send is desktop-only. The send button is disabled only while `dp-upload` is busy, never while a send is in flight. Presence in `mode="page"` stays on the page-global `PresenceManager`. An opt-in `ai-indicator="true"` attribute shows "Adjusting..." while `ai_pending` and "Adjusted by AI" when the revision lands. Styling is in `static/css/chat.css` using `variables.css` tokens only, responsive down to 360px.
|
||||
The messaging frontend is the single self-booting custom element `<dp-chat>` (`static/js/components/AppChat.js`), which replaced the old page-controller trio `MessagesLayout.js`/`MessagesSocket.js`/`MessageSearch.js` (all deleted) and `messages.css` (superseded by `static/css/chat.css`) - see `devplacepy/static/js/CLAUDE.md` for the component roster entry. `MobileNav.js` does **not** own the mobile pane (that duplicate was removed; `dp-chat` is the only pane controller). `templates/messages.html` renders `<dp-chat mode="page" self-uid="..." with-uid="..." conversations-url="/messages/conversations" search-url="/messages/search" send-url="/messages/send" ws-url="/messages/ws" ai-indicator="true">` wrapping the SAME server-rendered `.messages-list`/`.messages-main`/`.messages-thread`/`.message-bubble` markup as before (no-JS/crawler fallback), which the component *adopts* on `connectedCallback` rather than discarding. It owns a `ChatSocket` (`static/js/chat/ChatSocket.js`, exponential backoff, `4013` fast-path, 25s ping), sends over WS with an optimistic pending bubble keyed by `client_id`, reconciles on the echoed `message` frame, applies later `ai_processed` frames in place (compare `dp-content[data-source]`, never rendered `textContent`), injects the Report control on every live incoming bubble, catches up with `{type:"sync"}` on reconnect, and switches conversations without dropping the socket (`GET /messages?with_uid=` JSON + `history.pushState`). Older history loads via `GET /messages?with_uid=&before=` when the thread is scrolled to the top. Mobile: CSS hides the inactive pane from `with-uid` / `.show-list` before JS runs (no stacked FOUC); the composer does not auto-focus on coarse pointers; keyboard inset uses `visualViewport.offsetTop + height`; Enter-to-send is desktop-only. **The send button is disabled while `dp-upload` is busy AND while a genuine new send is in flight** (the send-lock, see "Send dedupe" above) - a `.failed`-bubble retry never engages it, so retrying an old message never blocks typing/sending a new one. Presence in `mode="page"` stays on the page-global `PresenceManager`. An opt-in `ai-indicator="true"` attribute shows "Adjusting..." while `ai_pending` and "Adjusted by AI" when the revision lands. Styling is in `static/css/chat.css` using `variables.css` tokens only, responsive down to 360px.
|
||||
|
||||
**Auto-scroll never fights a manual scroll-up.** `_startScrollWatcher`/`_stabilizeScroll` force `scrollTop = scrollHeight` on every thread mutation (new bubble, attachment/image finishing loading) ONLY while `_userAtBottom` is true, re-checked every `requestAnimationFrame` up to `STABILIZE_MAX_FRAMES`. The one thing that used to break this: while a multi-frame stabilize loop was in flight (`_stabilizePending`), the real `scroll` event listener (`_bindAutoScroll`) ignored EVERY scroll event outright, including a genuine user wheel/touch scroll-up that happened to land inside that window - so a rapid burst of incoming messages (each restarting the stabilize loop while attachments were still loading) could silently overwrite a user's attempt to scroll up to read history. The fix: every programmatic scroll (`_scrollThreadToEnd`, `_stabilizeScroll`) records the exact `scrollTop` it just set in `_lastForcedScrollTop`; the `scroll` listener only skips its own echo (`_stabilizePending && scrollTop === _lastForcedScrollTop`) and always processes a scroll that lands anywhere else, so a real user scroll-up is detected and `_userAtBottom` flips to `false` immediately, even mid-stabilization. Never gate scroll-intent detection on `_stabilizePending` alone again.
|
||||
|
||||
## Attachments stream live
|
||||
|
||||
|
||||
@@ -1,5 +1,9 @@
|
||||
# retoor <retoor@molodetz.nl>
|
||||
|
||||
from devplacepy.services.messaging.active_conversation import (
|
||||
is_actively_viewing,
|
||||
touch_active_conversation,
|
||||
)
|
||||
from devplacepy.services.messaging.hub import message_hub
|
||||
from devplacepy.services.messaging.persist import (
|
||||
message_frame,
|
||||
@@ -11,6 +15,7 @@ from devplacepy.services.messaging.relay import message_relay
|
||||
from devplacepy.services.messaging.tickets import issue_ticket, redeem_ticket
|
||||
|
||||
__all__ = [
|
||||
"is_actively_viewing",
|
||||
"issue_ticket",
|
||||
"message_frame",
|
||||
"message_hub",
|
||||
@@ -19,4 +24,5 @@ __all__ = [
|
||||
"push_content_revision",
|
||||
"redeem_ticket",
|
||||
"stamp_content_revision",
|
||||
"touch_active_conversation",
|
||||
]
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
# retoor <retoor@molodetz.nl>
|
||||
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
from typing import Optional
|
||||
|
||||
from devplacepy.database import get_table
|
||||
|
||||
WRITE_THROTTLE_SECONDS = 5
|
||||
FRESH_SECONDS = 20
|
||||
|
||||
_last_write: dict[str, float] = {}
|
||||
|
||||
|
||||
def touch_active_conversation(viewer_uid: str, with_uid: str) -> None:
|
||||
if not viewer_uid or not with_uid:
|
||||
return
|
||||
now = time.monotonic()
|
||||
key = f"{viewer_uid}:{with_uid}"
|
||||
if now - _last_write.get(key, 0.0) < WRITE_THROTTLE_SECONDS:
|
||||
return
|
||||
_last_write[key] = now
|
||||
get_table("users").update(
|
||||
{
|
||||
"uid": viewer_uid,
|
||||
"active_conversation_uid": with_uid,
|
||||
"active_conversation_at": datetime.now(timezone.utc).isoformat(),
|
||||
},
|
||||
["uid"],
|
||||
)
|
||||
|
||||
|
||||
def _seconds_since(stamped: Optional[str]) -> Optional[float]:
|
||||
if not stamped:
|
||||
return None
|
||||
try:
|
||||
seen = datetime.fromisoformat(stamped)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
if seen.tzinfo is None:
|
||||
seen = seen.replace(tzinfo=timezone.utc)
|
||||
return (datetime.now(timezone.utc) - seen).total_seconds()
|
||||
|
||||
|
||||
def is_actively_viewing(viewer_uid: str, other_uid: str) -> bool:
|
||||
if not viewer_uid or not other_uid:
|
||||
return False
|
||||
row = get_table("users").find_one(uid=viewer_uid)
|
||||
if not row or row.get("active_conversation_uid") != other_uid:
|
||||
return False
|
||||
elapsed = _seconds_since(row.get("active_conversation_at"))
|
||||
return elapsed is not None and elapsed < FRESH_SECONDS
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from datetime import datetime, timezone
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Any, Optional
|
||||
|
||||
from devplacepy.attachments import get_attachments, link_attachments
|
||||
@@ -18,6 +18,7 @@ from devplacepy.utils import (
|
||||
from devplacepy.services.audit import record as audit
|
||||
from devplacepy.services.correction import schedule_correction
|
||||
from devplacepy.services.ai_modifier import schedule_modification
|
||||
from devplacepy.services.messaging.active_conversation import is_actively_viewing
|
||||
from devplacepy.services.moderation.screening import (
|
||||
record as record_screening,
|
||||
refuse_if_blocked,
|
||||
@@ -27,6 +28,7 @@ from devplacepy.services.moderation.screening import (
|
||||
logger = logging.getLogger("messaging.persist")
|
||||
|
||||
MAX_CONTENT_LENGTH = 2000
|
||||
DEDUPE_WINDOW_SECONDS = 30
|
||||
|
||||
|
||||
def _slim_attachment(attachment: dict[str, Any]) -> dict[str, Any]:
|
||||
@@ -64,6 +66,18 @@ def message_frame(
|
||||
}
|
||||
|
||||
|
||||
def _find_recent_duplicate(sender_uid: str, client_id: str) -> Optional[dict[str, Any]]:
|
||||
cutoff = (
|
||||
datetime.now(timezone.utc) - timedelta(seconds=DEDUPE_WINDOW_SECONDS)
|
||||
).isoformat()
|
||||
row = get_table("messages").find_one(
|
||||
sender_uid=sender_uid,
|
||||
client_id=client_id,
|
||||
created_at={">=": cutoff},
|
||||
)
|
||||
return dict(row) if row else None
|
||||
|
||||
|
||||
def persist_message(
|
||||
sender: dict[str, Any],
|
||||
receiver_uid: str,
|
||||
@@ -72,23 +86,30 @@ def persist_message(
|
||||
*,
|
||||
request: Any = None,
|
||||
origin: str = "web",
|
||||
client_id: Optional[str] = None,
|
||||
) -> Optional[dict[str, Any]]:
|
||||
content = (content or "").strip()[:MAX_CONTENT_LENGTH]
|
||||
attachment_uids = attachment_uids or []
|
||||
if not content and not attachment_uids:
|
||||
return None
|
||||
|
||||
sender_uid = sender["uid"]
|
||||
client_id = (client_id or "").strip()[:64] or None
|
||||
if client_id:
|
||||
duplicate = _find_recent_duplicate(sender_uid, client_id)
|
||||
if duplicate is not None:
|
||||
return duplicate
|
||||
|
||||
receiver = get_table("users").find_one(uid=receiver_uid)
|
||||
if not receiver:
|
||||
return None
|
||||
|
||||
if sender["uid"] in get_blocked_uids(receiver_uid):
|
||||
if sender_uid in get_blocked_uids(receiver_uid):
|
||||
return None
|
||||
|
||||
screening = screen_fields("messages", {"content": content})
|
||||
refuse_if_blocked(screening)
|
||||
|
||||
sender_uid = sender["uid"]
|
||||
sender_username = sender.get("username", "")
|
||||
messages_table = get_table("messages")
|
||||
msg_uid = generate_uid()
|
||||
@@ -102,6 +123,7 @@ def persist_message(
|
||||
"read": False,
|
||||
"created_at": created_at,
|
||||
"updated_at": None,
|
||||
"client_id": client_id,
|
||||
}
|
||||
)
|
||||
|
||||
@@ -117,13 +139,14 @@ def persist_message(
|
||||
schedule_modification(sender, "messages", msg_uid, request)
|
||||
|
||||
if sender_uid != receiver_uid:
|
||||
create_notification(
|
||||
receiver_uid,
|
||||
"message",
|
||||
f"{sender_username} sent you a message",
|
||||
sender_uid,
|
||||
f"/messages?with_uid={sender_uid}",
|
||||
)
|
||||
if not is_actively_viewing(receiver_uid, sender_uid):
|
||||
create_notification(
|
||||
receiver_uid,
|
||||
"message",
|
||||
f"{sender_username} sent you a message",
|
||||
sender_uid,
|
||||
f"/messages?with_uid={sender_uid}",
|
||||
)
|
||||
clear_messages_cache(receiver_uid)
|
||||
|
||||
create_mention_notifications(
|
||||
@@ -182,6 +205,7 @@ def persist_message(
|
||||
"read": False,
|
||||
"created_at": created_at,
|
||||
"updated_at": None,
|
||||
"client_id": client_id,
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -42,6 +42,16 @@ Callers may send an optional `X-App-Reference` header to tag gateway calls by ap
|
||||
|
||||
**Per-user attribution:** the gateway also accepts a real user's `api_key` (Bearer / X-API-KEY / session), gated by `gateway_allow_users` (default **on**); `resolve_owner()` records `(owner_kind, owner_id)` per call, so usage is attributed and limitable per user. With `gateway_allow_users` off, only internal/access/admin keys work and per-user attribution never happens. Devii operating a signed-in user authenticates its LLM calls with that user's own `api_key` (set as the session's `ai_key` in `build_settings(..., owner_kind="user")`), so a user's full gateway spend (Devii and direct API calls) rolls up under their uid - surfaced admin-only on the profile page via `build_user_usage()` / `GET /admin/users/{uid}/ai-usage`. Guests keep the internal key.
|
||||
|
||||
### Failed-auth throttle (`services/openai_gateway/auth_throttle.py`)
|
||||
|
||||
A dedicated per-IP throttle protects `/openai/v1/*` against unauthenticated probing WITHOUT IP-allowlisting the endpoint (which would break legitimate external API consumers using `gateway_access_key`/`gateway_allow_users`). It is scoped ONLY to failed authentication attempts, never to general traffic - the existing global rate limiter already exempts `/openai` entirely (`main.py`), and this throttle does not change that exemption.
|
||||
|
||||
- **Config (`gateway_auth_throttle_enabled` default on, `gateway_auth_throttle_max_failures` default 10, `gateway_auth_throttle_window_seconds` default 60, all group "Access").** An admin can retune or disable the throttle with no code change, same as every other gateway setting.
|
||||
- **Counter.** `auth_throttle.py` is a small in-process sliding-window store (`dict[ip] -> list[float]` timestamps), the same shape as the global rate limiter's `_rate_limit_store` in `main.py`, but a separate, dedicated store local to the gateway rather than reusing that one (the two throttles key on different events - all mutating traffic vs. failed gateway auth - and `/openai` is deliberately exempt from the general limiter). `record_failure(ip, window)` appends a timestamp and returns the running in-window count; `is_throttled(ip, threshold, window)` checks without recording. Per-worker, like every other in-process cache in this codebase - not divided by worker count, so the effective aggregate threshold across `N` workers is up to `N x max_failures`; this is an accepted tradeoff (it still closes the probe, just at a slightly higher aggregate bound) rather than importing `main.WEB_WORKERS` into a leaf service module.
|
||||
- **`GatewayService.authorize()` records a failure on every deny path** (a cheap static-key/internal-key match still short-circuits first and is never counted), so a wrong or garbage key counts exactly like presenting nothing. **The fast 429 short-circuit fires only when the request carries no credential at all** (no `X-API-KEY`, no `Authorization` header of any scheme, no `session` cookie) AND the IP is already over threshold - this is the "skip the rest of `authorize()`" defense against wasted CPU on a blind flood. A request presenting ANY credential - even one that turns out invalid - always runs the full check, so it can never be blocked by another caller's failures sharing its IP (a NAT/proxy caller with its own valid key is never punished for a neighbor's failed probes); only a request presenting nothing is fast-denied once its IP has tripped.
|
||||
- **Audit.** Tripping the throttle (the exact request whose failure brings the window count to the configured threshold) records `ai.gateway_auth_throttle.tripped` (category `ai`, `result="denied"`, metadata `ip`/`failed_attempts`/`threshold`/`window_seconds`) via `audit.record(request, ...)`. Every subsequent fast-denied request in the same trip is silent (no audit spam under an actual flood) - the trip itself is the visible signal for admins.
|
||||
- **Internal traffic cannot trip this by construction.** Every internal consumer (Devii guests, AI correction/modifier, quiz grading, SEO metadata generation, news, bots) authenticates with `database.internal_gateway_key()`, which is the exact same value `authorize()` compares against as `gateway_internal_key` - that comparison is the FIRST check in `authorize()`, before the throttle is even consulted, so internal self-dial calls always succeed there and never reach the failure-recording branch.
|
||||
|
||||
## Config fields
|
||||
|
||||
All config is `config_fields` (upstream url/model/key, force-model, the Prompt group's `gateway_system_preamble` and `gateway_thinking` (default off), timeout, instances, vision url/model/key/cache/toggle, the Embeddings group (`gateway_embed_enabled`/`_url`/`_model`/`_key`), the auth toggles + static key + internal key, plus the Pricing/Reliability/Tracking groups); live metrics (requests/errors/in-flight/vision-calls/embed-calls/latency plus 24h cost/tokens/success rollups) via `collect_metrics`.
|
||||
@@ -57,6 +67,15 @@ Two behaviors:
|
||||
|
||||
Embeddings/passthrough are untouched by this composition step.
|
||||
|
||||
### Per-call operator preamble bypass (`bypass_preamble`, internal-only)
|
||||
|
||||
A structured/deterministic internal caller (one that expects a strict, machine-parsed output shape - grading JSON, a corrected text field, generated SEO metadata) can ask the gateway to skip `gateway_system_preamble` for that one call, since an operator preamble written for conversational tone can conflict with a task that demands exact output discipline. The date line and the client's own system content are unaffected either way.
|
||||
|
||||
- **Signal.** The chat request body carries `"bypass_preamble": true`. `GatewayRuntime.handle_chat` pops it off `body` before it is ever copied into the upstream `payload` (so it never leaks to the upstream provider) and only honors it when the caller also proved internal credentials for that same request (see below); otherwise it is silently discarded, exactly like an ordinary, unsupported field.
|
||||
- **Internal-only gate.** `GatewayService.internal_bypass_allowed(request, cfg)` requires the request to carry the `X-Gateway-Internal-Key` header equal to the configured `gateway_internal_key` setting. This is a SEPARATE header from `Authorization`/`X-API-KEY` on purpose: a caller that authenticates with a real user's own `api_key` (correction, the `@ai` modifier, quiz free-text grading - all per-user-attributed on purpose, for correct spend attribution) still proves it is DevPlace's own server code, not the user's browser or a malicious client holding that same api_key, by additionally presenting the internal key on this header. A caller that authenticates AS the internal key directly (SEO metadata generation, via `database.internal_gateway_key()`) can present the same header too - one mechanism covers both shapes of internal caller. `GatewayService.handle()` computes this boolean once per request and passes it into `handle_chat` as `bypass_allowed`; a request with no valid internal credentials gets `bypass_allowed=False` and the body field is ignored even if present - it can never come from an external API consumer or a public-facing client call.
|
||||
- **Callers that set it.** The four structured/deterministic internal callers all route through the single shared `services/correction.py::gateway_complete(..., bypass_preamble=True)` helper, which sets both the body field and the `X-Gateway-Internal-Key` header (via `database.internal_gateway_key()`) in one place: `services/correction.py::correct_text` (AI content correction), `services/ai_modifier.py::modify_text` (the `@ai` inline modifier), `services/quiz/grading.py::grade_free_text` (quiz free-text grading), and `services/jobs/seo_meta_service.py`'s generation call (SEO metadata). **Genuinely conversational internal callers are deliberately left unchanged** and keep receiving the operator preamble: Devii chat sessions and the docs chat do not call `gateway_complete` at all, so they are unaffected by construction.
|
||||
- **Other internal self-dial callers found but NOT updated** (each builds its own request instead of using `gateway_complete`, and each is a candidate for the same treatment in a follow-up): `services/gitea/enhance.py::enhance_ticket` and `services/gitea/planning.py::generate_plan` (both structured, single-shot ticket text generation), `services/dbapi/nl2sql.py::design_query` (NL-to-SQL, strictly structured), `services/deepsearch/llm.py`/`services/jobs/deepsearch/enhance.py` (planner/synthesis calls), and `services/jobs/isslop/agent/llm.py` (per-file classification). None of these were in scope for this change.
|
||||
|
||||
## Thinking default (fast path)
|
||||
|
||||
`thinking.py` (`apply_thinking`) runs in `handle_chat` after `stream_options` is stripped, and in the vision describe call. DeepSeek V4 **enables thinking by default**; leaving the payload alone is the slow path. The gateway therefore always writes an explicit thinking field in the upstream dialect:
|
||||
@@ -127,7 +146,7 @@ The human-facing **Quota rules** tab of `/admin/gateway` is a third, independent
|
||||
|
||||
## Embeddings
|
||||
|
||||
`POST /openai/v1/embeddings` exposes an OpenAI-compatible text-embeddings model. Clients send the generic model `molodetz~embed` (`config.INTERNAL_EMBED_MODEL`), which `handle_embeddings` remaps to `gateway_embed_model` exactly like chat remaps `molodetz` -> `gateway_model` (also remapped when `gateway_force_model` is on or the model is empty). It defaults to OpenRouter's `qwen/qwen3-embedding-8b` at `https://openrouter.ai/api/v1/embeddings` (`config.EMBED_*_DEFAULT`, $0.01 per 1M input tokens). `handle_embeddings` mirrors `handle_chat` but is simpler: no vision augmentation and no streaming - build the payload, forward via `_send`, and record one ledger row through the same `finalize(...)` closure. The config fields are the **Embeddings** group (`gateway_embed_enabled` default on, `gateway_embed_url`, `gateway_embed_model`, `gateway_embed_key`) plus the Pricing-group `gateway_embed_price_input_per_m`. `effective_config()` falls the embed key back to `gateway_vision_key` then `OPENROUTER_API_KEY` (NOT `gateway_api_key`: that is the DeepSeek chat upstream key, whereas embeddings target OpenRouter like vision does). Usage is recorded with **`backend="embed"`**; `usage.compute_cost` adds an `embed` branch (input-only, completion always 0, native OpenRouter `cost` still preferred) and `Pricing` gained `embed_input_per_m`. `analytics.py` groups by `backend` generically, so embed rows roll up automatically; `caching_savings` counts only **non-native** chat rows (native-priced rows did not use the configured cache-hit/miss rates, so folding them in would report a fictional saving). When `gateway_embed_enabled` is off the endpoint returns 503 with no ledger row.
|
||||
`POST /openai/v1/embeddings` exposes an OpenAI-compatible text-embeddings model. Clients send the generic model `molodetz~embed` (`config.INTERNAL_EMBED_MODEL`), which `handle_embeddings` remaps to `gateway_embed_model` exactly like chat remaps `molodetz` -> `gateway_model` (also remapped when `gateway_force_model` is on or the model is empty). It defaults to OpenRouter's `qwen/qwen3-embedding-8b` at `https://openrouter.ai/api/v1/embeddings` (`config.EMBED_*_DEFAULT`, $0.01 per 1M input tokens). `handle_embeddings` mirrors `handle_chat` but is simpler: no vision augmentation and no streaming - build the payload, forward via `_send`, and record one ledger row through the same `finalize(...)` closure. The config fields are the **Embeddings** group (`gateway_embed_enabled` default on, `gateway_embed_url`, `gateway_embed_model`, `gateway_embed_key`) plus the Pricing-group `gateway_embed_price_input_per_m`. `effective_config()` falls the embed key back to `gateway_vision_key` then `OPENROUTER_API_KEY` (NOT `gateway_api_key`: that is the DeepSeek chat upstream key, whereas embeddings target OpenRouter like vision does). Usage is recorded with **`backend="embed"`**; `usage.compute_cost` adds an `embed` branch (input-only, completion always 0, native OpenRouter `cost` still preferred) and `Pricing` gained `embed_input_per_m`. `analytics.py` groups by `backend` generically, so embed rows roll up automatically; `caching_savings` counts only **non-native** chat rows (native-priced rows did not use the configured cache-hit/miss rates, so folding them in would report a fictional saving). When `gateway_embed_enabled` is off the endpoint returns 503 with no ledger row. **When the resolved `gateway_embed_key` is still blank after the `effective_config()`/route-overlay merge** (all of `gateway_embed_key`, `gateway_vision_key`, and `OPENROUTER_API_KEY` are unset, or a per-route provider was configured with no `api_key` of its own), `handle_embeddings` returns the same `503` shape (`{"error": {"message": "Embeddings are not configured", "type": "embeddings_not_configured"}}`) with an `ai.gateway.call` audit row (`result="denied"`, `summary="no embeddings key configured"`) instead of forwarding an unauthenticated request upstream - mirrors the `gateway_embed_enabled` disabled-check immediately above it, no ledger row either. `routing.embed_overlay` only overlays `gateway_embed_key` when the matched route's provider actually has a non-blank `api_key`, so a misconfigured route can never downgrade an already-good fallback key to empty.
|
||||
|
||||
## Single point of truth for AI
|
||||
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
# retoor <retoor@molodetz.nl>
|
||||
|
||||
import time
|
||||
from collections import defaultdict
|
||||
|
||||
_failures: dict[str, list[float]] = defaultdict(list)
|
||||
_last_sweep = 0.0
|
||||
SWEEP_INTERVAL_SECONDS = 60.0
|
||||
|
||||
|
||||
def _window_slice(ip: str, window_seconds: int, now: float) -> list[float]:
|
||||
window_start = now - window_seconds
|
||||
timestamps = [t for t in _failures.get(ip, ()) if t > window_start]
|
||||
if timestamps:
|
||||
_failures[ip] = timestamps
|
||||
elif ip in _failures:
|
||||
del _failures[ip]
|
||||
return timestamps
|
||||
|
||||
|
||||
def _sweep(window_start: float) -> None:
|
||||
stale = [
|
||||
ip
|
||||
for ip, timestamps in _failures.items()
|
||||
if not timestamps or timestamps[-1] <= window_start
|
||||
]
|
||||
for ip in stale:
|
||||
del _failures[ip]
|
||||
|
||||
|
||||
def is_throttled(ip: str, threshold: int, window_seconds: int) -> bool:
|
||||
now = time.time()
|
||||
return len(_window_slice(ip, window_seconds, now)) >= threshold
|
||||
|
||||
|
||||
def record_failure(ip: str, window_seconds: int) -> int:
|
||||
global _last_sweep
|
||||
now = time.time()
|
||||
window_start = now - window_seconds
|
||||
if now - _last_sweep >= SWEEP_INTERVAL_SECONDS:
|
||||
_sweep(window_start)
|
||||
_last_sweep = now
|
||||
timestamps = _window_slice(ip, window_seconds, now)
|
||||
timestamps.append(now)
|
||||
_failures[ip] = timestamps
|
||||
return len(timestamps)
|
||||
|
||||
|
||||
def reset(ip: str) -> None:
|
||||
_failures.pop(ip, None)
|
||||
|
||||
|
||||
def clear() -> None:
|
||||
_failures.clear()
|
||||
global _last_sweep
|
||||
_last_sweep = 0.0
|
||||
@@ -6,6 +6,9 @@ TIMEOUT_DEFAULT = 300
|
||||
TIMEOUT_MIN = 300
|
||||
INSTANCES_DEFAULT = 4
|
||||
|
||||
AUTH_THROTTLE_MAX_FAILURES_DEFAULT = 10
|
||||
AUTH_THROTTLE_WINDOW_SECONDS_DEFAULT = 60
|
||||
|
||||
SYSTEM_PREAMBLE_DEFAULT = ""
|
||||
THINKING_DEFAULT = False
|
||||
|
||||
|
||||
@@ -350,9 +350,17 @@ class GatewayRuntime:
|
||||
return resp, None, timing
|
||||
|
||||
async def handle_chat(
|
||||
self, body: dict, cfg: dict, owner: tuple, user_agent: str, app_reference: str, log=None
|
||||
self,
|
||||
body: dict,
|
||||
cfg: dict,
|
||||
owner: tuple,
|
||||
user_agent: str,
|
||||
app_reference: str,
|
||||
log=None,
|
||||
bypass_allowed: bool = False,
|
||||
):
|
||||
log = log or (lambda message: None)
|
||||
bypass_preamble = bool(body.pop("bypass_preamble", False)) and bypass_allowed
|
||||
overlay = chat_overlay(body.get("model"), cfg)
|
||||
base_cfg = cfg
|
||||
if overlay:
|
||||
@@ -391,7 +399,8 @@ class GatewayRuntime:
|
||||
self.vision_calls += augmenter.calls
|
||||
vision_cost = augmenter.cost_usd
|
||||
|
||||
messages = apply_system_directives(messages, cfg.get("gateway_system_preamble", ""))
|
||||
preamble = "" if bypass_preamble else cfg.get("gateway_system_preamble", "")
|
||||
messages = apply_system_directives(messages, preamble)
|
||||
|
||||
requested = body.get("model")
|
||||
allow_client_model = bool(cfg.get("gateway_allow_client_model"))
|
||||
@@ -842,6 +851,35 @@ class GatewayRuntime:
|
||||
}
|
||||
},
|
||||
)
|
||||
if not cfg["gateway_embed_key"]:
|
||||
from devplacepy.services.audit import record as audit
|
||||
from devplacepy.services.openai_gateway.usage import audit_actor_for
|
||||
|
||||
actor_kind, actor_uid, actor_role = audit_actor_for(owner[0], owner[1])
|
||||
audit.record_system(
|
||||
"ai.gateway.call",
|
||||
actor_kind=actor_kind,
|
||||
actor_uid=actor_uid,
|
||||
actor_role=actor_role,
|
||||
origin="api",
|
||||
result="denied",
|
||||
summary="no embeddings key configured",
|
||||
metadata={
|
||||
"backend": "embed",
|
||||
"endpoint": "embeddings",
|
||||
"owner_kind": owner[0],
|
||||
"owner_id": owner[1],
|
||||
},
|
||||
)
|
||||
return JSONResponse(
|
||||
status_code=503,
|
||||
content={
|
||||
"error": {
|
||||
"message": "Embeddings are not configured",
|
||||
"type": "embeddings_not_configured",
|
||||
}
|
||||
},
|
||||
)
|
||||
client, sem = self._ensure(cfg)
|
||||
params = extract_params(body)
|
||||
handle_start = time.monotonic()
|
||||
@@ -866,13 +904,8 @@ class GatewayRuntime:
|
||||
"Content-Type": "application/json",
|
||||
**_attribution_headers(),
|
||||
**_extra_provider_headers(cfg),
|
||||
"Authorization": f"Bearer {cfg['gateway_embed_key']}",
|
||||
}
|
||||
if cfg["gateway_embed_key"]:
|
||||
headers["Authorization"] = f"Bearer {cfg['gateway_embed_key']}"
|
||||
else:
|
||||
log(
|
||||
"No upstream embeddings API key configured (gateway_embed_key / gateway_vision_key / OPENROUTER_API_KEY); upstream will likely reject the request"
|
||||
)
|
||||
|
||||
resp, exc, timing = await self._send(
|
||||
client,
|
||||
|
||||
@@ -11,16 +11,19 @@ from fastapi.responses import JSONResponse
|
||||
|
||||
from devplacepy.database import get_int_setting
|
||||
from devplacepy.services.base import BaseService, ConfigField
|
||||
from devplacepy.services.openai_gateway import config, quota
|
||||
from devplacepy.services.openai_gateway import auth_throttle, config, quota
|
||||
from devplacepy.services.openai_gateway.analytics import summary_metrics
|
||||
from devplacepy.services.openai_gateway.gateway import GatewayRuntime
|
||||
from devplacepy.services.openai_gateway.routing import model_store
|
||||
from devplacepy.utils import get_current_user
|
||||
from devplacepy.utils.auth import _request_has_auth
|
||||
from devplacepy.utils.guards import client_ip
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
APP_REFERENCE_PATTERN = re.compile(r"^[a-zA-Z0-9_.-]{1,30}$")
|
||||
DEFAULT_APP_REFERENCE = "default"
|
||||
INTERNAL_KEY_HEADER = "X-Gateway-Internal-Key"
|
||||
|
||||
USER_CONTENT_OWNER_KINDS = ("user", "admin")
|
||||
|
||||
@@ -324,6 +327,36 @@ class GatewayService(BaseService):
|
||||
"with this key. Clear it and restart to rotate.",
|
||||
group="Access",
|
||||
),
|
||||
ConfigField(
|
||||
"gateway_auth_throttle_enabled",
|
||||
"Failed-auth throttle",
|
||||
type="bool",
|
||||
default=True,
|
||||
help="Track failed authentication attempts per IP and block further "
|
||||
"unauthenticated attempts from an IP once it crosses the failure "
|
||||
"threshold within the window. Never blocks a request that presents "
|
||||
"valid credentials, regardless of what its IP has done.",
|
||||
group="Access",
|
||||
),
|
||||
ConfigField(
|
||||
"gateway_auth_throttle_max_failures",
|
||||
"Failed-auth threshold",
|
||||
type="int",
|
||||
default=config.AUTH_THROTTLE_MAX_FAILURES_DEFAULT,
|
||||
minimum=1,
|
||||
help="Failed authentication attempts allowed from one IP within the "
|
||||
"window before further unauthenticated attempts are blocked with 429.",
|
||||
group="Access",
|
||||
),
|
||||
ConfigField(
|
||||
"gateway_auth_throttle_window_seconds",
|
||||
"Failed-auth window (seconds)",
|
||||
type="int",
|
||||
default=config.AUTH_THROTTLE_WINDOW_SECONDS_DEFAULT,
|
||||
minimum=1,
|
||||
help="Sliding window the failed-auth threshold is counted over.",
|
||||
group="Access",
|
||||
),
|
||||
ConfigField(
|
||||
quota.FIELD_DEFAULT_USER,
|
||||
"Default per-user daily cap ($)",
|
||||
@@ -537,6 +570,24 @@ class GatewayService(BaseService):
|
||||
)
|
||||
return cfg
|
||||
|
||||
def _audit_throttle_tripped(
|
||||
self, request: Request, ip: str, count: int, threshold: int, window: int
|
||||
) -> None:
|
||||
from devplacepy.services.audit import record as audit
|
||||
|
||||
audit.record(
|
||||
request,
|
||||
"ai.gateway_auth_throttle.tripped",
|
||||
result="denied",
|
||||
summary=f"AI gateway auth throttle tripped for {ip} after {count} failed attempts",
|
||||
metadata={
|
||||
"ip": ip,
|
||||
"failed_attempts": count,
|
||||
"threshold": threshold,
|
||||
"window_seconds": window,
|
||||
},
|
||||
)
|
||||
|
||||
def authorize(self, request: Request) -> bool:
|
||||
cfg = self.get_config()
|
||||
if not cfg["gateway_require_auth"]:
|
||||
@@ -548,12 +599,35 @@ class GatewayService(BaseService):
|
||||
return True
|
||||
if presented and internal_key and presented == internal_key:
|
||||
return True
|
||||
throttle_enabled = cfg.get("gateway_auth_throttle_enabled", True)
|
||||
threshold = max(1, int(cfg.get("gateway_auth_throttle_max_failures", 10)))
|
||||
window = max(1, int(cfg.get("gateway_auth_throttle_window_seconds", 60)))
|
||||
has_credential = (
|
||||
bool(presented)
|
||||
or bool(request.cookies.get("session"))
|
||||
or _request_has_auth(request)
|
||||
)
|
||||
ip = client_ip(request, default="unknown")
|
||||
if (
|
||||
throttle_enabled
|
||||
and not has_credential
|
||||
and auth_throttle.is_throttled(ip, threshold, window)
|
||||
):
|
||||
raise HTTPException(
|
||||
status_code=429,
|
||||
detail="Too many failed authentication attempts. Try again later.",
|
||||
headers={"Retry-After": str(window)},
|
||||
)
|
||||
user = get_current_user(request)
|
||||
if user:
|
||||
if user.get("role") == "Admin" and cfg["gateway_allow_admins"]:
|
||||
return True
|
||||
if cfg["gateway_allow_users"]:
|
||||
return True
|
||||
if throttle_enabled:
|
||||
count = auth_throttle.record_failure(ip, window)
|
||||
if count == threshold:
|
||||
self._audit_throttle_tripped(request, ip, count, threshold, window)
|
||||
return False
|
||||
|
||||
def resolve_owner(self, request: Request) -> tuple:
|
||||
@@ -577,6 +651,13 @@ class GatewayService(BaseService):
|
||||
return (kind, user.get("uid") or "unknown")
|
||||
return ("anonymous", "anonymous")
|
||||
|
||||
def internal_bypass_allowed(self, request: Request, cfg: dict) -> bool:
|
||||
internal_key = cfg.get("gateway_internal_key")
|
||||
if not internal_key:
|
||||
return False
|
||||
presented = request.headers.get(INTERNAL_KEY_HEADER, "").strip()
|
||||
return bool(presented) and presented == internal_key
|
||||
|
||||
def user_content_owner(self, owner: tuple) -> str:
|
||||
if owner[0] in USER_CONTENT_OWNER_KINDS and owner[1]:
|
||||
return owner[1]
|
||||
@@ -722,7 +803,10 @@ class GatewayService(BaseService):
|
||||
if not isinstance(body, dict):
|
||||
self.log("Rejected chat request: JSON body was not an object")
|
||||
raise HTTPException(status_code=400, detail="Invalid JSON body")
|
||||
return await runtime.handle_chat(body, cfg, owner, user_agent, app_reference, self.log)
|
||||
bypass_allowed = self.internal_bypass_allowed(request, cfg)
|
||||
return await runtime.handle_chat(
|
||||
body, cfg, owner, user_agent, app_reference, self.log, bypass_allowed
|
||||
)
|
||||
if subpath == "embeddings" and request.method == "POST":
|
||||
try:
|
||||
body = await request.json()
|
||||
|
||||
@@ -62,6 +62,7 @@ def grade_free_text(api_key: str, question: dict, answer_text: str) -> scoring.G
|
||||
build_prompt(question, answer_text),
|
||||
QUIZ_GRADING_TIMEOUT_SECONDS,
|
||||
model=grading_model(),
|
||||
bypass_preamble=True,
|
||||
)
|
||||
except Exception as exc:
|
||||
logger.warning("Quiz AI grading failed: %s", exc)
|
||||
|
||||
@@ -404,6 +404,7 @@ class TelegramBridge:
|
||||
chat_id, "Your daily AI quota is reached (100%). Please try again later."
|
||||
)
|
||||
return
|
||||
devii.maybe_warn_quota_threshold("user", owner_id, owner_is_admin)
|
||||
session = devii.hub().get_or_create(
|
||||
"user",
|
||||
owner_id,
|
||||
|
||||
@@ -40,3 +40,4 @@ One global rule at the end of `base.css` collapses every animation/transition to
|
||||
- Every fluid grid/flex column sets `min-width: 0`, and every fluid grid track that holds content is written `minmax(0, 1fr)`, never a bare `1fr`. A `1fr` track's automatic minimum is the item's min-content size, so one unwrappable line inside a rendered code block (`.rendered-content pre`, `white-space: pre`) widens the track and blows the whole page open sideways. This is why a detail page (a `max-width` block, definite width, the `pre` scrolls inside it) survives content that destroys a listing grid. The rule covers a column that is fixed-width at desktop but becomes the single fluid column at a breakpoint (`.profile-sidebar`), and it is regression-tested by `assert_no_horizontal_overflow` in `tests/conftest.py`.
|
||||
- `!important` is allowed only for: the `.hidden`/`[hidden]` display utilities, the global reduced-motion rule, and the devii-avatar third-party-beating override. Anything else is a specificity problem to be fixed structurally.
|
||||
- Page-specific CSS lives in its own `static/css/*.css` loaded via `{% block extra_head %}`, never an inline `<style>` block; shared component styles live once (`components.css`, `feed.css` vote buttons) and are never redefined per page.
|
||||
- **Never give a sticky column its own CSS `max-height`/`overflow-y`.** A column shorter than the tallest column on the page renders a dead gap below it once capped (see root `CLAUDE.md`, commit `b475e7d6`, reverting exactly that). Independent scroll on a sticky column taller than the viewport is JS-driven: `static/js/StickyScrollPane.js` measures each managed element's real content height against the viewport space actually available and only then toggles `.sticky-scroll-active` (`overflow-y: auto`, defined once in `components.css`) plus a computed inline `max-height`. Add a new sticky column to its selector list in `Application.js` instead of writing CSS for it.
|
||||
|
||||
@@ -684,24 +684,6 @@ img {
|
||||
.topnav-user-dropdown:hover .dropdown-menu,
|
||||
.topnav-user-dropdown.open .dropdown-menu { display: block; }
|
||||
|
||||
.topnav-tools-dropdown { position: relative; }
|
||||
.topnav-tools-toggle { display: inline-flex; align-items: center; }
|
||||
.topnav-tools-dropdown.active .topnav-tools-toggle { color: var(--accent); }
|
||||
.topnav-tools-dropdown .dropdown-menu {
|
||||
display: none;
|
||||
position: absolute;
|
||||
top: 100%;
|
||||
right: 0;
|
||||
min-width: 180px;
|
||||
background: var(--bg-secondary);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: var(--radius);
|
||||
box-shadow: var(--shadow);
|
||||
padding: 0.25rem 0;
|
||||
z-index: var(--z-nav-drop);
|
||||
}
|
||||
.topnav-tools-dropdown:hover .dropdown-menu,
|
||||
.topnav-tools-dropdown.open .dropdown-menu { display: block; }
|
||||
.dropdown-item {
|
||||
display: block;
|
||||
padding: 0.5rem 0.875rem;
|
||||
@@ -1213,7 +1195,8 @@ body:has(.page-messages) {
|
||||
display: none;
|
||||
}
|
||||
|
||||
.topnav-tools-dropdown,
|
||||
.topnav-icon[href="/quizzes"],
|
||||
.topnav-icon[href="/battles"],
|
||||
.topnav-icon[href="/leaderboard"],
|
||||
.app-store-nav-link {
|
||||
display: none;
|
||||
|
||||
@@ -301,6 +301,11 @@ dp-upload[hidden] {
|
||||
border-color: var(--danger);
|
||||
}
|
||||
|
||||
.sticky-scroll-active {
|
||||
overflow-y: auto;
|
||||
overscroll-behavior: contain;
|
||||
}
|
||||
|
||||
emoji-picker {
|
||||
color-scheme: dark;
|
||||
--background: var(--bg-modal);
|
||||
|
||||
@@ -393,6 +393,71 @@
|
||||
color: var(--accent);
|
||||
}
|
||||
|
||||
.ds-history-list {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: var(--space-lg);
|
||||
}
|
||||
|
||||
.ds-history-item {
|
||||
margin-bottom: 0;
|
||||
}
|
||||
|
||||
.ds-history-item-head {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
gap: var(--space-md);
|
||||
margin-bottom: var(--space-sm);
|
||||
}
|
||||
|
||||
.ds-history-query {
|
||||
font-size: 1.05rem;
|
||||
margin: 0;
|
||||
}
|
||||
|
||||
.ds-history-status {
|
||||
text-transform: capitalize;
|
||||
font-size: 0.75rem;
|
||||
padding: 2px 10px;
|
||||
border-radius: var(--radius);
|
||||
background: var(--bg-card-hover);
|
||||
color: var(--text-muted);
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.ds-history-status.ds-status-done {
|
||||
background: rgba(var(--accent-rgb), 0.15);
|
||||
color: var(--success);
|
||||
}
|
||||
|
||||
.ds-history-status.ds-status-failed {
|
||||
color: var(--danger);
|
||||
}
|
||||
|
||||
.ds-history-status.ds-status-running,
|
||||
.ds-history-status.ds-status-pending {
|
||||
color: var(--warning);
|
||||
}
|
||||
|
||||
.ds-history-summary {
|
||||
color: var(--text-secondary);
|
||||
margin: 0 0 var(--space-md);
|
||||
}
|
||||
|
||||
.ds-history-item-foot {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
gap: var(--space-md);
|
||||
margin-top: var(--space-md);
|
||||
}
|
||||
|
||||
.ds-history-expired {
|
||||
color: var(--text-muted);
|
||||
font-size: 0.8125rem;
|
||||
}
|
||||
|
||||
.ds-exports {
|
||||
display: flex;
|
||||
gap: var(--space-sm);
|
||||
|
||||
@@ -218,6 +218,23 @@
|
||||
font-weight: 700;
|
||||
}
|
||||
|
||||
.param-nullable {
|
||||
font-size: 0.625rem;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.05em;
|
||||
color: var(--info);
|
||||
border: 1px solid var(--info);
|
||||
border-radius: 999px;
|
||||
padding: 0.05rem 0.4rem;
|
||||
}
|
||||
|
||||
.param-static {
|
||||
font-family: "SF Mono", Monaco, "Cascadia Code", monospace;
|
||||
font-size: 0.75rem;
|
||||
color: var(--text-muted);
|
||||
font-style: italic;
|
||||
}
|
||||
|
||||
.param-loc {
|
||||
font-size: 0.625rem;
|
||||
text-transform: uppercase;
|
||||
|
||||
@@ -376,3 +376,87 @@
|
||||
color: var(--text-muted);
|
||||
font-size: 0.8rem;
|
||||
}
|
||||
|
||||
.note-widget {
|
||||
position: relative;
|
||||
display: inline-flex;
|
||||
}
|
||||
|
||||
.note-btn.has-note {
|
||||
color: var(--accent);
|
||||
border-color: var(--accent);
|
||||
}
|
||||
|
||||
.note-editor {
|
||||
position: absolute;
|
||||
top: 100%;
|
||||
left: 0;
|
||||
z-index: var(--z-popover);
|
||||
width: 280px;
|
||||
margin-top: var(--space-xs);
|
||||
padding: var(--space-md);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: var(--radius);
|
||||
background: var(--bg-card);
|
||||
box-shadow: 0 4px 16px rgba(0, 0, 0, 0.35);
|
||||
}
|
||||
|
||||
.note-editor-textarea {
|
||||
width: 100%;
|
||||
min-height: 96px;
|
||||
resize: vertical;
|
||||
font-size: 0.85rem;
|
||||
}
|
||||
|
||||
.note-editor-actions {
|
||||
display: flex;
|
||||
justify-content: flex-end;
|
||||
gap: var(--space-sm);
|
||||
margin-top: var(--space-sm);
|
||||
}
|
||||
|
||||
.note-item {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: var(--space-sm);
|
||||
align-items: stretch;
|
||||
}
|
||||
|
||||
.note-item-link {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: var(--space-md);
|
||||
text-decoration: none;
|
||||
color: var(--text-primary);
|
||||
}
|
||||
|
||||
.note-item-content {
|
||||
margin: 0;
|
||||
color: var(--text-muted);
|
||||
font-size: 0.85rem;
|
||||
white-space: pre-wrap;
|
||||
}
|
||||
|
||||
.note-item-delete-form {
|
||||
align-self: flex-end;
|
||||
}
|
||||
|
||||
@keyframes engagement-busy-pulse {
|
||||
0%, 100% {
|
||||
opacity: 0.55;
|
||||
}
|
||||
50% {
|
||||
opacity: 0.9;
|
||||
}
|
||||
}
|
||||
|
||||
.post-action-btn.is-loading,
|
||||
.vote-star.is-loading,
|
||||
.reaction-chip.is-loading,
|
||||
.reaction-add-btn.is-loading,
|
||||
.reaction-palette-btn.is-loading,
|
||||
.bookmark-btn.is-loading,
|
||||
.poll-option.is-loading {
|
||||
pointer-events: none;
|
||||
animation: engagement-busy-pulse 0.9s ease-in-out infinite;
|
||||
}
|
||||
|
||||
@@ -231,6 +231,18 @@
|
||||
color: var(--text-primary);
|
||||
}
|
||||
|
||||
.gist-code-header-actions {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: var(--space-xs);
|
||||
}
|
||||
|
||||
.gist-code-block .gist-rendered-content {
|
||||
margin-bottom: 0;
|
||||
padding: 1rem;
|
||||
background: var(--bg-card);
|
||||
}
|
||||
|
||||
.gist-code-block pre {
|
||||
margin: 0;
|
||||
padding: 1rem;
|
||||
|
||||
@@ -132,13 +132,17 @@ export class ApiTester {
|
||||
buildParams() {
|
||||
const table = this.el("div", { class: "param-table" });
|
||||
for (const param of this.config.params) {
|
||||
const control = this.buildControl(param);
|
||||
this.inputs.push({ param, control });
|
||||
control.addEventListener("input", () => this.renderSnippets());
|
||||
control.addEventListener("change", () => this.renderSnippets());
|
||||
const isResponseField = param.location === "response";
|
||||
const control = isResponseField ? this.el("span", { class: "param-static", text: param.type }) : this.buildControl(param);
|
||||
if (!isResponseField) {
|
||||
this.inputs.push({ param, control });
|
||||
control.addEventListener("input", () => this.renderSnippets());
|
||||
control.addEventListener("change", () => this.renderSnippets());
|
||||
}
|
||||
const label = this.el("div", { class: "param-label" }, [
|
||||
this.el("span", { class: "param-name", text: param.name }),
|
||||
param.required ? this.el("span", { class: "param-required", text: "*" }) : null,
|
||||
param.nullable ? this.el("span", { class: "param-nullable", text: "nullable" }) : null,
|
||||
this.el("span", { class: "param-loc param-loc-" + param.location, text: param.location }),
|
||||
]);
|
||||
const allowed = param.type === "enum" && param.options && param.options.length
|
||||
|
||||
@@ -14,6 +14,7 @@ import { PushManager } from "./PushManager.js";
|
||||
import { CounterManager } from "./CounterManager.js";
|
||||
import { ReactionBar } from "./ReactionBar.js";
|
||||
import { BookmarkManager } from "./BookmarkManager.js";
|
||||
import { NoteManager } from "./NoteManager.js";
|
||||
import { PollManager } from "./PollManager.js";
|
||||
import { WarComposer } from "./WarComposer.js";
|
||||
import { ApiKeyManager } from "./ApiKeyManager.js";
|
||||
@@ -48,6 +49,7 @@ import { ScrollMemory } from "./ScrollMemory.js";
|
||||
import { GameFarm } from "./GameFarm.js";
|
||||
import { Accessibility } from "./Accessibility.js";
|
||||
import { OverflowTabs } from "./OverflowTabs.js";
|
||||
import { StickyScrollPane } from "./StickyScrollPane.js";
|
||||
import { AiAutoload } from "./autoload/AiAutoload.js";
|
||||
import "./components/AiInteraction.js";
|
||||
import "./components/AiStatus.js";
|
||||
@@ -87,6 +89,7 @@ class Application {
|
||||
this.counters = new CounterManager(this.pubsub);
|
||||
this.reactions = new ReactionBar();
|
||||
this.bookmarks = new BookmarkManager();
|
||||
this.notes = new NoteManager();
|
||||
this.polls = new PollManager();
|
||||
this.warComposer = new WarComposer();
|
||||
this.apiKey = new ApiKeyManager();
|
||||
@@ -118,6 +121,13 @@ class Application {
|
||||
this.editorLauncher = new EditorLauncher();
|
||||
this.gameFarm = new GameFarm();
|
||||
this.overflowTabs = new OverflowTabs();
|
||||
this.stickyScrollPane = new StickyScrollPane([
|
||||
".feed-right",
|
||||
".sidebar-card",
|
||||
".post-page-sidebar",
|
||||
".profile-sidebar",
|
||||
".leaderboard-page > aside",
|
||||
]);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
// retoor <retoor@molodetz.nl>
|
||||
|
||||
export class Avatar {
|
||||
static imgElement(username, size = 24) {
|
||||
static imgElement(username, size = 24, seed = null) {
|
||||
const img = document.createElement("img");
|
||||
img.src = `/avatar/multiavatar/${encodeURIComponent(username)}?size=${size}`;
|
||||
img.src = `/avatar/multiavatar/${encodeURIComponent(seed || username)}?size=${size}`;
|
||||
img.className = "avatar-img";
|
||||
img.style.width = `${size}px`;
|
||||
img.style.height = `${size}px`;
|
||||
|
||||
@@ -22,6 +22,6 @@ export class BookmarkManager extends OptimisticAction {
|
||||
if (label) {
|
||||
label.textContent = result.saved ? "Saved" : "Save";
|
||||
}
|
||||
});
|
||||
}, button);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -38,7 +38,7 @@ Conventions:
|
||||
|
||||
## Shared frontend utilities (do not re-implement these)
|
||||
|
||||
A small set of plain ES6 modules under `static/js/` own the cross-cutting patterns so feature code stays tiny. Reach for these instead of hand-rolling a loop, a fetch, or a click handler. `Http` (`static/js/Http.js`) is the single fetch helper (`getJson`, `sendForm`, and `send` which throws `error.message` on non-2xx or a `200 {ok:false}` body); every live-update loop uses `Poller` (`new Poller(fn, intervalMs, {pauseHidden})`); async-job status polls use `JobPoller.run(statusUrl, {onDone, onFailed, onTimeout})` (`ProjectForker`, `ZipDownloader`); click-to-POST engagement controllers (`VoteManager`/`ReactionBar`/`BookmarkManager`/`PollManager`) extend `OptimisticAction` and call `this.submit(url, params, errorTarget, render)`. Floating windows (container terminals and Devii) extend `FloatingWindow`. **Scroll restoration is `ScrollMemory`** (`static/js/ScrollMemory.js`, `app.scrollMemory`): per-tab (sessionStorage) positions keyed by exact `path+search`, restored ONLY on `back_forward`/`reload` navigations or a click on `a.back-link`/`[data-scroll-back]`/breadcrumb/previous-trail-URL links, applied via a layout-stable rAF loop that aborts on user input; it sets `history.scrollRestoration = "manual"` site-wide and upgrades query-less back-link hrefs to the exact previous URL - mark any "back to X" anchor with the `back-link` class and never hand-roll scroll persistence.
|
||||
A small set of plain ES6 modules under `static/js/` own the cross-cutting patterns so feature code stays tiny. Reach for these instead of hand-rolling a loop, a fetch, or a click handler. `Http` (`static/js/Http.js`) is the single fetch helper (`getJson`, `sendForm`, and `send` which throws `error.message` on non-2xx or a `200 {ok:false}` body); every live-update loop uses `Poller` (`new Poller(fn, intervalMs, {pauseHidden})`); async-job status polls use `JobPoller.run(statusUrl, {onDone, onFailed, onTimeout})` (`ProjectForker`, `ZipDownloader`); click-to-POST engagement controllers (`VoteManager`/`ReactionBar`/`BookmarkManager`/`PollManager`) extend `OptimisticAction` and call `this.submit(url, params, errorTarget, render)`. Floating windows (container terminals and Devii) extend `FloatingWindow`. **Independent scroll on tall sticky columns is `StickyScrollPane`** (`static/js/StickyScrollPane.js`, `app.stickyScrollPane`): constructed once in `Application.js` with the list of sticky-column selectors (`.feed-right`, `.sidebar-card`, `.post-page-sidebar`, `.profile-sidebar`, `.leaderboard-page > aside`), it measures each element's `scrollHeight` against the viewport space actually available below its computed `top` offset (a one-time probe-element measurement resolves the `--space-lg` gap token to pixels, so no literal is duplicated from CSS) and toggles the `.sticky-scroll-active` class (`overflow-y: auto` in `components.css`) plus an inline `max-height` ONLY when the column's natural content exceeds that space - a column that fits stays untouched `position: sticky` with no cap, so it never re-introduces the dead-space regression a pure CSS `max-height` approach caused (see `CLAUDE.md` root history on commit `b475e7d6`). Recomputes on `window resize` and on a shared `ResizeObserver` watching every managed element (catches content growth). Never add a second max-height/overflow recipe to a sticky column - extend the selector list passed to this one instance instead. **Scroll restoration is `ScrollMemory`** (`static/js/ScrollMemory.js`, `app.scrollMemory`): per-tab (sessionStorage) positions keyed by exact `path+search`, restored ONLY on `back_forward`/`reload` navigations or a click on `a.back-link`/`[data-scroll-back]`/breadcrumb/previous-trail-URL links, applied via a layout-stable rAF loop that aborts on user input; it sets `history.scrollRestoration = "manual"` site-wide and upgrades query-less back-link hrefs to the exact previous URL - mark any "back to X" anchor with the `back-link` class and never hand-roll scroll persistence.
|
||||
|
||||
Detail on each utility:
|
||||
|
||||
|
||||
@@ -77,6 +77,7 @@ export class DeviiTerminal {
|
||||
}
|
||||
return;
|
||||
}
|
||||
if (this.element && this.element.state !== "closed") return;
|
||||
if (this._canOpenOnGesture()) this.open();
|
||||
}
|
||||
|
||||
|
||||
@@ -7,6 +7,7 @@ export class DomUtils {
|
||||
this.initClipboardCopy();
|
||||
this.initShareButtons();
|
||||
this.initTogglers();
|
||||
this.initViewToggles();
|
||||
this.initStopPropagation();
|
||||
this.initReload();
|
||||
this.initCardNav();
|
||||
@@ -66,6 +67,18 @@ export class DomUtils {
|
||||
});
|
||||
}
|
||||
|
||||
initViewToggles() {
|
||||
DomUtils.onDataAttr("view-toggle", "click", (btn) => {
|
||||
const primary = document.getElementById(btn.dataset.viewToggle);
|
||||
const alt = document.getElementById(btn.dataset.viewToggleAlt);
|
||||
if (!primary || !alt) return;
|
||||
const showingAlt = alt.classList.contains("hidden");
|
||||
primary.classList.toggle("hidden", showingAlt);
|
||||
alt.classList.toggle("hidden", !showingAlt);
|
||||
btn.textContent = showingAlt ? btn.dataset.viewToggleLabelAlt : btn.dataset.viewToggleLabel;
|
||||
});
|
||||
}
|
||||
|
||||
initStopPropagation() {
|
||||
DomUtils.onDataAttr("stop-propagation", "click", (el, e) => e.stopPropagation());
|
||||
}
|
||||
|
||||
@@ -45,8 +45,11 @@ export class Http {
|
||||
return error;
|
||||
}
|
||||
|
||||
static async getJson(url) {
|
||||
const response = await fetch(url, { headers: { "Accept": "application/json" } });
|
||||
static async getJson(url, options = {}) {
|
||||
const response = await fetch(url, {
|
||||
headers: { "Accept": "application/json" },
|
||||
signal: options.signal,
|
||||
});
|
||||
if (!response.ok) {
|
||||
throw Http._error(`request failed with status ${response.status}`, response.status);
|
||||
}
|
||||
@@ -106,10 +109,15 @@ export class Http {
|
||||
}
|
||||
|
||||
static async sendDelete(url, options = {}) {
|
||||
const response = await fetch(url, {
|
||||
const init = {
|
||||
method: "DELETE",
|
||||
headers: { "Accept": "application/json" },
|
||||
});
|
||||
};
|
||||
if (options.body !== undefined) {
|
||||
init.headers["Content-Type"] = "application/json";
|
||||
init.body = JSON.stringify(options.body);
|
||||
}
|
||||
const response = await fetch(url, init);
|
||||
if (response.redirected && response.url.includes("/auth/login")) {
|
||||
Http.toLogin();
|
||||
return Http.suspend();
|
||||
|
||||
@@ -13,6 +13,7 @@ export class MentionInput {
|
||||
this.debounceTimer = null;
|
||||
this.lastMatch = null;
|
||||
this.nav = null;
|
||||
this.pendingController = null;
|
||||
this.build();
|
||||
}
|
||||
|
||||
@@ -69,9 +70,21 @@ export class MentionInput {
|
||||
this.debounceTimer = setTimeout(() => this.fetch(query), 200);
|
||||
}
|
||||
|
||||
isCurrentQuery(query) {
|
||||
return !!this.lastMatch && this.lastMatch.query === query;
|
||||
}
|
||||
|
||||
async fetch(query) {
|
||||
if (this.pendingController) {
|
||||
this.pendingController.abort();
|
||||
}
|
||||
const controller = new AbortController();
|
||||
this.pendingController = controller;
|
||||
try {
|
||||
const data = await Http.getJson("/profile/search?q=" + encodeURIComponent(query));
|
||||
const data = await Http.getJson("/profile/search?q=" + encodeURIComponent(query), { signal: controller.signal });
|
||||
if (!this.isCurrentQuery(query)) {
|
||||
return;
|
||||
}
|
||||
const results = data.results || [];
|
||||
if (results.length === 0) {
|
||||
this.hide();
|
||||
@@ -79,7 +92,14 @@ export class MentionInput {
|
||||
}
|
||||
this.render(results);
|
||||
} catch (e) {
|
||||
if (e.name === "AbortError") {
|
||||
return;
|
||||
}
|
||||
this.hide();
|
||||
} finally {
|
||||
if (this.pendingController === controller) {
|
||||
this.pendingController = null;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -92,7 +112,7 @@ export class MentionInput {
|
||||
item.dataset.username = r.username;
|
||||
const label = document.createElement("span");
|
||||
label.textContent = "@" + r.username;
|
||||
item.append(Avatar.imgElement(r.username), label);
|
||||
item.append(Avatar.imgElement(r.username, 24, r.avatar_seed), label);
|
||||
item.addEventListener("mousedown", (e) => {
|
||||
e.preventDefault();
|
||||
this.insert(r.username);
|
||||
|
||||
@@ -4,7 +4,6 @@ export class MobileNav {
|
||||
constructor() {
|
||||
this.initMobileNav();
|
||||
this.initProfileDropdown();
|
||||
this.initToolsDropdown();
|
||||
}
|
||||
|
||||
initMobileNav() {
|
||||
@@ -80,31 +79,4 @@ export class MobileNav {
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
initToolsDropdown() {
|
||||
const dropdown = document.querySelector(".topnav-tools-dropdown");
|
||||
if (!dropdown) return;
|
||||
const btn = dropdown.querySelector(".topnav-tools-toggle");
|
||||
if (!btn) return;
|
||||
|
||||
btn.addEventListener("click", (e) => {
|
||||
e.stopPropagation();
|
||||
const open = dropdown.classList.toggle("open");
|
||||
btn.setAttribute("aria-expanded", open ? "true" : "false");
|
||||
});
|
||||
|
||||
document.addEventListener("click", (e) => {
|
||||
if (!dropdown.contains(e.target)) {
|
||||
dropdown.classList.remove("open");
|
||||
btn.setAttribute("aria-expanded", "false");
|
||||
}
|
||||
});
|
||||
|
||||
document.addEventListener("keydown", (e) => {
|
||||
if (e.key === "Escape") {
|
||||
dropdown.classList.remove("open");
|
||||
btn.setAttribute("aria-expanded", "false");
|
||||
}
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
@@ -29,6 +29,7 @@ export class ModalManager {
|
||||
const modal = document.getElementById(modalId);
|
||||
if (modal) {
|
||||
modal.classList.add("visible");
|
||||
this.pushModalHistory(modal);
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -37,11 +38,13 @@ export class ModalManager {
|
||||
modal.addEventListener("click", (e) => {
|
||||
if (e.target === modal) {
|
||||
modal.classList.remove("visible");
|
||||
this.popModalHistory(modal);
|
||||
}
|
||||
});
|
||||
modal.querySelectorAll(".modal-close").forEach((closeBtn) => {
|
||||
closeBtn.addEventListener("click", () => {
|
||||
modal.classList.remove("visible");
|
||||
this.popModalHistory(modal);
|
||||
});
|
||||
});
|
||||
this.enhanceModal(modal);
|
||||
@@ -50,8 +53,30 @@ export class ModalManager {
|
||||
document.addEventListener("keydown", (e) => {
|
||||
if (e.key !== "Escape") return;
|
||||
const open = [...document.querySelectorAll(".modal-overlay.visible")].pop();
|
||||
if (open) open.classList.remove("visible");
|
||||
if (open) {
|
||||
open.classList.remove("visible");
|
||||
this.popModalHistory(open);
|
||||
}
|
||||
});
|
||||
|
||||
window.addEventListener("popstate", () => {
|
||||
const open = [...document.querySelectorAll(".modal-overlay.visible")].pop();
|
||||
if (!open) return;
|
||||
open._modalHistoryOpen = false;
|
||||
open.classList.remove("visible");
|
||||
});
|
||||
}
|
||||
|
||||
pushModalHistory(modal) {
|
||||
if (modal._modalHistoryOpen) return;
|
||||
modal._modalHistoryOpen = true;
|
||||
history.pushState({ modalOverlay: true }, "", location.href);
|
||||
}
|
||||
|
||||
popModalHistory(modal) {
|
||||
if (!modal._modalHistoryOpen) return;
|
||||
modal._modalHistoryOpen = false;
|
||||
history.back();
|
||||
}
|
||||
|
||||
enhanceModal(modal) {
|
||||
|
||||
@@ -0,0 +1,135 @@
|
||||
// retoor <retoor@molodetz.nl>
|
||||
|
||||
import { OptimisticAction } from "./OptimisticAction.js";
|
||||
|
||||
export class NoteManager extends OptimisticAction {
|
||||
constructor() {
|
||||
super();
|
||||
document.addEventListener("click", (event) => this.onClick(event));
|
||||
}
|
||||
|
||||
onClick(event) {
|
||||
const toggle = event.target.closest("[data-note-toggle]");
|
||||
if (toggle) {
|
||||
event.preventDefault();
|
||||
this.toggle(toggle);
|
||||
return;
|
||||
}
|
||||
const cancel = event.target.closest(".note-editor-cancel");
|
||||
if (cancel) {
|
||||
event.preventDefault();
|
||||
this.cancel(cancel);
|
||||
return;
|
||||
}
|
||||
const save = event.target.closest(".note-editor-save");
|
||||
if (save) {
|
||||
event.preventDefault();
|
||||
this.save(save);
|
||||
return;
|
||||
}
|
||||
const remove = event.target.closest(".note-editor-delete");
|
||||
if (remove) {
|
||||
event.preventDefault();
|
||||
this.remove(remove);
|
||||
}
|
||||
}
|
||||
|
||||
toggle(button) {
|
||||
const widget = button.closest(".note-widget");
|
||||
const editor = widget ? widget.querySelector(".note-editor") : null;
|
||||
if (!editor) {
|
||||
return;
|
||||
}
|
||||
const opening = editor.hidden;
|
||||
editor.hidden = !opening;
|
||||
button.setAttribute("aria-expanded", opening ? "true" : "false");
|
||||
if (opening) {
|
||||
const textarea = editor.querySelector(".note-editor-textarea");
|
||||
if (textarea) {
|
||||
textarea.focus();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
cancel(button) {
|
||||
const widget = button.closest(".note-widget");
|
||||
const editor = widget ? widget.querySelector(".note-editor") : null;
|
||||
const toggle = widget ? widget.querySelector("[data-note-toggle]") : null;
|
||||
if (!editor) {
|
||||
return;
|
||||
}
|
||||
const textarea = editor.querySelector(".note-editor-textarea");
|
||||
if (textarea) {
|
||||
textarea.value = textarea.dataset.saved || "";
|
||||
}
|
||||
editor.hidden = true;
|
||||
if (toggle) {
|
||||
toggle.setAttribute("aria-expanded", "false");
|
||||
}
|
||||
}
|
||||
|
||||
async save(button) {
|
||||
const widget = button.closest(".note-widget");
|
||||
const editor = widget ? widget.querySelector(".note-editor") : null;
|
||||
const textarea = editor ? editor.querySelector(".note-editor-textarea") : null;
|
||||
if (!widget || !textarea) {
|
||||
return;
|
||||
}
|
||||
const content = textarea.value.trim();
|
||||
if (!content) {
|
||||
return;
|
||||
}
|
||||
const type = widget.dataset.noteType;
|
||||
const uid = widget.dataset.noteUid;
|
||||
await this.submit(
|
||||
`/notes/${type}/${uid}`,
|
||||
{ content },
|
||||
widget,
|
||||
(result) => this.applySaved(widget, result.content),
|
||||
button
|
||||
);
|
||||
}
|
||||
|
||||
async remove(button) {
|
||||
const widget = button.closest(".note-widget");
|
||||
if (!widget) {
|
||||
return;
|
||||
}
|
||||
const type = widget.dataset.noteType;
|
||||
const uid = widget.dataset.noteUid;
|
||||
await this.submit(
|
||||
`/notes/${type}/${uid}/delete`,
|
||||
{},
|
||||
widget,
|
||||
() => this.applySaved(widget, ""),
|
||||
button
|
||||
);
|
||||
}
|
||||
|
||||
applySaved(widget, content) {
|
||||
const toggle = widget.querySelector("[data-note-toggle]");
|
||||
const editor = widget.querySelector(".note-editor");
|
||||
const textarea = editor ? editor.querySelector(".note-editor-textarea") : null;
|
||||
const label = toggle ? toggle.querySelector(".note-label") : null;
|
||||
const deleteBtn = editor ? editor.querySelector(".note-editor-delete") : null;
|
||||
if (textarea) {
|
||||
textarea.value = content;
|
||||
textarea.dataset.saved = content;
|
||||
}
|
||||
if (toggle) {
|
||||
toggle.classList.toggle("has-note", Boolean(content));
|
||||
}
|
||||
if (label) {
|
||||
label.textContent = content ? "Edit note" : "Add note";
|
||||
}
|
||||
if (deleteBtn) {
|
||||
deleteBtn.hidden = !content;
|
||||
}
|
||||
if (editor) {
|
||||
editor.hidden = true;
|
||||
}
|
||||
if (toggle) {
|
||||
toggle.setAttribute("aria-expanded", "false");
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -4,7 +4,19 @@ import { Http } from "./Http.js";
|
||||
import { Toast } from "./Toast.js";
|
||||
|
||||
export class OptimisticAction {
|
||||
async submit(url, params, errorTarget, render) {
|
||||
setBusy(target, busy) {
|
||||
if (!target) return;
|
||||
target.classList.toggle("is-loading", busy);
|
||||
target.disabled = busy;
|
||||
if (busy) {
|
||||
target.setAttribute("aria-busy", "true");
|
||||
} else {
|
||||
target.removeAttribute("aria-busy");
|
||||
}
|
||||
}
|
||||
|
||||
async submit(url, params, errorTarget, render, busyTarget = errorTarget) {
|
||||
this.setBusy(busyTarget, true);
|
||||
try {
|
||||
const result = await Http.sendForm(url, params, { silent: true });
|
||||
if (render) render(result);
|
||||
@@ -13,11 +25,14 @@ export class OptimisticAction {
|
||||
console.error("optimistic action failed", error);
|
||||
if (errorTarget) Toast.flash(errorTarget, "Error", 1500);
|
||||
return null;
|
||||
} finally {
|
||||
this.setBusy(busyTarget, false);
|
||||
}
|
||||
}
|
||||
|
||||
async submitOptimistic(url, params, errorTarget, apply, revert, reconcile) {
|
||||
async submitOptimistic(url, params, errorTarget, apply, revert, reconcile, busyTarget = errorTarget) {
|
||||
apply();
|
||||
this.setBusy(busyTarget, true);
|
||||
try {
|
||||
const result = await Http.sendForm(url, params, { silent: true });
|
||||
if (reconcile) reconcile(result);
|
||||
@@ -27,6 +42,8 @@ export class OptimisticAction {
|
||||
revert();
|
||||
if (errorTarget) Toast.flash(errorTarget, "Error", 1500);
|
||||
return null;
|
||||
} finally {
|
||||
this.setBusy(busyTarget, false);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -42,7 +42,7 @@ export class PollManager extends OptimisticAction {
|
||||
}
|
||||
const pollUid = poll.dataset.pollUid;
|
||||
const optionUid = option.dataset.optionUid;
|
||||
await this.submit(`/polls/${pollUid}/vote`, { option_uid: optionUid }, null, (result) => this.render(poll, result));
|
||||
await this.submit(`/polls/${pollUid}/vote`, { option_uid: optionUid }, null, (result) => this.render(poll, result), option);
|
||||
}
|
||||
|
||||
render(poll, result) {
|
||||
|
||||
@@ -10,11 +10,37 @@ export class PushManager {
|
||||
}
|
||||
this.userUid = document.body.dataset.userUid || "";
|
||||
this.triggers = Array.from(document.querySelectorAll("[data-push-enable]"));
|
||||
this.logoutLinks = Array.from(document.querySelectorAll("a[href='/auth/logout']"));
|
||||
this.bindTriggers();
|
||||
this.bindLogoutLinks();
|
||||
this.refreshTriggerVisibility();
|
||||
this.register(true).catch((error) => console.error("Push silent register failed:", error));
|
||||
}
|
||||
|
||||
bindLogoutLinks() {
|
||||
this.logoutLinks.forEach((link) => {
|
||||
link.addEventListener("click", (event) => {
|
||||
event.preventDefault();
|
||||
this.unregister().finally(() => {
|
||||
window.location.href = link.href;
|
||||
});
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
async unregister() {
|
||||
try {
|
||||
const registration = await navigator.serviceWorker.getRegistration();
|
||||
if (!registration) return;
|
||||
const subscription = await registration.pushManager.getSubscription();
|
||||
if (!subscription) return;
|
||||
await Http.sendDelete("/push.json", { body: { endpoint: subscription.endpoint }, silent: true });
|
||||
await subscription.unsubscribe();
|
||||
} catch (error) {
|
||||
console.error("Error unregistering push notifications:", error);
|
||||
}
|
||||
}
|
||||
|
||||
bindTriggers() {
|
||||
this.triggers.forEach((trigger) => {
|
||||
trigger.addEventListener("click", (event) => {
|
||||
|
||||
@@ -121,13 +121,14 @@ export class ReactionBar extends OptimisticAction {
|
||||
if (!bar) {
|
||||
return;
|
||||
}
|
||||
await this.reactWith(bar, trigger.dataset.reactionEmoji);
|
||||
await this.reactWith(bar, trigger.dataset.reactionEmoji, trigger);
|
||||
}
|
||||
|
||||
async reactWith(bar, emoji) {
|
||||
async reactWith(bar, emoji, busyTarget = null) {
|
||||
const type = bar.dataset.reactionType;
|
||||
const uid = bar.dataset.reactionUid;
|
||||
await this.submit(`/reactions/${type}/${uid}`, { emoji }, null, (result) => this.render(bar, result));
|
||||
const busy = busyTarget || bar.querySelector(".reaction-add-btn");
|
||||
await this.submit(`/reactions/${type}/${uid}`, { emoji }, null, (result) => this.render(bar, result), busy);
|
||||
this.closeAll();
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,81 @@
|
||||
// retoor <retoor@molodetz.nl>
|
||||
|
||||
export class StickyScrollPane {
|
||||
constructor(selectors, options = {}) {
|
||||
this.selectors = Array.isArray(selectors) ? selectors : [selectors];
|
||||
this.activeClass = options.activeClass || "sticky-scroll-active";
|
||||
this.gapVar = options.gapVar || "--space-lg";
|
||||
this.elements = new Set();
|
||||
this.scheduled = false;
|
||||
this.gapPx = this.resolveGapPx(this.gapVar);
|
||||
this.resizeObserver = typeof ResizeObserver !== "undefined"
|
||||
? new ResizeObserver(() => this.scheduleRecompute())
|
||||
: null;
|
||||
this.collect();
|
||||
window.addEventListener("resize", () => this.scheduleRecompute(), { passive: true });
|
||||
this.scheduleRecompute();
|
||||
}
|
||||
|
||||
resolveGapPx(varName) {
|
||||
const probe = document.createElement("div");
|
||||
probe.style.position = "absolute";
|
||||
probe.style.visibility = "hidden";
|
||||
probe.style.pointerEvents = "none";
|
||||
probe.style.height = `var(${varName})`;
|
||||
document.body.appendChild(probe);
|
||||
const px = probe.getBoundingClientRect().height;
|
||||
probe.remove();
|
||||
return px;
|
||||
}
|
||||
|
||||
collect() {
|
||||
this.selectors.forEach((selector) => {
|
||||
document.querySelectorAll(selector).forEach((el) => {
|
||||
if (this.elements.has(el)) return;
|
||||
this.elements.add(el);
|
||||
if (this.resizeObserver) this.resizeObserver.observe(el);
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
scheduleRecompute() {
|
||||
if (this.scheduled) return;
|
||||
this.scheduled = true;
|
||||
requestAnimationFrame(() => {
|
||||
this.scheduled = false;
|
||||
this.recomputeAll();
|
||||
});
|
||||
}
|
||||
|
||||
recomputeAll() {
|
||||
this.elements.forEach((el) => this.recompute(el));
|
||||
}
|
||||
|
||||
recompute(el) {
|
||||
const style = getComputedStyle(el);
|
||||
if (style.position !== "sticky" || style.display === "none") {
|
||||
this.disengage(el);
|
||||
return;
|
||||
}
|
||||
const top = parseFloat(style.top) || 0;
|
||||
const available = window.innerHeight - top - this.gapPx;
|
||||
const natural = el.scrollHeight;
|
||||
if (available > 0 && natural > available) {
|
||||
this.engage(el, available);
|
||||
} else {
|
||||
this.disengage(el);
|
||||
}
|
||||
}
|
||||
|
||||
engage(el, availablePx) {
|
||||
const height = `${Math.round(availablePx)}px`;
|
||||
if (el.style.maxHeight !== height) el.style.maxHeight = height;
|
||||
el.classList.add(this.activeClass);
|
||||
}
|
||||
|
||||
disengage(el) {
|
||||
if (!el.classList.contains(this.activeClass) && !el.style.maxHeight) return;
|
||||
el.classList.remove(this.activeClass);
|
||||
el.style.maxHeight = "";
|
||||
}
|
||||
}
|
||||
@@ -5,18 +5,18 @@ import { OptimisticAction } from "./OptimisticAction.js";
|
||||
export class VoteManager extends OptimisticAction {
|
||||
constructor() {
|
||||
super();
|
||||
this.initVoteButtons();
|
||||
document.addEventListener("click", (event) => this.onClick(event));
|
||||
}
|
||||
|
||||
initVoteButtons() {
|
||||
document.querySelectorAll('form[action^="/votes/"] button[type="submit"]').forEach((button) => {
|
||||
const form = button.closest("form");
|
||||
button.addEventListener("click", (event) => {
|
||||
event.preventDefault();
|
||||
event.stopPropagation();
|
||||
this.cast(form, button);
|
||||
});
|
||||
});
|
||||
onClick(event) {
|
||||
const button = event.target.closest('form[action^="/votes/"] button[type="submit"]');
|
||||
if (!button) {
|
||||
return;
|
||||
}
|
||||
const form = button.closest("form");
|
||||
event.preventDefault();
|
||||
event.stopPropagation();
|
||||
this.cast(form, button);
|
||||
}
|
||||
|
||||
cast(form, button) {
|
||||
|
||||
@@ -46,7 +46,10 @@ export class AppChat extends Component {
|
||||
this._pendingSends = new Map();
|
||||
this._failedSends = new Map();
|
||||
this._uploading = false;
|
||||
this._sendLocked = false;
|
||||
this._lockingSends = new Set();
|
||||
this._userAtBottom = true;
|
||||
this._lastForcedScrollTop = null;
|
||||
this._stabilizeFrames = 0;
|
||||
this._stabilizePending = false;
|
||||
this._lastTypingSent = 0;
|
||||
@@ -95,6 +98,7 @@ export class AppChat extends Component {
|
||||
this._bindAutoScroll();
|
||||
this._startScrollWatcher();
|
||||
this._ensureJumpButton();
|
||||
this._bindActiveConversation();
|
||||
this._connect();
|
||||
this._initPresence();
|
||||
this._scrollThreadToEnd();
|
||||
@@ -111,12 +115,16 @@ export class AppChat extends Component {
|
||||
if (this._mobileQuery && this._onMobileChange) {
|
||||
this._mobileQuery.removeEventListener("change", this._onMobileChange);
|
||||
}
|
||||
if (this._onWindowFocus) window.removeEventListener("focus", this._onWindowFocus);
|
||||
if (this._onVisibilityChange) document.removeEventListener("visibilitychange", this._onVisibilityChange);
|
||||
clearTimeout(this._typingHideTimer);
|
||||
clearTimeout(this._disconnectTimer);
|
||||
cancelAnimationFrame(this._autoGrowFrame);
|
||||
for (const entry of this._pendingSends.values()) clearTimeout(entry.timeoutId);
|
||||
this._pendingSends.clear();
|
||||
this._failedSends.clear();
|
||||
this._lockingSends.clear();
|
||||
this._sendLocked = false;
|
||||
if (this._presence) this._presence.stop();
|
||||
if (this._onPopState) window.removeEventListener("popstate", this._onPopState);
|
||||
}
|
||||
@@ -271,6 +279,7 @@ export class AppChat extends Component {
|
||||
if (!this.form || !this.input) return;
|
||||
this.form.addEventListener("submit", (event) => {
|
||||
event.preventDefault();
|
||||
if (this._sendLocked) return;
|
||||
this._sendViaSocket();
|
||||
});
|
||||
if (this.upload) {
|
||||
@@ -336,11 +345,17 @@ export class AppChat extends Component {
|
||||
|
||||
_refreshSendButton() {
|
||||
if (!this.sendBtn) return;
|
||||
const busy = this._uploading;
|
||||
const busy = this._uploading || this._sendLocked;
|
||||
this.sendBtn.disabled = busy;
|
||||
this.sendBtn.classList.toggle("is-sending", busy);
|
||||
}
|
||||
|
||||
_releaseSendLock(clientId) {
|
||||
if (!this._lockingSends.has(clientId)) return;
|
||||
this._lockingSends.delete(clientId);
|
||||
if (this._lockingSends.size === 0) this._sendLocked = false;
|
||||
}
|
||||
|
||||
_collectAttachments() {
|
||||
const hidden = this.form.querySelector('input[name="attachment_uids"]');
|
||||
if (!hidden || !hidden.value) return [];
|
||||
@@ -376,6 +391,10 @@ export class AppChat extends Component {
|
||||
|
||||
const timeoutId = window.setTimeout(() => this._failPendingSend(clientId), SEND_TIMEOUT_MS);
|
||||
this._pendingSends.set(clientId, { content, attachmentUids, timeoutId });
|
||||
if (!retryClientId) {
|
||||
this._lockingSends.add(clientId);
|
||||
this._sendLocked = true;
|
||||
}
|
||||
this._refreshSendButton();
|
||||
|
||||
if (!retryClientId) {
|
||||
@@ -414,6 +433,7 @@ export class AppChat extends Component {
|
||||
if (!entry) return;
|
||||
clearTimeout(entry.timeoutId);
|
||||
this._pendingSends.delete(clientId);
|
||||
this._releaseSendLock(clientId);
|
||||
this._refreshSendButton();
|
||||
const bubble = this.thread ? this.thread.querySelector(`.message-bubble[data-client-id="${clientId}"]`) : null;
|
||||
if (bubble && !bubble.dataset.msgUid) {
|
||||
@@ -434,6 +454,7 @@ export class AppChat extends Component {
|
||||
if (!entry) return;
|
||||
clearTimeout(entry.timeoutId);
|
||||
this._pendingSends.delete(clientId);
|
||||
this._releaseSendLock(clientId);
|
||||
this._refreshSendButton();
|
||||
}
|
||||
|
||||
@@ -881,9 +902,25 @@ export class AppChat extends Component {
|
||||
_markRead() {
|
||||
if (this._socketReady && this.withUid && this.socket && this._threadIsActive()) {
|
||||
this.socket.send({ type: "read", with_uid: this.withUid });
|
||||
this._sendActiveMarker();
|
||||
}
|
||||
}
|
||||
|
||||
_sendActiveMarker() {
|
||||
if (this._socketReady && this.withUid && this.socket && this._threadIsActive()) {
|
||||
this.socket.send({ type: "active", with_uid: this.withUid });
|
||||
}
|
||||
}
|
||||
|
||||
_bindActiveConversation() {
|
||||
this._onWindowFocus = () => this._sendActiveMarker();
|
||||
this._onVisibilityChange = () => {
|
||||
if (document.visibilityState === "visible") this._sendActiveMarker();
|
||||
};
|
||||
window.addEventListener("focus", this._onWindowFocus);
|
||||
document.addEventListener("visibilitychange", this._onVisibilityChange);
|
||||
}
|
||||
|
||||
_showTyping() {
|
||||
if (!this.typingEl) return;
|
||||
this.typingEl.hidden = false;
|
||||
@@ -989,6 +1026,7 @@ export class AppChat extends Component {
|
||||
_scrollThreadToEnd() {
|
||||
if (!this.thread || !this._userAtBottom) return;
|
||||
this.thread.scrollTop = this.thread.scrollHeight;
|
||||
this._lastForcedScrollTop = this.thread.scrollTop;
|
||||
}
|
||||
|
||||
_startScrollWatcher() {
|
||||
@@ -1020,6 +1058,7 @@ export class AppChat extends Component {
|
||||
}
|
||||
this._stabilizeFrames++;
|
||||
this.thread.scrollTop = this.thread.scrollHeight;
|
||||
this._lastForcedScrollTop = this.thread.scrollTop;
|
||||
const atBottom = this.thread.scrollHeight - this.thread.scrollTop - this.thread.clientHeight < 10;
|
||||
if (!atBottom) {
|
||||
this._stabilizePending = true;
|
||||
@@ -1035,13 +1074,16 @@ export class AppChat extends Component {
|
||||
_bindAutoScroll() {
|
||||
if (!this.thread) return;
|
||||
this.thread.addEventListener("scroll", () => {
|
||||
if (this._stabilizePending) return;
|
||||
const isOwnForcedScroll = this._stabilizePending && this.thread.scrollTop === this._lastForcedScrollTop;
|
||||
if (isOwnForcedScroll) return;
|
||||
const atBottom = this.thread.scrollHeight - this.thread.scrollTop - this.thread.clientHeight < AUTO_SCROLL_MARGIN_PX;
|
||||
this._userAtBottom = atBottom;
|
||||
if (atBottom) {
|
||||
this._newCount = 0;
|
||||
this._updateJumpButton();
|
||||
this._stabilizeScroll();
|
||||
} else {
|
||||
this._stabilizeFrames = 0;
|
||||
}
|
||||
if (this.thread.scrollTop < 48) this._loadOlder();
|
||||
}, { passive: true });
|
||||
@@ -1552,7 +1594,7 @@ export class AppChat extends Component {
|
||||
item.href = `/messages?with_uid=${r.uid}`;
|
||||
const label = document.createElement("span");
|
||||
label.textContent = r.username;
|
||||
item.append(Avatar.imgElement(r.username), label);
|
||||
item.append(Avatar.imgElement(r.username, 24, r.avatar_seed), label);
|
||||
dropdown.appendChild(item);
|
||||
}
|
||||
DomUtils.show(dropdown);
|
||||
|
||||
@@ -26,6 +26,7 @@
|
||||
{% endif %}
|
||||
<div class="comment-actions">
|
||||
<button type="button" class="comment-action-btn" data-action="reply"{{ guest_disabled(user) }}><span class="icon">π¬</span><span class="label"> Reply</span></button>
|
||||
<button type="button" class="comment-action-btn" data-share="#comment-{{ item.comment['uid'] }}"><span class="icon">π</span><span class="label"> Copy link</span></button>
|
||||
{% if owns(item.comment, user) %}
|
||||
<button type="button" class="comment-action-btn" data-action="edit" data-edit-url="/comments/edit/{{ item.comment['uid'] }}"><span class="icon">βοΈ</span><span class="label"> Edit</span></button>
|
||||
{% endif %}
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
<div class="note-widget" data-note-type="{{ _type }}" data-note-uid="{{ _uid }}">
|
||||
<button type="button" class="post-action-btn note-btn{% if _note %} has-note{% endif %}" data-note-toggle title="Personal note" aria-label="Personal note" aria-expanded="false"{{ guest_disabled(user) }}>
|
||||
<span class="note-icon">📝</span> <span class="note-label">{% if _note %}Edit note{% else %}Add note{% endif %}</span>
|
||||
</button>
|
||||
<div class="note-editor" hidden>
|
||||
<textarea class="note-editor-textarea" maxlength="4000" placeholder="Write a private note only you can see..." data-saved="{{ _note or '' }}">{{ _note or "" }}</textarea>
|
||||
<div class="note-editor-actions">
|
||||
<button type="button" class="btn btn-secondary btn-sm note-editor-cancel">Cancel</button>
|
||||
<button type="button" class="btn btn-danger btn-sm note-editor-delete"{% if not _note %} hidden{% endif %}>Delete</button>
|
||||
<button type="button" class="btn btn-primary btn-sm note-editor-save">Save</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -18,7 +18,7 @@
|
||||
{% if maturity_hidden(item.maturity, user) %}
|
||||
{% set _level = item.maturity %}{% include "_maturity_gate.html" %}
|
||||
{% else %}
|
||||
<div class="post-content rendered-content">{{ render_content(item.post['content'][:300] ~ ('...' if item.post['content']|length > 300 else ''), author_is_admin=is_admin(item.author)) }}</div>
|
||||
<div class="post-content rendered-content">{{ render_content(safe_truncate(item.post['content'], 300) ~ ('...' if item.post['content']|length > 300 else ''), author_is_admin=is_admin(item.author)) }}</div>
|
||||
{% endif %}
|
||||
|
||||
{% if item.project_link %}
|
||||
|
||||
@@ -74,22 +74,18 @@
|
||||
<a href="/news" class="topnav-link {{ nav_active(request, '/news') }}"><span class="icon">π°</span> News</a>
|
||||
<a href="/gists" class="topnav-link {{ nav_active(request, '/gists') }}"><span class="icon">π</span> Gists</a>
|
||||
<a href="/projects" class="topnav-link {{ nav_active(request, '/projects') }}"><span class="icon">π</span> Projects</a>
|
||||
<a href="/quizzes" class="topnav-link {{ nav_active(request, '/quizzes') }}"><span class="icon">π§©</span> Quizzes</a>
|
||||
<a href="/battles" class="topnav-link {{ nav_active(request, '/battles') }}"><span class="icon">⚔︎</span> Battles</a>
|
||||
{% if user %}<a href="/game" class="topnav-link {{ nav_active(request, '/game') }}"><span class="icon">π±</span> Farm</a>{% endif %}
|
||||
</div>
|
||||
<div class="topnav-right">
|
||||
<a href="/quizzes" class="topnav-icon {{ nav_active(request, '/quizzes') }}" title="Quizzes" aria-label="Quizzes">
|
||||
<span class="nav-bell" aria-hidden="true">π§©</span>
|
||||
</a>
|
||||
<a href="/battles" class="topnav-icon {{ nav_active(request, '/battles') }}" title="Battles" aria-label="Battles">
|
||||
<span class="nav-bell" aria-hidden="true">⚔︎</span>
|
||||
</a>
|
||||
<a href="/leaderboard" class="topnav-icon {{ nav_active(request, '/leaderboard') }}" title="Leaderboard" aria-label="Leaderboard">
|
||||
<span class="nav-bell" aria-hidden="true">π</span>
|
||||
</a>
|
||||
<div class="topnav-tools-dropdown {{ nav_active(request, '/tools') }}">
|
||||
<button type="button" class="topnav-icon topnav-tools-toggle" aria-expanded="false" aria-controls="tools-menu" title="Tools" aria-label="Tools"><span class="nav-bell" aria-hidden="true">π§°</span></button>
|
||||
<div class="dropdown-menu" id="tools-menu">
|
||||
<a href="/tools/seo" class="dropdown-item"><span class="icon">π</span> SEO Diagnostics</a>
|
||||
<a href="/tools/deepsearch" class="dropdown-item"><span class="icon">π§ </span> DeepSearch</a>
|
||||
<a href="/tools/isslop" class="dropdown-item"><span class="icon">π§ͺ</span> AI Usage Analyzer</a>
|
||||
</div>
|
||||
</div>
|
||||
{% if user %}
|
||||
{% set msg_unread = get_unread_messages(user["uid"]) %}
|
||||
<a href="/messages" class="topnav-icon {{ nav_active(request, '/messages') }}" data-counter="messages" data-counter-noun="messages" title="Messages" aria-label="{% if msg_unread %}{{ msg_unread }} unread messages{% else %}Messages, no unread{% endif %}">
|
||||
@@ -119,6 +115,7 @@
|
||||
<a href="/profile/{{ user['username'] }}" class="dropdown-item"><span class="icon">π€</span> Profile</a>
|
||||
<a href="/devii/" class="dropdown-item" data-devii-open><span class="icon">π€</span> Devii</a>
|
||||
<a href="/bookmarks/saved" class="dropdown-item"><span class="icon">π</span> Saved</a>
|
||||
<a href="/notes/saved" class="dropdown-item"><span class="icon">π</span> Notes</a>
|
||||
<a href="/auth/logout" class="dropdown-item"><span class="icon">πͺ</span> Logout</a>
|
||||
</div>
|
||||
</div>
|
||||
@@ -149,11 +146,6 @@
|
||||
<a href="/battles" class="topnav-mobile-link {{ nav_active(request, '/battles') }}"><span class="icon">⚔︎</span> Battles</a>
|
||||
<a href="/leaderboard" class="topnav-mobile-link {{ nav_active(request, '/leaderboard') }}"><span class="icon">π</span> Leaderboard</a>
|
||||
{% if user %}<a href="/game" class="topnav-mobile-link {{ nav_active(request, '/game') }}"><span class="icon">π±</span> Farm</a>{% endif %}
|
||||
<div class="topnav-mobile-divider"></div>
|
||||
<div class="topnav-mobile-section-label">Tools</div>
|
||||
<a href="/tools/seo" class="topnav-mobile-link {{ nav_active(request, '/tools/seo') }}"><span class="icon">π</span> SEO Diagnostics</a>
|
||||
<a href="/tools/deepsearch" class="topnav-mobile-link {{ nav_active(request, '/tools/deepsearch') }}"><span class="icon">π§ </span> DeepSearch</a>
|
||||
<a href="/tools/isslop" class="topnav-mobile-link {{ nav_active(request, '/tools/isslop') }}"><span class="icon">π§ͺ</span> AI Usage Analyzer</a>
|
||||
{% if user %}
|
||||
<a href="/messages" class="topnav-mobile-link {{ nav_active(request, '/messages') }}" data-counter="messages" data-counter-noun="messages" aria-label="{% if get_unread_messages(user['uid']) %}Messages, {{ get_unread_messages(user['uid']) }} unread{% else %}Messages, no unread{% endif %}"><span class="icon">βοΈ</span> Messages<span class="nav-badge nav-badge-inline" data-counter-badge {% if get_unread_messages(user["uid"]) == 0 %}hidden{% endif %} aria-hidden="true">{{ get_unread_messages(user["uid"]) }}</span></a>
|
||||
<a href="/notifications" class="topnav-mobile-link {{ nav_active(request, '/notifications') }}" data-counter="notifications" data-counter-noun="notifications" aria-label="{% if get_unread_count(user['uid']) %}Notifications, {{ get_unread_count(user['uid']) }} unread{% else %}Notifications, no unread{% endif %}"><span class="icon">π</span> Notifications<span class="nav-badge nav-badge-inline" data-counter-badge {% if get_unread_count(user["uid"]) == 0 %}hidden{% endif %} aria-hidden="true">{{ get_unread_count(user["uid"]) }}</span></a>
|
||||
@@ -184,6 +176,7 @@
|
||||
<a href="/profile/{{ user['username'] }}" class="topnav-mobile-link"><span class="icon">π€</span> Profile</a>
|
||||
<a href="/devii/" class="topnav-mobile-link" data-devii-open><span class="icon">π€</span> Devii</a>
|
||||
<a href="/bookmarks/saved" class="topnav-mobile-link {{ nav_active(request, '/bookmarks') }}"><span class="icon">π</span> Saved</a>
|
||||
<a href="/notes/saved" class="topnav-mobile-link {{ nav_active(request, '/notes') }}"><span class="icon">π</span> Notes</a>
|
||||
<a href="/auth/logout" class="topnav-mobile-link"><span class="icon">πͺ</span> Logout</a>
|
||||
{% else %}
|
||||
<div class="topnav-mobile-divider"></div>
|
||||
|
||||
@@ -90,5 +90,6 @@ Controllers find and enhance markup through declarative `data-*` attributes rath
|
||||
| `data-confirm` | Show a confirm dialog before the action. |
|
||||
| `data-copy` | Copy the referenced content to the clipboard. |
|
||||
| `data-share` | Copy a share URL. |
|
||||
| `data-view-toggle` / `data-view-toggle-alt` | Swap visibility between two elements (e.g. a raw/rendered pair), swapping the button's own label via `data-view-toggle-label` / `data-view-toggle-label-alt`. |
|
||||
| `data-mention` | Attach `@mention` autocomplete to a textarea. |
|
||||
</div>
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user