Commit Graph
26 Commits
Author SHA1 Message Date
retoorandClaude Sonnet 5.5 02ab747cd0 Code Farm: per-plot auto-replant pause and crop-lock check; community sync and node work
DevPlace CI / test (push) Failing after 44m45s
Continuous Delivery could never be turned off once a crop was planted. Add a
one-shot per-plot "stop after harvest" flag (POST /game/plot/stop, Devii tool
game_plot_stop, game_plots.stop_after_harvest), cleared by the next plant,
autoreplant claim or Refactor. Autoreplant now also respects crop locks through
the shared _crop_lock predicate. Plan and measurements in happyplanting.md.

Also includes the pending community sync, node network and related work in the
tree; the full suite passed (4999) with all of it.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-17 02:23:43 +02:00
retoorandClaude Sonnet 5 7880bf4b31 Fix container sync races that leaked orphan blobs; add a system-prune CLI command
sync_workspace (user-triggered) and the reconciler's sync_bidirectional_sync
could run concurrently for the same project, and store_upload's read-then-
write on a changed path meant two racing imports each wrote their own blob
while only one ever got referenced - the loser leaked forever. Combined with
no build-artifact exclusion, an actively-compiling workspace hit this
constantly and leaked 5.9M orphan blobs (~96GB) in production before it was
caught.

Closes it at the root: api._sync_dir_bidirectional_locked serializes both
call sites per-project (non-blocking - a project already mid-sync is simply
skipped until the next tick), and IMPORT_SKIP_NAMES/IMPORT_SKIP_EXTENSIONS
keep build output (build/, dist/, *.o, *.pyc, ...) out of the walk entirely.

Recovering what already leaked is a separate concern: a new CLI subcommand
(plus matching make targets) sweeps soft-deleted attachment/project-file
blobs and any blob with zero DB reference at all, plus orphaned container
workspace directories. run_maintenance_cleanup.sh wraps the existing
prune/clear commands for routine disk upkeep.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BWJy6PrMMt5hwWxQwia2rd
2026-09-08 03:43:49 +02:00
retoorandClaude Sonnet 5 4a13415b43 Fix blocking sync I/O on the event loop across containers, jobs, and xmlrpc
DevPlace CI / test (push) Failing after 26m53s
The container/workspace reachability probes (socket connect + HTTP check)
ran synchronously with real timeouts inside async request handlers and the
live-view relay's 3-4s ticks, freezing the whole event loop whenever a
container wasn't cleanly reachable - the likely cause of the periodic
app-wide stalls. Converted the probe chain (api._port_reachable/_http_probe,
editor_reachable, instance_runtime, provision.editor_ready/view) to real
async I/O and parallelized the admin container/workspace list decorators.

Also fixes: XmlrpcService.on_disable blocked up to 10s on a synchronous
subprocess.wait inside an async method (now matches TelegramService's
async-subprocess pattern); JobService._sweep_expired ran every job kind's
cleanup() - including shutil.rmtree on large directories - synchronously
on every tick, now offloaded via asyncio.to_thread for all job kinds at
once; and several smaller blocking reads/writes on request/service paths
(attachment-to-gitea mirroring, stealth chunked downloads, dbapi/isslop
file reads, job payload/report I/O) moved off the loop thread.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-03 12:59:10 +00:00
retoor 81d6aa68b6 Optimization 2026-09-03 08:47:57 +02:00
retoorandClaude Sonnet 5 57087536e5 Attribute Devii AI spend to its invoking action, fix quiz question-at-a-time review, DB API/isslop result routes, workspace docs, and drop redundant docstrings
- Route Devii-driven AI gateway cost to the action/tool that triggered
  it instead of a blanket "internal" bucket, so per-feature AI spend
  is attributable.
- Fix the quiz attempt review to show one previously-answered question
  at a time instead of all of them at once, and stop a quiz endpoint
  linked from the quiz flow from responding with raw JSON.
- Add DB API async query result route and AI Usage Analyzer annotated
  source/media routes, with traversal-safe uid/path handling and
  matching tests.
- Add Code Farm action audit logging (plant/harvest/buy-plot/upgrade/
  fertilize) and related admin workspace/services/trash/gateway route
  and doc touch-ups.
- Drop redundant docstrings from access_tokens.py per the no-comments
  convention.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VL9Xn57W5UR3HZbbuuzxdK
2026-09-03 08:47:57 +02:00
retoor 5774d83ece Add attachment management CRUD to the /uploads API
Complete the read and update faces of the signed-in user's attachment
management over the existing attachments table:

- GET /uploads: paginated list of the user's own attachments, newest
  first, with an optional linked/orphaned filter
- GET /uploads/{uid}: fetch one attachment (owner or admin)
- PATCH /uploads/{uid}: rename the display filename, always preserving
  the original extension (owner or admin, audited as attachment.rename)

Adds get_user_attachments/get_user_attachment data helpers, the
rename_attachment operation, AttachmentRenameForm, the UploadItemOut and
UploadsListOut schemas, the Devii tools list_attachments/get_attachment/
rename_attachment, expanded API reference documentation for the full
lifecycle including delete, and api-tier tests.
2026-07-26 16:46:41 +02:00
retoor c53e2a3319 Update 2026-07-19 18:57:43 +02:00
retoor 7bbaf51450 feat: add file attachment support to issue tracker with Gitea mirroring
Implement full attachment CRUD for issues and comments, restricted to open issues only. Attachments are stored locally and mirrored to the Gitea tracker via new `mirror_attachment_to_gitea` and `set_gitea_asset_id` functions. Add `gitea_asset_id` column to the attachments table, extend `IssueForm` and `IssueCommentForm` with `attachment_uids` field, and expose new API endpoints for listing, adding, and deleting issue attachments. Update agent configuration to include web research tools, and document the new capability in README, AGENTS.md, and CLAUDE.md.
2026-06-19 11:22:05 +00:00
retoor 9598a1b867 feat: add wildcard token support for allowed extensions and access token CLI commands
Introduce WILDCARD_TOKENS set in attachments.py to treat "*", ".*", "*.*" as wildcards that fall back to ALLOWED_UPLOAD_TYPES. Add cmd_token_issue and cmd_token_list CLI commands for issuing and listing DevPlace access tokens. Register access_tokens table in SOFT_DELETE_TABLES and initialize its columns and indexes in init_db. Replace Form() with Depends(json_or_form(...)) in admin backup, container, notification, settings, and user routers to support both JSON and form data.
2026-06-17 17:10:52 +00:00
retoor a618a95671 feat: remove .html and .svg from allowed upload types and MIME mappings
Remove HTML and SVG file extensions from the ALLOWED_UPLOAD_TYPES dictionary and their corresponding MIME type entries from MIME_TO_EXT in attachments.py, preventing users from uploading these potentially unsafe file formats through the API.
2026-06-16 06:50:16 +00:00
retoor 6b08761af5 feat: add audio file support and expand allowed upload types with new MIME mappings
Extend ALLOWED_UPLOAD_TYPES dict with audio formats (wav, flac, ogg, aac, wma, m4a), video formats (avi, mkv, flv, wmv, 3gp), document formats (doc, docx, xls, xlsx, ppt, pptx, odt, rtf), code formats (ts, java, cpp, c, h, rb, go, rs, sql, php, swift, kt), config formats (cfg, ini, log), archive formats (tar, gz, rar, 7z), and additional web formats (html, json, xml, yaml, yml, toml, sh, bat, svg). Add corresponding MIME_TO_EXT reverse mappings. Implement audio element rendering in ContentRenderer.js with audioExtRe regex and createAudioElement method. Add gallery-audio CSS class for audio player styling in attachments.css and media.css. Update _attachment_display.html and _media_gallery.html templates to render audio controls for is_audio attachments.
2026-06-15 15:08:10 +00:00
retoor f2b910ea75 fix: correct "bugs" to "issues" in routing table and README references across multiple documentation files 2026-06-14 07:48:10 +00:00
retoor 5c199474f3 fix: switch blob sharding from uuid7 leading hex to random tail in attachments, project_files, and zip_service 2026-06-13 19:12:54 +00:00
retoor 1294c95bed chore: consolidate runtime data layout under single data/ root directory
Migrate all runtime artifacts (database, uploads, VAPID keys, locks, bot state, container workspaces, zip staging) from scattered locations (`var/`, `devplacepy/static/uploads/`) into a unified `data/` directory. Update `config.py` as single source of truth with `DATA_PATHS` registry and `ensure_data_dirs()`, add `devplace migrate-data` CLI command with CRC verification and idempotent relocation, adjust `.dockerignore`, `.env.example`, `.gitignore`, `Dockerfile`, `Makefile`, `README.md`, `AGENTS.md`, `CLAUDE.md`, and all import paths in `attachments.py` to reference `config.UPLOADS_DIR`/`ATTACHMENTS_DIR` instead of computing from `STATIC_DIR`.
2026-06-13 19:06:43 +00:00
retoor 9a93debfc9 chore: document project-wide soft delete pattern and add deleted_by column to all tables 2026-06-11 20:36:47 +00:00
retoor 7fc9b0f715 chore: add retoor header to all devplacepy source files and update style agent header rule
The diff adds the mandatory `retoor <retoor@molodetz.nl>` header comment to every `.py` file under `devplacepy/` (including `__init__.py` and `routers/__init__.py`), and updates the `StyleAgent` in `agents/style.py` to instruct agents to only add the header on created or already-edited files rather than sweeping the entire repo. The `agents/base.py` operating protocol is also revised to reorder and clarify tool discipline rules.
2026-06-11 23:58:46 +00:00
retoor 3862958fae feat: add soft delete and media tab for user attachments with admin restore 2026-06-11 18:52:56 +00:00
retoor 3540bc8fa6 feat: add remote URL attachment support and project editing endpoint 2026-06-10 22:17:25 +00:00
retoor c4f2937415 fix: normalize unicode escape sequences and reformat multi-line expressions across codebase 2026-06-09 16:48:08 +00:00
retoor 97eb58fc19 feat: add project file system with CRUD, upload, inline editing, and video attachment support
- Add new `/projects/{slug}/files` endpoint group for per-project filesystem operations including directory and file CRUD, upload, and inline editing with public read and owner write access
- Extend attachment system to support video formats (webm, ogv, mov, m4v) with proper file icons and MIME types
- Implement configurable allowed file types via `allowed_file_types` site setting, replacing hardcoded `ALLOWED_UPLOAD_TYPES` with dynamic `allowed_extensions()` and `is_extension_allowed()` functions
- Add `delete_all_project_files()` call in `delete_content_item()` to clean up project files when a project is deleted
- Create database indexes on `project_files` table for `(project_uid, path)` and `(project_uid, parent_path)` to optimize file lookups
- Introduce `docs_prose.py` module with `render_prose()` function that renders Markdown content inside `data-render` divs using mistune, enabling dynamic prose rendering in documentation pages
- Enhance docs search with Markdown-aware text stripping (`_demarkdown()`) and improved HTML/script/style sanitization for better search indexing
- Update documentation API samples to reflect new attachment response fields (`is_image`, `is_video`, `mime_type`) and note video format support
- Update README to document the new project files endpoint and clarify AI gateway attribution for guest Devii sessions
2026-06-08 20:51:09 +00:00
retoor 02b07dce8e feat: add thumbnail_name field to attachment storage and refactor link/delete to batch SQL queries 2026-06-05 02:43:06 +00:00
retoor 4cdf49cf8f chore: remove deprecated demo make target and update attachment thumbnail resolution
- Remove `make demo` target from Makefile, AGENTS.md, and README.md
- Improve `_row_to_attachment` to glob for thumbnail files with any extension instead of hardcoding `.jpg`
- Rewrite `cmd_attachments_prune` to use time-based cutoff and `delete_attachment` helper instead of orphan detection
- Add cursor-based pagination to feed and notifications endpoints with `before` parameter
- Track `did_upvote` flag in vote handler to only notify on new upvotes, not vote changes
- Add `PAGE_SIZE` constant and `next_cursor` template variable to notifications page
- Implement Playwright tests for notification pagination with 30 seeded notifications
2026-05-28 22:45:07 +00:00
retoor 74571f7737 feat: replace legacy vote buttons with AJAX submission and dynamic count updates
Replace static form-based vote buttons with a new VoteManager class that submits votes via fetch, returns JSON with net/up/down/current value, and updates vote count spans using data-vote-count attributes across all templates. Add JSONResponse endpoint in votes router for AJAX requests, switch VoteManager import from Http to Toast for error feedback, and refactor link_attachments to handle comma-separated UIDs.
2026-05-27 19:06:18 +00:00
retoor 8eea7cc0c9 feat: add news sanitize CLI command and upload static file serving with content-disposition 2026-05-22 23:50:31 +00:00
retoor 34cae11630 fix: replace deprecated datetime.utcnow with timezone-aware datetime.now in all modules and fix comment count query column name 2026-05-14 02:12:19 +00:00
retoor aa88f18c03 chore: add docker infrastructure, gists feature, attachment system, and admin CLI tools
- Add .dockerignore, Dockerfile, and docker compose targets to Makefile for containerized deployment
- Implement gists router with CRUD operations, database schema, and polymorphic comment/vote reuse
- Create attachment upload system with thumbnail generation, MIME detection, and storage path management
- Add attachments_prune CLI command to clean orphaned attachment records and files
- Introduce rate limiting middleware with 60 requests per minute window
- Add custom 404 and 500 error handlers with SEO-optimized template responses
- Extend database initialization with gists and attachments indexes plus upload site settings defaults
- Update load_comments to include attachment mapping for comment resources
- Register gists and uploads routers in main application and update AGENTS.md documentation
2026-05-12 13:07:34 +00:00