Add pruning and retention policy for devii_lessons table to prevent unbounded growth #77

New Issue
Closed
opened 2026-07-13 00:37:31 +02:00 by retoor · 0 comments
retoor commented 2026-07-13 00:37:31 +02:00
Owner
Copy Link

Summary

The devii_lessons table has grown to 3827 records (soft-deleted) with no pruning, retention, or deduplication policy. Every reflect() call stores a permanent lesson used by recall() queries, causing performance degradation and accumulation of duplicate/irrelevant lessons.

Steps to Reproduce

  1. Call reflect() multiple times for the same user/guest session.
  2. Observe that each call inserts a new record into devii_lessons without any limit or deduplication.
  3. Run recall() queries and note that old, duplicate, or irrelevant lessons are still returned.
  4. Check the table row count; it grows unbounded per account/session.

Expected Behaviour

  • The devii_lessons table should have a maximum lesson count per user/guest session (e.g., 500 active lessons).
  • Lessons older than a configurable number of days should be automatically pruned.
  • Similar or duplicate lessons should be merged or deduplicated instead of stacking.
  • A mechanism (e.g., quality/thumbs-up system) should allow irrelevant lessons to be silently dropped.
  • recall() queries should remain performant as the table size is bounded.

Actual Behaviour

  • The table contains 3827 records (soft-deleted) with no cap, retention policy, or deduplication.
  • recall() queries slow down as the table grows.
  • Duplicate lessons about the same topic (e.g., container sync issues) accumulate.
  • Old, irrelevant lessons continue to match in recall results.
  • The database size is 44 KB and grows with every session.

Environment

Not provided.

Reported by retoor via DevPlace.

## Summary The `devii_lessons` table has grown to 3827 records (soft-deleted) with no pruning, retention, or deduplication policy. Every `reflect()` call stores a permanent lesson used by `recall()` queries, causing performance degradation and accumulation of duplicate/irrelevant lessons. ## Steps to Reproduce 1. Call `reflect()` multiple times for the same user/guest session. 2. Observe that each call inserts a new record into `devii_lessons` without any limit or deduplication. 3. Run `recall()` queries and note that old, duplicate, or irrelevant lessons are still returned. 4. Check the table row count; it grows unbounded per account/session. ## Expected Behaviour - The `devii_lessons` table should have a maximum lesson count per user/guest session (e.g., 500 active lessons). - Lessons older than a configurable number of days should be automatically pruned. - Similar or duplicate lessons should be merged or deduplicated instead of stacking. - A mechanism (e.g., quality/thumbs-up system) should allow irrelevant lessons to be silently dropped. - `recall()` queries should remain performant as the table size is bounded. ## Actual Behaviour - The table contains 3827 records (soft-deleted) with no cap, retention policy, or deduplication. - `recall()` queries slow down as the table grows. - Duplicate lessons about the same topic (e.g., container sync issues) accumulate. - Old, irrelevant lessons continue to match in recall results. - The database size is 44 KB and grows with every session. ## Environment Not provided. --- *Reported by **retoor** via DevPlace.*
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
No items
No Label
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: retoor/devplacepy#77
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?