feat: add distributed async dataset with unix socket server and refactor agent communication bus

Implement AsyncDataSet class supporting client-server model over Unix sockets with SQLite backend, including KV store, table management, and concurrent query handling. Rename `get_messages` to `receive_messages` in AgentCommunicationBus and update all callers. Remove deprecated `get_recommended_agent` function from agent_roles, `invalidate_tool` from tool_cache, and legacy `receive_messages` wrapper. Add multiplexer command routing in handlers with `/prompt` command support. Introduce comprehensive help documentation system for workflows. Update default API URLs to production endpoints and refactor adaptive context window calculation in AdvancedContextManager.
2025-11-06 14:15:06 +00:00
parent ab9c29467c
commit 2c3749fd58
48 changed files with 6395 additions and 1062 deletions
@@ -1,6 +1,7 @@
 import json
 import time

+from pr.commands.multiplexer_commands import MULTIPLEXER_COMMANDS
 from pr.autonomous import run_autonomous_mode
 from pr.core.api import list_models
 from pr.tools import read_file
@@ -13,6 +14,12 @@ def handle_command(assistant, command):
    command_parts = command.strip().split(maxsplit=1)
    cmd = command_parts[0].lower()

+    if cmd in MULTIPLEXER_COMMANDS:
+        # Pass the assistant object and the remaining arguments to the multiplexer command
+        return MULTIPLEXER_COMMANDS[cmd](
+            assistant, command_parts[1:] if len(command_parts) > 1 else []
+        )
+
    if cmd == "/edit":
        rp_editor = RPEditor(command_parts[1] if len(command_parts) > 1 else None)
        rp_editor.start()
@@ -23,6 +30,18 @@ def handle_command(assistant, command):
        if task:
            run_autonomous_mode(assistant, task)

+    elif cmd == "/prompt":
+        rp_editor = RPEditor(command_parts[1] if len(command_parts) > 1 else None)
+        rp_editor.start()
+        rp_editor.thread.join()
+        prompt_text = str(rp_editor.get_text())
+        rp_editor.stop()
+        rp_editor = None
+        if prompt_text.strip():
+            from pr.core.assistant import process_message
+
+            process_message(assistant, prompt_text)
+
    elif cmd == "/auto":
        if len(command_parts) < 2:
            print(f"{Colors.RED}Usage: /auto [task description]{Colors.RESET}")
@@ -0,0 +1,507 @@
+from pr.ui import Colors
+
+
+def get_workflow_help():
+    return f"""
+{Colors.BOLD}WORKFLOWS - AUTOMATED TASK EXECUTION{Colors.RESET}
+
+{Colors.BOLD}Overview:{Colors.RESET}
+Workflows enable automated execution of multi-step tasks by chaining together
+tool calls with defined execution logic, error handling, and conditional branching.
+
+{Colors.BOLD}Execution Modes:{Colors.RESET}
+  {Colors.CYAN}sequential{Colors.RESET}  - Steps execute one after another in order
+  {Colors.CYAN}parallel{Colors.RESET}    - All steps execute simultaneously
+  {Colors.CYAN}conditional{Colors.RESET} - Steps execute based on conditions and paths
+
+{Colors.BOLD}Workflow Components:{Colors.RESET}
+  {Colors.CYAN}Steps{Colors.RESET}        - Individual tool executions with parameters
+  {Colors.CYAN}Variables{Colors.RESET}    - Dynamic values available across workflow
+  {Colors.CYAN}Conditions{Colors.RESET}   - Boolean expressions controlling step execution
+  {Colors.CYAN}Paths{Colors.RESET}        - Success/failure routing between steps
+  {Colors.CYAN}Retries{Colors.RESET}      - Automatic retry count for failed steps
+  {Colors.CYAN}Timeouts{Colors.RESET}     - Maximum execution time per step (seconds)
+
+{Colors.BOLD}Variable Substitution:{Colors.RESET}
+  Reference previous step results:  ${{step.step_id}}
+  Reference workflow variables:     ${{var.variable_name}}
+
+  Example: {{"path": "${{step.get_path}}/output.txt"}}
+
+{Colors.BOLD}Workflow Commands:{Colors.RESET}
+  {Colors.CYAN}/workflows{Colors.RESET}           - List all available workflows
+  {Colors.CYAN}/workflow <name>{Colors.RESET}     - Execute a specific workflow
+
+{Colors.BOLD}Workflow Structure:{Colors.RESET}
+Each workflow consists of:
+  - Name and description
+  - Execution mode (sequential/parallel/conditional)
+  - List of steps with tool calls
+  - Optional variables for dynamic values
+  - Optional tags for categorization
+
+{Colors.BOLD}Step Configuration:{Colors.RESET}
+Each step includes:
+  - tool_name: Which tool to execute
+  - arguments: Parameters passed to the tool
+  - step_id: Unique identifier for the step
+  - condition: Optional boolean expression (conditional mode)
+  - on_success: List of step IDs to execute on success
+  - on_failure: List of step IDs to execute on failure
+  - retry_count: Number of automatic retries (default: 0)
+  - timeout_seconds: Maximum execution time (default: 300)
+
+{Colors.BOLD}Conditional Execution:{Colors.RESET}
+Conditions are Python expressions evaluated with access to:
+  - variables: All workflow variables
+  - results: All step results by step_id
+
+  Example condition: "results['check_file']['status'] == 'success'"
+
+{Colors.BOLD}Success/Failure Paths:{Colors.RESET}
+Steps can define:
+  - on_success: List of step IDs to execute if step succeeds
+  - on_failure: List of step IDs to execute if step fails
+
+This enables error recovery and branching logic within workflows.
+
+{Colors.BOLD}Workflow Storage:{Colors.RESET}
+Workflows are stored in the SQLite database and persist across sessions.
+Execution history and statistics are tracked for each workflow.
+
+{Colors.BOLD}Example Workflow Scenarios:{Colors.RESET}
+  - Sequential: Data pipeline (fetch → process → save → validate)
+  - Parallel: Multi-source aggregation (fetch A + fetch B + fetch C)
+  - Conditional: Error recovery (try main → on failure → fallback)
+
+{Colors.BOLD}Best Practices:{Colors.RESET}
+  - Use descriptive step_id names (e.g., "fetch_user_data")
+  - Set appropriate timeouts for long-running operations
+  - Add retry_count for network or flaky operations
+  - Use conditional paths for error handling
+  - Keep workflows focused on a single logical task
+  - Use variables for values shared across steps
+"""
+
+
+def get_agent_help():
+    return f"""
+{Colors.BOLD}AGENTS - SPECIALIZED AI ASSISTANTS{Colors.RESET}
+
+{Colors.BOLD}Overview:{Colors.RESET}
+Agents are specialized AI instances with specific roles, capabilities, and
+system prompts. Each agent maintains its own conversation history and can
+collaborate with other agents to complete complex tasks.
+
+{Colors.BOLD}Available Agent Roles:{Colors.RESET}
+  {Colors.CYAN}coding{Colors.RESET}          - Code writing, review, debugging, refactoring
+                     Temperature: 0.3 (precise)
+
+  {Colors.CYAN}research{Colors.RESET}        - Information gathering, analysis, documentation
+                     Temperature: 0.5 (balanced)
+
+  {Colors.CYAN}data_analysis{Colors.RESET}   - Data processing, statistical analysis, queries
+                     Temperature: 0.3 (precise)
+
+  {Colors.CYAN}planning{Colors.RESET}        - Task decomposition, workflow design, coordination
+                     Temperature: 0.6 (creative)
+
+  {Colors.CYAN}testing{Colors.RESET}         - Test design, quality assurance, validation
+                     Temperature: 0.4 (methodical)
+
+  {Colors.CYAN}documentation{Colors.RESET}   - Technical writing, API docs, user guides
+                     Temperature: 0.6 (creative)
+
+  {Colors.CYAN}orchestrator{Colors.RESET}    - Multi-agent coordination and task delegation
+                     Temperature: 0.5 (balanced)
+
+  {Colors.CYAN}general{Colors.RESET}         - General purpose assistance across domains
+                     Temperature: 0.7 (versatile)
+
+{Colors.BOLD}Agent Commands:{Colors.RESET}
+  {Colors.CYAN}/agent <role> <task>{Colors.RESET}        - Create agent and assign task
+  {Colors.CYAN}/agents{Colors.RESET}                      - Show all active agents
+  {Colors.CYAN}/collaborate <task>{Colors.RESET}          - Multi-agent collaboration
+
+{Colors.BOLD}Single Agent Usage:{Colors.RESET}
+Create a specialized agent and assign it a task:
+
+  /agent coding Implement a binary search function in Python
+  /agent research Find latest best practices for API security
+  /agent testing Create test cases for user authentication
+
+Each agent:
+  - Maintains its own conversation history
+  - Has access to role-specific tools
+  - Uses specialized system prompts
+  - Tracks task completion count
+  - Integrates with knowledge base
+
+{Colors.BOLD}Multi-Agent Collaboration:{Colors.RESET}
+Use multiple agents to work together on complex tasks:
+
+  /collaborate Build a REST API with tests and documentation
+
+This automatically:
+  1. Creates an orchestrator agent
+  2. Spawns coding, research, and planning agents
+  3. Orchestrator delegates subtasks to specialists
+  4. Agents communicate results back to orchestrator
+  5. Orchestrator integrates results and coordinates
+
+{Colors.BOLD}Agent Capabilities by Role:{Colors.RESET}
+
+{Colors.CYAN}Coding Agent:{Colors.RESET}
+  - Read, write, and modify files
+  - Execute Python code
+  - Run commands and build tools
+  - Index source directories
+  - Code review and refactoring
+
+{Colors.CYAN}Research Agent:{Colors.RESET}
+  - Web search and HTTP fetching
+  - Read documentation and files
+  - Query databases for information
+  - Analyze and synthesize findings
+
+{Colors.CYAN}Data Analysis Agent:{Colors.RESET}
+  - Database queries and operations
+  - Python execution for analysis
+  - File read/write for data
+  - Statistical processing
+
+{Colors.CYAN}Planning Agent:{Colors.RESET}
+  - Task decomposition and organization
+  - Database storage for plans
+  - File operations for documentation
+  - Dependency identification
+
+{Colors.CYAN}Testing Agent:{Colors.RESET}
+  - Test execution and validation
+  - Code reading and analysis
+  - Command execution for test runners
+  - Quality verification
+
+{Colors.CYAN}Documentation Agent:{Colors.RESET}
+  - Technical writing and formatting
+  - Web research for best practices
+  - File operations for docs
+  - Documentation organization
+
+{Colors.BOLD}Agent Session Management:{Colors.RESET}
+  - Each session has a unique ID
+  - Agents persist within a session
+  - View active agents with /agents
+  - Agents share knowledge base access
+  - Communication bus for agent messages
+
+{Colors.BOLD}Knowledge Base Integration:{Colors.RESET}
+When agents execute tasks:
+  - Relevant knowledge entries are automatically retrieved
+  - Top 3 matches are injected into agent context
+  - Knowledge access count is tracked
+  - Agents can update knowledge base
+
+{Colors.BOLD}Agent Communication:{Colors.RESET}
+Agents can send messages to each other:
+  - Request type: Ask another agent for help
+  - Response type: Answer another agent's request
+  - Notification type: Inform about events
+  - Coordination type: Synchronize work
+
+{Colors.BOLD}Best Practices:{Colors.RESET}
+  - Use specialized agents for focused tasks
+  - Use collaboration for complex multi-domain tasks
+  - Coding agent: Precise, deterministic code generation
+  - Research agent: Thorough information gathering
+  - Planning agent: Breaking down complex tasks
+  - Testing agent: Comprehensive test coverage
+  - Documentation agent: Clear user-facing docs
+
+{Colors.BOLD}Performance Characteristics:{Colors.RESET}
+  - Lower temperature: More deterministic (coding, data_analysis)
+  - Higher temperature: More creative (planning, documentation)
+  - Max tokens: 4096 for all agents
+  - Parallel execution: Multiple agents work simultaneously
+  - Message history: Full conversation context maintained
+
+{Colors.BOLD}Example Usage Patterns:{Colors.RESET}
+
+Single specialized agent:
+  /agent coding Refactor authentication module for better security
+
+Multi-agent collaboration:
+  /collaborate Create a data visualization dashboard with tests
+
+Agent inspection:
+  /agents                    # View all active agents and their stats
+"""
+
+
+def get_knowledge_help():
+    return f"""
+{Colors.BOLD}KNOWLEDGE BASE - PERSISTENT INFORMATION STORAGE{Colors.RESET}
+
+{Colors.BOLD}Overview:{Colors.RESET}
+The knowledge base provides persistent storage for information, facts, and
+context that can be accessed across sessions and by all agents.
+
+{Colors.BOLD}Commands:{Colors.RESET}
+  {Colors.CYAN}/remember <content>{Colors.RESET}      - Store information in knowledge base
+  {Colors.CYAN}/knowledge <query>{Colors.RESET}       - Search knowledge base
+  {Colors.CYAN}/stats{Colors.RESET}                    - Show knowledge base statistics
+
+{Colors.BOLD}Features:{Colors.RESET}
+  - Automatic categorization of content
+  - TF-IDF based semantic search
+  - Access frequency tracking
+  - Importance scoring
+  - Temporal relevance
+  - Category-based organization
+
+{Colors.BOLD}Categories:{Colors.RESET}
+Content is automatically categorized as:
+  - code: Code snippets and programming concepts
+  - api: API documentation and endpoints
+  - configuration: Settings and configurations
+  - documentation: General documentation
+  - fact: Factual information
+  - procedure: Step-by-step procedures
+  - general: Uncategorized information
+
+{Colors.BOLD}Search Capabilities:{Colors.RESET}
+  - Keyword matching with TF-IDF scoring
+  - Returns top relevant entries
+  - Considers access frequency
+  - Factors in recency
+  - Searches across all categories
+
+{Colors.BOLD}Integration:{Colors.RESET}
+  - Agents automatically query knowledge base
+  - Top matches injected into agent context
+  - Conversation history can be stored
+  - Facts extracted from interactions
+  - Persistent across sessions
+"""
+
+
+def get_cache_help():
+    return f"""
+{Colors.BOLD}CACHING SYSTEM - PERFORMANCE OPTIMIZATION{Colors.RESET}
+
+{Colors.BOLD}Overview:{Colors.RESET}
+The caching system optimizes performance by storing API responses and tool
+results, reducing redundant API calls and expensive operations.
+
+{Colors.BOLD}Commands:{Colors.RESET}
+  {Colors.CYAN}/cache{Colors.RESET}              - Show cache statistics
+  {Colors.CYAN}/cache clear{Colors.RESET}        - Clear all caches
+
+{Colors.BOLD}Cache Types:{Colors.RESET}
+
+{Colors.CYAN}API Cache:{Colors.RESET}
+  - Stores LLM API responses
+  - Key: Hash of messages + model + parameters
+  - Tracks token usage
+  - TTL: Configurable expiration
+  - Reduces API costs and latency
+
+{Colors.CYAN}Tool Cache:{Colors.RESET}
+  - Stores tool execution results
+  - Key: Tool name + arguments hash
+  - Per-tool statistics
+  - Hit count tracking
+  - Reduces redundant operations
+
+{Colors.BOLD}Cache Statistics:{Colors.RESET}
+  - Total entries (valid + expired)
+  - Valid entry count
+  - Expired entry count
+  - Total cached tokens (API cache)
+  - Cache hits per tool (Tool cache)
+  - Per-tool breakdown
+
+{Colors.BOLD}Benefits:{Colors.RESET}
+  - Reduced API costs
+  - Faster response times
+  - Lower rate limit usage
+  - Consistent results for identical queries
+  - Tool execution optimization
+"""
+
+
+def get_background_help():
+    return f"""
+{Colors.BOLD}BACKGROUND SESSIONS - CONCURRENT TASK EXECUTION{Colors.RESET}
+
+{Colors.BOLD}Overview:{Colors.RESET}
+Background sessions allow running long-running commands and processes while
+continuing to interact with the assistant. Sessions are monitored and can
+be managed through the assistant.
+
+{Colors.BOLD}Commands:{Colors.RESET}
+  {Colors.CYAN}/bg start <command>{Colors.RESET}       - Start command in background
+  {Colors.CYAN}/bg list{Colors.RESET}                   - List all background sessions
+  {Colors.CYAN}/bg status <name>{Colors.RESET}          - Show session status
+  {Colors.CYAN}/bg output <name>{Colors.RESET}          - View session output
+  {Colors.CYAN}/bg input <name> <text>{Colors.RESET}    - Send input to session
+  {Colors.CYAN}/bg kill <name>{Colors.RESET}            - Terminate session
+  {Colors.CYAN}/bg events{Colors.RESET}                 - Show recent events
+
+{Colors.BOLD}Features:{Colors.RESET}
+  - Automatic session monitoring
+  - Output capture and buffering
+  - Interactive input support
+  - Event notification system
+  - Session lifecycle management
+  - Background event detection
+
+{Colors.BOLD}Event Types:{Colors.RESET}
+  - session_started: New session created
+  - session_ended: Session terminated
+  - output_received: New output available
+  - possible_input_needed: May require user input
+  - high_output_volume: Large amount of output
+  - inactive_session: No activity for period
+
+{Colors.BOLD}Status Indicators:{Colors.RESET}
+The prompt shows background session count: You[2bg]>
+  - Number indicates active background sessions
+  - Updates automatically as sessions start/stop
+
+{Colors.BOLD}Use Cases:{Colors.RESET}
+  - Long-running build processes
+  - Web servers and daemons
+  - File watching and monitoring
+  - Batch processing tasks
+  - Test suites execution
+  - Development servers
+"""
+
+
+def get_full_help():
+    return f"""
+{Colors.BOLD}R - PROFESSIONAL AI ASSISTANT{Colors.RESET}
+
+{Colors.BOLD}BASIC COMMANDS{Colors.RESET}
+  exit, quit, q          - Exit the assistant
+  /help                  - Show this help message
+  /help workflows        - Detailed workflow documentation
+  /help agents           - Detailed agent documentation
+  /help knowledge        - Knowledge base documentation
+  /help cache            - Caching system documentation
+  /help background       - Background sessions documentation
+  /reset                 - Clear message history
+  /dump                  - Show message history as JSON
+  /verbose               - Toggle verbose mode
+  /models                - List available models
+  /tools                 - List available tools
+
+{Colors.BOLD}FILE OPERATIONS{Colors.RESET}
+  /review <file>         - Review a file
+  /refactor <file>       - Refactor code in a file
+  /obfuscate <file>      - Obfuscate code in a file
+
+{Colors.BOLD}AUTONOMOUS MODE{Colors.RESET}
+  {Colors.CYAN}/auto <task>{Colors.RESET}        - Enter autonomous mode for task completion
+                         Assistant works continuously until task is complete
+                         Max 50 iterations with automatic context management
+                         Press Ctrl+C twice to force exit
+
+{Colors.BOLD}WORKFLOWS{Colors.RESET}
+  {Colors.CYAN}/workflows{Colors.RESET}          - List all available workflows
+  {Colors.CYAN}/workflow <name>{Colors.RESET}    - Execute a specific workflow
+
+  Workflows enable automated multi-step task execution with:
+  - Sequential, parallel, or conditional execution
+  - Variable substitution and step dependencies
+  - Error handling and retry logic
+  - Success/failure path routing
+
+  For detailed documentation: /help workflows
+
+{Colors.BOLD}AGENTS{Colors.RESET}
+  {Colors.CYAN}/agent <role> <task>{Colors.RESET}     - Create specialized agent
+  {Colors.CYAN}/agents{Colors.RESET}                   - Show active agents
+  {Colors.CYAN}/collaborate <task>{Colors.RESET}       - Multi-agent collaboration
+
+  Available roles: coding, research, data_analysis, planning,
+                   testing, documentation, orchestrator, general
+
+  Each agent has specialized capabilities and system prompts.
+  Agents can work independently or collaborate on complex tasks.
+
+  For detailed documentation: /help agents
+
+{Colors.BOLD}KNOWLEDGE BASE{Colors.RESET}
+  {Colors.CYAN}/knowledge <query>{Colors.RESET}        - Search knowledge base
+  {Colors.CYAN}/remember <content>{Colors.RESET}       - Store information
+
+  Persistent storage for facts, procedures, and context.
+  Automatic categorization and TF-IDF search.
+  Integrated with agents for context injection.
+
+  For detailed documentation: /help knowledge
+
+{Colors.BOLD}SESSION MANAGEMENT{Colors.RESET}
+  {Colors.CYAN}/history{Colors.RESET}                  - Show conversation history
+  {Colors.CYAN}/cache{Colors.RESET}                    - Show cache statistics
+  {Colors.CYAN}/cache clear{Colors.RESET}              - Clear all caches
+  {Colors.CYAN}/stats{Colors.RESET}                    - Show system statistics
+
+{Colors.BOLD}BACKGROUND SESSIONS{Colors.RESET}
+  {Colors.CYAN}/bg start <command>{Colors.RESET}       - Start background session
+  {Colors.CYAN}/bg list{Colors.RESET}                  - List active sessions
+  {Colors.CYAN}/bg output <name>{Colors.RESET}         - View session output
+  {Colors.CYAN}/bg kill <name>{Colors.RESET}           - Terminate session
+
+  Run long-running processes while maintaining interactivity.
+  Automatic monitoring and event notifications.
+
+  For detailed documentation: /help background
+
+{Colors.BOLD}CONTEXT FILES{Colors.RESET}
+  .rcontext.txt          - Local project context (auto-loaded)
+  ~/.rcontext.txt        - Global context (auto-loaded)
+  -c, --context FILE     - Additional context files (command line)
+
+{Colors.BOLD}ENVIRONMENT VARIABLES{Colors.RESET}
+  OPENROUTER_API_KEY     - API key for OpenRouter
+  AI_MODEL               - Default model to use
+  API_URL                - Custom API endpoint
+  USE_TOOLS              - Enable/disable tools (default: 1)
+
+{Colors.BOLD}COMMAND-LINE FLAGS{Colors.RESET}
+  -i, --interactive      - Start in interactive mode
+  -v, --verbose          - Enable verbose output
+  --debug                - Enable debug logging
+  -m, --model MODEL      - Specify AI model
+  --no-syntax            - Disable syntax highlighting
+
+{Colors.BOLD}DATA STORAGE{Colors.RESET}
+  ~/.assistant_db.sqlite - SQLite database for persistence
+  ~/.assistant_history   - Command history
+  ~/.assistant_error.log - Error logs
+
+{Colors.BOLD}AVAILABLE TOOLS{Colors.RESET}
+Tools are functions the AI can call to interact with the system:
+  - File operations: read, write, list, mkdir, search/replace
+  - Command execution: run_command, interactive sessions
+  - Web operations: http_fetch, web_search, web_search_news
+  - Database: db_set, db_get, db_query
+  - Python execution: python_exec with persistent globals
+  - Code editing: open_editor, insert/replace text, diff/patch
+  - Process management: tail, kill, interactive control
+  - Agents: create, execute tasks, collaborate
+  - Knowledge: add, search, categorize entries
+
+{Colors.BOLD}GETTING HELP{Colors.RESET}
+  /help                  - This help message
+  /help workflows        - Workflow system details
+  /help agents           - Agent system details
+  /help knowledge        - Knowledge base details
+  /help cache            - Cache system details
+  /help background       - Background sessions details
+  /tools                 - List all available tools
+  /models                - List all available models
+"""