diff --git a/taicpp/docs/shell.md b/taicpp/docs/shell.md new file mode 100644 index 0000000..2a136f6 --- /dev/null +++ b/taicpp/docs/shell.md @@ -0,0 +1,80 @@ +# shell: processes, boxes, live output, speech + +Scope: `src/shell/` (namespace `detz::shell`). Source of truth is `../tai.py`; this file states what the C++ port +does today, how it is verified, and the gaps. Wiring for other modules: [wiring/shell.md](wiring/shell.md). + +## Command Output Rule (repository AGENTS.md) + +Every command an agent runs for the user streams live and is returned to the agent. The window is a fixed-height +message (`LiveStream`, or the chat TUI's `live` block through the `StreamSink` protocol). A command that stops on a +prompt (sudo, ssh, y/n) runs on a pty and asks through `ask(text, secret)`; the answer is typed into the command, a +decline (`nullopt`) sends end-of-file, and time spent waiting for the user never counts against the timeout. +Anything that runs a process goes through `RunLive` with `ShellIo::Stream` and `ShellIo::Ask`. No bare +`subprocess`-style calls for agent commands. + +## Caps (constants, all from tai.py) + +| constant | value | where | +|---|---|---| +| `kShellTimeout` (SHELL_TIMEOUT) | 120 s | runlive.h | +| `kRunOutputCap` (RUN_OUTPUT_CAP) | 4,000,000 chars, pty path only | runlive.h | +| `kRunPromptIdle` (RUN_PROMPT_IDLE) | 1.0 s of silence before a prompt is asked | runlive.h | +| `kRunMaxAsks` (RUN_MAX_ASKS) | 20 questions per command | runlive.h | +| `kRunAnswerCap` (RUN_ANSWER_CAP) | 1000 chars typed per answer | runlive.h | +| `kRunRows` x `kRunCols` | 24 x 120 pty window | pty.h | +| `kStreamHeight` (STREAM_HEIGHT) | 4 lines | livestream.h | +| `kShellSnapshotMaxFiles` | 25 files | targets.h | +| ShellSession caps | SID_MAX 64, CHAIN_CAP 64, ROWS_CAP 1000, TREE_DEPTH 8, TREE_KIDS 10, TREE_LINES 40 | session.h | + +## Parity table (tai.py symbol -> C++) + +| tai.py | C++ | notes | +|---|---|---| +| `run_live`, `run_live_pty` | `RunLive::Run`, `RunLive::RunPty` | pipe path with reader thread, pty path with prompt detection | +| `RUN_CTRL_RE`, `RUN_PROMPT_RE` | `RunLive::StripControl`, `RunLive::LooksLikePrompt` | hand-written scanners, same alternation | +| `PtyProc` | `PtyProc` | posix_openpt, TIOCSCTTY, TIOCSWINSZ, poll reader, echo probe | +| `subprocess.Popen/run`, `shutil.which` | `Popen`, `Subprocess::Run`, `Subprocess::Which` (proc.h) | errno reporting through a CLOEXEC pipe, Python `OSError`/`TimeoutExpired` texts | +| `LiveStream`, `stream_status`, `human_size` | `LiveStream`, `StreamStatus`, `HumanSize` | | +| `Spinner` | `Spinner` | stderr tty only, 80 ms frames, shimmer + clock when color is on | +| `LaneBoard` | `LaneBoard` | `list_agents` injected as `AgentLister` | +| `Glow` (every static method) | `Glow` (glow.h) | `describe_call` over `JsonValue`, `agent_row`/`lanes` over `AgentRow` | +| `Sandbox`, `container_engine`, `box_state`, `update_box_caps`, `ensure_box`, `box_exec`, `box_python`, `box_transcribe`, `box_speak`, `box_opencode_binary`, `box_opencode_run` | `Sandbox` (sandbox.h) | `BoxOpencodeExec` is the exec half; the caller runs `opencode_parse_events` | +| `BOX_CONTAINERFILE`, `BOX_STT`, `BOX_TTS`, `BOX_OPENCODE_INSTALL_SCRIPT` | `BoxData` (boxdata.cpp, generated from tai.py) | byte-identical, checked by the box parity case | +| `curl_hint`, `CURL_HINT` | `CurlHint`, `kCurlHint` | | +| `ShellSession` (current, register, tree_lines, report, ...) | `ShellSession` (session.h) | `SessionDb` interface replaces `store.db`; pid/boot/chain logic identical | +| `shell_target_paths`, `segment_targets` | `ShellTargets::Paths`, `ShellTargets::Segment` | shlex, glob and os.walk ported (`Shlex`, `PyGlob`) | +| `Tools.shell_stream`, `Tools.shell_ask` | `ShellIo::Stream`, `ShellIo::Ask` over `ShellApp` | | +| `edge_token`, `edge_timestamp`, `edge_ssml`, `edge_chunks`, `ws_send`, `ws_recv_exact`, `ws_recv`, `edge_skew`, `edge_connect`, `edge_synthesize` | `EdgeSpeech` (edge.h) | TLS bytes via libcurl connect-only, `EdgeSocket` seam for tests | +| `play_audio`, `listen_audio`, `RECORDERS`, `TRANSCRIBERS` | `Audio` (audio.h) | | + +Old invented API (`Sandbox::run(allow-list)`, `ShellSession::exec`, `PtyProc::spawn/read_all`, `Spinner::tick/done`, +`LiveStream::push/drain/joined`, `LaneBoard::set/render`, `Glow::highlight`, `speak_text`) is gone; tai.py has none of +it. Replacements are listed in wiring/shell.md. + +## How it is verified + +- `tests/shell_parity.py` builds `tests/shell_parity_driver.cpp` and runs every case through the driver and through + the same function of `../tai.py`, comparing bytes. Real processes, a real pty (including the real `su` password + prompt), real files, a scripted fake `podman`, a fake `/proc`, a fake websocket server. Fuzzed inputs for markdown, + wrap, truncate, diff, json. `python3 tests/shell_parity.py [--fuzz N] [--filter NAME] [--live] [-v]`. +- `tests/shell_test.cpp`: unit tests, build line in its header. +- Both build with `-Wall -Wextra` and must be warning-free; the parity run fails on any compiler warning. + +## Gaps and deviations (stated, not hidden) + +- Unicode classes: `\w`, `\d`, `str.isalnum`, `str.lower` use hand-written ranges (exact for ASCII, Latin-1, Greek, + Cyrillic basics; approximate beyond). Affects markdown italics boundaries, number tokens in `highlight_python`, + `complete_files` case folding for exotic scripts. +- The pipe path of `run_live` keeps all output in memory (no cap), exactly like tai.py; only the pty path is capped. +- After a timeout only the shell is killed (`Popen.kill`), as in tai.py. A grandchild that keeps the pipe open makes + `run_live` wait up to 10 s for the reader and leaves the grandchild running. +- `Spinner` and `LaneBoard` are time based; the parity run compares the first frames, the final clear and the frame + count within 2, not every frame. +- Edge speech: wire handling is verified offline against tai.py with a scripted server. The live service answered + `403 Forbidden` to tai.py and to the port alike when last tried (2026-10-25, this machine), so end-to-end audio + was not verified. +- `Sandbox::EnsureBox` image build and the opencode install were exercised only against a scripted fake engine, not + a real multi-minute podman build. +- `ShellIo::Ask` REPL fallback: Ctrl-C during the question is not mapped to a decline (tai.py maps + `KeyboardInterrupt`); the REPL owner decides signal handling. +- `BackendError` here is `detz::shell::BackendError`; the agent core converts it at the call site. diff --git a/taicpp/docs/util.md b/taicpp/docs/util.md new file mode 100644 index 0000000..a0ee84c --- /dev/null +++ b/taicpp/docs/util.md @@ -0,0 +1,34 @@ +# util: text, ansi, markdown, diff, json, paths + +Scope: `src/util/` (namespace `detz`). Source of truth `../tai.py`. All lengths and slices are Unicode code points +(Python `str` semantics) over UTF-8 `std::string`; invalid bytes decode to U+FFFD like `errors="replace"`. +Verified byte for byte against tai.py by `tests/shell_parity.py` (see [shell.md](shell.md)). + +| python (tai.py) | C++ (src/util/) | notes | +|---|---|---| +| `Ansi`, `FG_*`, `BG_*` | `Ansi`, `Palette` (ansi.h) | all codes incl. UNDERLINE, STRIKE, GRAY | +| `paint`, `color_enabled`, `strip_ansi`, `style_text`, `style_span`, `truncate_ansi`, `pad_tinted`, `wrap_ansi` | `AnsiText` (ansi.h) | `Paint`/`Paint2` free functions stay; they now honor NO_COLOR and stdout tty like `paint` | +| `shutil.get_terminal_size` | `AnsiText::TerminalWidth` | COLUMNS, then TIOCGWINSZ on stdout, then fallback | +| `truncate`, `capped_text`, `trunc_marker`, `read_capped` | `Truncate`, `CappedText`, `TruncMarker`, `ReadCapped` (text.h) | `CappedText` returns `Capped{present,text,size}`; `Truncate(tail=0)` keeps the whole text like `text[-0:]` | +| `chunk_lines`, `collapse_repeated_lines` | `ChunkLines`, `CollapseRepeatedLines` | `str.splitlines` semantics | +| `split_leading_tags`, `expand_file_mentions`, `complete_files` | same names in text.h | `os.path.expanduser`, `exists`, `abspath` via `PyPath` | +| `human_size`, `estimate_tokens`, `as_int`, `short_error`, `now_iso`, `tool_name_clean`, `parse_duration`, `format_delay` | text.h | | +| `md_inline`, `md_split_row`, `md_is_separator`, `md_align_cell`, `md_table_block`, `render_markdown`, `MD_BULLETS` | `Markdown` (markdown.h) | `re` rules matched by scanners with `re.sub` semantics | +| `render_diff`, `split_diff_lines`, `difflib.SequenceMatcher(autojunk=False)` | `DiffRender`, `SequenceMatcher` | opcodes, grouped opcodes, same match order | +| `highlight_python`, `PY_TOKEN_RE` | `PyHighlight` | string, comment, number, keyword, decorator, def/class name in regex order | +| `tool_args_text`, `tool_args_valid`, `action_signature` | `ToolArgs` (toolargs.h) | | +| `json.loads/dumps/str/repr` as used by Glow and tool args | `JsonValue` (jsonvalue.h) | ordered objects, duplicate key last wins, big ints kept as text, `ensure_ascii` dumps, float repr | +| `os.path.*` subset | `PyPath` (pypath.h) | join, normpath, abspath, basename, dirname, expanduser, realpath, listdir | +| `shlex.split(posix=True)` | `Shlex` (shlex.h) | | +| `glob.glob(recursive=True)`, `fnmatch`, `os.walk` | `PyGlob` (pyglob.h) | Python 3.14 glob algorithm | +| `hashlib.sha256`, `base64.b64encode` | `Digest` (digest.h) | | +| `str` helpers | `Utf8` (utf8.h) | strip, split, splitlines, expandtabs, lower, rjust, slices | +| http (urllib flows) | `HttpGet/HttpPost/HttpRequest/HttpPostSse` (http.h) | pre-existing libcurl layer, not touched in this milestone | + +`Glow` (describe_call, agent rows, status bar) lives in `src/shell` and is documented in [shell.md](shell.md). + +## Gaps + +- Non-ASCII classification (`IsWord`, `IsAlnum`, `Lower`) is table-light; exact for ASCII and the common European + and Cyrillic ranges, approximate elsewhere. +- `http.h` is the earlier libcurl wrapper; tai.py's HTTP flows live in the llm and agent modules, which own them. diff --git a/taicpp/docs/wiring/shell.md b/taicpp/docs/wiring/shell.md new file mode 100644 index 0000000..1d5641a --- /dev/null +++ b/taicpp/docs/wiring/shell.md @@ -0,0 +1,93 @@ +# wiring: shell and util, for the agent core and other callers + +What other modules call. Python reference: `Tools.run_shell`, `Tools.run_shell_box`, `Tools.shell_stream`, +`Tools.shell_ask` in `../tai.py`. Link flags: `-pthread -lcurl` (edge speech uses libcurl connect-only). Include +dirs: every `src/*` dir is already on the include path. + +## 1. Running a command for the user (run_shell) + +```cpp +#include "runlive.h" // RunLive, Command, RunLiveOptions, RunResult, kShellTimeout +#include "shellio.h" // ShellApp, ShellIo +#include "livestream.h" // StreamStatus +#include "sandbox.h" // CurlHint + +using namespace detz::shell; + +std::string RunShell(ShellApp& app, bool live, const std::string& command, const std::string& workdir, + const EnvMap& env /* process env + secrets */) { + std::unique_ptr stream = ShellIo::Stream(app, command, live); // Tools.shell_stream + RunLiveOptions o; + o.cwd = workdir; o.env = &env; + o.stream = stream.get(); + o.ask = ShellIo::Ask(app, command); // Tools.shell_ask (empty = nobody can answer) + RunResult r = RunLive::Run(Command::Shell(command), o); // run_live(command, shell=True, ...) + if (stream) stream->Close(StreamStatus(r.code, r.output, r.elapsed, r.timedOut)); + if (r.timedOut) return "error: timed out after " + std::to_string(kShellTimeout) + " seconds"; + // "exit %d\n" + spill_or_truncate(store, "output", "shell: " + command[:80], output.strip() or "(no output)", ["shell"], profile) + curl_hint(command) + return "exit " + std::to_string(r.code) + "\n" + /* spill_or_truncate(...) */ "" + CurlHint(command); +} +``` + +- `RunLive::Run` throws `detz::shell::OsError` (what() is Python's `str(OSError)`) when the program cannot start; + `run_shell_box` catches it as `"error: " + ShortError(...)`. `OsError` and `TimeoutExpired` derive from + `ProcessError`, the analog of `except (OSError, subprocess.SubprocessError)`. +- Box variant (`run_shell_box`): `Command::Argv({engine, "exec", ...extra, ask ? "-it" : "-i", Sandbox::kName, "sh", "-c", command})` + with `extra = {"--workdir", workdir, "-e", "NAME=value", ...}`; same stream/ask handling. +- `ShellApp` is the adapter for `self.app`: implement `Auto()`, `Persist()`, `Depth()`, `HasPrompter()`/`Prompt()`, + `MarkUserSteppedIn()`, `HasStreamFactory()`/`StreamFactory()` (the chat TUI returns its `TuiLiveStream`, which must + implement `StreamSink`: `Feed(line)`, optional `Pause()`, `Close(status)`), and `Redact(command)` + (`store.redact(command, profile)`, identity without a store). +- `StreamSink` is the Python `feed/pause/close` protocol. `LiveStream` is the REPL implementation. +- Never run an agent command any other way (Command Output Rule). Output is never shown twice: the stream is the + user's view, `RunResult.output` is the agent's. + +## 2. Snapshot targets before a destructive command + +`ShellTargets::Paths(command, workdir)` returns the existing files a command is about to change (25 max, absolute, +de-duplicated); feed them to `ReadCapped(path)` (`AUDIT_MAX_CHARS`) then `audit_event` / `upsert_file_record`. + +## 3. Types other modules supply + +| need | type | who provides | +|---|---|---| +| agent records for lanes / `Glow::AgentRowText` | `shell::AgentRow{id,status,task,elapsed,tools?,collected}` | agent core maps `list_agents()` rows | +| `LaneBoard` rows | `shell::AgentLister` = `std::function(const std::string& profile)>` | agent core (`list_agents`) | +| shell session persistence | `shell::SessionDb` (`Execute`, `Query`, `Commit`; throw `SessionDbError` on sqlite errors) | store adapts its sqlite handle; table `shell_sessions` as in settings/store spec | +| errors from box/edge helpers | `shell::BackendError(message, status)` | catch and rethrow as `llm::BackendError` where the Python raises `BackendError` | +| opencode in the box | `Sandbox::BoxOpencodeExec(engine, model, prompt, timeout, agent)` returns the process result; run `opencode_parse_events(out, rc, err)` on it | llm module | + +## 4. Speech + +```cpp +std::string audio = EdgeSpeech::Synthesize(text.substr(0, 8000), voice); // edge_synthesize; throws BackendError, OsError +Audio::Play(path); // play_audio +std::string heard = Audio::Listen(seconds, audio_dir); // listen_audio +std::string mp3 = Sandbox::BoxSpeak(engine, text, voice); // box_speak +std::string text2 = Sandbox::BoxTranscribe(engine, wavBytes); // box_transcribe +``` + +## 5. Replacements for the old placeholder API + +| old (removed) | use | +|---|---| +| `ShellSession sh; sh.exec(cmd, ms)` | `RunLive::Run(Command::Shell(cmd), o)`; `ShellSession` now means the shell identity (`ShellSession::Current()`, `Register`, `Report`) | +| `ShellSession::target_paths/segment_targets` | `ShellTargets::Paths/Segment` | +| `Sandbox(root, allow).run(cmd)` | `Sandbox::BoxExec(engine, argv)` / the box variant of run_shell | +| `PtyProc::spawn/read_all` | `PtyProc(argv, shellCommand, cwd, env)` + `Read/Write` or just `RunLive::RunPty` | +| `Spinner::tick/done` | `Spinner s("label"); s.Start(); ...; s.Stop();` | +| `LiveStream::push/drain/joined` | `LiveStream(height, width, sink, enabled)` with `Feed/Pause/Close` | +| `LaneBoard::set/render` | `LaneBoard(profile, lister)` with `Start/Stop`, `Glow::Lanes` for the rows | +| `Glow::highlight`, `speak_text` | none in tai.py; use `PyHighlight::Line`, `Glow::*`, `EdgeSpeech` | +| `CappedText(string)` returning string | `CappedText(optional)` returning `Capped{present,text,size}` | +| `detz::shell::curl_hint` | still available (inline alias of `CurlHint`) | + +`src/agent/exec.cpp` (3 call sites of `ShellSession::exec`) and `tests/test_main.cpp` (shell section, updated in this +milestone) were written against the old API; exec.cpp is outside the shell/util area and must be moved to +`RunLive` by the agent core. + +## 6. Threads + +`RunLive::Run` may be called from any thread (the burst test runs 32 concurrent runs). `Spinner`, `LaneBoard` start a +worker thread and join on `Stop()` / destruction. `Sandbox` caches (`caps` stamp, python path, opencode presence) are +mutex-guarded process state, like the Python module globals. diff --git a/taicpp/src/shell/.gitkeep b/taicpp/src/shell/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/taicpp/src/shell/audio.cpp b/taicpp/src/shell/audio.cpp new file mode 100644 index 0000000..4353538 --- /dev/null +++ b/taicpp/src/shell/audio.cpp @@ -0,0 +1,89 @@ +#include "audio.h" +#include "proc.h" +#include "text.h" +#include "utf8.h" +#include +#include +#include +namespace detz::shell { +const std::vector& Audio::Recorders() { + static const std::vector r = { + {"arecord", {"arecord", "-q", "-d", "{seconds}", "-f", "cd", "-t", "wav", "{path}"}}, + {"rec", {"rec", "-q", "{path}", "trim", "0", "{seconds}"}}, + {"ffmpeg", {"ffmpeg", "-y", "-v", "quiet", "-f", "alsa", "-i", "default", "-t", "{seconds}", "{path}"}}}; + return r; +} +const std::vector& Audio::Transcribers() { + static const std::vector t = {"whisper-cpp", "whisper-cli", "whisper", "faster-whisper"}; + return t; +} + +bool Audio::Play(const std::string& path) { + std::vector runner; +#ifdef __APPLE__ + if (!Subprocess::Which("afplay").empty()) runner = {"afplay", path}; + else +#endif + if (!Subprocess::Which("paplay").empty()) runner = {"paplay", path}; + else if (!Subprocess::Which("aplay").empty()) runner = {"aplay", "-q", path}; + else if (!Subprocess::Which("pw-play").empty()) runner = {"pw-play", path}; + else return false; + try { + SubprocessOptions o; + o.popen.argv = runner; + o.popen.out = StdMode::Null; + o.popen.err = StdMode::Null; + o.timeout = 300; + Subprocess::Run(o); + return true; + } catch (const ProcessError&) { + return false; + } +} + +std::string Audio::Listen(long seconds, const std::string& workdir) { + const Recorder* recorder = nullptr; + for (auto& item : Recorders()) if (!Subprocess::Which(item.binary).empty()) { recorder = &item; break; } + std::string transcriber; + for (auto& name : Transcribers()) if (!Subprocess::Which(name).empty()) { transcriber = name; break; } + if (!recorder || transcriber.empty()) { + std::vector missing; + if (!recorder) missing.push_back("audio recorder (install arecord, sox, or ffmpeg)"); + if (transcriber.empty()) missing.push_back("transcriber (install whisper-cpp or whisper)"); + return "listening unavailable, missing: " + Utf8::Join(missing, ", "); + } + std::string path = workdir + (workdir.empty() || workdir.back() == '/' ? "" : "/") + "listen-" + std::to_string((long long)std::time(nullptr)) + ".wav"; + std::vector command; + for (auto part : recorder->argv) { + part = Utf8::ReplaceAll(part, "{seconds}", std::to_string(seconds)); + command.push_back(Utf8::ReplaceAll(part, "{path}", path)); + } + CompletedProcess done; + std::string result; + bool failed = false; + try { + SubprocessOptions rec; + rec.popen.argv = command; + rec.popen.out = StdMode::Null; + rec.popen.err = StdMode::Null; + rec.timeout = (double)seconds + 15; + Subprocess::Run(rec); + SubprocessOptions tr; + tr.popen.argv = {transcriber, path}; + tr.popen.out = StdMode::Pipe; + tr.popen.err = StdMode::Pipe; + tr.timeout = 180; + done = Subprocess::Run(tr); + } catch (const ProcessError& exc) { + result = "listening failed: " + ShortError(exc.what(), "ProcessError"); + failed = true; + } + ::unlink(path.c_str()); + if (failed) return result; + IncrementalUtf8 d1, d2; UniversalNewlines n1, n2; + std::string err = n1.Feed(d1.Decode(done.err, true), true), out = n2.Feed(d2.Decode(done.out, true), true); + if (done.returncode != 0) return "transcriber error: " + Utf8::Head(err, 500); + std::string text = Utf8::Strip(out); + return text.empty() ? "heard nothing" : Truncate(text, 3000, 500); +} +} // namespace detz::shell diff --git a/taicpp/src/shell/audio.h b/taicpp/src/shell/audio.h new file mode 100644 index 0000000..4e8cc46 --- /dev/null +++ b/taicpp/src/shell/audio.h @@ -0,0 +1,17 @@ +#pragma once +// Audio: play_audio() and listen_audio() of tai.py, with RECORDERS / TRANSCRIBERS. +#include +#include +namespace detz::shell { +struct Recorder { std::string binary; std::vector argv; }; // argv holds {seconds} / {path} placeholders + +class Audio { + public: + static const std::vector& Recorders(); // RECORDERS + static const std::vector& Transcribers(); // TRANSCRIBERS + // play_audio(path): true when a player ran + static bool Play(const std::string& path); + // listen_audio(seconds, workdir): the transcript, or a one-line reason + static std::string Listen(long seconds, const std::string& workdir); +}; +} // namespace detz::shell diff --git a/taicpp/src/shell/boxdata.cpp b/taicpp/src/shell/boxdata.cpp new file mode 100644 index 0000000..e993ded --- /dev/null +++ b/taicpp/src/shell/boxdata.cpp @@ -0,0 +1,46 @@ +// Generated from tai.py by the porting step; texts are byte-identical to the Python constants. +#include "boxdata.h" +namespace detz::shell { +const char* BoxData::BoxContainerfile() { + static const char* const text = R"BOXDATA(FROM python:3.12-slim +ENV DEBIAN_FRONTEND=noninteractive PIP_NO_CACHE_DIR=1 +RUN apt-get update && apt-get install -y --no-install-recommends ffmpeg espeak-ng python3-venv && rm -rf /var/lib/apt/lists/* +RUN python -m venv /box/venv && /box/venv/bin/pip install --no-cache-dir faster-whisper edge-tts +RUN /box/venv/bin/python -c "from faster_whisper import WhisperModel; WhisperModel('tiny', device='cpu', compute_type='int8')" +COPY stt.py /box/stt.py +COPY tts.py /box/tts.py +CMD ["sleep", "infinity"] +)BOXDATA"; + return text; +} +const char* BoxData::BoxStt() { + static const char* const text = R"BOXDATA(import sys +from faster_whisper import WhisperModel +data = sys.stdin.buffer.read() +with open("/tmp/in.audio", "wb") as handle: + handle.write(data) +model = WhisperModel("tiny", device="cpu", compute_type="int8") +segments, _info = model.transcribe("/tmp/in.audio") +print(" ".join(segment.text for segment in segments).strip(), flush=True) +)BOXDATA"; + return text; +} +const char* BoxData::BoxTts() { + static const char* const text = R"BOXDATA(import asyncio +import sys +import edge_tts +async def speak(text, voice): + talk = edge_tts.Communicate(text, voice or "en-US-EmmaMultilingualNeural") + await talk.save("/tmp/out.mp3") +text = sys.stdin.read().strip() +voice = sys.argv[1] if len(sys.argv) > 1 else "" +asyncio.run(speak(text, voice)) +sys.stdout.buffer.write(open("/tmp/out.mp3", "rb").read()) +)BOXDATA"; + return text; +} +const char* BoxData::BoxOpencodeInstallScript() { + static const char* const text = R"BOXDATA(set -e; ARCH="$(dpkg --print-architecture)"; case "$ARCH" in amd64) NODE_ARCH=x64;; arm64) NODE_ARCH=arm64;; armhf) NODE_ARCH=armv7l;; i386) NODE_ARCH=x86;; *) echo "unsupported arch $ARCH" >&2; exit 1;; esac; apt-get update -qq && apt-get install -y -qq --no-install-recommends curl xz-utils ca-certificates; curl -fsSL "https://nodejs.org/dist/v20.19.2/node-v20.19.2-linux-$NODE_ARCH.tar.xz" -o /tmp/node.tar.xz; tar -xJf /tmp/node.tar.xz -C /usr/local --strip-components=1; rm /tmp/node.tar.xz; apt-get purge -y -qq curl xz-utils; apt-get autoremove -y -qq; rm -rf /var/lib/apt/lists/*; npm install -g opencode-ai --no-fund --no-audit)BOXDATA"; + return text; +} +} // namespace detz::shell diff --git a/taicpp/src/shell/boxdata.h b/taicpp/src/shell/boxdata.h new file mode 100644 index 0000000..5c86e04 --- /dev/null +++ b/taicpp/src/shell/boxdata.h @@ -0,0 +1,13 @@ +#pragma once +// BoxData: the embedded texts of the sandbox box (BOX_CONTAINERFILE, BOX_STT, BOX_TTS and +// BOX_OPENCODE_INSTALL_SCRIPT of tai.py). boxdata.cpp is generated from tai.py and checked +// byte for byte by the parity test. +namespace detz::shell { +class BoxData { + public: + static const char* BoxContainerfile(); + static const char* BoxStt(); + static const char* BoxTts(); + static const char* BoxOpencodeInstallScript(); +}; +} // namespace detz::shell diff --git a/taicpp/src/shell/edge.cpp b/taicpp/src/shell/edge.cpp new file mode 100644 index 0000000..6a85449 --- /dev/null +++ b/taicpp/src/shell/edge.cpp @@ -0,0 +1,359 @@ +#include "edge.h" +#include "digest.h" +#include "utf8.h" +#include +#include +#include +#include +#include +#include +#include +#include +#include +namespace detz::shell { +namespace { +std::string UrandomBytes(size_t n) { + std::string out(n, '\0'); + std::ifstream in("/dev/urandom", std::ios::binary); + in.read(out.data(), (std::streamsize)n); + return out; +} +std::string Hex(const std::string& raw) { + static const char* d = "0123456789abcdef"; + std::string o; + for (unsigned char c : raw) { o.push_back(d[c >> 4]); o.push_back(d[c & 15]); } + return o; +} +std::string UuidHex() { + std::string r = UrandomBytes(16); + r[6] = (char)((r[6] & 0x0f) | 0x40); // version 4 + r[8] = (char)((r[8] & 0x3f) | 0x80); // variant + return Hex(r); +} +double SystemNow() { + struct timespec ts; + clock_gettime(CLOCK_REALTIME, &ts); + return (double)ts.tv_sec + (double)ts.tv_nsec / 1e9; +} + +// TLS byte stream through libcurl's connect-only mode (curl does DNS, TCP and the TLS handshake). +class CurlTlsSocket : public EdgeSocket { + public: + CurlTlsSocket(const std::string& host, int port, double connectTimeout, double ioTimeout) : ioTimeout_(ioTimeout) { + curl_ = curl_easy_init(); + if (!curl_) throw OsError::Text("cannot create TLS connection"); + std::string url = "https://" + host + ":" + std::to_string(port) + "/"; + curl_easy_setopt(curl_, CURLOPT_URL, url.c_str()); + curl_easy_setopt(curl_, CURLOPT_CONNECT_ONLY, 1L); + curl_easy_setopt(curl_, CURLOPT_HTTP_VERSION, (long)CURL_HTTP_VERSION_1_1); // ALPN http/1.1: the handshake below is hand-written HTTP/1.1 + curl_easy_setopt(curl_, CURLOPT_CONNECTTIMEOUT_MS, (long)(connectTimeout * 1000)); + CURLcode rc = curl_easy_perform(curl_); + if (rc != CURLE_OK) { + std::string msg = curl_easy_strerror(rc); + curl_easy_cleanup(curl_); + curl_ = nullptr; + throw OsError::Text(msg, rc == CURLE_OPERATION_TIMEDOUT ? ETIMEDOUT : ECONNREFUSED); + } + } + ~CurlTlsSocket() override { Close(); } + void SendAll(const std::string& data) override { + size_t off = 0; + while (off < data.size()) { + size_t sent = 0; + CURLcode rc = curl_easy_send(curl_, data.data() + off, data.size() - off, &sent); + if (rc == CURLE_AGAIN) { Wait(false); continue; } + if (rc != CURLE_OK) throw OsError::Text(curl_easy_strerror(rc), EPIPE); + off += sent; + } + } + std::string Recv(size_t max) override { + std::string buf(max, '\0'); + while (true) { + size_t got = 0; + CURLcode rc = curl_easy_recv(curl_, buf.data(), max, &got); + if (rc == CURLE_AGAIN) { Wait(true); continue; } + if (rc == CURLE_UNSUPPORTED_PROTOCOL || (rc == CURLE_OK && got == 0)) return ""; + if (rc != CURLE_OK) throw OsError::Text(curl_easy_strerror(rc), ECONNRESET); + buf.resize(got); + return buf; + } + } + void Close() override { if (curl_) { curl_easy_cleanup(curl_); curl_ = nullptr; } } + private: + void Wait(bool forRead) { + curl_socket_t fd = CURL_SOCKET_BAD; + curl_easy_getinfo(curl_, CURLINFO_ACTIVESOCKET, &fd); + struct pollfd p = {(int)fd, (short)(forRead ? POLLIN : POLLOUT), 0}; + int rc = ::poll(&p, 1, (int)(ioTimeout_ * 1000)); + if (rc == 0) throw OsError::Text("timed out", ETIMEDOUT); + } + CURL* curl_{nullptr}; + double ioTimeout_; +}; + +std::string ZoneNameLower(const std::string& s) { return Utf8::Lower(s); } + +// time.strptime(text, "%a, %d %b %Y %H:%M:%S %Z") followed by time.mktime(); false on ValueError. +bool StrptimeMktime(const std::string& textIn, double* stamp) { + static const char* wd[] = {"monday", "tuesday", "wednesday", "thursday", "friday", "saturday", "sunday"}; + static const char* mon[] = {"january", "february", "march", "april", "may", "june", "july", "august", "september", "october", "november", "december"}; + std::string t = Utf8::Strip(textIn); + size_t i = 0, n = t.size(); + auto alpha = [&]() { size_t s = i; while (i < n && std::isalpha((unsigned char)t[i])) ++i; return Utf8::Lower(t.substr(s, i - s)); }; + auto spaces = [&]() { size_t s = i; while (i < n && Utf8::IsSpace((unsigned char)t[i])) ++i; return i > s; }; + auto num = [&](int maxDigits, int* out) { size_t s = i; int v = 0; while (i < n && i - s < (size_t)maxDigits && std::isdigit((unsigned char)t[i])) v = v * 10 + (t[i++] - '0'); *out = v; return i > s; }; + std::string w = alpha(); + bool wOk = false; // %a matches the abbreviated weekday names only + for (auto name : wd) if (w == std::string(name).substr(0, 3)) wOk = true; + if (!wOk || i >= n || t[i] != ',') return false; + ++i; + if (!spaces()) return false; + int day, year, hh, mm, ss; + { + size_t s0 = i; + if (!num(2, &day)) return false; + size_t cnt = i - s0; + if ((cnt == 2 && (day < 1 || day > 31)) || (cnt == 1 && day < 1)) return false; + } + if (!spaces()) return false; + std::string m = alpha(); + int month = -1; + for (int k = 0; k < 12; ++k) if (m == std::string(mon[k]).substr(0, 3)) month = k; // %b: abbreviated names only + if (month < 0 || !spaces()) return false; + if (!num(4, &year) || year < 1000) return false; + if (!spaces()) return false; + if (!num(2, &hh) || hh > 23 || i >= n || t[i] != ':') return false; + ++i; + if (!num(2, &mm) || mm > 59 || i >= n || t[i] != ':') return false; + ++i; + if (!num(2, &ss) || ss > 61) return false; + if (!spaces()) return false; + std::string zone = ZoneNameLower(t.substr(i)); + i = n; + tzset(); + int isdst = -1; + if (zone == "utc" || zone == "gmt") isdst = 0; + else if (zone == ZoneNameLower(tzname[0])) isdst = 0; + else if (daylight && zone == ZoneNameLower(tzname[1])) isdst = 1; + else return false; + std::tm tm{}; + tm.tm_year = year - 1900; tm.tm_mon = month; tm.tm_mday = day; tm.tm_hour = hh; tm.tm_min = mm; tm.tm_sec = ss > 59 ? 59 : ss; + tm.tm_isdst = isdst; + time_t r = std::mktime(&tm); + if (r == (time_t)-1) return false; + *stamp = (double)r + (ss > 59 ? ss - 59 : 0); + return true; +} +} // namespace + +EdgeSpeech::Env& EdgeSpeech::env() { + static Env e = {UrandomBytes, UuidHex, SystemNow}; + return e; +} + +std::string EdgeSpeech::Token(long skew) { + long long ticks = (long long)std::floor(env().now()) + skew + 11644473600LL; + ticks -= ticks % 300; + ticks *= 10000000LL; + std::string hex = Digest::Sha256Hex(std::to_string(ticks) + kTrustedToken); + for (auto& c : hex) c = (char)std::toupper((unsigned char)c); + return hex; +} + +std::string EdgeSpeech::Timestamp() { + std::time_t t = (std::time_t)std::floor(env().now()); + std::tm tm{}; + gmtime_r(&t, &tm); + char buf[96]; + std::strftime(buf, sizeof buf, "%a %b %d %Y %H:%M:%S GMT+0000 (Coordinated Universal Time)", &tm); + return buf; +} + +std::string EdgeSpeech::Ssml(const std::string& text, const std::string& voice) { + std::string safe = Utf8::ReplaceAll(Utf8::ReplaceAll(Utf8::ReplaceAll(text, "&", "&"), "<", "<"), ">", ">"); + safe = Utf8::ReplaceAll(Utf8::ReplaceAll(safe, "\r", " "), "\n", " "); + return "" + safe + ""; +} + +std::vector EdgeSpeech::Chunks(const std::string& text, size_t limit) { + // re.split(r"(?<=[.!?])\s+", text.strip()) + std::u32string u = Utf8::Decode(Utf8::Strip(text)); + std::vector sentences; + std::u32string cur; + size_t i = 0; + while (i < u.size()) { + if (Utf8::IsSpace(u[i]) && i > 0 && (u[i - 1] == '.' || u[i - 1] == '!' || u[i - 1] == '?')) { + size_t j = i; + while (j < u.size() && Utf8::IsSpace(u[j])) ++j; + sentences.push_back(Utf8::Encode(cur)); + cur.clear(); + i = j; + } else cur.push_back(u[i++]); + } + sentences.push_back(Utf8::Encode(cur)); + std::vector parts; + std::string current; + for (auto& sentence : sentences) { + std::string trial = Utf8::Strip(current + " " + sentence); + if (trial.size() > limit && !current.empty()) { + parts.push_back(current); + current = sentence; + } else current = trial; + } + if (!current.empty()) parts.push_back(current); + if (parts.empty()) parts.push_back(text); + return parts; +} + +void EdgeSpeech::WsSend(EdgeSocket& sock, const std::string& payload) { + std::string mask = env().randomBytes(4); + std::string header(1, (char)0x81); + size_t length = payload.size(); + if (length < 126) header.push_back((char)(0x80 | length)); + else if (length < 65536) { + header.push_back((char)(0x80 | 126)); + header.push_back((char)(length >> 8)); header.push_back((char)(length & 0xff)); + } else { + header.push_back((char)(0x80 | 127)); + for (int s = 56; s >= 0; s -= 8) header.push_back((char)((uint64_t)length >> s & 0xff)); + } + std::string masked(payload.size(), '\0'); + for (size_t k = 0; k < payload.size(); ++k) masked[k] = (char)((unsigned char)payload[k] ^ (unsigned char)mask[k % 4]); + sock.SendAll(header + mask + masked); +} + +namespace { +std::string RecvExact(EdgeSocket& sock, size_t count) { + std::string data; + while (data.size() < count) { + std::string piece = sock.Recv(count - data.size()); + if (piece.empty()) throw BackendError("edge connection closed"); + data += piece; + } + return data; +} +} // namespace + +std::pair EdgeSpeech::WsRecv(EdgeSocket& sock) { + int opcode = 0; + std::string payload; + bool first = true; + while (true) { + std::string head = RecvExact(sock, 2); + bool done = (unsigned char)head[0] & 0x80; + int part = (unsigned char)head[0] & 0x0F; + bool masked = (unsigned char)head[1] & 0x80; + uint64_t length = (unsigned char)head[1] & 0x7F; + if (length == 126) { + std::string b = RecvExact(sock, 2); + length = ((uint64_t)(unsigned char)b[0] << 8) | (unsigned char)b[1]; + } else if (length == 127) { + std::string b = RecvExact(sock, 8); + length = 0; + for (int k = 0; k < 8; ++k) length = (length << 8) | (unsigned char)b[k]; + } + std::string key = masked ? RecvExact(sock, 4) : ""; + std::string body = length ? RecvExact(sock, (size_t)length) : ""; + if (masked) for (size_t k = 0; k < body.size(); ++k) body[k] = (char)((unsigned char)body[k] ^ (unsigned char)key[k % 4]); + if (part == 0x8) throw BackendError("edge closed stream"); + if (part == 0x9) { sock.SendAll(std::string("\x8A\x00", 2)); continue; } + if ((part == 0x1 || part == 0x2) && first) { opcode = part; first = false; } + payload += body; + if (done) return {opcode, payload}; + } +} + +long EdgeSpeech::Skew(const std::string& headerText) { + // re.search(r"Date:\s*(.+?)\r?$", header_text, re.MULTILINE | re.IGNORECASE) + std::string low = Utf8::Lower(headerText); + size_t p = low.find("date:"); + if (p == std::string::npos) return 0; + size_t i = p + 5; + while (i < headerText.size() && Utf8::IsSpace((unsigned char)headerText[i])) ++i; + size_t e = i; + while (e < headerText.size() && headerText[e] != '\n') ++e; + if (e > i && headerText[e - 1] == '\r') --e; + if (e <= i) return 0; + double stamp; + if (!StrptimeMktime(headerText.substr(i, e - i), &stamp)) return 0; + return (long)std::trunc(stamp - env().now()); +} + +std::unique_ptr EdgeSpeech::Connect() { + long skew = 0; + for (int attempt = 0; attempt < 2; ++attempt) { + std::string path = std::string(kWsPath) + "?TrustedClientToken=" + kTrustedToken + "&ConnectionId=" + env().uuidHex() + "&Sec-MS-GEC=" + Token(skew) + "&Sec-MS-GEC-Version=1-" + kChromium; + std::unique_ptr sock = std::make_unique(kHost, 443, 20, 30); + std::string key = Digest::Base64(env().randomBytes(16)); + std::string muid; + { + std::string r = env().randomBytes(32); + for (unsigned char c : r) muid.push_back("0123456789ABCDEF"[c % 16]); + } + std::string handshake = std::string("GET ") + path + " HTTP/1.1\r\n" + + "Host: " + kHost + "\r\n" + + "Upgrade: websocket\r\n" + + "Connection: Upgrade\r\n" + + "Sec-WebSocket-Version: 13\r\n" + + "Sec-WebSocket-Key: " + key + "\r\n" + + "Pragma: no-cache\r\n" + + "Cache-Control: no-cache\r\n" + + "Origin: chrome-extension://jdiccldimpdaibmpdkjnbmckianbfold\r\n" + + "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36 Edg/143.0.0.0\r\n" + + "Accept-Encoding: gzip, deflate, br, zstd\r\n" + + "Accept-Language: en-US,en;q=0.9\r\n" + + "Cookie: muid=" + muid + ";\r\n\r\n"; + sock->SendAll(handshake); + std::string answer; + while (answer.find("\r\n\r\n") == std::string::npos) { + std::string piece = sock->Recv(4096); + if (piece.empty()) break; + answer += piece; + } + size_t eol = answer.find("\r\n"); + std::string firstLine = eol == std::string::npos ? answer : answer.substr(0, eol); + if (firstLine.find(" 101 ") != std::string::npos) return sock; + sock->Close(); + if (answer.find(" 403 ") != std::string::npos && attempt == 0) { + skew = Skew(answer); + continue; + } + throw BackendError("edge handshake failed: " + Utf8::Head(firstLine, 200)); + } + throw BackendError("edge handshake failed after retry"); +} + +std::string EdgeSpeech::SynthesizeOn(EdgeSocket& sock, const std::string& text, const std::string& voice) { + std::string audio; + std::string config = "X-Timestamp:" + Timestamp() + "\r\nContent-Type:application/json; charset=utf-8\r\nPath:speech.config\r\n\r\n" + + "{\"context\":{\"synthesis\":{\"audio\":{\"metadataoptions\":{\"sentenceBoundaryEnabled\":\"false\",\"wordBoundaryEnabled\":\"true\"},\"outputFormat\":\"audio-24khz-48kbitrate-mono-mp3\"}}}}"; + WsSend(sock, config); + for (auto& chunk : Chunks(text)) { + std::string message = "X-RequestId:" + env().uuidHex() + "\r\nContent-Type:application/ssml+xml\r\nX-Timestamp:" + Timestamp() + "Z\r\nPath:ssml\r\n\r\n" + Ssml(chunk, voice); + WsSend(sock, message); + while (true) { + auto [opcode, payload] = WsRecv(sock); + if (opcode == 0x2 && payload.size() > 2) { + size_t headLen = ((size_t)(unsigned char)payload[0] << 8) | (unsigned char)payload[1]; + audio += payload.size() > 2 + headLen ? payload.substr(2 + headLen) : ""; + } else if (opcode == 0x1 && Utf8::Encode(Utf8::Decode(payload)).find("Path:turn.end") != std::string::npos) { + break; + } + } + } + return audio; +} + +std::string EdgeSpeech::Synthesize(const std::string& text, const std::string& voice) { + std::unique_ptr sock = Connect(); + try { + std::string audio = SynthesizeOn(*sock, text, voice); + sock->Close(); + return audio; + } catch (...) { + sock->Close(); + throw; + } +} +} // namespace detz::shell diff --git a/taicpp/src/shell/edge.h b/taicpp/src/shell/edge.h new file mode 100644 index 0000000..6ea3ec6 --- /dev/null +++ b/taicpp/src/shell/edge.h @@ -0,0 +1,50 @@ +#pragma once +// EdgeSpeech: text to speech over the Edge read-aloud websocket (edge_token, edge_timestamp, +// edge_ssml, edge_chunks, ws_send, ws_recv, edge_skew, edge_connect, edge_synthesize of +// tai.py). The wire handling is hand-written like the Python (no websocket library); the TLS +// byte stream is libcurl in connect-only mode. EdgeSocket is the seam the tests use. +#include "errors.h" +#include "proc.h" +#include +#include +#include +#include +#include +namespace detz::shell { +class EdgeSocket { + public: + virtual ~EdgeSocket() = default; + virtual void SendAll(const std::string& data) = 0; + virtual std::string Recv(size_t max) = 0; // empty string = connection closed + virtual void Close() = 0; +}; + +class EdgeSpeech { + public: + static constexpr const char* kHost = "speech.platform.bing.com"; // EDGE_HOST + static constexpr const char* kWsPath = "/consumer/speech/synthesize/readaloud/edge/v1"; // EDGE_WS_PATH + static constexpr const char* kTrustedToken = "6A5AA1D4EAFF4E9FB37E23D68491D6F4"; // EDGE_TRUSTED_TOKEN + static constexpr const char* kChromium = "143.0.3650.75"; // EDGE_CHROMIUM + + // Entropy and clock, replaceable by tests (defaults: urandom, uuid4, time.time()). + struct Env { + std::function randomBytes; + std::function uuidHex; + std::function now; + }; + static Env& env(); + + static std::string Token(long skew = 0); + static std::string Timestamp(); + static std::string Ssml(const std::string& text, const std::string& voice); + static std::vector Chunks(const std::string& text, size_t limit = 4000); + static void WsSend(EdgeSocket& sock, const std::string& payload); + // (opcode, payload); throws BackendError("edge connection closed" / "edge closed stream") + static std::pair WsRecv(EdgeSocket& sock); + static long Skew(const std::string& headerText); + static std::unique_ptr Connect(); + // edge_synthesize(text, voice): MP3 bytes. SynthesizeOn runs the exchange on a given socket. + static std::string Synthesize(const std::string& text, const std::string& voice); + static std::string SynthesizeOn(EdgeSocket& sock, const std::string& text, const std::string& voice); +}; +} // namespace detz::shell diff --git a/taicpp/src/shell/errors.h b/taicpp/src/shell/errors.h new file mode 100644 index 0000000..8b41ac4 --- /dev/null +++ b/taicpp/src/shell/errors.h @@ -0,0 +1,14 @@ +#pragma once +// BackendError as far as the shell helpers (box_*, edge speech) raise it. The agent core maps it +// to its own backend error type at the call site (status carries the HTTP-like status, 0 if none). +#include +#include +namespace detz::shell { +class BackendError : public std::runtime_error { + public: + explicit BackendError(const std::string& message, int status = 0) : std::runtime_error(message), status_(status) {} + int status() const { return status_; } + private: + int status_; +}; +} // namespace detz::shell diff --git a/taicpp/src/shell/glow.cpp b/taicpp/src/shell/glow.cpp new file mode 100644 index 0000000..d6c7075 --- /dev/null +++ b/taicpp/src/shell/glow.cpp @@ -0,0 +1,306 @@ +#include "glow.h" +#include "ansi.h" +#include "pypath.h" +#include "text.h" +#include "toolargs.h" +#include "utf8.h" +#include +#include +namespace detz::shell { +namespace { +std::string Fmt(const char* f, double v) { char b[64]; std::snprintf(b, sizeof b, f, v); return b; } +std::string Join(const std::vector& v, const std::string& sep) { return Utf8::Join(v, sep); } +std::string Squash(const std::string& s) { return Join(Utf8::SplitWhitespace(s), " "); } // " ".join(s.split()) +std::string Trunc(const std::string& s, long width) { return AnsiText::Truncate(s, width); } +} // namespace + +bool Glow::Enabled() { return AnsiText::ColorEnabled(); } +bool Glow::StderrEnabled() { return AnsiText::StderrColorEnabled(); } +int Glow::Width(int fallback) { return AnsiText::TerminalWidth(fallback); } + +std::string Glow::AccentFor(const std::string& name) { + if (name == "shell") return Palette::kFgRun; + if (name == "fork") return Palette::kFgFork; + if (name == "poll") return Palette::kFgFile; + if (name == "write_file" || name == "edit_file" || name == "delete_file") return Ansi::kGreen; + if (name == "ask_user") return Palette::kFgOrange; + if (name == "store_secret" || name == "list_secrets" || name == "delete_secret") return Palette::kFgYolo; + return Ansi::kDim; +} +std::string Glow::Bar(const std::string& name) { return AccentFor(name) + "\xe2\x96\x8c" + Ansi::kReset + " "; } // ▌ + +std::string Glow::EventRow(const std::string& icon, const std::string& verb, const std::string& target, const std::string& meta, long width, const std::string& iconColor) { + std::string row = std::string(Ansi::kBold) + iconColor + icon + Ansi::kReset + " " + Ansi::kBold + verb + Ansi::kReset; + if (!target.empty()) row += std::string(" ") + Palette::kFgFile + target + Ansi::kReset; + if (!meta.empty()) row += std::string(" ") + Ansi::kDim + meta + Ansi::kReset; + return width ? Trunc(row, width) : row; +} +std::string Glow::PreviewRow(const std::string& text, long width) { + std::string row = std::string(Ansi::kDim) + "\xe2\x94\x82 " + text + Ansi::kReset; // │ + return width ? Trunc(row, width) : row; +} +std::string Glow::IconFor(const std::string& name) { + if (name == "shell") return "\xe2\x96\xb6"; // ▶ + if (name == "fork") return "\xe2\x9c\xa6"; // ✦ + if (name == "poll") return "\xe2\x9f\xb3"; // ⟳ + return "\xe2\x97\x87"; // ◇ +} +std::string Glow::Ellipsis(const std::string& text, long cap, bool head) { + std::u32string u = Utf8::Decode(text); + if ((long)u.size() <= cap || cap < 8) return text; + if (head) return Utf8::Encode(u.substr(0, cap - 1)) + "\xe2\x80\xa6"; // … + long keep = cap - 1, front = keep / 2; + return Utf8::Encode(u.substr(0, front)) + "\xe2\x80\xa6" + Utf8::Encode(u.substr(u.size() - (keep - front))); +} +std::string Glow::DoneRow(const std::string& verb, const std::string& target, long chars, double elapsedMs, bool failed, long width) { + std::string elapsed = elapsedMs >= 1000 ? Fmt("%.1f", elapsedMs / 1000.0) + "s" : std::to_string((long)elapsedMs) + "ms"; + std::string mark = failed ? std::string(Ansi::kRed) + "\xe2\x9c\x97" + Ansi::kReset : std::string(Ansi::kGreen) + "\xe2\x9c\x93" + Ansi::kReset; // ✗ ✓ + std::string tail = std::string(Ansi::kDim) + "\xc2\xb7 " + elapsed + " \xc2\xb7 " + std::to_string(chars) + " chars" + Ansi::kReset; // · + std::string left = std::string(Ansi::kBold) + "\xe2\x97\x86" + Ansi::kReset + " " + Ansi::kBold + verb + Ansi::kReset; // ◆ + if (!target.empty()) left += std::string(" ") + Palette::kFgFile + target + Ansi::kReset; + std::string dot = std::string(Ansi::kDim) + "\xc2\xb7" + Ansi::kReset; + if (width) { + long pl = (long)AnsiText::PlainLen(left), pr = (long)AnsiText::PlainLen(mark + " " + tail); + if (pl + 2 + pr <= width) return left + std::string((size_t)(width - pl - pr), ' ') + mark + " " + tail; + return Trunc(left + " " + dot + " " + mark + " " + tail, width); + } + return left + " " + dot + " " + mark + " " + tail; +} +std::string Glow::Hairline(long width) { + if (!width) width = Width(); + std::string line; + for (long i = 0; i < std::max(1L, width); ++i) line += "\xe2\x94\x80"; + return std::string(Ansi::kDim) + line + Ansi::kReset; +} +std::string Glow::Shimmer(const std::string& text, long pos, long span) { + if (text.empty()) return ""; + std::u32string u = Utf8::Decode(text); + long cycle = (long)u.size() + span * 2; + long center = (pos % cycle) - span; + std::string out; + for (long p = 0; p < (long)u.size(); ++p) { + long dist = std::labs(p - center); + std::string ch = Utf8::EncodeOne(u[p]); + if (dist == 0) out += std::string(Ansi::kBold) + Palette::kFgBright + ch; + else if (dist <= span) out += std::string(Ansi::kBold) + ch; + else out += std::string(Ansi::kDim) + ch; + } + return out + Ansi::kReset; +} + +CallDescription Glow::DescribeCall(const std::string& name, const std::string& rawArguments) { + bool ok = false; + JsonValue args = JsonValue::Parse(rawArguments.empty() ? "{}" : rawArguments, &ok); + bool parsed = ok && args.is_object(); + return Describe(name, parsed ? args : JsonValue::Object(), parsed, rawArguments); +} +CallDescription Glow::DescribeCall(const std::string& name, const JsonValue& rawArguments) { + bool parsed = rawArguments.is_object(); + return Describe(name, parsed ? rawArguments : JsonValue::Object(), parsed, parsed ? "" : ToolArgs::Text(rawArguments)); +} + +CallDescription Glow::Describe(const std::string& name, const JsonValue& argsIn, bool parsed, const std::string& rawText) { + JsonValue args = argsIn; + if (name == "store_secret" && args.Get("value")) args.Set("value", JsonValue::String("[hidden]")); + auto head = [&](const std::string& key, long cap = 80) -> std::string { + const JsonValue* v = args.Get(key); + if (!v || v->is_null()) return ""; + return Utf8::Head(Squash(v->Str()), cap); + }; + auto base = [&](const std::string& key) -> std::string { + std::string value = head(key); + std::string trimmed = value; + while (!trimmed.empty() && trimmed.back() == '/') trimmed.pop_back(); + std::string b = PyPath::Basename(trimmed); + return b.empty() ? value : b; + }; + std::string path = base("path"); + if (path.empty()) path = base("file"); + std::string command = head("command"), query = head("query"), url = head("url"); + std::string prompt = head("prompt", 60), task = head("task", 60), who = head("name"); + std::string ident = head("id"); + if (ident.empty()) ident = head("target"); + if (ident.empty()) ident = head("agent_id"); + std::string when = head("at"); + if (when.empty()) when = head("every").empty() ? "" : "every " + head("every"); + const JsonValue* specp = args.Get("spec"); + JsonValue spec = (specp && specp->is_object()) ? *specp : JsonValue::Object(); + std::string flowTitle; + { + const JsonValue* title = spec.Get("title"); + if (title && title->Truthy()) flowTitle = title->Str(); + else { + const JsonValue* screens = spec.Get("screens"); + long n = 0; + if (screens && screens->Truthy()) { + if (screens->is_array()) n = (long)screens->items().size(); + else if (screens->is_object()) n = (long)screens->members().size(); + else if (screens->is_string()) n = (long)Utf8::Len(screens->string_value()); + } + flowTitle = std::to_string(n) + " screens"; + } + } + std::string flow = Utf8::Head(Squash(flowTitle), 60); + struct Row { std::string verb, target, extra; }; + const std::map table = { + {"shell", {"Run", command, ""}}, + {"read_file", {"Read", path, ""}}, + {"write_file", {"Write", path, ""}}, + {"edit_file", {"Edit", path, ""}}, + {"delete_file", {"Delete", path, ""}}, + {"web_search", {"Search", query, ""}}, + {"web_fetch", {"Fetch", url, ""}}, + {"describe_image", {"See", path.empty() ? head("image") : path, ""}}, + {"speak", {"Speak", head("text", 60), ""}}, + {"listen", {"Listen", head("seconds"), ""}}, + {"remember", {"Remember", head("instruction", 60), ""}}, + {"recall", {"Recall", query, ""}}, + {"load_skill", {"Skill", who, ""}}, + {"create_skill", {"Skill", who, ""}}, + {"install_skill", {"Install", who.empty() ? head("source") : who, ""}}, + {"get_current_terminal_content", {"Capture", "terminal", ""}}, + {"list_terminals", {"Terminals", "", ""}}, + {"tail_terminals", {"Tail", "terminals", ""}}, + {"export_terminal_log", {"Export", "terminal log", ""}}, + {"plexer_send", {"Send", ident, ""}}, + {"plexer_read", {"Read", ident, ""}}, + {"plexer_type", {"Type", ident, ""}}, + {"plexer_status", {"Plexer", "", ""}}, + {"plexer_show", {"Show", flow, ""}}, + {"tui_screen", {"TUI", "screen", ""}}, + {"tui_keys", {"TUI", "keys", ""}}, + {"tui_type", {"TUI", "type", ""}}, + {"tui_click", {"TUI", "click", ""}}, + {"tui_goto", {"TUI", "goto", ""}}, + {"tui_result", {"TUI", "result", ""}}, + {"fork", {"Fork", task, ""}}, + {"poll", {"Poll", ident, ""}}, + {"ask_user", {"Ask", head("question", 60), ""}}, + {"sysinfo", {"Probe", "", ""}}, + {"store_secret", {"Seal", who, ""}}, + {"list_secrets", {"Seal", "list", ""}}, + {"delete_secret", {"Delete", who, ""}}, + {"schedule", {"Schedule", prompt, when}}, + {"unschedule", {"Unschedule", ident, ""}}, + {"schedules", {"Schedules", "", ""}}, + {"record_save", {"Save", head("title"), ""}}, + {"record_read", {"Read", ident, ""}}, + {"record_search", {"Search", query, ""}}, + {"record_delete", {"Delete", ident, ""}}, + {"graph_link", {"Link", "", ""}}, + {"graph_query", {"Graph", head("node"), ""}}, + {"audit", {"Audit", path, ""}}, + {"restore", {"Restore", ident, ""}}, + {"release", {"Release", head("part"), head("message", 60)}}, + {"tags", {"Tags", head("prefix"), ""}}, + {"install", {"Install", head("component"), ""}}, + {"create_bot", {"Bot", who, ""}}, + {"search", {"Search", query, ""}}, + }; + static const std::map primary = { + {"shell", "command"}, {"read_file", "path"}, {"write_file", "path"}, {"edit_file", "path"}, + {"delete_file", "path"}, {"web_search", "query"}, {"web_fetch", "url"}, {"describe_image", "path"}, + {"speak", "text"}, {"remember", "instruction"}, {"recall", "query"}, {"load_skill", "name"}, + {"create_skill", "name"}, {"install_skill", "source"}, {"fork", "task"}, {"ask_user", "question"}, + {"delete_secret", "name"}, {"schedule", "prompt"}, {"unschedule", "id"}, {"record_save", "title"}, + {"record_read", "id"}, {"record_search", "query"}, {"record_delete", "id"}, {"graph_query", "node"}, + {"audit", "path"}, {"restore", "id"}, {"release", "part"}, {"tags", "prefix"}, + {"install", "component"}, {"create_bot", "name"}, {"search", "query"}}; + auto argsText = [&]() -> std::string { + return Squash(parsed ? ToolArgs::Text(args) : (rawText.empty() ? std::string("{}") : rawText)); + }; + auto it = table.find(name); + if (it == table.end()) return {name, "", "", argsText()}; + const Row& row = it->second; + std::vector bits; + if (!row.extra.empty()) bits.push_back(row.extra); + bits.push_back(name); + std::string preview; + if (name == "poll" && !ident.empty()) preview = "agent " + ident; + else if (primary.count(name)) preview = head(primary.at(name), 200); + if (preview.empty()) preview = argsText(); + return {row.verb, Ellipsis(row.target, 60), Join(bits, " \xc2\xb7 "), preview}; +} + +std::string Glow::Statusbar(const std::string& version, const std::string& profile, const std::string& directory, bool yolo, long agents, long width) { + std::string home = PyPath::Expanduser("~"); + std::string shortDir = directory; + if (directory == home) shortDir = "~"; + else if (!home.empty() && Utf8::StartsWith(directory, home + "/")) shortDir = "~" + directory.substr(home.size()); + std::vector segs = { + std::string(Ansi::kBold) + Palette::kFgFile + "tai " + version + Ansi::kReset, + std::string(Palette::kFgFile) + profile + Ansi::kReset, + std::string(Ansi::kDim) + shortDir + Ansi::kReset}; + if (yolo) segs.push_back(std::string(Ansi::kBold) + Palette::kFgYolo + "YOLO" + Ansi::kReset); + if (agents) segs.push_back(std::string(Ansi::kYellow) + "+" + std::to_string(agents) + Ansi::kReset); + std::string dot = std::string(Ansi::kDim) + "\xc2\xb7" + Ansi::kReset; + if (!width) width = Width(); + auto joined = [&](const std::vector& v) { + std::vector padded; + for (auto& s : v) padded.push_back(" " + s + " "); + return Utf8::Strip(Join(padded, dot)); + }; + std::string flow = joined(segs); + if ((yolo || agents) && segs.size() > 3) { + std::string left = joined({segs[0], segs[1], segs[2]}); + std::string right = joined(std::vector(segs.begin() + 3, segs.end())); + long gap = width - (long)AnsiText::PlainLen(left) - (long)AnsiText::PlainLen(right); + if (gap >= 4) return left + std::string((size_t)gap, ' ') + right; + std::vector keep = {segs[0], segs[1]}; + keep.insert(keep.end(), segs.begin() + 3, segs.end()); + flow = joined(keep); + } + if ((long)AnsiText::PlainLen(flow) <= width) return flow; + std::string core = AnsiText::Truncate(flow, width); + // re.sub(r"(?:\033\[[0-9;]*m|[ ·])+$", "", core) + std::u32string u = Utf8::Decode(core); + std::vector ok(u.size() + 1, 0); + ok[u.size()] = 1; + for (long i = (long)u.size() - 1; i >= 0; --i) { + if ((u[i] == ' ' || u[i] == 0xB7) && ok[i + 1]) { ok[i] = 1; continue; } + if (u[i] == 0x1b && i + 1 < (long)u.size() && u[i + 1] == '[') { + size_t j = i + 2; + while (j < u.size() && ((u[j] >= '0' && u[j] <= '9') || u[j] == ';')) ++j; + if (j < u.size() && u[j] == 'm' && ok[j + 1]) ok[i] = 1; + } + } + size_t cut = u.size(); + for (size_t i = 0; i < u.size(); ++i) if (ok[i]) { cut = i; break; } + return Utf8::Encode(u.substr(0, cut)) + Ansi::kReset; +} + +std::string Glow::PromptText() { return std::string(Ansi::kBold) + Palette::kFgOrange + "\xe2\x80\xba " + Ansi::kReset; } // › + +std::string Glow::AgentRowText(const AgentRow& r, long width, const std::string& pulse, const std::string& suffix) { + std::string dot = Ansi::kDim; + if (r.status == "running") dot = Ansi::kYellow; + else if (r.status == "done") dot = Ansi::kGreen; + else if (r.status == "error") dot = Ansi::kRed; + else if (r.status == "timeout") dot = Palette::kFgOrange; + std::string task = Ellipsis(Squash(r.task), 50, true); + std::string row = " #" + std::to_string(r.id) + " " + pulse + dot + "\xe2\x97\x8f" + Ansi::kReset + " " + task + " " + Ansi::kDim + "\xc2\xb7 " + r.status + " \xc2\xb7 " + FormatDelay((long)r.elapsed) + Ansi::kReset; + if (r.tools && *r.tools == 0 && r.status != "running") row += std::string(" ") + Palette::kFgOrange + "\xc2\xb7 0 tools" + Ansi::kReset; + if (!suffix.empty()) row += suffix; + return width ? Trunc(row, width) : row; +} + +std::vector Glow::Lanes(const std::vector& records, long cap, long width, const std::string& pulse) { + std::vector running, pending, view; + for (auto& r : records) if (r.status == "running") running.push_back(&r); + for (auto& r : records) if ((r.status == "done" || r.status == "timeout" || r.status == "error") && !r.collected) pending.push_back(&r); + view = running; view.insert(view.end(), pending.begin(), pending.end()); + std::string head = "\xe2\x97\x87 lanes \xc2\xb7 " + std::to_string(running.size()) + " running"; // ◇ · + if (!pending.empty()) head += " \xc2\xb7 " + std::to_string(pending.size()) + " to collect"; + std::string headRow = std::string(Ansi::kDim) + head + Ansi::kReset; + std::vector rows = {width ? Trunc(headRow, width) : headRow}; + for (size_t i = 0; i < view.size() && (long)i < cap; ++i) { + const AgentRow* r = view[i]; + bool isPending = false; + for (auto* p : pending) if (p == r) isPending = true; + std::string suffix = isPending ? std::string(" ") + Ansi::kDim + "\xc2\xb7 to collect" + Ansi::kReset : ""; + rows.push_back(AgentRowText(*r, width, r->status == "running" ? pulse : "", suffix)); + } + if ((long)view.size() > cap) rows.push_back(std::string(Ansi::kDim) + " \xc2\xb7 +" + std::to_string(view.size() - cap) + " more" + Ansi::kReset); + return rows; +} +} // namespace detz::shell diff --git a/taicpp/src/shell/glow.h b/taicpp/src/shell/glow.h new file mode 100644 index 0000000..8c08065 --- /dev/null +++ b/taicpp/src/shell/glow.h @@ -0,0 +1,47 @@ +#pragma once +// Glow: the screenshot-style CLI event renderer of tai.py (class Glow). Every styling +// method always emits codes; call sites gate on Enabled(). +#include "jsonvalue.h" +#include +#include +#include +namespace detz::shell { +// One agent record as Glow.agent_row / Glow.lanes read it (list_agents() rows). +struct AgentRow { + long id{0}; + std::string status; // running | done | error | timeout + std::string task; + double elapsed{0}; // seconds (record.get("elapsed") or 0) + std::optional tools; // record.get("tools"); unset when the key is absent + bool collected{false}; +}; +// Glow.describe_call result: (verb, target, extra, preview) +struct CallDescription { std::string verb, target, extra, preview; }; + +class Glow { + public: + static bool Enabled(); + static bool StderrEnabled(); + static int Width(int fallback = 80); + static std::string AccentFor(const std::string& name); + static std::string Bar(const std::string& name); + // width 0 = no truncation + static std::string EventRow(const std::string& icon, const std::string& verb, const std::string& target, const std::string& meta = "", long width = 0, const std::string& iconColor = ""); + static std::string PreviewRow(const std::string& text, long width = 0); + static std::string IconFor(const std::string& name); + static std::string Ellipsis(const std::string& text, long cap = 60, bool head = false); + static std::string DoneRow(const std::string& verb, const std::string& target, long chars, double elapsedMs, bool failed = false, long width = 0); + static std::string Hairline(long width = 0); + static std::string Shimmer(const std::string& text, long pos, long span = 3); + // raw arguments as the model sent them (JSON text) + static CallDescription DescribeCall(const std::string& name, const std::string& rawArguments); + // raw arguments already parsed (any JSON value; an object counts as parsed) + static CallDescription DescribeCall(const std::string& name, const JsonValue& rawArguments); + static std::string Statusbar(const std::string& version, const std::string& profile, const std::string& directory, bool yolo = false, long agents = 0, long width = 0); + static std::string PromptText(); + static std::string AgentRowText(const AgentRow& record, long width = 0, const std::string& pulse = "", const std::string& suffix = ""); + static std::vector Lanes(const std::vector& records, long cap = 6, long width = 0, const std::string& pulse = ""); + private: + static CallDescription Describe(const std::string& name, const JsonValue& args, bool parsed, const std::string& rawText); +}; +} // namespace detz::shell diff --git a/taicpp/src/shell/lanes.cpp b/taicpp/src/shell/lanes.cpp new file mode 100644 index 0000000..823780f --- /dev/null +++ b/taicpp/src/shell/lanes.cpp @@ -0,0 +1,62 @@ +#include "lanes.h" +#include "ansi.h" +#include +namespace detz::shell { +const std::vector& LaneBoard::Pulse() { + static const std::vector p = {Ansi::kDim, "", Ansi::kBold, ""}; + return p; +} +LaneBoard::LaneBoard(std::string profile, AgentLister lister, int width, int cap, OutputSink* file) + : profile_(std::move(profile)), lister_(std::move(lister)), width_(width ? width : Glow::Width()), cap_(cap), file_(file) { + if (!file_) { ownedStdout_ = std::make_unique(stdout); file_ = ownedStdout_.get(); } +} +LaneBoard::~LaneBoard() { if (worker_.joinable()) { stop_ = true; worker_.join(); } } +bool LaneBoard::Available(bool quiet) { return !quiet && Glow::Enabled(); } +std::vector LaneBoard::Rows(const std::string& pulse) const { + return Glow::Lanes(lister_(profile_), cap_, width_, pulse); +} +void LaneBoard::Paint(const std::string& pulse) { + std::vector rows = Rows(pulse); + if (rows.empty()) return; + if (painted_ && painted_ != (int)rows.size()) { + for (auto& row : rows) file_->Write(row + "\n"); + file_->Flush(); + painted_ = (int)rows.size(); + return; + } + if (painted_) file_->Write("\033[" + std::to_string(painted_) + "A"); + for (auto& row : rows) file_->Write("\r\033[K" + row + "\n"); + file_->Flush(); + painted_ = (int)rows.size(); +} +void LaneBoard::Tick() { + size_t frame = 1; + while (!stop_) { + std::this_thread::sleep_for(std::chrono::duration(1.0 / kFps)); + if (stop_) break; + try { + Paint(Pulse()[frame % Pulse().size()]); + } catch (...) { + break; + } + ++frame; + } +} +void LaneBoard::Start() { + if (hasWorker_) return; + Paint(""); + hasWorker_ = true; + finished_ = false; + worker_ = std::thread([this] { Tick(); finished_ = true; }); +} +void LaneBoard::Stop() { + stop_ = true; + if (hasWorker_) { + hasWorker_ = false; + // join(timeout=2) + for (int i = 0; i < 200 && !finished_; ++i) std::this_thread::sleep_for(std::chrono::milliseconds(10)); + if (finished_) worker_.join(); else worker_.detach(); + } + Paint(""); +} +} // namespace detz::shell diff --git a/taicpp/src/shell/lanes.h b/taicpp/src/shell/lanes.h new file mode 100644 index 0000000..de016ca --- /dev/null +++ b/taicpp/src/shell/lanes.h @@ -0,0 +1,47 @@ +#pragma once +// LaneBoard: live lane block for poll waits: ticking elapsed, pulsing dots. Prints K lines +// once, then rewrites them in place at FPS until Stop() leaves a steady final frame. Rows +// only flip running to finished mid-wait; if the row count ever changes the board degrades +// to fresh static lines instead of mis-drawing over scrollback. (tai.py class LaneBoard.) +#include "glow.h" +#include "sink.h" +#include +#include +#include +#include +#include +#include +namespace detz::shell { +// list_agents(profile): the agent records of the profile (supplied by the agent core). +using AgentLister = std::function(const std::string& profile)>; + +class LaneBoard { + public: + static constexpr int kFps = 3; + // width 0 = Glow.width(); file null = stdout + LaneBoard(std::string profile, AgentLister lister, int width = 0, int cap = 6, OutputSink* file = nullptr); + ~LaneBoard(); + LaneBoard(const LaneBoard&) = delete; + LaneBoard& operator=(const LaneBoard&) = delete; + // LaneBoard.available(app): not app.quiet and Glow.enabled() + static bool Available(bool quiet); + std::vector Rows(const std::string& pulse = "") const; + void Start(); + void Stop(); + int painted() const { return painted_; } + static const std::vector& Pulse(); // PULSE = (DIM, "", BOLD, "") + private: + void Paint(const std::string& pulse); + void Tick(); + std::string profile_; + AgentLister lister_; + int width_, cap_; + OutputSink* file_; + std::unique_ptr ownedStdout_; + std::atomic stop_{false}; + std::thread worker_; + std::atomic finished_{false}; + bool hasWorker_{false}; + int painted_{0}; +}; +} // namespace detz::shell diff --git a/taicpp/src/shell/livestream.cpp b/taicpp/src/shell/livestream.cpp new file mode 100644 index 0000000..ded473d --- /dev/null +++ b/taicpp/src/shell/livestream.cpp @@ -0,0 +1,48 @@ +#include "livestream.h" +#include "ansi.h" +#include "text.h" +#include "utf8.h" +#include +#include +namespace detz::shell { +LiveStream::LiveStream(int height, int width, OutputSink* file, bool enabled) + : height_(std::max(1, height)), width_(width ? width : AnsiText::TerminalWidth(80)), file_(file), enabled_(enabled) { + if (!file_) { ownedStdout_ = std::make_unique(stdout); file_ = ownedStdout_.get(); } +} +void LiveStream::Feed(const std::string& line) { + if (!enabled_) return; + // line.split("\r")[-1].rstrip("\n").expandtabs(8) + std::string last = line; + size_t cr = last.rfind('\r'); + if (cr != std::string::npos) last = last.substr(cr + 1); + while (!last.empty() && last.back() == '\n') last.pop_back(); + std::string cleaned = AnsiText::Truncate(Utf8::ExpandTabs(last, 8), std::max(8, width_ - 2)); + lines_.push_back(cleaned); + if ((int)lines_.size() > height_) lines_.pop_front(); + if (shown_ < height_) { + file_->Write("\xe2\x94\x82 " + cleaned + "\n"); // │ + ++shown_; + } else { + file_->Write("\033[" + std::to_string(height_) + "A"); + for (auto& kept : lines_) file_->Write("\r\033[K\xe2\x94\x82 " + kept + "\n"); + } + file_->Flush(); +} +void LiveStream::Pause() { + lines_.clear(); + shown_ = 0; +} +void LiveStream::Close(const std::string& status) { + if (!enabled_) return; + if (shown_ == 0) file_->Write("\xe2\x94\x82 (no output)\n"); + file_->Write("\xe2\x94\x82 " + AnsiText::Truncate(status, std::max(8, width_ - 2)) + "\n"); + file_->Flush(); +} +std::string StreamStatus(int code, const std::string& output, double elapsed, bool timedOut) { + std::string detail = std::to_string(Utf8::SplitLines(output).size()) + " lines \xc2\xb7 " + HumanSize((long)Utf8::Encode(Utf8::Decode(output)).size()); + if (timedOut) return "timed out after 120s (killed) \xc2\xb7 " + detail; + char buf[64]; + std::snprintf(buf, sizeof buf, "%.1f", elapsed); + return "exit " + std::to_string(code) + " \xc2\xb7 " + buf + "s \xc2\xb7 " + detail; +} +} // namespace detz::shell diff --git a/taicpp/src/shell/livestream.h b/taicpp/src/shell/livestream.h new file mode 100644 index 0000000..bcb856d --- /dev/null +++ b/taicpp/src/shell/livestream.h @@ -0,0 +1,32 @@ +#pragma once +// LiveStream: the REPL's fixed-height scrolling window for a running command's output +// (tai.py class LiveStream), plus human_size / stream_status / truncate_ansi helpers. +#include "sink.h" +#include +#include +#include +namespace detz::shell { +constexpr int kStreamHeight = 4; // STREAM_HEIGHT + +class LiveStream : public StreamSink { + public: + // width 0 = terminal columns; file null = stdout + explicit LiveStream(int height = kStreamHeight, int width = 0, OutputSink* file = nullptr, bool enabled = true); + void Feed(const std::string& line) override; + // The terminal is about to be used for a question: leave the window where it is and + // start a fresh one below whatever gets printed. + void Pause() override; + void Close(const std::string& status) override; + int shown() const { return shown_; } + int height() const { return height_; } + private: + int height_, width_; + OutputSink* file_; + std::unique_ptr ownedStdout_; + bool enabled_; + std::deque lines_; + int shown_{0}; +}; +// stream_status(code, output, elapsed, timed_out) +std::string StreamStatus(int code, const std::string& output, double elapsed, bool timedOut); +} // namespace detz::shell diff --git a/taicpp/src/shell/proc.cpp b/taicpp/src/shell/proc.cpp new file mode 100644 index 0000000..39fe918 --- /dev/null +++ b/taicpp/src/shell/proc.cpp @@ -0,0 +1,352 @@ +#include "proc.h" +#include "utf8.h" +#include "jsonvalue.h" +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +extern char** environ; +namespace detz::shell { + +OsError::OsError(int err, const std::string& filename) + : ProcessError("[Errno " + std::to_string(err) + "] " + std::strerror(err) + (filename.empty() ? "" : ": '" + filename + "'")), err_(err) {} + +EnvMap EnvMap::FromEnviron() { + EnvMap m; + for (char** e = environ; e && *e; ++e) { + std::string kv = *e; + size_t eq = kv.find('=', 1); + if (eq == std::string::npos) continue; + m.Set(kv.substr(0, eq), kv.substr(eq + 1)); + } + return m; +} +bool EnvMap::Has(const std::string& key) const { + for (auto& kv : items_) if (kv.first == key) return true; + return false; +} +std::string EnvMap::Get(const std::string& key, const std::string& dflt) const { + for (auto& kv : items_) if (kv.first == key) return kv.second; + return dflt; +} +void EnvMap::Set(const std::string& key, const std::string& value) { + for (auto& kv : items_) if (kv.first == key) { kv.second = value; return; } + items_.emplace_back(key, value); +} +void EnvMap::SetDefault(const std::string& key, const std::string& value) { if (!Has(key)) items_.emplace_back(key, value); } +std::vector EnvMap::Envp() const { + std::vector out; + for (auto& kv : items_) out.push_back(kv.first + "=" + kv.second); + return out; +} + +double Subprocess::Now() { + struct timespec ts; + clock_gettime(CLOCK_REALTIME, &ts); + return (double)ts.tv_sec + (double)ts.tv_nsec / 1e9; +} +void Subprocess::Sleep(double seconds) { + if (seconds <= 0) return; + struct timespec ts; + ts.tv_sec = (time_t)seconds; + ts.tv_nsec = (long)((seconds - (double)ts.tv_sec) * 1e9); + nanosleep(&ts, nullptr); +} + +std::string Subprocess::Which(const std::string& name) { + auto ok = [](const std::string& p) { + struct stat st; + return ::stat(p.c_str(), &st) == 0 && !S_ISDIR(st.st_mode) && ::access(p.c_str(), X_OK) == 0; + }; + if (name.find('/') != std::string::npos) return ok(name) ? name : ""; + const char* penv = std::getenv("PATH"); + std::string path = penv ? penv : ":/bin:/usr/bin"; + if (path.empty()) return ""; + size_t i = 0; + while (i <= path.size()) { + size_t j = path.find(':', i); + if (j == std::string::npos) j = path.size(); + std::string dir = path.substr(i, j - i); + std::string full = dir.empty() ? name : dir + "/" + name; + if (ok(full)) return full; + i = j + 1; + } + return ""; +} + +namespace { +void CloseAllFromThree() { +#ifdef SYS_close_range + if (::syscall(SYS_close_range, 3u, ~0u, 4u /*CLOSE_RANGE_CLOEXEC*/) == 0) return; +#endif + long max = ::sysconf(_SC_OPEN_MAX); + if (max < 0 || max > 65536) max = 65536; + for (long fd = 3; fd < max; ++fd) { + int fl = ::fcntl((int)fd, F_GETFD); + if (fl >= 0) ::fcntl((int)fd, F_SETFD, fl | FD_CLOEXEC); + } +} +std::vector ExecutableList(const std::string& prog, const EnvMap* env) { + if (prog.find('/') != std::string::npos) return {prog}; + std::string path = env ? env->Get("PATH", ":/bin:/usr/bin") : (std::getenv("PATH") ? std::getenv("PATH") : ":/bin:/usr/bin"); + std::vector out; + size_t i = 0; + while (i <= path.size()) { + size_t j = path.find(':', i); + if (j == std::string::npos) j = path.size(); + std::string dir = path.substr(i, j - i); + out.push_back((dir.empty() ? "." : dir) + "/" + prog); + i = j + 1; + } + return out; +} +} // namespace + +Popen::Popen(const PopenOptions& o) { + std::vector argv = o.argv; + if (o.shellCommand) argv = {"/bin/sh", "-c", *o.shellCommand}; + if (argv.empty()) throw OsError(ENOENT, ""); + std::vector envp; + if (o.env) envp = o.env->Envp(); + std::vector exes = ExecutableList(argv[0], o.env); + int childIn = -1, childOut = -1, childErr = -1; + int parentIn = -1, parentOut = -1, parentErr = -1; + int devnull = -1; + auto nullfd = [&]() { if (devnull < 0) devnull = ::open("/dev/null", O_RDWR | O_CLOEXEC); return devnull; }; + auto mkpipe = [&](int* rd, int* wr) { int p[2]; if (::pipe2(p, O_CLOEXEC) != 0) throw OsError(errno, ""); *rd = p[0]; *wr = p[1]; }; + if (o.in == StdMode::Pipe) { int r, w; mkpipe(&r, &w); childIn = r; parentIn = w; } + else if (o.in == StdMode::Null) childIn = nullfd(); + else if (o.in == StdMode::Fd) childIn = o.inFd; + if (o.out == StdMode::Pipe) { int r, w; mkpipe(&r, &w); childOut = w; parentOut = r; } + else if (o.out == StdMode::Null) childOut = nullfd(); + else if (o.out == StdMode::Fd) childOut = o.outFd; + if (o.err == StdMode::Pipe) { int r, w; mkpipe(&r, &w); childErr = w; parentErr = r; } + else if (o.err == StdMode::Null) childErr = nullfd(); + else if (o.err == StdMode::Fd) childErr = o.errFd; + else if (o.err == StdMode::Merge) childErr = childOut; + int errpipe[2]; + if (::pipe2(errpipe, O_CLOEXEC) != 0) throw OsError(errno, ""); + // Marshal into C arrays before fork (no allocation in the child). + std::vector cargv, cenvp; + for (auto& a : argv) cargv.push_back(const_cast(a.c_str())); + cargv.push_back(nullptr); + for (auto& e : envp) cenvp.push_back(const_cast(e.c_str())); + cenvp.push_back(nullptr); + std::vector cexes; + for (auto& e : exes) cexes.push_back(e.c_str()); + const char* ccwd = o.cwd ? o.cwd->c_str() : nullptr; + pid_t pid = ::fork(); + if (pid < 0) { + int e = errno; + ::close(errpipe[0]); ::close(errpipe[1]); + throw OsError(e, ""); + } + if (pid == 0) { + auto fail = [&](char stage, int err) { + char buf[2] = {stage, (char)0}; + (void)!::write(errpipe[1], buf, 1); + (void)!::write(errpipe[1], &err, sizeof err); + ::_exit(255); + }; + // Move source fds out of the 0..2 range so the dup2s below cannot clobber each other. + auto lift = [&](int fd) { if (fd >= 0 && fd < 3) { int n = ::fcntl(fd, F_DUPFD_CLOEXEC, 10); return n; } return fd; }; + int si = lift(childIn), so = lift(childOut), se = (o.err == StdMode::Merge) ? -2 : lift(childErr); + if (si >= 0) ::dup2(si, 0); + if (so >= 0) ::dup2(so, 1); + if (se == -2) ::dup2(1, 2); else if (se >= 0) ::dup2(se, 2); + if (ccwd && ::chdir(ccwd) != 0) fail('C', errno); + ::signal(SIGPIPE, SIG_DFL); + ::signal(SIGXFSZ, SIG_DFL); + if (o.adoptCtty) { + if (::setsid() < 0) fail('T', errno); + if (::ioctl(0, TIOCSCTTY, 0) < 0) fail('T', errno); + } + CloseAllFromThree(); + int saved = 0; + for (const char* exe : cexes) { + if (o.env) ::execve(exe, cargv.data(), cenvp.data()); else ::execv(exe, cargv.data()); + if (errno != ENOENT && errno != ENOTDIR && saved == 0) saved = errno; + } + fail('E', saved ? saved : errno); + } + ::close(errpipe[1]); + if (devnull >= 0) ::close(devnull); + if (childIn >= 0 && o.in == StdMode::Pipe) ::close(childIn); + if (childOut >= 0 && o.out == StdMode::Pipe) ::close(childOut); + if (childErr >= 0 && o.err == StdMode::Pipe) ::close(childErr); + char stage[2] = {0, 0}; int err = 0; + ssize_t n; + do { n = ::read(errpipe[0], stage, 1); } while (n < 0 && errno == EINTR); + if (n == 1) { + ssize_t m; + do { m = ::read(errpipe[0], &err, sizeof err); } while (m < 0 && errno == EINTR); + } + ::close(errpipe[0]); + if (n == 1) { + int st; ::waitpid(pid, &st, 0); + for (int fd : {parentIn, parentOut, parentErr}) if (fd >= 0) ::close(fd); + throw OsError(err, stage[0] == 'C' ? (o.cwd ? *o.cwd : "") : (stage[0] == 'E' ? argv[0] : "")); + } + pid_ = pid; inFd_ = parentIn; outFd_ = parentOut; errFd_ = parentErr; +} + +Popen::~Popen() { + CloseStdin(); + if (outFd_ >= 0) ::close(outFd_); + if (errFd_ >= 0) ::close(errFd_); +} +void Popen::CloseStdin() { if (inFd_ >= 0) { ::close(inFd_); inFd_ = -1; } } +bool Popen::Poll() { + if (exited_) return true; + if (pid_ < 0) return true; + int st = 0; + pid_t r = ::waitpid(pid_, &st, WNOHANG); + if (r == 0) return false; + if (r < 0) { exited_ = true; returncode_ = 0; return true; } + exited_ = true; + returncode_ = WIFSIGNALED(st) ? -WTERMSIG(st) : WEXITSTATUS(st); + return true; +} +bool Popen::Wait(double timeoutSeconds) { + if (exited_) return true; + if (timeoutSeconds < 0) { + int st = 0; + pid_t r; + do { r = ::waitpid(pid_, &st, 0); } while (r < 0 && errno == EINTR); + exited_ = true; + returncode_ = r < 0 ? 0 : (WIFSIGNALED(st) ? -WTERMSIG(st) : WEXITSTATUS(st)); + return true; + } + double end = Subprocess::Now() + timeoutSeconds, delay = 0.0005; + while (true) { + if (Poll()) return true; + double remaining = end - Subprocess::Now(); + if (remaining <= 0) return false; + delay = std::min(delay * 2, 0.05); + Subprocess::Sleep(std::min(delay, remaining)); + } +} +void Popen::Kill() { if (pid_ > 0 && !exited_) ::kill(pid_, SIGKILL); } +void Popen::KillGroup() { + if (pid_ <= 0) return; + if (::killpg(pid_, SIGKILL) != 0 && !exited_) ::kill(pid_, SIGKILL); +} + +CompletedProcess Subprocess::Run(SubprocessOptions options) { + ::signal(SIGPIPE, SIG_IGN); // Python ignores SIGPIPE at startup; writes report EPIPE instead + if (options.input && options.popen.in == StdMode::Inherit) options.popen.in = StdMode::Pipe; + Popen proc(options.popen); + CompletedProcess done; + double endtime = options.timeout >= 0 ? Subprocess::Now() + options.timeout : -1; + auto remaining = [&]() { return endtime < 0 ? -1.0 : endtime - Subprocess::Now(); }; + auto expire = [&]() { + proc.Kill(); + proc.Wait(-1); + // str(TimeoutExpired): "Command '' timed out after seconds" + std::string cmd; + if (options.popen.shellCommand) cmd = *options.popen.shellCommand; + else { + JsonValue list = JsonValue::Array(); + for (auto& a : options.popen.argv) list.Push(JsonValue::String(a)); + cmd = list.Repr(); + } + std::string secs = options.timeout == (double)(long long)options.timeout ? std::to_string((long long)options.timeout) : JsonValue::Float(options.timeout).Repr(); + throw TimeoutExpired("Command '" + cmd + "' timed out after " + secs + " seconds"); + }; + std::string input = options.input.value_or(""); + size_t written = 0; + int inFd = proc.stdinFd(), outFd = proc.stdoutFd(), errFd = proc.stderrFd(); + if (inFd >= 0) { + ::fcntl(inFd, F_SETFL, ::fcntl(inFd, F_GETFL) | O_NONBLOCK); + if (input.empty()) proc.CloseStdin(), inFd = -1; + } + bool outOpen = outFd >= 0, errOpen = errFd >= 0; + while (inFd >= 0 || outOpen || errOpen) { + struct pollfd fds[3]; int n = 0, idx[3]; + if (inFd >= 0) { fds[n] = {inFd, POLLOUT, 0}; idx[n++] = 0; } + if (outOpen) { fds[n] = {outFd, POLLIN, 0}; idx[n++] = 1; } + if (errOpen) { fds[n] = {errFd, POLLIN, 0}; idx[n++] = 2; } + int wait = -1; + if (endtime >= 0) { + double r = remaining(); + if (r <= 0) expire(); + wait = (int)(r * 1000) + 1; + } + int rc = ::poll(fds, n, wait); + if (rc < 0) { if (errno == EINTR) continue; throw OsError(errno, ""); } + if (rc == 0) { if (endtime >= 0 && remaining() <= 0) expire(); continue; } + for (int k = 0; k < n; ++k) { + if (!fds[k].revents) continue; + if (idx[k] == 0) { + size_t chunk = std::min(input.size() - written, 512); + ssize_t w = ::write(inFd, input.data() + written, chunk); + if (w < 0 && (errno == EPIPE || errno == EBADF)) { proc.CloseStdin(); inFd = -1; continue; } + if (w > 0) written += (size_t)w; + if (written >= input.size()) { proc.CloseStdin(); inFd = -1; } + } else { + int fd = idx[k] == 1 ? outFd : errFd; + char buf[32768]; + ssize_t r = ::read(fd, buf, sizeof buf); + if (r > 0) (idx[k] == 1 ? done.out : done.err).append(buf, (size_t)r); + else if (r == 0 || (r < 0 && errno != EINTR && errno != EAGAIN)) { if (idx[k] == 1) outOpen = false; else errOpen = false; } + } + } + } + double r = remaining(); + if (endtime >= 0 && r <= 0 && !proc.Poll()) expire(); + if (!proc.Wait(endtime < 0 ? -1 : std::max(r, 0.0))) expire(); + done.returncode = proc.returncode(); + return done; +} + +std::string IncrementalUtf8::Decode(const std::string& bytes, bool final) { + std::string buf = pending_ + bytes; + pending_.clear(); + size_t keep = 0; + if (!final) { + size_t n = buf.size(); + for (size_t back = 1; back <= 3 && back <= n; ++back) { + unsigned char c = buf[n - back]; + if ((c & 0xC0) == 0x80) continue; // continuation: look further back + size_t need = c >= 0xF0 && c <= 0xF4 ? 4 : c >= 0xE0 && c <= 0xEF ? 3 : c >= 0xC2 && c <= 0xDF ? 2 : 0; + if (need && back < need) { + bool valid = true; + if (back >= 2) { + unsigned char c2 = buf[n - back + 1]; + unsigned char lo = 0x80, hi = 0xBF; + if (c == 0xE0) lo = 0xA0; else if (c == 0xED) hi = 0x9F; else if (c == 0xF0) lo = 0x90; else if (c == 0xF4) hi = 0x8F; + if (c2 < lo || c2 > hi) valid = false; + } + if (valid) keep = back; + } + break; + } + } + pending_ = buf.substr(buf.size() - keep); + buf.resize(buf.size() - keep); + return Utf8::Encode(Utf8::Decode(buf)); +} + +std::string UniversalNewlines::Feed(const std::string& text, bool final) { + std::string in = (pendingCr_ ? "\r" : "") + text; + pendingCr_ = false; + if (!final && !in.empty() && in.back() == '\r') { in.pop_back(); pendingCr_ = true; } + std::string out; + out.reserve(in.size()); + for (size_t i = 0; i < in.size(); ++i) { + if (in[i] == '\r') { out.push_back('\n'); if (i + 1 < in.size() && in[i + 1] == '\n') ++i; } + else out.push_back(in[i]); + } + if (final && pendingCr_) { out.push_back('\n'); pendingCr_ = false; } + return out; +} +} // namespace detz::shell diff --git a/taicpp/src/shell/proc.h b/taicpp/src/shell/proc.h new file mode 100644 index 0000000..fd2f108 --- /dev/null +++ b/taicpp/src/shell/proc.h @@ -0,0 +1,110 @@ +#pragma once +// Process primitives mirroring Python's subprocess as tai.py uses it: Popen (fork/exec with +// errno reporting, optional pty session), subprocess.run (capture, input, timeout), and +// shutil.which. Errors follow Python's classes: OsError (OSError), SubprocessError, +// TimeoutExpired. +#include +#include +#include +#include +#include +namespace detz::shell { + +class ProcessError : public std::runtime_error { using std::runtime_error::runtime_error; }; +// OSError: what() is str(exc), e.g. "[Errno 2] No such file or directory: 'nope'" +class OsError : public ProcessError { + public: + OsError(int err, const std::string& filename); + // an OSError subclass with its own text, e.g. socket.timeout: "timed out" + static OsError Text(const std::string& text, int err = 0) { return OsError(text, err, 0); } + int err() const { return err_; } + private: + OsError(const std::string& text, int err, int) : ProcessError(text), err_(err) {} + int err_; +}; +class SubprocessError : public ProcessError { using ProcessError::ProcessError; }; +class TimeoutExpired : public SubprocessError { using SubprocessError::SubprocessError; }; + +// Ordered environment (Python dict order matters for `env` output). +class EnvMap { + public: + static EnvMap FromEnviron(); + bool Has(const std::string& key) const; + std::string Get(const std::string& key, const std::string& dflt = "") const; + void Set(const std::string& key, const std::string& value); + void SetDefault(const std::string& key, const std::string& value); + std::vector Envp() const; + const std::vector>& items() const { return items_; } + private: + std::vector> items_; +}; + +enum class StdMode { Inherit, Null, Pipe, Merge /* stderr into stdout */, Fd /* explicit fd */ }; + +struct PopenOptions { + std::vector argv; // argv[0] resolved via PATH when it has no '/' + std::optional shellCommand; // shell=True: argv becomes {"/bin/sh","-c",cmd} + std::optional cwd; + const EnvMap* env{nullptr}; // null: inherit the process environment + StdMode in{StdMode::Inherit}, out{StdMode::Inherit}, err{StdMode::Inherit}; + int inFd{-1}, outFd{-1}, errFd{-1}; // used with StdMode::Fd + bool adoptCtty{false}; // setsid() + TIOCSCTTY on fd 0 (pty child) +}; + +class Popen { + public: + explicit Popen(const PopenOptions& options); // throws OsError when exec/chdir fails + ~Popen(); + Popen(const Popen&) = delete; + Popen& operator=(const Popen&) = delete; + int pid() const { return pid_; } + int stdinFd() const { return inFd_; } // parent end when StdMode::Pipe, else -1 + int stdoutFd() const { return outFd_; } + int stderrFd() const { return errFd_; } + void CloseStdin(); + // poll(): true when the child has exited (return code stored) + bool Poll(); + // wait(timeout): true when exited; timeout < 0 waits forever + bool Wait(double timeoutSeconds = -1); + void Kill(); // SIGKILL to the child only (Popen.kill) + void KillGroup(); // killpg(pid), falling back to kill + // Python returncode: exit status, or -signal + int returncode() const { return returncode_; } + bool exited() const { return exited_; } + private: + int pid_{-1}, inFd_{-1}, outFd_{-1}, errFd_{-1}, returncode_{0}; + bool exited_{false}; +}; + +struct SubprocessOptions { + PopenOptions popen; + std::optional input; // bytes written to the child's stdin + double timeout{-1}; // seconds, <0 none +}; +struct CompletedProcess { int returncode{0}; std::string out, err; }; + +class Subprocess { + public: + // subprocess.run(...): captures Pipe-mode streams; throws TimeoutExpired / OsError. + static CompletedProcess Run(SubprocessOptions options); + // shutil.which(name): first executable regular file on PATH, or "". + static std::string Which(const std::string& name); + static double Now(); // time.time() + static void Sleep(double seconds); +}; + +// Incremental UTF-8 decoder with errors="replace" (codecs incremental decoder, no final flush). +class IncrementalUtf8 { + public: + std::string Decode(const std::string& bytes, bool final = false); + private: + std::string pending_; +}; +// TextIOWrapper(universal_newlines=True) translation: \r\n and \r become \n across chunks. +class UniversalNewlines { + public: + std::string Feed(const std::string& text, bool final = false); + private: + bool pendingCr_{false}; +}; +} // namespace detz::shell diff --git a/taicpp/src/shell/pty.cpp b/taicpp/src/shell/pty.cpp new file mode 100644 index 0000000..47245a9 --- /dev/null +++ b/taicpp/src/shell/pty.cpp @@ -0,0 +1,80 @@ +#include "pty.h" +#include +#include +#include +#include +#include +#include +#include +namespace detz::shell { +bool PtyProc::Available() { return true; } + +PtyProc::PtyProc(const std::vector& argv, const std::optional& shellCommand, const std::optional& cwd, const EnvMap* env, int rows, int cols) { + int master = ::posix_openpt(O_RDWR | O_NOCTTY); + if (master < 0) throw OsError(errno, ""); + if (::grantpt(master) != 0 || ::unlockpt(master) != 0) { int e = errno; ::close(master); throw OsError(e, ""); } + char name[128]; + if (::ptsname_r(master, name, sizeof name) != 0) { int e = errno; ::close(master); throw OsError(e, ""); } + int slave = ::open(name, O_RDWR | O_NOCTTY); + if (slave < 0) { int e = errno; ::close(master); throw OsError(e, ""); } + ::fcntl(master, F_SETFD, FD_CLOEXEC); + try { + struct winsize ws = {}; + ws.ws_row = (unsigned short)rows; ws.ws_col = (unsigned short)cols; + ::ioctl(master, TIOCSWINSZ, &ws); + PopenOptions o; + o.argv = argv; + o.shellCommand = shellCommand; + o.cwd = cwd; + o.env = env; + o.in = o.out = o.err = StdMode::Fd; + o.inFd = o.outFd = o.errFd = slave; + o.adoptCtty = true; + proc_ = std::make_unique(o); + } catch (...) { + ::close(slave); + ::close(master); + throw; + } + ::close(slave); + master_ = master; +} +PtyProc::~PtyProc() { Close(); } + +PtyProc::ReadResult PtyProc::Read(double timeoutSeconds) { + ReadResult r; + struct pollfd pfd = {master_, POLLIN, 0}; + int ms = (int)(std::max(0.0, timeoutSeconds) * 1000.0); + int rc; + do { rc = ::poll(&pfd, 1, ms); } while (rc < 0 && errno == EINTR); + if (rc <= 0) return r; // select() reported nothing readable + char buf[65536]; + ssize_t n; + do { n = ::read(master_, buf, sizeof buf); } while (n < 0 && errno == EINTR); + if (n <= 0) { r.kind = ReadKind::End; return r; } // EOF, or OSError (EIO) once the slave closed + r.kind = ReadKind::Data; + r.data.assign(buf, (size_t)n); + return r; +} +void PtyProc::Write(const std::string& text) { WriteBytes(text); } +void PtyProc::WriteBytes(const std::string& data) { + size_t off = 0; + while (off < data.size()) { + ssize_t n = ::write(master_, data.data() + off, data.size() - off); + if (n < 0) { if (errno == EINTR) continue; throw OsError(errno, ""); } + off += (size_t)n; + } +} +void PtyProc::Resize(int rows, int cols) { + struct winsize ws = {}; + ws.ws_row = (unsigned short)rows; ws.ws_col = (unsigned short)cols; + if (::ioctl(master_, TIOCSWINSZ, &ws) != 0) throw OsError(errno, ""); +} +bool PtyProc::EchoOff() const { + struct termios t; + if (::tcgetattr(master_, &t) != 0) return false; + return !(t.c_lflag & ECHO); +} +void PtyProc::Kill() { if (proc_) proc_->KillGroup(); } +void PtyProc::Close() { if (master_ >= 0) { ::close(master_); master_ = -1; } } +} // namespace detz::shell diff --git a/taicpp/src/shell/pty.h b/taicpp/src/shell/pty.h new file mode 100644 index 0000000..da2f5e5 --- /dev/null +++ b/taicpp/src/shell/pty.h @@ -0,0 +1,38 @@ +#pragma once +// PtyProc: a child on its own pty with the pty as controlling terminal, so ssh, sudo and +// friends can prompt exactly as in a real terminal (tai.py class PtyProc). One reader +// (poll), one writer; the caller owns the loop. +#include "proc.h" +#include +#include +#include +#include +namespace detz::shell { +constexpr int kRunRows = 24; // RUN_ROWS +constexpr int kRunCols = 120; // RUN_COLS + +class PtyProc { + public: + enum class ReadKind { Nothing, Data, End }; + struct ReadResult { ReadKind kind{ReadKind::Nothing}; std::string data; }; + + static bool Available(); + // argv form (shellCommand unset) or shell form (shellCommand set, argv ignored) + PtyProc(const std::vector& argv, const std::optional& shellCommand, const std::optional& cwd, const EnvMap* env, int rows = kRunRows, int cols = kRunCols); + ~PtyProc(); + PtyProc(const PtyProc&) = delete; + PtyProc& operator=(const PtyProc&) = delete; + // read(timeout): Nothing when no bytes arrived in time, End at end of output (EOF or EIO). + ReadResult Read(double timeoutSeconds); + void Write(const std::string& text); + void WriteBytes(const std::string& data); + void Resize(int rows, int cols); + bool EchoOff() const; + void Kill(); + void Close(); + Popen& proc() { return *proc_; } + private: + int master_{-1}; + std::unique_ptr proc_; +}; +} // namespace detz::shell diff --git a/taicpp/src/shell/runlive.cpp b/taicpp/src/shell/runlive.cpp new file mode 100644 index 0000000..5e91f0b --- /dev/null +++ b/taicpp/src/shell/runlive.cpp @@ -0,0 +1,277 @@ +#include "runlive.h" +#include "pty.h" +#include "utf8.h" +#include +#include +#include +#include +#include +#include +#include +#include +namespace detz::shell { +namespace { +double Now() { return Subprocess::Now(); } + +PopenOptions MakeOptions(const Command& c, const RunLiveOptions& o) { + PopenOptions p; + p.argv = c.argv; + p.shellCommand = c.shell; + p.cwd = o.cwd; + p.env = o.env; + return p; +} + +// Shared between run_live and its reader thread; outlives both when the thread is abandoned. +struct PipeState { + std::mutex mu; + std::condition_variable cv; + std::deque> pending; // nullopt = end marker + std::vector chunks; + bool done{false}; +}; +} // namespace + +std::string RunLive::StripControl(const std::string& s) { + // \x1b\[[0-9;?]*[ -/]*[@-~] | \x1b\][^\x07]*\x07 | [\x00-\x08\x0b\x0c\x0e-\x1f\x7f] + std::string out; + size_t i = 0, n = s.size(); + while (i < n) { + unsigned char c = s[i]; + if (c == 0x1b && i + 1 < n && s[i + 1] == '[') { + size_t j = i + 2; + while (j < n && ((s[j] >= '0' && s[j] <= '9') || s[j] == ';' || s[j] == '?')) ++j; + while (j < n && s[j] >= ' ' && s[j] <= '/') ++j; + if (j < n && s[j] >= '@' && s[j] <= '~') { i = j + 1; continue; } + } + if (c == 0x1b && i + 1 < n && s[i + 1] == ']') { + size_t j = s.find('\x07', i + 2); + if (j != std::string::npos) { i = j + 1; continue; } + } + bool ctrl = c <= 0x08 || c == 0x0b || c == 0x0c || (c >= 0x0e && c <= 0x1f) || c == 0x7f; + if (ctrl) { ++i; continue; } + out.push_back((char)c); + ++i; + } + return out; +} + +bool RunLive::LooksLikePrompt(const std::string& text) { + std::string t = Utf8::Rstrip(text); + if (t.empty()) return false; + return std::string(":?>])$#%").find(t.back()) != std::string::npos; +} + +RunResult RunLive::Run(const Command& command, const RunLiveOptions& options) { + if (options.ask) return RunPty(command, options); + double started = Now(); + PopenOptions po = MakeOptions(command, options); + po.in = StdMode::Null; + po.out = StdMode::Pipe; + po.err = StdMode::Merge; + auto proc = std::make_unique(po); + auto state = std::make_shared(); + int readFd = ::dup(proc->stdoutFd()); // the reader owns its own descriptor + std::thread reader([state, readFd]() { + IncrementalUtf8 dec; + UniversalNewlines nl; + std::string buf; + auto emitLines = [&](bool flush) { + size_t pos; + while ((pos = buf.find('\n')) != std::string::npos) { + std::string line = buf.substr(0, pos + 1); + buf.erase(0, pos + 1); + std::lock_guard lock(state->mu); + state->chunks.push_back(line); + state->pending.push_back(line); + state->cv.notify_all(); + } + if (flush && !buf.empty()) { + std::lock_guard lock(state->mu); + state->chunks.push_back(buf); + state->pending.push_back(buf); + state->cv.notify_all(); + buf.clear(); + } + }; + char raw[8192]; + while (true) { + ssize_t n = ::read(readFd, raw, sizeof raw); + if (n < 0 && errno == EINTR) continue; + if (n <= 0) { + buf += nl.Feed(dec.Decode("", true), true); + emitLines(true); + break; + } + buf += nl.Feed(dec.Decode(std::string(raw, (size_t)n)), false); + emitLines(false); + } + ::close(readFd); + std::lock_guard lock(state->mu); + state->pending.push_back(std::nullopt); + state->done = true; + state->cv.notify_all(); + }); + bool timedOut = false; + auto reap = [&]() { + if (timedOut) proc->Kill(); + if (!proc->Wait(10)) { proc->Kill(); proc->Wait(-1); } + // reader.join(timeout=10) + { + std::unique_lock lock(state->mu); + state->cv.wait_for(lock, std::chrono::seconds(10), [&] { return state->done; }); + } + bool finished; + { std::lock_guard lock(state->mu); finished = state->done; } + if (finished) reader.join(); else reader.detach(); + }; + try { + double deadline = started + options.timeout; + while (true) { + double remaining = deadline - Now(); + if (remaining <= 0) { timedOut = true; break; } + std::optional item; + bool got = false; + { + std::unique_lock lock(state->mu); + state->cv.wait_for(lock, std::chrono::duration(std::min(0.05, remaining)), [&] { return !state->pending.empty(); }); + if (!state->pending.empty()) { item = state->pending.front(); state->pending.pop_front(); got = true; } + } + if (!got) continue; + if (!item) break; + if (options.stream) options.stream->Feed(*item); + } + } catch (...) { + proc->Kill(); + proc->Wait(-1); + reap(); + throw; + } + reap(); + if (!timedOut) { + while (true) { + std::optional item; + { + std::lock_guard lock(state->mu); + if (state->pending.empty()) break; + item = state->pending.front(); + state->pending.pop_front(); + } + if (!item) break; + if (options.stream) options.stream->Feed(*item); + } + } + RunResult r; + r.code = proc->returncode(); + { + std::lock_guard lock(state->mu); + for (auto& c : state->chunks) r.output += c; + } + r.elapsed = Now() - started; + r.timedOut = timedOut; + return r; +} + +RunResult RunLive::RunPty(const Command& command, const RunLiveOptions& options) { + double started = Now(); + EnvMap env = options.env ? *options.env : EnvMap::FromEnviron(); + env.SetDefault("PAGER", "cat"); + env.SetDefault("GIT_PAGER", "cat"); + PtyProc term(command.argv, command.shell, options.cwd, &env); + IncrementalUtf8 decoder; + std::vector chunks; + long size = 0; + std::string partial, asked; + int asks = 0; + bool declined = false, timedOut = false; + double deadline = started + options.timeout; + double lastOutput = Now(); + StreamSink* stream = options.stream; + + auto emit = [&](const std::string& raw) { + std::string r = raw; + while (!r.empty() && r.back() == '\r') r.pop_back(); + size_t cr = r.rfind('\r'); + if (cr != std::string::npos) r = r.substr(cr + 1); + std::string line = StripControl(r); + if (size < kRunOutputCap) { + chunks.push_back(line + "\n"); + size += (long)Utf8::Len(line) + 1; + } + if (stream) stream->Feed(line + "\n"); + }; + auto take = [&](const std::string& data) { + partial += decoder.Decode(data); + size_t pos; + while ((pos = partial.find('\n')) != std::string::npos) { + std::string line = partial.substr(0, pos); + partial.erase(0, pos + 1); + emit(line); + } + }; + auto finish = [&]() { + if (timedOut) term.Kill(); + if (!term.proc().Wait(10)) { term.Kill(); term.proc().Wait(-1); } + term.Close(); + }; + try { + while (true) { + double remaining = deadline - Now(); + if (remaining <= 0) { timedOut = true; break; } + PtyProc::ReadResult rd = term.Read(std::min(0.05, remaining)); + if (rd.kind == PtyProc::ReadKind::End) break; + if (rd.kind == PtyProc::ReadKind::Data) { + take(rd.data); + lastOutput = Now(); + continue; + } + if (term.proc().Poll()) { + while (true) { + PtyProc::ReadResult more = term.Read(0); + if (more.kind != PtyProc::ReadKind::Data) break; + take(more.data); + } + break; + } + std::string text = Utf8::Strip(StripControl(partial)); + if (declined || asks >= kRunMaxAsks || text.empty() || text == asked || Now() - lastOutput < kRunPromptIdle) continue; + bool secret = term.EchoOff(); + if (!secret && (!LooksLikePrompt(text) || partial.find('\r') != std::string::npos)) continue; + asked = text; + asks += 1; + if (stream) stream->Pause(); + double waited = Now(); + std::optional answer = options.ask(text, secret); + deadline += Now() - waited; + lastOutput = Now(); + std::string note; + if (!answer) { + declined = true; + term.Write("\x04"); + note = "\xc2\xbb " + text + " (no answer)"; // » + } else { + std::string a = Utf8::ReplaceAll(*answer, "\r", ""); + size_t nl = a.find('\n'); + if (nl != std::string::npos) a = a.substr(0, nl); + a = Utf8::Head(a, (long)kRunAnswerCap); + term.Write(a + "\n"); + note = "\xc2\xbb " + text + " " + (secret ? std::string("(hidden)") : a); + } + if (stream) stream->Feed(note + "\n"); + } + } catch (...) { + term.Kill(); + term.proc().Wait(-1); + finish(); + throw; + } + finish(); + if (!Utf8::Strip(partial).empty()) emit(partial); + RunResult r; + r.code = term.proc().returncode(); + for (auto& c : chunks) r.output += c; + r.elapsed = Now() - started; + r.timedOut = timedOut; + return r; +} +} // namespace detz::shell diff --git a/taicpp/src/shell/runlive.h b/taicpp/src/shell/runlive.h new file mode 100644 index 0000000..b3ac7e2 --- /dev/null +++ b/taicpp/src/shell/runlive.h @@ -0,0 +1,53 @@ +#pragma once +// RunLive: run_live() / run_live_pty() of tai.py. Every command an agent runs for the user +// streams live into a StreamSink and is returned to the agent. With an ask function the +// command runs on a pty; when it stops on a prompt (sudo, ssh, y/n) ask(text, secret) gets +// the question and its answer is typed into the command. nullopt declines: the command gets +// end-of-file. Time spent waiting for the user does not count against the timeout. +#include "proc.h" +#include "sink.h" +#include +#include +#include +#include +namespace detz::shell { +constexpr int kShellTimeout = 120; // SHELL_TIMEOUT +constexpr long kRunOutputCap = 4000000; // RUN_OUTPUT_CAP +constexpr double kRunPromptIdle = 1.0; // RUN_PROMPT_IDLE +constexpr int kRunMaxAsks = 20; // RUN_MAX_ASKS +constexpr size_t kRunAnswerCap = 1000; // RUN_ANSWER_CAP + +// ask(text, secret): the user's answer, or nullopt to decline. +using AskFn = std::function(const std::string& text, bool secret)>; + +// What to run: an argv list, or a shell command line (shell=True). +struct Command { + std::vector argv; + std::optional shell; + static Command Argv(std::vector a) { Command c; c.argv = std::move(a); return c; } + static Command Shell(std::string line) { Command c; c.shell = std::move(line); return c; } +}; + +// (returncode, output, elapsed seconds, timed_out) +struct RunResult { int code{0}; std::string output; double elapsed{0}; bool timedOut{false}; }; + +struct RunLiveOptions { + double timeout{kShellTimeout}; + std::optional cwd; + const EnvMap* env{nullptr}; // null: inherit + StreamSink* stream{nullptr}; + AskFn ask; // empty: no pty, no questions +}; + +class RunLive { + public: + // run_live(): pty path when ask is set, pipe path otherwise. Throws OsError when the + // command cannot start (missing program, bad cwd). + static RunResult Run(const Command& command, const RunLiveOptions& options = RunLiveOptions()); + static RunResult RunPty(const Command& command, const RunLiveOptions& options); + // RUN_CTRL_RE.sub("", text) + static std::string StripControl(const std::string& text); + // RUN_PROMPT_RE.search(text) is not None + static bool LooksLikePrompt(const std::string& text); +}; +} // namespace detz::shell diff --git a/taicpp/src/shell/sandbox.cpp b/taicpp/src/shell/sandbox.cpp new file mode 100644 index 0000000..1f2ae65 --- /dev/null +++ b/taicpp/src/shell/sandbox.cpp @@ -0,0 +1,234 @@ +#include "sandbox.h" +#include "ansi.h" +#include "boxdata.h" +#include "text.h" +#include "utf8.h" +#include +#include +#include +#include +#include +#include +#include +#include +namespace detz::shell { +const char* const kCurlHint = "hint: use the web_fetch tool for HTTP(S) (methods, headers, bodies, status) instead of curl/wget via shell"; + +namespace { +std::mutex gMu; +bool gCapsDone = false; // _BOX_CAPS_STAMP["done"] +std::map gPythonCache; // _BOX_PYTHON_CACHE +std::map gOpencodeCache; // _BOX_OPENCODE_CACHE + +// universal_newlines=True decode of captured bytes +std::string Text(const std::string& bytes) { + IncrementalUtf8 dec; UniversalNewlines nl; + return nl.Feed(dec.Decode(bytes, true), true); +} +SubprocessOptions Plain(std::vector argv, double timeout) { + SubprocessOptions o; + o.popen.argv = std::move(argv); + o.popen.out = StdMode::Pipe; + o.popen.err = StdMode::Pipe; + o.timeout = timeout; + return o; +} +bool IsWord(const std::string& s, size_t i) { // \w at byte i (unicode-aware enough for the boundary test) + if (i >= s.size()) return false; + unsigned char c = s[i]; + return c == '_' || c >= 0x80 || std::isalnum(c); +} +// re.search(r"\bcurl\b|\bwget\b", command) +bool HasFetchTool(const std::string& s) { + for (const char* w : {"curl", "wget"}) { + size_t n = std::char_traits::length(w), pos = 0; + while ((pos = s.find(w, pos)) != std::string::npos) { + bool before = pos > 0 && IsWord(s, pos - 1), after = IsWord(s, pos + n); + if (!before && !after) return true; + ++pos; + } + } + return false; +} +} // namespace + +std::string CurlHint(const std::string& command) { + if (HasFetchTool(command)) { + std::string low; + for (char c : command) low.push_back((char)std::tolower((unsigned char)c)); + if (low.find("http") != std::string::npos) return std::string("\n") + kCurlHint; + } + return ""; +} + +std::vector Sandbox::RunArgs() { return {"--memory", kMemory, "--cpus", kCpus, "--pids-limit", kPidsLimit}; } +std::vector Sandbox::UpdateArgs() { return {"--memory", kMemory, "--cpus", kCpus}; } + +std::string Sandbox::ContainerEngine() { + if (!Subprocess::Which("podman").empty()) return "podman"; + if (!Subprocess::Which("docker").empty()) return "docker"; + return ""; +} + +CompletedProcess Sandbox::BoxExec(const std::string& engine, const std::vector& argv, const std::vector& extra, const std::optional& input, double timeout) { + std::vector full = {engine, "exec"}; + full.insert(full.end(), extra.begin(), extra.end()); + full.push_back("-i"); + full.push_back(kName); + full.insert(full.end(), argv.begin(), argv.end()); + SubprocessOptions o = Plain(full, timeout); + o.input = input; + return Subprocess::Run(o); +} + +std::string Sandbox::BoxState(const std::string& engine) { + try { + CompletedProcess done = Subprocess::Run(Plain({engine, "inspect", "-f", "{{.State.Running}}", kName}, 15)); + if (done.returncode != 0) return "missing"; + return Utf8::Strip(Text(done.out)) == "true" ? "running" : "stopped"; + } catch (const ProcessError&) { + return "missing"; + } +} + +bool Sandbox::UpdateBoxCaps(const std::string& engine) { + const char* caps = std::getenv("TAI_BOX_CAPS"); + { + std::lock_guard lock(gMu); + if (gCapsDone || (caps && std::string(caps) == "0")) return false; + gCapsDone = true; + } + std::vector argv = {engine, "update"}; + for (auto& a : UpdateArgs()) argv.push_back(a); + argv.push_back(kName); + CompletedProcess done; + try { + done = Subprocess::Run(Plain(argv, 60)); + } catch (const ProcessError&) { + return false; + } + if (done.returncode != 0) { + std::printf("box caps not applied (%s); run `%s rm -f %s` so next use recreates the box with limits\n", + Utf8::Head(Utf8::Strip(Text(done.err)), 200).c_str(), engine.c_str(), kName); + std::fflush(stdout); + return false; + } + return true; +} + +std::string Sandbox::EnsureBox() { + std::string engine = ContainerEngine(); + if (engine.empty()) throw BackendError("no container engine found, install podman"); + std::string state = BoxState(engine); + if (state == "running") { UpdateBoxCaps(engine); return engine; } + if (state == "stopped") { + CompletedProcess done = Subprocess::Run(Plain({engine, "start", kName}, 60)); + if (done.returncode != 0) throw BackendError("cannot start box: " + Utf8::Head(Utf8::Strip(Text(done.err)), 200)); + UpdateBoxCaps(engine); + return engine; + } + std::printf("building sandbox image %s (one-time, takes minutes)...\n", kImage); + std::fflush(stdout); + namespace fs = std::filesystem; + std::string tmpl = (fs::temp_directory_path() / "tai-box-XXXXXX").string(); + std::vector buf(tmpl.begin(), tmpl.end()); + buf.push_back('\0'); + if (!::mkdtemp(buf.data())) throw OsError(errno, tmpl); + std::string build = buf.data(); + struct Cleanup { std::string dir; ~Cleanup() { std::error_code ec; fs::remove_all(dir, ec); } } cleanup{build}; + const std::pair files[] = {{"Containerfile", BoxData::BoxContainerfile()}, {"stt.py", BoxData::BoxStt()}, {"tts.py", BoxData::BoxTts()}}; + for (auto& f : files) { + std::ofstream out(build + "/" + f.first, std::ios::binary); + out << f.second; + } + SubprocessOptions b; + b.popen.argv = {engine, "build", "-t", kImage, build}; + b.timeout = 1800; + if (Subprocess::Run(b).returncode != 0) throw BackendError("box image build failed"); + std::vector run = {engine, "run", "-d", "--name", kName, "--restart", "unless-stopped"}; + for (auto& a : RunArgs()) run.push_back(a); + run.push_back(kImage); + CompletedProcess done = Subprocess::Run(Plain(run, 120)); + if (done.returncode != 0) throw BackendError("cannot start box: " + Utf8::Head(Utf8::Strip(Text(done.err)), 200)); + return engine; +} + +std::string Sandbox::BoxPython(const std::string& engine) { + { + std::lock_guard lock(gMu); + auto it = gPythonCache.find(engine); + if (it != gPythonCache.end()) return it->second; + } + std::string value; + try { + CompletedProcess probe = BoxExec(engine, {"test", "-x", kPython}, {}, std::nullopt, 30); + value = probe.returncode == 0 ? kPython : "python3"; + } catch (const ProcessError&) { + value = "python3"; + } + std::lock_guard lock(gMu); + gPythonCache[engine] = value; + return value; +} + +std::string Sandbox::BoxTranscribe(const std::string& engine, const std::string& audio, double timeout) { + CompletedProcess done = BoxExec(engine, {BoxPython(engine), "/box/stt.py"}, {}, audio, timeout); + if (done.returncode != 0) throw BackendError("box stt failed: " + Utf8::Head(Utf8::Encode(Utf8::Decode(done.err)), 300)); + return Utf8::Strip(Utf8::Encode(Utf8::Decode(done.out))); +} + +std::string Sandbox::BoxSpeak(const std::string& engine, const std::string& text, const std::string& voice) { + std::vector argv = {BoxPython(engine), "/box/tts.py"}; + if (!voice.empty()) argv.push_back(voice); + CompletedProcess done = BoxExec(engine, argv, {}, text, 180); + if (done.returncode != 0 || done.out.empty()) throw BackendError("box tts failed: " + Utf8::Head(Utf8::Encode(Utf8::Decode(done.err)), 300)); + return done.out; +} + +bool Sandbox::BoxOpencodeBinary(const std::string& engine) { + { + std::lock_guard lock(gMu); + auto it = gOpencodeCache.find(engine); + if (it != gOpencodeCache.end()) return it->second; + } + auto remember = [&](bool v) { std::lock_guard lock(gMu); gOpencodeCache[engine] = v; return v; }; + try { + CompletedProcess probe = BoxExec(engine, {"which", "opencode"}, {}, std::nullopt, 10); + if (probe.returncode == 0) return remember(true); + } catch (const ProcessError&) { + return remember(false); + } + auto note = [](const std::string& msg) { + std::string line = Paint(msg, Ansi::kDim) + "\n"; + std::fwrite(line.data(), 1, line.size(), stderr); + std::fflush(stderr); + }; + note(std::string("opencode: no local CLI on this machine; installing it into the existing sandbox " + "box (container '") + kName + "') instead, as a last-resort backend. Nothing outside that " + "container is touched. Streaming the install below:"); + int code; + try { + SubprocessOptions o; + o.popen.argv = {engine, "exec", kName, "sh", "-c", BoxData::BoxOpencodeInstallScript()}; + o.timeout = 600; + code = Subprocess::Run(o).returncode; + } catch (const ProcessError& exc) { + note(std::string("opencode box install failed: ") + ShortError(exc.what(), "ProcessError")); + return remember(false); + } + bool ok = code == 0; + note(std::string("opencode box install ") + (ok ? "succeeded" : "failed (exit " + std::to_string(code) + ")")); + return remember(ok); +} + +CompletedProcess Sandbox::BoxOpencodeExec(const std::string& engine, const std::string& model, const std::string& prompt, double timeout, const std::string& agent) { + std::vector argv = {"opencode", "run", "--pure", "--agent", agent, "--format", "json", "-m", "opencode/" + model}; + try { + return BoxExec(engine, argv, {}, prompt, timeout); + } catch (const TimeoutExpired&) { + throw BackendError("opencode (box) run timed out after " + std::to_string((long)timeout) + "s"); + } catch (const ProcessError& exc) { + throw BackendError("opencode (box) run failed: " + ShortError(exc.what(), "ProcessError")); + } +} +} // namespace detz::shell diff --git a/taicpp/src/shell/sandbox.h b/taicpp/src/shell/sandbox.h new file mode 100644 index 0000000..951a7a0 --- /dev/null +++ b/taicpp/src/shell/sandbox.h @@ -0,0 +1,47 @@ +#pragma once +// Sandbox: the container sandbox ("box") of tai.py: engine probe, box lifecycle and the +// box_* helpers (exec, python, transcribe, speak, opencode), plus curl_hint. +#include "errors.h" +#include "proc.h" +#include +#include +#include +#include +namespace detz::shell { +class Sandbox { + public: + static constexpr const char* kImage = "tai-box:latest"; // BOX_IMAGE + static constexpr const char* kName = "tai-box"; // BOX_NAME + static constexpr const char* kMemory = "2g"; // BOX_MEMORY + static constexpr const char* kCpus = "2.0"; // BOX_CPUS + static constexpr const char* kPidsLimit = "256"; // BOX_PIDS_LIMIT + static constexpr const char* kPython = "/box/venv/bin/python"; // BOX_PYTHON + static constexpr const char* kOpencodeNodeVersion = "20.19.2"; // BOX_OPENCODE_NODE_VERSION + + // BOX_RUN_ARGS / BOX_UPDATE_ARGS + static std::vector RunArgs(); + static std::vector UpdateArgs(); + + // Sandbox.container_engine(): "podman", "docker" or "". + static std::string ContainerEngine(); + // Sandbox.box_exec(engine, argv, extra, input_data, timeout): captured bytes. + static CompletedProcess BoxExec(const std::string& engine, const std::vector& argv, const std::vector& extra = {}, const std::optional& input = std::nullopt, double timeout = 180); + + static std::string BoxState(const std::string& engine); // missing | running | stopped + static bool UpdateBoxCaps(const std::string& engine); + // ensure_box(): the engine name; throws BackendError. + static std::string EnsureBox(); + static std::string BoxPython(const std::string& engine); + static std::string BoxTranscribe(const std::string& engine, const std::string& audio, double timeout = 300); + static std::string BoxSpeak(const std::string& engine, const std::string& text, const std::string& voice = ""); + static bool BoxOpencodeBinary(const std::string& engine); + // The exec half of box_opencode_run(); the caller parses the events (opencode_parse_events). + // Throws BackendError exactly where box_opencode_run does. + static CompletedProcess BoxOpencodeExec(const std::string& engine, const std::string& model, const std::string& prompt, double timeout, const std::string& agent); +}; + +// curl_hint(command): "\n" + CURL_HINT when a shell command fetches http with curl or wget. +std::string CurlHint(const std::string& command); +inline std::string curl_hint(const std::string& command) { return CurlHint(command); } +extern const char* const kCurlHint; // CURL_HINT +} // namespace detz::shell diff --git a/taicpp/src/shell/session.cpp b/taicpp/src/shell/session.cpp new file mode 100644 index 0000000..78c7b47 --- /dev/null +++ b/taicpp/src/shell/session.cpp @@ -0,0 +1,369 @@ +#include "session.h" +#include "jsonvalue.h" +#include "pypath.h" +#include "text.h" +#include "utf8.h" +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +namespace detz::shell { +namespace { +std::string Lstrip(const std::string& s, char c) { size_t i = 0; while (i < s.size() && s[i] == c) ++i; return s.substr(i); } +std::string Fmt(const char* f, const std::tm& tm) { char b[64]; std::strftime(b, sizeof b, f, &tm); return b; } +std::string FirstField(const std::string& text) { size_t p = text.find('\0'); return p == std::string::npos ? text : text.substr(0, p); } +bool ParseLong(const std::string& s, long* out) { + std::string t = Utf8::Strip(s); + if (t.empty()) return false; + size_t i = 0; bool neg = false; + if (t[i] == '+' || t[i] == '-') { neg = t[i] == '-'; ++i; } + if (i >= t.size()) return false; + long v = 0; + for (; i < t.size(); ++i) { + if (t[i] < '0' || t[i] > '9') return false; + v = v * 10 + (t[i] - '0'); + } + *out = neg ? -v : v; + return true; +} +SqlValue Opt(const std::optional& v) { return v ? SqlValue::Text(*v) : SqlValue::Null(); } +} // namespace + +ShellSession::ShellSession(std::string sid_, std::string shell_, long pid_, std::string started_, std::optional parent_, std::vector chain_) + : sid(std::move(sid_)), shell(std::move(shell_)), pid(pid_), started(std::move(started_)), parent(std::move(parent_)), chain(std::move(chain_)) {} + +bool ShellSession::IsShell(const std::string& n) { + static const std::set shells = {"bash", "zsh", "fish", "sh", "dash", "ksh", "tcsh", "nu"}; + return shells.count(n) > 0; +} +std::string ShellSession::Sanitize(const std::string& value) { + std::string o; + for (char c : value) if (std::isalnum((unsigned char)c) || c == '_' || c == '.' || c == '-') o.push_back(c); + return o.substr(0, kSidMax); +} +std::string ShellSession::ReadText(const std::string& path) { + std::ifstream in(path, std::ios::binary); + if (!in) return ""; + std::string raw(4096 * 4 + 8, '\0'); + in.read(raw.data(), (std::streamsize)raw.size()); + raw.resize((size_t)in.gcount()); + std::u32string u = Utf8::Decode(raw), t; + for (size_t i = 0; i < u.size(); ++i) { + if (u[i] == '\r') { t.push_back('\n'); if (i + 1 < u.size() && u[i + 1] == '\n') ++i; } + else t.push_back(u[i]); + } + if (t.size() > 4096) t.resize(4096); + return Utf8::Encode(t); +} +ShellSession::Stat ShellSession::ParseStat(const std::string& text) { + Stat st; + size_t close = text.rfind(')'); + if (close == std::string::npos) return st; + std::string head = text.substr(0, close), tail = text.substr(close + 1); + size_t open = head.find('('); + if (head.empty() || open == std::string::npos) return st; + st.comm = head.substr(open + 1); + std::vector fields = Utf8::SplitWhitespace(tail); + if (fields.size() < 20) return st; + long ppid, ticks; + if (!ParseLong(fields[1], &ppid) || !ParseLong(fields[19], &ticks)) return st; + st.ok = true; st.ppid = ppid; st.ticks = ticks; + return st; +} +std::optional ShellSession::BootTime(const std::string& procRoot) { + std::ifstream in(PyPath::Join(procRoot, "stat")); + if (!in) return std::nullopt; + std::string line; + while (std::getline(in, line)) { + if (Utf8::StartsWith(line, "btime ")) { + auto fields = Utf8::SplitWhitespace(line); + if (fields.size() < 2) return std::nullopt; + char* end = nullptr; + double v = std::strtod(fields[1].c_str(), &end); + if (end == fields[1].c_str() || *end) return std::nullopt; + return v; + } + } + return std::nullopt; +} +long ShellSession::ClkTick() { + long v = ::sysconf(_SC_CLK_TCK); + return v > 0 ? v : 100; +} +std::string ShellSession::BootTag(const std::string& procRoot) { + std::string raw = Utf8::Lower(ReadText(PyPath::Join(procRoot, "sys/kernel/random/boot_id"))), digest; + for (char c : raw) if ((c >= '0' && c <= '9') || (c >= 'a' && c <= 'f')) digest.push_back(c); + digest = digest.substr(0, 8); + if (!digest.empty()) return digest; + auto btime = BootTime(procRoot); + if (btime && *btime != 0) return Md5Hex(JsonValue::Float(*btime).Repr()).substr(0, 8); + return "deadbeef"; +} +std::string ShellSession::ShellName(const std::string& argv0, const std::string& comm) { + for (const std::string& cand : {Utf8::Lower(Lstrip(PyPath::Basename(FirstField(argv0)), '-')), Utf8::Lower(Lstrip(comm, '-'))}) + if (IsShell(cand)) return cand; + return ""; +} +std::vector ShellSession::ReadChain(long pid, const std::string& procRoot, std::optional btime, long tick, size_t cap) { + std::vector chain; + std::set seen; + long current = pid; + while (current && !seen.count(current) && chain.size() < cap) { + seen.insert(current); + Stat st = ParseStat(ReadText(PyPath::Join(PyPath::Join(procRoot, std::to_string(current)), "stat"))); + if (!st.ok) break; + std::string cmdline = ReadText(PyPath::Join(PyPath::Join(procRoot, std::to_string(current)), "cmdline")); + ChainEntry e; + e.pid = current; e.ppid = st.ppid; + e.comm = Utf8::Head(st.comm, 64); + e.argv0 = Utf8::Head(PyPath::Basename(FirstField(cmdline)), 64); + e.name = ShellName(e.argv0, st.comm); + e.start = (btime && *btime != 0) ? *btime + (double)st.ticks / (double)tick : 0.0; + e.startIso = e.start != 0 ? IsoUtc(e.start) : ""; + chain.push_back(e); + if (st.ppid <= 0 || st.ppid == current) break; + current = st.ppid; + } + return chain; +} +std::string ShellSession::ShellId(double startTs, long pid, const std::string& boot) { + std::time_t t = (std::time_t)std::floor(startTs); + std::tm tm{}; + localtime_r(&t, &tm); + return Fmt("%Y%m%d-%H%M%S", tm) + "-p" + std::to_string(pid) + "-" + boot; +} +std::string ShellSession::EphemeralId(double nowTs, long pid) { + std::time_t t = (std::time_t)std::floor(nowTs); + std::tm tm{}; + localtime_r(&t, &tm); + std::random_device rd; + char hex[8]; + std::snprintf(hex, sizeof hex, "%04x", (unsigned)(rd() & 0xffff)); + return Fmt("%Y%m%d-%H%M%S", tm) + "-e" + std::to_string(pid) + "-" + hex; +} +std::string ShellSession::IsoUtc(double ts) { + double sec = std::floor(ts), frac = ts - sec; + double us = std::nearbyint(frac * 1e6); + if (us >= 1e6) sec += 1; + std::time_t t = (std::time_t)sec; + std::tm tm{}; + gmtime_r(&t, &tm); + return Fmt("%Y-%m-%dT%H:%M:%S", tm) + "+00:00"; +} + +ShellSession ShellSession::Current(const EnvMap* env, long pid, const std::string& procRoot, double now, long clkTck) { + std::string rawEnv; + if (env) rawEnv = env->Get(kEnv); else if (const char* v = std::getenv(kEnv)) rawEnv = v; + std::string raw = Sanitize(rawEnv); + if (pid < 0) pid = (long)::getpid(); + if (now < 0) now = Subprocess::Now(); + auto btime = BootTime(procRoot); + long tick = clkTck ? clkTck : ClkTick(); + std::string boot = BootTag(procRoot); + std::vector chain = ReadChain(pid, procRoot, btime, tick, kChainCap); + for (auto& e : chain) if (!e.name.empty() && e.start != 0) e.sid = ShellId(e.start, e.pid, boot); + std::vector shells; + for (auto& e : chain) if (!e.sid.empty()) shells.push_back(&e); + ChainEntry* inner = shells.empty() ? nullptr : shells[0]; + ChainEntry* outer = shells.size() > 1 ? shells[1] : nullptr; + std::optional outerSid = outer ? std::optional(outer->sid) : std::nullopt; + if (!raw.empty()) { + // PID_RE = -p(\d+)- + long embedded = -1; bool found = false; + for (size_t i = 0; i + 2 < raw.size() && !found; ++i) { + if (raw[i] == '-' && raw[i + 1] == 'p') { + size_t j = i + 2; + while (j < raw.size() && raw[j] >= '0' && raw[j] <= '9') ++j; + if (j > i + 2 && j < raw.size() && raw[j] == '-') { + found = true; + embedded = j - (i + 2) > 17 ? -2 : std::strtol(raw.substr(i + 2, j - i - 2).c_str(), nullptr, 10); + } + } + } + if (!found) return ShellSession(raw, inner ? inner->name : "", inner ? inner->pid : pid, inner ? inner->startIso : "", std::nullopt, chain); + if (inner && embedded == inner->pid) return ShellSession(raw, inner->name, inner->pid, inner->startIso, outerSid, chain); + std::string fresh = inner ? inner->sid : EphemeralId(now, pid); + std::string started = inner ? inner->startIso : IsoUtc(now); + return ShellSession(fresh, inner ? inner->name : "", inner ? inner->pid : pid, started, raw, chain); + } + if (inner) return ShellSession(inner->sid, inner->name, inner->pid, inner->startIso, outerSid, chain); + return ShellSession(EphemeralId(now, pid), "", pid, IsoUtc(now), std::nullopt, chain); +} + +void ShellSession::Export() const { ::setenv(kEnv, sid.c_str(), 1); } + +void ShellSession::Register(SessionDb& db) const { + try { + std::string stamp = NowIso(); + std::vector shells; + for (auto& e : chain) if (!e.sid.empty()) shells.push_back(&e); + for (size_t pos = 0; pos < shells.size(); ++pos) { + std::optional nextSid = pos + 1 < shells.size() ? std::optional(shells[pos + 1]->sid) : std::nullopt; + db.Execute("INSERT OR IGNORE INTO shell_sessions (sid, parent, shell, pid, started) VALUES (?, ?, ?, ?, ?)", + {SqlValue::Text(shells[pos]->sid), Opt(nextSid), SqlValue::Text(shells[pos]->name), SqlValue::Int(shells[pos]->pid), SqlValue::Text(shells[pos]->startIso)}); + } + JsonValue arr = JsonValue::Array(); + for (auto& e : chain) { + JsonValue o = JsonValue::Object(); + o.Set("pid", JsonValue::Int(e.pid)); o.Set("ppid", JsonValue::Int(e.ppid)); + o.Set("comm", JsonValue::String(e.comm)); o.Set("argv0", JsonValue::String(e.argv0)); + o.Set("name", JsonValue::String(e.name)); o.Set("start", JsonValue::Float(e.start)); + o.Set("start_iso", JsonValue::String(e.startIso)); o.Set("sid", JsonValue::String(e.sid)); + arr.Push(o); + } + std::string chainJson = arr.Dump(false), cwd = PyPath::Realpath(PyPath::Getcwd()); + db.Execute("INSERT OR IGNORE INTO shell_sessions (sid, parent, shell, pid, started, last_seen, cwd, chain) VALUES (?, ?, ?, ?, ?, ?, ?, ?)", + {SqlValue::Text(sid), Opt(parent), SqlValue::Text(shell), SqlValue::Int(pid), SqlValue::Text(started), SqlValue::Text(stamp), SqlValue::Text(cwd), SqlValue::Text(chainJson)}); + db.Execute("UPDATE shell_sessions SET last_seen = ?, cwd = ?, parent = COALESCE(parent, ?), chain = ? WHERE sid = ?", + {SqlValue::Text(stamp), SqlValue::Text(cwd), Opt(parent), SqlValue::Text(chainJson), SqlValue::Text(sid)}); + db.Execute("DELETE FROM shell_sessions WHERE sid NOT IN (SELECT sid FROM shell_sessions ORDER BY COALESCE(last_seen, started, '') DESC LIMIT ?)", + {SqlValue::Int((long long)kRowsCap)}); + db.Commit(); + } catch (const SessionDbError&) { + } +} + +std::vector ShellSession::TreeLines(SessionDb& db, const std::string& sid) { + std::vector> rows; + try { + rows = db.Query("SELECT sid, parent, shell, pid FROM shell_sessions", {}); + } catch (const SessionDbError&) { + return {}; + } + std::map> bySid; + std::map> kids; + for (auto& item : rows) { + bySid[item[0].Str()] = item; + if (item[1].Truthy()) kids[item[1].Str()].push_back(item[0].Str()); + } + for (auto& kv : kids) std::sort(kv.second.begin(), kv.second.end()); + if (!bySid.count(sid)) return {}; + auto label = [&](const std::string& key) { + auto it = bySid.find(key); + if (it == bySid.end()) return key + " (pruned)"; + const auto& item = it->second; + return key + " " + (item[2].Truthy() ? item[2].Str() : "?") + (item[3].Truthy() ? " pid " + item[3].Str() : ""); + }; + std::vector upwards; + std::set seen = {sid}; + std::string node = sid; + while (upwards.size() < kTreeDepth) { + auto it = bySid.find(node); + if (it == bySid.end() || !it->second[1].Truthy()) break; + std::string parent = it->second[1].Str(); + if (seen.count(parent)) break; + seen.insert(parent); + upwards.push_back(parent); + node = parent; + } + std::vector lines; + size_t depth = 0; + for (auto it = upwards.rbegin(); it != upwards.rend(); ++it, ++depth) lines.push_back(std::string(depth * 2, ' ') + label(*it)); + lines.push_back(std::string(upwards.size() * 2, ' ') + "* " + label(sid)); + std::function(const std::string&, size_t, std::set)> below = + [&](const std::string& key, size_t d, std::set seenHere) { + std::vector out; + if (d > kTreeDepth || lines.size() + out.size() >= kTreeLines) return out; + auto kit = kids.find(key); + if (kit == kids.end()) return out; + size_t count = 0; + for (auto& child : kit->second) { + if (count++ >= kTreeKids) break; + if (seenHere.count(child) || lines.size() + out.size() >= kTreeLines) continue; + out.push_back(std::string(d * 2, ' ') + label(child)); + std::set next = seenHere; + next.insert(child); + auto sub = below(child, d + 1, next); + out.insert(out.end(), sub.begin(), sub.end()); + } + return out; + }; + auto extra = below(sid, upwards.size() + 1, seen); + lines.insert(lines.end(), extra.begin(), extra.end()); + if (lines.size() > kTreeLines) lines.resize(kTreeLines); + return lines; +} + +std::string ShellSession::Report(SessionDb& db, const std::string& sid) { + std::vector lines; + std::vector> found; + try { + if (!sid.empty()) found = db.Query("SELECT sid, parent, shell, pid, started, last_seen, cwd FROM shell_sessions WHERE sid = ?", {SqlValue::Text(sid)}); + } catch (const SessionDbError&) { + return "ctx unavailable (vault error)"; + } + auto orq = [](const SqlValue& v, const std::string& d) { return v.Truthy() ? v.Str() : d; }; + if (!found.empty()) { + const auto& row = found[0]; + lines.push_back("ctx: " + row[0].Str()); + lines.push_back("shell: " + orq(row[2], "?") + " (pid " + orq(row[3], "?") + ", started " + orq(row[4], "?") + ")"); + lines.push_back("parent: " + orq(row[1], "(none)")); + lines.push_back("cwd: " + orq(row[6], "?")); + } else if (!sid.empty()) { + lines.push_back("ctx: " + sid + " (unregistered)"); + } else { + lines.push_back("ctx: (legacy, no session)"); + } + auto tree = TreeLines(db, sid); + if (!tree.empty()) { + lines.push_back("tree:"); + lines.insert(lines.end(), tree.begin(), tree.end()); + } + std::vector> recent; + try { + recent = db.Query("SELECT sid, shell, last_seen FROM shell_sessions ORDER BY COALESCE(last_seen, started, '') DESC LIMIT 8", {}); + } catch (const SessionDbError&) { + } + if (!recent.empty()) { + lines.push_back("recent:"); + for (auto& item : recent) + lines.push_back(std::string(item[0].Str() == sid ? "*" : " ") + " " + item[0].Str() + " " + orq(item[1], "?") + " " + orq(item[2], "")); + } + return Utf8::Join(lines, "\n"); +} + +std::string ShellSession::Md5Hex(const std::string& data) { + static const uint32_t K[64] = { + 0xd76aa478, 0xe8c7b756, 0x242070db, 0xc1bdceee, 0xf57c0faf, 0x4787c62a, 0xa8304613, 0xfd469501, 0x698098d8, 0x8b44f7af, 0xffff5bb1, 0x895cd7be, 0x6b901122, 0xfd987193, 0xa679438e, 0x49b40821, + 0xf61e2562, 0xc040b340, 0x265e5a51, 0xe9b6c7aa, 0xd62f105d, 0x02441453, 0xd8a1e681, 0xe7d3fbc8, 0x21e1cde6, 0xc33707d6, 0xf4d50d87, 0x455a14ed, 0xa9e3e905, 0xfcefa3f8, 0x676f02d9, 0x8d2a4c8a, + 0xfffa3942, 0x8771f681, 0x6d9d6122, 0xfde5380c, 0xa4beea44, 0x4bdecfa9, 0xf6bb4b60, 0xbebfbc70, 0x289b7ec6, 0xeaa127fa, 0xd4ef3085, 0x04881d05, 0xd9d4d039, 0xe6db99e5, 0x1fa27cf8, 0xc4ac5665, + 0xf4292244, 0x432aff97, 0xab9423a7, 0xfc93a039, 0x655b59c3, 0x8f0ccc92, 0xffeff47d, 0x85845dd1, 0x6fa87e4f, 0xfe2ce6e0, 0xa3014314, 0x4e0811a1, 0xf7537e82, 0xbd3af235, 0x2ad7d2bb, 0xeb86d391}; + static const int S[64] = {7, 12, 17, 22, 7, 12, 17, 22, 7, 12, 17, 22, 7, 12, 17, 22, 5, 9, 14, 20, 5, 9, 14, 20, 5, 9, 14, 20, 5, 9, 14, 20, + 4, 11, 16, 23, 4, 11, 16, 23, 4, 11, 16, 23, 4, 11, 16, 23, 6, 10, 15, 21, 6, 10, 15, 21, 6, 10, 15, 21, 6, 10, 15, 21}; + uint32_t a0 = 0x67452301, b0 = 0xefcdab89, c0 = 0x98badcfe, d0 = 0x10325476; + std::string msg = data; + uint64_t bits = (uint64_t)data.size() * 8; + msg.push_back((char)0x80); + while (msg.size() % 64 != 56) msg.push_back('\0'); + for (int i = 0; i < 8; ++i) msg.push_back((char)((bits >> (8 * i)) & 0xff)); + for (size_t off = 0; off < msg.size(); off += 64) { + uint32_t M[16]; + for (int i = 0; i < 16; ++i) + M[i] = (uint32_t)(unsigned char)msg[off + i * 4] | ((uint32_t)(unsigned char)msg[off + i * 4 + 1] << 8) | ((uint32_t)(unsigned char)msg[off + i * 4 + 2] << 16) | ((uint32_t)(unsigned char)msg[off + i * 4 + 3] << 24); + uint32_t A = a0, B = b0, C = c0, D = d0; + for (int i = 0; i < 64; ++i) { + uint32_t F; int g; + if (i < 16) { F = (B & C) | (~B & D); g = i; } + else if (i < 32) { F = (D & B) | (~D & C); g = (5 * i + 1) % 16; } + else if (i < 48) { F = B ^ C ^ D; g = (3 * i + 5) % 16; } + else { F = C ^ (B | ~D); g = (7 * i) % 16; } + F = F + A + K[i] + M[g]; + A = D; D = C; C = B; + B = B + ((F << S[i]) | (F >> (32 - S[i]))); + } + a0 += A; b0 += B; c0 += C; d0 += D; + } + char out[33]; + uint32_t parts[4] = {a0, b0, c0, d0}; + for (int p = 0; p < 4; ++p) + for (int b = 0; b < 4; ++b) std::snprintf(out + p * 8 + b * 2, 3, "%02x", (parts[p] >> (8 * b)) & 0xff); + return std::string(out, 32); +} +} // namespace detz::shell diff --git a/taicpp/src/shell/session.h b/taicpp/src/shell/session.h new file mode 100644 index 0000000..7719490 --- /dev/null +++ b/taicpp/src/shell/session.h @@ -0,0 +1,92 @@ +#pragma once +// ShellSession: one interactive shell is one conversation context (tai.py class ShellSession). +// Every bash/zsh/fish instance gets a stable logical id derived from /proc: +// YYYYMMDD-HHMMSS-p-b (local start time, shell pid, 8 hex of the kernel boot id) +// with an ephemeral YYYYMMDD-HHMMSS-e- fallback. An explicit TAI_SESSION wins when +// it is a plain word or when its embedded -p- matches the nearest shell ancestor; an id +// that embeds another pid means a nested shell, so a fresh child id is derived and the +// inherited value becomes its parent link. register() upserts the shell_sessions rows; +// report()/tree_lines() render /ctx. +#include "proc.h" +#include +#include +#include +#include +#include +namespace detz::shell { + +// One column value of a shell_sessions row (sqlite NULL / INTEGER / TEXT). +struct SqlValue { + enum class Kind { Null, Int, Text } kind{Kind::Null}; + long long i{0}; + std::string s; + static SqlValue Null() { return SqlValue(); } + static SqlValue Int(long long v) { SqlValue x; x.kind = Kind::Int; x.i = v; return x; } + static SqlValue Text(std::string v) { SqlValue x; x.kind = Kind::Text; x.s = std::move(v); return x; } + bool Truthy() const { return kind == Kind::Int ? i != 0 : kind == Kind::Text ? !s.empty() : false; } + std::string Str() const { return kind == Kind::Int ? std::to_string(i) : kind == Kind::Text ? s : "None"; } +}; +// The slice of the vault ShellSession needs (store.db.execute/commit/fetch). Methods throw +// SessionDbError for sqlite3.Error / OSError / ValueError. +class SessionDbError : public std::runtime_error { using std::runtime_error::runtime_error; }; +class SessionDb { + public: + virtual ~SessionDb() = default; + virtual void Execute(const std::string& sql, const std::vector& params) = 0; + virtual std::vector> Query(const std::string& sql, const std::vector& params) = 0; + virtual void Commit() = 0; +}; + +// One process of the ancestry chain (the dicts of read_chain). +struct ChainEntry { + long pid{0}, ppid{0}; + std::string comm, argv0, name; + double start{0}; + std::string startIso, sid; +}; + +class ShellSession { + public: + static constexpr const char* kEnv = "TAI_SESSION"; // ENV + static constexpr size_t kSidMax = 64; // SID_MAX + static constexpr size_t kChainCap = 64; // CHAIN_CAP + static constexpr size_t kFilesCap = 20; // FILES_CAP + static constexpr size_t kRowsCap = 1000; // ROWS_CAP + static constexpr size_t kTreeDepth = 8; // TREE_DEPTH + static constexpr size_t kTreeKids = 10; // TREE_KIDS + static constexpr size_t kTreeLines = 40; // TREE_LINES + + ShellSession(std::string sid, std::string shell = "", long pid = 0, std::string started = "", std::optional parent = std::nullopt, std::vector chain = {}); + + std::string sid, shell; + long pid{0}; + std::string started; + std::optional parent; + std::vector chain; + + static bool IsShell(const std::string& name); // name in SHELLS + static std::string Sanitize(const std::string& value); + static std::string Coerce(const std::string& value) { return Sanitize(value); } + static std::string ReadText(const std::string& path); // first 4096 chars, "" on error + struct Stat { bool ok{false}; std::string comm; long ppid{0}, ticks{0}; }; + static Stat ParseStat(const std::string& text); + static std::optional BootTime(const std::string& procRoot); + static long ClkTick(); + static std::string BootTag(const std::string& procRoot); + static std::string ShellName(const std::string& argv0, const std::string& comm); + static std::vector ReadChain(long pid, const std::string& procRoot, std::optional btime, long tick, size_t cap); + static std::string ShellId(double startTs, long pid, const std::string& boot); + static std::string EphemeralId(double nowTs, long pid); + // ShellSession.current(env, pid, proc_root, now, clk_tck); env null = process environment, + // pid < 0 = this process, now < 0 = time.time(), clkTck 0 = sysconf. + static ShellSession Current(const EnvMap* env = nullptr, long pid = -1, const std::string& procRoot = "/proc", double now = -1, long clkTck = 0); + + void Export() const; // os.environ[TAI_SESSION] = sid + void Register(SessionDb& db) const; + static std::vector TreeLines(SessionDb& db, const std::string& sid); + static std::string Report(SessionDb& db, const std::string& sid); + // md5(text).hexdigest(), exposed for the parity test + static std::string Md5Hex(const std::string& data); + static std::string IsoUtc(double ts); // datetime.fromtimestamp(ts, utc).isoformat(timespec="seconds") +}; +} // namespace detz::shell diff --git a/taicpp/src/shell/shell.mk b/taicpp/src/shell/shell.mk new file mode 100644 index 0000000..c8c124f --- /dev/null +++ b/taicpp/src/shell/shell.mk @@ -0,0 +1 @@ +SHELL_SRCS = src/shell/audio.cpp src/shell/boxdata.cpp src/shell/edge.cpp src/shell/glow.cpp src/shell/lanes.cpp src/shell/livestream.cpp src/shell/proc.cpp src/shell/pty.cpp src/shell/runlive.cpp src/shell/sandbox.cpp src/shell/session.cpp src/shell/shellio.cpp src/shell/spinner.cpp src/shell/targets.cpp diff --git a/taicpp/src/shell/shellio.cpp b/taicpp/src/shell/shellio.cpp new file mode 100644 index 0000000..7927e9f --- /dev/null +++ b/taicpp/src/shell/shellio.cpp @@ -0,0 +1,73 @@ +#include "shellio.h" +#include "ansi.h" +#include "utf8.h" +#include +#include +#include +#include +#include +namespace detz::shell { +namespace { +// getpass.getpass(prompt): prompt on the tty, no echo, newline after +std::optional GetPass(const std::string& prompt) { + int fd = ::open("/dev/tty", O_RDWR | O_NOCTTY); + bool own = fd >= 0; + if (fd < 0) fd = 0; + struct termios old, quiet; + bool changed = false; + if (::tcgetattr(fd, &old) == 0) { + quiet = old; + quiet.c_lflag &= ~(tcflag_t)ECHO; + changed = ::tcsetattr(fd, TCSAFLUSH, &quiet) == 0; + } + int out = own ? fd : 2; + (void)!::write(out, prompt.data(), prompt.size()); + std::string line; + bool got = false, eof = false; + char c; + while (true) { + ssize_t n = ::read(fd, &c, 1); + if (n <= 0) { eof = true; break; } + got = true; + if (c == '\n') break; + line.push_back(c); + } + if (changed) ::tcsetattr(fd, TCSAFLUSH, &old); + (void)!::write(out, "\n", 1); + if (own) ::close(fd); + if (eof && !got) return std::nullopt; // EOFError + return line; +} +std::optional Input(const std::string& prompt) { + std::fputs(prompt.c_str(), stdout); + std::fflush(stdout); + std::string line; + if (!std::getline(std::cin, line)) return std::nullopt; // EOFError + return line; +} +} // namespace + +std::unique_ptr ShellIo::Stream(ShellApp& app, const std::string& command, bool live) { + if (app.HasStreamFactory()) return app.StreamFactory(app.Redact(command)); + if (live && ::isatty(1)) return std::make_unique(); + return nullptr; +} + +AskFn ShellIo::Ask(ShellApp& app, const std::string& command) { + if (app.Auto() || !app.Persist() || app.Depth() > 0) return nullptr; + ShellApp* a = &app; + auto answered = [a](std::optional value) { + if (value) a->MarkUserSteppedIn(); + return value; + }; + if (app.HasPrompter()) { + std::string head = Utf8::Head(command, 60); + return [a, head, answered](const std::string& text, bool) { return answered(a->Prompt("ask", head + " asks:\n" + text, {})); }; + } + if (!(::isatty(0) && ::isatty(1))) return nullptr; + return [answered](const std::string& text, bool secret) -> std::optional { + std::string prompt = Paint("\xc2\xbb " + text + " ", Ansi::kYellow); + return answered(secret ? GetPass(prompt) : Input(prompt)); + }; +} +} // namespace detz::shell diff --git a/taicpp/src/shell/shellio.h b/taicpp/src/shell/shellio.h new file mode 100644 index 0000000..ec79315 --- /dev/null +++ b/taicpp/src/shell/shellio.h @@ -0,0 +1,41 @@ +#pragma once +// ShellIo: Tools.shell_stream() and Tools.shell_ask() of tai.py. Where a running command's +// output goes live, and who answers when the command stops on a prompt (sudo, ssh, y/n). +// Every command an agent runs for the user must go through RunLive with these two (Command +// Output Rule in the repository AGENTS.md); never a bare subprocess. +#include "livestream.h" +#include "runlive.h" +#include +#include +#include +#include +namespace detz::shell { +// The attributes of `self.app` that shell_stream / shell_ask read. +class ShellApp { + public: + virtual ~ShellApp() = default; + virtual bool Auto() const { return false; } // app.auto + virtual bool Persist() const { return true; } // app.persist + virtual int Depth() const { return 0; } // app.depth (background workers > 0) + // app.prompter: prompter("ask", text, []) -> answer or nullopt; HasPrompter() false when unset + virtual bool HasPrompter() const { return false; } + virtual std::optional Prompt(const std::string& kind, const std::string& text, const std::vector& options) { (void)kind; (void)text; (void)options; return std::nullopt; } + virtual void MarkUserSteppedIn() {} // app.user_stepped_in = True + // app.stream_factory(redacted_command): the chat scene's own window; HasStreamFactory() false when unset. + // A factory may return null (no live window). + virtual bool HasStreamFactory() const { return false; } + virtual std::unique_ptr StreamFactory(const std::string& redactedCommand) { (void)redactedCommand; return nullptr; } + // store.redact(command, profile) when the app has a store, else identity + virtual std::string Redact(const std::string& command) { return command; } +}; + +class ShellIo { + public: + // shell_stream(command): the chat scene's window when the app provides one, else the REPL's + // scrolling LiveStream when `live` and stdout is a tty, else null. + static std::unique_ptr Stream(ShellApp& app, const std::string& command, bool live); + // shell_ask(command): ask(text, secret) for a command that stops on a prompt, or an empty + // function when nobody can answer (autonomous mode, background workers, non-interactive stdin). + static AskFn Ask(ShellApp& app, const std::string& command); +}; +} // namespace detz::shell diff --git a/taicpp/src/shell/sink.h b/taicpp/src/shell/sink.h new file mode 100644 index 0000000..8465f60 --- /dev/null +++ b/taicpp/src/shell/sink.h @@ -0,0 +1,41 @@ +#pragma once +// Output sinks: the "file" a LiveStream / LaneBoard writes to (sys.stdout by default). +// StreamSink is the feed/pause/close protocol run_live streams a command's lines into +// (LiveStream here, TuiLiveStream in the chat TUI). +#include +#include +#include +namespace detz::shell { +class OutputSink { + public: + virtual ~OutputSink() = default; + virtual void Write(const std::string& text) = 0; + virtual void Flush() = 0; +}; +class FileSink : public OutputSink { + public: + explicit FileSink(std::FILE* file) : file_(file) {} + void Write(const std::string& text) override { std::fwrite(text.data(), 1, text.size(), file_); } + void Flush() override { std::fflush(file_); } + private: + std::FILE* file_; +}; +// io.StringIO analog (tests, capture) +class StringSink : public OutputSink { + public: + void Write(const std::string& text) override { std::lock_guard lock(mu_); text_ += text; } + void Flush() override {} + std::string str() const { std::lock_guard lock(mu_); return text_; } + private: + mutable std::mutex mu_; + std::string text_; +}; +// A command's live output window: feed(line), pause() before a question, close(status). +class StreamSink { + public: + virtual ~StreamSink() = default; + virtual void Feed(const std::string& line) = 0; + virtual void Pause() {} + virtual void Close(const std::string& status) = 0; +}; +} // namespace detz::shell diff --git a/taicpp/src/shell/spinner.cpp b/taicpp/src/shell/spinner.cpp new file mode 100644 index 0000000..8e86a37 --- /dev/null +++ b/taicpp/src/shell/spinner.cpp @@ -0,0 +1,59 @@ +#include "spinner.h" +#include "ansi.h" +#include "glow.h" +#include "proc.h" +#include "utf8.h" +#include +#include +#include +#include +namespace detz::shell { +const char* const* Spinner::Frames() { + static const char* const f[10] = {"\xe2\xa0\x8b", "\xe2\xa0\x99", "\xe2\xa0\xb9", "\xe2\xa0\xb8", "\xe2\xa0\xbc", "\xe2\xa0\xb4", "\xe2\xa0\xa6", "\xe2\xa0\xa7", "\xe2\xa0\x87", "\xe2\xa0\x8f"}; + return f; +} +Spinner::Spinner(std::string label) : label_(std::move(label)) {} +Spinner::~Spinner() { if (worker_.joinable()) { stop_ = true; worker_.join(); } } + +void Spinner::Start() { + if (!::isatty(2)) return; + started_ = true; + worker_ = std::thread([this] { Spin(); finished_ = true; }); +} + +void Spinner::Spin() { + long pos = 0; + double started = Subprocess::Now(); + bool glow = Glow::StderrEnabled(); + size_t widest = Utf8::Len(label_) + 2; + while (!stop_) { + const char* frame = Frames()[pos % kFrameCount]; + if (glow) { + char clockBuf[48]; + std::snprintf(clockBuf, sizeof clockBuf, "\xc2\xb7 %.1fs", Subprocess::Now() - started); + std::string clock = clockBuf; + std::string line = std::string("\r") + Ansi::kBold + Ansi::kCyan + frame + " " + Glow::Shimmer(label_, pos) + " " + Ansi::kDim + clock + Ansi::kReset; + std::fwrite(line.data(), 1, line.size(), stderr); + widest = std::max(widest, Utf8::Len(label_) + Utf8::Len(clock) + 3); + } else { + std::string line = std::string("\r") + frame + " " + label_; + std::fwrite(line.data(), 1, line.size(), stderr); + } + std::fflush(stderr); + ++pos; + std::this_thread::sleep_for(std::chrono::milliseconds(80)); + } + std::string clear = "\r" + std::string(widest, ' ') + "\r"; + std::fwrite(clear.data(), 1, clear.size(), stderr); + std::fflush(stderr); +} + +void Spinner::Stop() { + stop_ = true; + if (worker_.joinable()) { + // join(timeout=1): the loop wakes within 80ms; detach if it is somehow stuck + for (int i = 0; i < 100 && !finished_; ++i) std::this_thread::sleep_for(std::chrono::milliseconds(10)); + if (finished_) worker_.join(); else worker_.detach(); + } +} +} // namespace detz::shell diff --git a/taicpp/src/shell/spinner.h b/taicpp/src/shell/spinner.h new file mode 100644 index 0000000..44ab195 --- /dev/null +++ b/taicpp/src/shell/spinner.h @@ -0,0 +1,28 @@ +#pragma once +// Spinner: the stderr "thinking" spinner of tai.py (class Spinner): braille frames, shimmering +// label and an elapsed clock when color is on; plain frame + label otherwise. Does nothing +// when stderr is not a tty. +#include +#include +#include +namespace detz::shell { +class Spinner { + public: + explicit Spinner(std::string label); + ~Spinner(); + Spinner(const Spinner&) = delete; + Spinner& operator=(const Spinner&) = delete; + void Start(); + void Stop(); + const std::string& label() const { return label_; } + static const char* const* Frames(); // FRAMES (10 braille frames) + static constexpr int kFrameCount = 10; + private: + void Spin(); + std::string label_; + std::atomic stop_{false}; + std::thread worker_; + std::atomic finished_{false}; + bool started_{false}; +}; +} // namespace detz::shell diff --git a/taicpp/src/shell/targets.cpp b/taicpp/src/shell/targets.cpp new file mode 100644 index 0000000..8166271 --- /dev/null +++ b/taicpp/src/shell/targets.cpp @@ -0,0 +1,117 @@ +#include "targets.h" +#include "pyglob.h" +#include "pypath.h" +#include "shlex.h" +#include "utf8.h" +#include +namespace detz::shell { +namespace { +bool OneOf(const std::string& s, std::initializer_list set) { + for (auto x : set) if (s == x) return true; + return false; +} +// re.fullmatch(r"[0-9.]+[smhd]?", s) +bool IsTimeout(const std::string& s) { + size_t i = 0; + while (i < s.size() && ((s[i] >= '0' && s[i] <= '9') || s[i] == '.')) ++i; + if (i == 0) return false; + if (i < s.size() && OneOf(std::string(1, s[i]), {"s", "m", "h", "d"})) ++i; + return i == s.size(); +} +// re.fullmatch(r"[0-9]+[KMGTPE]?[iB]?", s) +bool IsSize(const std::string& s) { + size_t i = 0; + while (i < s.size() && s[i] >= '0' && s[i] <= '9') ++i; + if (i == 0) return false; + if (i < s.size() && std::string("KMGTPE").find(s[i]) != std::string::npos) ++i; + if (i < s.size() && (s[i] == 'i' || s[i] == 'B')) ++i; + return i == s.size(); +} +} // namespace + +std::vector ShellTargets::Segment(const std::vector& tokens) { + std::vector targets; + size_t pos = 0; + while (pos < tokens.size()) { + const std::string& token = tokens[pos]; + bool assign = token.find('=') != std::string::npos && !(Utf8::StartsWith(token, ">") || Utf8::StartsWith(token, "<") || Utf8::StartsWith(token, "-") || Utf8::StartsWith(token, "/")); + if (assign) ++pos; + else if (OneOf(token, {"sudo", "command", "env", "nice", "nohup", "setsid"})) ++pos; + else if (token == "timeout" && pos + 1 < tokens.size() && IsTimeout(tokens[pos + 1])) pos += 2; + else break; + } + std::vector rest(tokens.begin() + (long)pos, tokens.end()); + for (size_t index = 0; index < rest.size(); ++index) { + if ((rest[index] == ">" || rest[index] == ">>") && index + 1 < rest.size()) { + const std::string& cand = rest[index + 1]; + if (!cand.empty() && !Utf8::StartsWith(cand, "&") && !Utf8::StartsWith(cand, "/dev/")) targets.push_back(cand); + } + } + if (rest.empty()) return targets; + std::string name = PyPath::Basename(rest[0]); + std::vector operands; + for (size_t k = 1; k < rest.size(); ++k) + if (!Utf8::StartsWith(rest[k], "-") || rest[k] == "-" || rest[k] == "--") operands.push_back(rest[k]); + std::vector filtered; + for (auto& item : operands) if (!OneOf(item, {">", ">>", "<", "--"})) filtered.push_back(item); + operands = filtered; + if (name == "rm") targets.insert(targets.end(), operands.begin(), operands.end()); + else if ((name == "mv" || name == "cp") && !operands.empty()) targets.push_back(operands.back()); + else if (name == "tee") targets.insert(targets.end(), operands.begin(), operands.end()); + else if (name == "truncate") { for (auto& item : operands) { if (IsSize(item)) continue; targets.push_back(item); } } + else if (name == "dd") { for (auto& item : operands) if (Utf8::StartsWith(item, "of=")) targets.push_back(item.substr(3)); } + else if (name == "shred") targets.insert(targets.end(), operands.begin(), operands.end()); + std::vector result; + for (auto& item : targets) if (!item.empty() && item != "-") result.push_back(item); + return result; +} + +std::vector ShellTargets::Paths(const std::string& command, const std::string& workdir) { + std::string spaced; // re.sub(r"([;&|()<>])", r" \1 ", command) + for (char c : command) { + if (std::string(";&|()<>").find(c) != std::string::npos) { spaced += ' '; spaced += c; spaced += ' '; } + else spaced += c; + } + std::vector tokens; + if (!Shlex::Split(spaced, &tokens)) return {}; + std::vector found, segment; + for (auto& token : tokens) { + if (OneOf(token, {";", "&&", "||", "|", "&", "(", ")"})) { + auto t = Segment(segment); + found.insert(found.end(), t.begin(), t.end()); + segment.clear(); + } else segment.push_back(token); + } + { auto t = Segment(segment); found.insert(found.end(), t.begin(), t.end()); } + std::vector resolved; + const size_t cap = kShellSnapshotMaxFiles; + for (auto& item : found) { + if (resolved.size() >= cap) break; + std::string base = PyPath::IsAbs(item) ? item : PyPath::Join(workdir, item); + std::vector hits; + if (item.find_first_of("*?[") != std::string::npos) { + hits = PyGlob::Glob(base); + std::sort(hits.begin(), hits.end()); + } else hits = {base}; + for (auto& hit : hits) { + if (resolved.size() >= cap) break; + if (PyPath::IsFile(hit)) resolved.push_back(PyPath::Abspath(hit)); + else if (PyPath::IsDir(hit)) { + PyGlob::Walk(hit, [&](const PyGlob::WalkEntry& e) { + std::vector files = e.files; + std::sort(files.begin(), files.end()); + for (auto& entry : files) { + if (resolved.size() >= cap) break; + std::string full = PyPath::Join(e.root, entry); + if (PyPath::IsFile(full)) resolved.push_back(PyPath::Abspath(full)); + } + return resolved.size() < cap; + }); + } + } + } + std::vector seen; + for (auto& item : resolved) if (std::find(seen.begin(), seen.end(), item) == seen.end()) seen.push_back(item); + return seen; +} +} // namespace detz::shell diff --git a/taicpp/src/shell/targets.h b/taicpp/src/shell/targets.h new file mode 100644 index 0000000..d3dd1a0 --- /dev/null +++ b/taicpp/src/shell/targets.h @@ -0,0 +1,15 @@ +#pragma once +// ShellTargets: shell_target_paths() and segment_targets() of tai.py: which existing files +// a shell command is about to change (rm, mv, cp, tee, truncate, dd of=, shred, > redirects). +#include +#include +namespace detz::shell { +constexpr size_t kShellSnapshotMaxFiles = 25; // SHELL_SNAPSHOT_MAX_FILES +class ShellTargets { + public: + // shell_target_paths(command, workdir): absolute, de-duplicated file paths, capped. + static std::vector Paths(const std::string& command, const std::string& workdir); + // segment_targets(tokens): raw target operands of one command segment. + static std::vector Segment(const std::vector& tokens); +}; +} // namespace detz::shell diff --git a/taicpp/src/util/.gitkeep b/taicpp/src/util/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/taicpp/src/util/ansi.cpp b/taicpp/src/util/ansi.cpp new file mode 100644 index 0000000..9cdf1ed --- /dev/null +++ b/taicpp/src/util/ansi.cpp @@ -0,0 +1,157 @@ +#include "ansi.h" +#include "utf8.h" +#include +#include +#include +namespace detz { +bool AnsiText::ColorEnabled() { + const char* nc = std::getenv("NO_COLOR"); + return !(nc && *nc) && ::isatty(1); +} +bool AnsiText::StderrColorEnabled() { + const char* nc = std::getenv("NO_COLOR"); + return !(nc && *nc) && ::isatty(2); +} +int AnsiText::TerminalWidth(int fallback) { + int columns = 0; + if (const char* env = std::getenv("COLUMNS")) { + char* end = nullptr; + long v = std::strtol(env, &end, 10); + if (end != env && *end == '\0' && v > 0) columns = (int)v; + } + if (columns <= 0) { + struct winsize ws; + if (::ioctl(1, TIOCGWINSZ, &ws) == 0) columns = ws.ws_col; + } + if (columns <= 0) columns = fallback; + return columns; +} +std::string AnsiText::Paint(const std::string& text, const std::vector& codes) { + if (!ColorEnabled()) return text; + std::string o; + for (auto& c : codes) o += c; + return o + text + Ansi::kReset; +} +size_t AnsiText::SgrAt(const std::string& t, size_t pos) { + if (pos >= t.size() || t[pos] != '\033' || pos + 1 >= t.size() || t[pos + 1] != '[') return 0; + size_t i = pos + 2; + while (i < t.size() && ((t[i] >= '0' && t[i] <= '9') || t[i] == ';')) ++i; + if (i < t.size() && t[i] == 'm') return i + 1 - pos; + return 0; +} +std::string AnsiText::Strip(const std::string& text) { + std::string o; o.reserve(text.size()); + size_t i = 0; + while (i < text.size()) { + size_t n = text[i] == '\033' ? SgrAt(text, i) : 0; + if (n) i += n; else o.push_back(text[i++]); + } + return o; +} +size_t AnsiText::PlainLen(const std::string& text) { return Utf8::Len(Strip(text)); } +std::string AnsiText::StyleText(const std::string& text, const std::vector& codes) { + std::string o; + for (auto& c : codes) o += c; + return o + text + Ansi::kReset; +} +std::string AnsiText::StyleSpan(const std::string& text, const std::vector& codes) { + std::string prefix; + for (auto& c : codes) prefix += c; + return prefix + Utf8::ReplaceAll(text, Ansi::kReset, std::string(Ansi::kReset) + prefix) + Ansi::kReset; +} +std::string AnsiText::Truncate(const std::string& text, long width) { + if ((long)PlainLen(text) <= width) return text; + std::string out; + long count = 0; + size_t pos = 0; + bool broke = false; + size_t i = 0; + while (i < text.size()) { + size_t n = text[i] == '\033' ? SgrAt(text, i) : 0; + if (!n) { ++i; continue; } + std::string chunk = text.substr(pos, i - pos); + long take = std::min((long)Utf8::Len(chunk), width - count); + out += Utf8::Head(chunk, take); + count += take; + if (count >= width) { pos = i; broke = true; break; } + out += text.substr(i, n); + pos = i + n; + i += n; + } + (void)broke; + if (count < width) out += Utf8::Head(Strip(text.substr(pos)), width - count); + return out + Ansi::kReset; +} +std::string AnsiText::PadTinted(const std::string& line, long width) { + long plain = (long)PlainLen(line); + if (plain < width) { + std::string fill((size_t)(width - plain), ' '); + std::string reset = Ansi::kReset; + if (Utf8::EndsWith(line, reset)) return line.substr(0, line.size() - reset.size()) + fill + reset; + return line + fill; + } + return line; +} +std::vector AnsiText::Wrap(const std::string& text, long width, const std::string& first_prefix, const std::string& next_prefix) { + // re.findall(r"\033\[[0-9;]*m|[^\s\033]+|\s+", text) + std::vector parts; + std::u32string u = Utf8::Decode(text); + size_t i = 0; + while (i < u.size()) { + char32_t c = u[i]; + if (c == 0x1b) { + size_t j = i + 1; + bool ok = j < u.size() && u[j] == '['; + if (ok) { + ++j; + while (j < u.size() && ((u[j] >= '0' && u[j] <= '9') || u[j] == ';')) ++j; + ok = j < u.size() && u[j] == 'm'; + } + if (ok) { parts.push_back(Utf8::Encode(u.substr(i, j + 1 - i))); i = j + 1; } else ++i; + } else if (Utf8::IsSpace(c)) { + size_t j = i; + while (j < u.size() && Utf8::IsSpace(u[j])) ++j; + parts.push_back(Utf8::Encode(u.substr(i, j - i))); + i = j; + } else { + size_t j = i; + while (j < u.size() && !Utf8::IsSpace(u[j]) && u[j] != 0x1b) ++j; + parts.push_back(Utf8::Encode(u.substr(i, j - i))); + i = j; + } + } + std::vector lines; + std::string current = first_prefix; + long length = (long)PlainLen(first_prefix); + long base = length; + std::string active; + const std::string reset = Ansi::kReset; + for (auto& part : parts) { + if (part.empty()) continue; + if (part[0] == '\033') { + current += part; + if (part == reset) active.clear(); else active += part; + continue; + } + if (Utf8::IsSpace(Utf8::Decode(part)[0])) { + if (length < width && length > base) { current += " "; length += 1; } + continue; + } + long plen = (long)Utf8::Len(part); + if (length + plen > width && length > base) { + if (!active.empty()) current += reset; + lines.push_back(Utf8::Rstrip(current)); + current = next_prefix + active; + length = (long)PlainLen(next_prefix); + base = length; + } + current += part; + length += plen; + } + if (!active.empty()) current += reset; + if (!Utf8::Strip(current).empty()) lines.push_back(Utf8::Rstrip(current)); + return lines; +} +std::string Paint(const std::string& text, const std::string& codes) { return AnsiText::Paint(text, {codes}); } +std::string Paint2(const std::string& text, const std::string& c1, const std::string& c2) { return AnsiText::Paint(text, {c1, c2}); } +} // namespace detz diff --git a/taicpp/src/util/ansi.h b/taicpp/src/util/ansi.h new file mode 100644 index 0000000..e3c4c68 --- /dev/null +++ b/taicpp/src/util/ansi.h @@ -0,0 +1,58 @@ +#pragma once +// Ansi: escape codes (tai.py class Ansi plus the module-level FG_/BG_ constants) +// and AnsiText, the static helpers around them (strip_ansi, style_text, style_span, +// truncate_ansi, pad_tinted, wrap_ansi, paint, color_enabled). +#include +#include +namespace detz { +struct Ansi { + static constexpr const char* kReset = "\033[0m"; + static constexpr const char* kBold = "\033[1m"; + static constexpr const char* kDim = "\033[2m"; + static constexpr const char* kUnderline = "\033[4m"; + static constexpr const char* kStrike = "\033[9m"; + static constexpr const char* kRed = "\033[31m"; + static constexpr const char* kGreen = "\033[32m"; + static constexpr const char* kYellow = "\033[33m"; + static constexpr const char* kBlue = "\033[34m"; + static constexpr const char* kMagenta = "\033[35m"; + static constexpr const char* kCyan = "\033[36m"; + static constexpr const char* kGray = "\033[90m"; +}; +// Module-level color constants of tai.py (BG_ADD ... FG_FORK). +struct Palette { + static constexpr const char* kBgAdd = "\033[48;5;236m"; + static constexpr const char* kBgDel = "\033[48;5;236m"; + static constexpr const char* kFgAdd = "\033[38;5;222m"; + static constexpr const char* kFgDel = "\033[38;5;203m"; + static constexpr const char* kFgFile = "\033[38;5;111m"; + static constexpr const char* kFgBright = "\033[97m"; + static constexpr const char* kFgYolo = "\033[38;5;211m"; + static constexpr const char* kFgOrange = "\033[38;5;214m"; + static constexpr const char* kFgRun = "\033[38;5;80m"; + static constexpr const char* kFgFork = "\033[38;5;135m"; +}; +class AnsiText { + public: + // not NO_COLOR and stdout is a tty + static bool ColorEnabled(); + // not NO_COLOR and stderr is a tty + static bool StderrColorEnabled(); + // shutil.get_terminal_size().columns (COLUMNS env, then the tty, then fallback) + static int TerminalWidth(int fallback = 80); + // paint(): codes + text + RESET, or the bare text when NO_COLOR or stdout is no tty + static std::string Paint(const std::string& text, const std::vector& codes); + // strip_ansi(): removes \033[...m sequences + static std::string Strip(const std::string& text); + static size_t PlainLen(const std::string& text); // len(strip_ansi(text)) + static std::string StyleText(const std::string& text, const std::vector& codes); + static std::string StyleSpan(const std::string& text, const std::vector& codes); + static std::string Truncate(const std::string& text, long width); + static std::string PadTinted(const std::string& line, long width); + static std::vector Wrap(const std::string& text, long width, const std::string& first_prefix = "", const std::string& next_prefix = ""); + // Length in bytes of an ANSI SGR sequence (\033[0-9;*m) starting at pos, or 0. + static size_t SgrAt(const std::string& text, size_t pos); +}; +std::string Paint(const std::string& text, const std::string& codes); +std::string Paint2(const std::string& text, const std::string& c1, const std::string& c2); +} // namespace detz diff --git a/taicpp/src/util/diffrender.cpp b/taicpp/src/util/diffrender.cpp new file mode 100644 index 0000000..cd7b9a2 --- /dev/null +++ b/taicpp/src/util/diffrender.cpp @@ -0,0 +1,113 @@ +#include "diffrender.h" +#include "ansi.h" +#include "glow.h" +#include "pyhighlight.h" +#include "sequencematcher.h" +#include "utf8.h" +#include +namespace detz { +namespace { +struct Row { std::string kind; long oldNo, newNo; std::string text; }; +long Count(const std::string& s, const std::string& mark) { + long n = 0; size_t p = 0; + while ((p = s.find(mark, p)) != std::string::npos) { ++n; p += mark.size(); } + return n; +} +} // namespace + +std::vector DiffRender::SplitLines(const std::string& text) { + std::vector lines; + if (text.empty()) return lines; + size_t i = 0; + while (true) { + size_t j = text.find('\n', i); + if (j == std::string::npos) { lines.push_back(text.substr(i)); break; } + lines.push_back(text.substr(i, j - i)); + i = j + 1; + } + if (!lines.empty() && lines.back().empty()) lines.pop_back(); + return lines; +} + +std::string DiffRender::Render(const std::string& path, const std::optional& oldText, const std::optional& newText, long width, bool colors) { + if (newText && oldText.value_or("") == *newText) return ""; + std::vector oldLines = SplitLines(oldText.value_or("")); + std::vector newLines = newText ? SplitLines(*newText) : std::vector(); + if (!width) width = AnsiText::TerminalWidth(80); + SequenceMatcher matcher(oldLines, newLines); + std::vector rows; + for (auto& group : matcher.GroupedOpcodes(kDiffContext)) { + if (!rows.empty()) rows.push_back({"gap", 0, 0, ""}); + for (auto& op : group) { + if (op.tag == "equal") { + for (long k = 0; k < op.i2 - op.i1; ++k) rows.push_back({"ctx", op.i1 + k + 1, op.j1 + k + 1, oldLines[op.i1 + k]}); + } else if (op.tag == "delete") { + for (long k = 0; k < op.i2 - op.i1; ++k) rows.push_back({"del", op.i1 + k + 1, 0, oldLines[op.i1 + k]}); + } else if (op.tag == "insert") { + for (long k = 0; k < op.j2 - op.j1; ++k) rows.push_back({"add", 0, op.j1 + k + 1, newLines[op.j1 + k]}); + } else { + for (long k = 0; k < op.i2 - op.i1; ++k) rows.push_back({"del", op.i1 + k + 1, 0, oldLines[op.i1 + k]}); + for (long k = 0; k < op.j2 - op.j1; ++k) rows.push_back({"add", 0, op.j1 + k + 1, newLines[op.j1 + k]}); + } + } + } + long added = 0, removed = 0; + for (auto& r : rows) { if (r.kind == "add") ++added; if (r.kind == "del") ++removed; } + std::string verb, meta; + if (!newText) { verb = "Deleted"; meta = "(deleted, " + std::to_string(oldLines.size()) + " lines)"; } + else if (oldLines.empty()) { verb = "Created"; meta = "(new file, " + std::to_string(newLines.size()) + " lines)"; } + else { verb = "Edited"; meta = "(+" + std::to_string(added) + " -" + std::to_string(removed) + ")"; } + std::string lowered = Utf8::Lower(path); + bool python = Utf8::EndsWith(lowered, ".py"); + long biggest = std::max({(long)oldLines.size(), (long)newLines.size(), 1}); + long numWidth = std::max(4, (long)std::to_string(biggest).size()); + long codeWidth = std::max(20, width - numWidth - 4); + std::string fence; + bool inFence = false; + std::vector out; + if (colors) out.push_back(shell::Glow::EventRow("\xe2\x97\x86", verb, path, meta, width)); // ◆ + else out.push_back("\xe2\x94\x82 \xe2\x94\x80\xe2\x94\x80 " + path + " " + meta); // │ ── + const std::string dimDots = "\xc2\xb7\xc2\xb7\xc2\xb7"; // ··· + for (size_t idx = 0; idx < rows.size() && idx < (size_t)kDiffMaxLines; ++idx) { + const Row& r = rows[idx]; + if (r.kind == "gap") { out.push_back(colors ? std::string(Ansi::kDim) + dimDots + Ansi::kReset : dimDots); continue; } + std::string code = Utf8::ExpandTabs(AnsiText::Strip(r.text), 8); + if (inFence) { + if (python && colors) code = std::string(Ansi::kYellow) + code + Ansi::kReset; + if (r.text.find(fence) != std::string::npos) inFence = false; + } else if (python && colors) { + bool opened = false; + for (const char* mark : {"\"\"\"", "'''"}) { + if (Count(r.text, mark) % 2 == 1) { + fence = mark; inFence = true; opened = true; + code = std::string(Ansi::kYellow) + code + Ansi::kReset; + break; + } + } + if (!opened) code = PyHighlight::Line(code); + } + long number = (r.kind == "ctx" || r.kind == "del") ? r.oldNo : r.newNo; + std::string gutter = Utf8::RJust(std::to_string(number), (size_t)numWidth); + if (!colors) { + const char* sign = r.kind == "ctx" ? " " : r.kind == "del" ? "-" : "+"; + out.push_back(gutter + " " + sign + " " + Utf8::Head(code, codeWidth)); + continue; + } + std::string dimmed = std::string(Ansi::kDim) + gutter + Ansi::kReset; + if (r.kind == "ctx") { + out.push_back(dimmed + " " + AnsiText::Truncate(code, codeWidth)); + } else if (r.kind == "del") { + std::string row = dimmed + " " + Ansi::kRed + Ansi::kBold + "-" + Ansi::kReset + " " + AnsiText::Truncate(code, codeWidth); + out.push_back(AnsiText::PadTinted(AnsiText::StyleSpan(row, {Palette::kBgDel, Palette::kFgDel}), width)); + } else { + std::string row = dimmed + " " + Palette::kFgAdd + Ansi::kBold + "+" + Ansi::kReset + " " + AnsiText::Truncate(code, codeWidth); + out.push_back(AnsiText::PadTinted(AnsiText::StyleSpan(row, {Palette::kBgAdd, Palette::kFgAdd}), width)); + } + } + if (rows.size() > (size_t)kDiffMaxLines) { + std::string note = "\xc2\xb7\xc2\xb7\xc2\xb7 " + std::to_string(rows.size() - kDiffMaxLines) + " more lines hidden"; + out.push_back(colors ? std::string(Ansi::kDim) + note + Ansi::kReset : note); + } + return Utf8::Join(out, "\n"); +} +} // namespace detz diff --git a/taicpp/src/util/diffrender.h b/taicpp/src/util/diffrender.h new file mode 100644 index 0000000..3a40631 --- /dev/null +++ b/taicpp/src/util/diffrender.h @@ -0,0 +1,15 @@ +#pragma once +// DiffRender: render_diff / split_diff_lines of tai.py. +#include +#include +#include +namespace detz { +constexpr int kDiffContext = 3; // DIFF_CONTEXT +constexpr int kDiffMaxLines = 120; // DIFF_MAX_LINES +class DiffRender { + public: + static std::vector SplitLines(const std::string& text); + // render_diff(path, old, new, width=0, colors=True); nullopt is Python None. + static std::string Render(const std::string& path, const std::optional& oldText, const std::optional& newText, long width = 0, bool colors = true); +}; +} // namespace detz diff --git a/taicpp/src/util/digest.cpp b/taicpp/src/util/digest.cpp new file mode 100644 index 0000000..f04b34a --- /dev/null +++ b/taicpp/src/util/digest.cpp @@ -0,0 +1,63 @@ +#include "digest.h" +#include +#include +namespace detz { +namespace { +inline uint32_t Rotr(uint32_t x, int n) { return (x >> n) | (x << (32 - n)); } +} // namespace +std::string Digest::Sha256Hex(const std::string& data) { + static const uint32_t K[64] = { + 0x428a2f98, 0x71374491, 0xb5c0fbcf, 0xe9b5dba5, 0x3956c25b, 0x59f111f1, 0x923f82a4, 0xab1c5ed5, 0xd807aa98, 0x12835b01, 0x243185be, 0x550c7dc3, 0x72be5d74, 0x80deb1fe, 0x9bdc06a7, 0xc19bf174, + 0xe49b69c1, 0xefbe4786, 0x0fc19dc6, 0x240ca1cc, 0x2de92c6f, 0x4a7484aa, 0x5cb0a9dc, 0x76f988da, 0x983e5152, 0xa831c66d, 0xb00327c8, 0xbf597fc7, 0xc6e00bf3, 0xd5a79147, 0x06ca6351, 0x14292967, + 0x27b70a85, 0x2e1b2138, 0x4d2c6dfc, 0x53380d13, 0x650a7354, 0x766a0abb, 0x81c2c92e, 0x92722c85, 0xa2bfe8a1, 0xa81a664b, 0xc24b8b70, 0xc76c51a3, 0xd192e819, 0xd6990624, 0xf40e3585, 0x106aa070, + 0x19a4c116, 0x1e376c08, 0x2748774c, 0x34b0bcb5, 0x391c0cb3, 0x4ed8aa4a, 0x5b9cca4f, 0x682e6ff3, 0x748f82ee, 0x78a5636f, 0x84c87814, 0x8cc70208, 0x90befffa, 0xa4506ceb, 0xbef9a3f7, 0xc67178f2}; + uint32_t h[8] = {0x6a09e667, 0xbb67ae85, 0x3c6ef372, 0xa54ff53a, 0x510e527f, 0x9b05688c, 0x1f83d9ab, 0x5be0cd19}; + std::string msg = data; + uint64_t bits = (uint64_t)data.size() * 8; + msg.push_back((char)0x80); + while (msg.size() % 64 != 56) msg.push_back('\0'); + for (int i = 7; i >= 0; --i) msg.push_back((char)((bits >> (8 * i)) & 0xff)); + for (size_t off = 0; off < msg.size(); off += 64) { + uint32_t w[64]; + for (int i = 0; i < 16; ++i) + w[i] = ((uint32_t)(unsigned char)msg[off + i * 4] << 24) | ((uint32_t)(unsigned char)msg[off + i * 4 + 1] << 16) | ((uint32_t)(unsigned char)msg[off + i * 4 + 2] << 8) | (uint32_t)(unsigned char)msg[off + i * 4 + 3]; + for (int i = 16; i < 64; ++i) { + uint32_t s0 = Rotr(w[i - 15], 7) ^ Rotr(w[i - 15], 18) ^ (w[i - 15] >> 3); + uint32_t s1 = Rotr(w[i - 2], 17) ^ Rotr(w[i - 2], 19) ^ (w[i - 2] >> 10); + w[i] = w[i - 16] + s0 + w[i - 7] + s1; + } + uint32_t a = h[0], b = h[1], c = h[2], d = h[3], e = h[4], f = h[5], g = h[6], hh = h[7]; + for (int i = 0; i < 64; ++i) { + uint32_t S1 = Rotr(e, 6) ^ Rotr(e, 11) ^ Rotr(e, 25); + uint32_t ch = (e & f) ^ (~e & g); + uint32_t t1 = hh + S1 + ch + K[i] + w[i]; + uint32_t S0 = Rotr(a, 2) ^ Rotr(a, 13) ^ Rotr(a, 22); + uint32_t maj = (a & b) ^ (a & c) ^ (b & c); + uint32_t t2 = S0 + maj; + hh = g; g = f; f = e; e = d + t1; d = c; c = b; b = a; a = t1 + t2; + } + h[0] += a; h[1] += b; h[2] += c; h[3] += d; h[4] += e; h[5] += f; h[6] += g; h[7] += hh; + } + char out[65]; + for (int i = 0; i < 8; ++i) std::snprintf(out + i * 8, 9, "%08x", h[i]); + return std::string(out, 64); +} +std::string Digest::Base64(const std::string& data) { + static const char* tbl = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"; + std::string out; + size_t i = 0; + while (i + 2 < data.size()) { + uint32_t v = ((unsigned char)data[i] << 16) | ((unsigned char)data[i + 1] << 8) | (unsigned char)data[i + 2]; + out += tbl[v >> 18]; out += tbl[(v >> 12) & 63]; out += tbl[(v >> 6) & 63]; out += tbl[v & 63]; + i += 3; + } + if (i + 1 == data.size()) { + uint32_t v = (unsigned char)data[i] << 16; + out += tbl[v >> 18]; out += tbl[(v >> 12) & 63]; out += "=="; + } else if (i + 2 == data.size()) { + uint32_t v = ((unsigned char)data[i] << 16) | ((unsigned char)data[i + 1] << 8); + out += tbl[v >> 18]; out += tbl[(v >> 12) & 63]; out += tbl[(v >> 6) & 63]; out += '='; + } + return out; +} +} // namespace detz diff --git a/taicpp/src/util/digest.h b/taicpp/src/util/digest.h new file mode 100644 index 0000000..b16eea0 --- /dev/null +++ b/taicpp/src/util/digest.h @@ -0,0 +1,10 @@ +#pragma once +// Digest: sha256 hexdigest and base64 (hashlib.sha256(...).hexdigest(), base64.b64encode). +#include +namespace detz { +class Digest { + public: + static std::string Sha256Hex(const std::string& data); + static std::string Base64(const std::string& data); +}; +} // namespace detz diff --git a/taicpp/src/util/http.cpp b/taicpp/src/util/http.cpp new file mode 100644 index 0000000..7fbd732 --- /dev/null +++ b/taicpp/src/util/http.cpp @@ -0,0 +1,40 @@ +#include "util/http.h" +#include +#include +#include +namespace detz { +namespace { +struct Buf { std::string data; std::map* hdrs{nullptr}; std::string* ctype{nullptr}; std::string sse; std::function* cb{nullptr}; std::string ev; std::string dt; }; +size_t wbody(char* p, size_t n, size_t m, void* u){ ((Buf*)u)->data.append(p,n*m); return n*m; } +size_t whdr(char* p, size_t n, size_t m, void* u){ Buf* b=(Buf*)u; std::string l(p,n*m); auto c=l.find(':'); if(c!=std::string::npos){ std::string k=l.substr(0,c),v=l.substr(c+1); while(!v.empty()&&(v.back()=='\r'||v.back()=='\n'))v.pop_back(); while(!v.empty()&&v[0]==' ')v.erase(v.begin()); if(b->hdrs)(*b->hdrs)[k]=v; if(k=="Content-Type"&&b->ctype)*b->ctype=v; } return n*m; } +size_t wsse(char* p, size_t n, size_t m, void* u){ Buf* b=(Buf*)u; b->sse.append(p,n*m); size_t pos; while((pos=b->sse.find("\n\n"))!=std::string::npos){ std::string blk=b->sse.substr(0,pos); b->sse.erase(0,pos+2); SseEvent e; std::istringstream s(blk); std::string ln; while(std::getline(s,ln)){ if(!ln.empty()&&ln.back()=='\r')ln.pop_back(); if(ln.rfind("event:",0)==0)e.event=ln.substr(6); else if(ln.rfind("data:",0)==0){ if(!e.data.empty())e.data+="\n"; e.data+=ln.substr(5);} } if(e.event.empty())e.event="message"; if((*b->cb)&&!e.data.empty())(*b->cb)(e); } b->data+=std::string(p,n*m); return n*m; } +} +HttpResponse HttpRequest(const std::string& method, const std::string& url, const std::string& body, const std::map& headers, long timeout_s){ + HttpResponse r; CURL* c=curl_easy_init(); if(!c) return r; Buf b; b.data=""; b.hdrs=&r.headers; b.ctype=&r.content_type; + curl_easy_setopt(c,CURLOPT_URL,url.c_str()); curl_easy_setopt(c,CURLOPT_WRITEFUNCTION,wbody); curl_easy_setopt(c,CURLOPT_WRITEDATA,&b); + curl_easy_setopt(c,CURLOPT_HEADERFUNCTION,whdr); curl_easy_setopt(c,CURLOPT_HEADERDATA,&b); + curl_easy_setopt(c,CURLOPT_TIMEOUT,timeout_s); curl_easy_setopt(c,CURLOPT_FOLLOWLOCATION,1L); + struct curl_slist* hs=nullptr; for(auto&[k,v]:headers) hs=curl_slist_append(hs,(k+": "+v).c_str()); + if(hs) curl_easy_setopt(c,CURLOPT_HTTPHEADER,hs); + if(method=="POST"){ curl_easy_setopt(c,CURLOPT_POST,1L); curl_easy_setopt(c,CURLOPT_POSTFIELDS,body.c_str()); curl_easy_setopt(c,CURLOPT_POSTFIELDSIZE,(long)body.size()); } + else if(method!="GET"){ curl_easy_setopt(c,CURLOPT_CUSTOMREQUEST,method.c_str()); if(!body.empty()){curl_easy_setopt(c,CURLOPT_POSTFIELDS,body.c_str());curl_easy_setopt(c,CURLOPT_POSTFIELDSIZE,(long)body.size());} } + CURLcode rc=curl_easy_perform(c); if(rc==CURLE_OK){ curl_easy_getinfo(c,CURLINFO_RESPONSE_CODE,&r.status); r.body=b.data; } + curl_slist_free_all(hs); curl_easy_cleanup(c); return r; +} +HttpResponse HttpGet(const std::string& u,const std::map& h,long t){ return HttpRequest("GET",u,"",h,t); } +HttpResponse HttpPost(const std::string& u,const std::string& b,const std::map& h,long t){ return HttpRequest("POST",u,b,h,t); } +long HttpPostSse(const std::string& url,const std::string& body,const std::map& headers,long timeout_s,const std::function& cb){ + CURL* c=curl_easy_init(); if(!c) return 0; Buf b; b.cb=(std::function*)&cb; + curl_easy_setopt(c,CURLOPT_URL,url.c_str()); curl_easy_setopt(c,CURLOPT_POST,1L); + curl_easy_setopt(c,CURLOPT_POSTFIELDS,body.c_str()); curl_easy_setopt(c,CURLOPT_POSTFIELDSIZE,(long)body.size()); + curl_easy_setopt(c,CURLOPT_WRITEFUNCTION,wsse); curl_easy_setopt(c,CURLOPT_WRITEDATA,&b); + curl_easy_setopt(c,CURLOPT_TIMEOUT,timeout_s); + struct curl_slist* hs=nullptr; for(auto&[k,v]:headers) hs=curl_slist_append(hs,(k+": "+v).c_str()); + if(hs) curl_easy_setopt(c,CURLOPT_HTTPHEADER,hs); + long st=0; if(curl_easy_perform(c)==CURLE_OK) curl_easy_getinfo(c,CURLINFO_RESPONSE_CODE,&st); + curl_slist_free_all(hs); curl_easy_cleanup(c); return st; +} +std::string JsonEscape(const std::string& s){ std::string o; for(char c:s){ switch(c){case '"':o+="\\\"";break;case '\\':o+="\\\\";break;case '\n':o+="\\n";break;case '\r':o+="\\r";break;case '\t':o+="\\t";break;default: if((unsigned char)c<0x20){char b[8];snprintf(b,8,"\\u%04x",c);o+=b;} else o+=c;} } return o; } +std::string JsonGetStr(const std::string& o,const std::string& k){ std::string p="\""+k+"\""; auto i=o.find(p); if(i==std::string::npos)return ""; i=o.find(':',i); if(i==std::string::npos)return ""; ++i; while(i +#include +#include +#include +namespace detz { +// Minimal HTTP client over libcurl (mirrors tai.py urllib flows: GET/POST, +// headers, timeouts, SSE streaming). No dependency beyond libcurl. +struct HttpResponse { long status{0}; std::string body; std::map headers; std::string content_type; }; +struct SseEvent { std::string event; std::string data; }; +HttpResponse HttpGet(const std::string& url, const std::map& headers = {}, long timeout_s = 60); +HttpResponse HttpPost(const std::string& url, const std::string& body, const std::map& headers = {}, long timeout_s = 60); +HttpResponse HttpRequest(const std::string& method, const std::string& url, const std::string& body, const std::map& headers, long timeout_s); +// Stream SSE: invokes cb per event, returns final HTTP status. +long HttpPostSse(const std::string& url, const std::string& body, const std::map& headers, long timeout_s, const std::function& cb); +// Tiny JSON helpers (escape/extract) mirroring tai.py json usage. +std::string JsonEscape(const std::string& s); +std::string JsonGetStr(const std::string& obj, const std::string& key); +long JsonGetInt(const std::string& obj, const std::string& key, long dflt = 0); +} diff --git a/taicpp/src/util/jsonvalue.cpp b/taicpp/src/util/jsonvalue.cpp new file mode 100644 index 0000000..032ae2c --- /dev/null +++ b/taicpp/src/util/jsonvalue.cpp @@ -0,0 +1,301 @@ +#include "jsonvalue.h" +#include "utf8.h" +#include +#include +#include +#include +#include +#include +namespace detz { +namespace { +class Parser { + public: + explicit Parser(const std::string& t) : s_(t) {} + bool Run(JsonValue* out) { + Ws(); + if (!Value(out, 0)) return false; + Ws(); + return i_ == s_.size(); + } + private: + const std::string& s_; + size_t i_{0}; + void Ws() { while (i_ < s_.size() && (s_[i_] == ' ' || s_[i_] == '\t' || s_[i_] == '\n' || s_[i_] == '\r')) ++i_; } + bool Lit(const char* w) { + size_t n = std::char_traits::length(w); + if (s_.compare(i_, n, w) == 0) { i_ += n; return true; } + return false; + } + bool Value(JsonValue* out, int depth) { + if (depth > 900 || i_ >= s_.size()) return false; + char c = s_[i_]; + if (c == '{') return Object(out, depth); + if (c == '[') return ArrayV(out, depth); + if (c == '"') { std::string v; if (!Str(&v)) return false; *out = JsonValue::String(v); return true; } + if (Lit("null")) { *out = JsonValue::Null(); return true; } + if (Lit("true")) { *out = JsonValue::Bool(true); return true; } + if (Lit("false")) { *out = JsonValue::Bool(false); return true; } + if (Lit("NaN")) { *out = JsonValue::Float(std::nan("")); return true; } + if (Lit("Infinity")) { *out = JsonValue::Float(INFINITY); return true; } + if (Lit("-Infinity")) { *out = JsonValue::Float(-INFINITY); return true; } + return Number(out); + } + bool Number(JsonValue* out) { + size_t st = i_; + if (i_ < s_.size() && s_[i_] == '-') ++i_; + if (i_ >= s_.size()) return false; + if (s_[i_] == '0') ++i_; + else if (s_[i_] >= '1' && s_[i_] <= '9') { while (i_ < s_.size() && std::isdigit((unsigned char)s_[i_])) ++i_; } + else return false; + bool fl = false; + if (i_ + 1 < s_.size() && s_[i_] == '.' && std::isdigit((unsigned char)s_[i_ + 1])) { + fl = true; ++i_; + while (i_ < s_.size() && std::isdigit((unsigned char)s_[i_])) ++i_; + } + if (i_ < s_.size() && (s_[i_] == 'e' || s_[i_] == 'E')) { + size_t j = i_ + 1; + if (j < s_.size() && (s_[j] == '+' || s_[j] == '-')) ++j; + if (j < s_.size() && std::isdigit((unsigned char)s_[j])) { + while (j < s_.size() && std::isdigit((unsigned char)s_[j])) ++j; + fl = true; i_ = j; + } + } + std::string t = s_.substr(st, i_ - st); + if (fl) { *out = JsonValue::Float(std::strtod(t.c_str(), nullptr)); return true; } + errno = 0; + long long v = std::strtoll(t.c_str(), nullptr, 10); + if (errno == ERANGE) { *out = JsonValue::BigInt(t); return true; } // Python ints are unbounded + *out = JsonValue::Int(v); + return true; + } + bool Hex4(char32_t* cp) { + if (i_ + 4 > s_.size()) return false; + char32_t v = 0; + for (int k = 0; k < 4; ++k) { + char c = s_[i_ + k]; int d; + if (c >= '0' && c <= '9') d = c - '0'; else if (c >= 'a' && c <= 'f') d = c - 'a' + 10; else if (c >= 'A' && c <= 'F') d = c - 'A' + 10; else return false; + v = v * 16 + d; + } + i_ += 4; *cp = v; return true; + } + bool Str(std::string* out) { + ++i_; // opening quote + std::string r; + while (true) { + if (i_ >= s_.size()) return false; + unsigned char c = s_[i_]; + if (c == '"') { ++i_; break; } + if (c < 0x20) return false; + if (c != '\\') { r.push_back((char)c); ++i_; continue; } + ++i_; + if (i_ >= s_.size()) return false; + char e = s_[i_++]; + switch (e) { + case '"': r += '"'; break; case '\\': r += '\\'; break; case '/': r += '/'; break; + case 'b': r += '\b'; break; case 'f': r += '\f'; break; case 'n': r += '\n'; break; + case 'r': r += '\r'; break; case 't': r += '\t'; break; + case 'u': { + char32_t cp; + if (!Hex4(&cp)) return false; + if (cp >= 0xD800 && cp <= 0xDBFF && i_ + 1 < s_.size() && s_[i_] == '\\' && s_[i_ + 1] == 'u') { + size_t save = i_; i_ += 2; char32_t lo; + if (Hex4(&lo) && lo >= 0xDC00 && lo <= 0xDFFF) cp = 0x10000 + ((cp - 0xD800) << 10) + (lo - 0xDC00); + else i_ = save; + } + if (cp >= 0xD800 && cp <= 0xDFFF) cp = 0xFFFD; // lone surrogate: not representable in UTF-8 + r += Utf8::EncodeOne(cp); + break; + } + default: return false; + } + } + *out = r; + return true; + } + bool ArrayV(JsonValue* out, int depth) { + ++i_; *out = JsonValue::Array(); Ws(); + if (i_ < s_.size() && s_[i_] == ']') { ++i_; return true; } + while (true) { + Ws(); JsonValue v; + if (!Value(&v, depth + 1)) return false; + out->Push(std::move(v)); Ws(); + if (i_ >= s_.size()) return false; + if (s_[i_] == ',') { ++i_; continue; } + if (s_[i_] == ']') { ++i_; return true; } + return false; + } + } + bool Object(JsonValue* out, int depth) { + ++i_; *out = JsonValue::Object(); Ws(); + if (i_ < s_.size() && s_[i_] == '}') { ++i_; return true; } + while (true) { + Ws(); + if (i_ >= s_.size() || s_[i_] != '"') return false; + std::string k; if (!Str(&k)) return false; + Ws(); + if (i_ >= s_.size() || s_[i_] != ':') return false; + ++i_; Ws(); + JsonValue v; + if (!Value(&v, depth + 1)) return false; + out->Set(k, std::move(v)); Ws(); + if (i_ >= s_.size()) return false; + if (s_[i_] == ',') { ++i_; continue; } + if (s_[i_] == '}') { ++i_; return true; } + return false; + } + } +}; + +std::string FloatRepr(double d) { + if (std::isnan(d)) return "nan"; + if (std::isinf(d)) return d < 0 ? "-inf" : "inf"; + char buf[64]; + int prec = 1; + for (; prec <= 17; ++prec) { + std::snprintf(buf, sizeof buf, "%.*e", prec - 1, d); + if (std::strtod(buf, nullptr) == d) break; + } + // buf = d.ddde[+-]XX ; rebuild in Python repr style + std::string m = buf; + size_t e = m.find('e'); + std::string mant = m.substr(0, e); + int ex = std::atoi(m.c_str() + e + 1); + bool neg = mant[0] == '-'; + if (neg) mant = mant.substr(1); + std::string digits; + for (char c : mant) if (c != '.') digits.push_back(c); + while (digits.size() > 1 && digits.back() == '0') digits.pop_back(); + std::string r; + if (ex < -4 || ex >= 16) { + r = digits.substr(0, 1); + if (digits.size() > 1) r += "." + digits.substr(1); + char eb[16]; std::snprintf(eb, sizeof eb, "e%c%02d", ex < 0 ? '-' : '+', std::abs(ex)); + r += eb; + } else if (ex >= 0) { + if ((int)digits.size() <= ex + 1) r = digits + std::string(ex + 1 - digits.size(), '0') + ".0"; + else r = digits.substr(0, ex + 1) + "." + digits.substr(ex + 1); + } else { + r = "0." + std::string(-ex - 1, '0') + digits; + } + return neg ? "-" + r : r; +} + +bool Printable(char32_t c) { + if (c < 0x20 || (c >= 0x7f && c <= 0xa0) || c == 0xad) return false; + if ((c >= 0x2000 && c <= 0x200f) || (c >= 0x2028 && c <= 0x202f) || (c >= 0x205f && c <= 0x206f)) return false; + if (c == 0x3000 || c == 0xfeff || (c >= 0xfff9 && c <= 0xfffb) || c == 0x1680) return false; + return true; +} + +std::string PyStrRepr(const std::string& s) { + char q = '\''; + if (s.find('\'') != std::string::npos && s.find('"') == std::string::npos) q = '"'; + std::string r(1, q); + for (char32_t c : Utf8::Decode(s)) { + if (c == (char32_t)q || c == '\\') { r += '\\'; r += (char)c; } + else if (c == '\n') r += "\\n"; + else if (c == '\r') r += "\\r"; + else if (c == '\t') r += "\\t"; + else if (!Printable(c)) { + char b[16]; + if (c <= 0xff) std::snprintf(b, sizeof b, "\\x%02x", (unsigned)c); + else if (c <= 0xffff) std::snprintf(b, sizeof b, "\\u%04x", (unsigned)c); + else std::snprintf(b, sizeof b, "\\U%08x", (unsigned)c); + r += b; + } else r += Utf8::EncodeOne(c); + } + r += q; + return r; +} + +std::string JsonStr(const std::string& s) { + std::string r = "\""; + for (char32_t c : Utf8::Decode(s)) { + char b[16]; + if (c == '"') r += "\\\""; else if (c == '\\') r += "\\\\"; + else if (c == '\n') r += "\\n"; else if (c == '\r') r += "\\r"; else if (c == '\t') r += "\\t"; + else if (c == '\b') r += "\\b"; else if (c == '\f') r += "\\f"; + else if (c < 0x20) { std::snprintf(b, sizeof b, "\\u%04x", (unsigned)c); r += b; } + else if (c < 0x7f) r.push_back((char)c); + else if (c <= 0xffff) { std::snprintf(b, sizeof b, "\\u%04x", (unsigned)c); r += b; } + else { + char32_t v = c - 0x10000; + std::snprintf(b, sizeof b, "\\u%04x\\u%04x", (unsigned)(0xd800 + (v >> 10)), (unsigned)(0xdc00 + (v & 0x3ff))); + r += b; + } + } + return r + "\""; +} +} // namespace + +JsonValue JsonValue::Parse(const std::string& text, bool* ok) { + JsonValue v; Parser p(text); + bool good = p.Run(&v); + if (ok) *ok = good; + return good ? v : JsonValue(); +} +const JsonValue* JsonValue::Get(const std::string& key) const { + for (auto& m : obj_) if (m.first == key) return &m.second; + return nullptr; +} +void JsonValue::Set(const std::string& key, JsonValue v) { + for (auto& m : obj_) if (m.first == key) { m.second = std::move(v); return; } + obj_.emplace_back(key, std::move(v)); +} +bool JsonValue::Truthy() const { + switch (type_) { + case Type::Null: return false; + case Type::Bool: return b_; + case Type::Int: return text_ != "0"; + case Type::Float: return d_ != 0; + case Type::String: return !text_.empty(); + case Type::Array: return !arr_.empty(); + case Type::Object: return !obj_.empty(); + } + return false; +} +std::string JsonValue::Dump(bool sortKeys) const { + switch (type_) { + case Type::Null: return "null"; + case Type::Bool: return b_ ? "true" : "false"; + case Type::Int: return text_; + case Type::Float: return std::isnan(d_) ? "NaN" : std::isinf(d_) ? (d_ < 0 ? "-Infinity" : "Infinity") : FloatRepr(d_); + case Type::String: return JsonStr(text_); + case Type::Array: { + std::string r = "["; + for (size_t i = 0; i < arr_.size(); ++i) { if (i) r += ", "; r += arr_[i].Dump(sortKeys); } + return r + "]"; + } + case Type::Object: { + std::vector*> ms; + for (auto& m : obj_) ms.push_back(&m); + if (sortKeys) std::sort(ms.begin(), ms.end(), [](auto* a, auto* b) { return Utf8::Decode(a->first) < Utf8::Decode(b->first); }); + std::string r = "{"; + for (size_t i = 0; i < ms.size(); ++i) { if (i) r += ", "; r += JsonStr(ms[i]->first) + ": " + ms[i]->second.Dump(sortKeys); } + return r + "}"; + } + } + return ""; +} +std::string JsonValue::Repr() const { + switch (type_) { + case Type::Null: return "None"; + case Type::Bool: return b_ ? "True" : "False"; + case Type::Int: return text_; + case Type::Float: return FloatRepr(d_); + case Type::String: return PyStrRepr(text_); + case Type::Array: { + std::string r = "["; + for (size_t i = 0; i < arr_.size(); ++i) { if (i) r += ", "; r += arr_[i].Repr(); } + return r + "]"; + } + case Type::Object: { + std::string r = "{"; + for (size_t i = 0; i < obj_.size(); ++i) { if (i) r += ", "; r += PyStrRepr(obj_[i].first) + ": " + obj_[i].second.Repr(); } + return r + "}"; + } + } + return ""; +} +std::string JsonValue::Str() const { return type_ == Type::String ? text_ : Repr(); } +} // namespace detz diff --git a/taicpp/src/util/jsonvalue.h b/taicpp/src/util/jsonvalue.h new file mode 100644 index 0000000..94ce676 --- /dev/null +++ b/taicpp/src/util/jsonvalue.h @@ -0,0 +1,55 @@ +#pragma once +// JsonValue: a small ordered JSON value with Python json.loads / json.dumps / str() / repr() +// behavior, for the helpers of tai.py that parse tool arguments (Glow.describe_call). +#include +#include +#include +namespace detz { +class JsonValue { + public: + enum class Type { Null, Bool, Int, Float, String, Array, Object }; + JsonValue() = default; + static JsonValue Null() { return JsonValue(); } + static JsonValue Bool(bool v) { JsonValue j; j.type_ = Type::Bool; j.b_ = v; return j; } + static JsonValue Int(long long v) { JsonValue j; j.type_ = Type::Int; j.text_ = std::to_string(v); j.i_ = v; return j; } + // an integer beyond 64 bits, kept as its decimal text + static JsonValue BigInt(std::string digits) { JsonValue j; j.type_ = Type::Int; j.text_ = std::move(digits); return j; } + static JsonValue Float(double v) { JsonValue j; j.type_ = Type::Float; j.d_ = v; return j; } + static JsonValue String(std::string v) { JsonValue j; j.type_ = Type::String; j.text_ = std::move(v); return j; } + static JsonValue Array() { JsonValue j; j.type_ = Type::Array; return j; } + static JsonValue Object() { JsonValue j; j.type_ = Type::Object; return j; } + + // json.loads(text); ok=false (and Null) on any ValueError. + static JsonValue Parse(const std::string& text, bool* ok); + + Type type() const { return type_; } + bool is_null() const { return type_ == Type::Null; } + bool is_object() const { return type_ == Type::Object; } + bool is_array() const { return type_ == Type::Array; } + bool is_string() const { return type_ == Type::String; } + bool bool_value() const { return b_; } + const std::string& string_value() const { return text_; } + const std::vector& items() const { return arr_; } + const std::vector>& members() const { return obj_; } + const JsonValue* Get(const std::string& key) const; + void Push(JsonValue v) { arr_.push_back(std::move(v)); } + void Set(const std::string& key, JsonValue v); + // Python truthiness (for "x or y" idioms) + bool Truthy() const; + + // json.dumps(value, sort_keys=?) with default separators and ensure_ascii. + std::string Dump(bool sortKeys = false) const; + // str(value) and repr(value) of the Python object json.loads would produce. + std::string Str() const; + std::string Repr() const; + + private: + Type type_{Type::Null}; + bool b_{false}; + long long i_{0}; + double d_{0}; + std::string text_; // string payload, or decimal digits for Int + std::vector arr_; + std::vector> obj_; +}; +} // namespace detz diff --git a/taicpp/src/util/markdown.cpp b/taicpp/src/util/markdown.cpp new file mode 100644 index 0000000..a57ba3d --- /dev/null +++ b/taicpp/src/util/markdown.cpp @@ -0,0 +1,444 @@ +#include "markdown.h" +#include "ansi.h" +#include "utf8.h" +#include +#include +#include +namespace detz { +namespace { +using U = std::u32string; +using Matcher = std::function; + +U Sub(const U& t, const Matcher& m) { + U out; + size_t i = 0; + while (i < t.size()) { + size_t end = 0; U repl; + if (m(t, i, end, repl) && end > i) { out += repl; i = end; } + else out.push_back(t[i++]); + } + return out; +} +bool Word(const U& t, size_t i) { return i < t.size() && Utf8::IsWord(t[i]); } +bool At(const U& t, size_t i, const char32_t* lit) { + size_t k = 0; + for (; lit[k]; ++k) if (i + k >= t.size() || t[i + k] != lit[k]) return false; + return true; +} +size_t LitLen(const char32_t* lit) { size_t k = 0; while (lit[k]) ++k; return k; } + +// open + [^\n]+ + close (optionally with \w lookbehind/lookahead guards) +Matcher Delimited(const char32_t* open, char32_t forbid, const char32_t* close, bool guarded, + std::function repl) { + return [=](const U& t, size_t i, size_t& end, U& out) { + if (guarded && i > 0 && Utf8::IsWord(t[i - 1])) return false; + if (!At(t, i, open)) return false; + size_t j = i + LitLen(open), s = j; + while (j < t.size() && t[j] != forbid && t[j] != '\n') ++j; + if (j == s || !At(t, j, close)) return false; + size_t e = j + LitLen(close); + if (guarded && Word(t, e)) return false; + end = e; + out = repl(t.substr(s, j - s)); + return true; + }; +} + +std::string E(const U& u) { return Utf8::Encode(u); } +U D(const std::string& s) { return Utf8::Decode(s); } + +// \s+ "title" and the closing paren of ![alt](url "title") / [text](url "title"); returns end or 0. +size_t LinkTail(const U& t, size_t k) { + if (k < t.size() && t[k] == ')') return k + 1; + if (k < t.size() && Utf8::IsSpace(t[k])) { + size_t m = k; + while (m < t.size() && Utf8::IsSpace(t[m])) ++m; + if (m < t.size() && t[m] == '"') { + size_t q = m + 1; + while (q < t.size() && t[q] != '"') ++q; + if (q < t.size() && q + 1 < t.size() && t[q + 1] == ')') return q + 2; + } + } + return 0; +} + +bool LinkMatch(const U& t, size_t i, bool image, U* label, U* url, size_t* end) { + size_t j = i; + if (image) { if (!At(t, j, U"![")) return false; j += 2; } + else { if (j >= t.size() || t[j] != '[') return false; j += 1; } + size_t ls = j; + while (j < t.size() && t[j] != ']' && t[j] != '\n') ++j; + if (j >= t.size() || t[j] != ']' || (!image && j == ls)) return false; + if (!At(t, j + 1, U"(")) return false; + size_t k = j + 2, us = k; + while (k < t.size() && t[k] != ')' && !Utf8::IsSpace(t[k])) ++k; + if (k == us) return false; + size_t e = LinkTail(t, k); + if (!e) return false; + *label = t.substr(ls, j - ls); + *url = t.substr(us, k - us); + *end = e; + return true; +} + +std::string Style(const std::string& text, std::initializer_list codes) { + std::vector v; + for (auto c : codes) v.push_back(c); + return AnsiText::StyleText(text, v); +} +std::string Span(const std::string& text, std::initializer_list codes) { + std::vector v; + for (auto c : codes) v.push_back(c); + return AnsiText::StyleSpan(text, v); +} + +struct ListMatch { bool ok{false}; U indent; std::string body; }; +ListMatch MatchList(const std::string& line, bool numbered) { + ListMatch m; + U u = D(line); + size_t i = 0, n = u.size(); + while (i < n && Utf8::IsSpace(u[i])) ++i; + size_t j = i; + if (numbered) { + while (j < n && u[j] >= '0' && u[j] <= '9') ++j; + if (j == i || j >= n || (u[j] != '.' && u[j] != ')')) return m; + ++j; + } else { + if (j >= n || (u[j] != '-' && u[j] != '*' && u[j] != '+')) return m; + ++j; + } + size_t k = j; + while (k < n && Utf8::IsSpace(u[k])) ++k; + if (k == j) return m; + size_t e = n; + while (e > k && Utf8::IsSpace(u[e - 1])) --e; + if (e == k) return m; + m.ok = true; m.indent = u.substr(0, i); m.body = E(u.substr(k, e - k)); + return m; +} + +bool IsRunOf(const std::string& s, char c, size_t minLen) { + if (s.size() < minLen) return false; + for (char x : s) if (x != c) return false; + return true; +} +} // namespace + +const std::vector& Markdown::Bullets() { + static const std::vector b = {"\xe2\x80\xa2", "\xe2\x97\xa6", "\xe2\x96\xaa"}; // • ◦ ▪ + return b; +} + +std::string Markdown::Inline(const std::string& input) { + std::vector stashed; + auto placeholder = [&](const std::string& value) { + stashed.push_back(value); + return D(std::string("\0", 1) + std::to_string(stashed.size() - 1) + std::string("\0", 1)); + }; + U t = D(input); + // 1. `code` + t = Sub(t, Delimited(U"`", '`', U"`", false, [&](const U& g) { return placeholder(Style(E(g), {Ansi::kCyan})); })); + // 2. backslash escapes + t = Sub(t, [&](const U& s, size_t i, size_t& end, U& out) { + static const std::u32string punct = U"!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~"; + if (s[i] != '\\' || i + 1 >= s.size() || punct.find(s[i + 1]) == U::npos) return false; + out = placeholder(E(s.substr(i + 1, 1))); + end = i + 2; + return true; + }); + // 3. images + t = Sub(t, [&](const U& s, size_t i, size_t& end, U& out) { + U label, url; + if (!LinkMatch(s, i, true, &label, &url, &end)) return false; + out = placeholder("[image: " + E(label.empty() ? url : label) + "]"); + return true; + }); + // 4. links + t = Sub(t, [&](const U& s, size_t i, size_t& end, U& out) { + U label, url; + if (!LinkMatch(s, i, false, &label, &url, &end)) return false; + out = D(Span(Inline(E(label)), {Ansi::kBlue, Ansi::kUnderline}) + Style(" (" + E(url) + ")", {Ansi::kDim})); + return true; + }); + // 5. **bold** + t = Sub(t, Delimited(U"**", '*', U"**", false, [&](const U& g) { return D(Style(E(g), {Ansi::kBold})); })); + // 6. __bold__ + t = Sub(t, Delimited(U"__", '_', U"__", true, [&](const U& g) { return D(Style(E(g), {Ansi::kBold})); })); + // 7. ~~strike~~ + t = Sub(t, Delimited(U"~~", '~', U"~~", false, [&](const U& g) { return D(Style(E(g), {Ansi::kStrike})); })); + // 8. *dim* + t = Sub(t, Delimited(U"*", '*', U"*", false, [&](const U& g) { return D(Style(E(g), {Ansi::kDim})); })); + // 9. _dim_ + t = Sub(t, Delimited(U"_", '_', U"_", true, [&](const U& g) { return D(Style(E(g), {Ansi::kDim})); })); + std::string text = E(t); + for (size_t pos = 0; pos < stashed.size(); ++pos) + text = Utf8::ReplaceAll(text, std::string("\0", 1) + std::to_string(pos) + std::string("\0", 1), stashed[pos]); + return text; +} + +std::vector Markdown::SplitRow(const std::string& input) { + std::string line = Utf8::Strip(input); + if (Utf8::StartsWith(line, "|")) line = line.substr(1); + if (Utf8::EndsWith(line, "|") && !Utf8::EndsWith(line, "\\|")) line.pop_back(); + std::vector cells; + std::string cur; + for (size_t i = 0; i < line.size(); ++i) { + if (line[i] == '|' && !(i > 0 && line[i - 1] == '\\')) { cells.push_back(cur); cur.clear(); } + else cur.push_back(line[i]); + } + cells.push_back(cur); + for (auto& c : cells) c = Utf8::Strip(Utf8::ReplaceAll(c, "\\|", "|")); + return cells; +} + +bool Markdown::IsSeparator(const std::string& line) { + auto cells = SplitRow(line); + if (cells.empty()) return false; + for (auto& c : cells) { + size_t a = 0, b = c.size(); + if (a < b && c[a] == ':') ++a; + if (b > a && c[b - 1] == ':') --b; + if (b <= a) return false; + for (size_t k = a; k < b; ++k) if (c[k] != '-') return false; + } + return true; +} + +std::string Markdown::AlignCell(const std::string& cell, long width, const std::string& align) { + long pad = std::max(0L, width - (long)AnsiText::PlainLen(cell)); + if (align == "right") return std::string((size_t)pad, ' ') + cell; + if (align == "center") { + long left = pad / 2; + return std::string((size_t)left, ' ') + cell + std::string((size_t)(pad - left), ' '); + } + return cell + std::string((size_t)pad, ' '); +} + +std::vector Markdown::TableBlock(std::vector header, const std::vector& aligns, std::vector> rows) { + size_t span = header.size(); + for (auto& r : rows) span = std::max(span, r.size()); + header.resize(span); + for (auto& r : rows) r.resize(span); + std::vector head; + std::vector> body; + for (auto& c : header) head.push_back(Inline(c)); + for (auto& r : rows) { std::vector row; for (auto& c : r) row.push_back(Inline(c)); body.push_back(row); } + std::vector widths(span, 0); + for (size_t p = 0; p < span; ++p) { + long w = (long)AnsiText::PlainLen(head[p]); + for (auto& r : body) w = std::max(w, (long)AnsiText::PlainLen(r[p])); + widths[p] = w; + } + auto fill = [&](const char* l, const char* mid, const char* r) { + std::string s = l; + for (size_t p = 0; p < span; ++p) { + if (p) s += mid; + for (long k = 0; k < widths[p] + 2; ++k) s += "\xe2\x94\x80"; // ─ + } + return Style(s + r, {Ansi::kDim}); + }; + std::string bar = Style("\xe2\x94\x82", {Ansi::kDim}); // │ + std::vector lines; + lines.push_back(fill("\xe2\x94\x8c", "\xe2\x94\xac", "\xe2\x94\x90")); + std::string l = bar; + for (size_t p = 0; p < span; ++p) { if (p) l += bar; l += " " + Span(AlignCell(head[p], widths[p], aligns[p]), {Ansi::kBold}) + " "; } + lines.push_back(l + bar); + lines.push_back(fill("\xe2\x94\x9c", "\xe2\x94\xbc", "\xe2\x94\xa4")); + for (auto& r : body) { + std::string row = bar; + for (size_t p = 0; p < span; ++p) { if (p) row += bar; row += " " + AlignCell(r[p], widths[p], aligns[p]) + " "; } + lines.push_back(row + bar); + } + lines.push_back(fill("\xe2\x94\x94", "\xe2\x94\xb4", "\xe2\x94\x98")); + return lines; +} + +std::string Markdown::Render(const std::string& input, long width, int color) { + bool col = color < 0 ? AnsiText::ColorEnabled() : color != 0; + if (!col) return input; + if (width < 0) width = AnsiText::TerminalWidth(80); + std::string text = Utf8::ExpandTabs(Utf8::ReplaceAll(input, "\r\n", "\n"), 8); + { // re.sub(r"", " \n", text, flags=re.IGNORECASE) + U u = D(text); + u = Sub(u, [](const U& s, size_t i, size_t& end, U& out) { + if (s[i] != '<' || i + 2 >= s.size() || (s[i + 1] != 'b' && s[i + 1] != 'B') || (s[i + 2] != 'r' && s[i + 2] != 'R')) return false; + size_t j = i + 3; + while (j < s.size() && Utf8::IsSpace(s[j])) ++j; + if (j < s.size() && s[j] == '/') ++j; + if (j >= s.size() || s[j] != '>') return false; + end = j + 1; out = U" \n"; + return true; + }); + text = E(u); + } + std::vector raw; + { + size_t i = 0; + while (true) { // text.split("\n") + size_t j = text.find('\n', i); + if (j == std::string::npos) { raw.push_back(text.substr(i)); break; } + raw.push_back(text.substr(i, j - i)); i = j + 1; + } + } + std::vector out; + size_t pos = 0, total = raw.size(); + while (pos < total) { + const std::string& line = raw[pos]; + std::string stripped = Utf8::Strip(line); + if (stripped.empty()) { ++pos; continue; } + // fenced code + { + std::string t = Utf8::Lstrip(line); + char c = t.empty() ? 0 : t[0]; + size_t k = 0; + if (c == '`' || c == '~') while (k < t.size() && t[k] == c) ++k; + bool fence = false; + if (k >= 3) { + std::u32string rest = D(t.substr(k)); + size_t a = 0, n = rest.size(); + while (a < n && Utf8::IsSpace(rest[a])) ++a; + while (a < n && !Utf8::IsSpace(rest[a])) ++a; + while (a < n && Utf8::IsSpace(rest[a])) ++a; + fence = a == n; + } + if (fence) { + ++pos; + while (pos < total) { + std::string closer = Utf8::Strip(raw[pos]); + if (!closer.empty() && IsRunOf(closer, c, 3)) { ++pos; break; } + out.push_back(!Utf8::Strip(raw[pos]).empty() ? " " + Style(raw[pos], {Ansi::kDim}) : ""); + ++pos; + } + out.push_back(""); + continue; + } + } + // table + if (line.find('|') != std::string::npos && pos + 1 < total && IsSeparator(raw[pos + 1])) { + auto header = SplitRow(line); + std::vector aligns; + for (auto& cell : SplitRow(raw[pos + 1])) { + if (Utf8::StartsWith(cell, ":") && Utf8::EndsWith(cell, ":") && cell.size() > 2) aligns.push_back("center"); + else if (Utf8::EndsWith(cell, ":")) aligns.push_back("right"); + else aligns.push_back("left"); + } + pos += 2; + std::vector> rows; + while (pos < total && raw[pos].find('|') != std::string::npos && !Utf8::Strip(raw[pos]).empty()) { + rows.push_back(SplitRow(raw[pos])); + ++pos; + } + size_t need = header.size(); + for (auto& r : rows) need = std::max(need, r.size()); + while (aligns.size() < need) aligns.push_back("left"); + for (auto& l : TableBlock(header, aligns, rows)) out.push_back(l); + out.push_back(""); + continue; + } + // heading + { + size_t h = 0; + while (h < stripped.size() && stripped[h] == '#') ++h; + if (h >= 1 && h <= 6 && h < stripped.size()) { + std::u32string rest = D(stripped.substr(h)); + size_t a = 0; + while (a < rest.size() && Utf8::IsSpace(rest[a])) ++a; + if (a > 0 && a < rest.size()) { + std::string body = Inline(E(rest.substr(a))); + if (h == 1) out.push_back(Span(body, {Ansi::kBold, Ansi::kCyan})); + else if (h == 2) out.push_back(Span(body, {Ansi::kBold})); + else out.push_back(Span(body, {Ansi::kBold, Ansi::kDim})); + out.push_back(""); + ++pos; + continue; + } + } + } + // horizontal rule + { + std::string nospace = Utf8::ReplaceAll(stripped, " ", ""); + if (IsRunOf(nospace, '*', 3) || IsRunOf(nospace, '-', 3) || IsRunOf(nospace, '_', 3)) { + std::string bar; + for (long k = 0; k < width; ++k) bar += "\xe2\x94\x80"; + out.push_back(Style(bar, {Ansi::kDim})); + out.push_back(""); + ++pos; + continue; + } + } + // blockquote + if (Utf8::StartsWith(stripped, ">")) { + std::vector> paras; + std::vector current; + while (pos < total && Utf8::StartsWith(Utf8::Strip(raw[pos]), ">")) { + U u = D(raw[pos]); + size_t i = 0; + while (i < u.size() && Utf8::IsSpace(u[i])) ++i; + size_t last = i; + while (i < u.size() && u[i] == '>') { ++i; if (i < u.size() && Utf8::IsSpace(u[i])) ++i; last = i; } + std::string inner = E(u.substr(last)); + std::string trimmed = Utf8::Strip(inner); + if (!trimmed.empty()) current.push_back(trimmed); + else if (!current.empty()) { paras.push_back(current); current.clear(); } + ++pos; + } + if (!current.empty()) paras.push_back(current); + std::string prefix = Style("\xe2\x94\x82 ", {Ansi::kDim}); + for (size_t index = 0; index < paras.size(); ++index) { + if (index) out.push_back(Style("\xe2\x94\x82", {Ansi::kDim})); + for (auto& w : AnsiText::Wrap(Inline(Utf8::Join(paras[index], " ")), width, prefix, prefix)) out.push_back(w); + } + out.push_back(""); + continue; + } + // lists + if (MatchList(line, false).ok || MatchList(line, true).ok) { + std::map counters; + while (pos < total) { + ListMatch b = MatchList(raw[pos], false), nu = MatchList(raw[pos], true); + if (!b.ok && !nu.ok) break; + long level; std::string marker, body; + if (b.ok) { + level = (long)b.indent.size() / 2; + marker = Bullets()[(size_t)level % Bullets().size()]; + body = b.body; + U bu = D(body); + if (bu.size() >= 4 && bu[0] == '[' && (bu[1] == ' ' || bu[1] == 'x' || bu[1] == 'X') && bu[2] == ']' && Utf8::IsSpace(bu[3])) { + size_t k = 3; + while (k < bu.size() && Utf8::IsSpace(bu[k])) ++k; + if (k < bu.size()) { + marker = (bu[1] == 'x' || bu[1] == 'X') ? "\xe2\x98\x91" : "\xe2\x98\x90"; // ☑ ☐ + body = E(bu.substr(k)); + } + } + } else { + level = (long)nu.indent.size() / 2; + counters[level] += 1; + for (auto it = counters.begin(); it != counters.end();) it = it->first > level ? counters.erase(it) : std::next(it); + marker = std::to_string(counters[level]) + "."; + body = nu.body; + } + std::string pad((size_t)(level * 2), ' '); + for (auto& w : AnsiText::Wrap(Inline(body), width, pad + marker + " ", pad + std::string(Utf8::Len(marker) + 1, ' '))) out.push_back(w); + ++pos; + } + out.push_back(""); + continue; + } + // paragraph + std::vector para; + while (pos < total && !Utf8::Strip(raw[pos]).empty()) { para.push_back(raw[pos]); ++pos; } + std::vector segments, current; + for (auto& chunk : para) { + current.push_back(Utf8::Strip(chunk)); + if (Utf8::EndsWith(chunk, " ")) { segments.push_back(Utf8::Join(current, " ")); current.clear(); } + } + if (!current.empty()) segments.push_back(Utf8::Join(current, " ")); + for (auto& seg : segments) + for (auto& w : AnsiText::Wrap(Inline(seg), width)) out.push_back(w); + out.push_back(""); + } + while (!out.empty() && out.back().empty()) out.pop_back(); + return Utf8::Join(out, "\n"); +} +} // namespace detz diff --git a/taicpp/src/util/markdown.h b/taicpp/src/util/markdown.h new file mode 100644 index 0000000..96ec495 --- /dev/null +++ b/taicpp/src/util/markdown.h @@ -0,0 +1,20 @@ +#pragma once +// Markdown: terminal markdown renderer, a literal port of tai.py md_inline, md_split_row, +// md_is_separator, md_align_cell, md_table_block and render_markdown. The regex rules of the +// Python source are matched by hand-written scanners with the same left-to-right, +// non-overlapping semantics as re.sub. +#include +#include +namespace detz { +class Markdown { + public: + static std::string Inline(const std::string& text); + static std::vector SplitRow(const std::string& line); + static bool IsSeparator(const std::string& line); + static std::string AlignCell(const std::string& cell, long width, const std::string& align); + static std::vector TableBlock(std::vector header, const std::vector& aligns, std::vector> rows); + // render_markdown(text, width=None, color=None): width < 0 means None, color -1 None / 0 False / 1 True. + static std::string Render(const std::string& text, long width = -1, int color = -1); + static const std::vector& Bullets(); // MD_BULLETS +}; +} // namespace detz diff --git a/taicpp/src/util/pyglob.cpp b/taicpp/src/util/pyglob.cpp new file mode 100644 index 0000000..e33c053 --- /dev/null +++ b/taicpp/src/util/pyglob.cpp @@ -0,0 +1,184 @@ +#include "pyglob.h" +#include "pypath.h" +#include "utf8.h" +#include +#include +#include +namespace detz { +namespace { +using U = std::u32string; + +// fnmatch matcher over code points: '*' any run, '?' any char, [seq] / [!seq] classes +bool ClassMatch(const U& pat, size_t i, size_t j, char32_t ch) { + // pat[i:j] is the class body (after '[' and before ']') + bool negate = false; + size_t s = i; + if (s < j && pat[s] == '!') { negate = true; ++s; } + if (!negate && s >= j) return false; // empty class never matches + if (negate && s >= j) return true; // [!] matches any char + bool hit = false; + size_t k = s; + while (k < j) { + if (k + 2 < j && pat[k + 1] == '-') { + char32_t lo = pat[k], hi = pat[k + 2]; + if (lo <= hi && ch >= lo && ch <= hi) hit = true; + k += 3; + } else { + if (pat[k] == ch) hit = true; + ++k; + } + } + return negate ? !hit : hit; +} + +bool Match(const U& pat, size_t p, const U& name, size_t n) { + while (p < pat.size()) { + char32_t c = pat[p]; + if (c == '*') { + while (p < pat.size() && pat[p] == '*') ++p; + if (p == pat.size()) return true; + for (size_t k = n; k <= name.size(); ++k) if (Match(pat, p, name, k)) return true; + return false; + } + if (c == '?') { if (n >= name.size()) return false; ++p; ++n; continue; } + if (c == '[') { + size_t i = p + 1, j = i; + if (j < pat.size() && pat[j] == '!') ++j; + if (j < pat.size() && pat[j] == ']') ++j; + while (j < pat.size() && pat[j] != ']') ++j; + if (j >= pat.size()) { // literal '[' + if (n >= name.size() || name[n] != '[') return false; + ++p; ++n; continue; + } + if (n >= name.size() || !ClassMatch(pat, i, j, name[n])) return false; + p = j + 1; ++n; continue; + } + if (n >= name.size() || name[n] != c) return false; + ++p; ++n; + } + return n == name.size(); +} + +std::string Join(const std::string& d, const std::string& b) { // glob._join + if (d.empty() || b.empty()) return d.empty() ? b : d; + return PyPath::Join(d, b); +} +void Split(const std::string& p, std::string* head, std::string* tail) { // os.path.split + size_t i = p.rfind('/'); + size_t cut = i == std::string::npos ? 0 : i + 1; + *head = p.substr(0, cut); + *tail = p.substr(cut); + if (!head->empty() && *head != std::string(head->size(), '/')) { + while (!head->empty() && head->back() == '/') head->pop_back(); + } +} +bool Lexists(const std::string& p) { struct stat st; return ::lstat(p.c_str(), &st) == 0; } + +std::vector ListDir(const std::string& dirname, bool dironly) { + std::vector out; + DIR* d = ::opendir(dirname.empty() ? "." : dirname.c_str()); + if (!d) return out; + while (struct dirent* e = ::readdir(d)) { + std::string n = e->d_name; + if (n == "." || n == "..") continue; + if (dironly) { + std::string full = (dirname.empty() ? "." : dirname) + "/" + n; + if (!PyPath::IsDir(full)) continue; + } + out.push_back(n); + } + ::closedir(d); + return out; +} +bool Hidden(const std::string& s) { return !s.empty() && s[0] == '.'; } + +void RList(const std::string& dirname, bool dironly, std::vector* out) { + for (auto& x : ListDir(dirname, dironly)) { + if (Hidden(x)) continue; + out->push_back(x); + std::string path = dirname.empty() ? x : Join(dirname, x); + std::vector sub; + RList(path, dironly, &sub); + for (auto& y : sub) out->push_back(Join(x, y)); + } +} + +std::vector Glob0(const std::string& dirname, const std::string& basename) { + if (!basename.empty()) { if (Lexists(Join(dirname, basename))) return {basename}; } + else if (PyPath::IsDir(dirname)) return {basename}; + return {}; +} +std::vector Glob1(const std::string& dirname, const std::string& pattern, bool dironly) { + std::vector out; + bool skipHidden = !Hidden(pattern); + for (auto& x : ListDir(dirname, dironly)) { + if (skipHidden && Hidden(x)) continue; + if (PyGlob::FnmatchCase(x, pattern)) out.push_back(x); + } + return out; +} +std::vector Glob2(const std::string& dirname, bool dironly) { + std::vector out; + if (dirname.empty() || PyPath::IsDir(dirname)) out.push_back(""); + RList(dirname, dironly, &out); + return out; +} +void IGlob(const std::string& pathname, bool dironly, std::vector* out) { + std::string dirname, basename; + Split(pathname, &dirname, &basename); + if (!PyGlob::HasMagic(pathname)) { + if (!basename.empty()) { if (Lexists(pathname)) out->push_back(pathname); } + else if (PyPath::IsDir(dirname)) out->push_back(pathname); + return; + } + if (dirname.empty()) { + auto names = basename == "**" ? Glob2("", dironly) : Glob1("", basename, dironly); + for (auto& n : names) out->push_back(n); + return; + } + std::vector dirs; + if (dirname != pathname && PyGlob::HasMagic(dirname)) IGlob(dirname, true, &dirs); + else dirs = {dirname}; + for (auto& d : dirs) { + std::vector names; + if (PyGlob::HasMagic(basename)) names = basename == "**" ? Glob2(d, dironly) : Glob1(d, basename, dironly); + else names = Glob0(d, basename); + for (auto& n : names) out->push_back(PyPath::Join(d, n)); + } +} +void WalkRec(const std::string& top, const std::function& visit, bool* stop) { + PyGlob::WalkEntry entry; + entry.root = top; + DIR* d = ::opendir(top.c_str()); + if (!d) return; + while (struct dirent* e = ::readdir(d)) { + std::string n = e->d_name; + if (n == "." || n == "..") continue; + if (PyPath::IsDir(PyPath::Join(top, n))) entry.dirs.push_back(n); else entry.files.push_back(n); + } + ::closedir(d); + if (!visit(entry)) { *stop = true; return; } + for (auto& dn : entry.dirs) { + std::string path = PyPath::Join(top, dn); + if (PyPath::IsLink(path)) continue; + WalkRec(path, visit, stop); + if (*stop) return; + } +} +} // namespace + +bool PyGlob::HasMagic(const std::string& s) { return s.find_first_of("*?[") != std::string::npos; } +bool PyGlob::FnmatchCase(const std::string& name, const std::string& pattern) { + return Match(Utf8::Decode(pattern), 0, Utf8::Decode(name), 0); +} +std::vector PyGlob::Glob(const std::string& pattern) { + std::vector out; + if (pattern.empty()) return out; + IGlob(pattern, false, &out); + return out; +} +void PyGlob::Walk(const std::string& top, const std::function& visit) { + bool stop = false; + WalkRec(top, visit, &stop); +} +} // namespace detz diff --git a/taicpp/src/util/pyglob.h b/taicpp/src/util/pyglob.h new file mode 100644 index 0000000..6c4318e --- /dev/null +++ b/taicpp/src/util/pyglob.h @@ -0,0 +1,18 @@ +#pragma once +// PyGlob: Python glob.glob(pattern, recursive=True), fnmatch.fnmatchcase and os.walk, posix only. +#include +#include +#include +namespace detz { +class PyGlob { + public: + static bool HasMagic(const std::string& s); // [*?[] present + static bool FnmatchCase(const std::string& name, const std::string& pattern); + // glob.glob(pattern, recursive=True), in generator order (callers sort) + static std::vector Glob(const std::string& pattern); + struct WalkEntry { std::string root; std::vector dirs, files; }; + // os.walk(top) topdown, followlinks=False, scandir order. The visitor returns false to stop + // (the generator is lazy in Python and callers break out early). + static void Walk(const std::string& top, const std::function& visit); +}; +} // namespace detz diff --git a/taicpp/src/util/pyhighlight.cpp b/taicpp/src/util/pyhighlight.cpp new file mode 100644 index 0000000..7c7793e --- /dev/null +++ b/taicpp/src/util/pyhighlight.cpp @@ -0,0 +1,126 @@ +#include "pyhighlight.h" +#include "ansi.h" +#include "utf8.h" +#include +namespace detz { +namespace { +using U = std::u32string; +enum class Kind { None, String, Comment, Number, Keyword, Decorator, DefName }; + +bool IsDigit(char32_t c) { return c >= '0' && c <= '9'; } +bool Word(const U& t, long i) { return i >= 0 && i < (long)t.size() && Utf8::IsWord(t[i]); } +bool Boundary(const U& t, long p) { return Word(t, p - 1) != Word(t, p); } +bool At(const U& t, size_t i, const char32_t* lit) { + for (size_t k = 0; lit[k]; ++k) if (i + k >= t.size() || t[i + k] != lit[k]) return false; + return true; +} + +// """ ... """ / ''' ... ''' (lazy, escapes `\.` do not cross a newline) +long Triple(const U& t, size_t s, const char32_t* q) { + if (!At(t, s, q)) return -1; + size_t i = s + 3; + while (true) { + if (At(t, i, q)) return (long)i + 3; + if (i >= t.size()) return -1; + if (t[i] == '\\') { if (i + 1 < t.size() && t[i + 1] != '\n') i += 2; else return -1; } + else ++i; + } +} +// "..." / '...' (single line) +long Simple(const U& t, size_t s, char32_t q) { + if (s >= t.size() || t[s] != q) return -1; + size_t i = s + 1; + while (i < t.size()) { + if (t[i] == '\\') { if (i + 1 < t.size() && t[i + 1] != '\n') i += 2; else break; } + else if (t[i] != q && t[i] != '\n') ++i; + else break; + } + return (i < t.size() && t[i] == q) ? (long)i + 1 : -1; +} +long Str(const U& t, size_t s) { + long e; + if ((e = Triple(t, s, U"\"\"\"")) >= 0) return e; + if ((e = Triple(t, s, U"'''")) >= 0) return e; + if ((e = Simple(t, s, '"')) >= 0) return e; + return Simple(t, s, '\''); +} +// \b\d[\d._]*(?:[eE][+-]?\d+)?[jJ]?\b with full backtracking +long Number(const U& t, size_t s) { + if (s >= t.size() || !IsDigit(t[s]) || !Boundary(t, (long)s)) return -1; + long kmax = 0; + while (s + 1 + kmax < t.size() && (IsDigit(t[s + 1 + kmax]) || t[s + 1 + kmax] == '.' || t[s + 1 + kmax] == '_')) ++kmax; + auto tail = [&](long p2) -> long { // [jJ]?\b + if (p2 < (long)t.size() && (t[p2] == 'j' || t[p2] == 'J') && Boundary(t, p2 + 1)) return p2 + 1; + if (Boundary(t, p2)) return p2; + return -1; + }; + for (long k = kmax; k >= 0; --k) { + long p1 = (long)s + 1 + k; + if (p1 < (long)t.size() && (t[p1] == 'e' || t[p1] == 'E')) { + for (int sign = 1; sign >= 0; --sign) { + long q = p1 + 1; + if (sign) { if (q < (long)t.size() && (t[q] == '+' || t[q] == '-')) ++q; else continue; } + long m = 0; + while (q + m < (long)t.size() && IsDigit(t[q + m])) ++m; + for (long d = m; d >= 1; --d) { long e = tail(q + d); if (e >= 0) return e; } + } + } + long e = tail(p1); + if (e >= 0) return e; + } + return -1; +} +bool Keyword(const U& t, size_t s, size_t* end) { + static const std::set kws = {"False", "None", "True", "and", "as", "assert", "async", "await", "break", "class", "continue", "def", "del", "elif", "else", "except", "finally", "for", "from", "global", "if", "import", "in", "is", "lambda", "nonlocal", "not", "or", "pass", "raise", "return", "try", "while", "with", "yield", "match", "case"}; + if (s >= t.size() || !Utf8::IsWord(t[s]) || !Boundary(t, (long)s)) return false; + size_t e = s; + while (e < t.size() && Utf8::IsWord(t[e])) ++e; + if (!kws.count(Utf8::Encode(t.substr(s, e - s)))) return false; + *end = e; + return true; +} +bool IdentStart(char32_t c) { return c == '_' || (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z'); } +} // namespace + +std::string PyHighlight::Line(const std::string& input) { + U t = Utf8::Decode(input); + std::string out; + size_t pos = 0, i = 0; + while (i < t.size()) { + Kind kind = Kind::None; + size_t end = 0; + long e; + if (t[i] == '"' || t[i] == '\'') { + if ((e = Str(t, i)) >= 0) { kind = Kind::String; end = (size_t)e; } + } + if (kind == Kind::None && t[i] == '#') { + end = i; while (end < t.size() && t[end] != '\n') ++end; + kind = Kind::Comment; + } + if (kind == Kind::None && (e = Number(t, i)) >= 0) { kind = Kind::Number; end = (size_t)e; } + if (kind == Kind::None && Keyword(t, i, &end)) kind = Kind::Keyword; + if (kind == Kind::None && t[i] == '@' && i + 1 < t.size() && IdentStart(t[i + 1])) { + end = i + 2; + while (end < t.size() && (Utf8::IsWord(t[end]) || t[end] == '.')) ++end; + kind = Kind::Decorator; + } + if (kind == Kind::None && IdentStart(t[i]) && ((i >= 4 && At(t, i - 4, U"def ")) || (i >= 6 && At(t, i - 6, U"class ")))) { + end = i + 1; + while (end < t.size() && Utf8::IsWord(t[end])) ++end; + kind = Kind::DefName; + } + if (kind == Kind::None) { ++i; continue; } + out += Utf8::Encode(t.substr(pos, i - pos)); + std::string text = Utf8::Encode(t.substr(i, end - i)); + const char* color = Ansi::kCyan; + if (kind == Kind::String) color = Ansi::kYellow; + else if (kind == Kind::Comment) color = Ansi::kGray; + else if (kind == Kind::Number) color = Palette::kFgOrange; + else if (kind == Kind::Keyword) color = Ansi::kMagenta; + out += std::string(color) + text + Ansi::kReset; + pos = i = end; + } + out += Utf8::Encode(t.substr(pos)); + return out; +} +} // namespace detz diff --git a/taicpp/src/util/pyhighlight.h b/taicpp/src/util/pyhighlight.h new file mode 100644 index 0000000..615171e --- /dev/null +++ b/taicpp/src/util/pyhighlight.h @@ -0,0 +1,10 @@ +#pragma once +// PyHighlight: highlight_python() of tai.py. PY_TOKEN_RE is matched by a hand-written scanner +// with the same alternation order (string, comment, number, keyword, decorator, defname). +#include +namespace detz { +class PyHighlight { + public: + static std::string Line(const std::string& line); +}; +} // namespace detz diff --git a/taicpp/src/util/pypath.cpp b/taicpp/src/util/pypath.cpp new file mode 100644 index 0000000..20e5749 --- /dev/null +++ b/taicpp/src/util/pypath.cpp @@ -0,0 +1,110 @@ +#include "pypath.h" +#include +#include +#include +#include +#include +#include +namespace detz { +std::string PyPath::Join(const std::string& a, const std::string& b) { + if (!b.empty() && b[0] == '/') return b; + std::string path = a; + if (path.empty() || path.back() == '/') path += b; else path += "/" + b; + return path; +} +std::string PyPath::Normpath(const std::string& p) { + if (p.empty()) return "."; + bool abs = p[0] == '/'; + int initial = 0; + if (abs) { initial = 1; if (p.size() >= 2 && p[1] == '/' && !(p.size() >= 3 && p[2] == '/')) initial = 2; } + std::vector comps, out; + size_t i = 0; + while (i <= p.size()) { + size_t j = p.find('/', i); + if (j == std::string::npos) j = p.size(); + comps.push_back(p.substr(i, j - i)); + i = j + 1; + } + for (auto& c : comps) { + if (c.empty() || c == ".") continue; + if (c != ".." || (!abs && out.empty()) || (!out.empty() && out.back() == "..")) out.push_back(c); + else if (!out.empty()) out.pop_back(); + } + std::string r; + for (size_t k = 0; k < out.size(); ++k) { if (k) r += "/"; r += out[k]; } + if (abs) r = std::string((size_t)initial, '/') + r; + return r.empty() ? "." : r; +} +std::string PyPath::Getcwd() { + char buf[PATH_MAX]; + return ::getcwd(buf, sizeof buf) ? std::string(buf) : std::string(); +} +std::string PyPath::Abspath(const std::string& p) { + return Normpath(IsAbs(p) ? p : Join(Getcwd(), p)); +} +std::string PyPath::Basename(const std::string& p) { + size_t i = p.rfind('/'); + return i == std::string::npos ? p : p.substr(i + 1); +} +std::string PyPath::Dirname(const std::string& p) { + size_t i = p.rfind('/'); + if (i == std::string::npos) return ""; + std::string head = p.substr(0, i + 1); + if (head != std::string(head.size(), '/')) { + while (!head.empty() && head.back() == '/') head.pop_back(); + } + return head; +} +std::string PyPath::Expanduser(const std::string& p) { + if (p.empty() || p[0] != '~') return p; + size_t i = p.find('/'); + if (i == std::string::npos) i = p.size(); + std::string user = p.substr(1, i - 1), home; + if (user.empty()) { + const char* h = std::getenv("HOME"); + if (h) home = h; + else { struct passwd* pw = ::getpwuid(::getuid()); if (!pw) return p; home = pw->pw_dir; } + } else { + struct passwd* pw = ::getpwnam(user.c_str()); + if (!pw) return p; + home = pw->pw_dir; + } + while (home.size() > 1 && home.back() == '/') home.pop_back(); + std::string rest = p.substr(i); + std::string r = home + rest; + return r.empty() ? "/" : r; +} +std::string PyPath::Realpath(const std::string& p) { + char buf[PATH_MAX]; + if (::realpath(p.c_str(), buf)) return buf; + // nonexistent tail: resolve the longest existing prefix, keep the rest (Python non-strict) + std::string abs = Abspath(p), tail; + while (abs != "/" && !abs.empty()) { + if (::realpath(abs.c_str(), buf)) { std::string r = buf; return tail.empty() ? r : Join(r, tail); } + tail = tail.empty() ? Basename(abs) : Basename(abs) + "/" + tail; + abs = Dirname(abs); + } + return Normpath(p); +} +bool PyPath::Exists(const std::string& p) { struct stat st; return ::stat(p.c_str(), &st) == 0; } +bool PyPath::IsFile(const std::string& p) { struct stat st; return ::stat(p.c_str(), &st) == 0 && S_ISREG(st.st_mode); } +bool PyPath::IsDir(const std::string& p) { struct stat st; return ::stat(p.c_str(), &st) == 0 && S_ISDIR(st.st_mode); } +bool PyPath::IsLink(const std::string& p) { struct stat st; return ::lstat(p.c_str(), &st) == 0 && S_ISLNK(st.st_mode); } +long PyPath::Getsize(const std::string& p) { struct stat st; return ::stat(p.c_str(), &st) == 0 ? (long)st.st_size : -1; } +std::vector PyPath::Listdir(const std::string& p, bool* ok) { + std::vector out; + DIR* d = ::opendir(p.c_str()); + if (!d) { if (ok) *ok = false; return out; } + while (struct dirent* e = ::readdir(d)) { + std::string n = e->d_name; + if (n != "." && n != "..") out.push_back(n); + } + ::closedir(d); + if (ok) *ok = true; + return out; +} +std::string PyPath::Getenv(const std::string& name, const std::string& dflt) { + const char* v = std::getenv(name.c_str()); + return v ? std::string(v) : dflt; +} +} // namespace detz diff --git a/taicpp/src/util/pypath.h b/taicpp/src/util/pypath.h new file mode 100644 index 0000000..4328017 --- /dev/null +++ b/taicpp/src/util/pypath.h @@ -0,0 +1,25 @@ +#pragma once +// PyPath: Python os.path semantics (lexical, posix) used by the shell helpers. +#include +#include +namespace detz { +class PyPath { + public: + static bool IsAbs(const std::string& p) { return !p.empty() && p[0] == '/'; } + static std::string Join(const std::string& a, const std::string& b); // os.path.join(a, b) + static std::string Normpath(const std::string& p); + static std::string Abspath(const std::string& p); + static std::string Basename(const std::string& p); + static std::string Dirname(const std::string& p); + static std::string Expanduser(const std::string& p); + static std::string Realpath(const std::string& p); + static std::string Getcwd(); + static bool Exists(const std::string& p); // follows symlinks + static bool IsFile(const std::string& p); + static bool IsDir(const std::string& p); + static bool IsLink(const std::string& p); + static long Getsize(const std::string& p); // -1 on error + static std::vector Listdir(const std::string& p, bool* ok); + static std::string Getenv(const std::string& name, const std::string& dflt = ""); +}; +} // namespace detz diff --git a/taicpp/src/util/sequencematcher.cpp b/taicpp/src/util/sequencematcher.cpp new file mode 100644 index 0000000..b5b1dda --- /dev/null +++ b/taicpp/src/util/sequencematcher.cpp @@ -0,0 +1,120 @@ +#include "sequencematcher.h" +#include +#include +#include +namespace detz { +SequenceMatcher::SequenceMatcher(std::vector a, std::vector b) : a_(std::move(a)), b_(std::move(b)) { + std::unordered_map ids; + bId_.reserve(b_.size()); + for (size_t j = 0; j < b_.size(); ++j) { + auto it = ids.find(b_[j]); + long id; + if (it == ids.end()) { id = (long)b2j_.size(); ids.emplace(b_[j], id); b2j_.emplace_back(); } + else id = it->second; + b2j_[id].push_back((long)j); + bId_.push_back(id); + } + aId_.reserve(a_.size()); + for (auto& line : a_) { + auto it = ids.find(line); + aId_.push_back(it == ids.end() ? -1 : it->second); + } +} + +MatchBlock SequenceMatcher::FindLongestMatch(long alo, long ahi, long blo, long bhi) const { + long besti = alo, bestj = blo, bestsize = 0; + std::unordered_map j2len, newj2len; + for (long i = alo; i < ahi; ++i) { + newj2len.clear(); + if (aId_[i] >= 0) { + for (long j : b2j_[aId_[i]]) { + if (j < blo) continue; + if (j >= bhi) break; + auto prev = j2len.find(j - 1); + long k = (prev == j2len.end() ? 0 : prev->second) + 1; + newj2len[j] = k; + if (k > bestsize) { besti = i - k + 1; bestj = j - k + 1; bestsize = k; } + } + } + j2len.swap(newj2len); + } + // extend over equal lines (no junk elements exist, so only the non-junk loops apply) + while (besti > alo && bestj > blo && a_[besti - 1] == b_[bestj - 1]) { --besti; --bestj; ++bestsize; } + while (besti + bestsize < ahi && bestj + bestsize < bhi && a_[besti + bestsize] == b_[bestj + bestsize]) ++bestsize; + return {besti, bestj, bestsize}; +} + +std::vector SequenceMatcher::MatchingBlocks() const { + long la = (long)a_.size(), lb = (long)b_.size(); + std::vector> queue = {{0, la, 0, lb}}; + std::vector blocks; + while (!queue.empty()) { + auto [alo, ahi, blo, bhi] = queue.back(); + queue.pop_back(); + MatchBlock x = FindLongestMatch(alo, ahi, blo, bhi); + if (x.size) { + blocks.push_back(x); + if (alo < x.a && blo < x.b) queue.emplace_back(alo, x.a, blo, x.b); + if (x.a + x.size < ahi && x.b + x.size < bhi) queue.emplace_back(x.a + x.size, ahi, x.b + x.size, bhi); + } + } + std::sort(blocks.begin(), blocks.end(), [](const MatchBlock& l, const MatchBlock& r) { + return std::tie(l.a, l.b, l.size) < std::tie(r.a, r.b, r.size); + }); + long i1 = 0, j1 = 0, k1 = 0; + std::vector merged; + for (auto& blk : blocks) { + if (i1 + k1 == blk.a && j1 + k1 == blk.b) k1 += blk.size; + else { + if (k1) merged.push_back({i1, j1, k1}); + i1 = blk.a; j1 = blk.b; k1 = blk.size; + } + } + if (k1) merged.push_back({i1, j1, k1}); + merged.push_back({la, lb, 0}); + return merged; +} + +std::vector SequenceMatcher::Opcodes() const { + long i = 0, j = 0; + std::vector answer; + for (auto& blk : MatchingBlocks()) { + std::string tag; + if (i < blk.a && j < blk.b) tag = "replace"; + else if (i < blk.a) tag = "delete"; + else if (j < blk.b) tag = "insert"; + if (!tag.empty()) answer.push_back({tag, i, blk.a, j, blk.b}); + i = blk.a + blk.size; j = blk.b + blk.size; + if (blk.size) answer.push_back({"equal", i - blk.size, i, j - blk.size, j}); + } + return answer; +} + +std::vector> SequenceMatcher::GroupedOpcodes(long n) const { + std::vector codes = Opcodes(); + if (codes.empty()) codes.push_back({"equal", 0, 1, 0, 1}); + if (codes.front().tag == "equal") { + Opcode& c = codes.front(); + c = {c.tag, std::max(c.i1, c.i2 - n), c.i2, std::max(c.j1, c.j2 - n), c.j2}; + } + if (codes.back().tag == "equal") { + Opcode& c = codes.back(); + c = {c.tag, c.i1, std::min(c.i2, c.i1 + n), c.j1, std::min(c.j2, c.j1 + n)}; + } + long nn = n + n; + std::vector> groups; + std::vector group; + for (auto c : codes) { + if (c.tag == "equal" && c.i2 - c.i1 > nn) { + group.push_back({c.tag, c.i1, std::min(c.i2, c.i1 + n), c.j1, std::min(c.j2, c.j1 + n)}); + groups.push_back(group); + group.clear(); + c.i1 = std::max(c.i1, c.i2 - n); + c.j1 = std::max(c.j1, c.j2 - n); + } + group.push_back(c); + } + if (!group.empty() && !(group.size() == 1 && group[0].tag == "equal")) groups.push_back(group); + return groups; +} +} // namespace detz diff --git a/taicpp/src/util/sequencematcher.h b/taicpp/src/util/sequencematcher.h new file mode 100644 index 0000000..2fdb64c --- /dev/null +++ b/taicpp/src/util/sequencematcher.h @@ -0,0 +1,21 @@ +#pragma once +// SequenceMatcher: Python difflib.SequenceMatcher(None, a, b, autojunk=False) over lists of +// lines (no junk heuristics), with get_matching_blocks, get_opcodes and get_grouped_opcodes. +#include +#include +namespace detz { +struct Opcode { std::string tag; long i1, i2, j1, j2; }; +struct MatchBlock { long a, b, size; }; +class SequenceMatcher { + public: + SequenceMatcher(std::vector a, std::vector b); + std::vector MatchingBlocks() const; + std::vector Opcodes() const; + std::vector> GroupedOpcodes(long n = 3) const; + private: + MatchBlock FindLongestMatch(long alo, long ahi, long blo, long bhi) const; + std::vector a_, b_; + std::vector> b2j_; // per distinct line id: ascending positions in b + std::vector aId_, bId_; // line id (or -1 when the line is not in b) +}; +} // namespace detz diff --git a/taicpp/src/util/shlex.cpp b/taicpp/src/util/shlex.cpp new file mode 100644 index 0000000..35c7a97 --- /dev/null +++ b/taicpp/src/util/shlex.cpp @@ -0,0 +1,48 @@ +#include "shlex.h" +namespace detz { +bool Shlex::Split(const std::string& s, std::vector* out) { + out->clear(); + size_t i = 0; + auto ws = [](char c) { return c == ' ' || c == '\t' || c == '\r' || c == '\n'; }; + while (true) { + // read_token(): state machine, whitespace_split=True, posix=True + char state = ' '; // ' ' between tokens, 'a' in a word, '\'' / '"' quoted, '\\' escape + char escapedstate = ' '; + bool quoted = false; + std::string token; + bool eof = false; + while (true) { + if (i >= s.size()) { + if (state == '\'' || state == '"') return false; // No closing quotation + if (state == '\\') return false; // No escaped character + eof = true; + break; + } + char c = s[i++]; + if (state == ' ') { + if (ws(c)) { if (!token.empty() || quoted) break; continue; } + if (c == '\\') { escapedstate = 'a'; state = c; } + else if (c == '\'' || c == '"') state = c; + else { token = std::string(1, c); state = 'a'; } + } else if (state == '\'' || state == '"') { + quoted = true; + if (c == state) state = 'a'; + else if (c == '\\' && state == '"') { escapedstate = state; state = c; } + else token += c; + } else if (state == '\\') { + if ((escapedstate == '\'' || escapedstate == '"') && c != state && c != escapedstate) token += state; + token += c; + state = escapedstate; + } else { // 'a' + if (ws(c)) { state = ' '; if (!token.empty() || quoted) break; continue; } + if (c == '\'' || c == '"') state = c; + else if (c == '\\') { escapedstate = 'a'; state = c; } + else token += c; + } + } + if (eof && token.empty() && !quoted) return true; + out->push_back(token); + if (eof) return true; + } +} +} // namespace detz diff --git a/taicpp/src/util/shlex.h b/taicpp/src/util/shlex.h new file mode 100644 index 0000000..39beb00 --- /dev/null +++ b/taicpp/src/util/shlex.h @@ -0,0 +1,11 @@ +#pragma once +// Shlex: Python shlex.split(text, posix=True) with comments disabled (the form tai.py uses). +#include +#include +namespace detz { +class Shlex { + public: + // false on ValueError ("No closing quotation" / "No escaped character") + static bool Split(const std::string& text, std::vector* out); +}; +} // namespace detz diff --git a/taicpp/src/util/text.cpp b/taicpp/src/util/text.cpp new file mode 100644 index 0000000..76e9dd5 --- /dev/null +++ b/taicpp/src/util/text.cpp @@ -0,0 +1,235 @@ +#include "text.h" +#include "pypath.h" +#include "utf8.h" +#include +#include +#include +#include +#include +namespace detz { + +std::string Truncate(const std::string& text, size_t head, size_t tail) { + std::u32string u = Utf8::Decode(text); + if (u.size() <= head + tail + 50) return text; + size_t skipped = u.size() - head - tail; + // text[-tail:] with tail == 0 is text[0:], the whole text (Python slicing quirk, kept) + std::u32string last = tail == 0 ? u : u.substr(u.size() - tail); + return Utf8::Encode(u.substr(0, head)) + "\n...[" + std::to_string(skipped) + " chars truncated]...\n" + Utf8::Encode(last); +} + +std::string TruncMarker(long trueBytes) { return "\n...[truncated, full size " + std::to_string(trueBytes) + " bytes]..."; } + +Capped CappedText(const std::optional& text) { + Capped r; + if (!text) return r; + r.present = true; + std::u32string u = Utf8::Decode(*text); + long bytes = (long)Utf8::Encode(u).size(); // text.encode("utf-8") after replacement decode + r.size = bytes; + if ((long)u.size() <= kAuditMaxChars) { r.text = Utf8::Encode(u); return r; } + r.text = Utf8::Encode(u.substr(0, kAuditMaxChars)) + TruncMarker(bytes); + return r; +} + +ReadCappedResult ReadCapped(const std::string& path, long cap) { + ReadCappedResult r; + long size = PyPath::Getsize(path); + if (size < 0) return r; + std::ifstream in(path, std::ios::binary); + if (!in) return r; + std::string raw((size_t)(cap + 1) * 4 + 8, '\0'); + in.read(raw.data(), (std::streamsize)raw.size()); + raw.resize((size_t)in.gcount()); + if (in.bad()) return r; + std::u32string u = Utf8::Decode(raw), t; // universal newlines: \r\n and \r become \n + for (size_t i = 0; i < u.size(); ++i) { + if (u[i] == '\r') { t.push_back('\n'); if (i + 1 < u.size() && u[i + 1] == '\n') ++i; } + else t.push_back(u[i]); + } + r.ok = true; + if ((long)t.size() > cap) { r.text = Utf8::Encode(t.substr(0, cap)); r.has_size = true; r.size = size; return r; } + r.text = Utf8::Encode(t); + return r; +} + +std::vector ChunkLines(const std::string& text, int maxLines, int overlapLines, int maxChars) { + std::vector lines = Utf8::SplitLines(text), chunks; + int n = (int)lines.size(); + if (n == 0) return chunks; + int start = 0; + while (start < n) { + int end = start, size = 0, count = 0; + while (end < n && count < maxLines && size < maxChars) { + size += (int)Utf8::Len(lines[end]) + 1; + count += 1; + end += 1; + } + std::string piece; + for (int i = start; i < end; ++i) { if (i > start) piece += "\n"; piece += lines[i]; } + piece = Utf8::Strip(piece); + if (!piece.empty()) chunks.push_back(piece); + if (end >= n) break; + start = std::max(start + 1, end - overlapLines); + } + return chunks; +} + +std::string CollapseRepeatedLines(const std::string& text, int threshold) { + std::vector lines = Utf8::SplitLines(text), out; + size_t n = lines.size(), i = 0; + while (i < n) { + size_t j = i; + while (j < n && lines[j] == lines[i]) ++j; + size_t run = j - i; + if ((long)run >= threshold) { + out.push_back(lines[i]); + out.push_back("... (repeated " + std::to_string(run - 1) + " more times) ..."); + } else { + for (size_t k = i; k < j; ++k) out.push_back(lines[k]); + } + i = j; + } + return Utf8::Join(out, "\n"); +} + +std::pair, std::string> SplitLeadingTags(const std::string& text) { + std::vector tags; + std::u32string rest = Utf8::Decode(text); + while (!rest.empty() && rest[0] == '@') { + size_t i = 1; + while (i < rest.size() && !Utf8::IsSpace(rest[i])) ++i; + if (i == 1) break; // @(\S+) needs at least one non-space char + tags.push_back(Utf8::Encode(rest.substr(1, i - 1))); + size_t j = i; + while (j < rest.size() && Utf8::IsSpace(rest[j])) ++j; // trailing \s* + rest = rest.substr(j); + } + return {tags, Utf8::Encode(rest)}; +} + +std::string ExpandFileMentions(const std::string& text, const std::string& directory) { + // FILE_MENTION_RE = (? 0 && std::string(",;:!?)").find(group[e - 1]) != std::string::npos) --e; + std::string core = group.substr(0, e), punct = group.substr(e); + std::string expanded = PyPath::Expanduser(core); + std::string path = PyPath::IsAbs(expanded) ? expanded : PyPath::Join(directory, expanded); + if (!core.empty() && PyPath::Exists(path)) out += Utf8::Decode(PyPath::Abspath(path) + punct); + else out += whole; + i = j; + } + return Utf8::Encode(out); +} + +std::vector CompleteFiles(const std::string& prefix, const std::string& directory, size_t cap) { + std::string base, part, sub; + if (prefix.find('/') != std::string::npos) { + size_t p = prefix.rfind('/'); + sub = prefix.substr(0, p); + part = prefix.substr(p + 1); + base = PyPath::Expanduser(PyPath::Join(directory, sub)); + } else { + base = PyPath::Expanduser(directory); + part = prefix; + } + bool ok = false; + std::vector entries = PyPath::Listdir(base, &ok), found; + if (!ok) return found; + std::sort(entries.begin(), entries.end()); + std::string lowpart = Utf8::Lower(part); + for (auto& entry : entries) { + if (!entry.empty() && entry[0] == '.' && !(part.size() && part[0] == '.')) continue; + if (!Utf8::StartsWith(Utf8::Lower(entry), lowpart)) continue; + std::string rel = prefix.find('/') == std::string::npos ? entry : sub + "/" + entry; + if (PyPath::IsDir(PyPath::Join(base, entry))) rel += "/"; + found.push_back(rel); + if (found.size() >= cap) break; + } + return found; +} + +std::string HumanSize(long num) { + if (num < 0) num = 0; + char buf[64]; + if (num < 1024) { std::snprintf(buf, sizeof buf, "%ld B", num); return buf; } + if (num < 1048576) { std::snprintf(buf, sizeof buf, "%.1f KB", (double)num / 1024); return buf; } + std::snprintf(buf, sizeof buf, "%.1f MB", (double)num / 1048576); + return buf; +} + +long EstimateTokens(const std::string& text) { return std::max(1, (long)Utf8::Len(text) / 4); } + +long AsInt(const std::string& value, long dflt) { + std::string s = Utf8::Strip(value); + if (s.empty()) return dflt; + size_t i = 0; bool neg = false; + if (s[i] == '+' || s[i] == '-') { neg = s[i] == '-'; ++i; } + if (i >= s.size()) return dflt; + long v = 0; bool lastDigit = false; + for (; i < s.size(); ++i) { + char c = s[i]; + if (c >= '0' && c <= '9') { v = v * 10 + (c - '0'); lastDigit = true; } + else if (c == '_' && lastDigit && i + 1 < s.size() && s[i + 1] >= '0' && s[i + 1] <= '9') lastDigit = false; + else return dflt; + } + return neg ? -v : v; +} + +std::string ShortError(const std::string& message, const std::string& typeName) { + return Utf8::Head(message.empty() ? typeName : message, 300); +} + +std::string NowIso() { + std::time_t t = std::time(nullptr); + std::tm tm{}; + gmtime_r(&t, &tm); + char buf[40]; + std::strftime(buf, sizeof buf, "%Y-%m-%dT%H:%M:%S+00:00", &tm); + return buf; +} + +std::string ToolNameClean(const std::string& rawName) { + size_t p = rawName.find("<|"); + return Utf8::Strip(p == std::string::npos ? rawName : rawName.substr(0, p)); +} + +long ParseDuration(const std::string& raw) { + std::string s = Utf8::Lower(Utf8::Strip(raw)); + size_t i = 0; + while (i < s.size() && s[i] >= '0' && s[i] <= '9') ++i; + if (i == 0) return -1; + long n = std::stol(s.substr(0, i)); + while (i < s.size() && Utf8::IsSpace((unsigned char)s[i])) ++i; + long mult = 1; + if (i < s.size()) { + if (i + 1 != s.size()) return -1; + switch (s[i]) { + case 's': mult = 1; break; + case 'm': mult = 60; break; + case 'h': mult = 3600; break; + case 'd': mult = 86400; break; + case 'w': mult = 604800; break; + default: return -1; + } + } + return n * mult; +} + +std::string FormatDelay(long seconds) { + if (seconds < 0) return "overdue"; + if (seconds < 60) return std::to_string(seconds) + "s"; + if (seconds < 3600) return std::to_string(seconds / 60) + "m"; + if (seconds < 86400) return std::to_string(seconds / 3600) + "h"; + return std::to_string(seconds / 86400) + "d"; +} + +} // namespace detz diff --git a/taicpp/src/util/text.h b/taicpp/src/util/text.h new file mode 100644 index 0000000..101583a --- /dev/null +++ b/taicpp/src/util/text.h @@ -0,0 +1,43 @@ +#pragma once +// Text helpers ported 1:1 from tai.py (truncate, capped_text, trunc_marker, read_capped, +// chunk_lines, collapse_repeated_lines, split_leading_tags, expand_file_mentions, +// complete_files, human_size, estimate_tokens, as_int, short_error, now_iso, +// tool_name_clean, parse_duration, format_delay). Lengths and slices are code points. +#include +#include +#include +#include +namespace detz { +constexpr long kAuditMaxChars = 20000; // AUDIT_MAX_CHARS +constexpr int kChunkMaxLines = 60; // CHUNK_MAX_LINES +constexpr int kChunkOverlapLines = 3; // CHUNK_OVERLAP_LINES +constexpr int kChunkMaxChars = 4000; // CHUNK_MAX_CHARS + +std::string Truncate(const std::string& text, size_t head = 1500, size_t tail = 500); + +// capped_text(): (None, None) for None, else (text or capped text, true utf-8 size). +struct Capped { bool present{false}; std::string text; long size{0}; }; +Capped CappedText(const std::optional& text); +std::string TruncMarker(long trueBytes); + +// read_capped(): (None, None) on error; (text, None) when it fits; (head, size) when cut. +struct ReadCappedResult { bool ok{false}; std::string text; bool has_size{false}; long size{0}; }; +ReadCappedResult ReadCapped(const std::string& path, long cap = kAuditMaxChars); + +std::vector ChunkLines(const std::string& text, int maxLines = kChunkMaxLines, int overlapLines = kChunkOverlapLines, int maxChars = kChunkMaxChars); +std::string CollapseRepeatedLines(const std::string& text, int threshold = 3); +std::pair, std::string> SplitLeadingTags(const std::string& text); +std::string ExpandFileMentions(const std::string& text, const std::string& directory); +std::vector CompleteFiles(const std::string& prefix, const std::string& directory, size_t cap = 100); + +std::string HumanSize(long num); +long EstimateTokens(const std::string& text); +// as_int(): int(value) or default. Accepts what Python int(str) accepts (sign, underscores, spaces). +long AsInt(const std::string& value, long dflt = 0); +std::string ShortError(const std::string& message, const std::string& typeName); +std::string NowIso(); +std::string ToolNameClean(const std::string& rawName); +// parse_duration(): seconds, or -1 for None. +long ParseDuration(const std::string& raw); +std::string FormatDelay(long seconds); +} // namespace detz diff --git a/taicpp/src/util/toolargs.cpp b/taicpp/src/util/toolargs.cpp new file mode 100644 index 0000000..850359f --- /dev/null +++ b/taicpp/src/util/toolargs.cpp @@ -0,0 +1,21 @@ +#include "toolargs.h" +#include "utf8.h" +namespace detz { +std::string ToolArgs::Text(const std::string& raw) { return raw.empty() ? "{}" : raw; } +std::string ToolArgs::Text(const JsonValue& raw) { + if (!raw.Truthy()) return "{}"; + if (raw.is_string()) return raw.string_value(); + return raw.Dump(false); +} +std::string ToolArgs::Valid(const std::string& arguments) { + bool ok = false; + JsonValue::Parse(arguments, &ok); + return ok ? arguments : "{}"; +} +std::string ToolArgs::Signature(const std::string& name, const std::string& rawArguments) { + bool ok = false; + JsonValue v = JsonValue::Parse(Text(rawArguments), &ok); + std::string flat = ok ? v.Dump(true) : rawArguments; + return name + " " + Utf8::Head(flat, 500); +} +} // namespace detz diff --git a/taicpp/src/util/toolargs.h b/taicpp/src/util/toolargs.h new file mode 100644 index 0000000..c6d9639 --- /dev/null +++ b/taicpp/src/util/toolargs.h @@ -0,0 +1,16 @@ +#pragma once +// ToolArgs: tool_args_text, tool_args_valid and action_signature of tai.py. +#include "jsonvalue.h" +#include +namespace detz { +class ToolArgs { + public: + // tool_args_text(raw): "{}" for falsy, the text for a str, json.dumps otherwise. + static std::string Text(const std::string& raw); + static std::string Text(const JsonValue& raw); + // tool_args_valid(arguments): arguments when json.loads accepts it, else "{}". + static std::string Valid(const std::string& arguments); + // action_signature(name, raw_arguments) + static std::string Signature(const std::string& name, const std::string& rawArguments); +}; +} // namespace detz diff --git a/taicpp/src/util/utf8.cpp b/taicpp/src/util/utf8.cpp new file mode 100644 index 0000000..270ae7d --- /dev/null +++ b/taicpp/src/util/utf8.cpp @@ -0,0 +1,140 @@ +#include "utf8.h" +namespace detz { +std::u32string Utf8::Decode(const std::string& s) { + std::u32string out; + size_t i = 0, n = s.size(); + while (i < n) { + unsigned char c = s[i]; + if (c < 0x80) { out.push_back(c); ++i; continue; } + int need = 0; char32_t cp = 0; unsigned char lo = 0x80, hi = 0xBF; + if (c >= 0xC2 && c <= 0xDF) { need = 1; cp = c & 0x1F; } + else if (c >= 0xE0 && c <= 0xEF) { need = 2; cp = c & 0x0F; if (c == 0xE0) lo = 0xA0; if (c == 0xED) hi = 0x9F; } + else if (c >= 0xF0 && c <= 0xF4) { need = 3; cp = c & 0x07; if (c == 0xF0) lo = 0x90; if (c == 0xF4) hi = 0x8F; } + else { out.push_back(0xFFFD); ++i; continue; } + size_t j = i + 1; int got = 0; + while (got < need && j < n) { + unsigned char d = s[j]; + unsigned char l = got == 0 ? lo : 0x80, h = got == 0 ? hi : 0xBF; + if (d < l || d > h) break; + cp = (cp << 6) | (d & 0x3F); ++j; ++got; + } + out.push_back(got == need ? cp : (char32_t)0xFFFD); + i = j; + } + return out; +} +std::string Utf8::EncodeOne(char32_t c) { + std::string o; + if (c < 0x80) o.push_back((char)c); + else if (c < 0x800) { o.push_back((char)(0xC0 | (c >> 6))); o.push_back((char)(0x80 | (c & 0x3F))); } + else if (c < 0x10000) { o.push_back((char)(0xE0 | (c >> 12))); o.push_back((char)(0x80 | ((c >> 6) & 0x3F))); o.push_back((char)(0x80 | (c & 0x3F))); } + else { o.push_back((char)(0xF0 | (c >> 18))); o.push_back((char)(0x80 | ((c >> 12) & 0x3F))); o.push_back((char)(0x80 | ((c >> 6) & 0x3F))); o.push_back((char)(0x80 | (c & 0x3F))); } + return o; +} +std::string Utf8::Encode(const std::u32string& s) { std::string o; for (char32_t c : s) o += EncodeOne(c); return o; } +size_t Utf8::Len(const std::string& s) { + for (unsigned char c : s) if (c >= 0x80) return Decode(s).size(); + return s.size(); +} +std::string Utf8::Slice(const std::string& s, long start, long end) { + std::u32string u = Decode(s); long n = (long)u.size(); + if (start < 0) { start += n; if (start < 0) start = 0; } else if (start > n) start = n; + if (end < 0) { end += n; if (end < 0) end = 0; } else if (end > n) end = n; + if (end <= start) return ""; + return Encode(u.substr(start, end - start)); +} +std::string Utf8::Head(const std::string& s, long n) { return Slice(s, 0, n); } +std::string Utf8::Tail(const std::string& s, long n) { return Slice(s, -n, 0x7fffffffL); } +bool Utf8::IsSpace(char32_t c) { + return (c >= 9 && c <= 13) || (c >= 0x1c && c <= 0x20) || c == 0x85 || c == 0xA0 || c == 0x1680 || + (c >= 0x2000 && c <= 0x200A) || c == 0x2028 || c == 0x2029 || c == 0x202F || c == 0x205F || c == 0x3000; +} +bool Utf8::IsAlnum(char32_t c) { + if (c < 0x80) return (c >= '0' && c <= '9') || (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z'); + if (c < 0xC0) return c == 0xAA || c == 0xB5 || c == 0xBA || c == 0xB2 || c == 0xB3 || c == 0xB9 || (c >= 0xBC && c <= 0xBE); + if (c == 0xD7 || c == 0xF7) return false; + if (c < 0x2B9) return true; + if (c < 0x370) return false; + if (c < 0x2000) return true; + if (c < 0x2C00) return false; + if (c >= 0x3000 && c <= 0x303F) return false; + if (c >= 0xFE00 && c <= 0xFE6F) return false; + if (c >= 0xFF00 && c <= 0xFF0F) return false; + if (c >= 0xFFF0 && c <= 0xFFFF) return false; + if (c >= 0x1F000 && c <= 0x1FAFF) return false; + return true; +} +bool Utf8::IsWord(char32_t c) { return c == '_' || IsAlnum(c); } +std::string Utf8::Lstrip(const std::string& s) { + std::u32string u = Decode(s); size_t a = 0; + while (a < u.size() && IsSpace(u[a])) ++a; + return Encode(u.substr(a)); +} +std::string Utf8::Rstrip(const std::string& s) { + std::u32string u = Decode(s); size_t b = u.size(); + while (b > 0 && IsSpace(u[b - 1])) --b; + return Encode(u.substr(0, b)); +} +std::string Utf8::Strip(const std::string& s) { return Rstrip(Lstrip(s)); } +std::vector Utf8::SplitLines(const std::string& s) { + std::u32string u = Decode(s); std::vector out; std::u32string cur; size_t i = 0; + while (i < u.size()) { + char32_t c = u[i]; + bool brk = c == '\n' || c == '\r' || c == 0x0B || c == 0x0C || c == 0x1C || c == 0x1D || c == 0x1E || c == 0x85 || c == 0x2028 || c == 0x2029; + if (!brk) { cur.push_back(c); ++i; continue; } + out.push_back(Encode(cur)); cur.clear(); + if (c == '\r' && i + 1 < u.size() && u[i + 1] == '\n') ++i; + ++i; + } + if (!cur.empty()) out.push_back(Encode(cur)); + return out; +} +std::vector Utf8::SplitWhitespace(const std::string& s) { + std::u32string u = Decode(s); std::vector out; std::u32string cur; + for (char32_t c : u) { + if (IsSpace(c)) { if (!cur.empty()) { out.push_back(Encode(cur)); cur.clear(); } } + else cur.push_back(c); + } + if (!cur.empty()) out.push_back(Encode(cur)); + return out; +} +std::string Utf8::ExpandTabs(const std::string& s, int tabsize) { + std::u32string u = Decode(s), o; long col = 0; + for (char32_t c : u) { + if (c == '\t') { + if (tabsize > 0) { long add = tabsize - (col % tabsize); o.append(add, U' '); col += add; } + } else { + o.push_back(c); + if (c == '\n' || c == '\r') col = 0; else ++col; + } + } + return Encode(o); +} +std::string Utf8::RJust(const std::string& s, size_t width) { + size_t n = Len(s); + return n >= width ? s : std::string(width - n, ' ') + s; +} +std::string Utf8::Lower(const std::string& s) { + std::u32string u = Decode(s); + for (auto& c : u) { + if (c >= 'A' && c <= 'Z') c += 32; + else if (c >= 0xC0 && c <= 0xDE && c != 0xD7) c += 32; + else if (c >= 0x391 && c <= 0x3A9 && c != 0x3A2) c += 32; + else if (c >= 0x410 && c <= 0x42F) c += 32; + else if (c >= 0x400 && c <= 0x40F) c += 80; + } + return Encode(u); +} +std::string Utf8::Join(const std::vector& parts, const std::string& sep) { + std::string o; + for (size_t i = 0; i < parts.size(); ++i) { if (i) o += sep; o += parts[i]; } + return o; +} +std::string Utf8::ReplaceAll(const std::string& s, const std::string& from, const std::string& to) { + if (from.empty()) return s; + std::string o; size_t pos = 0, f; + while ((f = s.find(from, pos)) != std::string::npos) { o.append(s, pos, f - pos); o += to; pos = f + from.size(); } + o.append(s, pos, std::string::npos); + return o; +} +} // namespace detz diff --git a/taicpp/src/util/utf8.h b/taicpp/src/util/utf8.h new file mode 100644 index 0000000..b1d255a --- /dev/null +++ b/taicpp/src/util/utf8.h @@ -0,0 +1,33 @@ +#pragma once +// Utf8: Python str semantics (code point indexing, str.strip/split/splitlines/expandtabs) +// over UTF-8 std::string. Invalid bytes decode to U+FFFD like errors="replace". +#include +#include +namespace detz { +class Utf8 { + public: + static std::u32string Decode(const std::string& s); + static std::string Encode(const std::u32string& s); + static std::string EncodeOne(char32_t c); + static size_t Len(const std::string& s); + // s[start:end] with Python slice clamping; negative indexes count from the end. + static std::string Slice(const std::string& s, long start, long end); + static std::string Head(const std::string& s, long n); // s[:n] + static std::string Tail(const std::string& s, long n); // s[-n:] (n > 0) + static bool IsSpace(char32_t c); // str.isspace + static bool IsWord(char32_t c); // re \w (unicode, approximate) + static bool IsAlnum(char32_t c); + static std::string Strip(const std::string& s); + static std::string Lstrip(const std::string& s); + static std::string Rstrip(const std::string& s); + static std::vector SplitLines(const std::string& s); // str.splitlines() + static std::vector SplitWhitespace(const std::string& s); // str.split() + static std::string ExpandTabs(const std::string& s, int tabsize = 8); + static std::string RJust(const std::string& s, size_t width); + static std::string Lower(const std::string& s); + static std::string Join(const std::vector& parts, const std::string& sep); + static std::string ReplaceAll(const std::string& s, const std::string& from, const std::string& to); + static bool StartsWith(const std::string& s, const std::string& p) { return s.compare(0, p.size(), p) == 0; } + static bool EndsWith(const std::string& s, const std::string& p) { return s.size() >= p.size() && s.compare(s.size() - p.size(), p.size(), p) == 0; } +}; +} // namespace detz diff --git a/taicpp/src/util/util.mk b/taicpp/src/util/util.mk new file mode 100644 index 0000000..91d7c13 --- /dev/null +++ b/taicpp/src/util/util.mk @@ -0,0 +1,2 @@ +UTIL_SRCS = src/util/ansi.cpp src/util/diffrender.cpp src/util/digest.cpp src/util/http.cpp src/util/jsonvalue.cpp src/util/markdown.cpp src/util/pyglob.cpp src/util/pyhighlight.cpp src/util/pypath.cpp src/util/sequencematcher.cpp src/util/shlex.cpp src/util/text.cpp src/util/toolargs.cpp src/util/utf8.cpp +CONFIG_SRCS = src/config/config.cpp diff --git a/taicpp/tests/shell_parity.py b/taicpp/tests/shell_parity.py new file mode 100644 index 0000000..2e0b4e6 --- /dev/null +++ b/taicpp/tests/shell_parity.py @@ -0,0 +1,1155 @@ +#!/usr/bin/env python3 +"""Differential test of the C++ shell/util port against ../tai.py. + +Builds build/shell_parity_driver from src/shell + src/util, runs every case through the driver +and through the Python function of tai.py, and compares the bytes. Real processes, a real pty, +real files: no mocks. Outputs go under build/ only. + + python3 tests/shell_parity.py [--filter NAME] [--fuzz N] [--seed S] [--no-build] [-v] +""" +import argparse +import codecs +import hashlib +import importlib.util +import io +import json +import os +import random +import re +import shutil +import subprocess +import sys +import tempfile +import time + +HERE = os.path.dirname(os.path.abspath(__file__)) +ROOT = os.path.dirname(HERE) +BUILD = os.path.join(ROOT, "build") +DRIVER = os.path.join(BUILD, "shell_parity_driver") +PY_FILE = os.path.join(os.path.dirname(ROOT), "tai.py") + +spec = importlib.util.spec_from_file_location("tai", PY_FILE) +tai = importlib.util.module_from_spec(spec) +sys.modules["tai"] = tai +spec.loader.exec_module(tai) + +RS, US = "\x1e", "\x1f" +NONE = "\x01NONE" + + +def build(): + os.makedirs(BUILD, exist_ok=True) + srcs = [] + for sub in ("src/util", "src/shell"): + for name in sorted(os.listdir(os.path.join(ROOT, sub))): + if name.endswith(".cpp") and name not in ("http.cpp",): + srcs.append(os.path.join(ROOT, sub, name)) + srcs.append(os.path.join(HERE, "shell_parity_driver.cpp")) + cmd = ["g++", "-std=c++23", "-Wall", "-Wextra", "-O1", "-I" + os.path.join(ROOT, "src/util"), "-I" + os.path.join(ROOT, "src/shell"), "-pthread"] + srcs + ["-lcurl", "-o", DRIVER] + done = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True) + if done.returncode != 0 or done.stdout.strip(): + print(done.stdout) + if done.returncode != 0: + sys.exit("driver build failed") + sys.exit("driver build produced warnings: the port must be warning-free") + + +def cpp(*args, timeout=60): + argv, spill = [], [] + for item in args: + item = str(item) + if len(item.encode("utf-8")) > 100000: # beyond the kernel's per-argument limit: pass by file + handle = tempfile.NamedTemporaryFile(dir=BUILD, delete=False) + handle.write(item.encode("utf-8")) + handle.close() + spill.append(handle.name) + item = "@@FILE:" + handle.name + argv.append(item) + try: + done = subprocess.run([DRIVER] + argv, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=timeout) + finally: + for name in spill: + os.unlink(name) + return done.stdout + + +def b(text): + return text.encode("utf-8", "replace") if isinstance(text, str) else text + + +# ---------- python side, one function per driver verb ---------- +def py_capped(text, flag): + out, size = tai.capped_text(None if flag == "none" else text) + return ("0" if out is None else "1") + RS + str(0 if size is None else size) + RS + (out or "") + + +def py_describe(name, raw): + verb, target, extra, preview = tai.Glow.describe_call(name, raw) + return "\n".join(json.dumps(x) for x in (verb, target, extra, preview)) + + +def py_json(text): + try: + value = json.loads(text) + except ValueError: + return "ERR" + return "\n".join((json.dumps(value), json.dumps(value, sort_keys=True), str(value), repr(value))) + + +def py_tool_args(text): + return "\n".join((tai.tool_args_text(text), tai.tool_args_valid(text), tai.action_signature("name", text))) + + +def py_sess_stat(text): + parsed = tai.ShellSession.parse_stat(text) + if parsed is None: + return "None" + return "%s %d %d" % (json.dumps(parsed[0]), parsed[1], parsed[2]) + + +def py_sess_current(env_value, pid, proc_root, now, clk): + env = {"TAI_SESSION": env_value} if env_value else {} + sess = tai.ShellSession.current(env=env, pid=int(pid), proc_root=proc_root, now=float(now), clk_tck=int(clk)) + chain = "".join("%d,%d,%s,%s,%s,%s,%s;" % (e["pid"], e["ppid"], e["comm"], e["argv0"], e["name"], e["start_iso"], e["sid"]) for e in sess.chain) + return "\n".join((json.dumps(sess.sid), json.dumps(sess.shell), str(sess.pid), json.dumps(sess.started), json.dumps(sess.parent) if sess.parent is not None else "null", chain)) + + +def py_live(height, width, status, *lines): + sink = io.StringIO() + stream = tai.LiveStream(int(height), int(width), sink, True) + for line in lines: + if line == "\x01PAUSE": + stream.pause() + else: + stream.feed(line) + stream.close(status) + return sink.getvalue() + + +class Recorder(object): + def __init__(self): + self.log = [] + + def feed(self, line): + self.log.append("FEED " + json.dumps(line) + "\n") + + def pause(self): + self.log.append("PAUSE\n") + + +def py_run(timeout, mode, ask, *rest): + rest = list(rest) + dash = rest.index("--") + answers, cmd = rest[:dash], rest[dash + 1:] + rec = Recorder() + nxt = [0] + + def answer(text, secret): + rec.log.append("ASK %s %s\n" % (json.dumps(text), "secret" if secret else "plain")) + if nxt[0] >= len(answers): + return None + value = answers[nxt[0]] + nxt[0] += 1 + if value.startswith("\x01SLEEP"): + delay, _, value = value[6:].partition(":") + time.sleep(int(delay) / 1000.0) + return None if value == "\x01DECLINE" else value + + old = tai.SHELL_TIMEOUT + try: + if mode == "shell": + result = tai.run_live(cmd[0], timeout=float(timeout), shell=True, stream=rec, ask=answer if ask == "1" else None) + else: + result = tai.run_live(cmd, timeout=float(timeout), shell=False, stream=rec, ask=answer if ask == "1" else None) + except OSError as exc: + rec.log.append("ERROR " + json.dumps(str(exc)) + "\n") + return "".join(rec.log) + code, output, _elapsed, timed_out = result + rec.log.append("RESULT %d %s %s\n" % (code, "timeout" if timed_out else "ok", json.dumps(output))) + return "".join(rec.log) + + +def py_lane(seq, width, cap): + lists = json.loads(seq) + calls = [0] + real = tai.list_agents + + def fake(profile): + value = lists[min(calls[0], len(lists) - 1)] + calls[0] += 1 + return value + + tai.list_agents = fake + try: + sink = io.StringIO() + board = tai.LaneBoard("prof", int(width), int(cap), sink) + board.start() + board.stop() + return sink.getvalue() + "\npainted=%d" % board.painted + finally: + tai.list_agents = real + + +PY = { + "truncate": lambda t, h, n: tai.truncate(t, int(h), int(n)), + "capped": py_capped, + "chunk": lambda t, a, c, d: RS.join(tai.chunk_lines(t, int(a), int(c), int(d))), + "collapse": lambda t, n: tai.collapse_repeated_lines(t, int(n)), + "tags": lambda t: US.join(tai.split_leading_tags(t)[0]) + RS + tai.split_leading_tags(t)[1], + "mentions": lambda t, d: tai.expand_file_mentions(t, d), + "complete": lambda p, d, c: "\n".join(tai.complete_files(p, d, int(c))), + "human_size": lambda n: tai.human_size(int(n)), + "stream_status": lambda c, o, e, t: tai.stream_status(int(c), o, float(e), t == "1"), + "duration": lambda r: str(tai.parse_duration(r)), + "delay": lambda n: tai.format_delay(int(n)), + "tokens": lambda t: str(tai.estimate_tokens(t)), + "as_int": lambda v, d: str(tai.as_int(v, int(d))), + "toolname": lambda t: tai.tool_name_clean(t), + "target_paths": lambda c, w: "\n".join(tai.shell_target_paths(c, w)), + "segment": lambda *t: "\n".join(tai.segment_targets(list(t))), + "curl_hint": lambda c: tai.curl_hint(c), + "strip_ansi": lambda t: tai.strip_ansi(t), + "wrap": lambda t, w, f, n: "\n".join(tai.wrap_ansi(t, int(w), f, n)), + "trunc_ansi": lambda t, w: tai.truncate_ansi(t, int(w)), + "pad_tinted": lambda t, w: tai.pad_tinted(t, int(w)), + "style_span": lambda t, c1, c2: tai.style_span(t, c1, c2), + "md_inline": lambda t: tai.md_inline(t), + "markdown": lambda t, w: tai.render_markdown(t, int(w), True), + "diff": lambda p, o, n, w, c: tai.render_diff(p, None if o == NONE else o, None if n == NONE else n, int(w), c == "1"), + "highlight": lambda t: tai.highlight_python(t), + "g_event": lambda i, v, t, m, w, c: tai.Glow.event_row(i, v, t, m, int(w), c), + "g_done": lambda v, t, c, e, f, w: tai.Glow.done_row(v, t, int(c), float(e), f == "1", int(w)), + "g_ellipsis": lambda t, c, h: tai.Glow.ellipsis(t, int(c), h == "1"), + "g_shimmer": lambda t, p, s: tai.Glow.shimmer(t, int(p), int(s)), + "g_describe": py_describe, + "g_status": lambda v, p, d, y, a, w: tai.Glow.statusbar(v, p, d, y == "1", int(a), int(w)), + "g_accent": lambda n: "%s|%s|%s" % (tai.Glow.accent_for(n), tai.Glow.bar(n), tai.Glow.icon_for(n)), + "g_hair": lambda w: tai.Glow.hairline(int(w)), + "g_prompt": lambda: tai.Glow.prompt_text(), + "g_preview": lambda t, w: tai.Glow.preview_row(t, int(w)), + "g_agent_row": lambda r, w, p, s: tai.Glow.agent_row(json.loads(r), int(w), p, s), + "g_lanes": lambda r, c, w, p: "\n".join(tai.Glow.lanes(json.loads(r), int(c), int(w), p)), + "json": py_json, + "tool_args": py_tool_args, + "md5": lambda t: hashlib.md5(t.encode("utf-8")).hexdigest(), + "sess_stat": py_sess_stat, + "sess_name": lambda a, c: tai.ShellSession.shell_name(a, c), + "sess_boot": lambda r: tai.ShellSession.boot_tag(r), + "sess_current": py_sess_current, + "live": py_live, + "run": py_run, + "lane": py_lane, +} + +CASES = [] # (label, func, args) + + +def case(func, *args, label=None, env=None): + CASES.append((label or func, func, tuple(str(a) for a in args), env)) + + +def corpus(): + texts = [ + "", "a", "plain text", "line one\nline two\n", "tab\there\tx", "unicode: \u00e9\u00e8 \u4e2d\u6587 \U0001f600 end", + "x" * 3000, "y" * 40 + "\n" + "z" * 5000, "crlf\r\nline\rcr only\n", "nbsp\u00a0space\u2003em", + "\x1b[31mred\x1b[0m plain \x1b[1;4mbold\x1b[0m tail", + "a\n\n\nb\n\n", "same\nsame\nsame\nsame\nother\nsame\nsame\n", + ] + for t in texts: + for head, tail in ((1500, 500), (10, 5), (0, 0), (3, 60)): + case("truncate", t, head, tail) + case("capped", t, "text") + case("chunk", t, 60, 3, 4000) + case("chunk", t, 2, 1, 30) + case("collapse", t, 3) + case("collapse", t, 2) + case("strip_ansi", t) + case("tokens", t) + case("capped", "", "none") + case("capped", "w" * 20001, "text") + case("capped", "\u00e9" * 20001, "text") + case("truncate", "\u00e9" * 600, 100, 100) + case("truncate", "x" * 2051, 1500, 500) + case("truncate", "x" * 2050, 1500, 500) + case("chunk", "\n".join("line %d" % i for i in range(300)), 60, 3, 4000) + case("chunk", "\n".join("l" * 100 for _ in range(100)), 60, 3, 4000) + case("chunk", "\n".join(["a", "", " ", "b"] * 20), 5, 2, 100) + case("collapse", "\n".join(["tick"] * 10 + ["done"]), 3) + for text in ("@a @b rest of it", "@onlytag", "no tag", "@a\t@b\n text", "@ spaced", "@", "@@x y", "", "@\u00e9 \u4e2d x"): + case("tags", text) + for raw in ("5", "5s", "10m", "2h", "1d", "3 w", " 7 ", "abc", "", "5x", "-5", "5 m", "5mm", "0", "007"): + case("duration", raw) + for n in (-5, 0, 1, 59, 60, 3599, 3600, 86399, 86400, 999999): + case("delay", n) + case("human_size", n) + for n in (1023, 1024, 1048575, 1048576, 123456789): + case("human_size", n) + for value in ("12", " 12 ", "+3", "-4", "1_0", "1__0", "_1", "1_", "abc", "", "3.5", "0x10"): + case("as_int", value, 7) + for name in ("shell", "shell<|channel|>commentary", " spaced ", "", "<|x"): + case("toolname", name) + for cmd in ("curl http://x", "wget https://y", "curl ftp://z", "echo curlhttp", "CURL HTTP://X", "mycurl http", "curl_x http", "x wget", "", "echo http curl"): + case("curl_hint", cmd) + case("stream_status", 0, "a\nb\nc", 1.234, 0) + case("stream_status", 3, "", 0.04, 0) + case("stream_status", 1, "x" * 2000, 99.96, 1) + case("stream_status", 0, "\u00e9\u00e8\n\u4e2d", 2.5, 0) + + +def corpus_ansi(): + samples = [ + "plain", "\x1b[31mred\x1b[0m", "\x1b[1m\x1b[36mbold cyan\x1b[0m trailing text here that is long enough to wrap", + "word " * 40, "a\u00e9\u4e2d long\u00a0word " * 8, "\x1b[2mdim\x1b[0m \x1b[4munder \x1b[0m end", "x" * 200, + "tab\tsep\nnew\nline", "", " ", "\x1b[9mstrike", "esc \x1b alone \x1b[ broken", "a\u3000b\u3000c " * 10, + ] + for s in samples: + for w in (0, 1, 8, 20, 40, 80): + case("trunc_ansi", s, w) + case("pad_tinted", s, w) + for w in (10, 25, 60): + case("wrap", s, w, "", "") + case("wrap", s, w, "> ", " ") + case("wrap", s, w, "\x1b[2m| \x1b[0m", "\x1b[2m| \x1b[0m") + case("style_span", s, "\x1b[44m", "\x1b[33m") + + +MD = [ + "# Title\n\nparagraph with `code` and **bold** and *dim* and ~~gone~~.\n", + "## Sub\n### Third\n#### Fourth\n####### seven\n#nospace\n#\n# \n", + "- one\n- two\n - nested\n - deeper\n- [ ] todo\n- [x] done\n- [X] done2\n", + "1. first\n2. second\n 1. inner\n 2. inner2\n3. third\n", + "> quote line\n> second line\n>\n> new para\n>> nested\n", + "```python\nprint('x')\n\nmore\n```\nafter\n~~~\nraw\n~~~\n", + "| a | b |\n|---|:-:|\n| 1 | 2 |\n| long cell here | x |\n", + "| a | b | c |\n|:--|--:|:-:|\n| 1 | 2 |\n| only |\n", + "***\n---\n___\n- - -\ntext\n", + "line one \nline two\nline three\n\npara two\n", + "[link](http://example.com) and ![img](a.png \"t\") and [x](y \"title\") and ![](p.png)\n", + "escaped \\* star \\` tick \\[no\\](x) snake_case_word _em_ __strong__ __x__y\n", + "
and
and
here\ttab\n", + "plain\ntext\ncontinues here\n# not a heading mid paragraph\n", + "**unclosed bold and `unclosed code and [broken](\n", + "mixed *a **b** c* and `**no**` and **`code`**\n", + "\u00e9\u00e8 \u4e2d\u6587 **bold \u00e9** `\u4e2d` [\u00e9](u)\n", + "1) paren\n2) paren two\n\n* star\n+ plus\n\ntext after\n", + " - indented bullet\n - more\n\n| not | table\n|just|text|\n", + "a | b\n--|--\nc | d\n", + "```\nunterminated fence\n", + "- item with a very long line that should wrap around the width of the terminal when rendered with a small width\n", + "~~~lang x\n``` inside\n~~~\n", +] + + +def corpus_markdown(): + for text in MD: + case("md_inline", text) + for width in (20, 40, 80): + case("markdown", text, width) + case("markdown", "", 80) + case("markdown", "\n\n\n", 80) + case("md_inline", "[nested **bold** link](http://u/**x**)") + case("md_inline", "`a` `b` \\`c\\` ![alt](u) [t](u)") + + +DIFF_OLD = "def f(x):\n return x + 1\n\n\nclass A:\n pass\n\nprint('end')\n" +DIFF_NEW = "def f(x):\n return x + 2\n\n\nclass A:\n \"\"\"doc\n string\"\"\"\n pass\n\n# comment 12 3.5e3\nprint('end')\n" + + +def corpus_diff(): + big_old = "\n".join("line %d" % i for i in range(300)) + "\n" + big_new = "\n".join(("line %d changed" % i if i % 7 == 0 else "line %d" % i) for i in range(300)) + "\n" + pairs = [ + ("a.py", DIFF_OLD, DIFF_NEW), ("a.txt", DIFF_OLD, DIFF_NEW), ("new.py", "", DIFF_NEW), ("gone.py", DIFF_OLD, NONE), + ("same.py", "x\n", "x\n"), ("e.py", "", ""), ("b.py", big_old, big_new), ("tabs.py", "a\tb\n", "a\tc\n"), + ("u.py", "caf\u00e9 = 1\n", "caf\u00e9 = 2\n"), ("ansi.py", "x \x1b[31my\x1b[0m\n", "x \x1b[31mz\x1b[0m\n"), + ("nonl.py", "a", "b"), ("deleteall.txt", "a\nb\n", ""), ("kw.py", "x = 1\n", "if x:\n import os as o # c\n def g(): return None\n"), + ("long.py", "a" * 400 + "\n", "b" * 400 + "\n"), + ("trip.py", "'''open\nmid\n'''\nx = 1\n", "'''open\nmid2\n'''\nx = 2\n"), + ] + for path, old, new in pairs: + for width in (0 + 100, 60): + for colors in (1, 0): + case("diff", path, old, new, width, colors, label="diff:" + path) + case("diff", "n.py", NONE, "x\n", 80, 1) + for line in ("def f(x, y=3):", "class Foo: # comment", "x = 'str' + \"dq\" + 12 + 0x1f + 1.5e3j", "@decorator.attr(1)", "'''triple''' in line", + " return None ", "s = \"unterminated", "# only comment", "print(\"a\\\"b\") # c", "w1 = 1_000 and 3x or x3", "async def go(): await y", + "\"\"\"doc\"\"\" + '''doc2'''", "d = {'k': 1} ", "name = 'caf\u00e9' # \u4e2d", "def two_spaces(): pass", "1.e5 2e+ 3e+4 5.5.5 0_0"): + case("highlight", line) + + +JSON_SAMPLES = [ + '{"a": 1, "b": [1, 2.5, "x"], "c": null, "d": true}', '[]', '{}', '"str"', '12', '-0', '1e5', '1.5E-3', '123456789012345678901234567890', + '{"k": {"z": 1, "a": [{"q": "\\u00e9\\n\\"x\\""}]}}', '{"dup": 1, "dup": 2}', 'nope', '', '{"a":', '[1,]', "{'a': 1}", 'NaN', 'Infinity', '-Infinity', + '"\\ud83d\\ude00"', '{"s": "it\'s"}', '{"s": "say \\"hi\\" it\'s"}', '[1e400]', '0.1', '1e16', '1e-5', '100.0', '[true, false, null]', ' {"a" : 1 } ', + '{"a": "\\u0007\\u007f\\u00a0"}', '"tab\\there"', '[[[[1]]]]', '{"b": 1, "a": 2}', '1.0', '5e-324', '0.30000000000000004', +] + + +def corpus_json(): + for text in JSON_SAMPLES: + case("json", text) + case("tool_args", text) + + +DESCRIBE = [ + ("shell", '{"command": "ls -la /tmp"}'), ("shell", '{"command": " many spaces\\nand newline "}'), ("shell", ""), ("shell", "not json"), ("shell", "[1,2]"), + ("read_file", '{"path": "/a/b/c.txt"}'), ("read_file", '{"path": "/a/b/"}'), ("read_file", '{"file": "rel/x.py"}'), ("write_file", '{"path": "/"}'), + ("edit_file", '{"path": "x.py", "old": "a", "new": "b"}'), ("web_fetch", '{"url": "http://example.com/" }'), ("web_search", '{"query": "q"}'), + ("describe_image", '{"image": "pic.png"}'), ("describe_image", '{"path": "/x/y.png", "image": "z"}'), ("speak", '{"text": "hello world"}'), + ("fork", '{"task": "' + "long task " * 20 + '"}'), ("poll", '{"agent_id": 7}'), ("poll", '{}'), ("poll", '{"target": "t"}'), ("ask_user", '{"question": "ok?"}'), + ("store_secret", '{"name": "k", "value": "topsecret"}'), ("store_secret", '{"name": "k"}'), ("delete_secret", '{"name": "k"}'), + ("schedule", '{"prompt": "do it", "at": "2026-01-01T00:00"}'), ("schedule", '{"prompt": "do it", "every": "5m"}'), ("schedule", '{"prompt": "p"}'), + ("release", '{"part": "patch", "message": "msg"}'), ("plexer_show", '{"spec": {"title": "My flow", "screens": [1,2]}}'), ("plexer_show", '{"spec": {"screens": [1,2,3]}}'), + ("plexer_show", '{"spec": {}}'), ("plexer_show", '{"spec": "str"}'), ("plexer_send", '{"id": "x"}'), ("record_save", '{"title": "T", "content": "C"}'), + ("unknown_tool", '{"a": 1}'), ("unknown_tool", ''), ("unknown_tool", 'raw text here'), ("shell", '{"command": null}'), ("shell", '{"command": 12}'), + ("shell", '{"command": ["a", "b"]}'), ("shell", '{"command": {"k": "v"}}'), ("shell", '{"command": true}'), ("tags", '{"prefix": "p"}'), + ("install", '{"component": "venv"}'), ("graph_query", '{"node": "n"}'), ("audit", '{"path": "/p/q"}'), ("listen", '{"seconds": 5}'), ("remember", '{"instruction": "x"}'), + ("load_skill", '{"name": "s"}'), ("install_skill", '{"source": "github"}'), ("create_bot", '{"name": "bot"}'), ("search", '{"query": "z"}'), + ("shell", '{"command": "' + "x" * 300 + '"}'), ("read_file", '{"path": "' + "d/" * 50 + 'file.txt"}'), ("tui_screen", '{}'), ("sysinfo", '{}'), ("unschedule", '{"id": 3}'), +] + + +def corpus_glow(): + for icon, verb, target, meta, width, color in (("\u25c6", "Edited", "path/to/file.py", "(+1 -2)", 0, ""), ("\u25c6", "Run", "ls", "", 40, "\x1b[32m"), ("x", "V", "", "m", 5, ""), + ("\u25c6", "Created", "a" * 80, "(new file, 10 lines)", 30, "")): + case("g_event", icon, verb, target, meta, width, color) + for verb, target, chars, ms, failed, width in (("Run", "ls", 120, 12.7, 0, 0), ("Run", "ls", 120, 1500, 0, 60), ("Run", "ls", 5, 999, 1, 60), ("Read", "f" * 100, 1, 2500.55, 0, 30), + ("Done", "", 0, 0, 0, 20), ("R", "t", 10, 1000, 0, 17)): + case("g_done", verb, target, chars, ms, failed, width) + for text, cap, head in (("short", 60, 0), ("x" * 100, 60, 0), ("y" * 100, 50, 1), ("abc" * 5, 7, 0), ("\u00e9" * 100, 20, 0), ("", 60, 0), ("a" * 61, 60, 0), ("path/" * 30, 33, 0)): + case("g_ellipsis", text, cap, head) + for text, pos in (("thinking", 0), ("thinking", 5), ("thinking", 17), ("", 3), ("\u00e9\u4e2d", 2), ("x", 100)): + case("g_shimmer", text, pos, 3) + case("g_shimmer", "merging memory", 7, 5) + for name, raw in DESCRIBE: + case("g_describe", name, raw) + for yolo, agents, width in ((0, 0, 80), (1, 0, 80), (0, 3, 80), (1, 2, 120), (1, 2, 40), (0, 0, 20), (1, 1, 10), (0, 0, 200)): + case("g_status", "1.44.2", "default", os.path.expanduser("~") + "/work/proj", yolo, agents, width) + case("g_status", "1.44.2", "prof", "/tmp/x", yolo, agents, width) + case("g_status", "1.44.2", "prof", os.path.expanduser("~"), yolo, agents, width) + for name in ("shell", "fork", "poll", "write_file", "edit_file", "delete_file", "ask_user", "store_secret", "list_secrets", "delete_secret", "other"): + case("g_accent", name) + case("g_hair", 0 + 30) + case("g_hair", 1) + case("g_prompt") + case("g_preview", "some text", 0) + case("g_preview", "some text that is long", 10) + recs = [ + {"id": 1, "status": "running", "task": "build the thing " * 5, "elapsed": 75, "tools": 3}, + {"id": 2, "status": "done", "task": "finished", "elapsed": 3.7, "tools": 0, "collected": False}, + {"id": 3, "status": "error", "task": "bad", "elapsed": 100000, "tools": 2, "collected": True}, + {"id": 4, "status": "timeout", "task": "slow\n\ttask x", "elapsed": 0}, + {"id": 5, "status": "weird", "task": "w", "elapsed": 7200}, + {"id": 6, "status": "done", "task": None, "elapsed": None, "tools": 0}, + ] + for rec in recs: + for width in (0, 50): + case("g_agent_row", json.dumps(rec), width, "\x1b[2m", " sfx") + for cap in (6, 2, 1): + for width in (0, 60): + case("g_lanes", json.dumps(recs), cap, width, "\x1b[1m") + case("g_lanes", json.dumps([]), 6, 0, "") + case("g_lanes", json.dumps([recs[2]]), 6, 0, "") + case("g_lanes", json.dumps(recs[:2]), 6, 40, "") + + +def make_proc_root(base): + """A fake /proc: chain 5000 (python) <- 4000 (bash) <- 3000 (-zsh) <- 1 (init).""" + root = os.path.join(base, "proc") + os.makedirs(os.path.join(root, "sys/kernel/random"), exist_ok=True) + with open(os.path.join(root, "stat"), "w") as handle: + handle.write("cpu 1 2 3\nbtime 1760000000\nprocesses 5\n") + with open(os.path.join(root, "sys/kernel/random/boot_id"), "w") as handle: + handle.write("ABCDEF12-3456-7890-abcd-ef1234567890\n") + procs = [(5000, "python3", 4000, 9000, "python3\0tai.py\0"), (4000, "bash", 3000, 8000, "bash\0--norc\0"), (3000, "zsh", 1, 7000, "-zsh\0"), (1, "systemd", 0, 100, "/sbin/init\0")] + for pid, comm, ppid, ticks, cmdline in procs: + os.makedirs(os.path.join(root, str(pid)), exist_ok=True) + fields = ["S", str(ppid)] + ["0"] * 17 + [str(ticks)] + ["0"] * 20 + with open(os.path.join(root, str(pid), "stat"), "w") as handle: + handle.write("%d (%s) %s\n" % (pid, comm, " ".join(fields))) + with open(os.path.join(root, str(pid), "cmdline"), "w", newline="") as handle: + handle.write(cmdline) + # a process whose comm contains parentheses and a nested shell + os.makedirs(os.path.join(root, "6000"), exist_ok=True) + with open(os.path.join(root, "6000", "stat"), "w") as handle: + handle.write("6000 (we(ird) name) S 5000 " + " ".join(["0"] * 17 + ["12345"] + ["0"] * 20) + "\n") + with open(os.path.join(root, "6000", "cmdline"), "w", newline="") as handle: + handle.write("/usr/bin/fish\0-l\0") + with open(os.path.join(root, "7000"), "w") as handle: + handle.write("") + return root + + +def corpus_session(base): + root = make_proc_root(base) + for text in ("1 (bash) S 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 5 6 7", "", "no parens", "(only) ", "12 (a b) S 4 " + " ".join(["0"] * 17 + ["99"]), + "12 (a)b) S 4 " + " ".join(["0"] * 17 + ["99"]), "12 (a) S x " + " ".join(["0"] * 17 + ["99"]), "12 (a) S 4 1"): + case("sess_stat", text) + for argv0, comm in (("bash", "bash"), ("-zsh", "zsh"), ("/bin/FISH", "x"), ("python3", "python3"), ("", "sh"), ("-", "-nu"), ("tcsh", ""), ("x", "-ksh")): + case("sess_name", argv0, comm) + case("sess_boot", root) + case("sess_boot", os.path.join(base, "nonexistent")) + for pid in (5000, 6000, 1, 99999): + for env in ("", "mypinned", "20250101-000000-p4000-abcdef12", "20250101-000000-p1234-abcdef12", "weird chars!@#", "x" * 100): + case("sess_current", env, pid, root, 1760001000.5, 100, label="sess_current:%s:%s" % (pid, env[:12])) + for text in ("", "a", "abc", "hello world" * 20, "\u00e9"): + case("md5", text) + # no boot_id: falls back to md5 of btime + alt = os.path.join(base, "proc2") + os.makedirs(alt, exist_ok=True) + shutil.copytree(os.path.join(root, "5000"), os.path.join(alt, "5000")) + with open(os.path.join(alt, "stat"), "w") as handle: + handle.write("btime 1760000123\n") + case("sess_boot", alt) + case("sess_current", "", 5000, alt, 1760001000.0, 100) + + +def corpus_targets(base): + work = os.path.join(base, "work") + os.makedirs(os.path.join(work, "sub/deeper"), exist_ok=True) + os.makedirs(os.path.join(work, "empty"), exist_ok=True) + for name in ("a.txt", "b.txt", "c.log", ".hidden", "sub/s1.txt", "sub/s2.py", "sub/deeper/d.txt", "name with space.txt"): + with open(os.path.join(work, name), "w") as handle: + handle.write("x") + os.symlink(os.path.join(work, "a.txt"), os.path.join(work, "link")) + for i in range(40): + with open(os.path.join(work, "sub", "many%02d.dat" % i), "w") as handle: + handle.write("x") + cmds = [ + "rm a.txt", "rm -rf sub", "rm -f *.txt", "rm sub/*", "rm **/*.txt", "rm sub/**", "rm [ab].txt", "rm ?.txt", "rm -- -weird", "mv a.txt b.txt", "cp a.txt x/y.txt", + "tee b.txt < in", "echo hi > a.txt", "echo hi >> b.txt", "echo hi > /dev/null", "echo x 2>&1 > c.log", "cat a >&2", "truncate -s 10 a.txt", "truncate -s 10K a.txt b.txt", + "dd if=/dev/zero of=a.txt bs=1", "shred a.txt", "sudo rm a.txt", "sudo -u x rm a.txt", "env FOO=1 rm a.txt", "FOO=1 BAR=2 rm a.txt", "timeout 5 rm a.txt", + "timeout 5s rm a.txt", "timeout x rm a.txt", "nice -n 5 rm a.txt", "command rm a.txt", "nohup rm a.txt", "setsid rm a.txt", "ls; rm a.txt; rm b.txt", "ls && rm a.txt || rm b.txt", + "ls | tee c.log", "(rm a.txt)", "rm 'name with space.txt'", 'rm "name with space.txt"', "rm name\\ with\\ space.txt", "rm 'unclosed", 'rm "unclosed', "rm trailing\\", + "rm a.txt &", "rm a.txt & rm b.txt", "rm nonexistent", "", " ", ";;", "rm", "mv", "mv a.txt", "cp a b c", "rm empty", "rm sub/deeper", "rm /dev/null", "rm .", "rm ..", "rm -r .", + "rm sub/many*", "rm sub", "rm link", "rm .hidden", "rm *", "rm .*", "tee", "tee -a a.txt", "echo > a.txt > b.txt", "echo >a.txt", "echo>b.txt", "FOO=bar", "env", "sudo", + "rm $HOME", "rm ~", "rm a.txt;rm b.txt", "rm a.txt|cat", "rm \"a\".txt", "rm a'.'txt", "rm '' a.txt", "rm \\\\", "echo 'a b' > 'b.txt'", "rm sub/../a.txt", "rm ./a.txt", + "rm " + os.path.join(work, "a.txt"), "rm /nonexistent/dir/*", "rm [", "rm [a", "rm [!a].txt", "rm [a-b].txt", "rm [b-a].txt", "rm *.t[x]t", "rm sub/*/*", "rm */", "rm sub/", + ] + for cmd in cmds: + case("target_paths", cmd, work, label="target_paths:" + cmd) + case("target_paths", "rm a.txt", "relative/workdir") + case("target_paths", "rm *", os.path.join(work, "sub")) + for tokens in (["rm", "a", "b"], ["mv", "x", "y"], ["cp"], [], [">", "f"], ["echo", ">", "&2"], ["echo", ">", "/dev/null"], ["A=1", "rm", "x"], ["sudo", "env", "A=1", "tee", "f"], + ["timeout", "10", "rm", "f"], ["timeout", "1.5m", "rm", "f"], ["truncate", "-s", "5", "f"], ["truncate", "1GiB", "f"], ["dd", "of=out", "if=in"], ["-", "x"], ["rm", "-", "x"], + ["=x", "rm", "y"], [">x", "rm"], ["/a=b", "rm"], ["rm", "-rf", "--", "-x"], ["shred", "-u", "z"], ["/bin/rm", "f"], ["tee", ">", "f"]): + case("segment", *tokens) + + # complete_files / mentions need a stable dir + comp = os.path.join(base, "comp") + os.makedirs(os.path.join(comp, "Alpha"), exist_ok=True) + os.makedirs(os.path.join(comp, "sub/inner"), exist_ok=True) + for name in ("alpha.txt", "Beta.md", ".dot", "Apple", "z.py", "sub/one.txt", "sub/Two.txt", "sub/.hid"): + with open(os.path.join(comp, name), "w") as handle: + handle.write("x") + for prefix in ("", "a", "A", "al", "B", ".", ".d", "sub/", "sub/o", "sub/T", "nothing", "sub/inner", "sub/inner/", "/", "zz", "z"): + case("complete", prefix, comp, 100) + case("complete", "", comp, 2) + case("complete", "a", "/nonexistent/dir", 100) + case("complete", "~/", comp, 5) + for text in ("see @alpha.txt please", "@alpha.txt,", "@alpha.txt) and @Beta.md!", "mail me@alpha.txt", "@nothing here", "@", "x @ y", "@sub/one.txt:", "@/etc/hostname", "@~", "@sub", "a\n@z.py\n@sub/Two.txt", + "@alpha.txt@Beta.md", "\u00e9 @alpha.txt \u4e2d", "(@alpha.txt)", "@.dot", "@.", "@..", "@sub/../z.py", "@@z.py", "@z.py?;"): + case("mentions", text, comp) + + +def corpus_live(): + case("live", 4, 40, "exit 0 done", "one\n", "two\n", "three\n") + case("live", 4, 40, "status", "1\n", "2\n", "3\n", "4\n", "5\n", "6\n") + case("live", 2, 30, "s", "tab\there\n", "carriage\rreturn\n", "\x1b[31mred\x1b[0m long line " + "x" * 80 + "\n", "last\n") + case("live", 3, 20, "s") + case("live", 3, 20, "s", "a\n", "\x01PAUSE", "b\n", "c\n", "d\n", "e\n") + case("live", 1, 8, "very long status line here that gets cut", "x\n") + case("live", 5, 100, "s", "\u00e9\u4e2d\u00e8\n", "plain\n\n", "trailing newlines\n\n\n") + + +def corpus_run(): + sh = "shell" + case("run", 10, sh, 0, "--", "echo hello; echo err >&2; exit 3") + case("run", 10, sh, 0, "--", "printf 'a\\rb\\r\\nc\\n'") + case("run", 10, sh, 0, "--", "printf 'no newline at end'") + case("run", 10, sh, 0, "--", "printf 'x\\ry'") + case("run", 10, sh, 0, "--", "printf 'bad \\xff\\xfe utf8 \\xe4\\xb8 cut\\n'") + case("run", 10, sh, 0, "--", "seq 1 5000") + case("run", 10, sh, 0, "--", "head -c 300000 /dev/zero | tr '\\0' 'a'; echo") + case("run", 10, sh, 0, "--", "for i in 1 2 3; do echo $i; sleep 0.05; done") + case("run", 10, sh, 0, "--", "exit 0") + case("run", 10, sh, 0, "--", "kill -9 $$") + case("run", 10, sh, 0, "--", "cat /nonexistent 2>&1; echo $?") + case("run", 10, sh, 0, "--", "echo $0; pwd >/dev/null; cat; echo eof-ok") + case("run", 1, sh, 0, "--", "echo before; sleep 5; echo after") + case("run", 10, "argv", 0, "--", "echo", "arg one", "two") + case("run", 10, "argv", 0, "--", "/nonexistent/program") + case("run", 10, "argv", 0, "--", "sh", "-c", "exit 7") + # pty path + case("run", 10, sh, 1, "--", "echo plain; exit 2") + case("run", 10, sh, 1, "--", "printf 'a\\rb\\n'; echo done") + case("run", 10, sh, 1, "hunter2", "--", "printf 'Password: '; read -r x; echo \"got:$x\"") + case("run", 10, sh, 1, "yes", "--", "printf 'Continue? (y/n) '; read -r x; echo \"answer:$x\"") + case("run", 10, sh, 1, "\x01DECLINE", "--", "printf 'Name: '; read -r x; echo \"read:$x:\"") + case("run", 10, sh, 1, "a", "b", "--", "printf 'First: '; read -r x; printf 'Second: '; read -r y; echo \"$x$y\"") + case("run", 10, sh, 1, "s3cret", "--", "stty -echo; printf 'Secret: '; read -r x; stty echo; echo; echo \"len:${#x}\"") + case("run", 10, sh, 1, "--", "echo not a prompt; sleep 1.3; echo end") + case("run", 10, sh, 1, "--", "printf 'no prompt chars here'; sleep 1.3; echo") + case("run", 10, sh, 1, "line1\nline2", "--", "printf 'Q: '; read -r x; echo \"[$x]\"") + case("run", 10, sh, 1, "x" * 1200, "--", "printf 'Q: '; read -r x; echo ${#x}") + case("run", 10, sh, 1, "--", "seq 1 20000") + case("run", 10, sh, 1, "--", "tty >/dev/null && echo has-tty; stty size; echo $PAGER $GIT_PAGER") + case("run", 10, sh, 1, "--", "exit 5") + case("run", 10, sh, 1, "--", "kill -9 $$") + case("run", 1, sh, 1, "--", "echo start; sleep 6; echo never") + case("run", 10, sh, 1, "--", "printf '\\033[31mred\\033[0m \\033]0;title\\007ok\\n'") + case("run", 10, sh, 1, "--", "printf 'caf\\xc3\\xa9 \\xe4\\xb8\\xad\\n'") + case("run", 10, "argv", 1, "--", "/nonexistent/program") + if shutil.which("su"): # a real password prompt from a real program (secret detection, EOF on decline) + case("run", 30, sh, 1, "nope", "--", "su root -c true; echo rc=$?", label="run:su-answer") + case("run", 30, sh, 1, "\x01DECLINE", "--", "su root -c true; echo rc=$?", label="run:su-decline") + case("run", 10, sh, 1, "answer", "--", "printf 'Proceed? '; read -r x; echo got:$x; printf 'Again? '; read -r y; echo got2:$y") + # waiting for the user does not count against the timeout: 2s budget, 3s of thinking + case("run", 2, sh, 1, "\x01SLEEP3000:slowanswer", "--", "printf 'Name: '; read -r x; echo \"hi $x\"") + case("run", 2, sh, 1, "\x01SLEEP3000:a", "\x01SLEEP1500:b", "--", "printf 'One: '; read -r x; printf 'Two: '; read -r y; echo \"$x$y\"") + # output cap on the pty path (RUN_OUTPUT_CAP), none on the pipe path + case("run", 60, sh, 1, "--", "seq 1 700000") + case("run", 60, sh, 0, "--", "seq 1 300000") + case("run", 10, sh, 1, "--", "printf 'x%.0s' $(seq 1 100); echo; printf 'tab\\there\\n'") + + +def corpus_lane(): + r1 = {"id": 1, "status": "running", "task": "alpha", "elapsed": 5, "tools": 1} + r2 = {"id": 2, "status": "done", "task": "beta", "elapsed": 9, "tools": 0} + r3 = {"id": 3, "status": "running", "task": "gamma", "elapsed": 1} + case("lane", json.dumps([[r1], [r1]]), 60, 6) + case("lane", json.dumps([[r1], [r1, r2]]), 60, 6) + case("lane", json.dumps([[r1, r2], [r1, r2, r3]]), 0 + 40, 2) + case("lane", json.dumps([[], [r1]]), 50, 6) + case("lane", json.dumps([[r1, r2, r3], [r1, r2, r3]]), 80, 1) + + +def pty_stderr(argv, env): + """Run argv with stderr on a pty (so isatty is true); return what it wrote there.""" + import pty as ptymod + master, slave = ptymod.openpty() + proc = subprocess.Popen(argv, stdout=subprocess.DEVNULL, stderr=slave, env=env) + os.close(slave) + data = b"" + while True: + try: + chunk = os.read(master, 65536) + except OSError: + break + if not chunk: + break + data += chunk + proc.wait() + os.close(master) + return data.decode("utf-8", "replace") + + +def spinner_cases(opts): + """Spinner under a real pty: same frames, same clear sequence. Frame count depends on + sleep granularity, so the first frames and the final clear are compared, plus the count within one.""" + failures = 0 + runner = ("import importlib.util, sys, time\n" + "spec = importlib.util.spec_from_file_location('tai', %r)\n" + "tai = importlib.util.module_from_spec(spec); sys.modules['tai'] = tai; spec.loader.exec_module(tai)\n" + "s = tai.Spinner(sys.argv[1]); s.start(); time.sleep(int(sys.argv[2]) / 1000.0); s.stop()\n") % PY_FILE + for label, ms, no_color in (("thinking", 500, False), ("merging memory", 700, False), ("plain label", 400, True), ("x", 300, False)): + env = dict(os.environ) + env.pop("NO_COLOR", None) + if no_color: + env["NO_COLOR"] = "1" + want = pty_stderr([sys.executable, "-c", runner, label, str(ms)], env) + got = pty_stderr([DRIVER, "spinner", label, str(ms)], env) + norm = lambda text: re.sub(r"\d+\.\ds", "T.Ts", text.replace("\r\n", "\n")) + w, g = norm(want).split("\r"), norm(got).split("\r") + ok = w[:4] == g[:4] and w[-2:] == g[-2:] and abs(len(w) - len(g)) <= 2 + print("%s spinner %r %dms (frames py=%d cpp=%d)" % ("ok " if ok else "FAIL", label, ms, len(w), len(g))) + if not ok: + failures += 1 + print(" want: %r" % w[:5]) + print(" got : %r" % g[:5]) + return failures + + +def burst_cases(): + """Concurrent runs through both paths: every run must equal the Python result.""" + from concurrent.futures import ThreadPoolExecutor + want = {} + for mode in ("0", "1"): + want[mode] = b(py_run(30, "shell", mode, "--", "seq 1 20000; echo done >&2")) + jobs = [(mode,) for _ in range(12) for mode in ("0", "1")] + with ThreadPoolExecutor(max_workers=24) as pool: + results = list(pool.map(lambda job: (job[0], cpp("run", 30, "shell", job[0], "--", "seq 1 20000; echo done >&2", timeout=120)), jobs)) + bad = sum(1 for mode, out in results if out != want[mode]) + print("%s burst: %d concurrent runs (pipe and pty), %d differ from tai.py" % ("ok " if not bad else "FAIL", len(results), bad)) + return 1 if bad else 0 + + +import contextlib + + +@contextlib.contextmanager +def fixed_env(now): + """Deterministic clock, urandom and uuid4 for the edge-speech functions (the C++ driver does the same).""" + counter = [0] + + class Uuid(object): + def __init__(self, n): + self.hex = "%032x" % n + + def uuid4(): + counter[0] += 1 + return Uuid(counter[0]) + + saved = (time.time, os.urandom, tai.uuid.uuid4) + real_gmtime = time.gmtime + time.gmtime = lambda secs=None: real_gmtime(float(now) if secs is None else secs) + time.time = lambda: float(now) + os.urandom = lambda n: bytes((i * 7 + 1) % 256 for i in range(n)) + tai.uuid.uuid4 = uuid4 + try: + yield + finally: + time.time, os.urandom, tai.uuid.uuid4 = saved + time.gmtime = real_gmtime + + +class FakeSock(object): + def __init__(self, inbox=b""): + self.inbox, self.sent = inbox, b"" + + def sendall(self, data): + self.sent += data + + def recv(self, n): + out, self.inbox = self.inbox[:n], self.inbox[n:] + return out + + def close(self): + pass + + +def py_ws_recv(hexframes): + sock = FakeSock(bytes.fromhex(hexframes)) + with fixed_env(0): + try: + opcode, payload = tai.ws_recv(sock) + head = "OK %d %s" % (opcode, payload.hex()) + except tai.BackendError as exc: + head = "ERR %s" % exc + return head + "|sent=" + sock.sent.hex() + + +def py_edge_session(frames, text, voice, now): + sock = FakeSock(bytes.fromhex(frames)) + real = tai.edge_connect + tai.edge_connect = lambda: sock + try: + with fixed_env(now): + try: + head = "OK " + tai.edge_synthesize(text, voice).hex() + except tai.BackendError as exc: + head = "ERR %s" % exc + finally: + tai.edge_connect = real + return head + "|sent=" + sock.sent.hex() + + +def py_ws_send(payload): + sock = FakeSock() + with fixed_env(0): + tai.ws_send(sock, payload) + return sock.sent.hex() + + +def py_edge_token(skew, now): + with fixed_env(now): + return tai.edge_token(int(skew)) + + +def py_edge_ts(now): + with fixed_env(now): + return tai.edge_timestamp() + + +def py_edge_skew(header, now): + with fixed_env(now): + return str(tai.edge_skew(header)) + + +def frame(opcode, payload, fin=True, mask=None, ext=None): + """One websocket frame as the server (unmasked) or a test (masked) would send it.""" + first = (0x80 if fin else 0) | opcode + n = len(payload) + if ext == 16 or (ext is None and 126 <= n < 65536): + head = bytes([first, (0x80 if mask else 0) | 126]) + n.to_bytes(2, "big") + elif ext == 64 or (ext is None and n >= 65536): + head = bytes([first, (0x80 if mask else 0) | 127]) + n.to_bytes(8, "big") + else: + head = bytes([first, (0x80 if mask else 0) | n]) + if mask: + payload = bytes(b ^ mask[i % 4] for i, b in enumerate(payload)) + return (head + mask + payload).hex() + return (head + payload).hex() + + +PY.update({ + "edge_token": py_edge_token, + "edge_ts": py_edge_ts, + "edge_ssml": lambda t, v: tai.edge_ssml(t, v), + "edge_chunks": lambda t, n: RS.join(tai.edge_chunks(t, int(n))), + "ws_send": py_ws_send, + "ws_recv": py_ws_recv, + "edge_skew": py_edge_skew, + "edge_session": py_edge_session, + "listen": lambda sec, work: tai.listen_audio(int(sec), work), + "play": lambda path: str(tai.play_audio(path)), +}) + + +def live_edge(): + """One real synthesis against the Edge service (network): sizes within 30%, same container header.""" + text, voice = "Hello from the parity test.", "en-US-EmmaMultilingualNeural" + try: + audio = tai.edge_synthesize(text, voice) + want = "OK" + size_py = len(audio) + except Exception as exc: # network down: report, do not fail the run + print("skip live edge synthesis (python side failed: %s)" % exc) + return 0 + out = cpp("edge_live", text, voice, timeout=90).decode("utf-8", "replace") + parts = out.split() + ok = parts[:1] == ["OK"] and abs(int(parts[1]) - size_py) <= 0.3 * size_py and parts[2] == audio[:3].hex() + print("%s live edge synthesis: python %d bytes, cpp %s" % ("ok " if ok else "FAIL", size_py, out)) + return 0 if ok else 1 + + +def corpus_edge(): + for skew in (0, 5, -3600, 7200): + for now in (0, 1760000000, 1760000299, 1760000300.7): + case("edge_token", skew, now) + for now in (0, 1760000000, 1760086399.5): + case("edge_ts", now) + for text in ("hello", "a < b & c > d", "line\nbreak\rcr", "it's \"q\"", "\u00e9\u4e2d"): + case("edge_ssml", text, "en-US-EmmaMultilingualNeural") + for text, limit in (("One. Two! Three? Four.", 4000), ("One sentence here. Another sentence there. And a third.", 25), (" padded text ", 4000), ("", 10), + ("no punctuation just words " * 10, 40), ("A. B. C. D.", 3), ("x" * 100, 10), ("Dr. Smith went. He left.\n\nNew para.", 30), ("\u00e9\u00e9\u00e9. \u00e9\u00e9\u00e9.", 7)): + case("edge_chunks", text, limit) + for payload in ("", "hi", "x" * 125, "x" * 126, "x" * 65535, "x" * 65536, "\u00e9\u4e2d"): + case("ws_send", payload, label="ws_send:%d" % len(payload)) + mask = bytes([9, 8, 7, 6]) + frames = { + "text": frame(1, b"Path:turn.end"), "binary-masked": frame(2, b"abc", mask=mask), "ext16": frame(2, b"z" * 300), "ext64": frame(2, b"y" * 70000, ext=64), + "ping-then-text": frame(9, b"") + frame(1, b"after"), "close": frame(8, b""), "fragmented": frame(1, b"par", fin=False) + frame(0, b"ts"), + "fragment-binary": frame(2, b"ab", fin=False) + frame(0, b"cd", fin=False) + frame(0, b"ef"), "truncated": frame(1, b"hello")[:-4], "empty": "", "one-byte": "81", + "ping-in-fragments": frame(1, b"a", fin=False) + frame(9, b"") + frame(0, b"b"), "pong-ignored": frame(10, b"x") + frame(1, b"y"), + } + for name, data in frames.items(): + case("ws_recv", data, label="ws_recv:" + name) + for header, now in (("HTTP/1.1 403 Forbidden\r\nDate: Mon, 06 Jan 2025 10:00:00 GMT\r\n\r\n", 1736157600), ("HTTP/1.1 403\r\ndate: Tue, 07 Jan 2025 10:00:01 UTC\r\nX: y\r\n\r\n", 1736157600), + ("HTTP/1.1 403\r\nDate: garbage\r\n\r\n", 1736157600), ("HTTP/1.1 403\r\n\r\n", 1736157600), ("Date:Mon, 6 Jan 2025 1:2:3 GMT", 1736157600), + ("Date: Mon, 06 Jan 2025 10:00:00 CET\r\n", 1736157600), ("Date: Mon, 32 Jan 2025 10:00:00 GMT\r\n", 1736157600), ("Date: Monday, 06 January 2025 10:00:00 GMT\r\n", 1736157600), + ("Date: Mon,06 Jan 2025 10:00:00 GMT\r\n", 1736157600), ("x\r\nDATE: Wed, 08 Jan 2025 23:59:59 GMT \r\n", 1736157600), ("Date: Mon, 06 Jan 2025 10:00:00\r\n", 1736157600)): + case("edge_skew", header, now) + turn_end = frame(1, b"X-RequestId:1\r\nPath:turn.end\r\n\r\n{}") + session = frame(1, b"Path:turn.start") + frame(2, (5).to_bytes(2, "big") + b"HDR:x" + b"AUDIO1") + frame(2, b"\x00\x01") + frame(9, b"") + frame(2, (0).to_bytes(2, "big") + b"AUDIO2") + turn_end + case("edge_session", session, "Hello there.", "en-US-X", 1760000000) + case("edge_session", session + session, "One sentence. " * 400, "en-US-X", 1760000000, label="edge_session:two-chunks") + case("edge_session", frame(1, b"nothing useful") + frame(8, b""), "Hi.", "v", 1760000000, label="edge_session:closed") + case("edge_session", "", "Hi.", "v", 1760000000, label="edge_session:eof") + case("edge_session", frame(2, b"\x00\x00") + turn_end, "Hi.", "v", 1760000000, label="edge_session:short-binary") + + +def corpus_audio(base): + """listen/play with fake recorder, transcriber and player binaries on private PATHs.""" + def make_bin(name, scripts): + bindir = os.path.join(base, name) + os.makedirs(bindir) + for binary, body in scripts.items(): + path = os.path.join(bindir, binary) + with open(path, "w") as handle: + handle.write("#!/bin/sh\n" + body + "\n") + os.chmod(path, 0o755) + return bindir + + work = os.path.join(base, "audio") + os.makedirs(work) + rec = 'for last; do :; done; echo rec > "$last"' + full = make_bin("bin_full", {"arecord": rec, "whisper": 'n=${1##*/}; echo "transcript of ${n%%-*}"', "paplay": "exit 0"}) + plain = make_bin("bin_plain", {"rec": rec, "whisper-cpp": 'echo " spoken words "', "aplay": "exit 3"}) + broken = make_bin("bin_broken", {"ffmpeg": rec, "whisper-cli": 'echo "model missing" >&2; exit 4', "pw-play": "exit 0"}) + empty = make_bin("bin_empty", {"arecord": rec, "faster-whisper": "true"}) + longtext = make_bin("bin_long", {"arecord": rec, "whisper": 'i=0; while [ $i -lt 700 ]; do printf "word%d " $i; i=$((i+1)); done; echo'}) + norec = make_bin("bin_norec", {"whisper": "true"}) + nothing = make_bin("bin_nothing", {}) + for name, path in (("full", full), ("plain", plain), ("broken", broken), ("empty", empty), ("long", longtext), ("norec", norec), ("nothing", nothing)): + env = {"PATH": path} + case("listen", 3, work, label="listen:" + name, env=env) + case("play", os.path.join(work, "x.mp3"), label="play:" + name, env=env) + case("listen", 3, work + "/", label="listen:trailing-slash", env={"PATH": full}) + case("listen", 3, os.path.join(base, "no-such-dir"), label="listen:bad-workdir", env={"PATH": full}) + + +FAKE_PODMAN = r"""#!/bin/sh +D="$FAKE_DIR" +log() { printf '%s\n' "$*" >> "$D/calls.log"; } +cmd="$1" +case "$cmd" in + inspect) log "inspect $*"; [ -f "$D/state" ] || exit 1; if [ "$(cat "$D/state")" = running ]; then echo true; else echo false; fi ;; + start) log "start $*"; if [ "$FAKE_START_FAIL" = 1 ]; then echo "start failed badly" >&2; exit 125; fi; echo running > "$D/state" ;; + update) log "update $*"; if [ "$FAKE_UPDATE_FAIL" = 1 ]; then echo "cgroup write denied" >&2; exit 1; fi ;; + build) log "build $2 $3 "; for f in Containerfile stt.py tts.py; do echo "--- $f" >> "$D/calls.log"; cat "$4/$f" >> "$D/calls.log"; done + if [ "$FAKE_BUILD_FAIL" = 1 ]; then exit 1; fi ;; + run) log "run $*"; if [ "$FAKE_RUN_FAIL" = 1 ]; then echo "run failed" >&2; exit 125; fi; echo running > "$D/state" ;; + exec) + log "exec $*" + while [ "$1" != "tai-box" ]; do shift; done; shift + case "$1" in + test) exit "${FAKE_TEST_RC:-0}" ;; + which) exit "${FAKE_WHICH_RC:-0}" ;; + sh) if [ "$2" = "-c" ]; then log "script-start"; log "$3"; exit "${FAKE_INSTALL_RC:-0}"; fi ;; + opencode) in="$(cat)"; log "stdin: $in"; echo "OC-OUT $*"; echo "OC-ERR" >&2; exit "${FAKE_OC_RC:-0}" ;; + /box/venv/bin/python|python3) + in="$(cat)"; log "stdin: $in" + case "$2" in + /box/stt.py) if [ "$FAKE_STT_FAIL" = 1 ]; then echo "stt broke" >&2; exit 2; fi; echo " heard: $in " ;; + /box/tts.py) if [ "$FAKE_TTS_FAIL" = 1 ]; then echo "tts broke" >&2; exit 2; fi; if [ "$FAKE_TTS_EMPTY" = 1 ]; then exit 0; fi; printf 'MP3[%s|%s]' "$in" "$3" ;; + esac ;; + *) echo "unknown exec $*" >&2; exit 9 ;; + esac ;; + *) echo "unknown $cmd" >&2; exit 9 ;; +esac +""" + + +def box_reset(): + tai._BOX_CAPS_STAMP["done"] = False + tai._BOX_PYTHON_CACHE.clear() + tai._BOX_OPENCODE_CACHE.clear() + + +def py_box(scenario, engine, *rest): + box_reset() + out = io.StringIO() + real_parse = tai.opencode_parse_events + tai.opencode_parse_events = lambda stdout_text, returncode, stderr_text: "PARSE|%d|%s|%s" % (returncode, stdout_text, stderr_text) + try: + with contextlib.redirect_stdout(out), contextlib.redirect_stderr(out): + try: + if scenario == "engine": + result = tai.container_engine() + elif scenario == "state": + result = tai.box_state(engine) + elif scenario == "ensure": + result = "OK " + tai.ensure_box() + elif scenario == "caps": + result = str(tai.update_box_caps(engine)) + elif scenario == "python": + result = tai.box_python(engine) + elif scenario == "stt": + result = "OK " + tai.box_transcribe(engine, rest[0].encode("utf-8")) + elif scenario == "tts": + result = "OK " + tai.box_speak(engine, rest[0], rest[1]).hex() + elif scenario == "ocbin": + result = str(tai.box_opencode_binary(engine)) + elif scenario == "ocrun": + result = tai.box_opencode_run(engine, rest[0], rest[1], float(rest[2]) if "." in rest[2] else int(rest[2])) + elif scenario == "exec": + done = tai.box_exec(engine, list(rest)) + result = "RC %d %s %s" % (done.returncode, json.dumps(done.stdout.decode("utf-8", "replace")), json.dumps(done.stderr.decode("utf-8", "replace"))) + except tai.BackendError as exc: + result = "ERR %s" % exc + except subprocess.TimeoutExpired as exc: + result = "TIMEOUT %s" % exc + except OSError as exc: + result = "OSERR %s" % exc + finally: + tai.opencode_parse_events = real_parse + return out.getvalue() + result + + +def corpus_box(base): + """Box lifecycle against a scripted fake `podman`: same calls, same prints, same errors as tai.py.""" + bindir = os.path.join(base, "fakebin") + os.makedirs(bindir) + with open(os.path.join(bindir, "podman"), "w") as handle: + handle.write(FAKE_PODMAN) + os.chmod(os.path.join(bindir, "podman"), 0o755) + sysbin = "/usr/bin:/bin" + on = {"PATH": bindir + ":" + sysbin, "FAKE_TEMPLATE_STATE": ""} + + def scenario(label, name, state=None, extra=None, args=()): + env = dict(on) + env["FAKE_TEMPLATE_STATE"] = state or "" + env.update(extra or {}) + case("box", name, "podman", *args, label="box:" + label, env=env) + + scenario("engine", "engine") + case("box", "engine", "podman", label="box:engine-none", env={"PATH": os.path.join(base, "emptybin"), "FAKE_TEMPLATE_STATE": ""}) + case("box", "ensure", "podman", label="box:ensure-no-engine", env={"PATH": os.path.join(base, "emptybin"), "FAKE_TEMPLATE_STATE": ""}) + for st in ("", "running", "stopped"): + scenario("state-" + (st or "missing"), "state", st) + scenario("ensure-running", "ensure", "running") + scenario("ensure-running-nocaps-env", "ensure", "running", {"TAI_BOX_CAPS": "0"}) + scenario("ensure-running-capfail", "ensure", "running", {"FAKE_UPDATE_FAIL": "1"}) + scenario("ensure-stopped", "ensure", "stopped") + scenario("ensure-stopped-startfail", "ensure", "stopped", {"FAKE_START_FAIL": "1"}) + scenario("ensure-missing-builds", "ensure", "") + scenario("ensure-build-fail", "ensure", "", {"FAKE_BUILD_FAIL": "1"}) + scenario("ensure-run-fail", "ensure", "", {"FAKE_RUN_FAIL": "1"}) + scenario("caps", "caps", "running") + scenario("caps-fail", "caps", "running", {"FAKE_UPDATE_FAIL": "1"}) + scenario("python-venv", "python", "running") + scenario("python-fallback", "python", "running", {"FAKE_TEST_RC": "1"}) + scenario("stt", "stt", "running", args=("hello audio",)) + scenario("stt-fail", "stt", "running", {"FAKE_STT_FAIL": "1"}, args=("x",)) + scenario("tts", "tts", "running", args=("say this \u00e9", "")) + scenario("tts-voice", "tts", "running", args=("say this", "en-US-X")) + scenario("tts-fail", "tts", "running", {"FAKE_TTS_FAIL": "1"}, args=("x", "")) + scenario("tts-empty", "tts", "running", {"FAKE_TTS_EMPTY": "1"}, args=("x", "")) + scenario("ocbin-present", "ocbin", "running") + scenario("ocbin-install-ok", "ocbin", "running", {"FAKE_WHICH_RC": "1"}) + scenario("ocbin-install-fail", "ocbin", "running", {"FAKE_WHICH_RC": "1", "FAKE_INSTALL_RC": "3"}) + scenario("ocrun", "ocrun", "running", args=("mimo-v2.6-flash-free", "the prompt \u00e9", "30")) + scenario("ocrun-fail", "ocrun", "running", {"FAKE_OC_RC": "5"}, args=("m", "p", "30")) + scenario("exec-echo", "exec", "running", args=("test", "-x", "/x")) + scenario("exec-unknown", "exec", "running", args=("frobnicate",)) + case("box", "state", "/nonexistent/engine", label="box:state-bad-engine", env=on) + case("box", "python", "/nonexistent/engine", label="box:python-bad-engine", env=on) + case("box", "ocbin", "/nonexistent/engine", label="box:ocbin-bad-engine", env=on) + case("box", "ocrun", "/nonexistent/engine", "m", "p", "30", label="box:ocrun-bad-engine", env=on) + case("box", "exec", "/nonexistent/engine", "ls", label="box:exec-bad-engine", env=on) + + +PY["box"] = py_box + + +# ---------- runner ---------- +def main(): + parser = argparse.ArgumentParser() + parser.add_argument("--filter", default="") + parser.add_argument("--fuzz", type=int, default=300) + parser.add_argument("--seed", type=int, default=1) + parser.add_argument("--no-build", action="store_true") + parser.add_argument("--live", action="store_true", help="also run the network synthesis against the Edge service") + parser.add_argument("-v", action="store_true") + opts = parser.parse_args() + if not opts.no_build: + build() + os.environ["TZ"] = "Europe/Amsterdam" + base = tempfile.mkdtemp(prefix="shell_parity_", dir=BUILD) + try: + os.environ["HOME"] = os.path.join(base, "home") + os.makedirs(os.environ["HOME"], exist_ok=True) + corpus() + corpus_ansi() + corpus_markdown() + corpus_diff() + corpus_json() + corpus_glow() + corpus_session(base) + corpus_targets(base) + corpus_live() + corpus_run() + corpus_lane() + corpus_edge() + corpus_audio(base) + corpus_box(base) + fuzz(opts) + failures = 0 + total = 0 + for label, func, args, case_env in CASES: + if opts.filter and opts.filter not in label and opts.filter not in func: + continue + total += 1 + cwd = os.getcwd() + case_env = dict(case_env or {}) + template_state = case_env.pop("FAKE_TEMPLATE_STATE", None) + saved_env = {k: os.environ.get(k) for k in list(case_env) + ["FAKE_DIR"]} + for key, value in case_env.items(): + os.environ[key] = value + + def fake_dir(side): + if template_state is None: + return None + path = os.path.join(base, "fake_%s_%d" % (side, total)) + os.makedirs(path) + if template_state: + with open(os.path.join(path, "state"), "w") as handle: + handle.write(template_state) + os.environ["FAKE_DIR"] = path + return path + + def fake_log(path, out): + if path is None: + return out + log = "" + if os.path.exists(os.path.join(path, "calls.log")): + log = open(os.path.join(path, "calls.log"), encoding="utf-8", errors="replace").read() + log = re.sub(r"/[^ \n]*/(tai-box-\w+|tmp\w+)", "", log) + return out + b"\n--calls--\n" + log.encode("utf-8") + pydir = fake_dir("py") + try: + want = fake_log(pydir, b(PY[func](*args))) + except Exception as exc: # the Python function raised: the driver must not crash either + want = b("PYEXC " + type(exc).__name__) + cppdir = fake_dir("cpp") + try: + got = fake_log(cppdir, cpp(func, *args)) + except subprocess.TimeoutExpired: + got = b"TIMEOUT" + os.chdir(cwd) + for key, value in saved_env.items(): + if value is None: + os.environ.pop(key, None) + else: + os.environ[key] = value + if func == "sess_current": + norm = lambda raw: re.sub(rb"(-e\d+-)[0-9a-f]{4}", rb"\1XXXX", raw) + got, want = norm(got), norm(want) + if got != want: + failures += 1 + print("FAIL %s %r" % (label, [a[:80] for a in args])) + print(" want: %r" % want[:400]) + print(" got : %r" % got[:400]) + elif opts.v: + print("ok %s" % label) + if not opts.filter: + failures += spinner_cases(opts) + failures += burst_cases() + if opts.live: + failures += live_edge() + print("%d cases, %d failures" % (total, failures)) + return 1 if failures else 0 + finally: + shutil.rmtree(base, ignore_errors=True) + + +ALPHABET = list("ab c\n\t`*_~#>-+|[]()!\\:0123456789.\u00e9\u4e2d") + ["\x1b[31m", "\x1b[0m", " ", "**", "__", "~~", "```", "- ", "1. ", "> ", "| a |", "|---|", "
", "](u)", "[x](", "![", "def ", "'''", '"""', "# c", "@", "\r"] + + +def fuzz(opts): + rng = random.Random(opts.seed) + + def text(maxlen=60): + return "".join(rng.choice(ALPHABET) for _ in range(rng.randint(0, maxlen))) + + for _ in range(opts.fuzz): + t = text() + case("md_inline", t, label="fuzz:md_inline") + case("markdown", t, rng.choice((10, 30, 80)), label="fuzz:markdown") + case("wrap", t, rng.choice((5, 12, 40)), rng.choice(("", "> ")), rng.choice(("", " ")), label="fuzz:wrap") + case("trunc_ansi", t, rng.randint(0, 30), label="fuzz:trunc_ansi") + case("highlight", t, label="fuzz:highlight") + case("truncate", t, rng.randint(0, 20), rng.randint(0, 20), label="fuzz:truncate") + case("chunk", t, rng.randint(1, 4), rng.randint(0, 2), rng.randint(5, 50), label="fuzz:chunk") + case("collapse", t, rng.randint(1, 4), label="fuzz:collapse") + case("strip_ansi", t, label="fuzz:strip_ansi") + old = "\n".join(text(12) for _ in range(rng.randint(0, 8))) + new = "\n".join(rng.choice(old.split("\n") + [text(12)]) if old else text(12) for _ in range(rng.randint(0, 8))) + case("diff", rng.choice(("f.py", "f.txt")), old, new, 60, rng.choice((0, 1)), label="fuzz:diff") + case("g_ellipsis", t, rng.randint(0, 30), rng.choice((0, 1)), label="fuzz:ellipsis") + case("json", rng.choice(JSON_SAMPLES[:12]) if rng.random() < 0.3 else t, label="fuzz:json") + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/taicpp/tests/shell_parity_driver.cpp b/taicpp/tests/shell_parity_driver.cpp new file mode 100644 index 0000000..a9433db --- /dev/null +++ b/taicpp/tests/shell_parity_driver.cpp @@ -0,0 +1,266 @@ +// shell_parity_driver: runs one shell/util function of the C++ port per invocation and prints a +// canonical text form. tests/shell_parity.py runs the same function of ../tai.py, prints the same +// form, and diffs the two byte for byte. +#include "ansi.h" +#include "audio.h" +#include "diffrender.h" +#include "edge.h" +#include "glow.h" +#include "jsonvalue.h" +#include "lanes.h" +#include "livestream.h" +#include "markdown.h" +#include "pyhighlight.h" +#include "runlive.h" +#include "sandbox.h" +#include "session.h" +#include "spinner.h" +#include "targets.h" +#include "text.h" +#include "toolargs.h" +#include "utf8.h" +#include +#include +#include +#include +#include +using namespace detz; +using namespace detz::shell; + +namespace { +const char kRs = '\x1e', kUs = '\x1f'; +std::string J(const std::string& s) { return JsonValue::String(s).Dump(); } +std::string Join(const std::vector& v, const std::string& sep) { return Utf8::Join(v, sep); } +void Out(const std::string& s) { std::fwrite(s.data(), 1, s.size(), stdout); } +long L(const std::string& s) { return std::strtol(s.c_str(), nullptr, 10); } +double D(const std::string& s) { return std::strtod(s.c_str(), nullptr); } +std::optional Opt(const std::string& s) { if (s == "\x01NONE") return std::nullopt; return s; } + +std::string FromHex(const std::string& h) { + std::string o; + for (size_t i = 0; i + 1 < h.size(); i += 2) o.push_back((char)std::strtol(h.substr(i, 2).c_str(), nullptr, 16)); + return o; +} +std::string ToHex(const std::string& raw) { + static const char* d = "0123456789abcdef"; + std::string o; + for (unsigned char c : raw) { o.push_back(d[c >> 4]); o.push_back(d[c & 15]); } + return o; +} +struct MemorySocket : EdgeSocket { + std::string inbox, sent; + void SendAll(const std::string& data) override { sent += data; } + std::string Recv(size_t max) override { std::string r = inbox.substr(0, max); inbox.erase(0, r.size()); return r; } + void Close() override {} +}; +long gUuidCounter = 0; +void FixEnv(double now) { + gUuidCounter = 0; + EdgeSpeech::env().now = [now] { return now; }; + EdgeSpeech::env().randomBytes = [](size_t n) { std::string o; for (size_t i = 0; i < n; ++i) o.push_back((char)((i * 7 + 1) % 256)); return o; }; + EdgeSpeech::env().uuidHex = [] { char b[40]; std::snprintf(b, sizeof b, "%032lx", ++gUuidCounter); return std::string(b); }; +} + +struct RecordingStream : StreamSink { + void Feed(const std::string& line) override { Out("FEED " + J(line) + "\n"); } + void Pause() override { Out("PAUSE\n"); } + void Close(const std::string&) override {} +}; + +AgentRow RowFrom(const JsonValue& v) { + AgentRow r; + auto num = [&](const char* k) { const JsonValue* x = v.Get(k); return x ? x->Str() : std::string(); }; + r.id = L(num("id")); + r.status = v.Get("status") ? v.Get("status")->Str() : ""; + r.task = v.Get("task") && !v.Get("task")->is_null() ? v.Get("task")->Str() : ""; + r.elapsed = D(num("elapsed")); + if (v.Get("tools") && !v.Get("tools")->is_null()) r.tools = L(num("tools")); + r.collected = v.Get("collected") && v.Get("collected")->Truthy(); + return r; +} +} // namespace + +int main(int argc, char** argv) { + std::vector a(argv + 1, argv + argc); + if (a.empty()) return 2; + for (auto& item : a) { // arguments beyond the kernel's per-argument limit arrive as "@@FILE:" + if (item.rfind("@@FILE:", 0) == 0) { + FILE* fh = std::fopen(item.c_str() + 7, "rb"); + if (!fh) return 2; + item.clear(); + char buf[65536]; + size_t n; + while ((n = std::fread(buf, 1, sizeof buf, fh)) > 0) item.append(buf, n); + std::fclose(fh); + } + } + const std::string f = a[0]; + auto arg = [&](size_t i) -> std::string { return i < a.size() ? a[i] : std::string(); }; + if (f == "truncate") Out(Truncate(arg(1), L(arg(2)), L(arg(3)))); + else if (f == "capped") { + Capped c = CappedText(arg(2) == "none" ? std::nullopt : std::optional(arg(1))); + Out(std::string(c.present ? "1" : "0") + std::string(1, kRs) + std::to_string(c.size) + std::string(1, kRs) + c.text); + } else if (f == "chunk") Out(Join(ChunkLines(arg(1), (int)L(arg(2)), (int)L(arg(3)), (int)L(arg(4))), std::string(1, kRs))); + else if (f == "collapse") Out(CollapseRepeatedLines(arg(1), (int)L(arg(2)))); + else if (f == "tags") { auto r = SplitLeadingTags(arg(1)); Out(Join(r.first, std::string(1, kUs)) + std::string(1, kRs) + r.second); } + else if (f == "mentions") Out(ExpandFileMentions(arg(1), arg(2))); + else if (f == "complete") Out(Join(CompleteFiles(arg(1), arg(2), (size_t)L(arg(3))), "\n")); + else if (f == "human_size") Out(HumanSize(L(arg(1)))); + else if (f == "stream_status") Out(StreamStatus((int)L(arg(1)), arg(2), D(arg(3)), arg(4) == "1")); + else if (f == "duration") { long v = ParseDuration(arg(1)); Out(v < 0 ? "None" : std::to_string(v)); } + else if (f == "delay") Out(FormatDelay(L(arg(1)))); + else if (f == "tokens") Out(std::to_string(EstimateTokens(arg(1)))); + else if (f == "as_int") Out(std::to_string(AsInt(arg(1), L(arg(2))))); + else if (f == "toolname") Out(ToolNameClean(arg(1))); + else if (f == "target_paths") Out(Join(ShellTargets::Paths(arg(1), arg(2)), "\n")); + else if (f == "segment") Out(Join(ShellTargets::Segment(std::vector(a.begin() + 1, a.end())), "\n")); + else if (f == "curl_hint") Out(CurlHint(arg(1))); + else if (f == "strip_ansi") Out(AnsiText::Strip(arg(1))); + else if (f == "wrap") Out(Join(AnsiText::Wrap(arg(1), L(arg(2)), arg(3), arg(4)), "\n")); + else if (f == "trunc_ansi") Out(AnsiText::Truncate(arg(1), L(arg(2)))); + else if (f == "pad_tinted") Out(AnsiText::PadTinted(arg(1), L(arg(2)))); + else if (f == "style_span") Out(AnsiText::StyleSpan(arg(1), {arg(2), arg(3)})); + else if (f == "md_inline") Out(Markdown::Inline(arg(1))); + else if (f == "markdown") Out(Markdown::Render(arg(1), L(arg(2)), 1)); + else if (f == "diff") Out(DiffRender::Render(arg(1), Opt(arg(2)), Opt(arg(3)), L(arg(4)), arg(5) == "1")); + else if (f == "highlight") Out(PyHighlight::Line(arg(1))); + else if (f == "g_event") Out(Glow::EventRow(arg(1), arg(2), arg(3), arg(4), L(arg(5)), arg(6))); + else if (f == "g_done") Out(Glow::DoneRow(arg(1), arg(2), L(arg(3)), D(arg(4)), arg(5) == "1", L(arg(6)))); + else if (f == "g_ellipsis") Out(Glow::Ellipsis(arg(1), L(arg(2)), arg(3) == "1")); + else if (f == "g_shimmer") Out(Glow::Shimmer(arg(1), L(arg(2)), L(arg(3)))); + else if (f == "g_describe") { + CallDescription c = Glow::DescribeCall(arg(1), arg(2)); + Out(J(c.verb) + "\n" + J(c.target) + "\n" + J(c.extra) + "\n" + J(c.preview)); + } else if (f == "g_status") Out(Glow::Statusbar(arg(1), arg(2), arg(3), arg(4) == "1", L(arg(5)), L(arg(6)))); + else if (f == "g_accent") Out(Glow::AccentFor(arg(1)) + "|" + Glow::Bar(arg(1)) + "|" + Glow::IconFor(arg(1))); + else if (f == "g_hair") Out(Glow::Hairline(L(arg(1)))); + else if (f == "g_prompt") Out(Glow::PromptText()); + else if (f == "g_preview") Out(Glow::PreviewRow(arg(1), L(arg(2)))); + else if (f == "g_agent_row") { + bool ok; JsonValue v = JsonValue::Parse(arg(1), &ok); + Out(Glow::AgentRowText(RowFrom(v), L(arg(2)), arg(3), arg(4))); + } else if (f == "g_lanes") { + bool ok; JsonValue v = JsonValue::Parse(arg(1), &ok); + std::vector rows; + for (auto& item : v.items()) rows.push_back(RowFrom(item)); + Out(Join(Glow::Lanes(rows, L(arg(2)), L(arg(3)), arg(4)), "\n")); + } else if (f == "json") { + bool ok; JsonValue v = JsonValue::Parse(arg(1), &ok); + Out(ok ? v.Dump(false) + "\n" + v.Dump(true) + "\n" + v.Str() + "\n" + v.Repr() : "ERR"); + } else if (f == "tool_args") Out(ToolArgs::Text(arg(1)) + "\n" + ToolArgs::Valid(arg(1)) + "\n" + ToolArgs::Signature("name", arg(1))); + else if (f == "md5") Out(ShellSession::Md5Hex(arg(1))); + else if (f == "sess_stat") { + auto st = ShellSession::ParseStat(arg(1)); + Out(st.ok ? J(st.comm) + " " + std::to_string(st.ppid) + " " + std::to_string(st.ticks) : "None"); + } else if (f == "sess_name") Out(ShellSession::ShellName(arg(1), arg(2))); + else if (f == "sess_boot") Out(ShellSession::BootTag(arg(1))); + else if (f == "sess_current") { + EnvMap env; if (!arg(1).empty()) env.Set("TAI_SESSION", arg(1)); + ShellSession s = ShellSession::Current(&env, L(arg(2)), arg(3), D(arg(4)), L(arg(5))); + std::string chain; + for (auto& e : s.chain) chain += std::to_string(e.pid) + "," + std::to_string(e.ppid) + "," + e.comm + "," + e.argv0 + "," + e.name + "," + e.startIso + "," + e.sid + ";"; + Out(J(s.sid) + "\n" + J(s.shell) + "\n" + std::to_string(s.pid) + "\n" + J(s.started) + "\n" + (s.parent ? J(*s.parent) : "null") + "\n" + chain); + } else if (f == "live") { + // live ... : LiveStream into a StringSink + StringSink sink; + LiveStream ls((int)L(arg(1)), (int)L(arg(2)), &sink, true); + for (size_t i = 4; i < a.size(); ++i) { if (a[i] == "\x01PAUSE") ls.Pause(); else ls.Feed(a[i]); } + ls.Close(arg(3)); + Out(sink.str()); + } else if (f == "run") { + // run -- command... (answers: text, or "\001DECLINE") + double timeout = D(arg(1)); + bool useShell = arg(2) == "shell"; + bool ask = arg(3) == "1"; + size_t dash = 4; + std::vector answers; + while (dash < a.size() && a[dash] != "--") answers.push_back(a[dash++]); + std::vector cmd(a.begin() + (long)dash + 1, a.end()); + RecordingStream stream; + RunLiveOptions opt; + opt.timeout = timeout; + opt.stream = &stream; + size_t next = 0; + if (ask) opt.ask = [&](const std::string& text, bool secret) -> std::optional { + Out("ASK " + J(text) + " " + (secret ? "secret" : "plain") + "\n"); + if (next >= answers.size()) return std::nullopt; + std::string ans = answers[next++]; + if (ans.rfind("\001SLEEP", 0) == 0) { // "\001SLEEP:": the user takes a while to answer + size_t colon = ans.find(':'); + usleep((useconds_t)L(ans.substr(6, colon - 6)) * 1000); + ans = ans.substr(colon + 1); + } + if (ans == "\001DECLINE") return std::nullopt; + return ans; + }; + try { + RunResult r = RunLive::Run(useShell ? Command::Shell(cmd[0]) : Command::Argv(cmd), opt); + Out("RESULT " + std::to_string(r.code) + " " + (r.timedOut ? "timeout" : "ok") + " " + J(r.output) + "\n"); + } catch (const std::exception& e) { + Out(std::string("ERROR ") + J(e.what()) + "\n"); + } + } else if (f == "box") { + ::dup2(1, 2); // the Python side captures stdout and stderr into one stream + const std::string sc = arg(1), engine = arg(2); + auto text = [](const std::string& bytes) { return Utf8::Encode(Utf8::Decode(bytes)); }; + try { + if (sc == "engine") Out(Sandbox::ContainerEngine()); + else if (sc == "state") Out(Sandbox::BoxState(engine)); + else if (sc == "ensure") Out("OK " + Sandbox::EnsureBox()); + else if (sc == "caps") Out(Sandbox::UpdateBoxCaps(engine) ? "True" : "False"); + else if (sc == "python") Out(Sandbox::BoxPython(engine)); + else if (sc == "stt") Out("OK " + Sandbox::BoxTranscribe(engine, arg(3))); + else if (sc == "tts") Out("OK " + ToHex(Sandbox::BoxSpeak(engine, arg(3), arg(4)))); + else if (sc == "ocbin") Out(Sandbox::BoxOpencodeBinary(engine) ? "True" : "False"); + else if (sc == "ocrun") { + CompletedProcess d = Sandbox::BoxOpencodeExec(engine, arg(3), arg(4), D(arg(5)), "build"); + Out("PARSE|" + std::to_string(d.returncode) + "|" + text(d.out) + "|" + text(d.err)); + } else if (sc == "exec") { + CompletedProcess d = Sandbox::BoxExec(engine, std::vector(a.begin() + 3, a.end())); + Out("RC " + std::to_string(d.returncode) + " " + J(text(d.out)) + " " + J(text(d.err))); + } + } catch (const BackendError& e) { Out(std::string("ERR ") + e.what()); + } catch (const TimeoutExpired& e) { Out(std::string("TIMEOUT ") + e.what()); + } catch (const OsError& e) { Out(std::string("OSERR ") + e.what()); + } + } else if (f == "edge_token") { FixEnv(D(arg(2))); Out(EdgeSpeech::Token(L(arg(1)))); + } else if (f == "edge_ts") { FixEnv(D(arg(1))); Out(EdgeSpeech::Timestamp()); + } else if (f == "edge_ssml") Out(EdgeSpeech::Ssml(arg(1), arg(2))); + else if (f == "edge_chunks") Out(Join(EdgeSpeech::Chunks(arg(1), (size_t)L(arg(2))), std::string(1, kRs))); + else if (f == "ws_send") { FixEnv(0); MemorySocket m; EdgeSpeech::WsSend(m, arg(1)); Out(ToHex(m.sent)); + } else if (f == "ws_recv") { + FixEnv(0); MemorySocket m; m.inbox = FromHex(arg(1)); + try { auto r = EdgeSpeech::WsRecv(m); Out("OK " + std::to_string(r.first) + " " + ToHex(r.second)); } + catch (const BackendError& e) { Out(std::string("ERR ") + e.what()); } + Out("|sent=" + ToHex(m.sent)); + } else if (f == "edge_skew") { FixEnv(D(arg(2))); Out(std::to_string(EdgeSpeech::Skew(arg(1)))); + } else if (f == "edge_session") { + FixEnv(D(arg(4))); MemorySocket m; m.inbox = FromHex(arg(1)); + try { std::string audio = EdgeSpeech::SynthesizeOn(m, arg(2), arg(3)); Out("OK " + ToHex(audio)); } + catch (const BackendError& e) { Out(std::string("ERR ") + e.what()); } + Out("|sent=" + ToHex(m.sent)); + } else if (f == "edge_live") { + try { std::string audio = EdgeSpeech::Synthesize(arg(1), arg(2)); Out("OK " + std::to_string(audio.size()) + " " + ToHex(audio.substr(0, 3))); } + catch (const std::exception& e) { Out(std::string("ERR ") + e.what()); } + } else if (f == "listen") Out(Audio::Listen(L(arg(1)), arg(2))); + else if (f == "play") Out(Audio::Play(arg(1)) ? "True" : "False"); + else if (f == "spinner") { + // spinner