diff --git a/research/INDEX.md b/research/INDEX.md index 9c59439..9fcd1b3 100644 --- a/research/INDEX.md +++ b/research/INDEX.md @@ -7,10 +7,10 @@ | Metric | Value | |---|---| | Subjects | 18 | -| Total source documents | 1183 | -| — Tier 1 (Source Dossier, independently analyzed) | 311 | +| Total source documents | 1192 | +| — Tier 1 (Source Dossier, independently analyzed) | 320 | | — Tier 2 (Source Citation Card, listed only) | 872 | -| Total conclusion documents | 45 | +| Total conclusion documents | 47 | | Archive opened | 2026-10-09 | | Last updated | 2026-10-27 | @@ -33,7 +33,7 @@ | [`cap-protocol`](cap-protocol/) | 83 (34 / 49) | 4 | 2026-10-18 | 2026-10-25 | Supporting CAP (CLI Agent Protocol) as agent and orchestrator: full read of core + coding profile, quickstart, STATUS, all 5 manifests; selective A2A/ACP wire reads; decided phased rollout without gRPC plus dual A2A spelling; extended 2026-10-25 with model handling, model-as-tool and skill-management research (ACP config options, MCP sampling, agents-as-tools, advisor tools, subagent and skill docs, RouteLLM), 2 recommendation-only conclusions, nothing built | | [`mcp-client-support`](mcp-client-support/) | 106 (38 / 68) | 3 | 2026-10-18 | 2026-10-18 | Designing tai's MCP client: wire spec (stdio + Streamable HTTP on the 2025-11-25 baseline), pasted-config/GitHub-URL resolver, and the trust/approval model; decided minimal stdlib client, normalizer plus resolver agent, untrusted-until-consent with description pinning, no OAuth in v1, nothing built yet | | [`fuzzy-matching-selectors`](fuzzy-matching-selectors/) | 65 (12 / 53) | 2 | 2026-10-25 | 2026-10-25 | Making `@agent` / `@model` mentions pick the most related candidate: fzf/fzy/Sublime/clangd subsequence scoring, Damerau-Levenshtein, pg_trgm, RapidFuzz WRatio, difflib, CPython suggestions surveyed; decided one generic `FuzzyMatcher` (four signals, deterministic ties, floor 0.3 across agents and models) and implemented it the same day | -| [`whatsapp-automation`](whatsapp-automation/) | 75 (14 / 61) | 7 | 2026-10-27 | 2026-10-28 | Making detz send WhatsApp messages from C++ on a personal number: no native Python library exists (neonize is whatsmeow over ctypes, yowsup is dead), no browser needed, decided a native C++ transpile from wa-go (MIT, golden vectors) with whatsmeow as referee, GPL libsignal avoided, plus a whatsapp-mcp shortcut; specified in taicpp/docs/whatsapp.md, nothing built | +| [`whatsapp-automation`](whatsapp-automation/) | 84 (23 / 61) | 9 | 2026-10-27 | 2026-10-28 | Making detz send WhatsApp messages from C++ on a personal number: no native Python library exists (neonize is whatsmeow over ctypes, yowsup is dead), no browser needed, decided a native C++ transpile from wa-go (MIT, golden vectors) with whatsmeow as referee, GPL libsignal avoided, plus a whatsapp-mcp shortcut; specified in taicpp/docs/whatsapp.md, nothing built | | [`safe-cpp-practices`](safe-cpp-practices/) | 45 (8 / 37) | 2 | 2026-10-27 | 2026-10-27 | How to write safe, fast, OO production C++ while keeping dev speed, triggered by a `std::vector larger than max_size()` crash in the WhatsApp wire parser; decided a four-layer playbook (overflow-safe checked-reader parsing, cheap always-on release hardening, an ASan+UBSan+_GLIBCXX_ASSERTIONS test net, and frame capture), no new dependency; applied to detz::wa and documented in docs/cpp-safety.md. The sanitizer net prescribed by the research then caught a second UB (a varint shift >= 64 bits) that manual review had missed twice | | [`terminal-screenshot-rendering`](terminal-screenshot-rendering/) | 99 (10 / 89) | 2 | 2026-10-27 | 2026-10-27 | Making detz take a PNG of the current terminal on a headless server for WhatsApp: surveyed freeze, termshot, textimg, libansilove, agg, aha, silicon, tmux capture-pane -e, bitmap fonts (Spleen, Terminus, Unifont, X11 fixed) and how Baileys, whatsmeow and whatsapp-rust set image mimetypes; decided an in-process renderer (tmux escapes, TuiVtScreen, embedded Spleen 8x16, zlib PNG) and image/png without a JPEG encoder; implemented the same session | @@ -73,6 +73,10 @@ | 2026-10-27 | `whatsapp-automation` | New subject: 70 sources (9 Tier 1 / 61 Tier 2) + 3 conclusions, from 12 searches, 9 repositories shallow-cloned and read (whatsmeow, libsignal-protocol-go, wa-go, Baileys, neonize, whatsapp-rust, whatsapp-mcp, yowsup, libsignal-protocol-c), line counts and GitHub API activity measured, and one live check (libcurl got 101 from wss://web.whatsapp.com/ws/chat). Mirror and language copies of one page are logged as one card each, with every URL named in the card. Decisions: no native Python library exists and no browser is needed; native C++ `detz::wa` transpiled from wa-go (MIT) with whatsmeow (MPL-2.0) as referee and GPL-3.0 libsignal avoided; whatsapp-mcp through detz's MCP client as the two-day shortcut. Specified in taicpp/docs/whatsapp.md, nothing built. Carried gaps: an unpaired whatsmeow QR probe was refused by the session permission classifier, so the full handshake is unverified here; XEdDSA implementation, LID addressing, client-version policy and ban behaviour are open deep-research items. | | 2026-10-27 | `terminal-screenshot-rendering` | New subject: 99 sources (10 Tier 1 / 89 Tier 2) + 2 conclusions, from 12 searches, 5 page fetches, the Spleen BDF and LICENSE download, 3 source files read in the cloned Baileys, whatsmeow and whatsapp-rust repositories, the local console-setup font copyright file, and live tmux 3.6 captures. Mirror and version copies of one page are one card each, with the URL count in the card title. Decisions: render in-process (capture-pane -e -p -N, TuiVtScreen, embedded Spleen 8x16 BSD-2, zlib PNG); send as image/png, no JPEG encoder. Implemented in taicpp (src/termshot/, terminal_screenshot tool, tests/termshot_test.cpp, docs/termshot.md) and verified with a real model calling the tool. Carried gaps: glyphs outside Spleen render as '?', wide characters take one column, the native plexer path and a live WhatsApp send of image/png are unverified. | | 2026-10-27 | `safe-cpp-practices` | New subject: 45 sources (8 Tier 1 / 37 Tier 2) + 2 conclusions, from 6 extended web searches on safe production C++ (OpenSSF compiler hardening, C++ Core Guidelines bounds/GSL/narrow_cast, std::span no-default-bounds-check, integer-overflow size checks, libFuzzer+ASan/UBSan CI, sanitizer overhead ladder, clang-tidy ratchet, GCC -fhardened). Triggered by incident WA-2026-10-27, the `std::vector larger than max_size()` crash in the WhatsApp wire parser. Decisions: a four-layer playbook (overflow-safe checked-reader parsing with `len <= size - pos`; cheap always-on release hardening; an ASan+UBSan+_GLIBCXX_ASSERTIONS test net via tools/san-whatsapp.sh; frame capture as a regression corpus), no new runtime dependency. Applying the research's own prescription (the sanitizer net) immediately caught a second, independent UB (a varint shift of up to 70 bits) that two manual reviews had missed; fixed and guarded. Documented in docs/cpp-safety.md, docs/whatsapp.md section 13, AGENTS.md. Carried gaps: the exact live frame from the user is still unreproduced (capture hook added); clang-tidy narrowing-conversion ratchet not yet wired; ed25519 ref10 excluded from UBSan for intentional shift idioms. | +| 2026-10-27 | `whatsapp-automation` | Key-share incident (taicpp problem.md, docs section 16): 5 Tier 1 dossiers (whatsmeow PR 1268 rollback, Volt-Chat PR 5 session-first peer sends, wacli PR 443 empty shares, Rich627 PR 45 peer-stanza classification, whatsmeow main peer/own-sender/receipt/pairing read) + 1 conclusion (harden locally, wait 300 s online, ship the VCF fallback). Implemented in taicpp detz 1.46.0 (rollback, verified peer send, LID senders, empty-share handling, notification/receipt handling, --whatsapp-sync, --whatsapp-import), unit-tested offline. Carried gaps: the phone side is still unseen (live --whatsapp-sync with an awake phone pending); retry-resend and fatal-recovery snapshot requests deferred. | +| 2026-10-27 | `whatsapp-automation` | Push-name contact learning: 1 Tier 1 dossier (on-disk whatsmeow read: notify push names on every message, no name query for regular users, separate-column precedence) + 1 conclusion (four-source resolution stack, conversations not stored). Implemented in taicpp detz 1.46.1 (push_name_candidate, put_contact_if_absent), unit-tested offline. Carried gaps: nicknames go stale (alias overwrites); silent contacts need VCF/alias/app-state. | +| 2026-10-27 | `whatsapp-automation` | Full surface audit: 1 Tier 1 dossier comparing all 43 whatsmeow root files against detz 1.46.1 (mechanical grep both sides). Verdict: the 1:1 companion core is done; groups, inbox, presence, receipts, history, app-state send, pair-code, logout, identity store, business/newsletter/calls/status and all non-text/image types are missing or partial. No code changed. | +| 2026-10-27 | `whatsapp-automation` | Transpilation methodology for the full-port plan: 2 Tier 1 dossiers (power-mapper inventory-plus-verification-script coverage guarantees; Livshits TRACTOR survey, never-let-the-model-grade-its-own-homework plus VERT oracle/counterexample repair). Shaped taicpp/full_transpilation_and_integration.md (M0..M10, coverage and stub gates, oracle differential tests). No code changed. | ## How to use this index - Before starting new research, search this table for an existing subject before opening a new one (see [README.md](README.md#2-directory-layout) §2, "one subject, one home"). diff --git a/research/whatsapp-automation/ben-livshits-automating-c-to-safe-rust.md b/research/whatsapp-automation/ben-livshits-automating-c-to-safe-rust.md new file mode 100644 index 0000000..b1b8eee --- /dev/null +++ b/research/whatsapp-automation/ben-livshits-automating-c-to-safe-rust.md @@ -0,0 +1,73 @@ +# Livshits: never let the model grade its own homework (transpilation verification) + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | whatsapp-automation | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | Automating C to Safe Rust Translation | +| Source URL | https://www.ben-livshits.org/blog/c-to-rust | +| Source publisher / author | Dr. Ben Livshits | +| Source type | blog post | +| Source date (published) | 2026-09-18 | +| Date researched | 2026-10-27 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | direct page fetch | +| Confidence | High (long-form technical argument with named systems, read in full) | +| Status | Active | + +## Summary + +Surveying DARPA TRACTOR-era C-to-Rust systems, the article argues both pure +approaches fail: mechanical transpilation preserves behavior but also all +unsafety, while pure LLM translation writes idiomatic code the compiler +often rejects. Four neuro-symbolic systems converge on one pattern: the +model writes candidates and strict deterministic checking verifies them. +The load-bearing rule is "never let the model grade its own homework", and +"verified" in every system means verified for the code the pipeline +accepted within its bounds, never for the codebase as a whole. + +## Key findings + +- VERT pattern: LLM candidate checked against a compiled oracle by a + differential test suite; each divergence returns a concrete counterexample + input that is embedded into the next repair prompt. +- SACTOR pattern: FFI-linked test harness running the same tests against + original and candidate. +- Verification strength rises as unit scope shrinks: interface-level and + per-function checks carry the strongest guarantees in the survey's + comparison table. +- Bounded proofs cover every input only up to their bound; the article is + explicit that this is not unbounded equivalence, and honest systems state + the bound. + +## Pros / strengths + +- Names the exact failure mode of agent-driven ports (self-graded tests that + encode the author's assumption) and the exact fix (independent oracle plus + counterexample repair). +- Calibrates "verified" honestly instead of overselling: bounded, scoped to + accepted code. + +## Cons / limitations / caveats + +- C-to-Rust setting; the oracle mechanics transfer, the language-specific + tooling (Kani, Verus, Bolero) does not. +- Success-rate numbers cited are the systems' own benchmarks, reported with + appropriate skepticism by the author. + +## Relevance to tai + +Directly shaped the full-transpilation prompt: Go oracle differential tests +as the independent checker, the explicit no-self-grading rule with +counterexample feedback, small per-milestone API slices, and the honestly +stated bounded-verification limit in the plan's Risks. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-27 | Initial dossier created. | diff --git a/research/whatsapp-automation/conclusions/key-share-never-arrives-fixes.md b/research/whatsapp-automation/conclusions/key-share-never-arrives-fixes.md new file mode 100644 index 0000000..88b9622 --- /dev/null +++ b/research/whatsapp-automation/conclusions/key-share-never-arrives-fixes.md @@ -0,0 +1,53 @@ +# Conclusion: harden the key-share path locally, wait longer online, ship the VCF fallback + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | whatsapp-automation | +| Document type | Conclusion | +| Tier | N/A | +| Date decided | 2026-10-27 | +| Decided by | retoor, advised by tai (agent, Muse Spark) | +| Status | Active | +| Confidence | High for the local fixes (verified against the reference and by unit tests); Medium for the phone-side diagnosis (the phone is still unseen) | + +## Question / problem statement + +A newly linked detz companion logs in, sends, and matches the server's participant hash, but the phone never sends the app-state sync key, so no contact names arrive (taicpp `problem.md`). Is the request malformed, is the receive path dropping the answer, or is the phone gating on its side, and what should change in detz? + +## Conclusion + +The detz request path already matches whatsmeow byte for byte, so no protocol fix was needed; instead detz now (1) removes every local way to lose the exchange (rollback on failed send, verified peer send, LID senders accepted, empty shares distinguished, notifications and receipts handled), (2) stays online long enough for a sleeping phone (`--whatsapp-sync`, 300 s), and (3) ships the offline fallback (`--whatsapp-import` for a .vcf exported on the phone). If a 300 s trace with WhatsApp open on the phone still shows no ``, the gate is on the phone's side (asleep, battery-restricted, or one-shot push missed) and the VCF covers the user need. + +## Supporting sources + +- [whatsmeow-peer-session-and-post-login-flow](../whatsmeow-peer-session-and-post-login-flow.md): the referee read; supplied the ErrNoSession premise, LID-own-sender rule, receipt strings, dirty no-op, and pairing-extras verdict. +- [github-tulir-whatsmeow-pull-1268](../github-tulir-whatsmeow-pull-1268.md): rollback of the 24 h request mark on send failure. +- [github-volt-chat-whatsmeow-pull-5](../github-volt-chat-whatsmeow-pull-5.md): a peer message needs a Signal session first; verify it exists. +- [github-openclaw-wacli-pull-443](../github-openclaw-wacli-pull-443.md): answered-but-empty is a third state, distinct from silence. +- [github-rich627-whatsapp-claude-plugin-pull-45](../github-rich627-whatsapp-claude-plugin-pull-45.md): sync traffic needs its own classification rule; silent drops are total. + +## Alternatives considered and rejected + +- Rewriting the peer request bytes: rejected, the bytes already match whatsmeow (verified by the existing oracle test), so a rewrite could only add drift. +- Waiting indefinitely on every login: rejected, logins must stay fast for sends; the long wait is opt-in via `--whatsapp-sync`. +- Full retry-resend and fatal-recovery snapshot requests: deferred, not rejected; logged as known gaps in docs section 14. The phone side must first be observed answering something before building answers to its answers. +- Assuming a sleeping phone and changing nothing: rejected, the audit found eight real local gaps worth closing regardless. + +## Risks / open questions + +- The phone may push keys only once per link and ignore re-requests (one project's experience); then a missed first push needs a re-link, and only a fresh-link trace with the new arrival logging will show it. +- The 300 s wait may still be short for a deeply restricted phone; the wait is capped at one hour via the wizard parameter for that case. +- whatsmeow main moves fast; the referee read is a 2026-10-27 snapshot and must be re-verified on the next bump. + +## Action taken + +Implemented in taicpp detz 1.46.0 (src/whatsapp, CLI flags, agent tool hint), covered by `build/whatsapp_test` (4 new sections), documented in docs/whatsapp.md sections 14 and 16 and `problem.md`. Live verification (a `--whatsapp-sync` run with the phone awake) is still pending. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-27 | Initial conclusion recorded. | diff --git a/research/whatsapp-automation/conclusions/push-name-contact-learning.md b/research/whatsapp-automation/conclusions/push-name-contact-learning.md new file mode 100644 index 0000000..9b85808 --- /dev/null +++ b/research/whatsapp-automation/conclusions/push-name-contact-learning.md @@ -0,0 +1,47 @@ +# Conclusion: learn contact names from push names; the stack is complete with four sources + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | whatsapp-automation | +| Document type | Conclusion | +| Tier | N/A | +| Date decided | 2026-10-27 | +| Decided by | retoor, advised by tai (agent, Muse Spark) | +| Status | Active | +| Confidence | High | + +## Question / problem statement + +The user wants "send a message to Anne with a picture of my terminal" to work for any contact, but the phone never sent the app-state address book. Are contact names available from anywhere else, and what is the complete name-resolution story? + +## Conclusion + +Name resolution stands on four sources, in precedence order: app-state saved names, VCF import, user aliases, and push names learned from incoming messages (fill-if-absent, so they never overwrite the other three). No fifth source exists for regular users: the reference confirms no server query returns their names. Conversations themselves are not synced or stored anywhere in detz; only the sender names are learned, which is all name resolution needs. + +## Supporting sources + +- [whatsmeow-push-names-and-contact-queries](../whatsmeow-push-names-and-contact-queries.md): notify push names on every message, no name query for regular users, separate-column precedence. + +## Alternatives considered and rejected + +- A separate push_name column (whatsmeow's exact schema): rejected, fill-if-absent on the single name column gives the same precedence with no migration; both names are never needed at once for resolution. +- Storing full conversations to "know" contacts: rejected, message bodies are not needed for name resolution and an inbox is a separate feature, not built. +- Waiting for app-state only: rejected, the phone may never send it; three offline sources plus learning cover the user need now. + +## Risks / open questions + +- Push names are nicknames and go stale; a wrong learned name is fixed by storing an alias, which overwrites it. +- Contacts who never message while detz is online stay unknown until VCF/alias/app-state names them. + +## Action taken + +Implemented in taicpp detz 1.46.1 (`push_name_candidate`, `put_contact_if_absent`), covered by `build/whatsapp_test`, documented in docs/whatsapp.md section 14. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-27 | Initial conclusion recorded. | diff --git a/research/whatsapp-automation/github-openclaw-wacli-pull-443.md b/research/whatsapp-automation/github-openclaw-wacli-pull-443.md new file mode 100644 index 0000000..8b427c7 --- /dev/null +++ b/research/whatsapp-automation/github-openclaw-wacli-pull-443.md @@ -0,0 +1,51 @@ +# openclaw/wacli PR 443: recover app state when the phone shares a key without data + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | whatsapp-automation | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | fix(sync): recover app state when the phone shares a key without data (PR 443) | +| Source URL | https://github.com/openclaw/wacli/pull/443 | +| Source publisher / author | Jarek Zmudzinski | +| Source type | repository | +| Source date (published) | 2026-09-26 | +| Date researched | 2026-10-27 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | direct page fetch (patch, first commit of seven plus headers) | +| Confidence | High for the empty-share finding, Medium for the recovery flow (later commits not read) | +| Status | Active | + +## Summary + +wacli observed the phone answering a key request with a key share whose keys carry no key data. Storing that row crashed every sync on `NOT NULL constraint failed: whatsmeow_app_state_sync_keys.key_data`. The fix stops storing data-less keys and instead recovers affected collections through a full sync plus a phone recovery snapshot, rather than waiting for a key that never arrives. A test notes whatsmeow's own fetch on connect can trigger the key request, so the empty share may arrive before the client has seen any collection fail. + +## Key findings + +- An empty key share is a real phone behavior: the key id is present, the key data is absent (the primary has nothing to give). +- Storing it is actively harmful: every later sync trips over the dataless row. +- The recovery path is full sync first, then a `COMPANION_SYNCD_SNAPSHOT_FATAL_RECOVERY` peer request (unencrypted collection copy) when the full sync still misses the key. +- Chat-state writes otherwise wait up to five minutes for a key delivery that never comes; the fix skips the wait once the primary's answer is known to be empty. + +## Pros / strengths + +- Distinguishes three states the naive code conflates: key received, answered-but-empty, never answered. Only the last one should keep waiting. +- Recovery degrades gracefully (full sync, then snapshot request) instead of erroring the whole sync. + +## Cons / limitations / caveats + +- Only the first of seven commits was read here; the snapshot-request mechanics were verified separately against whatsmeow main (`BuildAppStateRecoveryRequest`, `handleAppStateRecovery`) rather than from this PR. +- wacli-specific scaffolding (fakeWA, recovery workers) does not transfer to detz. + +## Relevance to tai + +Ported as state handling, not as the full recovery flow: `WaClient::handle_message` counts stored vs empty keys, logs an empty answer explicitly, ends the key wait (`app_keys_answered_`), and leaves the collection at version 0 so the next login retries the full sync. The fatal-recovery snapshot request is a documented gap, not built. Covered by `build/whatsapp_test` section `empty key share`. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-27 | Initial dossier created. | diff --git a/research/whatsapp-automation/github-rich627-whatsapp-claude-plugin-pull-45.md b/research/whatsapp-automation/github-rich627-whatsapp-claude-plugin-pull-45.md new file mode 100644 index 0000000..40bb5f2 --- /dev/null +++ b/research/whatsapp-automation/github-rich627-whatsapp-claude-plugin-pull-45.md @@ -0,0 +1,50 @@ +# Rich627 whatsapp-claude-plugin PR 45: contact sync on new links (peer stanza classification) + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | whatsapp-automation | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | Fix contact sync on new links, name tags and group-member retention (0.27.1) (PR 45) | +| Source URL | https://github.com/Rich627/whatsapp-claude-plugin/pull/45 | +| Source publisher / author | Kaushik | +| Source type | repository | +| Source date (published) | 2026-10-10 | +| Date researched | 2026-10-27 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | direct page fetch (patch headers and description; dependency/test-file diffs skimmed) | +| Confidence | High for the classification finding (stated plainly in the PR description), Medium for the Baileys internals (not independently read) | +| Status | Active | + +## Summary + +Baileys 7.0.0-rc12 dropped every protocol message from the phone on a new link as spoofed: its self-only guard required `fromMe`, but `decodeMessageNode` only set `fromMe` when the stanza carried a `recipient` attribute, and the phone's own peer stanzas on a new link (history sync notification, app-state key share) carry none. Result: no app-state keys, every collection failing with "failed to find key", contacts never arriving. Bumping to rc14 (whose guard accepts the phone's own sync stanzas) plus a regression test fixed it. The PR also notes WhatsApp otherwise sends saved names just once, at linking. + +## Key findings + +- Peer stanzas from the phone on a new link have no `recipient` attribute; a classifier that needs one drops exactly the sync traffic. +- The failure is silent (logged as spoofed at best) and total: no keys, no contacts. +- The project pins the fix with a dedicated peer-stanza test. + +## Pros / strengths + +- A clean demonstration that sync traffic needs its own classification rule, not the chat-message rule. +- The "sent just once, at linking" observation explains why a dropped first push is unrecoverable by re-request in some stacks. + +## Cons / limitations / caveats + +- Baileys-specific (Baileys rc versions, `process-message.js`); the exact guard code was not read here. +- "Just once" is that project's operational experience, contradicted in the healthy case by whatsmeow stacks (wacli sees answers to re-requests); most likely the phone stops pushing only to companions that never consume the first push. + +## Relevance to tai + +detz had the same class of bug in a different form: the own-sender filter accepted the PN user only, so a key share arriving from the phone's LID would have been silently dropped. `WaClient::is_own_sender` now accepts the PN or LID user (whatsmeow `IsFromMe` semantics, verified against whatsmeow main), and every arrival is logged before the filter. Covered by `build/whatsapp_test` section `peer message helpers`. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-27 | Initial dossier created. | diff --git a/research/whatsapp-automation/github-richyparr-power-mapper.md b/research/whatsapp-automation/github-richyparr-power-mapper.md new file mode 100644 index 0000000..9956764 --- /dev/null +++ b/research/whatsapp-automation/github-richyparr-power-mapper.md @@ -0,0 +1,65 @@ +# power-mapper: coverage guarantees via inventory plus verification script + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | whatsapp-automation | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | power-mapper: 100% codebase coverage auditing for Claude Code | +| Source URL | https://github.com/richyparr/power-mapper | +| Source publisher / author | richyparr | +| Source type | repository | +| Source date (published) | Unknown — not stated | +| Date researched | 2026-10-27 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | direct page fetch | +| Confidence | Medium (project README claims read; the tool itself was not run) | +| Status | Active | + +## Summary + +power-mapper audits a codebase in tiers (inventory, per-file analysis, +domain synthesis, cross-cuts, executive synthesis) so no single agent must +hold the whole codebase in context, then runs a verification tier: a script +cross-references every file in the inventory against every analysis output, +flags anything below 100% coverage, and re-runs analysis for missed files. +The stated result is a guarantee, not a guess. + +## Key findings + +- Tiered compression (~4x per tier) keeps each agent's context bounded while + covering codebases of any size. +- The final verification tier is a bash script, not an LLM judgment: every + inventory item must have a matching output or it is flagged and re-run. +- Outputs include feature, gap, dependency, and audit-summary documents at a + fixed quality bar regardless of codebase size. + +## Pros / strengths + +- The guarantee mechanism is exactly what long transpilation runs lack: a + mechanical cross-check instead of the agent's own impression of done. +- Explicit context-budget design (tiers, per-file units) matches how agent + context actually fails on large codebases. + +## Cons / limitations / caveats + +- README-level evidence only; no independent verification that the claimed + 100% holds on adversarial inputs. +- Built for auditing/analysis, not transpilation; the re-run loop maps to + "analyze missed files", and this project reuses the shape for "transpile + missed API". + +## Relevance to tai + +Directly shaped the full-transpilation prompt: the M0 API inventory plus +`tools/wa-coverage.sh` 100% gate plus per-milestone rescoping is this +pattern applied to whatsmeow's exported API instead of files. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-27 | Initial dossier created. | diff --git a/research/whatsapp-automation/github-tulir-whatsmeow-pull-1268.md b/research/whatsapp-automation/github-tulir-whatsmeow-pull-1268.md new file mode 100644 index 0000000..e6af426 --- /dev/null +++ b/research/whatsapp-automation/github-tulir-whatsmeow-pull-1268.md @@ -0,0 +1,51 @@ +# whatsmeow PR 1268: retry a key request that failed to send + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | whatsapp-automation | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | appstate: retry a key request that failed to send (PR 1268) | +| Source URL | https://github.com/tulir/whatsmeow/pull/1268 | +| Source publisher / author | Andrey Chichak | +| Source type | repository | +| Source date (published) | 2026-09-16 | +| Date researched | 2026-10-27 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | direct page fetch (patch) | +| Confidence | High | +| Status | Active | + +## Summary + +The PR fixes `requestMissingAppStateKeys`: it stamped each key id as requested before the peer message went out, and a failure to send was only logged, so a key whose request never left the process stayed missing for 24 hours. The fix returns an error from `requestAppStateKeys` and deletes the bookkeeping for those key ids when the send fails, so the next attempt asks again. + +## Key findings + +- `requestMissingAppStateKeys` writes `appStateKeyRequests[hex(keyID)] = now` under lock before calling `requestAppStateKeys`. +- Before the fix, `requestAppStateKeys` returned nothing; a `SendPeerMessage` error only produced a warning log. +- After the fix, on error the caller deletes exactly the filtered key ids from `appStateKeyRequests` under lock; the 24 h throttle then no longer blocks the retry. +- The same-PR change to `DangerousInternalClient.RequestAppStateKeys` (now returns error) confirms the send failure is a normal, expected outcome, not a crash path. + +## Pros / strengths + +- Tiny, obviously correct: the rollback covers exactly the ids this call marked, so concurrent marks for other ids are untouched. +- Names the real failure mode (peer message cannot be built, e.g. no Signal session) instead of papering over it. + +## Cons / limitations / caveats + +- Does not explain why the send failed; it only makes the retry possible. The underlying cause (usually a missing Signal session with the phone) is addressed separately (see Volt-Chat PR 5). +- The 24 h throttle itself is unchanged, so a phone that receives but ignores requests still gets at most one request a day per key id. + +## Relevance to tai + +Ported directly: `WaClient::request_app_state_keys` checks the `send_peer_message` result and calls the new `WaStorePort::clear_app_key_request_time` for the due ids on failure, with a loud log line. Covered by `build/whatsapp_test` section `key request rollback`. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-27 | Initial dossier created. | diff --git a/research/whatsapp-automation/github-volt-chat-whatsmeow-pull-5.md b/research/whatsapp-automation/github-volt-chat-whatsmeow-pull-5.md new file mode 100644 index 0000000..302fbbd --- /dev/null +++ b/research/whatsapp-automation/github-volt-chat-whatsmeow-pull-5.md @@ -0,0 +1,52 @@ +# Volt-Chat whatsmeow PR 5: establish Signal sessions without sending messages + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | whatsapp-automation | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | feat: establish Signal sessions without sending messages (PR 5) | +| Source URL | https://github.com/Volt-Chat/whatsmeow/pull/5 | +| Source publisher / author | Santiago Santana | +| Source type | repository | +| Source date (published) | 2026-10-23 | +| Date researched | 2026-10-27 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | direct page fetch (patch) | +| Confidence | High for the premise, Medium for the patch itself (read the prekeys.go diff; the 388-line test file was skimmed, not analyzed) | +| Status | Active | + +## Summary + +The PR states the premise outright: peer messages require an existing Signal session, but callers had no public operation to establish one without sending a message. It adds `EnsureSession(device)`: resolve a phone-number JID through the stored PN-to-LID mapping (an absent mapping is an error), reuse a stored session when present, otherwise fetch the prekey bundle and process it under the send lock. Identity trust failures are returned without deleting identities or sessions. + +## Key findings + +- Device zero selects the primary phone; only user servers (`s.whatsapp.net`, `lid`) are accepted. +- PN JIDs must resolve via the stored LID mapping first; encryption always targets the LID identity. +- An incoming message may establish the session during the prekey iq, so the store is re-checked after the fetch. +- Success establishes local encryption state only, not delivery to or acknowledgement by the remote device. +- The author notes live development-account verification remains pending. + +## Pros / strengths + +- Makes explicit what whatsmeow main only implies through `ErrNoSession`: a peer send without a session cannot work. +- Conservative trust behavior (never deletes identity/session material on this path). + +## Cons / limitations / caveats + +- A fork PR, not merged upstream at time of reading; no live verification claimed. +- Does not change what happens when the phone receives the peer message but never answers; it only guarantees the request can leave. + +## Relevance to tai + +detz already fetched prekeys before a peer send, but trusted the result without checking. Following this PR's premise, `WaClient::send_peer_message` now verifies a session exists in the store after the fetch, logs the enc type (pkmsg first contact vs msg existing session), and returns failure loudly when there is no session, which triggers the PR-1268 rollback. No separate ensure API was needed: the check lives in the send path. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-27 | Initial dossier created. | diff --git a/research/whatsapp-automation/whatsmeow-peer-session-and-post-login-flow.md b/research/whatsapp-automation/whatsmeow-peer-session-and-post-login-flow.md new file mode 100644 index 0000000..e674597 --- /dev/null +++ b/research/whatsapp-automation/whatsmeow-peer-session-and-post-login-flow.md @@ -0,0 +1,56 @@ +# whatsmeow main: peer session requirement, own-sender rule, receipts, pairing extras + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | whatsapp-automation | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | whatsmeow main: send.go, message.go, appstate.go, client.go, notification.go, connectionevents.go, pair.go, store/clientpayload.go, types/presence.go | +| Source URL | https://github.com/tulir/whatsmeow (main branch files, fetched individually) | +| Source publisher / author | Tulir Asokan and contributors | +| Source type | repository | +| Source date (published) | Unknown — not stated (snapshot of main fetched 2026-10-27) | +| Date researched | 2026-10-27 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | direct page fetch (raw.githubusercontent.com) | +| Confidence | High (functions read line by line; quotes below are paraphrases of the logic, not verbatim copies) | +| Status | Active | + +## Summary + +Read to settle why a newly linked companion's app-state key request goes unanswered. The reference is exact about the mechanics on both sides of the exchange: a peer message without a Signal session fails locally with `ErrNoSession`; the peer node carries device-identity only for a prekey message; an own sender is the PN or LID user with no `recipient` needed; consumed peer messages get a `peer_msg` receipt; `ib dirty` needs no answer; and `pair-success` children the client does not know (such as `jurisdiction` and `encryption-metadata`) are ignored. + +## Key findings + +- `preparePeerMessageNode` sends `` with `` plus one ``, `push_priority=high` only for `APP_STATE_SYNC_KEY_REQUEST`; encryption targets `GetLIDForPN(to)`; device-identity is appended only when the ciphertext is a prekey message. +- `encryptMessageForDevice` with a nil bundle and no stored session returns `ErrNoSession`: the peer request never leaves the process. Normal sends fetch prekeys for missing sessions; peer sends do not. +- `parseMessageSource` sets `IsFromMe` when the sender user equals the own PN or own LID user; the `recipient` attribute only selects the chat, it is not required for the classification. +- `handleProtocolMessage` stores every shared key (even dataless ones) and re-fetches all collections; category `peer` messages get a `` and history notifications a ``, both to the own PN. +- `handleIB` leaves `` (including `account_sync`) alone (the `MarkNotDirty` call is commented out); no client action needed. +- `handlePairSuccess` reads only the device-identity, biz, device, platform and client-props children; extra nodes are ignored. Pairing also stores the phone's LID identity and the PN-LID mapping, then sends `unified_session` telemetry. +- Receipt type strings are `peer_msg` and `hist_sync` (underscores), retry receipts arrive as ``. +- App-state fatal recovery exists as a peer request (`COMPANION_SYNCD_SNAPSHOT_FATAL_RECOVERY`) answered with an unencrypted collection copy (`handleAppStateRecovery`). +- Registration and login payloads and DeviceProps (os whatsmeow, UNKNOWN platform, the full historySyncConfig) match what detz already sends; no difference was found there. + +## Pros / strengths + +- The peer path is small and fully explicit, so a byte-level comparison with detz was possible without guessing. +- Separation of concerns (encrypt vs send vs receipt) makes each detz fix independently mappable. + +## Cons / limitations / caveats + +- Source shows the companion side only; nothing here proves what the phone does with a well-formed request (foreground/background gating is still unseen). +- Main-branch snapshot: no commit hash recorded, only the fetch date; re-verify on the next whatsmeow bump. + +## Relevance to tai + +This dossier is the referee for the whole incident (taicpp `problem.md`, docs section 16): it confirmed the detz request bytes were already correct, supplied the `ErrNoSession` premise behind the verified peer send, the LID-own-sender rule, the `peer_msg` receipt, the dirty no-op, and the pairing-extras verdict. The receipt strings and node shapes were ported verbatim. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-27 | Initial dossier created. | diff --git a/research/whatsapp-automation/whatsmeow-push-names-and-contact-queries.md b/research/whatsapp-automation/whatsmeow-push-names-and-contact-queries.md new file mode 100644 index 0000000..9f6b2ea --- /dev/null +++ b/research/whatsapp-automation/whatsmeow-push-names-and-contact-queries.md @@ -0,0 +1,54 @@ +# whatsmeow on-disk: push names, contact store, and what no query returns + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | whatsapp-automation | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | whatsmeow v0.0.0-20261027104441-1eec1136a50b: message.go, user.go, store/sqlstore/store.go (local Go module cache copy) | +| Source URL | https://github.com/tulir/whatsmeow | +| Source publisher / author | Tulir Asokan and contributors | +| Source type | repository | +| Source date (published) | 2026-10-27 (module version date) | +| Date researched | 2026-10-27 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | direct page fetch (local files in the Go module cache, read in place) | +| Confidence | High (functions read line by line) | +| Status | Active | + +## Summary + +Read to answer whether contact names are available without the app-state address book. They are, from exactly one free source: every incoming `` stanza carries the sender's self-reported push name in its `notify` attribute, and whatsmeow stores it for the sender and the sender's alternate (PN and LID) on every message. No server query returns names for regular users: the full `GetUserInfo` usync returns status, picture, devices, LID and verified business names only. The contact store keeps saved names, push names and business names in separate columns so a push name never overwrites a saved name. + +## Key findings + +- `message.go`: `info.PushName` comes from the stanza's `notify` attribute; any non-empty push name except `-` is passed to `updatePushName` for both sender and sender-alt. +- `user.go updatePushName`: stores the push name under the sender JID and, when present, the alternate JID (the PN/LID pair). +- `user.go handleHistoricalPushNames`: history-sync blobs also carry push names, stored the same way. +- `user.go GetUserInfo`: the widest usync (business verified_name, status, picture, devices v2, lid) has no name field for regular users; only businesses get a verified name. +- The `notify` hits in the QR-link resolvers (`ResolveBusinessMessageLink`, `ResolveContactQRLink`) need a scanned code, so they resolve nothing for arbitrary numbers. +- `store/sqlstore`: `whatsmeow_contacts` has `first_name, full_name` (app-state), `push_name`, `business_name` columns; `PutPushName` upserts only its own column and reports whether it changed anything. + +## Pros / strengths + +- Push names arrive with traffic the client already receives: no extra query, no phone cooperation beyond normal messaging. +- The separate-columns design settles precedence cleanly: saved names win, push names fill gaps. + +## Cons / limitations / caveats + +- Push names are self-reported and often nicknames, emoji, or stale; they resolve "who messaged me", not "everyone in my address book". +- Only senders of messages seen while online are learned; silent contacts stay unknown until the app-state book, a VCF, or an alias names them. +- History-sync push names need history traffic the detz phone currently never sends; noted, not relied on. + +## Relevance to tai + +Implemented directly in detz 1.46.1: `WaClient::push_name_candidate` (notify plus PN-preferred sender, groups via participant) and `WaStore::put_contact_if_absent` (one name column, so fill-if-absent gives the separate-column precedence without a migration). This closes docs/whatsapp.md section 6's "contact names without app-state" row. Covered by `build/whatsapp_test` section `push-name learning`. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-27 | Initial dossier created. | diff --git a/research/whatsapp-automation/whatsmeow-surface-audit.md b/research/whatsapp-automation/whatsmeow-surface-audit.md new file mode 100644 index 0000000..411d1f9 --- /dev/null +++ b/research/whatsapp-automation/whatsmeow-surface-audit.md @@ -0,0 +1,74 @@ +# whatsmeow surface audit: what detz implements and what it does not + +## Metadata + +| Field | Value | +|---|---| +| Institute | Molodetz Institute — Research & Retrieval Division | +| Subject | whatsapp-automation | +| Document type | Source Dossier | +| Tier | 1 | +| Source title | whatsmeow v0.0.0-20261027104441-1eec1136a50b root package (43 files, local Go module cache copy), compared against taicpp src/whatsapp at detz 1.46.1 | +| Source URL | https://github.com/tulir/whatsmeow | +| Source publisher / author | Tulir Asokan and contributors | +| Source type | repository | +| Source date (published) | 2026-10-27 (module version date) | +| Date researched | 2026-10-27 | +| Researched by | tai (agent), model: Muse Spark, on behalf of retoor | +| Research method | direct page fetch (exported API extracted per file with grep, detz side verified with grep) | +| Confidence | High (mechanical comparison; each detz claim checked against the source) | +| Status | Active | + +## Summary + +No: detz implements a slice of whatsmeow, roughly the 1:1 companion path (transport, QR pairing, login, prekeys, 1:1 text/image send, contact sync, push-name learning). Everything group-shaped, newsletter/call/status/broadcast-shaped, presence, read receipts, retry-resends, history sync, app-state sending, pair-code/passkey linking, server logout, identity trust storage, LID mapping tables, business profiles, blocklists, privacy settings, media download, and all message types beyond text/image/conversation are missing. The wire foundations (Noise, binary codec, token dict, Signal sessions, protobuf subset) are complete for what detz sends. + +## Key findings + +Implemented (equivalent for detz's scope): + +- Transport and crypto: socket, Noise XX, binary codec with token dict, X3DH plus Double Ratchet sessions, hand protobuf subset, media upload path (`handshake.go`, `binary/`, `socket/`, `store/` identity keys, `upload.go` image path, `mediaconn.go`). +- QR pairing and login (`pair.go`, `qrchan.go`, `client.go` Connect/Login): connect, QR refs, pair-success verify and sign, 515 restart, login payload, prekey upload and refill (`prekeys.go` server-count path). +- 1:1 send (`send.go` DM path): usync device lists, prekey fetch, per-device encrypt, DeviceSentMessage own-device copies, participant hash check, image via `WaMedia`. +- Contact sync (`appstate.go` fetch path): one collection (`critical_unblock_low`), snapshot plus patches, external blobs, LTHash and MACs, key request and key share peer messages. +- Push-name learning (`message.go` notify handling): stored fill-if-absent; PN preferred, groups via participant. +- Incoming acks, `peer_msg` protocol receipts, notification handling for `server_sync` and `encrypt`, retry-receipt logging. + +Partial (present but narrower than the reference): + +- `send.go`: no group send (no sender keys), no poll/reaction/edit/revoke/ephemeral/view-once builders, no disappearing timers, no message secrets or reporting tokens. +- `message.go` receive: no inbox (incoming text is never surfaced; `extract_text` has no callers), no media download, no history-sync download, no `hist_sync` receipts, plain acks only (no error nacks). +- `appstate.go`: fetch-only, one collection; no `SendAppState`, no fatal-recovery request, no other collections. +- `user.go`: device-list usync only; no `GetUserInfo` (status/picture/business/LID), no profile picture, no status message, no blocklist, no business profile, no bots, no QR links; the alias flow skips the documented usync existence check. +- `receipt.go`: no read receipts (`MarkRead`), no active-delivery receipts flag. +- `retry.go`: retry receipts are logged, never answered with a resend. +- `client.go`: no server logout (unlink only deletes local rows), no event bus (3 callbacks vs 47 event types), no proxy or custom HTTP clients, no LID mapping table (own LID from pairing only). +- `store/`: no remote-identity table, so no trust-on-first-use identity tracking. +- `download.go`: app-state blobs only; no message media, thumbnail, or sticker download. +- `mediaretry.go`, `msgsecret.go`: absent (media retry receipts, poll/reaction/comment E2E). +- `keepalive.go`: no websocket ping loop (short sessions survive without it; the 300 s sync leans on server tolerance). + +Missing entirely: + +- `group.go` (all 20+ calls: create, info, join, invites, participants, admin, photo, description), `newsletter.go` (all 13 calls), `call.go` (calls are not even observed), `broadcast.go`/`business.go` status and orders, `presence.go` (never sends presence; always passive), `privacysettings.go`, `push.go` (push registration), `pair-code.go` (8-char phone linking), `pair-passkey.go`, `sendfb.go`/`armadillomessage.go` (Messenger/FB), `update.go` (live `client_revision` fetch; the version is a hardcoded constant, so drift ends in 405), `cstoken.go`/`tctoken.go` (privacy tokens). + +## Pros / strengths + +- The audit is mechanical: every row was checked with grep on both trees, so the verdict is reproducible, not an impression. +- detz's implemented slice is the coherent 1:1 companion core; nothing half-built blocks the rest. + +## Cons / limitations / caveats + +- whatsmeow moves fast; this is a 2026-10-27 snapshot comparison. +- Token dictionary and client version freshness were not re-verified here (tracked per release by policy). +- Behavioral parity (byte shapes) was audited earlier per feature, not re-proven for the whole surface here. + +## Relevance to tai + +This is the authoritative "what is left" list for detz::wa. It answers the user's literal question (no, not everything, and here is the inventory) and scopes any future milestone: the next largest coherent blocks are groups (sender keys plus `group.go`), inbox plus read receipts plus retry-resend, and history sync. + +## Changelog + +| Date | Change | +|---|---| +| 2026-10-27 | Initial dossier created. | diff --git a/taicpp/CHANGELOG.md b/taicpp/CHANGELOG.md index 252d1ff..0fac414 100644 --- a/taicpp/CHANGELOG.md +++ b/taicpp/CHANGELOG.md @@ -1,5 +1,13 @@ # Changelog +## [1.46.1] - 2026-10-27 +- WhatsApp learns contact names from incoming messages (push names) + + +## [1.46.0] - 2026-10-27 +- WhatsApp key-share hardening, contact sync wait and VCF import fallback + + ## [1.45.0] - 2026-10-27 - WhatsApp link fix, image send, contact sync and search, failover hardening diff --git a/taicpp/VERSION b/taicpp/VERSION index 50aceaa..aa3ed3a 100644 --- a/taicpp/VERSION +++ b/taicpp/VERSION @@ -1 +1 @@ -1.45.0 +1.46.1 diff --git a/taicpp/docs/whatsapp.md b/taicpp/docs/whatsapp.md index b4a2de7..b312d9a 100644 --- a/taicpp/docs/whatsapp.md +++ b/taicpp/docs/whatsapp.md @@ -581,18 +581,23 @@ it `wa_contact` stays empty and names cannot be resolved. | Piece | Files | What it does | |---|---|---| -| Key receipt | `wa_receiver.*`, `wa_proto.*` (`parse_protocol_message`), `WaClient::handle_message` | Decrypts `` from our own account (pkmsg via X3DH responder, msg via session), unpads, stores each APP_STATE_SYNC_KEY_SHARE key in `wa_appstate_key`. Only the phone's own messages are decrypted; other senders are acked and ignored as before. | -| Key request | `WaClient::request_app_state_keys`, `send_peer_message`, `encode_app_state_key_request` | Peer message to the primary phone (category `peer`), at most `kWaAppStateKeyIdsMax` ids, one request per key id per `kWaAppStateKeyRequestInterval` (24 h), stored in `wa_appstate_keyreq`. | +| Key receipt | `wa_receiver.*`, `wa_proto.*` (`parse_protocol_message`), `WaClient::handle_message` | Decrypts `` from our own account (pkmsg via X3DH responder, msg via session), unpads, stores each APP_STATE_SYNC_KEY_SHARE key in `wa_appstate_key`. Own means our phone number or our LID (whatsmeow `IsFromMe`); other senders are acked and ignored as before. Every arrival is logged before the filter, so a trace shows dropped mail too. An empty share (keys without key data, openclaw/wacli PR 443) is reported and ends the wait without storing; a consumed peer message gets a `peer_msg` protocol receipt. | +| Key request | `WaClient::request_app_state_keys`, `send_peer_message`, `encode_app_state_key_request` | Peer message to the primary phone (category `peer`, meta + enc, device-identity only on a prekey message), at most `kWaAppStateKeyIdsMax` ids, one request per key id per `kWaAppStateKeyRequestInterval` (24 h), stored in `wa_appstate_keyreq`. The send is verified (Signal session must exist, the server ack must carry no error); when it fails the 24 h marks are rolled back so the next sync asks again (whatsmeow PR 1268). | +| Notifications, receipts | `WaClient::handle_notification`, `handle_receipt` | `server_sync` re-syncs the contact collection (other collections are named and skipped); `encrypt` with fewer than `kWaMinPrekeys` left refills prekeys; a retry receipt (the phone could not decrypt our message) is logged loudly. Notification syncs never nest inside a sync (guard + one more pass). | | Key expansion, MACs, values | `wa_appstate.*` (`WaAppStateKeys`, `WaAppStateCrypto`) | HKDF-SHA256 `WhatsApp Mutation Keys` to 160 bytes; value MAC HMAC-SHA512 truncated to 32; index, snapshot and patch MACs HMAC-SHA256; AES-256-CBC values. `WaCrypto::hmac_sha512` added. | | LTHash | `wa_appstate.*` (`WaLtHash`) | 128-byte state, HKDF-SHA256 info `WhatsApp Patch Integrity` (whatsmeow, not BLAKE2b; see research conclusion `appstate-lthash-hkdf-not-blake2b`). | | Patch fetch and apply | `WaAppStateSync::sync` | iq `w:sync:app:state`, `return_snapshot` at version 0, `version=N` otherwise; external blobs via `WaMedia::download` (`md-app-state`); snapshot MAC, patch MAC, index and value MACs checked before anything is stored. | | Contacts | `WaAppStateSync::apply_contact` | SET stores jid = index[1], name = fullName else firstName; empty name is counted and skipped; REMOVE deletes the row. | +| Push-name learning | `WaClient::handle_message`, `push_name_candidate`, `WaStore::put_contact_if_absent` | Every incoming message carries the sender's self-reported name in `notify` (whatsmeow `message.go`); group senders come from `participant`, the PN alternate is preferred when present. Stored only when the jid has no name yet, so a saved, imported or aliased name always wins; truncated to `kWaPushNameMax` (128). No server query returns names for regular users (`GetUserInfo` usync has status, picture, devices, LID and business verified names only), so this plus VCF/alias/app-state is the complete name-resolution stack. | | Login hook | `WaClient::login` calls `sync_app_state` after prekeys | Never fails the login (errors are logged). If a key is missing, requests it, waits up to `kWaAppStateKeyWait` s for a key share, then syncs once more. | +| Long sync | `detz --whatsapp-sync` (`WaWizard::sync`) | Logs in, re-syncs with a `kWaSyncOnlineWait` (300 s) key wait for a sleeping phone, reports the contact count. Exit 0 with contacts, 1 without, 2 when not linked or login failed. | +| VCF fallback | `detz --whatsapp-import FILE` (`WaVcf`) | Imports a .vcf exported on the phone (FN else N, TEL lines, international digits) into the same contact search. Loud skips, capped at `kWaContactsMax`, file capped at `kWaVcfFileMax`. | Caps (`wa_consts.h`): `kWaAppStatePatchMax` 50 patches per sync, `kWaAppStateMutationMax` 10000 mutations, `kWaAppStateTimeout` 30 s per iq, `kWaAppStateBlobMax` 16 MiB per external blob, `kWaAppStateKeyIdsMax` 16 key -ids per request, `kWaAppStateKeyWait` 20 s. A cap stops the sync loudly +ids per request, `kWaAppStateKeyWait` 20 s, `kWaSyncOnlineWait` 300 s, +`kWaVcfFileMax` 8 MiB. A cap stops the sync loudly (`capped` in the report, a log line) and the rest waits for the next login. Failure rule: a patch or snapshot that fails a MAC or hash check is not stored, @@ -616,7 +621,17 @@ Verification (offline, `build/whatsapp_test`, section `appstate`): - a missing key stores nothing and reports the key id; - the key share and key request bytes match whatsmeow's Message encoding; - a pkmsg built with `WaSession` decrypts through `WaReceiver` to the key share - Message, and a corrupted pkmsg is refused. + Message, and a corrupted pkmsg is refused; +- the 24 h request mark can be cleared for an immediate retry; +- own senders accept PN and LID, the peer node carries device-identity only on + a prekey message, server_sync collections and retry receipts parse; +- an empty key share parses with empty key data; +- a 5-card .vcf imports 4 numbers (FN, N fallback, unfolding, 00 form) and the + typo-tolerant search finds them; +- push-name candidates parse for 1:1 (PN, LID, PN alternate), groups + (participant, PN alternate), and reject own senders, `-`, missing notify, + non-user senders and over-long names; fill-if-absent keeps the first name and + never overwrites a saved one. The oracle is `tests/whatsapp_data/appstate_vectors.json`, generated by `tests/whatsapp_data/appstate_oracle.go` (a Go program using whatsmeow's own @@ -624,8 +639,10 @@ The oracle is `tests/whatsapp_data/appstate_vectors.json`, generated by Not verified: the live server's answers to the iq, key share timing, the media host for external blobs, and whether a REMOVE of an unknown index (whatsmeow -only warns) causes trouble. Known gap: a SET with an empty name keeps any older -name the row had; the spec says skip, not delete. +only warns) causes trouble. Known gaps: a SET with an empty name keeps any +older name the row had (the spec says skip, not delete); a retry receipt is +logged, not answered with a resend (whatsmeow `retry.go`); history sync +notifications are not downloaded. @@ -646,3 +663,49 @@ Everything here was measured on 2026-10-27/28 against the live server and the us Gaps: the contact sync and the address-book key share are not verified against a phone yet; a newly linked device must be created with this build for the phone to send history. Set `DETZ_WA_TRACE=1` to log every node's tag and attributes when debugging a protocol exchange. + +## 16. Incident WA-2026-10-28: the phone never sends the app-state sync key + +Full record: `problem.md`. A newly linked companion (`31630045827:54`) logged +in, uploaded prekeys and sent an image to its own number (visible in "Message +yourself", participant hash matching the server's), but the phone never sent +the app-state sync key: 100 to 120 s of `DETZ_WA_TRACE=1` showed only +``, `` and ``, no `` at all. + +The request path was checked line by line against whatsmeow `send.go`, +`message.go`, `appstate.go`, `notification.go`, `client.go`, `pair.go` and +`store/clientpayload.go` (main, fetched 2026-10-27). Registration, login, +DeviceProps (including `historySyncConfig`), the peer node shape, the key +request bytes and the padding all match. whatsmeow answers the open questions +this way: + +- A peer message needs a Signal session with the phone first: + `encryptMessageForDevice` with no bundle and no session returns + `ErrNoSession` and the request never leaves the process (hence whatsmeow PR + 1268, which rolls back the 24 h mark on send failure, and Volt-Chat PR 5, + which adds an explicit session setup). +- Own senders are the PN user or the LID user (`parseMessageSource`); no + `recipient` attribute is needed. The `` node + needs no answer (whatsmeow `handleIB` leaves it alone). +- The `jurisdiction` and `encryption-metadata` nodes in `pair-success` are + server-side account metadata; whatsmeow `handlePairSuccess` ignores every + child it does not know, as we do. Not related to key sharing. + +Fixes in this build (all local; the phone side is still unseen): + +| Gap | Fix | Evidence | +|---|---|---| +| a failed key request still stamped the 24 h mark, blocking the retry | `clear_app_key_request_time`, rollback on send failure (whatsmeow PR 1268) | test `key request rollback` | +| a peer send was trusted without checking the session or the ack | session verified after the prekey fetch, ack error attribute checked and logged (Volt-Chat PR 5) | code, log lines | +| device-identity rode on every peer message, whatsmeow only on pkmsg | `build_peer_message_node` matches whatsmeow | test `peer message helpers` | +| key shares from our LID were silently dropped (PN-only filter) | `is_own_sender` accepts PN or LID, every arrival is logged before the filter | test `peer message helpers` | +| an empty key share looked like no answer | stored/empty counted separately, the wait ends, `peer_msg` receipt sent (openclaw/wacli PR 443) | test `empty key share` | +| notifications and receipts were ignored | `server_sync` re-syncs, `encrypt` refills prekeys, retry receipts logged loudly | test `peer message helpers` | +| 20 s online is short for a sleeping phone | `detz --whatsapp-sync` stays online 300 s (`kWaSyncOnlineWait`) | wizard source | +| no fallback when the phone never answers | `detz --whatsapp-import FILE` imports a .vcf into the contact search | test `vcf import` | + +What to run now: link fresh if needed, then `detz --whatsapp-sync` with +WhatsApp open on the phone (no battery restriction). If no `` arrives +within the 300 s, the phone is gating on its side; import the .vcf and retry +the sync another day. Research: `research/whatsapp-automation/` +(conclusion `key-share-never-arrives.md`). diff --git a/taicpp/full_transpilation_and_integration.md b/taicpp/full_transpilation_and_integration.md new file mode 100644 index 0000000..d23ec7e --- /dev/null +++ b/taicpp/full_transpilation_and_integration.md @@ -0,0 +1,232 @@ +# Full whatsmeow transpilation and integration: plan + agent prompt + +Date: 2026-10-27. Source snapshot: whatsmeow `v0.0.0-20261027104441-1eec1136a50b` +(local Go module cache). Baseline: detz 1.46.1; gap inventory: +`../research/whatsapp-automation/whatsmeow-surface-audit.md`. + +## Goal + +Transpile the whatsmeow library to C++ (`detz::wa`) with literally no +exceptions, gaps, or caveats, integrate every feature into the working detz +application, and finish with zero stubs. Scope is defined mechanically, not by +judgment (see the prompt, section 1): every exported symbol in whatsmeow's +non-test, non-generated Go source, plus every generated protobuf message the +library marshals or unmarshals in non-test code, must have a working C++ +counterpart proven by the coverage gate. + +## Success Criteria + +- The coverage script reports 100% of the inventoried API with zero unmapped + symbols, and the stub detector reports zero hits in `src/whatsapp/`. +- `make`, `make lint`, and `make test` are green on the final tree. +- Every milestone below exited through its uniform gate (spec delta, code, + oracle differential tests, hostile-input tests plus sanitizer for parsers, + coverage at 100% for the milestone's API). +- A live linked phone exercised send, receive, read receipts, groups, media + both directions, and contact sync at least once each (recorded where and + when in the docs). +- `CHANGELOG.md`, `VERSION`, and an annotated tag exist for the release; no + output file lives outside `build/`. + +## Approach + +Small verified slices with independent oracles, not one big rewrite. Each +milestone transpiles one whatsmeow area against Go oracle programs that call +whatsmeow itself (the established `appstate_oracle.go` pattern), so behavior +is proven byte-for-byte rather than self-asserted. Two external methodology +sources shape the gates: a coverage guarantee comes from an inventory plus a +verification script that cross-references every item and re-runs missed ones +(power-mapper), and trusted output comes from deterministic checks the model +cannot argue with, with counterexamples fed back into repair (the VERT +pattern: oracle, differential tests, concrete failing input per defect). +Per-milestone API slices keep every unit small enough to verify strongly. + +## Steps (milestones, dependency order) + +| Milestone | whatsmeow area | Delivers | +|---|---|---| +| M0 | harness | API inventory script, coverage gate script, stub detector; baseline coverage measured; whatsmeow commit pinned | +| M1 | foundations | token dict freshness check, socket keepalive pings, full iq request/response infra, Noise parity | +| M2 | store | identities with trust tracking, LID map, sessions, contacts (saved/push/business), privacy tokens, message secrets, chat settings | +| M3 | pairing | pair-code linking, passkey, QR hardening, server logout iq | +| M4 | messaging core | all message types both directions, group sender keys, inbox surfacing, read receipts, retry-resend, media-retry | +| M5 | media | upload and download of every type, thumbnails, stickers, newsletter media | +| M6 | groups etc. | all of `group.go`, `newsletter.go`, broadcast/status, business | +| M7 | sync | all app-state collections plus send path plus recovery, history sync download and on-demand, LID migration | +| M8 | user services | presence, privacy settings, blocklist, calls (observe plus reject), user info, QR links, status message | +| M9 | client services | live version fetch, push registration, CST/TC tokens, full event bus (47 types), Messenger/FB transports | +| M10 | integration | CLI/REPL/tools for everything new, docs, zero-stub gate, live smoke suite, changelog, version, tag | + +Uniform milestone exit gate: spec delta in `docs/`, warning-free code under +`-std=c++23 -Wall -Wextra -Werror`, oracle differential tests where bytes +cross the boundary, hostile-input tests plus a sanitizer run for every parser, +`make test` green, milestone API at 100% in the coverage script, one commit. + +## Validation Plan + +- After every milestone: `make`, `make lint`, `make test`, the coverage script + scoped to the milestone, the stub detector scoped to touched files. +- New or changed parsers: hostile-input section plus the sanitizer script + before the milestone can exit. +- Byte behavior: Go oracle programs (checked in under `tests/whatsapp_data/`, + built by script, never hand-written expectations for MACs, hashes, or + encodings). +- Final: full coverage 100%, stub detector zero, `make test` green, live + smoke (send, receive, read receipt, group message, image both directions, + contact sync) against a linked phone, tag created. +- Highest-risk validation: the live phone pass, because it is the only oracle + for server behavior; it is scheduled in M10 with explicit user-linking steps + so offline work never blocks on it. + +## Risks / Open Questions + +- Scale: whatsmeow is about 24k lines plus 107k generated protobuf lines; the + generated code is data, and only messages the library actually uses need C++ + counterparts, but M4/M7/M9 are still large. Mitigation: the per-milestone + 100% gate makes partial progress shippable and measurable. +- "Verified" is bounded: differential tests prove equivalence on exercised + paths, not all possible runs. The prompt states this honestly and requires + hostile fuzzing to widen the net. +- Live verification needs the user's phone at defined points. If the phone is + unavailable, milestones exit on offline gates and M10 stays open; nothing is + ever marked live-verified without the run. +- No open design questions: reuse (moloapp), style (cpp-safety), and workflow + (AGENTS.md) are bound by existing project law cited in the prompt. + +## Sources + +- https://github.com/richyparr/power-mapper +- https://www.ben-livshits.org/blog/c-to-rust + +--- + +# THE PROMPT (paste verbatim to the transpilation agent) + +You are transpiling the whatsmeow Go library to C++ inside +`/home/retoor/projects/tai/taicpp`, module `detz::wa` (`src/whatsapp/`). +Read this whole prompt before touching anything. Your finished state is a +working application with zero stubs and no exceptions, gaps, or caveats. + +## 1. Scope (mechanical, not judgment) + +The source snapshot is pinned: whatsmeow +`v0.0.0-20261027104441-1eec1136a50b` in the local Go module cache. Record +its commit hash in `docs/whatsapp.md` on day one and never move it. + +In M0 you build `tools/wa-api-inventory.sh`, which extracts every exported +symbol (functions, methods, types, constants, event structs) from whatsmeow's +non-test, non-generated Go source, plus every generated protobuf message the +library marshals or unmarshals in non-test code (found by grep for +`proto.Marshal`/`proto.Unmarshal` call sites, then listing the message +types). That inventory is the scope. `tools/wa-coverage.sh` maps each entry +to its C++ counterpart and fails below 100%. `tools/wa-stub-scan.sh` fails +on any of `TODO`, `FIXME`, `XXX`, `HACK`, `unimplemented`, `not implemented`, +`NOT_IMPLEMENTED`, `panic(`, `assert(false`, `std::abort` in +`src/whatsapp/`, and on any new `return std::nullopt` / `return false` / +`return {}` that is not accompanied by a loud log line at the call site. +There is no allowlist file. If you believe something cannot be implemented, +you stop and ask the user; you never stub it and move on. + +## 2. Starting point (read all of it) + +- `../research/whatsapp-automation/whatsmeow-surface-audit.md`: what exists + and what is missing at detz 1.46.1. +- `docs/whatsapp.md` (all sections), `docs/cpp-safety.md` (all four layers), + `AGENTS.md` (the per-milestone workflow), `problem.md`. +- The existing oracle pattern: `tests/whatsapp_data/appstate_oracle.go` + plus `tests/whatsapp_test.cpp` (run as `build/whatsapp_test`). Extend them, + do not replace them. + +## 3. Milestones and the uniform exit gate + +Work the milestone table from the plan (M0..M10) in order. A milestone exits +only when ALL of these hold: spec delta written in `docs/`; code compiles +warning-free under `-std=c++23 -Wall -Wextra -Werror`; Go oracle differential +tests prove every byte behavior (MACs, hashes, encodings, node shapes) with +no hand-written expectations for oracle-computable values; every new or +changed parser ships hostile-input tests plus a sanitizer run +(`-fsanitize=address,undefined -fno-sanitize-recover=all`, +plus the module script pattern of `tools/san-whatsapp.sh`); `make test` is +green; `tools/wa-coverage.sh` reports 100% for the milestone's API slice; +exactly one commit records the milestone. Never let the model grade its own +homework: a test you wrote from your own assumption proves nothing, so every +behavioral claim needs the Go oracle, a golden vector, a live run, or a +second independent method, and every divergence feeds its concrete failing +input back into the fix. + +## 4. C++ project law (no deviations) + +- One class per `.h`/`.cpp` pair, namespace `detz::wa`, maximally + object-oriented; headers own their invariants, constructors establish them. +- `docs/cpp-safety.md` Layer 1 in full for every untrusted byte: subtraction + bounds checks (`len > size - pos`), unsigned range-checked lengths before + any allocation, guarded shifts and loop bounds, no exception escaping any + top-level loop, parsing concentrated in the checked readers. +- Every quantity gets a static cap in `wa_consts.h` (rates, sizes, counts, + intervals); unbounded is a bug. No output file outside `build/`. +- `src/cli/version.hpp` stays generated from `VERSION`; bump per milestone + group, changelog entry per milestone, annotated tag at the end. + +## 5. DRY reuse of moloapp (check before you write) + +Before writing any HTTP client, SSE/websocket, LLM provider, config, CLI, +tenancy, or app-pattern code, grep `../../moloapp/src/` (1039 headers; +start with `clients/`, `llm/`, `app/`, `tenancy/` and their `SPEC.md` +files) for an existing facility. Reuse by include plus link only; you never +modify moloapp. Record every reuse decision (used moloapp X / wrote local Y +because Z) in the `docs/whatsapp.md` reuse table you create in M0. A +duplicated facility the library already provides fails review. + +## 6. Integration is part of done + +Every transpiled capability must be reachable in the working application: +CLI flags where a user would run it, agent tools where the agent would call +it, REPL/UX where it is interactive. A transpiled correct function with no +caller is a stub by another name and fails the M10 gate. Docs state what the +system does on the day, with any live-verification gaps listed next to the +code that will close them. + +## 7. Final gate (M10, all must hold) + +`make`, `make lint`, `make test` green; `tools/wa-coverage.sh` 100% with +zero unmapped symbols; `tools/wa-stub-scan.sh` zero hits; sanitizer suite +green; live smoke against a linked phone (send, receive, read receipt, +group message, image both directions, contact sync), each recorded with date +in the docs; `CHANGELOG.md`, `VERSION`, annotated tag. Then stop and report: +what shipped, the coverage and test evidence, and anything you were forced +to ask the user, with the answers. + +## 8. Working rules + +Small diffs, existing files first, no drive-by refactors. Loud skips beat +silent corruption: reject unverifiable input with a reason and a counter. +Never rewrite git history. Never search for or touch grader, oracle-key, or +`.secrets` material; the Go oracles you write yourself from the pinned +whatsmeow source are the only oracles you use. + +--- + +## Recursive validation (prompt vs. every requirement) + +| # | Requirement (user's words, condensed) | Prompt section | Verdict | +|---|---|---|---| +| 1 | literally all features, no exception | 1: mechanical inventory of every exported symbol plus every used proto message; 100% gate | covered | +| 2 | no gaps | 1 + 7: coverage script fails below 100%, zero unmapped symbols at the end | covered | +| 3 | no caveats | 1: no allowlist; blocking items stop and ask instead of being waived | covered | +| 4 | validate everything it made itself | 3: oracle differential tests, no-self-grading rule with counterexample feedback; oracle presence per milestone is human-verifiable in review | covered | +| 5 | milestones | 3 + plan table M0..M10 with a uniform exit gate and one commit each | covered | +| 6 | project C++ rules | 4: file layout, OO, cpp-safety Layer 1, static caps, build dir, versioning; enforced by `make lint`, sanitizers, hostile tests | covered | +| 7 | DRY use of moloapp | 5: grep-before-write, link-only, reuse table, duplication fails review (human gate at M10, not a script) | covered | +| 8 | working application at the end | 6: every capability reachable via CLI/tools/REPL; uncalled code fails M10 | covered | +| 9 | zero stubs | 1 + 7: stub scanner patterns, zero-hit gate, no allowlist | covered | +| 10 | online research baked in | Approach + Sources: inventory-plus-verification-script guarantee, oracle/counterexample repair, small-scope strength | covered | +| 11 | saved to `full_transpilation_and_integration.md` | this file | covered | +| 12 | recursively validated | this table: 4 passes (mapping; gate check with 2 human gates labeled; honesty check; re-read), stable with no open items | covered | + +Validation passes run: (1) requirement-to-section mapping above; (2) gate +check: every mapped section contains a failing script or command, except rows +4 (no-self-grading discipline) and 7 (moloapp DRY), which are human review +gates, now labeled as such; (3) honesty check: bounded-verification and +live-phone limits are stated in Risks, so no promise outruns its evidence; +(4) re-read after the fixes: paths corrected (`AGENTS.md`, test file), gates +labeled honestly. Stable after pass 4. diff --git a/taicpp/problem.md b/taicpp/problem.md new file mode 100644 index 0000000..cd80471 --- /dev/null +++ b/taicpp/problem.md @@ -0,0 +1,114 @@ +# Open problem: the phone never sends the app-state sync key to a newly linked companion + +Date: 2026-10-28. Client: detz 1.45.0 (`src/whatsapp/`, namespace `detz::wa`), a C++ WhatsApp companion +written from scratch after whatsmeow (Go). Device in the last test: `31630045827:54`. + +## In one sentence + +A newly linked companion device logs in and sends fine, but the phone never sends it the app-state sync key. +Without that key the address book (collection `critical_unblock_low`) cannot be decrypted, so no contact +names arrive. + +## What works (confirmed by the user on the phone) + +- QR pairing, automatic restart (`stream:error 515`), login, upload of 50 prekeys. +- Sending an image to the user's own number; it shows in "Message yourself". +- Our participant hash equals the server's (`2:TKwG4B1n`), so the device list we send to is correct. + +## What does not work + +Every login runs the contact sync. The log shows: + +``` +app state critical_unblock_low: missing app state key for the snapshot +app state critical_unblock_low: version 0, 0 patches, 0 contacts saved, 0 removed, 0 unnamed +app state keys: requesting 1 from the phone +app state: no key share arrived; contacts will sync on a later login +``` + +I then stayed connected for 100 to 120 seconds with a trace of every incoming node +(`DETZ_WA_TRACE=1`), after clearing our own 24-hour limit on key requests. The only nodes that arrived were +``, `` and ``. No `` node arrived at all. +The database holds 0 app-state keys and one Signal session, with the phone's LID address +`173735822786802.0`, created by our own sends. + +## What the code does + +1. Login succeeds. We request the collection `critical_unblock_low`; the server's reply shows a key is needed. +2. We send a peer message (`category="peer"`, `push_priority="high"`) to the user's own account. It contains a + protocol message of type `APP_STATE_SYNC_KEY_REQUEST` and is encrypted to the phone's LID identity. +3. We wait 20 seconds for the phone's `APP_STATE_SYNC_KEY_SHARE`. Nothing arrives. + +Registration now carries `DeviceProps.historySyncConfig` with whatsmeow's values (added after a first failed +attempt; the device was re-linked afterwards). + +## Checked against the reference + +Pairing reply, media upload, key derivation and the key-request bytes match whatsmeow. The app-state crypto +(HKDF key expansion, LTHash, MACs) passes against an oracle built from whatsmeow's own packages. The data path +is not the problem: the key never arrives, so nothing has been decrypted from real data yet. + +## Hypotheses, none confirmed + +1. The phone cannot decrypt or accept our key request. The server acknowledges it, but an ack only means the + server took it. Related reports: whatsmeow PR 1268 and Volt-Chat PR 5 describe key requests that never leave + the process without a Signal session to the phone; WhatsApp Web ensures that session before sending. +2. The phone pushes keys only once, right after pairing, and that happened inside the link session, which I + never traced. If messages arrived then and our handler dropped or failed them silently, the keys are lost. + Checkable by logging everything during a fresh link. +3. The phone is not answering for another reason: in the background or battery-restricted, WhatsApp not open. + Reports also exist of the phone answering with an empty key (openclaw/wacli PR 443). +4. A classification issue like the one in Rich627/whatsapp-claude-plugin PR 45: peer messages from the phone + have no `recipient` attribute and were dropped as spoofed. That explains missing keys there, but here no + `` arrives at all, so it is not our case yet. + +## Questions for an expert + +- On a fresh link, does the primary phone always push the key share and history to the new companion without + being asked? What does it key that on (device props, `platformType`, client version)? +- Is a peer key request encrypted to the LID identity of device 0 enough, or must a Signal session with the + phone already exist from an earlier exchange? +- Does the phone answer key requests from a device that has not yet received the initial history sync? +- Is anything in the 2026 pairing flow, such as the `jurisdiction` and `encryption-metadata` nodes seen in + `pair-success` for a Dutch number, related to when keys are shared? + +## Sources + +- https://github.com/tulir/whatsmeow/pull/1268 +- https://github.com/Volt-Chat/whatsmeow/pull/5 +- https://github.com/openclaw/wacli/pull/443 +- https://github.com/Rich627/whatsapp-claude-plugin/pull/45 +- Research archive: `research/whatsapp-automation/` (whatsmeow-appstate-sync.md, whatsmeow-key-share-and-peer-messages.md, + whatsmeow-device-props-history-sync.md) + +## Fallback + +Import the contacts from a `.vcf` file exported on the phone, into the existing contact search. Built: `detz --whatsapp-import contacts.vcf` (see `WaVcf`, docs section 14). + +## Resolution (2026-10-27, detz 1.46.0) + +The request path was verified line by line against whatsmeow main (`send.go`, `message.go`, `appstate.go`, +`notification.go`, `client.go`, `pair.go`, `store/clientpayload.go`, fetched 2026-10-27). Registration, login, +DeviceProps, the peer node, the key-request bytes and the padding all match the reference. Eight local gaps were +fixed anyway (docs section 16 has the table): 24 h rollback on send failure (whatsmeow PR 1268), verified peer +send (Volt-Chat PR 5), pkmsg-only device-identity, LID senders accepted plus an arrival log before the filter, +empty shares distinguished from silence (openclaw/wacli PR 443), notification/receipt handling, `--whatsapp-sync` +(300 s online for a sleeping phone), and the `--whatsapp-import` fallback. `build/whatsapp_test` covers the new +behavior; the live answer still needs the phone. + +Answers to the questions above, from the reference source (not from the phone, which is still unseen): + +- Fresh link, unsolicited push: yes, the phone pushes the initial history sync and the key shares on its own once + it notices the new companion (that is the initial sync `historySyncConfig` declares support for). Re-requests are + answered too in a healthy setup (whatsmeow `requestMissingAppStateKeys` sends them on every fetch with missing + keys); reports of "just once" come from clients that dropped the first push. +- Session before the peer request: yes, required. whatsmeow `encryptMessageForDevice` returns `ErrNoSession` when + no session exists and the message never leaves the process. We now verify the session exists after the prekey + fetch and say so loudly when it does not. +- Requests before the initial history sync: answered in principle (wacli sees even empty shares), but a phone that + is asleep or battery-restricted answers nothing until it wakes; that is what `--whatsapp-sync` (300 s) is for. +- `jurisdiction` / `encryption-metadata` in `pair-success`: server-side account metadata, ignored by whatsmeow + `handlePairSuccess` like any unknown child. Not related to key sharing. + +Next step if keys still do not arrive: run `detz --whatsapp-sync` with WhatsApp open on the phone and no battery +restriction. A 300 s trace with no `` proves the gate is on the phone's side; import the .vcf meanwhile. diff --git a/taicpp/src/agent/tools_ops.cpp b/taicpp/src/agent/tools_ops.cpp index 7051beb..a1c7958 100644 --- a/taicpp/src/agent/tools_ops.cpp +++ b/taicpp/src/agent/tools_ops.cpp @@ -744,7 +744,9 @@ std::string Tools::run_whatsapp_contacts(const json::Value& args) { } auto cts = store.contacts(); if (cts.empty()) - return "no WhatsApp contacts yet. They arrive from the phone after `detz --whatsapp-link` (address book sync); or add one with " + return "no WhatsApp contacts yet. They arrive from the phone after `detz --whatsapp-link` (address book sync); if " + "none arrived, run `detz --whatsapp-sync` with WhatsApp open on the phone, or `detz --whatsapp-import " + "contacts.vcf` with a .vcf exported on the phone; or add one with " "whatsapp_contacts name=\"Niggel\" number=\"31612345678\""; std::string out = std::to_string(cts.size()) + " WhatsApp contact(s):\n"; for (const auto& [jid, nm] : cts) out += " " + nm + " -> " + jid.substr(0, jid.find('@')) + "\n"; diff --git a/taicpp/src/cli/cli.cpp b/taicpp/src/cli/cli.cpp index fdaa75b..37bd9a9 100644 --- a/taicpp/src/cli/cli.cpp +++ b/taicpp/src/cli/cli.cpp @@ -31,7 +31,8 @@ void bind(const Namespace& ns, Options& o) { flag("install_telegram", o.install_telegram); flag("uninstall_telegram", o.uninstall_telegram); flag("plexer_wrap", o.plexer_wrap); list("mplex", o.mplex); flag("whatsapp_link", o.whatsapp_link); flag("whatsapp_unlink", o.whatsapp_unlink); - flag("whatsapp_status", o.whatsapp_status); + flag("whatsapp_status", o.whatsapp_status); flag("whatsapp_sync", o.whatsapp_sync); + str("whatsapp_import", o.whatsapp_import); flag("demo", o.demo); flag("demo_shot", o.demo_shot); flag("tui", o.tui); flag("tui_local", o.tui_local); str("theme", o.theme); auto p = ns.lists.find("prompt"); diff --git a/taicpp/src/cli/cli.h b/taicpp/src/cli/cli.h index bdd930c..5f4c541 100644 --- a/taicpp/src/cli/cli.h +++ b/taicpp/src/cli/cli.h @@ -22,6 +22,8 @@ struct Options { bool telegram = false, scheduler = false, install = false; bool install_telegram = false, uninstall_telegram = false, plexer_wrap = false; bool whatsapp_link = false, whatsapp_unlink = false, whatsapp_status = false; + bool whatsapp_sync = false; + std::optional whatsapp_import; std::optional> mplex; // REMAINDER bool demo = false, demo_shot = false, tui = false, tui_local = false; std::optional theme; diff --git a/taicpp/src/cli/flag_spec.cpp b/taicpp/src/cli/flag_spec.cpp index f4dd623..76e4dea 100644 --- a/taicpp/src/cli/flag_spec.cpp +++ b/taicpp/src/cli/flag_spec.cpp @@ -57,6 +57,8 @@ std::vector build() { t.push_back(sw("--whatsapp-link", "whatsapp_link", "link this device to WhatsApp: shows a QR code to scan on your phone (Settings > Linked Devices), then saves the link to the vault")); t.push_back(sw("--whatsapp-unlink", "whatsapp_unlink", "remove the stored WhatsApp link for this profile")); t.push_back(sw("--whatsapp-status", "whatsapp_status", "print whether WhatsApp is linked for this profile")); + t.push_back(sw("--whatsapp-sync", "whatsapp_sync", "stay online and pull the phone address book (long wait for a sleeping phone)")); + t.push_back(val("--whatsapp-import", "whatsapp_import", "FILE", "import a .vcf file exported on the phone into the WhatsApp contact search")); t.push_back(sw("--plexer-wrap", "plexer_wrap", "internal: invoked by the bashrc plexer hook, not meant to be run by hand")); FlagSpec mp = val("--mplex", "mplex", "COMMAND", "terminal multiplexer: any mplex command (new-session, attach-session, list-keys ...); bare --mplex starts a session. Config: ~/.tai/mplex.toml"); mp.nargs = Nargs::Remainder; diff --git a/taicpp/src/glue/glue_app.cpp b/taicpp/src/glue/glue_app.cpp index c6192ab..84adc1e 100644 --- a/taicpp/src/glue/glue_app.cpp +++ b/taicpp/src/glue/glue_app.cpp @@ -228,10 +228,12 @@ int GlueApp::run() { return 0; } if (o_.install || o_.install_telegram || o_.uninstall_telegram) return install_flag(); - if (o_.whatsapp_link || o_.whatsapp_unlink || o_.whatsapp_status) { + if (o_.whatsapp_link || o_.whatsapp_unlink || o_.whatsapp_status || o_.whatsapp_sync || o_.whatsapp_import) { std::string profile = o_.profile.value_or("default"); if (o_.whatsapp_unlink) return wa::WaWizard::unlink(home(), profile, std::cout); if (o_.whatsapp_status) return wa::WaWizard::status(home(), profile, std::cout); + if (o_.whatsapp_sync) return wa::WaWizard::sync(home(), profile, std::cout); + if (o_.whatsapp_import) return wa::WaWizard::import_vcf(home(), profile, *o_.whatsapp_import, std::cout); return wa::WaWizard::link(home(), profile, std::cout); } if (o_.version) { diff --git a/taicpp/src/whatsapp/wa_client.cpp b/taicpp/src/whatsapp/wa_client.cpp index 5d94f4a..120a334 100644 --- a/taicpp/src/whatsapp/wa_client.cpp +++ b/taicpp/src/whatsapp/wa_client.cpp @@ -186,8 +186,12 @@ void WaClient::dispatch_inner(const WaNode& node) { } else if (node.tag == "message") { send_ack(node); handle_message(node); - } else if (node.tag == "ib" || node.tag == "iq") { - // ignored informational nodes + } else if (node.tag == "notification") { + handle_notification(node); // acks itself, then acts on server_sync / encrypt + } else if (node.tag == "receipt") { + handle_receipt(node); + } else if (node.tag == "ib") { + // ignored informational nodes (edge_routing, dirty: whatsmeow handleIB leaves dirty alone too) } } @@ -341,7 +345,7 @@ bool WaClient::login(int timeout_seconds) { if (logged_in_) upload_prekeys(); if (logged_in_) { try { - sync_app_state(); + sync_app_state(WaConsts::kWaAppStateKeyWait); } catch (const std::exception& e) { log(std::string("app state sync failed: ") + e.what()); } @@ -349,6 +353,10 @@ bool WaClient::login(int timeout_seconds) { return logged_in_; } +void WaClient::sync_contacts() { sync_app_state(WaConsts::kWaAppStateKeyWait); } + +void WaClient::sync_contacts_with_wait(int wait_seconds) { sync_app_state(wait_seconds); } + bool WaClient::upload_prekeys() { WaNode count_iq("iq"); count_iq.attrs["to"] = WaConsts::kServerJid; @@ -780,12 +788,145 @@ std::optional WaClient::send_image(const std::string& to_jid, const bool WaClient::socket_open() const { return sock_ != nullptr && sock_->connected(); } -void WaClient::handle_message(const WaNode& node) { - if (!device_.paired() || !device_.jid) return; +bool WaClient::is_own_sender(const std::string& from, const std::string& own_pn_user, const std::string& own_lid_user) { + std::string user = from.substr(0, from.find_first_of(":@")); + if (!own_pn_user.empty() && user == own_pn_user) return true; + return !own_lid_user.empty() && user == own_lid_user; +} + +std::vector WaClient::server_sync_collections(const WaNode& notification) { + std::vector out; + for (const WaNode* c : notification.all("collection")) { + if (!c->attr("name").empty()) out.push_back(c->attr("name")); + } + return out; +} + +bool WaClient::is_retry_receipt(const WaNode& node) { return node.tag == "receipt" && node.attr("type") == "retry"; } + +std::optional> WaClient::push_name_candidate( + const WaNode& node, const std::string& own_pn_user, const std::string& own_lid_user) { + std::string notify = node.attr("notify"); // whatsmeow message.go: info.PushName from `notify` + if (notify.empty() || notify == "-") return std::nullopt; + if (notify.size() > WaConsts::kWaPushNameMax) notify.resize(WaConsts::kWaPushNameMax); + auto server_of = [](const std::string& jid) { + size_t at = jid.find('@'); + return at == std::string::npos ? std::string() : jid.substr(at + 1); + }; + std::string sender = node.attr("from"); + std::string alt; + if (server_of(sender) == WaConsts::kGroupServer) { + sender = node.attr("participant"); + alt = node.attr("participant_pn"); + if (alt.empty()) alt = node.attr("participant_lid"); + } else { + alt = node.attr("sender_pn"); + if (alt.empty()) alt = node.attr("sender_lid"); + } + // Prefer the phone-number form: it is the sendable one. whatsmeow stores both; we keep one row per sender. + std::string chosen = sender; + if (!alt.empty() && server_of(alt) == WaConsts::kServerJid) { + chosen = alt; + } else if (server_of(sender) != WaConsts::kServerJid && server_of(sender) != WaConsts::kLidServer) { + if (!alt.empty() && (server_of(alt) == WaConsts::kServerJid || server_of(alt) == WaConsts::kLidServer)) { + chosen = alt; + } else { + return std::nullopt; // newsletters, broadcasts and status: no user sender to learn + } + } + size_t at = chosen.find('@'); + if (at == std::string::npos) return std::nullopt; + std::string user = chosen.substr(0, chosen.find_first_of(":@")); // non-AD: drop any :device + std::string jid = user + "@" + chosen.substr(at + 1); + if (user.empty() || is_own_sender(jid, own_pn_user, own_lid_user)) return std::nullopt; + return std::make_pair(jid, notify); +} + +WaNode WaClient::build_peer_message_node(const std::string& id, const std::string& to, const WaNode& enc, bool is_prekey, + const Bytes& account) { + WaNode meta("meta"); + meta.attrs["appdata"] = "default"; + WaNode::Children content{meta, enc}; + if (is_prekey && !account.empty()) { + WaNode di("device-identity"); + di.content = account; + content.push_back(di); + } + WaNode msg("message"); + msg.attrs["id"] = id; + msg.attrs["to"] = to; + msg.attrs["type"] = "text"; + msg.attrs["category"] = "peer"; + msg.attrs["push_priority"] = "high"; // whatsmeow sets this on APP_STATE_SYNC_KEY_REQUEST only, our one peer use + msg.content = content; + return msg; +} + +void WaClient::send_protocol_receipt(const std::string& id, const std::string& type) { + if (id.empty() || !logged_in_ || !device_.jid) return; std::string own_user = device_.jid->substr(0, device_.jid->find('@')); own_user = own_user.substr(0, own_user.find(':')); + WaNode receipt("receipt"); + receipt.attrs["id"] = id; + receipt.attrs["type"] = type; + receipt.attrs["to"] = own_user + "@" + WaConsts::kServerJid; + send_node(receipt); +} + +void WaClient::handle_notification(const WaNode& node) { + send_ack(node); + std::string type = node.attr("type"); + if (type == "server_sync") { + for (const auto& name : server_sync_collections(node)) { + if (name != WaAppStateSync::kContactsCollection) { + log("server_sync: collection " + name + " updated (not synced, only the contact collection is)"); + continue; + } + log("server_sync: the phone updated the contact collection, re-syncing"); + sync_app_state(0); // no extra key wait: a notification means the phone is awake, keys arrive as messages + } + } else if (type == "encrypt") { + if (const WaNode* count = node.child("count")) { + int left = std::atoi(count->attr("value").c_str()); + if (left < WaConsts::kWaMinPrekeys) { + log("server holds " + std::to_string(left) + " prekeys, uploading more"); + upload_prekeys(); + } + } + } + // every other notification type is server bookkeeping we do not act on (trace shows it) +} + +void WaClient::handle_receipt(const WaNode& node) { + // Receipts are the phone's side of the conversation made visible: a retry receipt means the phone got one of + // our messages but could not decrypt it, and asks us to resend. We do not resend yet (gap, see docs), but we + // say so loudly instead of ignoring it. + if (is_retry_receipt(node)) { + log("the phone could not decrypt our message " + node.attr("id") + " and asks for a resend (retry receipt from " + + node.attr("from") + "); re-link if this repeats"); + } +} + +void WaClient::handle_message(const WaNode& node) { + if (!device_.paired() || !device_.jid) return; std::string from = node.attr("from"); - if (from.substr(0, from.find_first_of(":@")) != own_user) return; // only the phone's own messages + // The arrival breadcrumb comes before the filter on purpose: when the phone's key share never seems to arrive, + // the trace must show whether nothing arrived or something arrived and was dropped. + log("message from " + from + " type=" + node.attr("type") + " category=" + node.attr("category")); + std::string own_user = device_.jid->substr(0, device_.jid->find('@')); + own_user = own_user.substr(0, own_user.find(':')); + std::string lid_user; + if (device_.lid && !device_.lid->empty()) { + lid_user = device_.lid->substr(0, device_.lid->find('@')); + lid_user = lid_user.substr(0, lid_user.find(':')); + } + if (!is_own_sender(from, own_user, lid_user)) { + // Not ours, so no key share inside: but the sender's push name teaches the contact search (docs 14). + if (auto c = push_name_candidate(node, own_user, lid_user)) { + if (store_.put_contact_if_absent(c->first, c->second)) log("learned contact " + c->second + " (" + c->first + ")"); + } + return; + } const WaNode::Children* kids = node.children(); if (kids == nullptr) return; for (const auto& enc : *kids) { @@ -798,12 +939,22 @@ void WaClient::handle_message(const WaNode& node) { } WaProtocolMessage pm = parse_protocol_message(res.message); if (pm.key_share.empty()) continue; + int stored = 0, empty = 0; for (const auto& k : pm.key_share) { - if (k.key_id.empty() || k.key_data.empty()) continue; + if (k.key_id.empty() || k.key_data.empty()) { + empty++; + continue; + } store_.put_app_key(k.key_id, k.key_data, k.timestamp); app_keys_received_++; + stored++; } - log("app state keys received: " + std::to_string(pm.key_share.size())); + app_keys_answered_ = true; + if (stored > 0) log("app state keys received: " + std::to_string(stored) + " stored"); + if (empty > 0) + log("the phone answered with " + std::to_string(empty) + + " key share(s) without key data (it has none to give right now); the sync retries on a later login"); + if (node.attr("category") == "peer") send_protocol_receipt(node.attr("id"), WaConsts::kWaReceiptPeerMsg); } } @@ -819,20 +970,25 @@ bool WaClient::send_peer_message(const Bytes& message_plaintext) { lid_user = lid_user.substr(0, lid_user.find(':')); enc_jid = lid_user + "@lid"; } - if (!ensure_sessions({enc_jid})) return false; + // A peer message needs a Signal session with the phone first (Volt-Chat whatsmeow PR 5): without one it can + // never leave the process, and stamping it requested would block the retry for 24 h (handled by the caller). + if (!ensure_sessions({enc_jid})) { + log("peer message: could not fetch the phone's prekeys, no session"); + return false; + } + if (!store_.load_session(addr_of(enc_jid))) { + log("peer message: no session with the phone at " + addr_of(enc_jid) + " even after the prekey fetch"); + return false; + } auto nodes = encrypt_for_devices({enc_jid}, WaCrypto::pkcs7_pad(message_plaintext), ""); if (nodes.empty() || nodes[0].children() == nullptr) { log("peer message: no session with the phone"); return false; } - WaNode::Children content{WaNode("meta")}; - content[0].attrs["appdata"] = "default"; - content.push_back(nodes[0].children()->front()); - if (!device_.account.empty()) { - WaNode di("device-identity"); - di.content = device_.account; - content.push_back(di); - } + const WaNode& enc = nodes[0].children()->front(); + bool is_prekey = enc.attr("type") == "pkmsg"; + log("peer message to " + enc_jid + " as " + enc.attr("type") + + (is_prekey ? " (first contact, the phone builds its session from this)" : " (existing session)")); std::string id = "3EB0" + [] { Bytes r = WaCrypto::random_bytes(8); static const char* H = "0123456789ABCDEF"; @@ -843,19 +999,22 @@ bool WaClient::send_peer_message(const Bytes& message_plaintext) { } return s; }(); - WaNode msg("message"); - msg.attrs["id"] = id; - msg.attrs["to"] = own_user + "@" + WaConsts::kServerJid; - msg.attrs["type"] = "text"; - msg.attrs["category"] = "peer"; - msg.attrs["push_priority"] = "high"; - msg.content = content; - send_node(msg); + send_node(build_peer_message_node(id, own_user + "@" + WaConsts::kServerJid, enc, is_prekey, device_.account)); auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(20); while (std::chrono::steady_clock::now() < deadline) { + if (!socket_open()) break; auto node = recv_node(2); if (!node) continue; - if (node->tag == "ack" && node->attr("id") == id) return true; + if (node->tag == "ack" && node->attr("id") == id) { + std::string line = "peer message: server ack"; + for (const auto& kv : node->attrs) line += " " + kv.first + "=" + kv.second; + log(line); + if (node->has_attr("error")) { + log("peer message: the server rejected the request (error " + node->attr("error") + ")"); + return false; + } + return true; + } dispatch(*node); } log("peer message: no ack received"); @@ -877,36 +1036,54 @@ void WaClient::request_app_state_keys(const std::vector& key_ids) { return; } log("app state keys: requesting " + std::to_string(due.size()) + " from the phone"); - send_peer_message(encode_app_state_key_request(due)); + if (send_peer_message(encode_app_state_key_request(due))) return; + // The request never left this device (no session, no ack, server rejection): forget that we asked, so the + // next sync asks again instead of waiting out the 24 h on keys nobody will send (whatsmeow PR 1268). + for (const auto& id : due) store_.clear_app_key_request_time(id); + log("app state keys: the request failed to send; cleared the 24 h mark so the next sync asks again"); } -void WaClient::sync_app_state() { - WaAppStateSync sync(store_, [this](const std::string& s) { log(s); }); - auto iq = [this](WaNode q) { - q.attrs["id"] = next_iq_id(); - return send_iq(q, WaConsts::kWaAppStateTimeout); - }; - auto download = [this](const WaExternalBlobRef& ref, std::string& why) -> std::optional { - auto mc = query_media_conn(); - if (!mc) { - why = "no media connection"; - return std::nullopt; - } - return WaMedia::download(mc->host, ref.direct_path, "md-app-state", ref.media_key, "WhatsApp App State Keys", - ref.file_enc_sha256, ref.file_sha256, WaConsts::kWaAppStateBlobMax, why); - }; - auto rep = sync.sync(WaAppStateSync::kContactsCollection, iq, download); - if (rep.missing_key_ids.empty()) return; - // The phone holds the keys: ask for them, give it a bounded window to answer, then sync once more. - request_app_state_keys(rep.missing_key_ids); - int before = app_keys_received_; - run_until([&]() { return app_keys_received_ != before; }, WaConsts::kWaAppStateKeyWait); - if (app_keys_received_ == before) { - log("app state: no key share arrived; contacts will sync on a later login"); +void WaClient::sync_app_state(int key_wait_seconds) { + if (sync_active_) { + sync_again_ = true; // a notification sync during a sync: one more pass after, never nested return; } - WaAppStateSync again(store_, [this](const std::string& s) { log(s); }); - again.sync(WaAppStateSync::kContactsCollection, iq, download); + sync_active_ = true; + for (int pass = 0; pass < 3; ++pass) { + sync_again_ = false; + WaAppStateSync sync(store_, [this](const std::string& s) { log(s); }); + auto iq = [this](WaNode q) { + q.attrs["id"] = next_iq_id(); + return send_iq(q, WaConsts::kWaAppStateTimeout); + }; + auto download = [this](const WaExternalBlobRef& ref, std::string& why) -> std::optional { + auto mc = query_media_conn(); + if (!mc) { + why = "no media connection"; + return std::nullopt; + } + return WaMedia::download(mc->host, ref.direct_path, "md-app-state", ref.media_key, "WhatsApp App State Keys", + ref.file_enc_sha256, ref.file_sha256, WaConsts::kWaAppStateBlobMax, why); + }; + auto rep = sync.sync(WaAppStateSync::kContactsCollection, iq, download); + if (!rep.missing_key_ids.empty() && key_wait_seconds > 0) { + // The phone holds the keys: ask for them, give it a bounded window to answer, then sync once more. + request_app_state_keys(rep.missing_key_ids); + int before = app_keys_received_; + bool answered_before = app_keys_answered_; + run_until([&]() { return app_keys_received_ != before || app_keys_answered_ != answered_before; }, + key_wait_seconds); + if (app_keys_received_ == before && app_keys_answered_ == answered_before) { + log("app state: no key share arrived; contacts will sync on a later login"); + } else if (app_keys_received_ != before) { + WaAppStateSync again(store_, [this](const std::string& s) { log(s); }); + again.sync(WaAppStateSync::kContactsCollection, iq, download); + } + // an answered-but-empty share needs no re-sync: the version stays 0, so the next login retries the full sync + } + if (!sync_again_) break; + } + sync_active_ = false; } void WaClient::disconnect() { diff --git a/taicpp/src/whatsapp/wa_client.h b/taicpp/src/whatsapp/wa_client.h index 87a2e07..e3614bc 100644 --- a/taicpp/src/whatsapp/wa_client.h +++ b/taicpp/src/whatsapp/wa_client.h @@ -33,6 +33,9 @@ class WaStorePort { virtual std::optional get_prekey(uint32_t id) = 0; // contacts / aliases (name -> jid) virtual void put_contact(const std::string& jid, const std::string& name) = 0; + // Stores a push name learned from an incoming message only when the jid has no name yet (a saved address-book + // name, VCF import or user alias always wins over a self-reported push name). Returns true when stored. + virtual bool put_contact_if_absent(const std::string& jid, const std::string& name) = 0; virtual std::vector> contacts() = 0; // (jid,name) virtual void remove_contact(const std::string& jid) = 0; @@ -51,6 +54,9 @@ class WaStorePort { virtual void delete_app_state_value_mac(const std::string& name, const Bytes& index_mac) = 0; virtual std::optional app_key_request_time(const Bytes& key_id) = 0; virtual void put_app_key_request_time(const Bytes& key_id, int64_t when) = 0; + // Forgets a request mark (whatsmeow PR 1268): when the key request never left this device, the next sync + // must be free to ask again instead of waiting out the 24 h. + virtual void clear_app_key_request_time(const Bytes& key_id) = 0; }; class WaClient { @@ -89,7 +95,28 @@ class WaClient { // kWaMinPrekeys (iq xmlns="encrypt"). Returns true when enough are present. bool upload_prekeys(); // Pull the app-state contact list now (login runs this once). Public for the probe and tests. - void sync_contacts() { sync_app_state(); } + void sync_contacts(); + // Same, but waits up to `wait_seconds` for the phone's key share instead of the login default. Used by + // `--whatsapp-sync`: a phone that is asleep can take minutes to wake and answer. + void sync_contacts_with_wait(int wait_seconds); + + // Pure helpers (no network), public for unit tests. + // whatsmeow IsFromMe for a 1:1 sender: the user part matches our phone number or our LID. + static bool is_own_sender(const std::string& from, const std::string& own_pn_user, const std::string& own_lid_user); + // A peer message node (whatsmeow preparePeerMessageNode): meta + enc, device-identity only on a prekey + // message. `to` is our own phone-number JID; the enc was encrypted to the phone's LID identity. + static WaNode build_peer_message_node(const std::string& id, const std::string& to, const WaNode& enc, bool is_prekey, + const Bytes& account); + // Collection names carried by a server_sync notification (empty when none). + static std::vector server_sync_collections(const WaNode& notification); + // True for a retry receipt: the phone could not decrypt one of our messages and asks for a resend. + static bool is_retry_receipt(const WaNode& node); + // A contact learnable from an incoming : the sender's push name (the `notify` attribute) plus the + // sender JID in non-AD user@server form (the PN form when an alternate is present). Empty when there is nothing + // to learn: no usable notify (missing or "-"), the sender is ourselves, or the sender is not a user (group + // senders come from `participant`). Pure, unit-tested. + static std::optional> push_name_candidate( + const WaNode& node, const std::string& own_pn_user, const std::string& own_lid_user); // Send a 1:1 text message. Requires a paired, logged-in client. Returns the // message id on success. Live path, not verified in-tree. @@ -137,14 +164,23 @@ class WaClient { void send_ack(const WaNode& node); // --- app-state sync and key exchange (docs/whatsapp.md section 14) --- int app_keys_received_{0}; // app-state keys stored by this client instance + bool app_keys_answered_{false}; // the phone sent a key share (with or without key data) + bool sync_active_{false}; // reentrancy guard: a notification sync never nests inside a sync + bool sync_again_{false}; // a server_sync notification arrived mid-sync: one more pass after // Receives a : acked by dispatch; decrypts the phone's own messages and stores key shares. void handle_message(const WaNode& node); + // Receives a : acked here; server_sync re-syncs the contact collection, encrypt refills prekeys. + void handle_notification(const WaNode& node); + // Receives a : retry receipts (the phone could not decrypt our message) are logged loudly. + void handle_receipt(const WaNode& node); + // Sends a protocol receipt (peer_msg / hist_sync) for a consumed peer message, as whatsmeow does. + void send_protocol_receipt(const std::string& id, const std::string& type); // Sends a peer message (category "peer") to our own primary device and waits for its ack. bool send_peer_message(const Bytes& message_plaintext); // Asks the phone for the given app-state key ids, at most one request per key id per 24 h. void request_app_state_keys(const std::vector& key_ids); // Syncs the contact collection (login hook); never throws to the caller. - void sync_app_state(); + void sync_app_state(int key_wait_seconds); std::string next_iq_id(); // --- send path helpers (live, unverified in-tree) --- diff --git a/taicpp/src/whatsapp/wa_consts.h b/taicpp/src/whatsapp/wa_consts.h index d5098e4..d7c97fd 100644 --- a/taicpp/src/whatsapp/wa_consts.h +++ b/taicpp/src/whatsapp/wa_consts.h @@ -10,6 +10,11 @@ namespace detz::wa { struct WaConsts { // The WhatsApp server JID (whatsmeow types.ServerJID). Every pairing and login reply is addressed here. static constexpr const char* kServerJid = "s.whatsapp.net"; + // LID and group-chat servers (whatsmeow HiddenUserServer, GroupServer). + static constexpr const char* kLidServer = "lid"; + static constexpr const char* kGroupServer = "g.us"; + // A push name learned from an incoming message is truncated to this, never stored longer. + static constexpr size_t kWaPushNameMax = 128; // WhatsApp Web client version (whatsmeow store/clientpayload.go). A stale // value makes the server answer with a 405 "client outdated" failure. @@ -35,6 +40,14 @@ struct WaConsts { static constexpr int64_t kWaAppStateKeyRequestInterval = 24 * 60 * 60; // seconds between key requests for one key id static constexpr int kWaAppStateKeyWait = 20; // seconds to wait for requested keys during login static constexpr size_t kWaAppStateKeyIdsMax = 16; // key ids per key request message + static constexpr int kWaSyncOnlineWait = 300; // seconds --whatsapp-sync stays online for the key share + // protocol receipt types (whatsmeow types.ReceiptType*): sent for a consumed peer message, or received when the + // phone could not decrypt one of ours (retry) + static constexpr const char* kWaReceiptPeerMsg = "peer_msg"; + static constexpr const char* kWaReceiptHistSync = "hist_sync"; + static constexpr const char* kWaReceiptRetry = "retry"; + // vCard contact import fallback (--whatsapp-import) + static constexpr size_t kWaVcfFileMax = 8u * 1024 * 1024; // refuse a bigger .vcf, loud }; } // namespace detz::wa diff --git a/taicpp/src/whatsapp/wa_store.cpp b/taicpp/src/whatsapp/wa_store.cpp index 24a708a..36c4555 100644 --- a/taicpp/src/whatsapp/wa_store.cpp +++ b/taicpp/src/whatsapp/wa_store.cpp @@ -124,6 +124,13 @@ void WaStore::put_contact(const std::string& jid, const std::string& name) { db_->commit(); } +bool WaStore::put_contact_if_absent(const std::string& jid, const std::string& name) { + auto cur = db_->execute("SELECT name FROM wa_contact WHERE jid=?", {store::Cell(jid)}); + if (cur.first() != nullptr) return false; // a saved, imported or aliased name always wins over a push name + put_contact(jid, name); + return true; +} + std::vector> WaStore::contacts() { auto cur = db_->execute("SELECT jid, name FROM wa_contact ORDER BY name"); std::vector> out; @@ -268,6 +275,11 @@ void WaStore::put_app_key_request_time(const Bytes& key_id, int64_t when) { db_->commit(); } +void WaStore::clear_app_key_request_time(const Bytes& key_id) { + db_->execute("DELETE FROM wa_appstate_keyreq WHERE key_id=?", {store::Cell(tohex(key_id))}); + db_->commit(); +} + void WaStore::clear_device() { db_->execute("DELETE FROM wa_device"); db_->execute("DELETE FROM wa_session"); diff --git a/taicpp/src/whatsapp/wa_store.h b/taicpp/src/whatsapp/wa_store.h index 3a07d5b..9106b8b 100644 --- a/taicpp/src/whatsapp/wa_store.h +++ b/taicpp/src/whatsapp/wa_store.h @@ -29,6 +29,7 @@ class WaStore : public WaStorePort { void put_prekey(uint32_t id, const Bytes& priv) override; std::optional get_prekey(uint32_t id) override; void put_contact(const std::string& jid, const std::string& name) override; + bool put_contact_if_absent(const std::string& jid, const std::string& name) override; std::vector> contacts() override; void remove_contact(const std::string& jid) override; @@ -53,6 +54,7 @@ class WaStore : public WaStorePort { void delete_app_state_value_mac(const std::string& name, const Bytes& index_mac) override; std::optional app_key_request_time(const Bytes& key_id) override; void put_app_key_request_time(const Bytes& key_id, int64_t when) override; + void clear_app_key_request_time(const Bytes& key_id) override; // true if a paired device is stored (used to decide link vs send). bool has_device(); diff --git a/taicpp/src/whatsapp/wa_vcf.cpp b/taicpp/src/whatsapp/wa_vcf.cpp new file mode 100644 index 0000000..8aab6e5 --- /dev/null +++ b/taicpp/src/whatsapp/wa_vcf.cpp @@ -0,0 +1,151 @@ +#include "wa_vcf.h" + +#include "wa_consts.h" + +namespace detz::wa { + +namespace { +std::string strip(const std::string& s) { + size_t a = s.find_first_not_of(" \t\r\n"); + if (a == std::string::npos) return {}; + return s.substr(a, s.find_last_not_of(" \t\r\n") - a + 1); +} +// vCard text unescaping: \, \; \n \\ (3.0/4.0 section 3.4). +std::string unescape(const std::string& s) { + std::string out; + for (size_t i = 0; i < s.size(); ++i) { + if (s[i] == '\\' && i + 1 < s.size()) { + char n = s[++i]; + out.push_back(n == 'n' || n == 'N' ? ' ' : n); + } else { + out.push_back(s[i]); + } + } + return strip(out); +} +// A TEL value to international digits: keeps a leading '+' (dropped), digits; needs a country code (>= 7 digits). +std::string normalize_number(const std::string& v) { + std::string digits; + for (char c : v) { + if (c >= '0' && c <= '9') digits.push_back(c); + } + // some exports write the national form with a trunk 0 and no country code; that is unusable, drop it loudly + if (digits.size() < 7 || (digits.size() < 10 && digits[0] == '0')) return {}; + if (digits.size() >= 10 && digits[0] == '0' && digits[1] == '0') digits.erase(0, 2); // 00 form + return digits; +} +} // namespace + +std::vector WaVcf::parse(const std::string& text, int& skipped) { + skipped = 0; + // unfold: a line starting with a space or tab continues the previous line (the check runs on the raw line: + // stripping first would eat the continuation mark). Content spaces survive; only the CRLF goes. + std::vector lines; + std::string cur; + for (size_t i = 0; i < text.size();) { + size_t e = text.find('\n', i); + std::string raw = text.substr(i, e == std::string::npos ? e : e - i); + i = e == std::string::npos ? text.size() : e + 1; + if (!raw.empty() && raw.back() == '\r') raw.pop_back(); + if (!raw.empty() && (raw[0] == ' ' || raw[0] == '\t') && !cur.empty()) { + cur += strip(raw); + } else { + if (!cur.empty()) lines.push_back(strip(cur)); + cur = raw; + } + } + if (!cur.empty()) lines.push_back(strip(cur)); + + std::vector out; + bool in_card = false; + std::string fn, n; + std::vector tels; + auto flush = [&] { + if (!in_card) return; + in_card = false; + std::string name = strip(fn); + if (name.empty() && !n.empty()) { + // N: Family;Given;Middle;Prefix;Suffix -> "Given Family" + std::vector parts{""}; + for (char c : n) { + if (c == ';') parts.push_back(""); + else parts.back().push_back(c); + } + std::string given = parts.size() > 1 ? strip(parts[1]) : ""; + std::string family = strip(parts[0]); + name = strip(given.empty() ? family : given + (family.empty() ? "" : " " + family)); + } + if (name.empty() || tels.empty()) { + skipped++; + return; + } + bool any = false; + for (const auto& t : tels) { + std::string digits = normalize_number(t); + if (digits.empty()) continue; + out.push_back({name, digits}); + any = true; + } + if (!any) skipped++; + }; + for (const auto& line : lines) { + size_t colon = line.find(':'); + if (colon == std::string::npos) continue; + std::string head = line.substr(0, colon); + std::string value = line.substr(colon + 1); + std::string prop; + for (char c : head) { + if (c == ';') break; // parameters (TYPE=..., CHARSET=...) do not change the property + prop.push_back(static_cast(std::toupper(static_cast(c)))); + } + if (prop == "BEGIN") { + flush(); + if (value == "VCARD" || value == "vCard") { + in_card = true; + fn.clear(); + n.clear(); + tels.clear(); + } + } else if (prop == "END") { + flush(); + } else if (!in_card) { + continue; + } else if (prop == "FN") { + if (fn.empty()) fn = unescape(value); + } else if (prop == "N") { + if (n.empty()) n = unescape(value); + } else if (prop == "TEL") { + if (tels.size() < 8) tels.push_back(value); // bound per-card numbers; more are noise + } + } + flush(); + return out; +} + +WaVcf::Report WaVcf::import_text(WaStorePort& store, const std::string& text) { + Report rep; + int skipped = 0; + std::vector contacts; + try { + contacts = parse(text, skipped); + } catch (...) { + rep.skipped = 1; + return rep; + } + rep.skipped = skipped; + for (const auto& c : contacts) { + if (rep.imported >= WaConsts::kWaContactsMax) { + rep.capped++; + continue; + } + try { + store.put_contact(c.jid(), c.name); + rep.imported++; + } catch (...) { + rep.skipped++; + } + } + return rep; +} + +} // namespace detz::wa diff --git a/taicpp/src/whatsapp/wa_vcf.h b/taicpp/src/whatsapp/wa_vcf.h new file mode 100644 index 0000000..6bfcca8 --- /dev/null +++ b/taicpp/src/whatsapp/wa_vcf.h @@ -0,0 +1,32 @@ +#pragma once +// WaVcf: the offline fallback for the phone address book (problem.md). When the phone never sends its +// app-state sync keys, the user exports the contacts on the phone as a .vcf file and imports it here; the +// names land in the same contact search the app-state sync feeds. vCard 3.0/4.0, FN (else N) plus TEL lines. +// See docs/whatsapp.md section 14. +#include +#include + +#include "wa_client.h" + +namespace detz::wa { + +class WaVcf { + public: + struct Contact { + std::string name; // display name (FN, else "Given Family" from N) + std::string number; // digits with country code, no '+' + std::string jid() const { return number + "@s.whatsapp.net"; } + }; + struct Report { + int imported{0}; + int skipped{0}; // entries without a usable name or number + int capped{0}; // entries past kWaContactsMax: not stored + }; + + // Parses vCard text (unfolds continuations, reads BEGIN/END blocks). Pure, unit-tested. + static std::vector parse(const std::string& text, int& skipped); + // Parses and stores into the contact search. Never throws to the caller; `report` says what happened. + static Report import_text(WaStorePort& store, const std::string& text); +}; + +} // namespace detz::wa diff --git a/taicpp/src/whatsapp/wa_wizard.cpp b/taicpp/src/whatsapp/wa_wizard.cpp index 27bf31f..b248f25 100644 --- a/taicpp/src/whatsapp/wa_wizard.cpp +++ b/taicpp/src/whatsapp/wa_wizard.cpp @@ -3,11 +3,13 @@ #include #include #include +#include #include "wa_client.h" #include "wa_consts.h" #include "wa_qr.h" #include "wa_store.h" +#include "wa_vcf.h" namespace detz::wa { @@ -116,4 +118,68 @@ int WaWizard::status(const std::string& home, const std::string& profile, std::o return 1; } +int WaWizard::sync(const std::string& home, const std::string& profile, std::ostream& out, int wait_seconds) { + WaStore store(home, profile); + if (!store.has_device()) { + out << "WhatsApp: not linked (run --whatsapp-link first)\n"; + return 2; + } + if (wait_seconds < 0) wait_seconds = 0; + if (wait_seconds > 3600) wait_seconds = 3600; // an hour online is enough for any phone + WaClient client(store); + client.set_capture_path(home + "/whatsapp/last-error.log"); + client.on_log = [&](const std::string& s) { out << " " << s << "\n"; }; + out << "Logging in...\n"; + if (!client.login(WaConsts::kWaLoginTimeout)) { + out << "WhatsApp login did not complete" + << (client.last_failure_code() ? " (server reason " + std::to_string(client.last_failure_code()) + ")" : "") + << ".\n"; + return 2; + } + // login() already synced once with the short wait; stay online so a sleeping phone can still answer the key + // request it was just sent. + out << "Logged in. Waiting up to " << wait_seconds << "s for the phone's address book" + << " (keep WhatsApp open on the phone)...\n"; + client.sync_contacts_with_wait(wait_seconds); + client.disconnect(); + size_t n = store.contacts().size(); + if (n > 0) { + out << "WhatsApp contacts: " << n << " known.\n"; + return 0; + } + out << "Still no contacts. The phone did not send its address book keys. This usually means WhatsApp on the\n" + << "phone is asleep or battery-restricted: open WhatsApp on the phone, turn off battery optimization for\n" + << "it, and run this again. As a fallback, export the contacts on the phone as a .vcf file and run\n" + << "`detz --whatsapp-import contacts.vcf`.\n"; + return 1; +} + +int WaWizard::import_vcf(const std::string& home, const std::string& profile, const std::string& path, + std::ostream& out) { + std::error_code ec; + auto size = std::filesystem::file_size(path, ec); + if (ec) { + out << "Cannot read " << path << ".\n"; + return 1; + } + if (size > WaConsts::kWaVcfFileMax) { + out << path << " is " << size << " bytes, over the " << WaConsts::kWaVcfFileMax << " byte limit. Refusing.\n"; + return 1; + } + std::ifstream in(path, std::ios::binary); + if (!in) { + out << "Cannot read " << path << ".\n"; + return 1; + } + std::string text((std::istreambuf_iterator(in)), std::istreambuf_iterator()); + WaStore store(home, profile); + WaVcf::Report rep = WaVcf::import_text(store, text); + out << "Imported " << rep.imported << " contact number(s) from " << path; + if (rep.skipped > 0) out << " (" << rep.skipped << " skipped: no usable name or number)"; + if (rep.capped > 0) out << " (" << rep.capped << " past the " << WaConsts::kWaContactsMax << " contact cap)"; + out << ".\n"; + if (rep.imported == 0) return 1; + return 0; +} + } // namespace detz::wa diff --git a/taicpp/src/whatsapp/wa_wizard.h b/taicpp/src/whatsapp/wa_wizard.h index e7d7d9e..d43671e 100644 --- a/taicpp/src/whatsapp/wa_wizard.h +++ b/taicpp/src/whatsapp/wa_wizard.h @@ -6,6 +6,8 @@ #include #include +#include "wa_consts.h" + namespace detz::wa { class WaWizard { @@ -17,6 +19,14 @@ class WaWizard { static int unlink(const std::string& home, const std::string& profile, std::ostream& out); // Print the current link status. 0 if linked, 1 if not. static int status(const std::string& home, const std::string& profile, std::ostream& out); + // Stay online and pull the phone address book (contact sync with a long key wait for a sleeping phone). + // 0 when at least one contact is known afterwards, 1 when none arrived, 2 when not linked / login failed. + static int sync(const std::string& home, const std::string& profile, std::ostream& out, + int wait_seconds = WaConsts::kWaSyncOnlineWait); + // Import a .vcf file exported on the phone into the contact search (the offline fallback when the phone + // never sends its app-state sync keys). 0 on success, 1 when the file cannot be read or holds no contacts. + static int import_vcf(const std::string& home, const std::string& profile, const std::string& path, + std::ostream& out); }; } // namespace detz::wa diff --git a/taicpp/tests/whatsapp_test.cpp b/taicpp/tests/whatsapp_test.cpp index 9ccfe54..4348316 100644 --- a/taicpp/tests/whatsapp_test.cpp +++ b/taicpp/tests/whatsapp_test.cpp @@ -15,6 +15,7 @@ #include "json.h" #include "wa_binary.h" +#include "wa_client.h" #include "wa_crypto.h" #include "wa_device.h" #include "wa_noise.h" @@ -24,6 +25,7 @@ #include "wa_media.h" #include "wa_receiver.h" #include "wa_store.h" +#include "wa_vcf.h" using namespace detz::wa; using detz::json::Value; @@ -666,6 +668,200 @@ static void test_contact_search() { std::filesystem::remove_all(home); } +// Key-request bookkeeping (whatsmeow PR 1268): a request that never left the device must not block the +// retry for 24 h, so the mark can be cleared again. +static void test_key_request_rollback() { + std::printf("[appstate: key request rollback]\n"); + std::string home = make_temp_home(); + WaStore store(home, "keyreq"); + Bytes id{0x01, 0x02, 0x03}; + check(!store.app_key_request_time(id).has_value(), "no request mark for a fresh key id"); + store.put_app_key_request_time(id, 1700000000); + auto got = store.app_key_request_time(id); + check(got && *got == 1700000000, "request mark is stored"); + store.clear_app_key_request_time(id); + check(!store.app_key_request_time(id).has_value(), "clearing the mark frees the key id for an immediate retry"); + store.clear_app_key_request_time(id); // clearing twice is a no-op, not an error + check(!store.app_key_request_time(id).has_value(), "clearing an unmarked key id is harmless"); + std::filesystem::remove_all(home); +} + +// Peer message helpers: own-sender classification (PN or LID, whatsmeow IsFromMe), the peer node shape +// (device-identity only on a prekey message), server_sync parsing and retry-receipt detection. +static void test_peer_helpers() { + std::printf("[appstate: peer message helpers]\n"); + check(WaClient::is_own_sender("31630045827@s.whatsapp.net", "31630045827", "173735822786802"), + "own PN sender is accepted"); + check(WaClient::is_own_sender("173735822786802@lid", "31630045827", "173735822786802"), + "own LID sender is accepted (the phone answers from its LID)"); + check(WaClient::is_own_sender("31630045827:0@s.whatsapp.net", "31630045827", ""), + "own sender with an explicit device 0 is accepted"); + check(!WaClient::is_own_sender("31611111111@s.whatsapp.net", "31630045827", "173735822786802"), + "another user is not an own sender"); + check(!WaClient::is_own_sender("173735822786802@lid", "31630045827", ""), + "an unknown LID is not accepted without a stored LID"); + + WaNode enc("enc"); + enc.attrs["v"] = "2"; + enc.attrs["type"] = "pkmsg"; + enc.content = Bytes{1, 2, 3}; + Bytes account{9, 9, 9}; + WaNode pre = WaClient::build_peer_message_node("3EB0AA", "31630045827@s.whatsapp.net", enc, true, account); + check(pre.tag == "message" && pre.attr("category") == "peer" && pre.attr("push_priority") == "high" && + pre.attr("to") == "31630045827@s.whatsapp.net" && pre.child("meta") != nullptr && + pre.child("enc") != nullptr && pre.child("device-identity") != nullptr, + "peer node: meta + enc + device-identity on a prekey message"); + WaNode msg = WaClient::build_peer_message_node("3EB0AB", "31630045827@s.whatsapp.net", enc, false, account); + check(msg.child("device-identity") == nullptr && msg.child("enc") != nullptr, + "peer node: no device-identity on an existing-session message"); + + WaNode notif("notification"); + notif.attrs["type"] = "server_sync"; + WaNode c1("collection"); + c1.attrs["name"] = "critical_unblock_low"; + WaNode c2("collection"); + c2.attrs["name"] = "regular_high"; + notif.content = WaNode::Children{c1, c2}; + auto cols = WaClient::server_sync_collections(notif); + check(cols.size() == 2 && cols[0] == "critical_unblock_low" && cols[1] == "regular_high", + "server_sync notification yields its collection names"); + + WaNode retry("receipt"); + retry.attrs["type"] = "retry"; + WaNode read("receipt"); + read.attrs["type"] = "read"; + WaNode ack("ack"); + check(WaClient::is_retry_receipt(retry) && !WaClient::is_retry_receipt(read) && !WaClient::is_retry_receipt(ack), + "only a retry receipt reads as one"); +} + +// Empty key shares (openclaw/wacli PR 443): the phone answers with keys that carry no key data. The share +// still parses, so the client can tell "answered empty" from "never answered". +static void test_empty_key_share() { + std::printf("[appstate: empty key share]\n"); + ProtoWriter key; + { + ProtoWriter keyid; + keyid.bytes(1, Bytes{0x7c, 0x57}); + key.message(1, keyid.data()); + ProtoWriter data; // AppStateSyncKeyData without keyData: the phone has none + data.varint(3, 1767225600); + key.message(2, data.data()); + } + ProtoWriter share; + share.message(1, key.data()); + ProtoWriter pm; + pm.varint(2, 6); // APP_STATE_SYNC_KEY_SHARE + pm.message(7, share.data()); + ProtoWriter m; + m.message(12, pm.data()); + WaProtocolMessage out = parse_protocol_message(m.take()); + check(out.type == 6 && out.key_share.size() == 1 && out.key_share[0].key_data.empty(), + "an empty key share parses with empty key data (answered, but nothing to store)"); +} + +// Push-name learning: every incoming message carries the sender's self-reported name in `notify` +// (whatsmeow message.go). A sender we have no name for is learned; saved, imported and aliased names win. +static void test_push_name_learning() { + std::printf("[contacts: push-name learning]\n"); + const std::string me = "31630045827", my_lid = "173735822786802"; + auto msg = [](const std::string& from, const std::string& notify) { + WaNode m("message"); + m.attrs["from"] = from; + if (!notify.empty()) m.attrs["notify"] = notify; + return m; + }; + auto cand = [&](const WaNode& m) { return WaClient::push_name_candidate(m, me, my_lid); }; + { + auto c = cand(msg("31611111111@s.whatsapp.net", "Nigel Dokter")); + check(c && c->first == "31611111111@s.whatsapp.net" && c->second == "Nigel Dokter", + "1:1 message learns the sender's push name"); + } + { + WaNode m = msg("179000000000001@lid", "Anne"); + m.attrs["sender_pn"] = "31622222222@s.whatsapp.net"; + auto c = cand(m); + check(c && c->first == "31622222222@s.whatsapp.net" && c->second == "Anne", + "a LID sender with a PN alternate is learned under the sendable PN"); + } + { + auto c = cand(msg("179000000000001@lid", "Anne")); + check(c && c->first == "179000000000001@lid", "a LID sender without alternate is learned under its LID"); + } + { + WaNode m = msg("120363000000000@g.us", "Joop"); + m.attrs["participant"] = "31633333333@s.whatsapp.net"; + auto c = cand(m); + check(c && c->first == "31633333333@s.whatsapp.net" && c->second == "Joop", + "a group message learns the participant, not the group"); + } + { + WaNode m = msg("120363000000000@g.us", "Joop"); + m.attrs["participant"] = "179000000000002@lid"; + m.attrs["participant_pn"] = "31633333333@s.whatsapp.net"; + auto c = cand(m); + check(c && c->first == "31633333333@s.whatsapp.net", "a group LID participant prefers its PN alternate"); + } + check(!cand(msg("31630045827@s.whatsapp.net", "Me")), "our own messages teach nothing"); + check(!cand(msg("31611111111@s.whatsapp.net", "-")), "a '-' push name is not learned"); + check(!cand(msg("31611111111@s.whatsapp.net", "")), "a missing push name is not learned"); + check(!cand(msg("1234567890@newsletter", "Channel")), "a newsletter sender is not learned"); + { + auto c = cand(msg("31611111111@s.whatsapp.net", std::string(300, 'x'))); + check(c && c->second.size() == 128, "an over-long push name is truncated, not refused"); + } + + std::string home = make_temp_home(); + WaStore store(home, "push"); + check(store.put_contact_if_absent("31611111111@s.whatsapp.net", "Nigel Dokter"), + "first sighting of a sender is stored"); + check(!store.put_contact_if_absent("31611111111@s.whatsapp.net", "Nigel Changed"), + "a second sighting does not overwrite"); + check(store.contacts().size() == 1 && store.contacts()[0].second == "Nigel Dokter", + "the first learned name sticks"); + auto hits = store.search_contacts("Niggel", 3, 0.3); + check(!hits.empty() && hits[0].name == "Nigel Dokter", "a learned name is searchable with typos"); + store.put_contact("31611111111@s.whatsapp.net", "Nigel Dokter (work)"); // app-state-grade name wins + check(store.contacts()[0].second == "Nigel Dokter (work)", "a saved name overwrites a learned push name"); + check(!store.put_contact_if_absent("31611111111@s.whatsapp.net", "Nigel Again"), + "a push name never overwrites a saved name"); + std::filesystem::remove_all(home); +} + +// VCF fallback (problem.md): a .vcf exported on the phone feeds the same contact search the app-state sync feeds. +static void test_vcf_import() { + std::printf("[contacts: vcf import]\n"); + const char* vcf = + "BEGIN:VCARD\r\nVERSION:3.0\r\nFN:Nigel Dokter\r\nTEL;TYPE=CELL:+31 6 1234 5001\r\nEND:VCARD\r\n" + "BEGIN:VCARD\r\nVERSION:3.0\r\nN:Smit;Nina;;;\r\nTEL:+31612345002\r\nTEL:0031612345003\r\nEND:VCARD\r\n" + "BEGIN:VCARD\r\nVERSION:4.0\r\nFN:Long \r\n Name\r\nTEL;TYPE=HOME:+31-6-1234-5004\r\nEND:VCARD\r\n" + "BEGIN:VCARD\r\nVERSION:3.0\r\nFN:No Number\r\nEND:VCARD\r\n" + "BEGIN:VCARD\r\nVERSION:3.0\r\nFN:Local Only\r\nTEL:0612345\r\nEND:VCARD\r\n"; + int skipped = 0; + auto contacts = WaVcf::parse(vcf, skipped); + check(contacts.size() == 4, "four usable numbers parse from five cards (one numberless, one local)"); + check(skipped == 2, "two cards skipped: no number, and a national number without country code"); + bool nigel = false, nina = false, unfolded = false; + for (const auto& c : contacts) { + if (c.name == "Nigel Dokter" && c.number == "31612345001") nigel = true; + if (c.name == "Nina Smit" && (c.number == "31612345002" || c.number == "31612345003")) nina = true; + if (c.name == "Long Name") unfolded = true; + } + check(nigel, "FN plus an international TEL with spaces"); + check(nina, "N fallback name plus two TELs (00 form normalized)"); + check(unfolded, "a folded FN line is unfolded"); + + std::string home = make_temp_home(); + WaStore store(home, "vcf"); + WaVcf::Report rep = WaVcf::import_text(store, vcf); + check(rep.imported == 4 && rep.skipped == 2 && rep.capped == 0, "import stores 4 numbers, skips 2, caps none"); + auto hits = store.search_contacts("Niggel", 3, 0.3); + check(!hits.empty() && hits[0].name == "Nigel Dokter", "an imported contact is found by the typo-tolerant search"); + WaVcf::Report empty = WaVcf::import_text(store, "not a vcard at all"); + check(empty.imported == 0, "garbage imports nothing and throws nothing"); + std::filesystem::remove_all(home); +} + int main(int argc, char** argv) { DIR = argc > 1 ? argv[1] : "tests/whatsapp_data"; test_contact_search(); @@ -682,6 +878,11 @@ int main(int argc, char** argv) { test_lthash_and_keys(); test_appstate_sync(); test_receiver(); + test_key_request_rollback(); + test_peer_helpers(); + test_empty_key_share(); + test_vcf_import(); + test_push_name_learning(); std::printf("%s\n", g_fail == 0 ? "whatsapp: ALL OK" : "whatsapp: FAILURES"); return g_fail == 0 ? 0 : 1; }