detz: baseline 1.46.1 pre-transpilation
This commit is contained in:
@@ -1,5 +1,13 @@
|
||||
# Changelog
|
||||
|
||||
## [1.46.1] - 2026-10-27
|
||||
- WhatsApp learns contact names from incoming messages (push names)
|
||||
|
||||
|
||||
## [1.46.0] - 2026-10-27
|
||||
- WhatsApp key-share hardening, contact sync wait and VCF import fallback
|
||||
|
||||
|
||||
## [1.45.0] - 2026-10-27
|
||||
- WhatsApp link fix, image send, contact sync and search, failover hardening
|
||||
|
||||
|
||||
+1
-1
@@ -1 +1 @@
|
||||
1.45.0
|
||||
1.46.1
|
||||
|
||||
+69
-6
@@ -581,18 +581,23 @@ it `wa_contact` stays empty and names cannot be resolved.
|
||||
|
||||
| Piece | Files | What it does |
|
||||
|---|---|---|
|
||||
| Key receipt | `wa_receiver.*`, `wa_proto.*` (`parse_protocol_message`), `WaClient::handle_message` | Decrypts `<enc>` from our own account (pkmsg via X3DH responder, msg via session), unpads, stores each APP_STATE_SYNC_KEY_SHARE key in `wa_appstate_key`. Only the phone's own messages are decrypted; other senders are acked and ignored as before. |
|
||||
| Key request | `WaClient::request_app_state_keys`, `send_peer_message`, `encode_app_state_key_request` | Peer message to the primary phone (category `peer`), at most `kWaAppStateKeyIdsMax` ids, one request per key id per `kWaAppStateKeyRequestInterval` (24 h), stored in `wa_appstate_keyreq`. |
|
||||
| Key receipt | `wa_receiver.*`, `wa_proto.*` (`parse_protocol_message`), `WaClient::handle_message` | Decrypts `<enc>` from our own account (pkmsg via X3DH responder, msg via session), unpads, stores each APP_STATE_SYNC_KEY_SHARE key in `wa_appstate_key`. Own means our phone number or our LID (whatsmeow `IsFromMe`); other senders are acked and ignored as before. Every arrival is logged before the filter, so a trace shows dropped mail too. An empty share (keys without key data, openclaw/wacli PR 443) is reported and ends the wait without storing; a consumed peer message gets a `peer_msg` protocol receipt. |
|
||||
| Key request | `WaClient::request_app_state_keys`, `send_peer_message`, `encode_app_state_key_request` | Peer message to the primary phone (category `peer`, meta + enc, device-identity only on a prekey message), at most `kWaAppStateKeyIdsMax` ids, one request per key id per `kWaAppStateKeyRequestInterval` (24 h), stored in `wa_appstate_keyreq`. The send is verified (Signal session must exist, the server ack must carry no error); when it fails the 24 h marks are rolled back so the next sync asks again (whatsmeow PR 1268). |
|
||||
| Notifications, receipts | `WaClient::handle_notification`, `handle_receipt` | `server_sync` re-syncs the contact collection (other collections are named and skipped); `encrypt` with fewer than `kWaMinPrekeys` left refills prekeys; a retry receipt (the phone could not decrypt our message) is logged loudly. Notification syncs never nest inside a sync (guard + one more pass). |
|
||||
| Key expansion, MACs, values | `wa_appstate.*` (`WaAppStateKeys`, `WaAppStateCrypto`) | HKDF-SHA256 `WhatsApp Mutation Keys` to 160 bytes; value MAC HMAC-SHA512 truncated to 32; index, snapshot and patch MACs HMAC-SHA256; AES-256-CBC values. `WaCrypto::hmac_sha512` added. |
|
||||
| LTHash | `wa_appstate.*` (`WaLtHash`) | 128-byte state, HKDF-SHA256 info `WhatsApp Patch Integrity` (whatsmeow, not BLAKE2b; see research conclusion `appstate-lthash-hkdf-not-blake2b`). |
|
||||
| Patch fetch and apply | `WaAppStateSync::sync` | iq `w:sync:app:state`, `return_snapshot` at version 0, `version=N` otherwise; external blobs via `WaMedia::download` (`md-app-state`); snapshot MAC, patch MAC, index and value MACs checked before anything is stored. |
|
||||
| Contacts | `WaAppStateSync::apply_contact` | SET stores jid = index[1], name = fullName else firstName; empty name is counted and skipped; REMOVE deletes the row. |
|
||||
| Push-name learning | `WaClient::handle_message`, `push_name_candidate`, `WaStore::put_contact_if_absent` | Every incoming message carries the sender's self-reported name in `notify` (whatsmeow `message.go`); group senders come from `participant`, the PN alternate is preferred when present. Stored only when the jid has no name yet, so a saved, imported or aliased name always wins; truncated to `kWaPushNameMax` (128). No server query returns names for regular users (`GetUserInfo` usync has status, picture, devices, LID and business verified names only), so this plus VCF/alias/app-state is the complete name-resolution stack. |
|
||||
| Login hook | `WaClient::login` calls `sync_app_state` after prekeys | Never fails the login (errors are logged). If a key is missing, requests it, waits up to `kWaAppStateKeyWait` s for a key share, then syncs once more. |
|
||||
| Long sync | `detz --whatsapp-sync` (`WaWizard::sync`) | Logs in, re-syncs with a `kWaSyncOnlineWait` (300 s) key wait for a sleeping phone, reports the contact count. Exit 0 with contacts, 1 without, 2 when not linked or login failed. |
|
||||
| VCF fallback | `detz --whatsapp-import FILE` (`WaVcf`) | Imports a .vcf exported on the phone (FN else N, TEL lines, international digits) into the same contact search. Loud skips, capped at `kWaContactsMax`, file capped at `kWaVcfFileMax`. |
|
||||
|
||||
Caps (`wa_consts.h`): `kWaAppStatePatchMax` 50 patches per sync,
|
||||
`kWaAppStateMutationMax` 10000 mutations, `kWaAppStateTimeout` 30 s per iq,
|
||||
`kWaAppStateBlobMax` 16 MiB per external blob, `kWaAppStateKeyIdsMax` 16 key
|
||||
ids per request, `kWaAppStateKeyWait` 20 s. A cap stops the sync loudly
|
||||
ids per request, `kWaAppStateKeyWait` 20 s, `kWaSyncOnlineWait` 300 s,
|
||||
`kWaVcfFileMax` 8 MiB. A cap stops the sync loudly
|
||||
(`capped` in the report, a log line) and the rest waits for the next login.
|
||||
|
||||
Failure rule: a patch or snapshot that fails a MAC or hash check is not stored,
|
||||
@@ -616,7 +621,17 @@ Verification (offline, `build/whatsapp_test`, section `appstate`):
|
||||
- a missing key stores nothing and reports the key id;
|
||||
- the key share and key request bytes match whatsmeow's Message encoding;
|
||||
- a pkmsg built with `WaSession` decrypts through `WaReceiver` to the key share
|
||||
Message, and a corrupted pkmsg is refused.
|
||||
Message, and a corrupted pkmsg is refused;
|
||||
- the 24 h request mark can be cleared for an immediate retry;
|
||||
- own senders accept PN and LID, the peer node carries device-identity only on
|
||||
a prekey message, server_sync collections and retry receipts parse;
|
||||
- an empty key share parses with empty key data;
|
||||
- a 5-card .vcf imports 4 numbers (FN, N fallback, unfolding, 00 form) and the
|
||||
typo-tolerant search finds them;
|
||||
- push-name candidates parse for 1:1 (PN, LID, PN alternate), groups
|
||||
(participant, PN alternate), and reject own senders, `-`, missing notify,
|
||||
non-user senders and over-long names; fill-if-absent keeps the first name and
|
||||
never overwrites a saved one.
|
||||
|
||||
The oracle is `tests/whatsapp_data/appstate_vectors.json`, generated by
|
||||
`tests/whatsapp_data/appstate_oracle.go` (a Go program using whatsmeow's own
|
||||
@@ -624,8 +639,10 @@ The oracle is `tests/whatsapp_data/appstate_vectors.json`, generated by
|
||||
|
||||
Not verified: the live server's answers to the iq, key share timing, the media
|
||||
host for external blobs, and whether a REMOVE of an unknown index (whatsmeow
|
||||
only warns) causes trouble. Known gap: a SET with an empty name keeps any older
|
||||
name the row had; the spec says skip, not delete.
|
||||
only warns) causes trouble. Known gaps: a SET with an empty name keeps any
|
||||
older name the row had (the spec says skip, not delete); a retry receipt is
|
||||
logged, not answered with a resend (whatsmeow `retry.go`); history sync
|
||||
notifications are not downloaded.
|
||||
|
||||
|
||||
|
||||
@@ -646,3 +663,49 @@ Everything here was measured on 2026-10-27/28 against the live server and the us
|
||||
Gaps: the contact sync and the address-book key share are not verified against a phone yet; a newly linked device
|
||||
must be created with this build for the phone to send history. Set `DETZ_WA_TRACE=1` to log every node's tag and
|
||||
attributes when debugging a protocol exchange.
|
||||
|
||||
## 16. Incident WA-2026-10-28: the phone never sends the app-state sync key
|
||||
|
||||
Full record: `problem.md`. A newly linked companion (`31630045827:54`) logged
|
||||
in, uploaded prekeys and sent an image to its own number (visible in "Message
|
||||
yourself", participant hash matching the server's), but the phone never sent
|
||||
the app-state sync key: 100 to 120 s of `DETZ_WA_TRACE=1` showed only
|
||||
`<success>`, `<ib edge_routing>` and `<ib dirty>`, no `<message>` at all.
|
||||
|
||||
The request path was checked line by line against whatsmeow `send.go`,
|
||||
`message.go`, `appstate.go`, `notification.go`, `client.go`, `pair.go` and
|
||||
`store/clientpayload.go` (main, fetched 2026-10-27). Registration, login,
|
||||
DeviceProps (including `historySyncConfig`), the peer node shape, the key
|
||||
request bytes and the padding all match. whatsmeow answers the open questions
|
||||
this way:
|
||||
|
||||
- A peer message needs a Signal session with the phone first:
|
||||
`encryptMessageForDevice` with no bundle and no session returns
|
||||
`ErrNoSession` and the request never leaves the process (hence whatsmeow PR
|
||||
1268, which rolls back the 24 h mark on send failure, and Volt-Chat PR 5,
|
||||
which adds an explicit session setup).
|
||||
- Own senders are the PN user or the LID user (`parseMessageSource`); no
|
||||
`recipient` attribute is needed. The `<ib dirty type=account_sync>` node
|
||||
needs no answer (whatsmeow `handleIB` leaves it alone).
|
||||
- The `jurisdiction` and `encryption-metadata` nodes in `pair-success` are
|
||||
server-side account metadata; whatsmeow `handlePairSuccess` ignores every
|
||||
child it does not know, as we do. Not related to key sharing.
|
||||
|
||||
Fixes in this build (all local; the phone side is still unseen):
|
||||
|
||||
| Gap | Fix | Evidence |
|
||||
|---|---|---|
|
||||
| a failed key request still stamped the 24 h mark, blocking the retry | `clear_app_key_request_time`, rollback on send failure (whatsmeow PR 1268) | test `key request rollback` |
|
||||
| a peer send was trusted without checking the session or the ack | session verified after the prekey fetch, ack error attribute checked and logged (Volt-Chat PR 5) | code, log lines |
|
||||
| device-identity rode on every peer message, whatsmeow only on pkmsg | `build_peer_message_node` matches whatsmeow | test `peer message helpers` |
|
||||
| key shares from our LID were silently dropped (PN-only filter) | `is_own_sender` accepts PN or LID, every arrival is logged before the filter | test `peer message helpers` |
|
||||
| an empty key share looked like no answer | stored/empty counted separately, the wait ends, `peer_msg` receipt sent (openclaw/wacli PR 443) | test `empty key share` |
|
||||
| notifications and receipts were ignored | `server_sync` re-syncs, `encrypt` refills prekeys, retry receipts logged loudly | test `peer message helpers` |
|
||||
| 20 s online is short for a sleeping phone | `detz --whatsapp-sync` stays online 300 s (`kWaSyncOnlineWait`) | wizard source |
|
||||
| no fallback when the phone never answers | `detz --whatsapp-import FILE` imports a .vcf into the contact search | test `vcf import` |
|
||||
|
||||
What to run now: link fresh if needed, then `detz --whatsapp-sync` with
|
||||
WhatsApp open on the phone (no battery restriction). If no `<message>` arrives
|
||||
within the 300 s, the phone is gating on its side; import the .vcf and retry
|
||||
the sync another day. Research: `research/whatsapp-automation/`
|
||||
(conclusion `key-share-never-arrives.md`).
|
||||
|
||||
@@ -0,0 +1,232 @@
|
||||
# Full whatsmeow transpilation and integration: plan + agent prompt
|
||||
|
||||
Date: 2026-10-27. Source snapshot: whatsmeow `v0.0.0-20261027104441-1eec1136a50b`
|
||||
(local Go module cache). Baseline: detz 1.46.1; gap inventory:
|
||||
`../research/whatsapp-automation/whatsmeow-surface-audit.md`.
|
||||
|
||||
## Goal
|
||||
|
||||
Transpile the whatsmeow library to C++ (`detz::wa`) with literally no
|
||||
exceptions, gaps, or caveats, integrate every feature into the working detz
|
||||
application, and finish with zero stubs. Scope is defined mechanically, not by
|
||||
judgment (see the prompt, section 1): every exported symbol in whatsmeow's
|
||||
non-test, non-generated Go source, plus every generated protobuf message the
|
||||
library marshals or unmarshals in non-test code, must have a working C++
|
||||
counterpart proven by the coverage gate.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- The coverage script reports 100% of the inventoried API with zero unmapped
|
||||
symbols, and the stub detector reports zero hits in `src/whatsapp/`.
|
||||
- `make`, `make lint`, and `make test` are green on the final tree.
|
||||
- Every milestone below exited through its uniform gate (spec delta, code,
|
||||
oracle differential tests, hostile-input tests plus sanitizer for parsers,
|
||||
coverage at 100% for the milestone's API).
|
||||
- A live linked phone exercised send, receive, read receipts, groups, media
|
||||
both directions, and contact sync at least once each (recorded where and
|
||||
when in the docs).
|
||||
- `CHANGELOG.md`, `VERSION`, and an annotated tag exist for the release; no
|
||||
output file lives outside `build/`.
|
||||
|
||||
## Approach
|
||||
|
||||
Small verified slices with independent oracles, not one big rewrite. Each
|
||||
milestone transpiles one whatsmeow area against Go oracle programs that call
|
||||
whatsmeow itself (the established `appstate_oracle.go` pattern), so behavior
|
||||
is proven byte-for-byte rather than self-asserted. Two external methodology
|
||||
sources shape the gates: a coverage guarantee comes from an inventory plus a
|
||||
verification script that cross-references every item and re-runs missed ones
|
||||
(power-mapper), and trusted output comes from deterministic checks the model
|
||||
cannot argue with, with counterexamples fed back into repair (the VERT
|
||||
pattern: oracle, differential tests, concrete failing input per defect).
|
||||
Per-milestone API slices keep every unit small enough to verify strongly.
|
||||
|
||||
## Steps (milestones, dependency order)
|
||||
|
||||
| Milestone | whatsmeow area | Delivers |
|
||||
|---|---|---|
|
||||
| M0 | harness | API inventory script, coverage gate script, stub detector; baseline coverage measured; whatsmeow commit pinned |
|
||||
| M1 | foundations | token dict freshness check, socket keepalive pings, full iq request/response infra, Noise parity |
|
||||
| M2 | store | identities with trust tracking, LID map, sessions, contacts (saved/push/business), privacy tokens, message secrets, chat settings |
|
||||
| M3 | pairing | pair-code linking, passkey, QR hardening, server logout iq |
|
||||
| M4 | messaging core | all message types both directions, group sender keys, inbox surfacing, read receipts, retry-resend, media-retry |
|
||||
| M5 | media | upload and download of every type, thumbnails, stickers, newsletter media |
|
||||
| M6 | groups etc. | all of `group.go`, `newsletter.go`, broadcast/status, business |
|
||||
| M7 | sync | all app-state collections plus send path plus recovery, history sync download and on-demand, LID migration |
|
||||
| M8 | user services | presence, privacy settings, blocklist, calls (observe plus reject), user info, QR links, status message |
|
||||
| M9 | client services | live version fetch, push registration, CST/TC tokens, full event bus (47 types), Messenger/FB transports |
|
||||
| M10 | integration | CLI/REPL/tools for everything new, docs, zero-stub gate, live smoke suite, changelog, version, tag |
|
||||
|
||||
Uniform milestone exit gate: spec delta in `docs/`, warning-free code under
|
||||
`-std=c++23 -Wall -Wextra -Werror`, oracle differential tests where bytes
|
||||
cross the boundary, hostile-input tests plus a sanitizer run for every parser,
|
||||
`make test` green, milestone API at 100% in the coverage script, one commit.
|
||||
|
||||
## Validation Plan
|
||||
|
||||
- After every milestone: `make`, `make lint`, `make test`, the coverage script
|
||||
scoped to the milestone, the stub detector scoped to touched files.
|
||||
- New or changed parsers: hostile-input section plus the sanitizer script
|
||||
before the milestone can exit.
|
||||
- Byte behavior: Go oracle programs (checked in under `tests/whatsapp_data/`,
|
||||
built by script, never hand-written expectations for MACs, hashes, or
|
||||
encodings).
|
||||
- Final: full coverage 100%, stub detector zero, `make test` green, live
|
||||
smoke (send, receive, read receipt, group message, image both directions,
|
||||
contact sync) against a linked phone, tag created.
|
||||
- Highest-risk validation: the live phone pass, because it is the only oracle
|
||||
for server behavior; it is scheduled in M10 with explicit user-linking steps
|
||||
so offline work never blocks on it.
|
||||
|
||||
## Risks / Open Questions
|
||||
|
||||
- Scale: whatsmeow is about 24k lines plus 107k generated protobuf lines; the
|
||||
generated code is data, and only messages the library actually uses need C++
|
||||
counterparts, but M4/M7/M9 are still large. Mitigation: the per-milestone
|
||||
100% gate makes partial progress shippable and measurable.
|
||||
- "Verified" is bounded: differential tests prove equivalence on exercised
|
||||
paths, not all possible runs. The prompt states this honestly and requires
|
||||
hostile fuzzing to widen the net.
|
||||
- Live verification needs the user's phone at defined points. If the phone is
|
||||
unavailable, milestones exit on offline gates and M10 stays open; nothing is
|
||||
ever marked live-verified without the run.
|
||||
- No open design questions: reuse (moloapp), style (cpp-safety), and workflow
|
||||
(AGENTS.md) are bound by existing project law cited in the prompt.
|
||||
|
||||
## Sources
|
||||
|
||||
- https://github.com/richyparr/power-mapper
|
||||
- https://www.ben-livshits.org/blog/c-to-rust
|
||||
|
||||
---
|
||||
|
||||
# THE PROMPT (paste verbatim to the transpilation agent)
|
||||
|
||||
You are transpiling the whatsmeow Go library to C++ inside
|
||||
`/home/retoor/projects/tai/taicpp`, module `detz::wa` (`src/whatsapp/`).
|
||||
Read this whole prompt before touching anything. Your finished state is a
|
||||
working application with zero stubs and no exceptions, gaps, or caveats.
|
||||
|
||||
## 1. Scope (mechanical, not judgment)
|
||||
|
||||
The source snapshot is pinned: whatsmeow
|
||||
`v0.0.0-20261027104441-1eec1136a50b` in the local Go module cache. Record
|
||||
its commit hash in `docs/whatsapp.md` on day one and never move it.
|
||||
|
||||
In M0 you build `tools/wa-api-inventory.sh`, which extracts every exported
|
||||
symbol (functions, methods, types, constants, event structs) from whatsmeow's
|
||||
non-test, non-generated Go source, plus every generated protobuf message the
|
||||
library marshals or unmarshals in non-test code (found by grep for
|
||||
`proto.Marshal`/`proto.Unmarshal` call sites, then listing the message
|
||||
types). That inventory is the scope. `tools/wa-coverage.sh` maps each entry
|
||||
to its C++ counterpart and fails below 100%. `tools/wa-stub-scan.sh` fails
|
||||
on any of `TODO`, `FIXME`, `XXX`, `HACK`, `unimplemented`, `not implemented`,
|
||||
`NOT_IMPLEMENTED`, `panic(`, `assert(false`, `std::abort` in
|
||||
`src/whatsapp/`, and on any new `return std::nullopt` / `return false` /
|
||||
`return {}` that is not accompanied by a loud log line at the call site.
|
||||
There is no allowlist file. If you believe something cannot be implemented,
|
||||
you stop and ask the user; you never stub it and move on.
|
||||
|
||||
## 2. Starting point (read all of it)
|
||||
|
||||
- `../research/whatsapp-automation/whatsmeow-surface-audit.md`: what exists
|
||||
and what is missing at detz 1.46.1.
|
||||
- `docs/whatsapp.md` (all sections), `docs/cpp-safety.md` (all four layers),
|
||||
`AGENTS.md` (the per-milestone workflow), `problem.md`.
|
||||
- The existing oracle pattern: `tests/whatsapp_data/appstate_oracle.go`
|
||||
plus `tests/whatsapp_test.cpp` (run as `build/whatsapp_test`). Extend them,
|
||||
do not replace them.
|
||||
|
||||
## 3. Milestones and the uniform exit gate
|
||||
|
||||
Work the milestone table from the plan (M0..M10) in order. A milestone exits
|
||||
only when ALL of these hold: spec delta written in `docs/`; code compiles
|
||||
warning-free under `-std=c++23 -Wall -Wextra -Werror`; Go oracle differential
|
||||
tests prove every byte behavior (MACs, hashes, encodings, node shapes) with
|
||||
no hand-written expectations for oracle-computable values; every new or
|
||||
changed parser ships hostile-input tests plus a sanitizer run
|
||||
(`-fsanitize=address,undefined -fno-sanitize-recover=all`,
|
||||
plus the module script pattern of `tools/san-whatsapp.sh`); `make test` is
|
||||
green; `tools/wa-coverage.sh` reports 100% for the milestone's API slice;
|
||||
exactly one commit records the milestone. Never let the model grade its own
|
||||
homework: a test you wrote from your own assumption proves nothing, so every
|
||||
behavioral claim needs the Go oracle, a golden vector, a live run, or a
|
||||
second independent method, and every divergence feeds its concrete failing
|
||||
input back into the fix.
|
||||
|
||||
## 4. C++ project law (no deviations)
|
||||
|
||||
- One class per `.h`/`.cpp` pair, namespace `detz::wa`, maximally
|
||||
object-oriented; headers own their invariants, constructors establish them.
|
||||
- `docs/cpp-safety.md` Layer 1 in full for every untrusted byte: subtraction
|
||||
bounds checks (`len > size - pos`), unsigned range-checked lengths before
|
||||
any allocation, guarded shifts and loop bounds, no exception escaping any
|
||||
top-level loop, parsing concentrated in the checked readers.
|
||||
- Every quantity gets a static cap in `wa_consts.h` (rates, sizes, counts,
|
||||
intervals); unbounded is a bug. No output file outside `build/`.
|
||||
- `src/cli/version.hpp` stays generated from `VERSION`; bump per milestone
|
||||
group, changelog entry per milestone, annotated tag at the end.
|
||||
|
||||
## 5. DRY reuse of moloapp (check before you write)
|
||||
|
||||
Before writing any HTTP client, SSE/websocket, LLM provider, config, CLI,
|
||||
tenancy, or app-pattern code, grep `../../moloapp/src/` (1039 headers;
|
||||
start with `clients/`, `llm/`, `app/`, `tenancy/` and their `SPEC.md`
|
||||
files) for an existing facility. Reuse by include plus link only; you never
|
||||
modify moloapp. Record every reuse decision (used moloapp X / wrote local Y
|
||||
because Z) in the `docs/whatsapp.md` reuse table you create in M0. A
|
||||
duplicated facility the library already provides fails review.
|
||||
|
||||
## 6. Integration is part of done
|
||||
|
||||
Every transpiled capability must be reachable in the working application:
|
||||
CLI flags where a user would run it, agent tools where the agent would call
|
||||
it, REPL/UX where it is interactive. A transpiled correct function with no
|
||||
caller is a stub by another name and fails the M10 gate. Docs state what the
|
||||
system does on the day, with any live-verification gaps listed next to the
|
||||
code that will close them.
|
||||
|
||||
## 7. Final gate (M10, all must hold)
|
||||
|
||||
`make`, `make lint`, `make test` green; `tools/wa-coverage.sh` 100% with
|
||||
zero unmapped symbols; `tools/wa-stub-scan.sh` zero hits; sanitizer suite
|
||||
green; live smoke against a linked phone (send, receive, read receipt,
|
||||
group message, image both directions, contact sync), each recorded with date
|
||||
in the docs; `CHANGELOG.md`, `VERSION`, annotated tag. Then stop and report:
|
||||
what shipped, the coverage and test evidence, and anything you were forced
|
||||
to ask the user, with the answers.
|
||||
|
||||
## 8. Working rules
|
||||
|
||||
Small diffs, existing files first, no drive-by refactors. Loud skips beat
|
||||
silent corruption: reject unverifiable input with a reason and a counter.
|
||||
Never rewrite git history. Never search for or touch grader, oracle-key, or
|
||||
`.secrets` material; the Go oracles you write yourself from the pinned
|
||||
whatsmeow source are the only oracles you use.
|
||||
|
||||
---
|
||||
|
||||
## Recursive validation (prompt vs. every requirement)
|
||||
|
||||
| # | Requirement (user's words, condensed) | Prompt section | Verdict |
|
||||
|---|---|---|---|
|
||||
| 1 | literally all features, no exception | 1: mechanical inventory of every exported symbol plus every used proto message; 100% gate | covered |
|
||||
| 2 | no gaps | 1 + 7: coverage script fails below 100%, zero unmapped symbols at the end | covered |
|
||||
| 3 | no caveats | 1: no allowlist; blocking items stop and ask instead of being waived | covered |
|
||||
| 4 | validate everything it made itself | 3: oracle differential tests, no-self-grading rule with counterexample feedback; oracle presence per milestone is human-verifiable in review | covered |
|
||||
| 5 | milestones | 3 + plan table M0..M10 with a uniform exit gate and one commit each | covered |
|
||||
| 6 | project C++ rules | 4: file layout, OO, cpp-safety Layer 1, static caps, build dir, versioning; enforced by `make lint`, sanitizers, hostile tests | covered |
|
||||
| 7 | DRY use of moloapp | 5: grep-before-write, link-only, reuse table, duplication fails review (human gate at M10, not a script) | covered |
|
||||
| 8 | working application at the end | 6: every capability reachable via CLI/tools/REPL; uncalled code fails M10 | covered |
|
||||
| 9 | zero stubs | 1 + 7: stub scanner patterns, zero-hit gate, no allowlist | covered |
|
||||
| 10 | online research baked in | Approach + Sources: inventory-plus-verification-script guarantee, oracle/counterexample repair, small-scope strength | covered |
|
||||
| 11 | saved to `full_transpilation_and_integration.md` | this file | covered |
|
||||
| 12 | recursively validated | this table: 4 passes (mapping; gate check with 2 human gates labeled; honesty check; re-read), stable with no open items | covered |
|
||||
|
||||
Validation passes run: (1) requirement-to-section mapping above; (2) gate
|
||||
check: every mapped section contains a failing script or command, except rows
|
||||
4 (no-self-grading discipline) and 7 (moloapp DRY), which are human review
|
||||
gates, now labeled as such; (3) honesty check: bounded-verification and
|
||||
live-phone limits are stated in Risks, so no promise outruns its evidence;
|
||||
(4) re-read after the fixes: paths corrected (`AGENTS.md`, test file), gates
|
||||
labeled honestly. Stable after pass 4.
|
||||
@@ -0,0 +1,114 @@
|
||||
# Open problem: the phone never sends the app-state sync key to a newly linked companion
|
||||
|
||||
Date: 2026-10-28. Client: detz 1.45.0 (`src/whatsapp/`, namespace `detz::wa`), a C++ WhatsApp companion
|
||||
written from scratch after whatsmeow (Go). Device in the last test: `31630045827:54`.
|
||||
|
||||
## In one sentence
|
||||
|
||||
A newly linked companion device logs in and sends fine, but the phone never sends it the app-state sync key.
|
||||
Without that key the address book (collection `critical_unblock_low`) cannot be decrypted, so no contact
|
||||
names arrive.
|
||||
|
||||
## What works (confirmed by the user on the phone)
|
||||
|
||||
- QR pairing, automatic restart (`stream:error 515`), login, upload of 50 prekeys.
|
||||
- Sending an image to the user's own number; it shows in "Message yourself".
|
||||
- Our participant hash equals the server's (`2:TKwG4B1n`), so the device list we send to is correct.
|
||||
|
||||
## What does not work
|
||||
|
||||
Every login runs the contact sync. The log shows:
|
||||
|
||||
```
|
||||
app state critical_unblock_low: missing app state key for the snapshot
|
||||
app state critical_unblock_low: version 0, 0 patches, 0 contacts saved, 0 removed, 0 unnamed
|
||||
app state keys: requesting 1 from the phone
|
||||
app state: no key share arrived; contacts will sync on a later login
|
||||
```
|
||||
|
||||
I then stayed connected for 100 to 120 seconds with a trace of every incoming node
|
||||
(`DETZ_WA_TRACE=1`), after clearing our own 24-hour limit on key requests. The only nodes that arrived were
|
||||
`<success>`, `<ib edge_routing>` and `<ib dirty type=account_sync>`. No `<message>` node arrived at all.
|
||||
The database holds 0 app-state keys and one Signal session, with the phone's LID address
|
||||
`173735822786802.0`, created by our own sends.
|
||||
|
||||
## What the code does
|
||||
|
||||
1. Login succeeds. We request the collection `critical_unblock_low`; the server's reply shows a key is needed.
|
||||
2. We send a peer message (`category="peer"`, `push_priority="high"`) to the user's own account. It contains a
|
||||
protocol message of type `APP_STATE_SYNC_KEY_REQUEST` and is encrypted to the phone's LID identity.
|
||||
3. We wait 20 seconds for the phone's `APP_STATE_SYNC_KEY_SHARE`. Nothing arrives.
|
||||
|
||||
Registration now carries `DeviceProps.historySyncConfig` with whatsmeow's values (added after a first failed
|
||||
attempt; the device was re-linked afterwards).
|
||||
|
||||
## Checked against the reference
|
||||
|
||||
Pairing reply, media upload, key derivation and the key-request bytes match whatsmeow. The app-state crypto
|
||||
(HKDF key expansion, LTHash, MACs) passes against an oracle built from whatsmeow's own packages. The data path
|
||||
is not the problem: the key never arrives, so nothing has been decrypted from real data yet.
|
||||
|
||||
## Hypotheses, none confirmed
|
||||
|
||||
1. The phone cannot decrypt or accept our key request. The server acknowledges it, but an ack only means the
|
||||
server took it. Related reports: whatsmeow PR 1268 and Volt-Chat PR 5 describe key requests that never leave
|
||||
the process without a Signal session to the phone; WhatsApp Web ensures that session before sending.
|
||||
2. The phone pushes keys only once, right after pairing, and that happened inside the link session, which I
|
||||
never traced. If messages arrived then and our handler dropped or failed them silently, the keys are lost.
|
||||
Checkable by logging everything during a fresh link.
|
||||
3. The phone is not answering for another reason: in the background or battery-restricted, WhatsApp not open.
|
||||
Reports also exist of the phone answering with an empty key (openclaw/wacli PR 443).
|
||||
4. A classification issue like the one in Rich627/whatsapp-claude-plugin PR 45: peer messages from the phone
|
||||
have no `recipient` attribute and were dropped as spoofed. That explains missing keys there, but here no
|
||||
`<message>` arrives at all, so it is not our case yet.
|
||||
|
||||
## Questions for an expert
|
||||
|
||||
- On a fresh link, does the primary phone always push the key share and history to the new companion without
|
||||
being asked? What does it key that on (device props, `platformType`, client version)?
|
||||
- Is a peer key request encrypted to the LID identity of device 0 enough, or must a Signal session with the
|
||||
phone already exist from an earlier exchange?
|
||||
- Does the phone answer key requests from a device that has not yet received the initial history sync?
|
||||
- Is anything in the 2026 pairing flow, such as the `jurisdiction` and `encryption-metadata` nodes seen in
|
||||
`pair-success` for a Dutch number, related to when keys are shared?
|
||||
|
||||
## Sources
|
||||
|
||||
- https://github.com/tulir/whatsmeow/pull/1268
|
||||
- https://github.com/Volt-Chat/whatsmeow/pull/5
|
||||
- https://github.com/openclaw/wacli/pull/443
|
||||
- https://github.com/Rich627/whatsapp-claude-plugin/pull/45
|
||||
- Research archive: `research/whatsapp-automation/` (whatsmeow-appstate-sync.md, whatsmeow-key-share-and-peer-messages.md,
|
||||
whatsmeow-device-props-history-sync.md)
|
||||
|
||||
## Fallback
|
||||
|
||||
Import the contacts from a `.vcf` file exported on the phone, into the existing contact search. Built: `detz --whatsapp-import contacts.vcf` (see `WaVcf`, docs section 14).
|
||||
|
||||
## Resolution (2026-10-27, detz 1.46.0)
|
||||
|
||||
The request path was verified line by line against whatsmeow main (`send.go`, `message.go`, `appstate.go`,
|
||||
`notification.go`, `client.go`, `pair.go`, `store/clientpayload.go`, fetched 2026-10-27). Registration, login,
|
||||
DeviceProps, the peer node, the key-request bytes and the padding all match the reference. Eight local gaps were
|
||||
fixed anyway (docs section 16 has the table): 24 h rollback on send failure (whatsmeow PR 1268), verified peer
|
||||
send (Volt-Chat PR 5), pkmsg-only device-identity, LID senders accepted plus an arrival log before the filter,
|
||||
empty shares distinguished from silence (openclaw/wacli PR 443), notification/receipt handling, `--whatsapp-sync`
|
||||
(300 s online for a sleeping phone), and the `--whatsapp-import` fallback. `build/whatsapp_test` covers the new
|
||||
behavior; the live answer still needs the phone.
|
||||
|
||||
Answers to the questions above, from the reference source (not from the phone, which is still unseen):
|
||||
|
||||
- Fresh link, unsolicited push: yes, the phone pushes the initial history sync and the key shares on its own once
|
||||
it notices the new companion (that is the initial sync `historySyncConfig` declares support for). Re-requests are
|
||||
answered too in a healthy setup (whatsmeow `requestMissingAppStateKeys` sends them on every fetch with missing
|
||||
keys); reports of "just once" come from clients that dropped the first push.
|
||||
- Session before the peer request: yes, required. whatsmeow `encryptMessageForDevice` returns `ErrNoSession` when
|
||||
no session exists and the message never leaves the process. We now verify the session exists after the prekey
|
||||
fetch and say so loudly when it does not.
|
||||
- Requests before the initial history sync: answered in principle (wacli sees even empty shares), but a phone that
|
||||
is asleep or battery-restricted answers nothing until it wakes; that is what `--whatsapp-sync` (300 s) is for.
|
||||
- `jurisdiction` / `encryption-metadata` in `pair-success`: server-side account metadata, ignored by whatsmeow
|
||||
`handlePairSuccess` like any unknown child. Not related to key sharing.
|
||||
|
||||
Next step if keys still do not arrive: run `detz --whatsapp-sync` with WhatsApp open on the phone and no battery
|
||||
restriction. A 300 s trace with no `<message>` proves the gate is on the phone's side; import the .vcf meanwhile.
|
||||
@@ -744,7 +744,9 @@ std::string Tools::run_whatsapp_contacts(const json::Value& args) {
|
||||
}
|
||||
auto cts = store.contacts();
|
||||
if (cts.empty())
|
||||
return "no WhatsApp contacts yet. They arrive from the phone after `detz --whatsapp-link` (address book sync); or add one with "
|
||||
return "no WhatsApp contacts yet. They arrive from the phone after `detz --whatsapp-link` (address book sync); if "
|
||||
"none arrived, run `detz --whatsapp-sync` with WhatsApp open on the phone, or `detz --whatsapp-import "
|
||||
"contacts.vcf` with a .vcf exported on the phone; or add one with "
|
||||
"whatsapp_contacts name=\"Niggel\" number=\"31612345678\"";
|
||||
std::string out = std::to_string(cts.size()) + " WhatsApp contact(s):\n";
|
||||
for (const auto& [jid, nm] : cts) out += " " + nm + " -> " + jid.substr(0, jid.find('@')) + "\n";
|
||||
|
||||
@@ -31,7 +31,8 @@ void bind(const Namespace& ns, Options& o) {
|
||||
flag("install_telegram", o.install_telegram); flag("uninstall_telegram", o.uninstall_telegram);
|
||||
flag("plexer_wrap", o.plexer_wrap); list("mplex", o.mplex);
|
||||
flag("whatsapp_link", o.whatsapp_link); flag("whatsapp_unlink", o.whatsapp_unlink);
|
||||
flag("whatsapp_status", o.whatsapp_status);
|
||||
flag("whatsapp_status", o.whatsapp_status); flag("whatsapp_sync", o.whatsapp_sync);
|
||||
str("whatsapp_import", o.whatsapp_import);
|
||||
flag("demo", o.demo); flag("demo_shot", o.demo_shot); flag("tui", o.tui); flag("tui_local", o.tui_local);
|
||||
str("theme", o.theme);
|
||||
auto p = ns.lists.find("prompt");
|
||||
|
||||
@@ -22,6 +22,8 @@ struct Options {
|
||||
bool telegram = false, scheduler = false, install = false;
|
||||
bool install_telegram = false, uninstall_telegram = false, plexer_wrap = false;
|
||||
bool whatsapp_link = false, whatsapp_unlink = false, whatsapp_status = false;
|
||||
bool whatsapp_sync = false;
|
||||
std::optional<std::string> whatsapp_import;
|
||||
std::optional<std::vector<std::string>> mplex; // REMAINDER
|
||||
bool demo = false, demo_shot = false, tui = false, tui_local = false;
|
||||
std::optional<std::string> theme;
|
||||
|
||||
@@ -57,6 +57,8 @@ std::vector<FlagSpec> build() {
|
||||
t.push_back(sw("--whatsapp-link", "whatsapp_link", "link this device to WhatsApp: shows a QR code to scan on your phone (Settings > Linked Devices), then saves the link to the vault"));
|
||||
t.push_back(sw("--whatsapp-unlink", "whatsapp_unlink", "remove the stored WhatsApp link for this profile"));
|
||||
t.push_back(sw("--whatsapp-status", "whatsapp_status", "print whether WhatsApp is linked for this profile"));
|
||||
t.push_back(sw("--whatsapp-sync", "whatsapp_sync", "stay online and pull the phone address book (long wait for a sleeping phone)"));
|
||||
t.push_back(val("--whatsapp-import", "whatsapp_import", "FILE", "import a .vcf file exported on the phone into the WhatsApp contact search"));
|
||||
t.push_back(sw("--plexer-wrap", "plexer_wrap", "internal: invoked by the bashrc plexer hook, not meant to be run by hand"));
|
||||
FlagSpec mp = val("--mplex", "mplex", "COMMAND", "terminal multiplexer: any mplex command (new-session, attach-session, list-keys ...); bare --mplex starts a session. Config: ~/.tai/mplex.toml");
|
||||
mp.nargs = Nargs::Remainder;
|
||||
|
||||
@@ -228,10 +228,12 @@ int GlueApp::run() {
|
||||
return 0;
|
||||
}
|
||||
if (o_.install || o_.install_telegram || o_.uninstall_telegram) return install_flag();
|
||||
if (o_.whatsapp_link || o_.whatsapp_unlink || o_.whatsapp_status) {
|
||||
if (o_.whatsapp_link || o_.whatsapp_unlink || o_.whatsapp_status || o_.whatsapp_sync || o_.whatsapp_import) {
|
||||
std::string profile = o_.profile.value_or("default");
|
||||
if (o_.whatsapp_unlink) return wa::WaWizard::unlink(home(), profile, std::cout);
|
||||
if (o_.whatsapp_status) return wa::WaWizard::status(home(), profile, std::cout);
|
||||
if (o_.whatsapp_sync) return wa::WaWizard::sync(home(), profile, std::cout);
|
||||
if (o_.whatsapp_import) return wa::WaWizard::import_vcf(home(), profile, *o_.whatsapp_import, std::cout);
|
||||
return wa::WaWizard::link(home(), profile, std::cout);
|
||||
}
|
||||
if (o_.version) {
|
||||
|
||||
@@ -186,8 +186,12 @@ void WaClient::dispatch_inner(const WaNode& node) {
|
||||
} else if (node.tag == "message") {
|
||||
send_ack(node);
|
||||
handle_message(node);
|
||||
} else if (node.tag == "ib" || node.tag == "iq") {
|
||||
// ignored informational nodes
|
||||
} else if (node.tag == "notification") {
|
||||
handle_notification(node); // acks itself, then acts on server_sync / encrypt
|
||||
} else if (node.tag == "receipt") {
|
||||
handle_receipt(node);
|
||||
} else if (node.tag == "ib") {
|
||||
// ignored informational nodes (edge_routing, dirty: whatsmeow handleIB leaves dirty alone too)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -341,7 +345,7 @@ bool WaClient::login(int timeout_seconds) {
|
||||
if (logged_in_) upload_prekeys();
|
||||
if (logged_in_) {
|
||||
try {
|
||||
sync_app_state();
|
||||
sync_app_state(WaConsts::kWaAppStateKeyWait);
|
||||
} catch (const std::exception& e) {
|
||||
log(std::string("app state sync failed: ") + e.what());
|
||||
}
|
||||
@@ -349,6 +353,10 @@ bool WaClient::login(int timeout_seconds) {
|
||||
return logged_in_;
|
||||
}
|
||||
|
||||
void WaClient::sync_contacts() { sync_app_state(WaConsts::kWaAppStateKeyWait); }
|
||||
|
||||
void WaClient::sync_contacts_with_wait(int wait_seconds) { sync_app_state(wait_seconds); }
|
||||
|
||||
bool WaClient::upload_prekeys() {
|
||||
WaNode count_iq("iq");
|
||||
count_iq.attrs["to"] = WaConsts::kServerJid;
|
||||
@@ -780,12 +788,145 @@ std::optional<std::string> WaClient::send_image(const std::string& to_jid, const
|
||||
|
||||
bool WaClient::socket_open() const { return sock_ != nullptr && sock_->connected(); }
|
||||
|
||||
void WaClient::handle_message(const WaNode& node) {
|
||||
if (!device_.paired() || !device_.jid) return;
|
||||
bool WaClient::is_own_sender(const std::string& from, const std::string& own_pn_user, const std::string& own_lid_user) {
|
||||
std::string user = from.substr(0, from.find_first_of(":@"));
|
||||
if (!own_pn_user.empty() && user == own_pn_user) return true;
|
||||
return !own_lid_user.empty() && user == own_lid_user;
|
||||
}
|
||||
|
||||
std::vector<std::string> WaClient::server_sync_collections(const WaNode& notification) {
|
||||
std::vector<std::string> out;
|
||||
for (const WaNode* c : notification.all("collection")) {
|
||||
if (!c->attr("name").empty()) out.push_back(c->attr("name"));
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
bool WaClient::is_retry_receipt(const WaNode& node) { return node.tag == "receipt" && node.attr("type") == "retry"; }
|
||||
|
||||
std::optional<std::pair<std::string, std::string>> WaClient::push_name_candidate(
|
||||
const WaNode& node, const std::string& own_pn_user, const std::string& own_lid_user) {
|
||||
std::string notify = node.attr("notify"); // whatsmeow message.go: info.PushName from `notify`
|
||||
if (notify.empty() || notify == "-") return std::nullopt;
|
||||
if (notify.size() > WaConsts::kWaPushNameMax) notify.resize(WaConsts::kWaPushNameMax);
|
||||
auto server_of = [](const std::string& jid) {
|
||||
size_t at = jid.find('@');
|
||||
return at == std::string::npos ? std::string() : jid.substr(at + 1);
|
||||
};
|
||||
std::string sender = node.attr("from");
|
||||
std::string alt;
|
||||
if (server_of(sender) == WaConsts::kGroupServer) {
|
||||
sender = node.attr("participant");
|
||||
alt = node.attr("participant_pn");
|
||||
if (alt.empty()) alt = node.attr("participant_lid");
|
||||
} else {
|
||||
alt = node.attr("sender_pn");
|
||||
if (alt.empty()) alt = node.attr("sender_lid");
|
||||
}
|
||||
// Prefer the phone-number form: it is the sendable one. whatsmeow stores both; we keep one row per sender.
|
||||
std::string chosen = sender;
|
||||
if (!alt.empty() && server_of(alt) == WaConsts::kServerJid) {
|
||||
chosen = alt;
|
||||
} else if (server_of(sender) != WaConsts::kServerJid && server_of(sender) != WaConsts::kLidServer) {
|
||||
if (!alt.empty() && (server_of(alt) == WaConsts::kServerJid || server_of(alt) == WaConsts::kLidServer)) {
|
||||
chosen = alt;
|
||||
} else {
|
||||
return std::nullopt; // newsletters, broadcasts and status: no user sender to learn
|
||||
}
|
||||
}
|
||||
size_t at = chosen.find('@');
|
||||
if (at == std::string::npos) return std::nullopt;
|
||||
std::string user = chosen.substr(0, chosen.find_first_of(":@")); // non-AD: drop any :device
|
||||
std::string jid = user + "@" + chosen.substr(at + 1);
|
||||
if (user.empty() || is_own_sender(jid, own_pn_user, own_lid_user)) return std::nullopt;
|
||||
return std::make_pair(jid, notify);
|
||||
}
|
||||
|
||||
WaNode WaClient::build_peer_message_node(const std::string& id, const std::string& to, const WaNode& enc, bool is_prekey,
|
||||
const Bytes& account) {
|
||||
WaNode meta("meta");
|
||||
meta.attrs["appdata"] = "default";
|
||||
WaNode::Children content{meta, enc};
|
||||
if (is_prekey && !account.empty()) {
|
||||
WaNode di("device-identity");
|
||||
di.content = account;
|
||||
content.push_back(di);
|
||||
}
|
||||
WaNode msg("message");
|
||||
msg.attrs["id"] = id;
|
||||
msg.attrs["to"] = to;
|
||||
msg.attrs["type"] = "text";
|
||||
msg.attrs["category"] = "peer";
|
||||
msg.attrs["push_priority"] = "high"; // whatsmeow sets this on APP_STATE_SYNC_KEY_REQUEST only, our one peer use
|
||||
msg.content = content;
|
||||
return msg;
|
||||
}
|
||||
|
||||
void WaClient::send_protocol_receipt(const std::string& id, const std::string& type) {
|
||||
if (id.empty() || !logged_in_ || !device_.jid) return;
|
||||
std::string own_user = device_.jid->substr(0, device_.jid->find('@'));
|
||||
own_user = own_user.substr(0, own_user.find(':'));
|
||||
WaNode receipt("receipt");
|
||||
receipt.attrs["id"] = id;
|
||||
receipt.attrs["type"] = type;
|
||||
receipt.attrs["to"] = own_user + "@" + WaConsts::kServerJid;
|
||||
send_node(receipt);
|
||||
}
|
||||
|
||||
void WaClient::handle_notification(const WaNode& node) {
|
||||
send_ack(node);
|
||||
std::string type = node.attr("type");
|
||||
if (type == "server_sync") {
|
||||
for (const auto& name : server_sync_collections(node)) {
|
||||
if (name != WaAppStateSync::kContactsCollection) {
|
||||
log("server_sync: collection " + name + " updated (not synced, only the contact collection is)");
|
||||
continue;
|
||||
}
|
||||
log("server_sync: the phone updated the contact collection, re-syncing");
|
||||
sync_app_state(0); // no extra key wait: a notification means the phone is awake, keys arrive as messages
|
||||
}
|
||||
} else if (type == "encrypt") {
|
||||
if (const WaNode* count = node.child("count")) {
|
||||
int left = std::atoi(count->attr("value").c_str());
|
||||
if (left < WaConsts::kWaMinPrekeys) {
|
||||
log("server holds " + std::to_string(left) + " prekeys, uploading more");
|
||||
upload_prekeys();
|
||||
}
|
||||
}
|
||||
}
|
||||
// every other notification type is server bookkeeping we do not act on (trace shows it)
|
||||
}
|
||||
|
||||
void WaClient::handle_receipt(const WaNode& node) {
|
||||
// Receipts are the phone's side of the conversation made visible: a retry receipt means the phone got one of
|
||||
// our messages but could not decrypt it, and asks us to resend. We do not resend yet (gap, see docs), but we
|
||||
// say so loudly instead of ignoring it.
|
||||
if (is_retry_receipt(node)) {
|
||||
log("the phone could not decrypt our message " + node.attr("id") + " and asks for a resend (retry receipt from " +
|
||||
node.attr("from") + "); re-link if this repeats");
|
||||
}
|
||||
}
|
||||
|
||||
void WaClient::handle_message(const WaNode& node) {
|
||||
if (!device_.paired() || !device_.jid) return;
|
||||
std::string from = node.attr("from");
|
||||
if (from.substr(0, from.find_first_of(":@")) != own_user) return; // only the phone's own messages
|
||||
// The arrival breadcrumb comes before the filter on purpose: when the phone's key share never seems to arrive,
|
||||
// the trace must show whether nothing arrived or something arrived and was dropped.
|
||||
log("message from " + from + " type=" + node.attr("type") + " category=" + node.attr("category"));
|
||||
std::string own_user = device_.jid->substr(0, device_.jid->find('@'));
|
||||
own_user = own_user.substr(0, own_user.find(':'));
|
||||
std::string lid_user;
|
||||
if (device_.lid && !device_.lid->empty()) {
|
||||
lid_user = device_.lid->substr(0, device_.lid->find('@'));
|
||||
lid_user = lid_user.substr(0, lid_user.find(':'));
|
||||
}
|
||||
if (!is_own_sender(from, own_user, lid_user)) {
|
||||
// Not ours, so no key share inside: but the sender's push name teaches the contact search (docs 14).
|
||||
if (auto c = push_name_candidate(node, own_user, lid_user)) {
|
||||
if (store_.put_contact_if_absent(c->first, c->second)) log("learned contact " + c->second + " (" + c->first + ")");
|
||||
}
|
||||
return;
|
||||
}
|
||||
const WaNode::Children* kids = node.children();
|
||||
if (kids == nullptr) return;
|
||||
for (const auto& enc : *kids) {
|
||||
@@ -798,12 +939,22 @@ void WaClient::handle_message(const WaNode& node) {
|
||||
}
|
||||
WaProtocolMessage pm = parse_protocol_message(res.message);
|
||||
if (pm.key_share.empty()) continue;
|
||||
int stored = 0, empty = 0;
|
||||
for (const auto& k : pm.key_share) {
|
||||
if (k.key_id.empty() || k.key_data.empty()) continue;
|
||||
if (k.key_id.empty() || k.key_data.empty()) {
|
||||
empty++;
|
||||
continue;
|
||||
}
|
||||
store_.put_app_key(k.key_id, k.key_data, k.timestamp);
|
||||
app_keys_received_++;
|
||||
stored++;
|
||||
}
|
||||
log("app state keys received: " + std::to_string(pm.key_share.size()));
|
||||
app_keys_answered_ = true;
|
||||
if (stored > 0) log("app state keys received: " + std::to_string(stored) + " stored");
|
||||
if (empty > 0)
|
||||
log("the phone answered with " + std::to_string(empty) +
|
||||
" key share(s) without key data (it has none to give right now); the sync retries on a later login");
|
||||
if (node.attr("category") == "peer") send_protocol_receipt(node.attr("id"), WaConsts::kWaReceiptPeerMsg);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -819,20 +970,25 @@ bool WaClient::send_peer_message(const Bytes& message_plaintext) {
|
||||
lid_user = lid_user.substr(0, lid_user.find(':'));
|
||||
enc_jid = lid_user + "@lid";
|
||||
}
|
||||
if (!ensure_sessions({enc_jid})) return false;
|
||||
// A peer message needs a Signal session with the phone first (Volt-Chat whatsmeow PR 5): without one it can
|
||||
// never leave the process, and stamping it requested would block the retry for 24 h (handled by the caller).
|
||||
if (!ensure_sessions({enc_jid})) {
|
||||
log("peer message: could not fetch the phone's prekeys, no session");
|
||||
return false;
|
||||
}
|
||||
if (!store_.load_session(addr_of(enc_jid))) {
|
||||
log("peer message: no session with the phone at " + addr_of(enc_jid) + " even after the prekey fetch");
|
||||
return false;
|
||||
}
|
||||
auto nodes = encrypt_for_devices({enc_jid}, WaCrypto::pkcs7_pad(message_plaintext), "");
|
||||
if (nodes.empty() || nodes[0].children() == nullptr) {
|
||||
log("peer message: no session with the phone");
|
||||
return false;
|
||||
}
|
||||
WaNode::Children content{WaNode("meta")};
|
||||
content[0].attrs["appdata"] = "default";
|
||||
content.push_back(nodes[0].children()->front());
|
||||
if (!device_.account.empty()) {
|
||||
WaNode di("device-identity");
|
||||
di.content = device_.account;
|
||||
content.push_back(di);
|
||||
}
|
||||
const WaNode& enc = nodes[0].children()->front();
|
||||
bool is_prekey = enc.attr("type") == "pkmsg";
|
||||
log("peer message to " + enc_jid + " as " + enc.attr("type") +
|
||||
(is_prekey ? " (first contact, the phone builds its session from this)" : " (existing session)"));
|
||||
std::string id = "3EB0" + [] {
|
||||
Bytes r = WaCrypto::random_bytes(8);
|
||||
static const char* H = "0123456789ABCDEF";
|
||||
@@ -843,19 +999,22 @@ bool WaClient::send_peer_message(const Bytes& message_plaintext) {
|
||||
}
|
||||
return s;
|
||||
}();
|
||||
WaNode msg("message");
|
||||
msg.attrs["id"] = id;
|
||||
msg.attrs["to"] = own_user + "@" + WaConsts::kServerJid;
|
||||
msg.attrs["type"] = "text";
|
||||
msg.attrs["category"] = "peer";
|
||||
msg.attrs["push_priority"] = "high";
|
||||
msg.content = content;
|
||||
send_node(msg);
|
||||
send_node(build_peer_message_node(id, own_user + "@" + WaConsts::kServerJid, enc, is_prekey, device_.account));
|
||||
auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(20);
|
||||
while (std::chrono::steady_clock::now() < deadline) {
|
||||
if (!socket_open()) break;
|
||||
auto node = recv_node(2);
|
||||
if (!node) continue;
|
||||
if (node->tag == "ack" && node->attr("id") == id) return true;
|
||||
if (node->tag == "ack" && node->attr("id") == id) {
|
||||
std::string line = "peer message: server ack";
|
||||
for (const auto& kv : node->attrs) line += " " + kv.first + "=" + kv.second;
|
||||
log(line);
|
||||
if (node->has_attr("error")) {
|
||||
log("peer message: the server rejected the request (error " + node->attr("error") + ")");
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
dispatch(*node);
|
||||
}
|
||||
log("peer message: no ack received");
|
||||
@@ -877,36 +1036,54 @@ void WaClient::request_app_state_keys(const std::vector<Bytes>& key_ids) {
|
||||
return;
|
||||
}
|
||||
log("app state keys: requesting " + std::to_string(due.size()) + " from the phone");
|
||||
send_peer_message(encode_app_state_key_request(due));
|
||||
if (send_peer_message(encode_app_state_key_request(due))) return;
|
||||
// The request never left this device (no session, no ack, server rejection): forget that we asked, so the
|
||||
// next sync asks again instead of waiting out the 24 h on keys nobody will send (whatsmeow PR 1268).
|
||||
for (const auto& id : due) store_.clear_app_key_request_time(id);
|
||||
log("app state keys: the request failed to send; cleared the 24 h mark so the next sync asks again");
|
||||
}
|
||||
|
||||
void WaClient::sync_app_state() {
|
||||
WaAppStateSync sync(store_, [this](const std::string& s) { log(s); });
|
||||
auto iq = [this](WaNode q) {
|
||||
q.attrs["id"] = next_iq_id();
|
||||
return send_iq(q, WaConsts::kWaAppStateTimeout);
|
||||
};
|
||||
auto download = [this](const WaExternalBlobRef& ref, std::string& why) -> std::optional<Bytes> {
|
||||
auto mc = query_media_conn();
|
||||
if (!mc) {
|
||||
why = "no media connection";
|
||||
return std::nullopt;
|
||||
}
|
||||
return WaMedia::download(mc->host, ref.direct_path, "md-app-state", ref.media_key, "WhatsApp App State Keys",
|
||||
ref.file_enc_sha256, ref.file_sha256, WaConsts::kWaAppStateBlobMax, why);
|
||||
};
|
||||
auto rep = sync.sync(WaAppStateSync::kContactsCollection, iq, download);
|
||||
if (rep.missing_key_ids.empty()) return;
|
||||
// The phone holds the keys: ask for them, give it a bounded window to answer, then sync once more.
|
||||
request_app_state_keys(rep.missing_key_ids);
|
||||
int before = app_keys_received_;
|
||||
run_until([&]() { return app_keys_received_ != before; }, WaConsts::kWaAppStateKeyWait);
|
||||
if (app_keys_received_ == before) {
|
||||
log("app state: no key share arrived; contacts will sync on a later login");
|
||||
void WaClient::sync_app_state(int key_wait_seconds) {
|
||||
if (sync_active_) {
|
||||
sync_again_ = true; // a notification sync during a sync: one more pass after, never nested
|
||||
return;
|
||||
}
|
||||
WaAppStateSync again(store_, [this](const std::string& s) { log(s); });
|
||||
again.sync(WaAppStateSync::kContactsCollection, iq, download);
|
||||
sync_active_ = true;
|
||||
for (int pass = 0; pass < 3; ++pass) {
|
||||
sync_again_ = false;
|
||||
WaAppStateSync sync(store_, [this](const std::string& s) { log(s); });
|
||||
auto iq = [this](WaNode q) {
|
||||
q.attrs["id"] = next_iq_id();
|
||||
return send_iq(q, WaConsts::kWaAppStateTimeout);
|
||||
};
|
||||
auto download = [this](const WaExternalBlobRef& ref, std::string& why) -> std::optional<Bytes> {
|
||||
auto mc = query_media_conn();
|
||||
if (!mc) {
|
||||
why = "no media connection";
|
||||
return std::nullopt;
|
||||
}
|
||||
return WaMedia::download(mc->host, ref.direct_path, "md-app-state", ref.media_key, "WhatsApp App State Keys",
|
||||
ref.file_enc_sha256, ref.file_sha256, WaConsts::kWaAppStateBlobMax, why);
|
||||
};
|
||||
auto rep = sync.sync(WaAppStateSync::kContactsCollection, iq, download);
|
||||
if (!rep.missing_key_ids.empty() && key_wait_seconds > 0) {
|
||||
// The phone holds the keys: ask for them, give it a bounded window to answer, then sync once more.
|
||||
request_app_state_keys(rep.missing_key_ids);
|
||||
int before = app_keys_received_;
|
||||
bool answered_before = app_keys_answered_;
|
||||
run_until([&]() { return app_keys_received_ != before || app_keys_answered_ != answered_before; },
|
||||
key_wait_seconds);
|
||||
if (app_keys_received_ == before && app_keys_answered_ == answered_before) {
|
||||
log("app state: no key share arrived; contacts will sync on a later login");
|
||||
} else if (app_keys_received_ != before) {
|
||||
WaAppStateSync again(store_, [this](const std::string& s) { log(s); });
|
||||
again.sync(WaAppStateSync::kContactsCollection, iq, download);
|
||||
}
|
||||
// an answered-but-empty share needs no re-sync: the version stays 0, so the next login retries the full sync
|
||||
}
|
||||
if (!sync_again_) break;
|
||||
}
|
||||
sync_active_ = false;
|
||||
}
|
||||
|
||||
void WaClient::disconnect() {
|
||||
|
||||
@@ -33,6 +33,9 @@ class WaStorePort {
|
||||
virtual std::optional<Bytes> get_prekey(uint32_t id) = 0;
|
||||
// contacts / aliases (name -> jid)
|
||||
virtual void put_contact(const std::string& jid, const std::string& name) = 0;
|
||||
// Stores a push name learned from an incoming message only when the jid has no name yet (a saved address-book
|
||||
// name, VCF import or user alias always wins over a self-reported push name). Returns true when stored.
|
||||
virtual bool put_contact_if_absent(const std::string& jid, const std::string& name) = 0;
|
||||
virtual std::vector<std::pair<std::string, std::string>> contacts() = 0; // (jid,name)
|
||||
virtual void remove_contact(const std::string& jid) = 0;
|
||||
|
||||
@@ -51,6 +54,9 @@ class WaStorePort {
|
||||
virtual void delete_app_state_value_mac(const std::string& name, const Bytes& index_mac) = 0;
|
||||
virtual std::optional<int64_t> app_key_request_time(const Bytes& key_id) = 0;
|
||||
virtual void put_app_key_request_time(const Bytes& key_id, int64_t when) = 0;
|
||||
// Forgets a request mark (whatsmeow PR 1268): when the key request never left this device, the next sync
|
||||
// must be free to ask again instead of waiting out the 24 h.
|
||||
virtual void clear_app_key_request_time(const Bytes& key_id) = 0;
|
||||
};
|
||||
|
||||
class WaClient {
|
||||
@@ -89,7 +95,28 @@ class WaClient {
|
||||
// kWaMinPrekeys (iq xmlns="encrypt"). Returns true when enough are present.
|
||||
bool upload_prekeys();
|
||||
// Pull the app-state contact list now (login runs this once). Public for the probe and tests.
|
||||
void sync_contacts() { sync_app_state(); }
|
||||
void sync_contacts();
|
||||
// Same, but waits up to `wait_seconds` for the phone's key share instead of the login default. Used by
|
||||
// `--whatsapp-sync`: a phone that is asleep can take minutes to wake and answer.
|
||||
void sync_contacts_with_wait(int wait_seconds);
|
||||
|
||||
// Pure helpers (no network), public for unit tests.
|
||||
// whatsmeow IsFromMe for a 1:1 sender: the user part matches our phone number or our LID.
|
||||
static bool is_own_sender(const std::string& from, const std::string& own_pn_user, const std::string& own_lid_user);
|
||||
// A peer message node (whatsmeow preparePeerMessageNode): meta + enc, device-identity only on a prekey
|
||||
// message. `to` is our own phone-number JID; the enc was encrypted to the phone's LID identity.
|
||||
static WaNode build_peer_message_node(const std::string& id, const std::string& to, const WaNode& enc, bool is_prekey,
|
||||
const Bytes& account);
|
||||
// Collection names carried by a server_sync notification (empty when none).
|
||||
static std::vector<std::string> server_sync_collections(const WaNode& notification);
|
||||
// True for a retry receipt: the phone could not decrypt one of our messages and asks for a resend.
|
||||
static bool is_retry_receipt(const WaNode& node);
|
||||
// A contact learnable from an incoming <message>: the sender's push name (the `notify` attribute) plus the
|
||||
// sender JID in non-AD user@server form (the PN form when an alternate is present). Empty when there is nothing
|
||||
// to learn: no usable notify (missing or "-"), the sender is ourselves, or the sender is not a user (group
|
||||
// senders come from `participant`). Pure, unit-tested.
|
||||
static std::optional<std::pair<std::string, std::string>> push_name_candidate(
|
||||
const WaNode& node, const std::string& own_pn_user, const std::string& own_lid_user);
|
||||
|
||||
// Send a 1:1 text message. Requires a paired, logged-in client. Returns the
|
||||
// message id on success. Live path, not verified in-tree.
|
||||
@@ -137,14 +164,23 @@ class WaClient {
|
||||
void send_ack(const WaNode& node);
|
||||
// --- app-state sync and key exchange (docs/whatsapp.md section 14) ---
|
||||
int app_keys_received_{0}; // app-state keys stored by this client instance
|
||||
bool app_keys_answered_{false}; // the phone sent a key share (with or without key data)
|
||||
bool sync_active_{false}; // reentrancy guard: a notification sync never nests inside a sync
|
||||
bool sync_again_{false}; // a server_sync notification arrived mid-sync: one more pass after
|
||||
// Receives a <message>: acked by dispatch; decrypts the phone's own messages and stores key shares.
|
||||
void handle_message(const WaNode& node);
|
||||
// Receives a <notification>: acked here; server_sync re-syncs the contact collection, encrypt refills prekeys.
|
||||
void handle_notification(const WaNode& node);
|
||||
// Receives a <receipt>: retry receipts (the phone could not decrypt our message) are logged loudly.
|
||||
void handle_receipt(const WaNode& node);
|
||||
// Sends a protocol receipt (peer_msg / hist_sync) for a consumed peer message, as whatsmeow does.
|
||||
void send_protocol_receipt(const std::string& id, const std::string& type);
|
||||
// Sends a peer message (category "peer") to our own primary device and waits for its ack.
|
||||
bool send_peer_message(const Bytes& message_plaintext);
|
||||
// Asks the phone for the given app-state key ids, at most one request per key id per 24 h.
|
||||
void request_app_state_keys(const std::vector<Bytes>& key_ids);
|
||||
// Syncs the contact collection (login hook); never throws to the caller.
|
||||
void sync_app_state();
|
||||
void sync_app_state(int key_wait_seconds);
|
||||
std::string next_iq_id();
|
||||
|
||||
// --- send path helpers (live, unverified in-tree) ---
|
||||
|
||||
@@ -10,6 +10,11 @@ namespace detz::wa {
|
||||
struct WaConsts {
|
||||
// The WhatsApp server JID (whatsmeow types.ServerJID). Every pairing and login reply is addressed here.
|
||||
static constexpr const char* kServerJid = "s.whatsapp.net";
|
||||
// LID and group-chat servers (whatsmeow HiddenUserServer, GroupServer).
|
||||
static constexpr const char* kLidServer = "lid";
|
||||
static constexpr const char* kGroupServer = "g.us";
|
||||
// A push name learned from an incoming message is truncated to this, never stored longer.
|
||||
static constexpr size_t kWaPushNameMax = 128;
|
||||
|
||||
// WhatsApp Web client version (whatsmeow store/clientpayload.go). A stale
|
||||
// value makes the server answer with a 405 "client outdated" failure.
|
||||
@@ -35,6 +40,14 @@ struct WaConsts {
|
||||
static constexpr int64_t kWaAppStateKeyRequestInterval = 24 * 60 * 60; // seconds between key requests for one key id
|
||||
static constexpr int kWaAppStateKeyWait = 20; // seconds to wait for requested keys during login
|
||||
static constexpr size_t kWaAppStateKeyIdsMax = 16; // key ids per key request message
|
||||
static constexpr int kWaSyncOnlineWait = 300; // seconds --whatsapp-sync stays online for the key share
|
||||
// protocol receipt types (whatsmeow types.ReceiptType*): sent for a consumed peer message, or received when the
|
||||
// phone could not decrypt one of ours (retry)
|
||||
static constexpr const char* kWaReceiptPeerMsg = "peer_msg";
|
||||
static constexpr const char* kWaReceiptHistSync = "hist_sync";
|
||||
static constexpr const char* kWaReceiptRetry = "retry";
|
||||
// vCard contact import fallback (--whatsapp-import)
|
||||
static constexpr size_t kWaVcfFileMax = 8u * 1024 * 1024; // refuse a bigger .vcf, loud
|
||||
};
|
||||
|
||||
} // namespace detz::wa
|
||||
|
||||
@@ -124,6 +124,13 @@ void WaStore::put_contact(const std::string& jid, const std::string& name) {
|
||||
db_->commit();
|
||||
}
|
||||
|
||||
bool WaStore::put_contact_if_absent(const std::string& jid, const std::string& name) {
|
||||
auto cur = db_->execute("SELECT name FROM wa_contact WHERE jid=?", {store::Cell(jid)});
|
||||
if (cur.first() != nullptr) return false; // a saved, imported or aliased name always wins over a push name
|
||||
put_contact(jid, name);
|
||||
return true;
|
||||
}
|
||||
|
||||
std::vector<std::pair<std::string, std::string>> WaStore::contacts() {
|
||||
auto cur = db_->execute("SELECT jid, name FROM wa_contact ORDER BY name");
|
||||
std::vector<std::pair<std::string, std::string>> out;
|
||||
@@ -268,6 +275,11 @@ void WaStore::put_app_key_request_time(const Bytes& key_id, int64_t when) {
|
||||
db_->commit();
|
||||
}
|
||||
|
||||
void WaStore::clear_app_key_request_time(const Bytes& key_id) {
|
||||
db_->execute("DELETE FROM wa_appstate_keyreq WHERE key_id=?", {store::Cell(tohex(key_id))});
|
||||
db_->commit();
|
||||
}
|
||||
|
||||
void WaStore::clear_device() {
|
||||
db_->execute("DELETE FROM wa_device");
|
||||
db_->execute("DELETE FROM wa_session");
|
||||
|
||||
@@ -29,6 +29,7 @@ class WaStore : public WaStorePort {
|
||||
void put_prekey(uint32_t id, const Bytes& priv) override;
|
||||
std::optional<Bytes> get_prekey(uint32_t id) override;
|
||||
void put_contact(const std::string& jid, const std::string& name) override;
|
||||
bool put_contact_if_absent(const std::string& jid, const std::string& name) override;
|
||||
std::vector<std::pair<std::string, std::string>> contacts() override;
|
||||
void remove_contact(const std::string& jid) override;
|
||||
|
||||
@@ -53,6 +54,7 @@ class WaStore : public WaStorePort {
|
||||
void delete_app_state_value_mac(const std::string& name, const Bytes& index_mac) override;
|
||||
std::optional<int64_t> app_key_request_time(const Bytes& key_id) override;
|
||||
void put_app_key_request_time(const Bytes& key_id, int64_t when) override;
|
||||
void clear_app_key_request_time(const Bytes& key_id) override;
|
||||
|
||||
// true if a paired device is stored (used to decide link vs send).
|
||||
bool has_device();
|
||||
|
||||
@@ -0,0 +1,151 @@
|
||||
#include "wa_vcf.h"
|
||||
|
||||
#include "wa_consts.h"
|
||||
|
||||
namespace detz::wa {
|
||||
|
||||
namespace {
|
||||
std::string strip(const std::string& s) {
|
||||
size_t a = s.find_first_not_of(" \t\r\n");
|
||||
if (a == std::string::npos) return {};
|
||||
return s.substr(a, s.find_last_not_of(" \t\r\n") - a + 1);
|
||||
}
|
||||
// vCard text unescaping: \, \; \n \\ (3.0/4.0 section 3.4).
|
||||
std::string unescape(const std::string& s) {
|
||||
std::string out;
|
||||
for (size_t i = 0; i < s.size(); ++i) {
|
||||
if (s[i] == '\\' && i + 1 < s.size()) {
|
||||
char n = s[++i];
|
||||
out.push_back(n == 'n' || n == 'N' ? ' ' : n);
|
||||
} else {
|
||||
out.push_back(s[i]);
|
||||
}
|
||||
}
|
||||
return strip(out);
|
||||
}
|
||||
// A TEL value to international digits: keeps a leading '+' (dropped), digits; needs a country code (>= 7 digits).
|
||||
std::string normalize_number(const std::string& v) {
|
||||
std::string digits;
|
||||
for (char c : v) {
|
||||
if (c >= '0' && c <= '9') digits.push_back(c);
|
||||
}
|
||||
// some exports write the national form with a trunk 0 and no country code; that is unusable, drop it loudly
|
||||
if (digits.size() < 7 || (digits.size() < 10 && digits[0] == '0')) return {};
|
||||
if (digits.size() >= 10 && digits[0] == '0' && digits[1] == '0') digits.erase(0, 2); // 00<intl> form
|
||||
return digits;
|
||||
}
|
||||
} // namespace
|
||||
|
||||
std::vector<WaVcf::Contact> WaVcf::parse(const std::string& text, int& skipped) {
|
||||
skipped = 0;
|
||||
// unfold: a line starting with a space or tab continues the previous line (the check runs on the raw line:
|
||||
// stripping first would eat the continuation mark). Content spaces survive; only the CRLF goes.
|
||||
std::vector<std::string> lines;
|
||||
std::string cur;
|
||||
for (size_t i = 0; i < text.size();) {
|
||||
size_t e = text.find('\n', i);
|
||||
std::string raw = text.substr(i, e == std::string::npos ? e : e - i);
|
||||
i = e == std::string::npos ? text.size() : e + 1;
|
||||
if (!raw.empty() && raw.back() == '\r') raw.pop_back();
|
||||
if (!raw.empty() && (raw[0] == ' ' || raw[0] == '\t') && !cur.empty()) {
|
||||
cur += strip(raw);
|
||||
} else {
|
||||
if (!cur.empty()) lines.push_back(strip(cur));
|
||||
cur = raw;
|
||||
}
|
||||
}
|
||||
if (!cur.empty()) lines.push_back(strip(cur));
|
||||
|
||||
std::vector<Contact> out;
|
||||
bool in_card = false;
|
||||
std::string fn, n;
|
||||
std::vector<std::string> tels;
|
||||
auto flush = [&] {
|
||||
if (!in_card) return;
|
||||
in_card = false;
|
||||
std::string name = strip(fn);
|
||||
if (name.empty() && !n.empty()) {
|
||||
// N: Family;Given;Middle;Prefix;Suffix -> "Given Family"
|
||||
std::vector<std::string> parts{""};
|
||||
for (char c : n) {
|
||||
if (c == ';') parts.push_back("");
|
||||
else parts.back().push_back(c);
|
||||
}
|
||||
std::string given = parts.size() > 1 ? strip(parts[1]) : "";
|
||||
std::string family = strip(parts[0]);
|
||||
name = strip(given.empty() ? family : given + (family.empty() ? "" : " " + family));
|
||||
}
|
||||
if (name.empty() || tels.empty()) {
|
||||
skipped++;
|
||||
return;
|
||||
}
|
||||
bool any = false;
|
||||
for (const auto& t : tels) {
|
||||
std::string digits = normalize_number(t);
|
||||
if (digits.empty()) continue;
|
||||
out.push_back({name, digits});
|
||||
any = true;
|
||||
}
|
||||
if (!any) skipped++;
|
||||
};
|
||||
for (const auto& line : lines) {
|
||||
size_t colon = line.find(':');
|
||||
if (colon == std::string::npos) continue;
|
||||
std::string head = line.substr(0, colon);
|
||||
std::string value = line.substr(colon + 1);
|
||||
std::string prop;
|
||||
for (char c : head) {
|
||||
if (c == ';') break; // parameters (TYPE=..., CHARSET=...) do not change the property
|
||||
prop.push_back(static_cast<char>(std::toupper(static_cast<unsigned char>(c))));
|
||||
}
|
||||
if (prop == "BEGIN") {
|
||||
flush();
|
||||
if (value == "VCARD" || value == "vCard") {
|
||||
in_card = true;
|
||||
fn.clear();
|
||||
n.clear();
|
||||
tels.clear();
|
||||
}
|
||||
} else if (prop == "END") {
|
||||
flush();
|
||||
} else if (!in_card) {
|
||||
continue;
|
||||
} else if (prop == "FN") {
|
||||
if (fn.empty()) fn = unescape(value);
|
||||
} else if (prop == "N") {
|
||||
if (n.empty()) n = unescape(value);
|
||||
} else if (prop == "TEL") {
|
||||
if (tels.size() < 8) tels.push_back(value); // bound per-card numbers; more are noise
|
||||
}
|
||||
}
|
||||
flush();
|
||||
return out;
|
||||
}
|
||||
|
||||
WaVcf::Report WaVcf::import_text(WaStorePort& store, const std::string& text) {
|
||||
Report rep;
|
||||
int skipped = 0;
|
||||
std::vector<Contact> contacts;
|
||||
try {
|
||||
contacts = parse(text, skipped);
|
||||
} catch (...) {
|
||||
rep.skipped = 1;
|
||||
return rep;
|
||||
}
|
||||
rep.skipped = skipped;
|
||||
for (const auto& c : contacts) {
|
||||
if (rep.imported >= WaConsts::kWaContactsMax) {
|
||||
rep.capped++;
|
||||
continue;
|
||||
}
|
||||
try {
|
||||
store.put_contact(c.jid(), c.name);
|
||||
rep.imported++;
|
||||
} catch (...) {
|
||||
rep.skipped++;
|
||||
}
|
||||
}
|
||||
return rep;
|
||||
}
|
||||
|
||||
} // namespace detz::wa
|
||||
@@ -0,0 +1,32 @@
|
||||
#pragma once
|
||||
// WaVcf: the offline fallback for the phone address book (problem.md). When the phone never sends its
|
||||
// app-state sync keys, the user exports the contacts on the phone as a .vcf file and imports it here; the
|
||||
// names land in the same contact search the app-state sync feeds. vCard 3.0/4.0, FN (else N) plus TEL lines.
|
||||
// See docs/whatsapp.md section 14.
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "wa_client.h"
|
||||
|
||||
namespace detz::wa {
|
||||
|
||||
class WaVcf {
|
||||
public:
|
||||
struct Contact {
|
||||
std::string name; // display name (FN, else "Given Family" from N)
|
||||
std::string number; // digits with country code, no '+'
|
||||
std::string jid() const { return number + "@s.whatsapp.net"; }
|
||||
};
|
||||
struct Report {
|
||||
int imported{0};
|
||||
int skipped{0}; // entries without a usable name or number
|
||||
int capped{0}; // entries past kWaContactsMax: not stored
|
||||
};
|
||||
|
||||
// Parses vCard text (unfolds continuations, reads BEGIN/END blocks). Pure, unit-tested.
|
||||
static std::vector<Contact> parse(const std::string& text, int& skipped);
|
||||
// Parses and stores into the contact search. Never throws to the caller; `report` says what happened.
|
||||
static Report import_text(WaStorePort& store, const std::string& text);
|
||||
};
|
||||
|
||||
} // namespace detz::wa
|
||||
@@ -3,11 +3,13 @@
|
||||
#include <atomic>
|
||||
#include <csignal>
|
||||
#include <filesystem>
|
||||
#include <fstream>
|
||||
|
||||
#include "wa_client.h"
|
||||
#include "wa_consts.h"
|
||||
#include "wa_qr.h"
|
||||
#include "wa_store.h"
|
||||
#include "wa_vcf.h"
|
||||
|
||||
namespace detz::wa {
|
||||
|
||||
@@ -116,4 +118,68 @@ int WaWizard::status(const std::string& home, const std::string& profile, std::o
|
||||
return 1;
|
||||
}
|
||||
|
||||
int WaWizard::sync(const std::string& home, const std::string& profile, std::ostream& out, int wait_seconds) {
|
||||
WaStore store(home, profile);
|
||||
if (!store.has_device()) {
|
||||
out << "WhatsApp: not linked (run --whatsapp-link first)\n";
|
||||
return 2;
|
||||
}
|
||||
if (wait_seconds < 0) wait_seconds = 0;
|
||||
if (wait_seconds > 3600) wait_seconds = 3600; // an hour online is enough for any phone
|
||||
WaClient client(store);
|
||||
client.set_capture_path(home + "/whatsapp/last-error.log");
|
||||
client.on_log = [&](const std::string& s) { out << " " << s << "\n"; };
|
||||
out << "Logging in...\n";
|
||||
if (!client.login(WaConsts::kWaLoginTimeout)) {
|
||||
out << "WhatsApp login did not complete"
|
||||
<< (client.last_failure_code() ? " (server reason " + std::to_string(client.last_failure_code()) + ")" : "")
|
||||
<< ".\n";
|
||||
return 2;
|
||||
}
|
||||
// login() already synced once with the short wait; stay online so a sleeping phone can still answer the key
|
||||
// request it was just sent.
|
||||
out << "Logged in. Waiting up to " << wait_seconds << "s for the phone's address book"
|
||||
<< " (keep WhatsApp open on the phone)...\n";
|
||||
client.sync_contacts_with_wait(wait_seconds);
|
||||
client.disconnect();
|
||||
size_t n = store.contacts().size();
|
||||
if (n > 0) {
|
||||
out << "WhatsApp contacts: " << n << " known.\n";
|
||||
return 0;
|
||||
}
|
||||
out << "Still no contacts. The phone did not send its address book keys. This usually means WhatsApp on the\n"
|
||||
<< "phone is asleep or battery-restricted: open WhatsApp on the phone, turn off battery optimization for\n"
|
||||
<< "it, and run this again. As a fallback, export the contacts on the phone as a .vcf file and run\n"
|
||||
<< "`detz --whatsapp-import contacts.vcf`.\n";
|
||||
return 1;
|
||||
}
|
||||
|
||||
int WaWizard::import_vcf(const std::string& home, const std::string& profile, const std::string& path,
|
||||
std::ostream& out) {
|
||||
std::error_code ec;
|
||||
auto size = std::filesystem::file_size(path, ec);
|
||||
if (ec) {
|
||||
out << "Cannot read " << path << ".\n";
|
||||
return 1;
|
||||
}
|
||||
if (size > WaConsts::kWaVcfFileMax) {
|
||||
out << path << " is " << size << " bytes, over the " << WaConsts::kWaVcfFileMax << " byte limit. Refusing.\n";
|
||||
return 1;
|
||||
}
|
||||
std::ifstream in(path, std::ios::binary);
|
||||
if (!in) {
|
||||
out << "Cannot read " << path << ".\n";
|
||||
return 1;
|
||||
}
|
||||
std::string text((std::istreambuf_iterator<char>(in)), std::istreambuf_iterator<char>());
|
||||
WaStore store(home, profile);
|
||||
WaVcf::Report rep = WaVcf::import_text(store, text);
|
||||
out << "Imported " << rep.imported << " contact number(s) from " << path;
|
||||
if (rep.skipped > 0) out << " (" << rep.skipped << " skipped: no usable name or number)";
|
||||
if (rep.capped > 0) out << " (" << rep.capped << " past the " << WaConsts::kWaContactsMax << " contact cap)";
|
||||
out << ".\n";
|
||||
if (rep.imported == 0) return 1;
|
||||
return 0;
|
||||
}
|
||||
|
||||
} // namespace detz::wa
|
||||
|
||||
@@ -6,6 +6,8 @@
|
||||
#include <ostream>
|
||||
#include <string>
|
||||
|
||||
#include "wa_consts.h"
|
||||
|
||||
namespace detz::wa {
|
||||
|
||||
class WaWizard {
|
||||
@@ -17,6 +19,14 @@ class WaWizard {
|
||||
static int unlink(const std::string& home, const std::string& profile, std::ostream& out);
|
||||
// Print the current link status. 0 if linked, 1 if not.
|
||||
static int status(const std::string& home, const std::string& profile, std::ostream& out);
|
||||
// Stay online and pull the phone address book (contact sync with a long key wait for a sleeping phone).
|
||||
// 0 when at least one contact is known afterwards, 1 when none arrived, 2 when not linked / login failed.
|
||||
static int sync(const std::string& home, const std::string& profile, std::ostream& out,
|
||||
int wait_seconds = WaConsts::kWaSyncOnlineWait);
|
||||
// Import a .vcf file exported on the phone into the contact search (the offline fallback when the phone
|
||||
// never sends its app-state sync keys). 0 on success, 1 when the file cannot be read or holds no contacts.
|
||||
static int import_vcf(const std::string& home, const std::string& profile, const std::string& path,
|
||||
std::ostream& out);
|
||||
};
|
||||
|
||||
} // namespace detz::wa
|
||||
|
||||
@@ -15,6 +15,7 @@
|
||||
|
||||
#include "json.h"
|
||||
#include "wa_binary.h"
|
||||
#include "wa_client.h"
|
||||
#include "wa_crypto.h"
|
||||
#include "wa_device.h"
|
||||
#include "wa_noise.h"
|
||||
@@ -24,6 +25,7 @@
|
||||
#include "wa_media.h"
|
||||
#include "wa_receiver.h"
|
||||
#include "wa_store.h"
|
||||
#include "wa_vcf.h"
|
||||
|
||||
using namespace detz::wa;
|
||||
using detz::json::Value;
|
||||
@@ -666,6 +668,200 @@ static void test_contact_search() {
|
||||
std::filesystem::remove_all(home);
|
||||
}
|
||||
|
||||
// Key-request bookkeeping (whatsmeow PR 1268): a request that never left the device must not block the
|
||||
// retry for 24 h, so the mark can be cleared again.
|
||||
static void test_key_request_rollback() {
|
||||
std::printf("[appstate: key request rollback]\n");
|
||||
std::string home = make_temp_home();
|
||||
WaStore store(home, "keyreq");
|
||||
Bytes id{0x01, 0x02, 0x03};
|
||||
check(!store.app_key_request_time(id).has_value(), "no request mark for a fresh key id");
|
||||
store.put_app_key_request_time(id, 1700000000);
|
||||
auto got = store.app_key_request_time(id);
|
||||
check(got && *got == 1700000000, "request mark is stored");
|
||||
store.clear_app_key_request_time(id);
|
||||
check(!store.app_key_request_time(id).has_value(), "clearing the mark frees the key id for an immediate retry");
|
||||
store.clear_app_key_request_time(id); // clearing twice is a no-op, not an error
|
||||
check(!store.app_key_request_time(id).has_value(), "clearing an unmarked key id is harmless");
|
||||
std::filesystem::remove_all(home);
|
||||
}
|
||||
|
||||
// Peer message helpers: own-sender classification (PN or LID, whatsmeow IsFromMe), the peer node shape
|
||||
// (device-identity only on a prekey message), server_sync parsing and retry-receipt detection.
|
||||
static void test_peer_helpers() {
|
||||
std::printf("[appstate: peer message helpers]\n");
|
||||
check(WaClient::is_own_sender("31630045827@s.whatsapp.net", "31630045827", "173735822786802"),
|
||||
"own PN sender is accepted");
|
||||
check(WaClient::is_own_sender("173735822786802@lid", "31630045827", "173735822786802"),
|
||||
"own LID sender is accepted (the phone answers from its LID)");
|
||||
check(WaClient::is_own_sender("31630045827:0@s.whatsapp.net", "31630045827", ""),
|
||||
"own sender with an explicit device 0 is accepted");
|
||||
check(!WaClient::is_own_sender("31611111111@s.whatsapp.net", "31630045827", "173735822786802"),
|
||||
"another user is not an own sender");
|
||||
check(!WaClient::is_own_sender("173735822786802@lid", "31630045827", ""),
|
||||
"an unknown LID is not accepted without a stored LID");
|
||||
|
||||
WaNode enc("enc");
|
||||
enc.attrs["v"] = "2";
|
||||
enc.attrs["type"] = "pkmsg";
|
||||
enc.content = Bytes{1, 2, 3};
|
||||
Bytes account{9, 9, 9};
|
||||
WaNode pre = WaClient::build_peer_message_node("3EB0AA", "31630045827@s.whatsapp.net", enc, true, account);
|
||||
check(pre.tag == "message" && pre.attr("category") == "peer" && pre.attr("push_priority") == "high" &&
|
||||
pre.attr("to") == "31630045827@s.whatsapp.net" && pre.child("meta") != nullptr &&
|
||||
pre.child("enc") != nullptr && pre.child("device-identity") != nullptr,
|
||||
"peer node: meta + enc + device-identity on a prekey message");
|
||||
WaNode msg = WaClient::build_peer_message_node("3EB0AB", "31630045827@s.whatsapp.net", enc, false, account);
|
||||
check(msg.child("device-identity") == nullptr && msg.child("enc") != nullptr,
|
||||
"peer node: no device-identity on an existing-session message");
|
||||
|
||||
WaNode notif("notification");
|
||||
notif.attrs["type"] = "server_sync";
|
||||
WaNode c1("collection");
|
||||
c1.attrs["name"] = "critical_unblock_low";
|
||||
WaNode c2("collection");
|
||||
c2.attrs["name"] = "regular_high";
|
||||
notif.content = WaNode::Children{c1, c2};
|
||||
auto cols = WaClient::server_sync_collections(notif);
|
||||
check(cols.size() == 2 && cols[0] == "critical_unblock_low" && cols[1] == "regular_high",
|
||||
"server_sync notification yields its collection names");
|
||||
|
||||
WaNode retry("receipt");
|
||||
retry.attrs["type"] = "retry";
|
||||
WaNode read("receipt");
|
||||
read.attrs["type"] = "read";
|
||||
WaNode ack("ack");
|
||||
check(WaClient::is_retry_receipt(retry) && !WaClient::is_retry_receipt(read) && !WaClient::is_retry_receipt(ack),
|
||||
"only a retry receipt reads as one");
|
||||
}
|
||||
|
||||
// Empty key shares (openclaw/wacli PR 443): the phone answers with keys that carry no key data. The share
|
||||
// still parses, so the client can tell "answered empty" from "never answered".
|
||||
static void test_empty_key_share() {
|
||||
std::printf("[appstate: empty key share]\n");
|
||||
ProtoWriter key;
|
||||
{
|
||||
ProtoWriter keyid;
|
||||
keyid.bytes(1, Bytes{0x7c, 0x57});
|
||||
key.message(1, keyid.data());
|
||||
ProtoWriter data; // AppStateSyncKeyData without keyData: the phone has none
|
||||
data.varint(3, 1767225600);
|
||||
key.message(2, data.data());
|
||||
}
|
||||
ProtoWriter share;
|
||||
share.message(1, key.data());
|
||||
ProtoWriter pm;
|
||||
pm.varint(2, 6); // APP_STATE_SYNC_KEY_SHARE
|
||||
pm.message(7, share.data());
|
||||
ProtoWriter m;
|
||||
m.message(12, pm.data());
|
||||
WaProtocolMessage out = parse_protocol_message(m.take());
|
||||
check(out.type == 6 && out.key_share.size() == 1 && out.key_share[0].key_data.empty(),
|
||||
"an empty key share parses with empty key data (answered, but nothing to store)");
|
||||
}
|
||||
|
||||
// Push-name learning: every incoming message carries the sender's self-reported name in `notify`
|
||||
// (whatsmeow message.go). A sender we have no name for is learned; saved, imported and aliased names win.
|
||||
static void test_push_name_learning() {
|
||||
std::printf("[contacts: push-name learning]\n");
|
||||
const std::string me = "31630045827", my_lid = "173735822786802";
|
||||
auto msg = [](const std::string& from, const std::string& notify) {
|
||||
WaNode m("message");
|
||||
m.attrs["from"] = from;
|
||||
if (!notify.empty()) m.attrs["notify"] = notify;
|
||||
return m;
|
||||
};
|
||||
auto cand = [&](const WaNode& m) { return WaClient::push_name_candidate(m, me, my_lid); };
|
||||
{
|
||||
auto c = cand(msg("31611111111@s.whatsapp.net", "Nigel Dokter"));
|
||||
check(c && c->first == "31611111111@s.whatsapp.net" && c->second == "Nigel Dokter",
|
||||
"1:1 message learns the sender's push name");
|
||||
}
|
||||
{
|
||||
WaNode m = msg("179000000000001@lid", "Anne");
|
||||
m.attrs["sender_pn"] = "31622222222@s.whatsapp.net";
|
||||
auto c = cand(m);
|
||||
check(c && c->first == "31622222222@s.whatsapp.net" && c->second == "Anne",
|
||||
"a LID sender with a PN alternate is learned under the sendable PN");
|
||||
}
|
||||
{
|
||||
auto c = cand(msg("179000000000001@lid", "Anne"));
|
||||
check(c && c->first == "179000000000001@lid", "a LID sender without alternate is learned under its LID");
|
||||
}
|
||||
{
|
||||
WaNode m = msg("120363000000000@g.us", "Joop");
|
||||
m.attrs["participant"] = "31633333333@s.whatsapp.net";
|
||||
auto c = cand(m);
|
||||
check(c && c->first == "31633333333@s.whatsapp.net" && c->second == "Joop",
|
||||
"a group message learns the participant, not the group");
|
||||
}
|
||||
{
|
||||
WaNode m = msg("120363000000000@g.us", "Joop");
|
||||
m.attrs["participant"] = "179000000000002@lid";
|
||||
m.attrs["participant_pn"] = "31633333333@s.whatsapp.net";
|
||||
auto c = cand(m);
|
||||
check(c && c->first == "31633333333@s.whatsapp.net", "a group LID participant prefers its PN alternate");
|
||||
}
|
||||
check(!cand(msg("31630045827@s.whatsapp.net", "Me")), "our own messages teach nothing");
|
||||
check(!cand(msg("31611111111@s.whatsapp.net", "-")), "a '-' push name is not learned");
|
||||
check(!cand(msg("31611111111@s.whatsapp.net", "")), "a missing push name is not learned");
|
||||
check(!cand(msg("1234567890@newsletter", "Channel")), "a newsletter sender is not learned");
|
||||
{
|
||||
auto c = cand(msg("31611111111@s.whatsapp.net", std::string(300, 'x')));
|
||||
check(c && c->second.size() == 128, "an over-long push name is truncated, not refused");
|
||||
}
|
||||
|
||||
std::string home = make_temp_home();
|
||||
WaStore store(home, "push");
|
||||
check(store.put_contact_if_absent("31611111111@s.whatsapp.net", "Nigel Dokter"),
|
||||
"first sighting of a sender is stored");
|
||||
check(!store.put_contact_if_absent("31611111111@s.whatsapp.net", "Nigel Changed"),
|
||||
"a second sighting does not overwrite");
|
||||
check(store.contacts().size() == 1 && store.contacts()[0].second == "Nigel Dokter",
|
||||
"the first learned name sticks");
|
||||
auto hits = store.search_contacts("Niggel", 3, 0.3);
|
||||
check(!hits.empty() && hits[0].name == "Nigel Dokter", "a learned name is searchable with typos");
|
||||
store.put_contact("31611111111@s.whatsapp.net", "Nigel Dokter (work)"); // app-state-grade name wins
|
||||
check(store.contacts()[0].second == "Nigel Dokter (work)", "a saved name overwrites a learned push name");
|
||||
check(!store.put_contact_if_absent("31611111111@s.whatsapp.net", "Nigel Again"),
|
||||
"a push name never overwrites a saved name");
|
||||
std::filesystem::remove_all(home);
|
||||
}
|
||||
|
||||
// VCF fallback (problem.md): a .vcf exported on the phone feeds the same contact search the app-state sync feeds.
|
||||
static void test_vcf_import() {
|
||||
std::printf("[contacts: vcf import]\n");
|
||||
const char* vcf =
|
||||
"BEGIN:VCARD\r\nVERSION:3.0\r\nFN:Nigel Dokter\r\nTEL;TYPE=CELL:+31 6 1234 5001\r\nEND:VCARD\r\n"
|
||||
"BEGIN:VCARD\r\nVERSION:3.0\r\nN:Smit;Nina;;;\r\nTEL:+31612345002\r\nTEL:0031612345003\r\nEND:VCARD\r\n"
|
||||
"BEGIN:VCARD\r\nVERSION:4.0\r\nFN:Long \r\n Name\r\nTEL;TYPE=HOME:+31-6-1234-5004\r\nEND:VCARD\r\n"
|
||||
"BEGIN:VCARD\r\nVERSION:3.0\r\nFN:No Number\r\nEND:VCARD\r\n"
|
||||
"BEGIN:VCARD\r\nVERSION:3.0\r\nFN:Local Only\r\nTEL:0612345\r\nEND:VCARD\r\n";
|
||||
int skipped = 0;
|
||||
auto contacts = WaVcf::parse(vcf, skipped);
|
||||
check(contacts.size() == 4, "four usable numbers parse from five cards (one numberless, one local)");
|
||||
check(skipped == 2, "two cards skipped: no number, and a national number without country code");
|
||||
bool nigel = false, nina = false, unfolded = false;
|
||||
for (const auto& c : contacts) {
|
||||
if (c.name == "Nigel Dokter" && c.number == "31612345001") nigel = true;
|
||||
if (c.name == "Nina Smit" && (c.number == "31612345002" || c.number == "31612345003")) nina = true;
|
||||
if (c.name == "Long Name") unfolded = true;
|
||||
}
|
||||
check(nigel, "FN plus an international TEL with spaces");
|
||||
check(nina, "N fallback name plus two TELs (00 form normalized)");
|
||||
check(unfolded, "a folded FN line is unfolded");
|
||||
|
||||
std::string home = make_temp_home();
|
||||
WaStore store(home, "vcf");
|
||||
WaVcf::Report rep = WaVcf::import_text(store, vcf);
|
||||
check(rep.imported == 4 && rep.skipped == 2 && rep.capped == 0, "import stores 4 numbers, skips 2, caps none");
|
||||
auto hits = store.search_contacts("Niggel", 3, 0.3);
|
||||
check(!hits.empty() && hits[0].name == "Nigel Dokter", "an imported contact is found by the typo-tolerant search");
|
||||
WaVcf::Report empty = WaVcf::import_text(store, "not a vcard at all");
|
||||
check(empty.imported == 0, "garbage imports nothing and throws nothing");
|
||||
std::filesystem::remove_all(home);
|
||||
}
|
||||
|
||||
int main(int argc, char** argv) {
|
||||
DIR = argc > 1 ? argv[1] : "tests/whatsapp_data";
|
||||
test_contact_search();
|
||||
@@ -682,6 +878,11 @@ int main(int argc, char** argv) {
|
||||
test_lthash_and_keys();
|
||||
test_appstate_sync();
|
||||
test_receiver();
|
||||
test_key_request_rollback();
|
||||
test_peer_helpers();
|
||||
test_empty_key_share();
|
||||
test_vcf_import();
|
||||
test_push_name_learning();
|
||||
std::printf("%s\n", g_fail == 0 ? "whatsapp: ALL OK" : "whatsapp: FAILURES");
|
||||
return g_fail == 0 ? 0 : 1;
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user