49 lines
2.8 KiB
Markdown
49 lines
2.8 KiB
Markdown
 
|
# Asyncio: How to merge two streams — discuss.python.org
|
||
|
|
|
||
|
|
## Metadata
|
||
|
|
|
||
|
|
| Field | Value |
|
||
|
|
|---|---|
|
||
|
|
| Institute | Molodetz Institute — Research & Retrieval Division |
|
||
|
|
| Subject | async-pty-process-testing |
|
||
|
|
| Document type | Source Dossier |
|
||
|
|
| Tier | 1 |
|
||
|
|
| Source title | Asyncio: How to merge two streams |
|
||
|
|
| Source URL | https://discuss.python.org/t/asyncio-how-to-merge-two-streams/106616 |
|
||
|
|
| Source publisher / author | Python Discourse community |
|
||
|
|
| Source type | forum thread |
|
||
|
|
| Source date (published) | Unknown — not stated |
|
||
|
|
| Date researched | 2026-10-09 |
|
||
|
|
| Researched by | tai (agent), model: Claude Sonnet 5, on behalf of retoor |
|
||
|
|
| Research method | web search synthesis |
|
||
|
|
| Confidence | Medium |
|
||
|
|
| Status | Active |
|
||
|
|
|
||
|
|
## Summary
|
||
|
|
|
||
|
|
A Python Discourse thread discussing how to merge two independent asyncio streams (e.g. a subprocess's stdout and stderr) into one consumable sequence. The community-arrived-at pattern is one background task per stream, each pushing tagged chunks into a shared `asyncio.Queue`, with a single consumer draining the queue — but participants in the thread itself described the resulting plumbing as messy and said they were no longer comfortable maintaining it.
|
||
|
|
|
||
|
|
## Key findings
|
||
|
|
|
||
|
|
- The tagged-queue pattern (one task per stream, push `(stream_name, chunk)` tuples into a shared queue, one consumer loop) is the community's working answer, confirmed to function.
|
||
|
|
- Multiple participants in the same thread independently reported the approach grew complex enough to become a maintenance burden, and the thread never converged on one canonical, widely-endorsed implementation.
|
||
|
|
- `asyncio`'s own standard pattern naturally supports running several tasks (and even several subprocesses) concurrently via `asyncio.gather`, which the tagged-queue approach builds on top of.
|
||
|
|
|
||
|
|
## Pros / strengths
|
||
|
|
|
||
|
|
- Honest about the pattern's real-world maintainability cost, not just its mechanical correctness — useful signal for a framework design decision, not just a working snippet.
|
||
|
|
|
||
|
|
## Cons / limitations / caveats
|
||
|
|
|
||
|
|
- No single reference implementation to point to; "write your own small, well-tested helper" was effectively the thread's own conclusion, which is not a strong endorsement of the pattern as a reusable abstraction.
|
||
|
|
|
||
|
|
## Relevance to tai
|
||
|
|
|
||
|
|
This is the direct reason `tai_test.py` does **not** implement a generic multi-stream-merge utility. Its actual need — combining a subprocess's stdout and stderr into one stream for pipe-mode tests — is fully satisfied by the much simpler `stderr=asyncio.subprocess.STDOUT` argument to `create_subprocess_exec`, which merges them at the OS/subprocess level before asyncio ever sees two separate streams, sidestepping the complexity this thread describes entirely.
|
||
|
|
|
||
|
|
## Changelog
|
||
|
|
|
||
|
|
| Date | Change |
|
||
|
|
|---|---|
|
||
|
|
| 2026-10-09 | Initial dossier created. |
|