Somewhere in the last sprint, a developer previewed a roughly ten-megabyte log file inside an Influxx SSH Worktree, kept typing in the terminal pane next to it, and watched their own keystrokes disappear for seconds at a time. The remote machine wasn't overloaded and the network wasn't dropping packets — a single shared communication channel was simply treating a keystroke's echo exactly like ten megabytes of file data, because both had joined the same line and the line doesn't care who is in a hurry. This is the story of that channel, the 4,195,592 bytes we found queued in front of one keystroke, and the textbook networking bug that had been sitting in plain sight since the day we shipped remote worktrees.
What SSH Worktrees Actually Move Over the Wire
Influxx runs and orchestrates AI coding-agent CLIs — Claude Code, Codex, Cursor, GitHub Copilot, Gemini, Grok, Droid, Amp, OpenCode, and more — side by side, each in its own isolated git worktree, with notes and agents living in one cockpit instead of a browser tab and a separate notes app. Many of our users don't do their work on the laptop in front of them; their repository, their build tools, and often their agents live on a remote machine reached over SSH. SSH Worktrees exists to make that setup feel identical to working locally.
To do that, Influxx starts a small relay process on the remote machine and keeps one connection open to it for the life of a session. Keystrokes, terminal output, and bulk operations — reading a file for a preview, streaming a large diff — all travel across that same connection. Opening additional raw channels over SSH isn't free, so multiplexing everything onto one already-open connection was a reasonable simplification. It's also exactly the kind of simplification networking engineers have been writing postmortems about for decades.
The Symptom: A Keyboard That Stops Listening
The first reports were vague, the way these things usually are: typing felt "laggy," but only sometimes, and only over SSH Worktrees, never in a fully local session. Engineers tried reproducing it by typing into an empty terminal and couldn't — the connection looked fine. The pattern only appeared once someone noticed what else was running in the same session: a large file open in the preview pane, or a sizeable diff loading elsewhere in the same worktree. Typing while either was in flight was when the keyboard seemed to stop listening, sometimes for a couple of seconds at a stretch.
That detail — the freeze correlated with an operation the typing user hadn't necessarily triggered — was the thread worth pulling.
"I'd open a big log file in the preview pane just to skim it, then go back to the terminal to keep working, and my own typing would disappear for a second or two before it all showed up at once. I assumed my SSH connection was flaky until I noticed it only happened when that file was loading."
— Diego Farrell, backend engineer running Influxx against a shared build server
The Number That Made It Real: 4,195,592 Bytes
We instrumented the channel to answer one question: when a keystroke's echo sits in the queue, how much data is in front of it? With a roughly ten-megabyte file open for preview, the answer was up to 4,195,592 bytes — north of four megabytes of file data queued ahead of a few characters of terminal output. Because the channel delivered bytes in strict, single-file order, that keystroke echo couldn't go out until every one of those 4,195,592 bytes had gone first.
Translated into what a person at a keyboard actually feels: typing could freeze for multiple seconds at a time, triggered by something the person wasn't even doing themselves — a file preview loading, a diff streaming, in the background of the same SSH session.
Head-of-Line Blocking, Named
What we had was a clean, almost textbook case of head-of-line blocking — a classic networking failure mode where one big, low-urgency thing goes first only because it asked first, and blocks a small, high-urgency thing that arrived a moment later. It shows up anywhere a small, fast payload and a large, slow one share a single ordered queue: the queue has no concept of who is waiting on the result, only of who arrived first.
"Once we saw the byte count, this stopped being a mystery and became a name we already knew. We had one ordered pipe carrying a keystroke echo a human is staring at in real time, and a file transfer nobody watches byte by byte — and we'd given the pipe no way to tell them apart. First in, first out is a fine rule until the thing that arrived first is a ten-megabyte file and the thing right behind it is the letter the user just typed."
— Marcus Webb, Principal Systems Engineer at ETAPX
Why Typing Is the One Latency You Can't Hide
Plenty of latency in a developer tool is forgivable. A file preview that takes an extra half-second is mildly annoying at worst — most people aren't watching the clock while it loads. A keystroke is different: a person typing is in a tight, continuous feedback loop with the characters appearing on screen, and any perceptible gap between pressing a key and seeing it echoed back reads as broken, not slow. It's one of the few places in the app where latency is felt instantly, by a human whose fingers are already moving to the next key — exactly why fixing terminal input lag over SSH became a priority, not a someday-ticket.
Anatomy of the Fix
Once the shape of the problem was clear, the work fell into a fairly linear sequence.
- Reports come in describing intermittent typing lag: exclusively over SSH Worktrees, never in local sessions.
- Reproduction narrows the trigger: a large file preview or big diff loading elsewhere in the same session.
- Instrumentation measures queue depth: up to 4,195,592 bytes queued ahead of a single keystroke echo during a roughly ten-megabyte preview.
- Root cause is confirmed as head-of-line blocking: one strictly ordered channel, shared by bulk and interactive traffic, with no concept of priority.
- A second, unrelated bug turns up: the frame decoder re-concatenates its entire buffer on every new chunk, scaling quadratically with backlog size.
- The channel is split into two lanes: a bulk lane with real backpressure and a capped flow-control window, and an interactive lane admitted ahead of queued bulk data.
- The frame decoder is rewritten: to eliminate the repeated re-concatenation entirely.
- One limitation is documented rather than quietly left unfixed: a single oversized message can still occupy the channel atomically, deliberately deferred to a future protocol change.
Fix One: A Bulk Lane That Actually Respects Backpressure
The first change gave bulk data — file reads for previews, large diffs, anything meant to be consumed later rather than watched in real time — its own lane, separate from interactive terminal traffic. Two things make that lane behave:
- Real backpressure instead of blind queuing: if the underlying network write reports it can't accept more data right now, the bulk lane waits for a "drained" signal before sending anything else, instead of stacking up data regardless of whether the connection can move it.
- A credit-based flow-control window: the bulk lane caps how many chunks of bulk data can be in flight at once — up to four chunks, roughly one megabyte of in-flight data, instead of an unbounded amount.
Neither change makes a large transfer faster. Both make it smaller and more polite while it happens, which turns out to be the part that matters.
Fix Two: Letting Interactive Bytes Cut the Line
The second change fixes the symptom a person typing actually experiences. Terminal keystroke and output data is now treated as high priority: it's admitted to the channel immediately, ahead of any bulk data still waiting its turn, instead of sitting behind it in strict arrival order. A keystroke's echo no longer waits behind megabytes of an unrelated file transfer just because that transfer reached the queue a moment earlier — it goes first because it matters more right now, not because it arrived first.
Combined, the two changes are the difference between a queue and a priority system: bulk transfers still complete in full and still can't overrun the connection, but they no longer hold a keystroke hostage while they do it.
The Second Bug: A Quadratic Hiding in the Frame Decoder
While tracing the channel, we found a second, unrelated performance bug in the same code path — the part decoding incoming data frames off the wire. Every time a new chunk arrived, the decoder re-concatenated the entire buffer of already-arrived, not-yet-processed data before looking for a complete frame inside it. That's fine when the backlog is small. It stops being fine as the backlog grows, because the cost scales with the square of how much unprocessed data is already sitting there — every new chunk becomes more expensive to absorb, purely from copying memory that had already been copied moments before.
In practice, processing one roughly ten-megabyte transfer could trigger on the order of eighty megabytes of unnecessary memory copying, purely from repeatedly re-joining a buffer that mostly didn't need to move. We rewrote the decoder to stop re-concatenating on every arrival, removing that overhead and leaving the new priority lane far less backlog to fight through.
What We Didn't Fix
The honest part of this postmortem is what we didn't solve. The lanes, backpressure, and priority admission all operate on messages competing for the channel — which covers the overwhelming majority of real traffic, where a bulk operation is broken into many chunks a keystroke can now cut in front of. But the protocol still has a maximum message size, on the order of sixteen megabytes, and a single message near that ceiling is still transmitted as one atomic, unsplittable unit.
That means one unusually large message — an outsized git-diff response is the realistic example — can still delay a keystroke's echo by roughly however long that one message takes to move across the connection, because the protocol doesn't yet support splitting a single logical message into smaller pieces mid-flight. We know about this case. We chose not to fix it in the same pass.
"Splitting a message that's already on the wire is a different, harder problem than prioritizing between messages that haven't gone out yet. The lane and priority work solved the case actually causing the reported freezes — lots of ordinary-sized messages competing for one channel. A single sixteen-megabyte diff is real but much rarer, and fixing it means changing how the protocol frames a message, not just how we schedule frames we already have. We'd rather ship what helps almost everyone now and be upfront about what's still in progress."
— Marcus Webb, Principal Systems Engineer at ETAPX
Frequently Asked Questions
Was this bug unique to SSH Worktrees, or does it affect local sessions too?
It was specific to the remote path. The shared, strictly ordered channel that caused the head-of-line blocking connects the local app to the relay process on the remote machine — the piece SSH Worktrees adds. Local sessions don't route terminal output and bulk file reads through that same channel, so the multi-second freezes described here were remote-only.
Does the fix mean file previews or big diffs are now slower?
No. The bulk lane's backpressure and its four-chunk, roughly one-megabyte flow-control window govern how much data can be in flight at once, not how much total data moves or how fast the connection carries it. A large preview or diff still completes in full; it just can't flood the channel unconditionally while doing it.
Why not just make every message small from the start?
Because almost no ordinary operation produces a message anywhere near the maximum size, a generous framing ceiling was a sensible trade-off when SSH Worktrees was first built. The cost of sharing one channel at all was invisible until we instrumented a direct reproduction — a roughly ten-megabyte preview alongside active typing — and saw the queue depth for ourselves.
What's still left unfixed, and when would I notice it?
A single message that itself approaches the roughly sixteen-megabyte ceiling — an unusually large git diff is the realistic trigger — is still sent as one atomic, unsplittable unit, so a keystroke queued behind it waits roughly as long as that message takes to transfer. We've deliberately deferred this because fixing it needs true chunking within a single message, a deeper protocol change than the lane-and-priority fix that resolved the far more common case.
How was the 4,195,592-byte figure actually measured?
We instrumented the shared channel during a direct reproduction: open a roughly ten-megabyte file for preview in an SSH Worktree, type in the terminal in the same session, and trace how many bytes were already queued ahead of a keystroke's echo the moment it was sent. It wasn't estimated — it was observed.
Does this also help with things like agent output streaming, not just previews and diffs?
Yes. The fix operates at the channel level, not for any one feature — anything sending bulk data through the relay connection, including large agent output, benefits from the same lane separation and priority admission. File previews and diffs just happened to be the operations that made the bug easy to reproduce and measure.
None of this required exotic tooling — just tracing one channel closely enough to see what was actually queued in front of a keystroke, and naming the pattern once we saw it. Head-of-line blocking is old enough that most of us studied it before we ever worked here, which made it almost embarrassing to find sitting in our own SSH path. Typing is the one interaction in Influxx that every user touches constantly and expects zero ceremony around, which is why this fix mattered as much as it did — and why the one piece we left for later, a single oversized message on the wire, is now a documented limitation rather than a mystery someone has to rediscover.
