# Timothy Part 26: One Terminal Event and Five Ways to Lose It

> Source: https://www.sumonselim.com/timothy-part26-one-terminal-event-and-five-ways-to-lose-it
> Author: Muhammad Sumon Molla Selim
> Published: 2026-08-06
> Tag: Timothy
> Summary: A slow model on a CPU-only box made streaming chat turns vanish with zero trace: no assistant message, no error, no evidence the turn ran. Five commits trace one causal chain, defending a single contract in the gateway: a stream ends with exactly one terminal event.

A real chat turn ran for roughly thirty minutes against a remote model on a CPU-only homelab box. When it ended, the session had nothing to show for it. No assistant message. No error event. No `pending_state` partial. The `session_events` log for that turn was empty, and `turn_active` had silently flipped to false. From the database's point of view, the turn never happened.

This post is the story of the five commits that closed that hole, one layer at a time. Each fix revealed the next one underneath it. By the end the original symptom was impossible, but only because four different producers of the same event had been patched individually, and a fifth, structural backstop had been welded in underneath all of them.

## The contract being defended

Everything in this story comes back to one comment in the gateway's stream package:

```go
// internal/gateway/stream/events.go
// Channel contract: a stream ends with exactly one terminal event:
// "done" (provider finished, possibly preceded by "incomplete" when
// the stream was cut off) or "error" (nothing more is coming), and
// the channel closes immediately after.
```

Every provider, no matter its wire format, translates into this normalized vocabulary. Every consumer downstream of it, the gateway's own router, brain's gateway client, the agent loop, the chat persistence path, the missions runner, the SSE handler that faces the browser, assumes that contract holds. A stream that ends with zero events, or two terminals, or a terminal that never gets delivered, is not a well-formed empty answer. It's a broken turn.

D-044 is the durable-evidence half of the same promise: a turn that ran must leave either real content or a real failure event in `session_events`, the append-only log from [Part 6](https://www.sumonselim.com/articles/timothy-part6-sessions-that-survive-event-sourcing-the-backend). It must never vanish silently. The whole post is about defending that one line of comment, at every layer it got broken.

## Chapter 1: the proxy that kills idle connections

The only externally caused bug in this chain was also the one that exposed the rest of them.

[Part 21](https://www.sumonselim.com/articles/timothy-part21-pdf-attachments-and-a-turn-that-outlives-the-request) detached a chat turn from its HTTP request. A turn now runs on its own context with a 30-minute ceiling, and closing the browser tab no longer kills the answer that was almost done. That was the right trade, but it had a consequence nobody had seen yet: a detached turn can run long enough to trip a completely different class of timeout.

Cloudflare's 524, and similar proxy timeouts on other front-ends, do not fire on total request duration. They fire on a gap with no bytes flowing. A slow reasoning model on an overloaded CPU can go on the order of 100 seconds between tokens on a long turn. From the proxy's vantage point, nothing has moved for a minute and a half. The connection looks dead. So the proxy kills it, mid-generation, and the turn that was about to produce its next chunk gets cut off for the sin of being slow.

The fix is a heartbeat. Every 20 seconds, the SSE handler writes a comment line that carries no event name at all:

```go
// internal/brain/api/api.go
// streamHeartbeat keeps a chat SSE connection alive through proxies
// that kill an idle one (Cloudflare's ~100s 524 fires on a gap with no
// bytes flowing, not on total duration): a slow model can go well
// past that between tokens. A comment line carries no event, so it
// never reaches the client's SSE parser or reducer.
const streamHeartbeat = 20 * time.Second
```

The relay loop now selects between two cases, the events channel and the ticker:

```go
// internal/brain/api/api.go
ticker := time.NewTicker(streamHeartbeat)
defer ticker.Stop()
for {
    select {
    case ev, ok := <-events:
        if !ok {
            send(m)
            return
        }
        // ... forward ev to the client ...
    case <-ticker.C:
        _, _ = fmt.Fprintf(w, ": ping\n\n")
        flusher.Flush()
    }
}
```

A `: ping` line is an SSE comment. It has no event name, so the browser's `EventSource` parser discards it, and Timothy's web client reducer never sees it surface as an event. It exists purely to put bytes on the wire often enough that the proxy stops counting the gap. The same ticker got wired into the `/live` re-attach stream, because a client that reconnects mid-turn needs the same protection.

The second half of the fix is on the client. A dropped connection is no longer a hard failure. Because a Timothy turn runs detached from any single HTTP request (D-042, from Part 21), it is very likely still running server-side when the network blips or the laptop sleeps. So the web client reattaches:

```tsx
// web/src/pages/Chat.tsx
// A transport-level failure (dropped connection, proxy timeout,
// laptop sleep) throws a plain error, not ChatError: the turn
// itself runs detached from this request (D-042), so it's very
// likely still alive server-side. Reattach via /live instead of
// surfacing a false "failed" state.
toast.info('Connection dropped - reattaching')
handedOffToLive = true
attachLive(sessionRef.current)
```

Only a plain `TypeError` (the transport got cut) triggers this path. A definite `ChatError`, a real server response with a status code, still falls back to a transcript refetch, because that's a genuine failure, not a network event. `attachLive` itself caps retries at `maxLiveReconnects = 5` so a turn that is genuinely gone doesn't loop the reattach forever.

This was the only chapter where the enemy was outside the system. Once turns could survive 30 minutes, they exposed everything else.

## Chapters 2 and 3: the provider and the gateway that forgot to tell the client

The heartbeat kept the proxy from killing the connection. But a slow model on a CPU-only box could still drop on its own, when the remote Ollama process hiccuped or the host ran out of memory. And when it did, the next two holes in the contract opened up, one at the provider layer and one at the gateway.

The provider layer has a small helper called `emit` that every HTTP driver uses to push normalized events onto its channel:

```go
// internal/gateway/provider/httpx.go
// emit sends an event unless ctx is done; it reports whether the send
// happened. Streams must never block forever on an abandoned consumer.
func emit(ctx context.Context, ch chan<- stream.StreamEvent, ev stream.StreamEvent) bool {
    select {
    case ch <- ev:
        return true
    case <-ctx.Done():
        return false
    }
}
```

That `ctx.Done()` case is the right instinct for most events. A stream whose consumer has gone away should not block forever on a send nobody will read. But it is the wrong instinct for the terminal error that `runStream` emits when the upstream connection dies, because that terminal is the only evidence the attempt failed. When the upstream connection (remote Ollama) drops at nearly the same instant the parent context cancels, the terminal-error emit loses its race against `ctx.Done()` and returns `false`. The provider channel closes having delivered zero events. No chunk. No done. No error. A well-formed empty SSE response, from the gateway's point of view.

The gateway's `streamAttempt` then ranged over that channel with a `for ev := range ch` loop, and the loop body never ran once. The local result stayed all zero values. Nothing was sent to the client. The result was indistinguishable from a legitimate empty completion, so nothing failed over and nothing surfaced. Brain fell back to a generic "stream ended without a terminal event" message that told nobody anything.

The fix in #156 tracks whether any terminal was ever seen:

```go
// internal/gateway/api/api.go
sawTerminal := false
for ev := range ch {
    switch ev.Type {
    case stream.EventError:
        sawTerminal = true
        // ...
    case stream.EventIncomplete:
        sawTerminal = true
        // ...
    case stream.EventDone:
        sawTerminal = true
        // ...
    }
}

// A provider channel can close having delivered nothing at all (no
// chunk, no EventDone, no EventError) if runStream's own terminal
// emit lost its race against the parent context right as the
// upstream connection cut (httpx.go's emit silently no-ops when its
// ctx is done).
if !sawTerminal && !res.failed {
    res.failed, res.reason = true, "provider stream closed without a terminal event"
    res.entry.ErrorCode = "stream_cut"
}
```

With `res.failed = true` and nothing having streamed, the chain can fail over to the next provider. The ledger records `stream_cut`. The client gets a reason.

But #156 only mutated `streamAttempt`'s local result. That's correct when no content reached the client, because failover handles it from there. The problem is what happens once content already streamed. Once bytes have crossed the wire to the client, failover is off the table (the gateway's own bright line: errors after content are relayed honestly, never silently restarted elsewhere). In that case #156 recorded `stream_cut` in the ledger correctly, but the SSE stream just closed after the last chunk with no terminal frame sent to the client at all.

Brain's `persistTurn` knows how to handle a real `EventError` frame. It just never got one. A mid-stream cut against a real 30-minute attempt left zero `session_events` for the turn: no assistant message, no error, `turn_active` silently flipped false. The turn ceased to exist in the durable record.

That was #157. The fix mirrors the existing `EventError` case's streamed-vs-not-streamed split. If content already streamed, send an explicit terminal error frame:

```go
// internal/gateway/api/api.go
if !sawTerminal && !res.failed {
    res.failed, res.reason = true, "provider stream closed without a terminal event"
    res.entry.ErrorCode = "stream_cut"
    if res.streamed {
        send(stream.StreamEvent{Type: stream.EventError, Err: &stream.StreamError{
            Code:    "stream_cut",
            Message: res.reason,
        }})
    }
}
```

The gateway now tells the client when a stream cuts mid-flight, whether or not bytes have crossed. That left the brain-side hole.

## Chapters 4 and 5: the brain-side race, and the root cause underneath

Brain's gateway client, `gwclient`, had its own synthetic terminal for exactly this case. If the HTTP read failed without a real terminal having arrived, it would synthesize a `gateway_stream_cut` error event and push it onto the channel. The persistence path would then have something to record.

The guard on that synthesis was the problem:

```go
// internal/brain/gwclient/gwclient.go (before #158)
if err != nil && !sawTerminal && ctx.Err() == nil {
```

The context passed into `gwclient.Stream` is `turnCtx`, brain's own 30-minute turn ceiling. When a real long turn's deadline expires at nearly the same instant as a genuine upstream failure, `ctx.Err()` is already non-nil by the time the read error surfaces. So the guard skipped the fallback entirely. `persistTurn` had nothing. The turn vanished.

This is the exact race the synthetic terminal existed to catch, and the guard was skipping it on the exact condition that made the race fire.

#158 dropped the `ctx.Err() == nil` condition, so the synthesis always attempts delivery. It still raced the send against `ctx.Done()` so it would never block forever on an abandoned consumer:

```go
// internal/brain/gwclient/gwclient.go (#158)
if err != nil && !sawTerminal {
    select {
    case ch <- stream.StreamEvent{Type: stream.EventError, Err: &stream.StreamError{
        Code: "gateway_stream_cut", Message: err.Error(), Retryable: true,
    }}:
    case <-ctx.Done():
    }
}
```

Here is the honest sharp edge. That fix's regression test failed once in CI despite passing locally. The failure was not a flake in the test. It was the test correctly reporting that the fix was nondeterministic. Once `ctx` is already done, the delivery `select` races `ch <-` against `ctx.Done()`. Both cases are ready. Go picks randomly. A single attempt can land either way depending on the scheduler.

The test was rewritten to run 40 independent attempts and require the terminal to win at least once:

```go
// internal/brain/gwclient/gwclient_test.go (#158)
// The fix's delivery races ch<- against ctx.Done() once ctx is
// already done when the read error surfaces (never blocking
// forever on an abandoned consumer is the whole point of racing
// against ctx.Done() at all), so a single attempt can't
// distinguish "fixed, lost this particular race" from "the old,
// unconditionally-skipped bug". Run many independent attempts and
// require the terminal to win at least once.
const attempts = 40
delivered := 0
for i := 0; i < attempts; i++ {
    // ... set up gateway stub, cancel ctx, close release, count deliveries ...
}
if delivered == 0 {
    t.Fatalf("gateway_stream_cut never delivered across %d attempts, want at least one", attempts)
}
```

Confirmed against a real 30-minute turn: gateway logged completion with `error_code=provider_error` right after the gateway-side fixes landed, yet still produced no session event. The brain-side synthetic terminal was getting dropped on the floor. And the probabilistic test was the warning sign that #158 was not the real fix, just a closer approximation of one.

The real root cause was that four different producers treated end-of-turn delivery as a best-effort `select` racing `ctx.Done()`. When the 30-minute turn ceiling and a slow provider's `request_timeout` (D-041) landed at the same instant, both `select` cases were ready, Go picked randomly, and a lost coin-flip dropped the only evidence a turn existed. #159 is the commit that stopped racing.

The same design, applied at every site:

```go
// internal/brain/gwclient/gwclient.go (#159)
// Delivery deliberately does NOT race ctx.Done(): with both select
// cases ready (deadline fired, consumer mid-receive) Go picks
// randomly, and that coin flip has eaten the only evidence of a real
// ~30min turn. Every consumer of this channel drains until close, so
// the send succeeds immediately in practice; the timeout only bounds
// a hypothetically abandoned consumer.
if err != nil && !sawTerminal {
    select {
    case ch <- stream.StreamEvent{Type: stream.EventError, Err: &stream.StreamError{
        Code: "gateway_stream_cut", Message: err.Error(), Retryable: true,
    }}:
    case <-time.After(5 * time.Second):
    }
}
```

The 5-second bound replaces `ctx.Done()`. The justification is an invariant: every consumer of this channel drains until close, so the send succeeds immediately in practice. The timeout only exists to bound a hypothetically abandoned consumer that should not exist. That is a deliberate trade. A genuinely wedged consumer holds a goroutine for 5 seconds longer than the old `ctx.Done()` race would have. In return, a frequently-eaten terminal stops getting eaten. The test reverted to a single deterministic assertion, because the fix itself was deterministic again.

The agent loop got the same treatment. A new closure, `emitFinal`, delivers the usage-and-terminal pair without racing `ctx.Done()`:

```go
// internal/brain/loop/agent.go
emit := func(ev stream.StreamEvent) {
    emitMu.Lock()
    defer emitMu.Unlock()
    select {
    case out <- ev:
    case <-ctx.Done():     // the race that ate terminals
    }
}
// emitFinal delivers end-of-turn events (the usage/terminal pair)
// without racing ctx.Done(): when the turn deadline fires at the
// same instant the terminal is sent, both select cases in emit are
// ready and Go picks randomly; a lost flip drops the only evidence
// the turn ever existed.
emitFinal := func(ev stream.StreamEvent) {
    emitMu.Lock()
    defer emitMu.Unlock()
    select {
    case out <- ev:
    case <-time.After(5 * time.Second):
    }
}
```

All four terminal-producing exit points in the loop, gateway failure, normal completion, force-synthesis, and max-iterations, now route through `emitFinal`. The loop's synthesized `EventIncomplete` can no longer be coin-flipped away.

And then the structural backstop, in `persistTurn`. If every upstream producer lost its terminal and there is no text and no failure, synthesize one:

```go
// internal/brain/chat/chat.go
// No failure AND no text at all means every upstream producer lost
// its terminal on the way here (deadline racing the stream cut).
// Synthesize one rather than append nothing: a turn that ran must
// never leave zero events. A flushed partial (text non-empty,
// already checkpointed) keeps today's behavior: the pending state
// is the evidence.
if failure == nil && text == "" {
    failure = &session.TurnFailed{
        Code:    "turn_aborted",
        Message: "turn ended with no terminal event (deadline exceeded or stream cut)",
    }
}
```

Zero-evidence turns are now structurally impossible. Even if every producer above this point loses its race, the persistence path itself refuses to append nothing.

The missions runner got the same `sawTerminal` discipline. A bare channel close is now an infra error instead of a clean empty verdict. Before this, a silent cut read as a missing sentinel, and the harness burned a recovery re-run plus a forced retry: two full model turns for one silent infrastructure failure. `EventIncomplete` is now an infra error too, not a short clean answer. And an `EventError` with a nil `Err` no longer panics (the runner used to dereference `ev.Err.Message` unconditionally; chat's `noteFailure` already guarded this, the runner didn't).

Two more fixes shipped in the same commit because they shared a failure mode with the terminal-loss class: a single bad event silently killing a turn.

The loop's `executeOne` now recovers a panicking tool into an error result for the model, rather than letting it crash the process:

```go
// internal/brain/loop/agent.go
// A panicking tool must become feedback the model can read (D-009),
// never a process crash: executeOne runs inside an errgroup worker,
// and an unrecovered panic there takes down the whole brain, every
// active turn and mission with it.
content, status := func() (content, status string) {
    defer func() {
        if p := recover(); p != nil {
            a.logger.Error("tool panicked", "tool", call.Name, "panic", p, "stack", string(debug.Stack()))
            content, status = fmt.Sprintf("tool %s failed with an internal error: %v", call.Name, p), "error"
        }
    }()
    return a.resolveAndRun(ctx, exec, sessionID, call, toolNames, unattended, emit)
}()
```

That one is unrelated to streaming. It shares the shape: a single unexpected event takes down the whole turn. An errgroup worker panic does not just kill its own turn, it takes the whole brain with it.

And `TURN_TIMEOUT` is now an env-gated override on the compiled 30-minute ceiling, with a floor:

```go
// cmd/brain/main.go
// TURN_TIMEOUT raises the detached-turn ceiling above the compiled
// 30m default, needed when a route serves a slow CPU-only backend
// whose provider request_timeout (D-041) would otherwise collide
// with the brain ceiling and manufacture a deadline-vs-terminal
// race at the exact same instant. Env-gated feature, default off.
if v := os.Getenv("TURN_TIMEOUT"); v != "" {
    d, err := time.ParseDuration(v)
    if err != nil {
        app.Log.Warn("invalid TURN_TIMEOUT ignored", "value", v, "error", err)
    } else if err := svc.SetTurnTimeout(d); err != nil {
        app.Log.Warn("TURN_TIMEOUT rejected", "value", v, "error", err)
    } else {
        app.Log.Info("turn timeout overridden", "timeout", d)
    }
}
```

The floor is enforced in `SetTurnTimeout`: values at or below 10 minutes are rejected, because 10 minutes is the permission-park timeout, the window a turn spends waiting on a user's yes/no on a tool call. A ceiling at or below that floor means a parked ask could never be answered in time. This is opt-in, not on by default, because raising the global turn ceiling has a real cost: longer-lived detached goroutines and a larger memory hold per active turn. It exists for the slow CPU-only backend whose `request_timeout` keeps colliding with the turn ceiling, attacking the root cause directly instead of patching the race it produces.

## The recurring shape

This is the same species of bug the series keeps running into, last seen in [Part 22](https://www.sumonselim.com/articles/timothy-part22-connector-sensitivity-and-a-same-day-bug): a mechanism has more than one enforcement point, the layers live in different packages, and nothing forces them to be touched together.

The contract in `events.go` is one sentence. Defending it took five commits across four packages: the provider helper, the gateway stream attempt, the brain gateway client, the agent loop, the chat persistence path, the missions runner. Each layer could be closed individually, and each individual closure was correct, but the contract only held once every layer was closed. The gateway fix in #157 was correct on its own terms. It still left the brain-side hole in #158 wide open, because the gateway recording `stream_cut` in its ledger does not help a `persistTurn` that never receives an error frame.

The test harness itself has a small defense against this class regressing silently. The `scriptedAgent` fake in `runner_test.go` used to let any batch of events close the channel bare. It now auto-appends an `EventDone` to any batch that doesn't already end on a terminal, because the real `loop.Agent` guarantees exactly one terminal before close. Only the terminal-loss regression tests opt out, with `rawClose: true`. Every other test gets the contract enforced by the fake.

## What the chain cost, and what it gained

The honest accounting. The 5-second bounded send trades a rare stuck goroutine for a frequently-eaten terminal. A genuinely wedged consumer, which by the drain-until-close invariant should not exist, holds a goroutine 5 seconds longer than the old `ctx.Done()` race would have. That is the cost. It is deliberate.

The `turn_aborted` synthesis in `persistTurn` is a structural safety net. The user always sees something. But it is the last backstop in a chain of four producers, and if it fires at any frequency, an upstream terminal-delivery path has regressed. The code comment says so plainly: it exists because every upstream producer lost its terminal on the way here. It is not license to stop defending the producers above it.

`TURN_TIMEOUT` is opt-in, not a default, because raising the global turn ceiling has real cost in goroutine lifetime and memory hold.

And #158 was nondeterministic by design. Its probabilistic test was a warning sign that the root cause was still one commit away. #159 made the fix deterministic by making the code deterministic. The test reversion, from 40 attempts to a single assertion, is the proof that the underlying race was gone.

What the system gained is simple to say and took five commits to make true. A turn that ran must always leave a trace. The contract in `events.go`, exactly one terminal event per stream, is now defended at every layer it crosses. The original symptom, a 30-minute turn vanishing with zero `session_events`, is structurally impossible: every producer that could drop the terminal has been given a bounded delivery path, and the persistence path itself refuses to append nothing as a last resort.

The durable record was always the point. The `session_events` log from Part 6 is only worth anything if it is a complete record, and a complete record is one where a turn that ran is a turn that appears.

---

*Landed across #155 through #159, `214c529` through `4cbd3eb` on `main`.*