# Timothy Part 3: The LLM Gateway, Providers as Data, Streaming, and a Cost Ledger > Source: https://www.sumonselim.com/timothy-part3-the-llm-gateway-providers-as-data-streaming-cost-ledger > Author: Muhammad Sumon Molla Selim > Published: 2026-07-10 > Tag: Timothy > Summary: The gateway is the only component that ever touches an API key or subscription. I built it to treat providers as rows in Postgres, normalize every response into one event stream, and record a receipt for every token in a cost ledger, even when things go wrong. The first request that ever flowed through Timothy's gateway cost me nothing, not because it was free, but because the ledger refused to guess. **Why a cost ledger that records NULL rather than guessing prices?** GLM-4.7 took 11 input tokens, answered with 204 output tokens in 11,988 ms, and the cost column stayed `NULL`: I had not configured prices yet, and a ledger that invents numbers is worse than no ledger (D-013). Once prices landed, the next short chat turn recorded $0.000873. That is the level of accounting I wanted from day one. In [Part 2](https://www.sumonselim.com/articles/timothy-part2-the-architecture-three-go-services-and-a-rule) I laid out the architecture and the rule that shapes everything: deterministic code orchestrates; each LLM session does exactly one thing. The first real code I wrote was the **gateway**. It is the only service that ever sees a credential. Everything else (the brain, memoryd, clients) talks to it over a clean internal interface and never learns an API key or how to call Anthropic versus xAI. ## Providers Are Data, Not Code **Why store providers, models, and routing as data in Postgres instead of hard-coded in Go?** I did not want a new Go file every time I added a model or a provider (D-004). Instead, providers live in Postgres. ```sql -- migrations/0002_gateway.sql CREATE TABLE IF NOT EXISTS providers ( id uuid PRIMARY KEY DEFAULT gen_random_uuid(), name text UNIQUE NOT NULL, kind text NOT NULL CHECK (kind IN ('api', 'cli')), driver text NOT NULL, base_url text NOT NULL DEFAULT '', ... models jsonb NOT NULL DEFAULT '[]', credential_ref text NOT NULL DEFAULT '', enabled boolean NOT NULL DEFAULT false, ... ); ``` A row for Anthropic looks like this at seed time: - driver = "anthropic" - credential_ref = "ANTHROPIC_API_KEY" - models = array of objects with id, capabilities, and prices (USD per million tokens) The same table holds Z.AI and xAI as "openaicompat" drivers with their base URLs and different price structures. CLI drivers (actual `claude -p` or similar) will land in the same table later. At startup the gateway reads the enabled rows, resolves each `credential_ref` against the environment (or later Vault), and builds a live `Registry`. If the ref is empty, the provider is marked unhealthy and routing skips it automatically. ## One Normalized Stream for Every Provider **Why normalize every provider into a single event vocabulary instead of leaking their native formats?** Different providers speak different protocols. Anthropic has its own SSE shape, tool-use blocks, and prompt-caching headers. OpenAI-compatible endpoints are close but never identical (D-013). I refused to let that leak into the rest of the system. ```go // internal/gateway/stream/events.go const ( EventChunk EventType = "chunk" EventReasoningChunk EventType = "reasoning_chunk" EventToolStart EventType = "tool_start" EventToolEnd EventType = "tool_end" EventUsage EventType = "usage" EventRetry EventType = "retry" EventIncomplete EventType = "incomplete" EventDone EventType = "done" EventError EventType = "error" ) type StreamEvent struct { Type EventType Text string ToolCall *ToolCallEvent Usage *Usage Err *StreamError Retry *RetryInfo Meta *Meta // attached on the terminal "done" } ``` The nine event types cover everything a consumer needs to render honestly: | Event | Meaning | | :-- | :-- | | `chunk` / `reasoning_chunk` | assistant text / thinking text | | `tool_start` / `tool_end` | a tool call opened / its arguments completed | | `usage` | provider-reported token accounting | | `retry` | transient failure, driver retrying, shown, not hidden | | `incomplete` | the stream was cut or truncated before finishing | | `done` / `error` | exactly one of these terminates every stream | Every driver, anthropic or openaicompat, must turn its native output into this vocabulary and obey one strict contract: > The channel ends with **exactly one** terminal event ("done" or "error") and then closes. No second terminal. No "done" followed by more chunks. The brain and the web client can rely on it. Inside the drivers I added real hardening: - Hard per-request timeout that cancels the wire call. - Retries on 429/5xx/timeout, at most three, with exponential backoff plus jitter, surfaced as `retry` events so the caller sees the turbulence. - If the stream is cut after any content has been sent, the driver emits `incomplete` or `error` instead of silently retrying to another provider. - Only failures *before* the first byte allow failover down the chain. ## Data-Driven Routing and Failover A request arrives with a `task_category` (reasoning, coding, mini, summarize, realtime, embedding) and an optional `model_hint`. The gateway looks up the enabled chain for that category: ```json // example chain for "coding" [ {"provider_id": "...", "model": "claude-sonnet-5"}, {"provider_id": "...", "model": "glm-4.7"} ] ``` It walks the chain in order. For each entry it checks: - The provider is enabled and healthy (credential resolved and non-empty). - The model declares the required capability (chat or embeddings). The first usable attempt wins. If every candidate is skipped, the caller gets a structured `NoRouteError` that names every rejection reason. A `POST /internal/reload` or a 30-second background poll rebuilds the snapshot atomically. The old snapshot keeps serving in-flight requests while the new one takes over. ## Every Call Produces a Receipt I wanted a real cost ledger, not "we'll add metrics later." ```sql CREATE TABLE IF NOT EXISTS cost_ledger ( id uuid PRIMARY KEY, ts timestamptz NOT NULL DEFAULT now(), provider text NOT NULL, model text NOT NULL, task_category text NOT NULL, session_id text, ... input_tokens integer, output_tokens integer, cache_read_tokens integer, cache_write_tokens integer, latency_ms integer NOT NULL, status text NOT NULL, error_code text, cost_usd numeric(12, 6) ); ``` The gateway writes one row for every attempt, success or failure. Prices come from the `models` jsonb on the provider row. If a model has no prices, `cost_usd` is left null, never guessed. Cost is computed at request time and **frozen into the row**: when a provider changes prices later, history stays truthful. Two hard-won details live in this table. **Cached tokens mean different things per provider.** OpenAI-style APIs *include* cached tokens inside `prompt_tokens`; Anthropic reports cache reads separately from `input_tokens`. Add the two columns naively and you double-count every cache hit on OpenAI-compatible providers. The drivers normalize to the split representation before a row is ever written. **The zero-latency ledger.** A refactor of the streaming path returned a result struct by value while a deferred function stamped the latency onto the dead local copy, the classic Go `defer`-after-copy footgun. Every streamed request recorded `latency_ms = 0`. My tests asserted tokens, status, and cost; none asserted latency, so everything stayed green. I only caught it by querying the actual rows and reading `0` where 6-second GLM responses should have been. The fix was a named return value and a test against a deliberately slow fake, and a lesson I keep relearning: verify the running system, not the code. The ID can be pre-generated before the stream starts so the `meta` event that closes the SSE can include the ledger row identifier. Clients can show it immediately. ## The Internal API Surface The gateway exposes a narrow HTTP surface inside the compose network: - `POST /v1/stream`: the main one. Body carries task_category, optional model_hint, system prompt, messages, tools. Returns SSE of `StreamEvent` JSON frames. - `POST /v1/embed` - `GET /v1/providers`: current config plus health - `POST /internal/reload` No auth on these routes. The credential boundary is the gateway itself. ## What It Feels Like in Practice With the stack running and one provider enabled: ```bash # internal-only, from inside the compose network curl -N -X POST http://gateway:8081/v1/stream \ -H 'content-type: application/json' \ -d '{ "task_category": "mini", "messages": [{"role":"user","content":"Explain pgvector in one sentence."}] }' ``` You get a stream of frames: ``` data: {"type":"chunk","text":"pgvector turns Postgres into a vector database by adding..."}\n\n data: {"type":"usage","usage":{"input_tokens":18,"output_tokens":42}}\n\n data: {"type":"done","meta":{"provider":"anthropic","model":"claude-sonnet-5","ledger_id":"..."}}\n\n ``` A row appears in `cost_ledger` with the exact token counts the provider reported, the measured latency, and the computed cost based on the prices stored in the provider's model list. Disable the first provider in the chain via SQL, hit reload, and the next request automatically tries the fallback (or fails cleanly with the full list of skipped candidates). ## Why This Shape Treating providers as data means adding a new model or even a new cheap provider is a SQL insert plus a price update, no deploy. The normalized stream means the brain never contains "if anthropic" branches. The ledger means I have a receipt for every model call, which will matter the first time I look at the dashboard and think "wait, that much?" The gateway is deliberately boring infrastructure. That is the point. In the next post I wire the brain on top of it: bearer authentication for the public API, the pass-through chat that turns one user message into a gateway call, and the terminal `meta` event that tells the client which provider actually answered and how much it cost.