Except there was a gap. Every streaming call cost $0.00 in the ledger. Silently. No error, no warning – just wrong numbers.

I found the gap, fixed it, and got the PR merged. This is the story of how streaming cost tracking actually works.

What AgentBudget does

AgentBudget is an open-source SDK that puts a hard dollar limit on AI agent sessions. Two lines, no infrastructure:

1import agentbudget
2agentbudget.init("$5.00")

It works by monkey-patching Completions.create (OpenAI) and Messages.create (Anthropic) at init time. Every call goes through a wrapper that looks up the model cost, accumulates spend, and raises BudgetExhausted when the limit is hit.

The non-streaming path worked fine. The streaming path did not.

The bug

When stream=True, the patched method receives a Stream object back from the SDK – not a completed response with token counts. The original code had no handling for this case. It silently passed the stream through untracked.

Every streaming call registered as $0.00 in the ledger. Any agent using streaming could blow past its budget with zero resistance. The circuit breaker was blind to half its traffic.

Why streaming is harder to cost

Non-streaming is straightforward: the response comes back with a usage object containing prompt_tokens and completion_tokens. Multiply by the per-token rate, done.

Streaming returns a sequence of chunks. Token counts are not available upfront – they only appear after the stream is exhausted. For OpenAI specifically, usage data on the final chunk is opt-in: you need stream_options={"include_usage": True} in the request, otherwise the final chunk carries no usage data and there is nothing to cost.

Anthropic handles it differently. Usage data is always present, split across two events: message_start carries input tokens, message_delta carries output tokens at the end.

The fix: transparent wrapper classes

The solution is four wrapper classes that intercept the stream, yield every chunk unchanged, and record cost after the iterator is exhausted:

Each wrapper is fully transparent to the caller. The existing code pattern – for chunk in stream: or async for chunk in stream: or with client.stream(...) as stream: – continues to work without any changes.

The wrapper accumulates token counts as chunks arrive, then records the cost in the session ledger once the stream ends.

For OpenAI:

 1class _OpenAIStreamWrapper:
 2    def __init__(self, stream, session, model):
 3        self._stream = stream
 4        self._session = session
 5        self._model = model
 6
 7    def __iter__(self):
 8        for chunk in self._stream:
 9            # Track usage from the final chunk
10            if chunk.usage:
11                cost = price(self._model, chunk.usage)
12                self._session.record(cost)
13            yield chunk
14
15    # Full protocol: __enter__, __exit__, close()

For Anthropic, the accumulation spans two event types:

 1class _AnthropicStreamWrapper:
 2    def __iter__(self):
 3        input_tokens = 0
 4        output_tokens = 0
 5        for event in self._stream:
 6            if event.type == "message_start":
 7                input_tokens = event.message.usage.input_tokens
 8            elif event.type == "message_delta":
 9                output_tokens = event.usage.output_tokens
10            yield event
11        cost = price(self._model, input_tokens, output_tokens)
12        self._session.record(cost)

The detection of whether a response is a stream happens via isinstance checks guarded by ImportError, so the wrapper degrades gracefully if only one SDK is installed.

Implementing the full protocol

The tricky part is not the cost tracking logic – it is making the wrapper implement the full stream protocol correctly.

Python streams support multiple usage patterns: iterator (for chunk in stream), context manager (with stream), async iterator, async context manager, and explicit close()/aclose(). Missing any of these means certain callers will break with an AttributeError or a resource leak.

The wrapper needs to implement all of them:

 1# Sync: iterator + context manager + close
 2def __iter__(self): ...
 3def __enter__(self): return self
 4def __exit__(self, *args): self.close()
 5def close(self): self._stream.close()
 6
 7# Async: async iterator + async context manager + aclose
 8async def __aiter__(self): ...
 9async def __aenter__(self): return self
10async def __aexit__(self, *args): await self.aclose()
11async def aclose(self): await self._stream.aclose()

Getting this wrong is invisible in unit tests with mocks but breaks immediately against real SDK objects.

Tests

The PR added 19 tests covering:

• Cost recorded correctly after stream exhaustion
• Chunks with no usage data skipped without error
• Passthrough identity – every chunk the caller sees is unchanged
• BudgetExhausted propagates correctly mid-stream
• Both for chunk in stream: and with client.stream() as stream: patterns
• close() and aclose() delegation to the underlying stream
• Drop-in mode (agentbudget.init()) wrapping end-to-end
• No-session noop – if no budget session is active, streaming is fully transparent

All 129 pre-existing tests continued to pass. 148 total passing at merge.

The known limitation

OpenAI streaming requires the caller to pass stream_options={"include_usage": True}. Without it, the final chunk carries no usage data, and the cost is silently $0.

This is OpenAI’s API behaviour – usage data on streams is opt-in. The limitation is documented in the module docstring. A future improvement could auto-inject the option at the patch layer, but that requires modifying the outbound request, which is a bigger change than this PR intended to be.

Anthropic has no equivalent limitation – usage is always present.

What I learned

Transparent wrappers need the full protocol. It is tempting to implement just __iter__ and call it done. But real callers use context managers, explicit close, and async variants. Implementing the minimum breaks production code in ways that are hard to debug.

Streaming costs are provider-specific. OpenAI and Anthropic have different conventions for where usage data appears in the stream. Any SDK that claims to support both needs to handle both separately.

Test against real SDK objects, not just mocks. The isinstance detection and chunk structure only behave correctly against live SDK classes. Mock-based tests passed for the wrong reasons until I verified against an actual installed SDK.

Links

• PR #6 – feat: track costs for streaming OpenAI and Anthropic responses
• AgentBudget on GitHub
• PyPI

Except it doesn’t. It concatenates them all into one call: search_documentsget_usersearch_documents. The agent hallucinates, retries, and loops. I stared at the telemetry for twenty minutes before I realized the bug wasn’t in the agent — it was in the gateway’s streaming accumulator.

This is the story of how I found and fixed a parallel tool call merging bug in Bifrost, a 2.6k-star open-source AI gateway.

What is Bifrost?

Bifrost is a high-performance AI gateway written in Go. It sits between your application and LLM providers (OpenAI, Anthropic, Bedrock, Vertex, etc.), providing a unified OpenAI-compatible API with automatic failover, load balancing, semantic caching, and streaming support. Think of it as an intelligent reverse proxy for AI — one API, fifteen providers, ~11µs overhead at 5k RPS.

The relevant piece here is the streaming accumulator. When you make a streaming chat completion request, the LLM sends back a sequence of Server-Sent Events (SSE), each containing a small delta — a few tokens of text, or a fragment of a tool call’s arguments. The accumulator’s job is to reassemble these deltas into complete messages.

The bug: recency-based routing

Here’s the scenario. You ask Claude or GPT to call two tools in parallel:

User: "Search for compliance circulars about RBI and also get the user's profile"

The model decides to call both search_circulars and get_user_profile simultaneously. During streaming, the provider sends interleaved deltas:

Delta 1: tool_call[0] name="search" (index=0, id=call_abc)
Delta 2: tool_call[1] name="get_us" (index=1, id=call_xyz)
Delta 3: tool_call[0] args='{"query": "RBI'   (index=0)
Delta 4: tool_call[1] args='{"user_id": "u'   (index=1)
Delta 5: tool_call[0] args=' circulars"}'      (index=0)
Delta 6: tool_call[1] args='123"}'             (index=1)

Each delta carries an index (and optionally an id) that identifies which tool call it belongs to. The expected behavior: two distinct tool calls with their own names and arguments.

The actual behavior in Bifrost’s accumulator before the fix:

1// Old logic (simplified)
2if deltaToolCall.Function.Name != nil {
3    // New tool call — create it
4    existingToolCalls = append(existingToolCalls, newToolCall)
5} else {
6    // Argument delta — append to the LAST tool call
7    toolCallToModify = &existingToolCalls[len(existingToolCalls)-1]
8    toolCallToModify.Function.Arguments += deltaToolCall.Function.Arguments
9}

The routing logic was based on recency: if a delta has a Name, it’s a new tool call; otherwise, append arguments to the most recent one. This works perfectly when tool calls arrive sequentially. It breaks completely when they’re interleaved.

With the deltas above:

• Delta 3 (index=0 args) arrives → appended to the last tool call, which is get_user_profile (index=1). Wrong target.
• Delta 5 (index=0 args) arrives → again appended to get_user_profile. Now get_user_profile has arguments from two different calls mashed together.

The result: concatenated tool names, merged arguments, and an agent stuck in a retry loop trying to invoke search_circularsget_user_profile — a tool that doesn’t exist.

Finding the bug

I hit this in production. Our compliance platform uses an AI agent that calls multiple tools in parallel — searching circulars, fetching regulatory data, looking up user context. The agent kept failing with tool resolution errors, and the telemetry showed impossible tool names:

search_circularsearch_circularsearch_circular

At first I suspected the agent’s tool-calling logic. Then I looked at the raw SSE stream from the provider — the deltas were correct. Each one had the right index and id. The corruption was happening between the provider response and the final accumulated message.

I traced it to accumulateToolCallsInMessage in framework/streaming/accumulator.go. The function had no concept of tool call identity. It used the presence of a Name field as a heuristic for “new vs. continuation,” which is fragile and incorrect when deltas arrive out of order.

The fix

The fix is conceptually simple: route deltas by identity, not by recency.

Each tool call delta carries two potential identifiers:

1. tool_call.id — a string like call_abc123 (set by OpenAI/Anthropic)
2. tool_call.index — an integer position (0, 1, 2, …)

The new logic builds lookup maps on existing tool calls and uses them to route each incoming delta to the correct target:

1// Build lookup maps for identity-based routing
2idToIndex := make(map[string]int, len(existingToolCalls))
3indexToIndex := make(map[uint16]int, len(existingToolCalls))
4for i, toolCall := range existingToolCalls {
5    if toolCall.ID != nil && *toolCall.ID != "" {
6        idToIndex[*toolCall.ID] = i
7    }
8    indexToIndex[toolCall.Index] = i
9}

For each incoming delta, routing follows a priority chain:

 1for _, deltaToolCall := range deltaToolCalls {
 2    index := -1
 3
 4    // Priority 1: match by tool_call.id
 5    if deltaToolCall.ID != nil && *deltaToolCall.ID != "" {
 6        if existingIndex, ok := idToIndex[*deltaToolCall.ID]; ok {
 7            index = existingIndex
 8        }
 9    }
10
11    // Priority 2: match by tool_call.index
12    if index == -1 {
13        if existingIndex, ok := indexToIndex[deltaToolCall.Index]; ok {
14            index = existingIndex
15        }
16    }
17
18    // Priority 3: no match → create new tool call (if name present)
19    if index == -1 {
20        if deltaToolCall.Function.Name == nil {
21            // Can't create a tool call without a name
22            continue
23        }
24        newToolCall := schemas.ChatAssistantMessageToolCall{
25            Index:    deltaToolCall.Index,
26            ID:       deltaToolCall.ID,
27            Function: /* ... */,
28        }
29        existingToolCalls = append(existingToolCalls, newToolCall)
30        // Update maps for future deltas
31        indexToIndex[newToolCall.Index] = len(existingToolCalls) - 1
32        if newToolCall.ID != nil {
33            idToIndex[*newToolCall.ID] = len(existingToolCalls) - 1
34        }
35        continue
36    }
37
38    // Append arguments to the correct tool call
39    existingToolCalls[index].Function.Arguments += deltaToolCall.Function.Arguments
40}

The key insight is that the maps are updated as new tool calls are created, so subsequent deltas for the same call always find their target — even if they arrive many chunks later.

The same bug, twice

During code review, a reviewer (Pratham-Mishra04) pointed out that the same recency-based routing existed in a second code path: the Responses API streaming handler in responses.go.

The Responses API is OpenAI’s newer streaming format (the /v1/responses endpoint). It uses ItemID instead of tool_call.index, but the accumulation logic had the same flaw — FunctionCallArgumentsDelta events were always appended to the last message:

1// Old responses.go logic
2if resp.Delta != nil && len(messages) > 0 {
3    // Always targets the last message — wrong for parallel calls
4    a.appendFunctionArgumentsDeltaToResponsesMessage(
5        &messages[len(messages)-1], *resp.Delta)
6}

The fix follows the same pattern: search for a message matching resp.ItemID, and only fall back to the last message when no ItemID is present. If an ItemID exists but no message matches, create a stub message to hold the delta rather than merging it into the wrong call:

 1if resp.Delta != nil && len(messages) > 0 {
 2    targetIdx := len(messages) - 1
 3    if resp.ItemID != nil {
 4        targetIdx = -1
 5        for i := len(messages) - 1; i >= 0; i-- {
 6            if messages[i].ID != nil && *messages[i].ID == *resp.ItemID {
 7                targetIdx = i
 8                break
 9            }
10        }
11        if targetIdx == -1 {
12            // Create stub to avoid merging into wrong message
13            id := *resp.ItemID
14            messages = append(messages, schemas.ResponsesMessage{ID: &id})
15            targetIdx = len(messages) - 1
16        }
17    }
18    a.appendFunctionArgumentsDeltaToResponsesMessage(
19        &messages[targetIdx], *resp.Delta)
20}

Testing parallel accumulation

The regression test simulates the exact interleaving pattern that triggers the bug:

 1func TestAccumulateToolCallsInterleavedParallel(t *testing.T) {
 2    // ...
 3    toolCallID0 := "call_0"
 4    toolCallID1 := "call_1"
 5    toolNameAdd := "add"
 6    toolNameMultiply := "multiply"
 7
 8    // Simulate interleaved deltas
 9    accumulate(makeDelta(0, &toolCallID0, &toolNameAdd, ""))       // create call 0
10    accumulate(makeDelta(1, &toolCallID1, &toolNameMultiply, ""))  // create call 1
11    accumulate(makeDelta(0, nil, nil, `{"a": 1`))                  // args for call 0
12    accumulate(makeDelta(1, nil, nil, `{"a": 2`))                  // args for call 1
13    accumulate(makeDelta(0, nil, nil, `, "b": 3}`))                // more args for call 0
14    accumulate(makeDelta(1, nil, nil, `, "b": 4}`))                // more args for call 1
15
16    // Assert: two distinct tool calls with correct arguments
17    assert(addCall.Arguments == `{"a": 1, "b": 3}`)
18    assert(multiplyCall.Arguments == `{"a": 2, "b": 4}`)
19}

Without the fix, add would end up with multiply’s arguments concatenated in. With the fix, each call accumulates only its own deltas.

A second test covers the Responses API path with ItemID-based routing, using the same interleaving pattern but with OutputItemAdded + FunctionCallArgumentsDelta events.

The PR lifecycle

I filed Issue #1829 with reproduction details from production telemetry, then opened PR #1830 with the fix and tests.

The review process was smooth:

1. First commit: fix the Chat Completions accumulator + regression test
2. Reviewer feedback: same bug exists in Responses API — please fix there too + add changelog entries
3. Second commit: fix Responses API routing + regression test + changelogs
4. Merged by akshaydeo (Bifrost maintainer)

Total: 3 commits, ~280 lines of Go across 5 files. Same-day turnaround from issue to merge.

What I learned

1. Recency is not identity

The original code used position (last element) as a proxy for identity. This is a common pattern in sequential streaming — and it works, until the stream isn’t sequential. The fix is to use the actual identity field that the protocol already provides.

This is a general principle: if the protocol gives you an ID field, use it. Don’t rely on ordering assumptions that hold today but break when concurrency enters.

2. The same bug pattern repeats across API surfaces

Bifrost supports two streaming formats: Chat Completions (the classic tool_calls array) and Responses API (the newer ItemID-based format). Both had the same routing flaw because the accumulation logic was duplicated rather than shared. The reviewer catching this in a different file was the most valuable part of the review.

3. OSS contributions are about context, not complexity

The fix itself is straightforward — build a map, use it for lookup. The hard part was:

• Finding the bug in a codebase I’d never seen before
• Understanding why the existing approach was wrong
• Testing with realistic interleaved deltas
• Responding to review feedback about a parallel code path I hadn’t initially noticed

The code change was ~100 lines. The context-building was hours.

4. Production telemetry is the best bug report

The issue report included the exact malformed tool name from production: search_circularsearch_circularsearch_circular. That single string made the bug immediately reproducible and the fix trivially verifiable. If you’re filing a bug, include the malformed output — it’s worth more than a paragraph of description.

Links

• PR: fix: preserve parallel tool call deltas — #1830
• Issue: Streaming tool call deltas merge in parallel tool calling — #1829
• Project: Bifrost AI Gateway — 2.6k ⭐, fastest enterprise AI gateway