Except it doesn’t. It concatenates them all into one call: search_documentsget_usersearch_documents. The agent hallucinates, retries, and loops. I stared at the telemetry for twenty minutes before I realized the bug wasn’t in the agent — it was in the gateway’s streaming accumulator.

This is the story of how I found and fixed a parallel tool call merging bug in Bifrost, a 2.6k-star open-source AI gateway.

What is Bifrost?

Bifrost is a high-performance AI gateway written in Go. It sits between your application and LLM providers (OpenAI, Anthropic, Bedrock, Vertex, etc.), providing a unified OpenAI-compatible API with automatic failover, load balancing, semantic caching, and streaming support. Think of it as an intelligent reverse proxy for AI — one API, fifteen providers, ~11µs overhead at 5k RPS.

The relevant piece here is the streaming accumulator. When you make a streaming chat completion request, the LLM sends back a sequence of Server-Sent Events (SSE), each containing a small delta — a few tokens of text, or a fragment of a tool call’s arguments. The accumulator’s job is to reassemble these deltas into complete messages.

The bug: recency-based routing

Here’s the scenario. You ask Claude or GPT to call two tools in parallel:

User: "Search for compliance circulars about RBI and also get the user's profile"

The model decides to call both search_circulars and get_user_profile simultaneously. During streaming, the provider sends interleaved deltas:

Delta 1: tool_call[0] name="search" (index=0, id=call_abc)
Delta 2: tool_call[1] name="get_us" (index=1, id=call_xyz)
Delta 3: tool_call[0] args='{"query": "RBI'   (index=0)
Delta 4: tool_call[1] args='{"user_id": "u'   (index=1)
Delta 5: tool_call[0] args=' circulars"}'      (index=0)
Delta 6: tool_call[1] args='123"}'             (index=1)

Each delta carries an index (and optionally an id) that identifies which tool call it belongs to. The expected behavior: two distinct tool calls with their own names and arguments.

The actual behavior in Bifrost’s accumulator before the fix:

1// Old logic (simplified)
2if deltaToolCall.Function.Name != nil {
3    // New tool call — create it
4    existingToolCalls = append(existingToolCalls, newToolCall)
5} else {
6    // Argument delta — append to the LAST tool call
7    toolCallToModify = &existingToolCalls[len(existingToolCalls)-1]
8    toolCallToModify.Function.Arguments += deltaToolCall.Function.Arguments
9}

The routing logic was based on recency: if a delta has a Name, it’s a new tool call; otherwise, append arguments to the most recent one. This works perfectly when tool calls arrive sequentially. It breaks completely when they’re interleaved.

With the deltas above:

• Delta 3 (index=0 args) arrives → appended to the last tool call, which is get_user_profile (index=1). Wrong target.
• Delta 5 (index=0 args) arrives → again appended to get_user_profile. Now get_user_profile has arguments from two different calls mashed together.

The result: concatenated tool names, merged arguments, and an agent stuck in a retry loop trying to invoke search_circularsget_user_profile — a tool that doesn’t exist.

Finding the bug

I hit this in production. Our compliance platform uses an AI agent that calls multiple tools in parallel — searching circulars, fetching regulatory data, looking up user context. The agent kept failing with tool resolution errors, and the telemetry showed impossible tool names:

search_circularsearch_circularsearch_circular

At first I suspected the agent’s tool-calling logic. Then I looked at the raw SSE stream from the provider — the deltas were correct. Each one had the right index and id. The corruption was happening between the provider response and the final accumulated message.

I traced it to accumulateToolCallsInMessage in framework/streaming/accumulator.go. The function had no concept of tool call identity. It used the presence of a Name field as a heuristic for “new vs. continuation,” which is fragile and incorrect when deltas arrive out of order.

The fix

The fix is conceptually simple: route deltas by identity, not by recency.

Each tool call delta carries two potential identifiers:

1. tool_call.id — a string like call_abc123 (set by OpenAI/Anthropic)
2. tool_call.index — an integer position (0, 1, 2, …)

The new logic builds lookup maps on existing tool calls and uses them to route each incoming delta to the correct target:

1// Build lookup maps for identity-based routing
2idToIndex := make(map[string]int, len(existingToolCalls))
3indexToIndex := make(map[uint16]int, len(existingToolCalls))
4for i, toolCall := range existingToolCalls {
5    if toolCall.ID != nil && *toolCall.ID != "" {
6        idToIndex[*toolCall.ID] = i
7    }
8    indexToIndex[toolCall.Index] = i
9}

For each incoming delta, routing follows a priority chain:

 1for _, deltaToolCall := range deltaToolCalls {
 2    index := -1
 3
 4    // Priority 1: match by tool_call.id
 5    if deltaToolCall.ID != nil && *deltaToolCall.ID != "" {
 6        if existingIndex, ok := idToIndex[*deltaToolCall.ID]; ok {
 7            index = existingIndex
 8        }
 9    }
10
11    // Priority 2: match by tool_call.index
12    if index == -1 {
13        if existingIndex, ok := indexToIndex[deltaToolCall.Index]; ok {
14            index = existingIndex
15        }
16    }
17
18    // Priority 3: no match → create new tool call (if name present)
19    if index == -1 {
20        if deltaToolCall.Function.Name == nil {
21            // Can't create a tool call without a name
22            continue
23        }
24        newToolCall := schemas.ChatAssistantMessageToolCall{
25            Index:    deltaToolCall.Index,
26            ID:       deltaToolCall.ID,
27            Function: /* ... */,
28        }
29        existingToolCalls = append(existingToolCalls, newToolCall)
30        // Update maps for future deltas
31        indexToIndex[newToolCall.Index] = len(existingToolCalls) - 1
32        if newToolCall.ID != nil {
33            idToIndex[*newToolCall.ID] = len(existingToolCalls) - 1
34        }
35        continue
36    }
37
38    // Append arguments to the correct tool call
39    existingToolCalls[index].Function.Arguments += deltaToolCall.Function.Arguments
40}

The key insight is that the maps are updated as new tool calls are created, so subsequent deltas for the same call always find their target — even if they arrive many chunks later.

The same bug, twice

During code review, a reviewer (Pratham-Mishra04) pointed out that the same recency-based routing existed in a second code path: the Responses API streaming handler in responses.go.

The Responses API is OpenAI’s newer streaming format (the /v1/responses endpoint). It uses ItemID instead of tool_call.index, but the accumulation logic had the same flaw — FunctionCallArgumentsDelta events were always appended to the last message:

1// Old responses.go logic
2if resp.Delta != nil && len(messages) > 0 {
3    // Always targets the last message — wrong for parallel calls
4    a.appendFunctionArgumentsDeltaToResponsesMessage(
5        &messages[len(messages)-1], *resp.Delta)
6}

The fix follows the same pattern: search for a message matching resp.ItemID, and only fall back to the last message when no ItemID is present. If an ItemID exists but no message matches, create a stub message to hold the delta rather than merging it into the wrong call:

 1if resp.Delta != nil && len(messages) > 0 {
 2    targetIdx := len(messages) - 1
 3    if resp.ItemID != nil {
 4        targetIdx = -1
 5        for i := len(messages) - 1; i >= 0; i-- {
 6            if messages[i].ID != nil && *messages[i].ID == *resp.ItemID {
 7                targetIdx = i
 8                break
 9            }
10        }
11        if targetIdx == -1 {
12            // Create stub to avoid merging into wrong message
13            id := *resp.ItemID
14            messages = append(messages, schemas.ResponsesMessage{ID: &id})
15            targetIdx = len(messages) - 1
16        }
17    }
18    a.appendFunctionArgumentsDeltaToResponsesMessage(
19        &messages[targetIdx], *resp.Delta)
20}

Testing parallel accumulation

The regression test simulates the exact interleaving pattern that triggers the bug:

 1func TestAccumulateToolCallsInterleavedParallel(t *testing.T) {
 2    // ...
 3    toolCallID0 := "call_0"
 4    toolCallID1 := "call_1"
 5    toolNameAdd := "add"
 6    toolNameMultiply := "multiply"
 7
 8    // Simulate interleaved deltas
 9    accumulate(makeDelta(0, &toolCallID0, &toolNameAdd, ""))       // create call 0
10    accumulate(makeDelta(1, &toolCallID1, &toolNameMultiply, ""))  // create call 1
11    accumulate(makeDelta(0, nil, nil, `{"a": 1`))                  // args for call 0
12    accumulate(makeDelta(1, nil, nil, `{"a": 2`))                  // args for call 1
13    accumulate(makeDelta(0, nil, nil, `, "b": 3}`))                // more args for call 0
14    accumulate(makeDelta(1, nil, nil, `, "b": 4}`))                // more args for call 1
15
16    // Assert: two distinct tool calls with correct arguments
17    assert(addCall.Arguments == `{"a": 1, "b": 3}`)
18    assert(multiplyCall.Arguments == `{"a": 2, "b": 4}`)
19}

Without the fix, add would end up with multiply’s arguments concatenated in. With the fix, each call accumulates only its own deltas.

A second test covers the Responses API path with ItemID-based routing, using the same interleaving pattern but with OutputItemAdded + FunctionCallArgumentsDelta events.

The PR lifecycle

I filed Issue #1829 with reproduction details from production telemetry, then opened PR #1830 with the fix and tests.

The review process was smooth:

1. First commit: fix the Chat Completions accumulator + regression test
2. Reviewer feedback: same bug exists in Responses API — please fix there too + add changelog entries
3. Second commit: fix Responses API routing + regression test + changelogs
4. Merged by akshaydeo (Bifrost maintainer)

Total: 3 commits, ~280 lines of Go across 5 files. Same-day turnaround from issue to merge.

What I learned

1. Recency is not identity

The original code used position (last element) as a proxy for identity. This is a common pattern in sequential streaming — and it works, until the stream isn’t sequential. The fix is to use the actual identity field that the protocol already provides.

This is a general principle: if the protocol gives you an ID field, use it. Don’t rely on ordering assumptions that hold today but break when concurrency enters.

2. The same bug pattern repeats across API surfaces

Bifrost supports two streaming formats: Chat Completions (the classic tool_calls array) and Responses API (the newer ItemID-based format). Both had the same routing flaw because the accumulation logic was duplicated rather than shared. The reviewer catching this in a different file was the most valuable part of the review.

3. OSS contributions are about context, not complexity

The fix itself is straightforward — build a map, use it for lookup. The hard part was:

• Finding the bug in a codebase I’d never seen before
• Understanding why the existing approach was wrong
• Testing with realistic interleaved deltas
• Responding to review feedback about a parallel code path I hadn’t initially noticed

The code change was ~100 lines. The context-building was hours.

4. Production telemetry is the best bug report

The issue report included the exact malformed tool name from production: search_circularsearch_circularsearch_circular. That single string made the bug immediately reproducible and the fix trivially verifiable. If you’re filing a bug, include the malformed output — it’s worth more than a paragraph of description.

Links

• PR: fix: preserve parallel tool call deltas — #1830
• Issue: Streaming tool call deltas merge in parallel tool calling — #1829
• Project: Bifrost AI Gateway — 2.6k ⭐, fastest enterprise AI gateway

This is the story of how I tracked down four distinct memory leaks across two Python services running on Kubernetes. Each one was a different flavor of “silent accumulation,” and each one taught me something about how easy it is to bleed memory in async Python without any obvious errors.

No names. No product details. Just the bugs, the fixes, and the lessons.

The setup

We had a multi-service platform running on Kubernetes. Two services are relevant here:

• Service A: A document management service. It handled CRUD operations, search indexing, audit logging, and authorization precomputation. Built with FastAPI and async Python, backed by MongoDB and a vector store.

• Service B: A PDF processing service. It parsed documents, extracted text via OCR, and broke content into structured clauses. Heavy on PyMuPDF (fitz) for PDF rendering and Tesseract for OCR.

Both services ran as single-container pods with resource limits. Service A had a 2Gi memory limit. Service B had 4Gi.

Both were getting OOMKilled. Regularly.

The symptoms

Service A’s memory graph looked like this:

Memory (MB)
2048 |                                              ╱ OOMKilled
     |                                           ╱
1536 |                                        ╱
     |                                     ╱
1024 |                                  ╱
     |                               ╱
 512 |                            ╱
     |                         ╱
   0 |________________________╱
     0h        6h        12h        18h        24h

A steady climb at roughly 200MB per hour. No spikes. No correlation with traffic. Just a ramp.

Service B was different - it’d spike during document processing batches and never come back down:

Memory (MB)
4096 |          ╱╲        ╱╲
     |        ╱   ╲     ╱   ╲    ╱ OOMKilled
3072 |      ╱      ╲  ╱      ╲ ╱
     |    ╱         ╲╱        ╱
2048 |  ╱                   ╱
     |╱                   ╱
1024 |                  ╱
     |               ╱
   0 |______________╱
     0h     2h     4h     6h     8h     10h

Every batch pushed memory up. It came down a little, but never to baseline. Classic leak with GC partially reclaiming some objects but not the real offender.

Leak #1: The coroutines that never ran

This was the most insidious one. It looked like perfectly normal Python code:

 1class BaseDoc:
 2    def _write_audit_log(self, action, old_data, new_data):
 3        enqueue_audit_create(self, action, old_data, new_data)
 4
 5    async def save(self, *args, **kwargs):
 6        old = await self.get(self.id)
 7        result = await super().save(*args, **kwargs)
 8        self._write_audit_log("update", old, result)
 9        await create_snapshots(self.id)
10        await insert_versions_docs(self.id)
11        return result

Spot the bug?

_write_audit_log calls enqueue_audit_create(), which is an async function. But _write_audit_log is not async, and it doesn’t await the call. It just calls it.

When you call an async function without await, Python creates a coroutine object but never schedules it. The coroutine sits in memory, holding references to all its arguments - in this case, entire document objects with old_data and new_data.

1Coroutine Object (0x7f...)
2  ├── State: CREATED (never scheduled)
3  └── Frame Object
4       ├── Arg: self (BaseDoc instance, 2KB)
5       ├── Arg: old_data (Dict, 5MB)  <-- LEAKED
6       └── Arg: new_data (Dict, 5MB)  <-- LEAKED
7       └── Local Vars: (captured closure scope)

Python will eventually warn you about this (RuntimeWarning: coroutine was never awaited), but only if you have warnings enabled and you’re looking at the right log. In a noisy production service running dozens of requests per second, these warnings get buried.

The same pattern appeared elsewhere:

 1# In the CRUD endpoints
 2async def create_document(request):
 3    doc = await Document.create(request.data)
 4    enqueue_audit_create(doc)       # ← async, not awaited
 5    qdrant_upserts(doc)             # ← async, not awaited
 6    return doc
 7
 8async def update_document(request):
 9    doc = await Document.update(request.data)
10    enqueue_audit_update(doc)       # ← async, not awaited
11    qdrant_upserts(doc)             # ← async, not awaited
12    return doc

Each unawaited call created a coroutine holding the full document. For a service processing hundreds of documents per hour, that’s hundreds of document-sized objects pinned in memory with no way to be garbage collected.

The fix

Straightforward once you see it:

 1class BaseDoc:
 2    async def _write_audit_log(self, action, old_data, new_data):
 3        await enqueue_audit_create(self, action, old_data, new_data)
 4
 5    async def save(self, *args, **kwargs):
 6        old = await self.get(self.id)
 7        result = await super().save(*args, **kwargs)
 8        await self._write_audit_log("update", old, result)
 9        await create_snapshots(self.id)
10        await insert_versions_docs(self.id)
11        return result

And in the CRUD endpoints:

1async def create_document(request):
2    doc = await Document.create(request.data)
3    await enqueue_audit_create(doc)
4    await qdrant_upserts(doc)
5    return doc

The lesson

Unawaited coroutines are silent memory leaks. They don’t crash. They don’t raise. They just accumulate. And because they hold references to their arguments, they can keep large objects alive long after the request that created them is done.

If you’re running async Python, add this to your linting:

1# In your pytest conftest.py or startup
2import warnings
3warnings.filterwarnings("error", category=RuntimeWarning, message="coroutine.*was never awaited")

This turns the warning into an exception, which makes it impossible to miss.

Leak #2: Creating a new MongoClient on every authorization check

The authorization layer needed to precompute user access contexts. To do this, it queried MongoDB directly (not through the async ODM) using PyMongo’s synchronous MongoClient.

The code looked like this:

 1class AuthorizationPrecompute:
 2    def get_user_permissions(self, user_id):
 3        client = MongoClient(MONGO_URI)
 4        db = client["main_db"]
 5        permissions = list(db.permissions.find({"user": user_id}))
 6        client.close()
 7        return permissions
 8
 9    def get_role_policies(self, role_id):
10        client = MongoClient(MONGO_URI)
11        db = client["main_db"]
12        policies = list(db.policies.find({"role": role_id}))
13        client.close()
14        return policies
15
16    def get_org_settings(self, org_id):
17        client = MongoClient(MONGO_URI)
18        db = client["main_db"]
19        settings = db.settings.find_one({"org": org_id})
20        client.close()
21        return settings
22
23    # ... 10 more methods, each creating its own MongoClient

Thirteen methods. Thirteen MongoClient() instantiations per authorization check.

Each MongoClient creates a connection pool, background SDAM monitoring threads, and internal caches. Even with client.close(), the cleanup isn’t instant - Python’s garbage collector has to reclaim the thread stacks, socket buffers, and internal data structures.

And if an exception happens between MongoClient() and client.close(), the client never gets closed at all. The connection pool lives on, holding threads and sockets until the GC eventually (maybe) collects them.

At 50 requests/second, each triggering an auth check, we were creating 650 MongoClient instances per second. Most were closed, but the transient memory pressure and leaked exceptions meant we were always carrying a few hundred orphaned connection pools.

The fix

One shared client with a connection pool:

 1class AuthorizationPrecompute:
 2    _client = None
 3
 4    @classmethod
 5    def _get_client(cls):
 6        if cls._client is None:
 7            cls._client = MongoClient(
 8                MONGO_URI,
 9                maxPoolSize=20,
10                serverSelectionTimeoutMS=5000
11            )
12        return cls._client
13
14    def get_user_permissions(self, user_id):
15        db = self._get_client()["main_db"]
16        return list(db.permissions.find({"user": user_id}))
17
18    def get_role_policies(self, role_id):
19        db = self._get_client()["main_db"]
20        return list(db.policies.find({"role": role_id}))

Connection overhead went from ~50ms per call (TCP handshake + auth each time) to ~5ms (reused connection from pool).

The lesson

Each MongoClient is an entire connection pool, not a single connection. Creating one per function call is like starting a new database server for every query. Always share a single client instance (or at most one per thread/worker). And if you can’t guarantee close() will be called (i.e., exceptions exist), you definitely need a shared instance.

Leak #3: Cache entries that never expire

The authorization layer also had an in-memory cache for user access contexts:

 1class UserAccessContextCache:
 2    def __init__(self, ttl_seconds=300):
 3        self._cache = {}
 4        self._ttl = ttl_seconds
 5
 6    def get(self, key):
 7        entry = self._cache.get(key)
 8        if entry and time.time() - entry["created"] < self._ttl:
 9            return entry["value"]
10        return None
11
12    def set(self, key, value):
13        self._cache[key] = {
14            "value": value,
15            "created": time.time()
16        }

See the problem?

When get() finds an expired entry, it just returns None. It doesn’t delete the entry. The stale data sits in the dict forever.

And set() only adds entries. Nothing ever removes them.

Over time, this cache grows without bound. Every unique user who hits the service gets a cache entry that is never cleaned up. The access context objects themselves can be large - they contain permission trees, role hierarchies, and org settings.

With 200MB of stale cache entries after a day of operation, this was a significant contributor to the memory ramp.

The fix

Add proactive cleanup:

 1class UserAccessContextCache:
 2    def __init__(self, ttl_seconds=300, cleanup_interval=60):
 3        self._cache = {}
 4        self._ttl = ttl_seconds
 5        self._cleanup_interval = cleanup_interval
 6        self._last_cleanup = time.time()
 7
 8    def _cleanup_expired(self):
 9        now = time.time()
10        if now - self._last_cleanup < self._cleanup_interval:
11            return
12        self._last_cleanup = now
13        expired_keys = [
14            k for k, v in self._cache.items()
15            if now - v["created"] >= self._ttl
16        ]
17        for k in expired_keys:
18            del self._cache[k]
19
20    def get(self, key):
21        self._cleanup_expired()
22        entry = self._cache.get(key)
23        if entry and time.time() - entry["created"] < self._ttl:
24            return entry["value"]
25        # Delete on read if expired
26        if key in self._cache:
27            del self._cache[key]
28        return None
29
30    def set(self, key, value):
31        self._cleanup_expired()
32        self._cache[key] = {
33            "value": value,
34            "created": time.time()
35        }

Cleanup runs at most once per minute and removes all entries older than TTL.

The lesson

Every in-memory cache needs an eviction strategy. If there’s no mechanism to remove old entries, you have a memory leak with extra steps. “TTL on read” is not enough - if users stop requesting a key, that entry stays forever.

Use cachetools.TTLCache or similar if you don’t want to roll your own. Better yet, if the cache is big enough to matter, put it in Redis where you can set EXPIRE and forget about it.

Leak #4: PyMuPDF documents that never closed

Service B was the PDF processing service. It used PyMuPDF (fitz) to render pages and extract text. The endpoints looked like this:

 1@app.post("/parse_pdf")
 2async def parse_pdf(file: UploadFile):
 3    content = await file.read()
 4    doc = fitz.open(stream=content, filetype="pdf")
 5
 6    pages = []
 7    for page_num in range(len(doc)):
 8        page = doc[page_num]
 9        text = page.get_text()
10        pix = page.get_pixmap(dpi=300)
11        image = pix.tobytes("png")
12        pages.append({"text": text, "image": image})
13
14    return {"pages": pages}

No doc.close(). No context manager. No try/finally.

PyMuPDF is a C library wrapper. When you call fitz.open(), it allocates memory in C-land for the document structure, page data, fonts, images, and rendering buffers. Python’s garbage collector can eventually reclaim the Python wrapper object, but the C-side memory is only freed when doc.close() is explicitly called.

While the wrapper theoretically has a __del__ method, relying on it in a high-throughput async loop is dangerous. Python’s GC is lazy and may not run fast enough to keep up with the rate of C-memory allocation, leading to OOMs before the GC even realizes it needs to clean up.

For a 50-page PDF at 300 DPI, the pixmap rendering alone can consume 200–500MB of memory. If an exception occurs mid-processing (corrupt page, unsupported font), the doc handle leaks entirely.

The fix

Two changes: proper resource cleanup, and parallel OCR with controlled concurrency.

 1@app.post("/parse_pdf")
 2async def parse_pdf(file: UploadFile):
 3    content = await file.read()
 4    doc = fitz.open(stream=content, filetype="pdf")
 5    try:
 6        pages = []
 7        for page_num in range(len(doc)):
 8            page = doc[page_num]
 9            text = page.get_text()
10            pix = page.get_pixmap(dpi=300)
11            image = pix.tobytes("png")
12            # Explicitly free the pixmap
13            pix = None
14            pages.append({"text": text, "image": image})
15        return {"pages": pages}
16    finally:
17        doc.close()

This was applied to all four endpoints in the service: /parse_pdf, /process_clauses, /process_clauses_sync, and /extract_links.

We also added parallel strip processing for OCR (controlled by feature flags):

 1import os
 2from concurrent.futures import ThreadPoolExecutor
 3
 4ENABLE_PARALLEL_OCR = os.getenv("ENABLE_PARALLEL_OCR", "false").lower() == "true"
 5STRIP_OCR_WORKERS = int(os.getenv("STRIP_OCR_WORKERS", "8"))
 6
 7def process_page_strips(page, strips):
 8    if not ENABLE_PARALLEL_OCR:
 9        return [ocr_strip(page, s) for s in strips]
10
11    with ThreadPoolExecutor(max_workers=STRIP_OCR_WORKERS) as pool:
12        futures = [pool.submit(ocr_strip, page, s) for s in strips]
13        return [f.result() for f in futures]

The deployment

We didn’t flip everything on at once:

1. Deploy with ENABLE_PARALLEL_OCR=false - just the doc.close() fixes
2. Monitor memory for 24 hours to confirm the leak is fixed
3. Bump resource limits from 4Gi to 6Gi to accommodate parallel processing headroom
4. Enable parallel OCR: ENABLE_PARALLEL_OCR=true

Memory dropped from a steady 4GB to a stable 1.8GB. That’s a 55% reduction from just closing document handles properly.

The lesson

C extension libraries manage their own memory. Python’s GC has no visibility into fitz, numpy, Pillow, or any other C-backed library. If the library provides a close(), use it. If it supports context managers, use with. And always wrap C-heavy processing in try/finally.

The combined impact

Here’s the before and after across both services:

Service A went from daily OOMKills to zero restarts. Service B went from crashing every 8–10 hours to running for weeks without a restart.

How I found them

No fancy tooling. Here’s the actual process:

1. Look at the memory graph. Linear growth = accumulation leak. Sawtooth that doesn’t return to baseline = handle/resource leak.

2. Read the code. Seriously. I read every function in the hot paths and asked “what gets allocated here that doesn’t get freed?”

3. grep for patterns. Once I found one unawaited coroutine, I searched for the pattern across the entire codebase:

1# Find async functions called without await
2rg "^\s+[a-z_]+\(" --type py | grep -v "await " | grep -v "def " | grep -v "#"

Not perfect, but it flagged the 13 MongoClient instantiations immediately.

4. Check warnings. Enabled RuntimeWarning filtering and got 6 unawaited coroutine warnings within the first minute of startup.

5. Kubernetes events. kubectl describe pod shows OOMKilled with the exact memory at death. Compare that to your resource limits and you know how fast it’s growing.

1kubectl get events --field-selector reason=OOMKilling \
2  --sort-by='.lastTimestamp' -n your-namespace

A checklist for your services

If you’re running Python services on Kubernetes and seeing memory growth, check these in order:

• Unawaited coroutines. Search for RuntimeWarning: coroutine.*was never awaited in your logs. Better yet, make it an error.
• MongoClient instantiation. grep -rn "MongoClient(" your_code/ - if it appears in more than one place (especially inside functions, not module level), you probably have a leak.
• In-memory caches without eviction. Any dict that grows with usage and has no del or size cap is a leak. TTLCache, LRU, or maxsize - pick one.
• C extension resources. PyMuPDF, Pillow, lxml, pdfplumber - anything that wraps C needs explicit close() or with statements.
• Large objects in exception handlers. If your except block captures the full traceback and logs the local variables, those locals (which may include entire documents) are pinned until the log entry is flushed.
• Background tasks holding request data. If asyncio.create_task() captures a closure over request objects, those objects live until the task completes.

Final thought

None of these bugs produced an error. No stack trace. No warning (well, one did, but it was buried). The services ran fine - they just got slower and slower until Kubernetes killed them, then they restarted and the cycle began again.

Memory leaks in garbage-collected languages are not the “forgot to call free()” kind. They’re the “I’m holding a reference I didn’t know about” kind. The GC is doing its job perfectly - it’s keeping alive exactly what you told it to.

You just told it wrong.

Why increasing maxPoolSize breaks your database. A deep dive into MongoDB connection pooling, SDAM, WiredTiger tickets, and preventing transaction pinning.

Introduction

In the MongoDB ecosystem, the “connection pool” is often treated as a set-and-forget configuration within MongoClient. However, when latency spikes or the database becomes unresponsive during high traffic, the pool is usually the first mechanism to break.

Unlike relational databases that often sit behind proxies like PgBouncer, MongoDB drivers are “smart”—they maintain direct connections to multiple nodes in a Replica Set or Sharded Cluster. This complexity means exhaustion isn’t just about running out of sockets; it’s about topology monitoring, transaction pinning, and the brutal reality of the thread-per-connection model on the server side.

How MongoDB Connects and Executes Queries

Before diving into pool exhaustion, it’s worth understanding the lifecycle of a MongoDB connection and query. The process is more involved than a simple socket open/close.

Connection Establishment

When your application calls MongoClient.connect(), the driver doesn’t immediately open a socket. Instead, it:

1. TCP Socket Creation: Establishes a TCP/IP connection to the MongoDB server (default port 27017). This is a standard three-way handshake at the network layer.

2. Wire Protocol Handshake: The client sends a hello command (formerly isMaster in older versions) to the server. This serves multiple purposes:

   • Verifies the server is a MongoDB instance
   • Retrieves server capabilities (wire protocol version, supported authentication mechanisms)
   • In MongoDB 4.4+, can include speculativeAuthenticate to start authentication concurrently, reducing latency

3. Authentication: If credentials are provided, the client authenticates using mechanisms like SCRAM (Salted Challenge Response Authentication Mechanism):

   • Client sends saslStart with chosen mechanism and initial payload
   • Server responds with a challenge
   • Client computes and sends proof derived from the challenge
   • Server verifies and grants access

This entire process—TCP handshake, TLS negotiation (if enabled), handshake, and authentication—typically takes 10-50ms depending on network latency and encryption overhead.

Query Execution Flow

Once authenticated, the connection is ready for operations. When you execute a query:

1. Connection Checkout: The driver checks out a connection from the pool (or waits if none are available—this is where exhaustion manifests).

2. Message Construction: The query is serialized into MongoDB’s wire protocol format. Modern MongoDB uses the OP_MSG opcode (introduced in 3.6), which includes:

   • Standard message header (request ID, response-to ID)
   • Flags indicating message properties
   • Sections containing the actual command/query data
   • Optional checksum for integrity

3. Network Transmission: The message is sent over the TCP socket to the server.

4. Server Processing: The mongod process:

   • Receives the message on a dedicated thread (in the thread-per-connection model)
   • Parses the query
   • Plans execution strategy
   • Accesses data through WiredTiger (subject to ticket limits)
   • Creates a cursor for read operations

5. Response: The server sends results back over the same connection. For large result sets, the driver uses cursors to fetch documents in batches (default batch size is 101 documents).

6. Connection Return: After the operation completes, the connection is returned to the pool for reuse—unless it’s pinned to a transaction (see “The Specific Pathology: Pinning” below).

Why This Matters for Pooling

Each of these steps has latency and resource costs. The connection establishment overhead (10-50ms) is why pooling exists: reusing connections avoids paying this cost for every operation. However, the server-side thread allocation means each connection has a real memory and scheduling cost on mongod, which is why blindly increasing pool size backfires.

Connection Pool Internals

Understanding how the pool manages connections internally is crucial for diagnosing exhaustion. The pool maintains connections in different states:

Connection States

1. Idle: Connections that are established and authenticated but not currently in use. These sit in the pool, ready for immediate checkout. They consume server-side resources (thread + memory) but are available for operations.

2. Checked Out: Connections actively executing an operation. When you call collection.find(), the driver checks out a connection, performs the operation, then returns it to idle—unless it’s pinned (see “The Specific Pathology: Pinning”).

3. Pending: Connections in the process of being established (TCP handshake, authentication). These don’t count toward maxPoolSize until fully established.

4. Closed: Connections that have been terminated, either due to errors, idle timeout, or explicit closure.

The Wait Queue

When all connections are checked out and the pool has reached maxPoolSize, new requests enter a wait queue. This queue has its own limits:

• waitQueueTimeoutMS: How long a request waits for a connection before throwing ConnectionPoolCheckoutFailed. Default is 0 (infinite wait—dangerous in production). Always set this to fail fast rather than hang indefinitely.

If the wait queue fills up, your application starts failing requests. This is often the first symptom of pool exhaustion, not the root cause.

Connection Lifecycle

The connection lifecycle follows these states and transitions:

[Not Created]
    │
    │ connect()
    ▼
[Pending] ── TCP handshake, authentication ──► [Idle]
    │                                              │
    │                                              │ Driver request
    │                                              ▼
    │                                        [Checked Out]
    │                                              │
    │                                              │ Operation complete
    │                                              │
    │                                              ▼
    │                                          [Idle] ◄──┐
    │                                              │     │
    │                                              │     │ Return to pool
    │                                              │     │
    │                                              │     │
    │                                              │ startTransaction()
    │                                              ▼     │
    │                                          [Pinned] ──┘
    │                                              │
    │                                              │ ʕ•ᴥ•ʔ DANGEROUS:
    │                                              │ Exclusive to session
    │                                              │ until commit/abort
    │                                              │
    │                                              │ commit/abortTransaction()
    │                                              ▼
    │                                          [Idle]
    │                                              │
    │                                              │ maxIdleTimeMS reached
    │                                              ▼
    │                                          [Closed]
    │                                              │
    └──────────────────────────────────────────────┘

State Descriptions:

• Pending: Connection is being established (TCP handshake, authentication)
• Idle: Connection is established and available in the pool for immediate use
• Checked Out: Connection is actively executing an operation
• Pinned: Connection is reserved for a transaction (removed from pool until transaction completes)
• Closed: Connection has been terminated

The pool lazily creates connections up to maxPoolSize. If minPoolSize > 0, it pre-warms that many connections at startup. Connections idle longer than maxIdleTimeMS are closed to free server resources.

SDAM: The Hidden Connection Overhead

MongoDB drivers implement SDAM (Server Discovery and Monitoring), which maintains connections to all nodes in a Replica Set or Sharded Cluster, not just the primary.

How SDAM Works

1. Initial Discovery: On MongoClient.connect(), the driver connects to seed addresses and sends hello commands to discover all replica set members.

2. Continuous Monitoring: Every heartbeatFrequencyMS (default 10 seconds), the driver sends hello to each known node to:

   • Detect role changes (primary elections, step-downs)
   • Check server capabilities
   • Update topology description

3. Topology Events: When a primary election occurs, the driver:

   • Closes connections to the old primary
   • Opens connections to the new primary
   • Potentially drains and recreates pools

The Connection Multiplier

If you have a 3-node replica set and set maxPoolSize=50, you’re not creating 50 connections—you’re creating up to 150 connections (50 per node). SDAM maintains separate pools for each server in the topology.

During a primary election, the driver may temporarily exceed this as it transitions between nodes. If you’re running 20 app instances, that’s 3,000 potential connections during failover, even if only one node is serving traffic.

Takeaway: Your maxPoolSize is per-server, not per-client. In replica sets, multiply by the number of nodes you’re monitoring.

Connection Pool Configuration Options

Here are the key options that affect pool behavior:

• maxPoolSize: Maximum connections per server. Default 100. This is the most commonly misconfigured setting.

• minPoolSize: Minimum idle connections to maintain. Default 0. Setting this > 0 pre-warms the pool but consumes server resources even when idle.

• maxIdleTimeMS: Close idle connections after this duration. Default 0 (never close). Useful in serverless where you want aggressive cleanup.

• waitQueueTimeoutMS: How long to wait for a connection before failing. Default 0 (wait forever—dangerous in production). Always set this to fail fast rather than hang indefinitely.

• connectTimeoutMS: Timeout for initial connection establishment. Default 30 seconds.

• socketTimeoutMS: Timeout for individual operations. Default 0 (no timeout). Different from connection checkout timeout.

• heartbeatFrequencyMS: How often SDAM checks server status. Default 10 seconds. Lower values detect failovers faster but increase network overhead.

• TCP Keepalive: Often overlooked. If your app sits behind a firewall or load balancer (e.g., AWS Network Load Balancer) with a 350s idle timeout, and your pool has maxIdleTimeMS > 350s, the LB will silently drop the connection. The driver won’t know until it tries to write to the socket and fails. Important: socketTimeoutMS does not send keepalive probes. Best Practice: Set maxIdleTimeMS to be slightly lower than your infrastructure’s timeout (e.g., 120000ms). On Linux, also tune net.ipv4.tcp_keepalive_time to be less than the cloud provider’s timeout (usually < 120 seconds).

Example Configuration (Node.js):

1const client = new MongoClient(uri, {
2  maxPoolSize: 10,              // Small pool for 16-core DB
3  minPoolSize: 2,               // Pre-warm 2 connections
4  maxIdleTimeMS: 30000,         // Close idle after 30s
5  waitQueueTimeoutMS: 5000,     // Fail after 5s wait
6  connectTimeoutMS: 10000,      // 10s connection timeout
7  socketTimeoutMS: 45000,       // 45s operation timeout
8  heartbeatFrequencyMS: 10000   // Check every 10s
9});

The Cost of a Connection in mongod

While MongoDB has moved toward a more asynchronous networking model in recent versions, historically (and effectively, for resource planning), it relies on a synchronous thread-per-connection model.

When your application opens a new connection:

1. Network: TCP handshake + TLS negotiation (CPU intensive).
2. Authentication: SCRAM or X.509 handshakes occur.
3. Memory: The mongod process allocates stack memory (typically ~1MB) for the thread handling that connection.
4. Context: The internal serviceExecutor must schedule this thread.

If you set maxPoolSize=100 on 50 Kubernetes pods, you are allowing 5,000 potential concurrent threads on the Primary. Before you hit hardware limits, you will likely hit WiredTiger ticket exhaustion.

The Math: Sizing for WiredTiger

The naive approach is to increase maxPoolSize when you see ConnectionPoolCheckoutFailed errors. This is usually wrong.

WiredTiger (the storage engine) uses “tickets” to control concurrency. In MongoDB versions prior to 7.0, there are 128 read tickets and 128 write tickets by default. MongoDB 7.0+ uses dynamic ticketing that adjusts based on workload, but still caps at 128 per type. Note on MongoDB 7.0+: While the hard cap remains 128 tickets, the dynamic algorithm often lowers this limit (e.g., to 60 or 80) in real-time to preserve latency. This means you might see queuing before you hit 128 concurrent active queries. If 1,000 connections try to read simultaneously, 872 of them are just queuing inside the kernel, consuming RAM but doing zero work.

The optimal pool size formula remains tied to hardware capability, not traffic volume:

If your database has 16 cores and you have 20 application instances, a maxPoolSize of 100 is overkill. You might actually achieve higher throughput with a pool size of 2 or 3 per instance, preventing the database from spending all its cycles on context switching.

Note: The (Cores * 2) rule applies strictly to the active executing threads. You can have a larger pool to handle network jitter, but waitQueueTimeoutMS is your safety valve. If you need 100 connections to saturate the CPU, your queries are likely too slow (unindexed). Fix the queries, don’t inflate the pool.

Workload Considerations:

• CPU-Bound Workloads: The formula above applies directly. Each active connection should map to an executing thread.
• IO-Bound Workloads: If the working set doesn’t fit in RAM and disk I/O is the bottleneck, a slightly larger pool might help hide I/O latency, though typically minPoolSize is more relevant here to avoid establishing connections during I/O spikes.
• Atlas Proxy / Serverless: If using MongoDB Atlas Serverless or the Atlas Proxy, the connection limits are handled differently (via a proxy layer), and the “100 limit” on the client side is less about server stress and more about client-side throttling.

The Specific Pathology: “Pinning”

Since you are versed in Mongo, you know the drivers implement the SDAM (Server Discovery and Monitoring) specification. But the most dangerous feature regarding pooling is Client-Side Operation Pinning, specifically with Transactions.

In a standard read/write, the driver grabs a connection, executes, and returns it. In a Transaction (Multi-Document ACID):

1. session.startTransaction() is called.
2. The driver pins a connection to that session.
3. That connection is removed from the pool and reserved exclusively for this session until commitTransaction or abortTransaction is called.

The Exhaustion Scenario

If you have maxPoolSize=50 and an API endpoint that takes 2 seconds to execute some logic inside a transaction, you can only serve 25 requests per second. The 26th request will wait. If the logic hangs (e.g., waiting for a third-party HTTP call), you will drain the pool instantly, causing a cascading failure across the app.

ʕ•ᴥ•ʔ Critical Rule: Never perform network I/O or long computations inside a MongoDB transaction. Transactions should be atomic, fast, and only contain database operations.

Practical Examples: The Right Way vs. The Wrong Way

Proper Client Initialization

Node.js - Express (Correct):

 1// app.js - Initialize once at startup
 2const { MongoClient } = require('mongodb');
 3
 4let client;
 5let db;
 6
 7async function initDatabase() {
 8  client = new MongoClient(uri, {
 9    maxPoolSize: 10,
10    waitQueueTimeoutMS: 5000
11  });
12  await client.connect();
13  db = client.db('myapp');
14  console.log('Database connected');
15}
16
17// Initialize before starting server
18initDatabase().then(() => {
19  app.listen(3000);
20});
21
22// Use the same client instance everywhere
23app.get('/users', async (req, res) => {
24  const users = await db.collection('users').find({}).toArray();
25  res.json(users);
26});

Node.js - Express (Wrong - Creates New Pool Per Request):

1// ʕ•̀ω•́ʔ DON'T DO THIS
2app.get('/users', async (req, res) => {
3  const client = new MongoClient(uri);  // New pool every request!
4  await client.connect();
5  const db = client.db('myapp');
6  const users = await db.collection('users').find({}).toArray();
7  await client.close();  // Closes pool, but too late
8  res.json(users);
9});

Python - Gunicorn (Correct):

 1# app.py - Create client after fork
 2from pymongo import MongoClient
 3
 4client = None
 5db = None
 6
 7def on_starting(server):
 8    # This runs in the master process
 9    pass
10
11def when_ready(server):
12    # This runs in each worker after fork
13    global client, db
14    client = MongoClient(uri, maxPoolSize=10)
15    db = client['myapp']
16
17@app.route('/users')
18def get_users():
19    users = list(db.users.find({}))
20    return jsonify(users)

Python - Gunicorn (Wrong - Fork-Unsafe):

 1# ʕ•̀ω•́ʔ DON'T DO THIS
 2from pymongo import MongoClient
 3
 4# Created in master process - NOT fork-safe!
 5client = MongoClient(uri)
 6db = client['myapp']
 7
 8@app.route('/users')
 9def get_users():
10    # Child processes share parent's file descriptors = deadlocks
11    # This often results in AutoReconnect errors or 
12    # 'file descriptor bad' errors because the socket 
13    # is shared across memory boundaries
14    users = list(db.users.find({}))
15    return jsonify(users)

Java - Spring Boot:

 1spring:
 2  data:
 3    mongodb:
 4      uri: "mongodb://user:pass@host1:27017,host2:27017/?replicaSet=myRS&maxPoolSize=10&minPoolSize=2&maxIdleTimeMS=30000"
 5
 6// Or if using a Configurer Bean (safe way to customize without breaking auto-config)
 7@Configuration
 8public class MongoConfig {
 9    @Bean
10    public MongoClientSettingsBuilderCustomizer customizer() {
11        return builder -> builder.applyToConnectionPoolSettings(pool -> {
12            pool.maxSize(10);
13            pool.maxWaitTime(5, TimeUnit.SECONDS);
14        });
15    }
16}

The Health Check Trap:

ʕ•ᴥ•ʔ Warning: Do not create a new MongoClient inside your Kubernetes readiness/liveness probe. This is a classic DOS attack on your own database. The probe runs every 10s; if it creates a new connection each time without proper closure, you will exhaust the server limits in minutes. Use the singleton client to run a lightweight command like db.command({ ping: 1 }).

Monitoring Pool Metrics

Node.js - Command Monitoring:

 1const { MongoClient } = require('mongodb');
 2
 3const client = new MongoClient(uri, {
 4  monitorCommands: true  // Enable command monitoring
 5});
 6
 7client.on('commandStarted', (event) => {
 8  console.log('Command:', event.commandName);
 9});
10
11client.on('commandSucceeded', (event) => {
12  console.log('Duration:', event.duration, 'ms');
13});
14
15// For connection pool metrics, use pool monitoring events
16client.on('connectionCheckedOut', (event) => {
17  if (event.duration > 100) {  // Checkout took > 100ms
18    console.warn('Slow connection checkout:', event.duration, 'ms');
19  }
20});
21
22client.on('commandFailed', (event) => {
23  if (event.error.code === 'ConnectionPoolCheckoutFailed') {
24    console.error('Pool exhausted!');
25  }
26});

Go - Pool Monitoring:

 1import (
 2    "go.mongodb.org/mongo-driver/event"
 3    "go.mongodb.org/mongo-driver/mongo"
 4)
 5
 6poolMonitor := &event.PoolMonitor{
 7    Event: func(evt *event.PoolEvent) {
 8        switch evt.Type {
 9        case event.ConnectionCreated:
10            fmt.Println("Connection created")
11        case event.ConnectionCheckedOut:
12            fmt.Println("Connection checked out")
13        case event.ConnectionCheckoutFailed:
14            fmt.Println("Checkout failed:", evt.Reason)
15        case event.ConnectionReturned:
16            fmt.Println("Connection returned")
17        }
18    },
19}
20
21clientOptions := options.Client().
22    ApplyURI(uri).
23    SetPoolMonitor(poolMonitor).
24    SetMaxPoolSize(10)
25
26client, _ := mongo.Connect(context.TODO(), clientOptions)

Driver Nuances & Pitfalls

Node.js (Mongoose / Native Driver)

The Node driver is asynchronous, but the pool logic is strict.

• The Singleton Mistake: A common pattern in serverless or improper Express setup is calling MongoClient.connect() inside the route handler. This creates a new pool for every request, rapidly hitting the Atlas connection limit (e.g., 1,500 connections on M10). Always initialize the client once at startup.
• Unified Topology: The Node.js driver (v4.0+) uses Unified Topology by default (it’s the only option). This keeps a background monitoring thread that can detect step-downs. If your pool churns during a primary election, check if your heartbeatFrequencyMS is tuned correctly.

Python (PyMongo)

• Fork Safety: MongoClient is not fork-safe. If you use gunicorn or uwsgi with forking, you must create the client after the fork (e.g., inside the worker process initialization), or you will end up with child processes trying to use the parent’s file descriptors, leading to deadlocks and obscure socket errors. This often manifests as AutoReconnect errors or “file descriptor bad” errors.
• Wait Queue: In PyMongo 4.0+, waitQueueMultiple was removed. Use waitQueueTimeoutMS to control how long threads wait for connections. Limit your application’s thread pool size to control concurrency.

Java (Spring Boot / MongoDB Java Driver)

• Automatic Pooling: The MongoTemplate handles pooling automatically. Ensure spring.data.mongodb.database settings do not override defaults with aggressive timeouts. The Java driver’s default maxPoolSize is 100, which is often too high for production.

Go (mongo-go-driver)

• The Go driver respects context. If your Connect or Find context times out, the connection might be closed or returned to the pool depending on the stage.
• Monitor PoolEvent via the SetPoolMonitor option to visualize exactly when connections are created vs. checked out.

Serverless & Atlas Implications

In MongoDB Atlas, connection limits are hard tiers.

• M0 (Free Tier): 500 connections per node
• M10: 1,500 connections per node
• M20: 3,000 connections per node
• Serverless Instances: These bill by “Read Processing Units” but still require connection management.

If you are running on AWS Lambda or Vercel, you cannot maintain a persistent pool. Every “warm” container might keep a pool of 10 open. 100 concurrent Lambdas = 1,000 connections. Solution: You typically cannot use standard pooling here. You must aggressively close connections (set maxIdleTimeMS very low) or use the Atlas Data API (HTTP based) instead of the TCP driver to avoid socket exhaustion.

Monitoring & Diagnosis

How do you know it’s a pool issue and not a slow query?

1. Check the Server:

   1db.serverStatus().connections
   2// look for { "current": 1200, "available": 50 }
   

   If current is high, your app is leaking connections or scaling too wide.

2. Check the Queue: Monitor WiredTiger ticket usage via db.serverStatus().wiredTiger.concurrentTransactions or connection counts. If active connections significantly exceed your pool size or WiredTiger tickets are exhausted, your latency is due to queuing, not network speed.

3. Zombie Connections: When a client crashes hard (OOM Kill) without sending a TCP FIN, the server keeps the connection open until the server-side tcp_keepalive times out. In high-churn Kubernetes environments, this can exhaust the server’s file descriptors even if the active pool count seems low. This reinforces the need for aggressive maxIdleTimeMS and proper TCP keepalive configuration.

4. Driver Metrics: Enable command monitoring in your driver. If connectionCheckoutTime spikes while commandDuration (server execution time) stays low, the problem is client-side exhaustion. The database is fast; the line to get to it is slow.

5. Connection Churn Rate: Monitor the rate of new connections (connectionsCreated per second). If this is high (>10/sec per node) while your total pool count is stable, your maxIdleTimeMS is likely too low. You are burning server CPU on SSL handshakes rather than reusing connections. Check db.serverStatus().connections.created and compare it to your connection pool turnover rate.

Conclusion

In MongoDB, “Connection Pool Exhaustion” is often a misnomer for “Transaction Pinning” or “Overscaling.” Because the drivers are complex state machines interacting with Replica Sets, simply bumping maxPoolSize masks the problem until it crashes mongod via memory pressure or ticket exhaustion.

The fix is almost always:

1. Lower the pool size to match hardware (threads <= cores).
2. Shorten transaction scopes to microseconds.
3. Treat the MongoClient as a sacred singleton.