This post covers three CI/CD problems I solved with Docker Buildx, SHA-based image tagging, and dynamic workflow generation — and one thing I broke along the way.

Why latest is a lie

Container registries store images by tag. Tags are mutable pointers. When you docker push myapp:latest, you’re moving a pointer — the old image behind latest is now unreferenced. If two services share the tag latest or two people push to the same tag concurrently, you get a race condition on a mutable pointer. This is exactly the class of bug that version control systems solved decades ago.

The fundamental problem: image tags have no inherent ordering, no history, and no integrity guarantee. They’re a name, not a version.

The solution is to use a content-addressable identifier you already have: the Git commit SHA.

SHA-based image tagging

Every commit has a unique SHA-1 hash. Using it as the image tag creates an injective mapping from source code state → deployed artifact:

commit a1b2c3d → image registry/service:a1b2c3d → K8s pod spec

The implementation in GitHub Actions:

1- name: Derive image tag from commit
2  run: |
3    # Short SHA is sufficient — collision probability
4    # across 2^28 commits is negligible for any single repo
5    echo "IMAGE_TAG=${GITHUB_SHA::7}" >> $GITHUB_ENV

This gives you two things that latest never can:

1. Bijective traceability. Given a running pod, you can recover the exact source:

1$ kubectl get pod $POD -o jsonpath='{.spec.containers[0].image}'
2123456789.dkr.ecr.us-east-1.amazonaws.com/myservice:a1b2c3d
3
4$ git show a1b2c3d --stat

2. Immutable deployment records. The tag a1b2c3d either exists or it doesn’t. It can’t silently change meaning.

Propagating the tag to Kubernetes manifests

A tagged image is useless if the deployment spec still references latest. We close the loop by having CI update the manifest and commit the change:

 1- name: Patch deployment manifest
 2  run: |
 3    sed -i "s|image: $REGISTRY/$SERVICE:.*|image: $REGISTRY/$SERVICE:${IMAGE_TAG}|" \
 4      k8s/$ENVIRONMENT/${SERVICE}-deployment.yaml
 5
 6- name: Commit manifest update
 7  run: |
 8    git config user.name "ci-bot"
 9    git config user.email "ci-bot@noreply"
10    git add k8s/
11    git diff --cached --quiet || \
12      git commit -m "deploy($ENVIRONMENT): $SERVICE → $IMAGE_TAG"
13    git push

Now git log k8s/staging/ is your deployment history. Every deployment is a commit. Every rollback is git revert. This is the minimal viable GitOps pattern — your Git repo is the single source of truth for cluster state.

The git diff --cached --quiet guard prevents empty commits when nothing actually changed (e.g., the service wasn’t rebuilt).

Docker Buildx: layer caching in CI

The default docker build in CI has no cache. Every run downloads the base image, reinstalls all dependencies, copies all source files. For a Python service with 200+ packages, this is 8–12 minutes of work repeated identically on every push.

Docker Buildx provides two things that fix this: persistent builder instances and pluggable cache backends.

How Docker layer caching works

Docker images are a stack of layers. Each Dockerfile instruction creates a layer. Docker caches layers by their input hash — if the instruction and all preceding layers are identical, the cached layer is reused.

Layer 4: COPY . .                    ← changes every commit
Layer 3: RUN pip install -r req.txt  ← changes weekly
Layer 2: RUN apt-get install gcc     ← changes monthly
Layer 1: FROM python:3.11-slim       ← changes rarely

Cache invalidation is top-down: if Layer 2 changes, Layers 3 and 4 are invalidated too, even if requirements.txt hasn’t changed. This means Dockerfile instruction ordering directly determines cache efficiency.

The optimal ordering principle: sort instructions by change frequency, ascending. System deps → language deps → application source.

 1FROM python:3.11-slim AS base
 2WORKDIR /app
 3
 4# Layer 1: System dependencies (changes ~monthly)
 5RUN apt-get update && apt-get install -y --no-install-recommends \
 6    gcc libpq-dev && rm -rf /var/lib/apt/lists/*
 7
 8# Layer 2: Python dependencies (changes ~weekly)
 9COPY requirements.txt .
10RUN pip install --no-cache-dir -r requirements.txt
11
12# Layer 3: Application source (changes every commit)
13COPY . .
14
15CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

GitHub Actions cache backend

The key Buildx configuration:

 1- name: Set up Docker Buildx
 2  uses: docker/setup-buildx-action@v3
 3
 4- name: Build and push
 5  uses: docker/build-push-action@v5
 6  with:
 7    context: services/${{ matrix.service }}
 8    push: true
 9    tags: |
10      ${{ env.REGISTRY }}/${{ matrix.service }}:${{ env.IMAGE_TAG }}
11    cache-from: type=gha
12    cache-to: type=gha,mode=max

type=gha uses GitHub Actions’ built-in cache as the layer storage backend. This is important because CI runners are ephemeral — without an external cache backend, every run starts cold.

mode=max is critical and often missed. By default, Buildx only exports layers from the final stage. mode=max exports all intermediate layers, including those from multi-stage build stages. Without it, your pip install layer from a builder stage won’t be cached.

Under the hood, Buildx serializes each layer as a content-addressed blob and stores it via the GitHub Actions cache API. On subsequent runs, it checks the cache by layer digest before executing the instruction.

Cache efficiency in practice

The difference is dramatic:

The dominant cost axis shifts from build time to cache lookup time. For a source-only change, the only layer that rebuilds is COPY . ., which takes seconds.

Multi-stage builds and image size

If your final image includes build tools (gcc, python3-dev, header files), you’re shipping dead weight. Multi-stage builds separate the build environment from the runtime environment:

 1# Stage 1: Build (includes compilers, headers)
 2FROM python:3.11-slim AS builder
 3WORKDIR /build
 4RUN apt-get update && apt-get install -y gcc libffi-dev
 5COPY requirements.txt .
 6RUN pip install --no-cache-dir --prefix=/install -r requirements.txt
 7
 8# Stage 2: Runtime (minimal)
 9FROM python:3.11-slim
10WORKDIR /app
11COPY --from=builder /install /usr/local
12COPY . .
13CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

The --prefix=/install flag tells pip to install into a separate directory tree, which we then COPY --from=builder into the clean runtime image. The final image has no gcc, no header files, no pip cache.

Impact: image sizes dropped from 1.2GB to ~450MB. This matters for pull times during rollouts — a 450MB image pulls in ~15 seconds vs ~45 seconds for 1.2GB. During a rolling update across 6 replicas, that’s 3 minutes saved per deployment.

Dynamic service discovery in CI

Hardcoding a service list in your workflow is a maintenance hazard:

1# ❌ This will drift
2strategy:
3  matrix:
4    service: [auth, api, worker, processor]

When someone deletes processor/ and forgets to update the workflow, CI breaks. When someone adds a new service and forgets to add it, it never gets built. Both happen eventually.

Filesystem-driven matrix generation

Instead of declaring services, discover them:

 1jobs:
 2  discover:
 3    runs-on: ubuntu-latest
 4    outputs:
 5      services: ${{ steps.find.outputs.services }}
 6    steps:
 7      - uses: actions/checkout@v4
 8      - id: find
 9        run: |
10          services=$(find services/ -name Dockerfile -maxdepth 2 \
11            | xargs -I{} dirname {} \
12            | xargs -I{} basename {} \
13            | jq -R -s -c 'split("\n") | map(select(length > 0))')
14          echo "services=$services" >> $GITHUB_OUTPUT
15
16  build:
17    needs: discover
18    strategy:
19      matrix:
20        service: ${{ fromJson(needs.discover.outputs.services) }}

The contract becomes: a directory under services/ with a Dockerfile is a deployable service. No other registration needed. Add a service → add a directory with a Dockerfile. Remove a service → remove the directory.

Change detection: only build what’s affected

Even with caching, there’s no reason to build 8 services when 1 file changed. The git diff between the previous and current commit tells you what changed:

 1- uses: actions/checkout@v4
 2  with:
 3    fetch-depth: 0  # Required for accurate git diff
 4
 5- id: changes
 6  run: |
 7    changed=$(git diff --name-only ${{ github.event.before }} ${{ github.sha }})
 8    if echo "$changed" | grep -q "^services/${{ matrix.service }}/"; then
 9      echo "build=true" >> $GITHUB_OUTPUT
10    elif echo "$changed" | grep -q "^libs/shared/"; then
11      echo "build=true" >> $GITHUB_OUTPUT
12    else
13      echo "build=false" >> $GITHUB_OUTPUT
14    fi
15
16- name: Build and push
17  if: steps.changes.outputs.build == 'true'
18  uses: docker/build-push-action@v5
19  # ...

The fetch-depth: 0 is essential. GitHub Actions performs a shallow clone by default (fetch-depth: 1), which means git diff doesn’t have enough history and will treat everything as “new.” We discovered this when our changelogs started showing every file in the repo as changed.

The libs/shared/ check handles transitive dependencies — if a shared library changes, all services that could depend on it get rebuilt.

Edge cases that will bite you

ECR lifecycle policies

SHA tags accumulate. At 5 pushes/day × 8 services × ~800MB average image size, that’s 32GB/day of new images. Without cleanup, you’ll hit storage budget limits within weeks.

 1{
 2  "rules": [
 3    {
 4      "rulePriority": 1,
 5      "description": "Keep last 20 tagged images",
 6      "selection": {
 7        "tagStatus": "tagged",
 8        "countType": "imageCountMoreThan",
 9        "countNumber": 20
10      },
11      "action": { "type": "expire" }
12    },
13    {
14      "rulePriority": 2,
15      "description": "Expire untagged images after 24h",
16      "selection": {
17        "tagStatus": "untagged",
18        "countType": "sinceImagePushed",
19        "countUnit": "days",
20        "countNumber": 1
21      },
22      "action": { "type": "expire" }
23    }
24  ]
25}

20 tagged images gives you ~4 days of rollback runway at 5 pushes/day, which is plenty. Untagged images (intermediate layers from multi-stage builds) expire after 24 hours.

Concurrent manifest commits

If two workflow runs try to commit manifest changes simultaneously, the second git push fails. A retry loop with rebase handles this:

 1- name: Push manifest update
 2  run: |
 3    for attempt in 1 2 3; do
 4      git pull --rebase origin main
 5      git add k8s/
 6      git diff --cached --quiet && exit 0
 7      git commit -m "deploy: $SERVICE → $IMAGE_TAG"
 8      git push && exit 0
 9      echo "Push failed (attempt $attempt), retrying..."
10      sleep $((attempt * 2))
11    done
12    echo "::error::Manifest push failed after 3 attempts"
13    exit 1

The exponential backoff (sleep $((attempt * 2))) avoids thundering herd on concurrent pushes. In practice, this retries successfully on the first attempt 99% of the time.

Build context bloat

Docker sends the entire build context directory to the daemon before building. In a monorepo, that context can be enormous if you’re not careful. Each service needs a .dockerignore:

**/__pycache__
*.pyc
.git
.env
tests/
docs/
*.md
node_modules/

Without this, we were sending ~500MB of context for services that only needed ~20MB of source. The context upload alone added 2 minutes to every build.

Silent auth failures

ECR login can silently succeed without actually authenticating (e.g., when secrets aren’t available in fork PRs). The build proceeds, spends 10 minutes compiling, and only fails at docker push.

Add an explicit access check early:

1- name: Verify registry access
2  run: |
3    aws ecr describe-repositories \
4      --repository-names "${{ matrix.service }}" > /dev/null 2>&1 \
5    || { echo "::error::ECR access failed for ${{ matrix.service }}"; exit 1; }

Fail at second 5, not minute 12.

The pipeline end-to-end

git push origin main
        │
        ▼
┌──────────────┐
│   Discover   │  find services/ -name Dockerfile
│   Services   │  → ["auth", "api", "worker"]
└──────┬───────┘
       │
       ▼
┌─────────────────────────────────────────────────┐
│  Per-service matrix job (parallel)              │
│                                                 │
│  1. Checkout (fetch-depth: 0)                   │
│  2. git diff → should this service build?       │
│     └── no → skip remaining steps               │
│  3. Verify ECR access                           │
│  4. Set up Buildx                               │
│  5. Build + push (SHA tag, GHA layer cache)     │
│  6. sed manifest with new SHA                   │
│  7. git commit + push manifest (retry loop)     │
└──────┬──────────────────────────────────────────┘
       │
       ▼
┌──────────────┐
│  GitOps sync │  ArgoCD / Flux / kubectl apply
│  picks up    │  detects manifest diff → rolling update
└──────────────┘

Typical wall-clock time for a single-service source change: ~90 seconds from push to new pod running.

Key metrics

Checklist

• Tag with commit SHA, not latest. Mutable tags are mutable bugs.
• Order Dockerfile by change frequency. System deps → language deps → source. Cache invalidation is top-down.
• Use cache-from: type=gha with mode=max. Without mode=max, intermediate stage layers aren’t cached.
• fetch-depth: 0 on checkout. Shallow clones break git diff and changelog generation.
• Discover services from the filesystem. Hardcoded lists drift. Dockerfiles are the contract.
• Skip unchanged services. git diff + path matching. Don’t forget shared library paths as rebuild triggers.
• Set ECR lifecycle policies. 20 tagged images + 24h untagged expiry is a reasonable starting point.
• .dockerignore per service. Build context size ∝ upload time. Ship source, not tests.
• Verify registry auth early. Don’t spend 10 minutes building an image you can’t push.
• Retry manifest pushes. Concurrent CI runs will race on git push. Rebase + retry handles it.

This is the story of how I tracked down four distinct memory leaks across two Python services running on Kubernetes. Each one was a different flavor of “silent accumulation,” and each one taught me something about how easy it is to bleed memory in async Python without any obvious errors.

No names. No product details. Just the bugs, the fixes, and the lessons.

The setup

We had a multi-service platform running on Kubernetes. Two services are relevant here:

• Service A: A document management service. It handled CRUD operations, search indexing, audit logging, and authorization precomputation. Built with FastAPI and async Python, backed by MongoDB and a vector store.

• Service B: A PDF processing service. It parsed documents, extracted text via OCR, and broke content into structured clauses. Heavy on PyMuPDF (fitz) for PDF rendering and Tesseract for OCR.

Both services ran as single-container pods with resource limits. Service A had a 2Gi memory limit. Service B had 4Gi.

Both were getting OOMKilled. Regularly.

The symptoms

Service A’s memory graph looked like this:

Memory (MB)
2048 |                                              ╱ OOMKilled
     |                                           ╱
1536 |                                        ╱
     |                                     ╱
1024 |                                  ╱
     |                               ╱
 512 |                            ╱
     |                         ╱
   0 |________________________╱
     0h        6h        12h        18h        24h

A steady climb at roughly 200MB per hour. No spikes. No correlation with traffic. Just a ramp.

Service B was different - it’d spike during document processing batches and never come back down:

Memory (MB)
4096 |          ╱╲        ╱╲
     |        ╱   ╲     ╱   ╲    ╱ OOMKilled
3072 |      ╱      ╲  ╱      ╲ ╱
     |    ╱         ╲╱        ╱
2048 |  ╱                   ╱
     |╱                   ╱
1024 |                  ╱
     |               ╱
   0 |______________╱
     0h     2h     4h     6h     8h     10h

Every batch pushed memory up. It came down a little, but never to baseline. Classic leak with GC partially reclaiming some objects but not the real offender.

Leak #1: The coroutines that never ran

This was the most insidious one. It looked like perfectly normal Python code:

 1class BaseDoc:
 2    def _write_audit_log(self, action, old_data, new_data):
 3        enqueue_audit_create(self, action, old_data, new_data)
 4
 5    async def save(self, *args, **kwargs):
 6        old = await self.get(self.id)
 7        result = await super().save(*args, **kwargs)
 8        self._write_audit_log("update", old, result)
 9        await create_snapshots(self.id)
10        await insert_versions_docs(self.id)
11        return result

Spot the bug?

_write_audit_log calls enqueue_audit_create(), which is an async function. But _write_audit_log is not async, and it doesn’t await the call. It just calls it.

When you call an async function without await, Python creates a coroutine object but never schedules it. The coroutine sits in memory, holding references to all its arguments - in this case, entire document objects with old_data and new_data.

1Coroutine Object (0x7f...)
2  ├── State: CREATED (never scheduled)
3  └── Frame Object
4       ├── Arg: self (BaseDoc instance, 2KB)
5       ├── Arg: old_data (Dict, 5MB)  <-- LEAKED
6       └── Arg: new_data (Dict, 5MB)  <-- LEAKED
7       └── Local Vars: (captured closure scope)

Python will eventually warn you about this (RuntimeWarning: coroutine was never awaited), but only if you have warnings enabled and you’re looking at the right log. In a noisy production service running dozens of requests per second, these warnings get buried.

The same pattern appeared elsewhere:

 1# In the CRUD endpoints
 2async def create_document(request):
 3    doc = await Document.create(request.data)
 4    enqueue_audit_create(doc)       # ← async, not awaited
 5    qdrant_upserts(doc)             # ← async, not awaited
 6    return doc
 7
 8async def update_document(request):
 9    doc = await Document.update(request.data)
10    enqueue_audit_update(doc)       # ← async, not awaited
11    qdrant_upserts(doc)             # ← async, not awaited
12    return doc

Each unawaited call created a coroutine holding the full document. For a service processing hundreds of documents per hour, that’s hundreds of document-sized objects pinned in memory with no way to be garbage collected.

The fix

Straightforward once you see it:

 1class BaseDoc:
 2    async def _write_audit_log(self, action, old_data, new_data):
 3        await enqueue_audit_create(self, action, old_data, new_data)
 4
 5    async def save(self, *args, **kwargs):
 6        old = await self.get(self.id)
 7        result = await super().save(*args, **kwargs)
 8        await self._write_audit_log("update", old, result)
 9        await create_snapshots(self.id)
10        await insert_versions_docs(self.id)
11        return result

And in the CRUD endpoints:

1async def create_document(request):
2    doc = await Document.create(request.data)
3    await enqueue_audit_create(doc)
4    await qdrant_upserts(doc)
5    return doc

The lesson

Unawaited coroutines are silent memory leaks. They don’t crash. They don’t raise. They just accumulate. And because they hold references to their arguments, they can keep large objects alive long after the request that created them is done.

If you’re running async Python, add this to your linting:

1# In your pytest conftest.py or startup
2import warnings
3warnings.filterwarnings("error", category=RuntimeWarning, message="coroutine.*was never awaited")

This turns the warning into an exception, which makes it impossible to miss.

Leak #2: Creating a new MongoClient on every authorization check

The authorization layer needed to precompute user access contexts. To do this, it queried MongoDB directly (not through the async ODM) using PyMongo’s synchronous MongoClient.

The code looked like this:

 1class AuthorizationPrecompute:
 2    def get_user_permissions(self, user_id):
 3        client = MongoClient(MONGO_URI)
 4        db = client["main_db"]
 5        permissions = list(db.permissions.find({"user": user_id}))
 6        client.close()
 7        return permissions
 8
 9    def get_role_policies(self, role_id):
10        client = MongoClient(MONGO_URI)
11        db = client["main_db"]
12        policies = list(db.policies.find({"role": role_id}))
13        client.close()
14        return policies
15
16    def get_org_settings(self, org_id):
17        client = MongoClient(MONGO_URI)
18        db = client["main_db"]
19        settings = db.settings.find_one({"org": org_id})
20        client.close()
21        return settings
22
23    # ... 10 more methods, each creating its own MongoClient

Thirteen methods. Thirteen MongoClient() instantiations per authorization check.

Each MongoClient creates a connection pool, background SDAM monitoring threads, and internal caches. Even with client.close(), the cleanup isn’t instant - Python’s garbage collector has to reclaim the thread stacks, socket buffers, and internal data structures.

And if an exception happens between MongoClient() and client.close(), the client never gets closed at all. The connection pool lives on, holding threads and sockets until the GC eventually (maybe) collects them.

At 50 requests/second, each triggering an auth check, we were creating 650 MongoClient instances per second. Most were closed, but the transient memory pressure and leaked exceptions meant we were always carrying a few hundred orphaned connection pools.

The fix

One shared client with a connection pool:

 1class AuthorizationPrecompute:
 2    _client = None
 3
 4    @classmethod
 5    def _get_client(cls):
 6        if cls._client is None:
 7            cls._client = MongoClient(
 8                MONGO_URI,
 9                maxPoolSize=20,
10                serverSelectionTimeoutMS=5000
11            )
12        return cls._client
13
14    def get_user_permissions(self, user_id):
15        db = self._get_client()["main_db"]
16        return list(db.permissions.find({"user": user_id}))
17
18    def get_role_policies(self, role_id):
19        db = self._get_client()["main_db"]
20        return list(db.policies.find({"role": role_id}))

Connection overhead went from ~50ms per call (TCP handshake + auth each time) to ~5ms (reused connection from pool).

The lesson

Each MongoClient is an entire connection pool, not a single connection. Creating one per function call is like starting a new database server for every query. Always share a single client instance (or at most one per thread/worker). And if you can’t guarantee close() will be called (i.e., exceptions exist), you definitely need a shared instance.

Leak #3: Cache entries that never expire

The authorization layer also had an in-memory cache for user access contexts:

 1class UserAccessContextCache:
 2    def __init__(self, ttl_seconds=300):
 3        self._cache = {}
 4        self._ttl = ttl_seconds
 5
 6    def get(self, key):
 7        entry = self._cache.get(key)
 8        if entry and time.time() - entry["created"] < self._ttl:
 9            return entry["value"]
10        return None
11
12    def set(self, key, value):
13        self._cache[key] = {
14            "value": value,
15            "created": time.time()
16        }

See the problem?

When get() finds an expired entry, it just returns None. It doesn’t delete the entry. The stale data sits in the dict forever.

And set() only adds entries. Nothing ever removes them.

Over time, this cache grows without bound. Every unique user who hits the service gets a cache entry that is never cleaned up. The access context objects themselves can be large - they contain permission trees, role hierarchies, and org settings.

With 200MB of stale cache entries after a day of operation, this was a significant contributor to the memory ramp.

The fix

Add proactive cleanup:

 1class UserAccessContextCache:
 2    def __init__(self, ttl_seconds=300, cleanup_interval=60):
 3        self._cache = {}
 4        self._ttl = ttl_seconds
 5        self._cleanup_interval = cleanup_interval
 6        self._last_cleanup = time.time()
 7
 8    def _cleanup_expired(self):
 9        now = time.time()
10        if now - self._last_cleanup < self._cleanup_interval:
11            return
12        self._last_cleanup = now
13        expired_keys = [
14            k for k, v in self._cache.items()
15            if now - v["created"] >= self._ttl
16        ]
17        for k in expired_keys:
18            del self._cache[k]
19
20    def get(self, key):
21        self._cleanup_expired()
22        entry = self._cache.get(key)
23        if entry and time.time() - entry["created"] < self._ttl:
24            return entry["value"]
25        # Delete on read if expired
26        if key in self._cache:
27            del self._cache[key]
28        return None
29
30    def set(self, key, value):
31        self._cleanup_expired()
32        self._cache[key] = {
33            "value": value,
34            "created": time.time()
35        }

Cleanup runs at most once per minute and removes all entries older than TTL.

The lesson

Every in-memory cache needs an eviction strategy. If there’s no mechanism to remove old entries, you have a memory leak with extra steps. “TTL on read” is not enough - if users stop requesting a key, that entry stays forever.

Use cachetools.TTLCache or similar if you don’t want to roll your own. Better yet, if the cache is big enough to matter, put it in Redis where you can set EXPIRE and forget about it.

Leak #4: PyMuPDF documents that never closed

Service B was the PDF processing service. It used PyMuPDF (fitz) to render pages and extract text. The endpoints looked like this:

 1@app.post("/parse_pdf")
 2async def parse_pdf(file: UploadFile):
 3    content = await file.read()
 4    doc = fitz.open(stream=content, filetype="pdf")
 5
 6    pages = []
 7    for page_num in range(len(doc)):
 8        page = doc[page_num]
 9        text = page.get_text()
10        pix = page.get_pixmap(dpi=300)
11        image = pix.tobytes("png")
12        pages.append({"text": text, "image": image})
13
14    return {"pages": pages}

No doc.close(). No context manager. No try/finally.

PyMuPDF is a C library wrapper. When you call fitz.open(), it allocates memory in C-land for the document structure, page data, fonts, images, and rendering buffers. Python’s garbage collector can eventually reclaim the Python wrapper object, but the C-side memory is only freed when doc.close() is explicitly called.

While the wrapper theoretically has a __del__ method, relying on it in a high-throughput async loop is dangerous. Python’s GC is lazy and may not run fast enough to keep up with the rate of C-memory allocation, leading to OOMs before the GC even realizes it needs to clean up.

For a 50-page PDF at 300 DPI, the pixmap rendering alone can consume 200–500MB of memory. If an exception occurs mid-processing (corrupt page, unsupported font), the doc handle leaks entirely.

The fix

Two changes: proper resource cleanup, and parallel OCR with controlled concurrency.

 1@app.post("/parse_pdf")
 2async def parse_pdf(file: UploadFile):
 3    content = await file.read()
 4    doc = fitz.open(stream=content, filetype="pdf")
 5    try:
 6        pages = []
 7        for page_num in range(len(doc)):
 8            page = doc[page_num]
 9            text = page.get_text()
10            pix = page.get_pixmap(dpi=300)
11            image = pix.tobytes("png")
12            # Explicitly free the pixmap
13            pix = None
14            pages.append({"text": text, "image": image})
15        return {"pages": pages}
16    finally:
17        doc.close()

This was applied to all four endpoints in the service: /parse_pdf, /process_clauses, /process_clauses_sync, and /extract_links.

We also added parallel strip processing for OCR (controlled by feature flags):

 1import os
 2from concurrent.futures import ThreadPoolExecutor
 3
 4ENABLE_PARALLEL_OCR = os.getenv("ENABLE_PARALLEL_OCR", "false").lower() == "true"
 5STRIP_OCR_WORKERS = int(os.getenv("STRIP_OCR_WORKERS", "8"))
 6
 7def process_page_strips(page, strips):
 8    if not ENABLE_PARALLEL_OCR:
 9        return [ocr_strip(page, s) for s in strips]
10
11    with ThreadPoolExecutor(max_workers=STRIP_OCR_WORKERS) as pool:
12        futures = [pool.submit(ocr_strip, page, s) for s in strips]
13        return [f.result() for f in futures]

The deployment

We didn’t flip everything on at once:

1. Deploy with ENABLE_PARALLEL_OCR=false - just the doc.close() fixes
2. Monitor memory for 24 hours to confirm the leak is fixed
3. Bump resource limits from 4Gi to 6Gi to accommodate parallel processing headroom
4. Enable parallel OCR: ENABLE_PARALLEL_OCR=true

Memory dropped from a steady 4GB to a stable 1.8GB. That’s a 55% reduction from just closing document handles properly.

The lesson

C extension libraries manage their own memory. Python’s GC has no visibility into fitz, numpy, Pillow, or any other C-backed library. If the library provides a close(), use it. If it supports context managers, use with. And always wrap C-heavy processing in try/finally.

The combined impact

Here’s the before and after across both services:

Service A went from daily OOMKills to zero restarts. Service B went from crashing every 8–10 hours to running for weeks without a restart.

How I found them

No fancy tooling. Here’s the actual process:

1. Look at the memory graph. Linear growth = accumulation leak. Sawtooth that doesn’t return to baseline = handle/resource leak.

2. Read the code. Seriously. I read every function in the hot paths and asked “what gets allocated here that doesn’t get freed?”

3. grep for patterns. Once I found one unawaited coroutine, I searched for the pattern across the entire codebase:

1# Find async functions called without await
2rg "^\s+[a-z_]+\(" --type py | grep -v "await " | grep -v "def " | grep -v "#"

Not perfect, but it flagged the 13 MongoClient instantiations immediately.

4. Check warnings. Enabled RuntimeWarning filtering and got 6 unawaited coroutine warnings within the first minute of startup.

5. Kubernetes events. kubectl describe pod shows OOMKilled with the exact memory at death. Compare that to your resource limits and you know how fast it’s growing.

1kubectl get events --field-selector reason=OOMKilling \
2  --sort-by='.lastTimestamp' -n your-namespace

A checklist for your services

If you’re running Python services on Kubernetes and seeing memory growth, check these in order:

• Unawaited coroutines. Search for RuntimeWarning: coroutine.*was never awaited in your logs. Better yet, make it an error.
• MongoClient instantiation. grep -rn "MongoClient(" your_code/ - if it appears in more than one place (especially inside functions, not module level), you probably have a leak.
• In-memory caches without eviction. Any dict that grows with usage and has no del or size cap is a leak. TTLCache, LRU, or maxsize - pick one.
• C extension resources. PyMuPDF, Pillow, lxml, pdfplumber - anything that wraps C needs explicit close() or with statements.
• Large objects in exception handlers. If your except block captures the full traceback and logs the local variables, those locals (which may include entire documents) are pinned until the log entry is flushed.
• Background tasks holding request data. If asyncio.create_task() captures a closure over request objects, those objects live until the task completes.

Final thought

None of these bugs produced an error. No stack trace. No warning (well, one did, but it was buried). The services ran fine - they just got slower and slower until Kubernetes killed them, then they restarted and the cycle began again.

Memory leaks in garbage-collected languages are not the “forgot to call free()” kind. They’re the “I’m holding a reference I didn’t know about” kind. The GC is doing its job perfectly - it’s keeping alive exactly what you told it to.

You just told it wrong.