The whole stack runs on a budget VPS – 1 vCPU, 2.9GB RAM, 30GB disk, $22/year. All three services are running with room to spare.

Initial Setup

SSH key auth first, password auth off:

1# local: generate key if you don't have one
2ssh-keygen -t ed25519 -C "your_email@example.com"
3# paste ~/.ssh/id_ed25519.pub into your provider's SSH key field
4
5# on the server: disable password login
6sed -i 's/^PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
7systemctl restart sshd

Test key login before disabling password auth. If you lock yourself out, you’re recovering via the provider’s console.

Swap – critical on 3GB RAM:

1fallocate -l 2G /swapfile
2chmod 600 /swapfile
3mkswap /swapfile
4swapon /swapfile
5echo '/swapfile none swap sw 0 0' >> /etc/fstab

Without this, CouchDB and Conduit together will occasionally OOM under load.

Firewall:

1ufw allow ssh
2ufw allow 80/tcp
3ufw allow 443/tcp
4ufw allow 3478        # TURN (UDP+TCP)
5ufw allow 5349        # TURN TLS
6ufw allow 49152:65535/udp  # TURN relay range
7ufw enable

The TURN ports are for Matrix voice/video – skip them if you’re not setting up coturn.

Docker:

1curl -fsSL https://get.docker.com | sh

Service 1: Obsidian LiveSync (CouchDB)

LiveSync uses CouchDB as the sync backend. Install it directly on the host rather than in Docker – it’s simpler to configure and easier to troubleshoot.

1apt install -y couchdb

The installer prompts for admin credentials. Set them and keep them somewhere safe.

Configure /opt/couchdb/etc/local.ini:

 1[chttpd]
 2bind_address = 127.0.0.1
 3
 4[couch_httpd_auth]
 5require_valid_user = true
 6
 7[httpd]
 8enable_cors = true
 9
10[cors]
11origins = app://obsidian.md, capacitor://localhost, http://localhost
12credentials = true
13headers = accept, authorization, content-type, origin, referer
14methods = GET, PUT, POST, HEAD, DELETE
15max_age = 3600

1systemctl restart couchdb
2curl -X PUT http://admin:PASSWORD@localhost:5984/obsidian-sync

nginx config:

 1server {
 2    listen 80;
 3    server_name sync.your.domain;
 4    location / {
 5        proxy_pass http://127.0.0.1:5984;
 6        proxy_set_header Host $host;
 7        proxy_set_header X-Real-IP $remote_addr;
 8        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 9    }
10}

Get a cert: certbot --nginx -d sync.your.domain

In Obsidian: install the LiveSync community plugin, set the CouchDB URI to your domain, enter credentials, set a database name and an E2E passphrase. Enable sync. The passphrase encrypts vault content before it leaves your device – CouchDB never sees plaintext.

Service 2: Private Matrix Chat (Conduit + coturn)

Conduit is a Matrix homeserver written in Rust – much lighter than Synapse, which would be too heavy for 3GB RAM. I run it with federation disabled – this is a personal instance, not connected to the public Matrix network.

/opt/conduit/docker-compose.yml:

 1services:
 2  conduit:
 3    image: matrixconduit/matrix-conduit:latest
 4    restart: unless-stopped
 5    volumes:
 6      - conduit_data:/var/lib/matrix-conduit
 7    environment:
 8      CONDUIT_SERVER_NAME: "chat.your.domain"
 9      CONDUIT_DATABASE_BACKEND: "rocksdb"
10      CONDUIT_ALLOW_REGISTRATION: "true"      # flip to false after accounts created
11      CONDUIT_ALLOW_FEDERATION: "false"
12      CONDUIT_MAX_REQUEST_SIZE: "20000000"
13      CONDUIT_LOG: "warn"
14    ports:
15      - "127.0.0.1:6167:6167"
16
17volumes:
18  conduit_data:

1cd /opt/conduit && docker compose up -d

For voice and video calls, Matrix uses WebRTC – which needs a TURN server to traverse NAT. coturn handles this:

1apt install -y coturn
2openssl rand -hex 32   # generate a static auth secret, keep it

/etc/turnserver.conf:

listening-port=3478
tls-listening-port=5349
fingerprint
use-auth-secret
static-auth-secret=YOUR_SECRET
realm=chat.your.domain
cert=/etc/letsencrypt/live/chat.your.domain/fullchain.pem
pkey=/etc/letsencrypt/live/chat.your.domain/privkey.pem
min-port=49152
max-port=65535

nginx config:

 1server {
 2    listen 80;
 3    server_name chat.your.domain;
 4    location / {
 5        proxy_pass http://127.0.0.1:6167;
 6        proxy_set_header Host $host;
 7        proxy_set_header X-Real-IP $remote_addr;
 8        proxy_set_header Upgrade $http_upgrade;
 9        proxy_set_header Connection "upgrade";
10    }
11}

Once Conduit is up: go to Element Web, set homeserver to your domain, register your accounts while ALLOW_REGISTRATION is true, then flip it to false and restart the container. Anyone who finds your homeserver URL can register while it’s open.

Service 3: Calibre-Web

Calibre-Web gives you a browser-based library for EPUB, PDF, and MOBI files. The Docker image is from linuxserver.io.

Critical step before starting the container: Calibre-Web needs a properly initialized metadata.db. Not an empty file, not a directory – a valid Calibre library database. It must be created by calibredb.

1apt install -y calibre   # ~300MB, takes a few minutes
2mkdir -p /opt/calibre/books
3calibredb --with-library /opt/calibre/books list   # initializes metadata.db
4chown -R 1000:1000 /opt/calibre/books              # container runs as uid 1000
5chmod -R 755 /opt/calibre/books

Do this before starting the container. If you start it first and hit “invalid path” or “DB not writable”, this is why.

/opt/calibre/docker-compose.yml:

 1services:
 2  calibre-web:
 3    image: lscr.io/linuxserver/calibre-web:latest
 4    restart: unless-stopped
 5    environment:
 6      - PUID=1000
 7      - PGID=1000
 8      - TZ=Asia/Kolkata
 9    volumes:
10      - calibre_config:/config
11      - /opt/calibre/books:/books
12    ports:
13      - "127.0.0.1:8083:8083"
14
15volumes:
16  calibre_config:

nginx config:

1server {
2    listen 80;
3    server_name books.your.domain;
4    location / {
5        proxy_pass http://127.0.0.1:8083;
6        proxy_set_header Host $host;
7        proxy_set_header X-Real-IP $remote_addr;
8    }
9}

Default login is admin / admin123. Change it immediately. Set library path to /books.

Cloudflare in Front

All three subdomains go through Cloudflare. DNS A records point to the VPS IP, proxy status set to orange cloud (proxied) for all three. SSL mode set to Full – not Full Strict, since the Let’s Encrypt cert on origin is valid enough and Full Strict adds complexity for no real benefit here.

certbot handles the Let’s Encrypt certs on the origin:

1certbot --nginx -d sync.your.domain
2certbot --nginx -d chat.your.domain
3certbot --nginx -d books.your.domain

Resource Usage

All three services running simultaneously:

Swap barely touched. Disk at ~6GB of 30GB. Plenty of headroom for adding services.

Pitfalls

The whole thing took an afternoon to set up. $22/year for services I fully control and can inspect end to end.

This post covers what I set up, what broke, and what actually works now.

Starting point

The provider gives you a root password on first boot. First thing I did was generate an ed25519 SSH key pair locally, paste the public key into the VPS control panel, add a Host personal entry to ~/.ssh/config, and verify key auth works before touching anything else. Then disabling password login:

1sed -i 's/^PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
2systemctl restart sshd

Don’t close your existing SSH session before confirming you can open a new one with key auth. Classic footgun.

Before any services: add swap. 2.9GB of RAM is workable but tight once Docker gets involved.

1fallocate -l 2G /swapfile
2chmod 600 /swapfile
3mkswap /swapfile
4swapon /swapfile
5echo '/swapfile none swap sw 0 0' >> /etc/fstab

Then update packages, set up UFW to allow SSH, HTTP, HTTPS, and the TURN port range (3478, 5349, 49152:65535), and drop everything else.

After all that, the box looked like this:

OS: Ubuntu 24.04 LTS
RAM: 2.9GB (470MB used, 2.4GB free)
Disk: 30GB (2.4GB used, 26GB free)
Swap: 2GB (none used yet)
CPU: 1 core

Plenty of room to work with.

Service 1: Obsidian LiveSync

CouchDB as the backend for Obsidian’s LiveSync community plugin. The plugin syncs your vault across devices using a CouchDB instance you control. End-to-end encrypted via a passphrase set in the plugin – CouchDB stores ciphertext only.

Install CouchDB natively (not Docker – LiveSync requires specific CouchDB config that’s simpler to manage directly):

1apt install -y couchdb

The installer asks for admin credentials. Set them and remember them. Then configure CouchDB to accept external connections by editing /opt/couchdb/etc/local.ini:

 1[chttpd]
 2bind_address = 127.0.0.1  ; nginx will proxy, no need to expose directly
 3
 4[couch_httpd_auth]
 5require_valid_user = true
 6
 7[httpd]
 8enable_cors = true
 9
10[cors]
11origins = app://obsidian.md, capacitor://localhost, http://localhost
12credentials = true
13headers = accept, authorization, content-type, origin, referer
14methods = GET, PUT, POST, HEAD, DELETE
15max_age = 3600

Then create the sync database:

1curl -X PUT http://admin:password@localhost:5984/obsidian-sync

nginx proxies the sync subdomain to localhost:5984. Cloudflare sits in front with Full SSL mode. In the Obsidian LiveSync plugin settings, point it at your CouchDB subdomain, enter the credentials, set your E2E passphrase, and enable sync.

It just works after that. Notes push from Mac, pull on iPhone, no iCloud dependency.

Service 2: Private Matrix Chat

Conduit is a Matrix homeserver written in Rust. It’s small (62MB RAM), single-binary, and handles everything a private two-person server needs. Element Web as the client – works in any browser, no app required.

Docker Compose setup:

 1services:
 2  conduit:
 3    image: matrixconduit/matrix-conduit:latest
 4    restart: unless-stopped
 5    volumes:
 6      - conduit_data:/var/lib/matrix-conduit
 7    environment:
 8      CONDUIT_SERVER_NAME: "your.chat.domain"
 9      CONDUIT_DATABASE_BACKEND: "rocksdb"
10      CONDUIT_ALLOW_REGISTRATION: "true"
11      CONDUIT_ALLOW_FEDERATION: "false"
12      CONDUIT_MAX_REQUEST_SIZE: "20000000"
13      CONDUIT_TRUSTED_SERVERS: '["matrix.org"]'
14      CONDUIT_LOG: "warn"
15    ports:
16      - "127.0.0.1:6167:6167"
17
18volumes:
19  conduit_data:

ALLOW_FEDERATION: false is important for a private server. No traffic leaves your machine to federate with matrix.org or anyone else.

For Element Web, the easiest route is the hosted version at app.element.io – just point it at your homeserver URL when logging in. If you want it self-hosted too, serve the static build from nginx.

For voice and video calls, WebRTC needs a TURN server or calls fail behind NAT. coturn handles this:

1apt install -y coturn

Configure /etc/turnserver.conf:

listening-port=3478
tls-listening-port=5349
fingerprint
use-auth-secret
static-auth-secret=<generate with: openssl rand -hex 32>
realm=your.chat.domain
cert=/etc/letsencrypt/live/your.chat.domain/fullchain.pem
pkey=/etc/letsencrypt/live/your.chat.domain/privkey.pem

Add the TURN server details in Conduit’s config, then tell Element where to find it in the Synapse well-known file. After that, voice and video calls work reliably regardless of NAT.

Create user accounts via the Conduit admin API or by registering through Element while ALLOW_REGISTRATION: true. After both accounts are created, set ALLOW_REGISTRATION: false so no new accounts can be made.

Service 3: Personal Book Library

Calibre-Web in Docker. Serves EPUB, PDF, MOBI with a clean browser interface. Can read books directly in the browser without downloading.

The setup hit two non-obvious problems worth documenting.

Problem 1: Calibre-Web needs a real metadata.db

The container starts and asks for a library path. I pointed it at /books (mapped to /opt/calibre/books on the host). It said “invalid path.”

The issue is that Calibre-Web expects a valid Calibre library database (metadata.db) to already exist at that path. An empty directory is not enough. The database needs to have the correct schema – not just the file.

The fix is to install Calibre on the host and use calibredb to initialize the library properly:

1apt install -y calibre  # takes a while, ~300MB
2calibredb --with-library /opt/calibre/books list

The calibredb list call creates a properly initialized metadata.db if one doesn’t exist. After this, /books becomes a valid library path.

Problem 2: Permissions

Even after the database exists, Calibre-Web may refuse to write to it. The linuxserver/calibre-web Docker image runs as user 1000 internally. If the files on the host are owned by root, the container can’t write.

1chown -R 1000:1000 /opt/calibre/books
2chmod -R 755 /opt/calibre/books

After both fixes, the setup page accepted /books and the library came up clean.

Docker Compose:

 1services:
 2  calibre-web:
 3    image: lscr.io/linuxserver/calibre-web:latest
 4    restart: unless-stopped
 5    environment:
 6      - PUID=1000
 7      - PGID=1000
 8      - TZ=Asia/Kolkata
 9    volumes:
10      - calibre_config:/config
11      - /opt/calibre/books:/books
12    ports:
13      - "127.0.0.1:8083:8083"
14
15volumes:
16  calibre_config:

Default login after first start: admin / admin123. Change it immediately.

nginx and Cloudflare

All three services sit behind nginx. Each subdomain gets a server block that proxies to the local port:

1server {
2    listen 80;
3    server_name your.sync.domain;
4    location / {
5        proxy_pass http://127.0.0.1:5984;
6        proxy_set_header Host $host;
7        proxy_set_header X-Real-IP $remote_addr;
8    }
9}

Cloudflare handles DNS and acts as the TLS terminator in Full mode. Let’s Encrypt issues certs on each subdomain via certbot. The certs auto-renew – I won’t have to touch them.

Credentials stored in pass on the Mac, encrypted with a GPG key. No plaintext passwords in config files or notes.

Resource usage after all three services

Service          RAM
---------        ----
Calibre-Web      159MB
Conduit          62MB
CouchDB          ~80MB
nginx + system   ~453MB
---------        ------
Total            754MB / 2.9GB  (26% used)
Swap used        780KB
Disk used        ~6GB / 30GB

74% of RAM still free. The 2GB swap sits untouched. There’s room for more things.

What’s next

The daily note has some ideas: run Bifrost on the VPS and wire up a blog-writing pipeline, use the unlimited bandwidth for a VPN, set up a small agent. The RAM budget will tighten but there’s headroom to experiment.

For $22/year, the box is already earning its keep.

Tools: Ubuntu 24.04, Docker, nginx, Cloudflare, Let’s Encrypt, CouchDB, Conduit (Matrix), Element Web, Calibre-Web, coturn, UFW, pass + GPG, certbot.

References

• LandChad.net – practical self-hosting guides, covers nginx, certbot, and many services: https://landchad.net
• Conduit (Matrix homeserver in Rust): https://conduit.rs
• Obsidian LiveSync plugin: https://github.com/vrtmrz/obsidian-livesync
• Calibre-Web: https://github.com/janeczku/calibre-web
• coturn TURN server: https://github.com/coturn/coturn

This post covers three CI/CD problems I solved with Docker Buildx, SHA-based image tagging, and dynamic workflow generation — and one thing I broke along the way.

Why latest is a lie

Container registries store images by tag. Tags are mutable pointers. When you docker push myapp:latest, you’re moving a pointer — the old image behind latest is now unreferenced. If two services share the tag latest or two people push to the same tag concurrently, you get a race condition on a mutable pointer. This is exactly the class of bug that version control systems solved decades ago.

The fundamental problem: image tags have no inherent ordering, no history, and no integrity guarantee. They’re a name, not a version.

The solution is to use a content-addressable identifier you already have: the Git commit SHA.

SHA-based image tagging

Every commit has a unique SHA-1 hash. Using it as the image tag creates an injective mapping from source code state → deployed artifact:

commit a1b2c3d → image registry/service:a1b2c3d → K8s pod spec

The implementation in GitHub Actions:

1- name: Derive image tag from commit
2  run: |
3    # Short SHA is sufficient — collision probability
4    # across 2^28 commits is negligible for any single repo
5    echo "IMAGE_TAG=${GITHUB_SHA::7}" >> $GITHUB_ENV

This gives you two things that latest never can:

1. Bijective traceability. Given a running pod, you can recover the exact source:

1$ kubectl get pod $POD -o jsonpath='{.spec.containers[0].image}'
2123456789.dkr.ecr.us-east-1.amazonaws.com/myservice:a1b2c3d
3
4$ git show a1b2c3d --stat

2. Immutable deployment records. The tag a1b2c3d either exists or it doesn’t. It can’t silently change meaning.

Propagating the tag to Kubernetes manifests

A tagged image is useless if the deployment spec still references latest. We close the loop by having CI update the manifest and commit the change:

 1- name: Patch deployment manifest
 2  run: |
 3    sed -i "s|image: $REGISTRY/$SERVICE:.*|image: $REGISTRY/$SERVICE:${IMAGE_TAG}|" \
 4      k8s/$ENVIRONMENT/${SERVICE}-deployment.yaml
 5
 6- name: Commit manifest update
 7  run: |
 8    git config user.name "ci-bot"
 9    git config user.email "ci-bot@noreply"
10    git add k8s/
11    git diff --cached --quiet || \
12      git commit -m "deploy($ENVIRONMENT): $SERVICE → $IMAGE_TAG"
13    git push

Now git log k8s/staging/ is your deployment history. Every deployment is a commit. Every rollback is git revert. This is the minimal viable GitOps pattern — your Git repo is the single source of truth for cluster state.

The git diff --cached --quiet guard prevents empty commits when nothing actually changed (e.g., the service wasn’t rebuilt).

Docker Buildx: layer caching in CI

The default docker build in CI has no cache. Every run downloads the base image, reinstalls all dependencies, copies all source files. For a Python service with 200+ packages, this is 8–12 minutes of work repeated identically on every push.

Docker Buildx provides two things that fix this: persistent builder instances and pluggable cache backends.

How Docker layer caching works

Docker images are a stack of layers. Each Dockerfile instruction creates a layer. Docker caches layers by their input hash — if the instruction and all preceding layers are identical, the cached layer is reused.

Layer 4: COPY . .                    ← changes every commit
Layer 3: RUN pip install -r req.txt  ← changes weekly
Layer 2: RUN apt-get install gcc     ← changes monthly
Layer 1: FROM python:3.11-slim       ← changes rarely

Cache invalidation is top-down: if Layer 2 changes, Layers 3 and 4 are invalidated too, even if requirements.txt hasn’t changed. This means Dockerfile instruction ordering directly determines cache efficiency.

The optimal ordering principle: sort instructions by change frequency, ascending. System deps → language deps → application source.

 1FROM python:3.11-slim AS base
 2WORKDIR /app
 3
 4# Layer 1: System dependencies (changes ~monthly)
 5RUN apt-get update && apt-get install -y --no-install-recommends \
 6    gcc libpq-dev && rm -rf /var/lib/apt/lists/*
 7
 8# Layer 2: Python dependencies (changes ~weekly)
 9COPY requirements.txt .
10RUN pip install --no-cache-dir -r requirements.txt
11
12# Layer 3: Application source (changes every commit)
13COPY . .
14
15CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

GitHub Actions cache backend

The key Buildx configuration:

 1- name: Set up Docker Buildx
 2  uses: docker/setup-buildx-action@v3
 3
 4- name: Build and push
 5  uses: docker/build-push-action@v5
 6  with:
 7    context: services/${{ matrix.service }}
 8    push: true
 9    tags: |
10      ${{ env.REGISTRY }}/${{ matrix.service }}:${{ env.IMAGE_TAG }}
11    cache-from: type=gha
12    cache-to: type=gha,mode=max

type=gha uses GitHub Actions’ built-in cache as the layer storage backend. This is important because CI runners are ephemeral — without an external cache backend, every run starts cold.

mode=max is critical and often missed. By default, Buildx only exports layers from the final stage. mode=max exports all intermediate layers, including those from multi-stage build stages. Without it, your pip install layer from a builder stage won’t be cached.

Under the hood, Buildx serializes each layer as a content-addressed blob and stores it via the GitHub Actions cache API. On subsequent runs, it checks the cache by layer digest before executing the instruction.

Cache efficiency in practice

The difference is dramatic:

The dominant cost axis shifts from build time to cache lookup time. For a source-only change, the only layer that rebuilds is COPY . ., which takes seconds.

Multi-stage builds and image size

If your final image includes build tools (gcc, python3-dev, header files), you’re shipping dead weight. Multi-stage builds separate the build environment from the runtime environment:

 1# Stage 1: Build (includes compilers, headers)
 2FROM python:3.11-slim AS builder
 3WORKDIR /build
 4RUN apt-get update && apt-get install -y gcc libffi-dev
 5COPY requirements.txt .
 6RUN pip install --no-cache-dir --prefix=/install -r requirements.txt
 7
 8# Stage 2: Runtime (minimal)
 9FROM python:3.11-slim
10WORKDIR /app
11COPY --from=builder /install /usr/local
12COPY . .
13CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

The --prefix=/install flag tells pip to install into a separate directory tree, which we then COPY --from=builder into the clean runtime image. The final image has no gcc, no header files, no pip cache.

Impact: image sizes dropped from 1.2GB to ~450MB. This matters for pull times during rollouts — a 450MB image pulls in ~15 seconds vs ~45 seconds for 1.2GB. During a rolling update across 6 replicas, that’s 3 minutes saved per deployment.

Dynamic service discovery in CI

Hardcoding a service list in your workflow is a maintenance hazard:

1# ❌ This will drift
2strategy:
3  matrix:
4    service: [auth, api, worker, processor]

When someone deletes processor/ and forgets to update the workflow, CI breaks. When someone adds a new service and forgets to add it, it never gets built. Both happen eventually.

Filesystem-driven matrix generation

Instead of declaring services, discover them:

 1jobs:
 2  discover:
 3    runs-on: ubuntu-latest
 4    outputs:
 5      services: ${{ steps.find.outputs.services }}
 6    steps:
 7      - uses: actions/checkout@v4
 8      - id: find
 9        run: |
10          services=$(find services/ -name Dockerfile -maxdepth 2 \
11            | xargs -I{} dirname {} \
12            | xargs -I{} basename {} \
13            | jq -R -s -c 'split("\n") | map(select(length > 0))')
14          echo "services=$services" >> $GITHUB_OUTPUT
15
16  build:
17    needs: discover
18    strategy:
19      matrix:
20        service: ${{ fromJson(needs.discover.outputs.services) }}

The contract becomes: a directory under services/ with a Dockerfile is a deployable service. No other registration needed. Add a service → add a directory with a Dockerfile. Remove a service → remove the directory.

Change detection: only build what’s affected

Even with caching, there’s no reason to build 8 services when 1 file changed. The git diff between the previous and current commit tells you what changed:

 1- uses: actions/checkout@v4
 2  with:
 3    fetch-depth: 0  # Required for accurate git diff
 4
 5- id: changes
 6  run: |
 7    changed=$(git diff --name-only ${{ github.event.before }} ${{ github.sha }})
 8    if echo "$changed" | grep -q "^services/${{ matrix.service }}/"; then
 9      echo "build=true" >> $GITHUB_OUTPUT
10    elif echo "$changed" | grep -q "^libs/shared/"; then
11      echo "build=true" >> $GITHUB_OUTPUT
12    else
13      echo "build=false" >> $GITHUB_OUTPUT
14    fi
15
16- name: Build and push
17  if: steps.changes.outputs.build == 'true'
18  uses: docker/build-push-action@v5
19  # ...

The fetch-depth: 0 is essential. GitHub Actions performs a shallow clone by default (fetch-depth: 1), which means git diff doesn’t have enough history and will treat everything as “new.” We discovered this when our changelogs started showing every file in the repo as changed.

The libs/shared/ check handles transitive dependencies — if a shared library changes, all services that could depend on it get rebuilt.

Edge cases that will bite you

ECR lifecycle policies

SHA tags accumulate. At 5 pushes/day × 8 services × ~800MB average image size, that’s 32GB/day of new images. Without cleanup, you’ll hit storage budget limits within weeks.

 1{
 2  "rules": [
 3    {
 4      "rulePriority": 1,
 5      "description": "Keep last 20 tagged images",
 6      "selection": {
 7        "tagStatus": "tagged",
 8        "countType": "imageCountMoreThan",
 9        "countNumber": 20
10      },
11      "action": { "type": "expire" }
12    },
13    {
14      "rulePriority": 2,
15      "description": "Expire untagged images after 24h",
16      "selection": {
17        "tagStatus": "untagged",
18        "countType": "sinceImagePushed",
19        "countUnit": "days",
20        "countNumber": 1
21      },
22      "action": { "type": "expire" }
23    }
24  ]
25}

20 tagged images gives you ~4 days of rollback runway at 5 pushes/day, which is plenty. Untagged images (intermediate layers from multi-stage builds) expire after 24 hours.

Concurrent manifest commits

If two workflow runs try to commit manifest changes simultaneously, the second git push fails. A retry loop with rebase handles this:

 1- name: Push manifest update
 2  run: |
 3    for attempt in 1 2 3; do
 4      git pull --rebase origin main
 5      git add k8s/
 6      git diff --cached --quiet && exit 0
 7      git commit -m "deploy: $SERVICE → $IMAGE_TAG"
 8      git push && exit 0
 9      echo "Push failed (attempt $attempt), retrying..."
10      sleep $((attempt * 2))
11    done
12    echo "::error::Manifest push failed after 3 attempts"
13    exit 1

The exponential backoff (sleep $((attempt * 2))) avoids thundering herd on concurrent pushes. In practice, this retries successfully on the first attempt 99% of the time.

Build context bloat

Docker sends the entire build context directory to the daemon before building. In a monorepo, that context can be enormous if you’re not careful. Each service needs a .dockerignore:

**/__pycache__
*.pyc
.git
.env
tests/
docs/
*.md
node_modules/

Without this, we were sending ~500MB of context for services that only needed ~20MB of source. The context upload alone added 2 minutes to every build.

Silent auth failures

ECR login can silently succeed without actually authenticating (e.g., when secrets aren’t available in fork PRs). The build proceeds, spends 10 minutes compiling, and only fails at docker push.

Add an explicit access check early:

1- name: Verify registry access
2  run: |
3    aws ecr describe-repositories \
4      --repository-names "${{ matrix.service }}" > /dev/null 2>&1 \
5    || { echo "::error::ECR access failed for ${{ matrix.service }}"; exit 1; }

Fail at second 5, not minute 12.

The pipeline end-to-end

git push origin main
        │
        ▼
┌──────────────┐
│   Discover   │  find services/ -name Dockerfile
│   Services   │  → ["auth", "api", "worker"]
└──────┬───────┘
       │
       ▼
┌─────────────────────────────────────────────────┐
│  Per-service matrix job (parallel)              │
│                                                 │
│  1. Checkout (fetch-depth: 0)                   │
│  2. git diff → should this service build?       │
│     └── no → skip remaining steps               │
│  3. Verify ECR access                           │
│  4. Set up Buildx                               │
│  5. Build + push (SHA tag, GHA layer cache)     │
│  6. sed manifest with new SHA                   │
│  7. git commit + push manifest (retry loop)     │
└──────┬──────────────────────────────────────────┘
       │
       ▼
┌──────────────┐
│  GitOps sync │  ArgoCD / Flux / kubectl apply
│  picks up    │  detects manifest diff → rolling update
└──────────────┘

Typical wall-clock time for a single-service source change: ~90 seconds from push to new pod running.

Key metrics

Checklist

• Tag with commit SHA, not latest. Mutable tags are mutable bugs.
• Order Dockerfile by change frequency. System deps → language deps → source. Cache invalidation is top-down.
• Use cache-from: type=gha with mode=max. Without mode=max, intermediate stage layers aren’t cached.
• fetch-depth: 0 on checkout. Shallow clones break git diff and changelog generation.
• Discover services from the filesystem. Hardcoded lists drift. Dockerfiles are the contract.
• Skip unchanged services. git diff + path matching. Don’t forget shared library paths as rebuild triggers.
• Set ECR lifecycle policies. 20 tagged images + 24h untagged expiry is a reasonable starting point.
• .dockerignore per service. Build context size ∝ upload time. Ship source, not tests.
• Verify registry auth early. Don’t spend 10 minutes building an image you can’t push.
• Retry manifest pushes. Concurrent CI runs will race on git push. Rebase + retry handles it.

This is the story of how I tracked down four distinct memory leaks across two Python services running on Kubernetes. Each one was a different flavor of “silent accumulation,” and each one taught me something about how easy it is to bleed memory in async Python without any obvious errors.

No names. No product details. Just the bugs, the fixes, and the lessons.

The setup

We had a multi-service platform running on Kubernetes. Two services are relevant here:

• Service A: A document management service. It handled CRUD operations, search indexing, audit logging, and authorization precomputation. Built with FastAPI and async Python, backed by MongoDB and a vector store.

• Service B: A PDF processing service. It parsed documents, extracted text via OCR, and broke content into structured clauses. Heavy on PyMuPDF (fitz) for PDF rendering and Tesseract for OCR.

Both services ran as single-container pods with resource limits. Service A had a 2Gi memory limit. Service B had 4Gi.

Both were getting OOMKilled. Regularly.

The symptoms

Service A’s memory graph looked like this:

Memory (MB)
2048 |                                              ╱ OOMKilled
     |                                           ╱
1536 |                                        ╱
     |                                     ╱
1024 |                                  ╱
     |                               ╱
 512 |                            ╱
     |                         ╱
   0 |________________________╱
     0h        6h        12h        18h        24h

A steady climb at roughly 200MB per hour. No spikes. No correlation with traffic. Just a ramp.

Service B was different - it’d spike during document processing batches and never come back down:

Memory (MB)
4096 |          ╱╲        ╱╲
     |        ╱   ╲     ╱   ╲    ╱ OOMKilled
3072 |      ╱      ╲  ╱      ╲ ╱
     |    ╱         ╲╱        ╱
2048 |  ╱                   ╱
     |╱                   ╱
1024 |                  ╱
     |               ╱
   0 |______________╱
     0h     2h     4h     6h     8h     10h

Every batch pushed memory up. It came down a little, but never to baseline. Classic leak with GC partially reclaiming some objects but not the real offender.

Leak #1: The coroutines that never ran

This was the most insidious one. It looked like perfectly normal Python code:

 1class BaseDoc:
 2    def _write_audit_log(self, action, old_data, new_data):
 3        enqueue_audit_create(self, action, old_data, new_data)
 4
 5    async def save(self, *args, **kwargs):
 6        old = await self.get(self.id)
 7        result = await super().save(*args, **kwargs)
 8        self._write_audit_log("update", old, result)
 9        await create_snapshots(self.id)
10        await insert_versions_docs(self.id)
11        return result

Spot the bug?

_write_audit_log calls enqueue_audit_create(), which is an async function. But _write_audit_log is not async, and it doesn’t await the call. It just calls it.

When you call an async function without await, Python creates a coroutine object but never schedules it. The coroutine sits in memory, holding references to all its arguments - in this case, entire document objects with old_data and new_data.

1Coroutine Object (0x7f...)
2  ├── State: CREATED (never scheduled)
3  └── Frame Object
4       ├── Arg: self (BaseDoc instance, 2KB)
5       ├── Arg: old_data (Dict, 5MB)  <-- LEAKED
6       └── Arg: new_data (Dict, 5MB)  <-- LEAKED
7       └── Local Vars: (captured closure scope)

Python will eventually warn you about this (RuntimeWarning: coroutine was never awaited), but only if you have warnings enabled and you’re looking at the right log. In a noisy production service running dozens of requests per second, these warnings get buried.

The same pattern appeared elsewhere:

 1# In the CRUD endpoints
 2async def create_document(request):
 3    doc = await Document.create(request.data)
 4    enqueue_audit_create(doc)       # ← async, not awaited
 5    qdrant_upserts(doc)             # ← async, not awaited
 6    return doc
 7
 8async def update_document(request):
 9    doc = await Document.update(request.data)
10    enqueue_audit_update(doc)       # ← async, not awaited
11    qdrant_upserts(doc)             # ← async, not awaited
12    return doc

Each unawaited call created a coroutine holding the full document. For a service processing hundreds of documents per hour, that’s hundreds of document-sized objects pinned in memory with no way to be garbage collected.

The fix

Straightforward once you see it:

 1class BaseDoc:
 2    async def _write_audit_log(self, action, old_data, new_data):
 3        await enqueue_audit_create(self, action, old_data, new_data)
 4
 5    async def save(self, *args, **kwargs):
 6        old = await self.get(self.id)
 7        result = await super().save(*args, **kwargs)
 8        await self._write_audit_log("update", old, result)
 9        await create_snapshots(self.id)
10        await insert_versions_docs(self.id)
11        return result

And in the CRUD endpoints:

1async def create_document(request):
2    doc = await Document.create(request.data)
3    await enqueue_audit_create(doc)
4    await qdrant_upserts(doc)
5    return doc

The lesson

Unawaited coroutines are silent memory leaks. They don’t crash. They don’t raise. They just accumulate. And because they hold references to their arguments, they can keep large objects alive long after the request that created them is done.

If you’re running async Python, add this to your linting:

1# In your pytest conftest.py or startup
2import warnings
3warnings.filterwarnings("error", category=RuntimeWarning, message="coroutine.*was never awaited")

This turns the warning into an exception, which makes it impossible to miss.

Leak #2: Creating a new MongoClient on every authorization check

The authorization layer needed to precompute user access contexts. To do this, it queried MongoDB directly (not through the async ODM) using PyMongo’s synchronous MongoClient.

The code looked like this:

 1class AuthorizationPrecompute:
 2    def get_user_permissions(self, user_id):
 3        client = MongoClient(MONGO_URI)
 4        db = client["main_db"]
 5        permissions = list(db.permissions.find({"user": user_id}))
 6        client.close()
 7        return permissions
 8
 9    def get_role_policies(self, role_id):
10        client = MongoClient(MONGO_URI)
11        db = client["main_db"]
12        policies = list(db.policies.find({"role": role_id}))
13        client.close()
14        return policies
15
16    def get_org_settings(self, org_id):
17        client = MongoClient(MONGO_URI)
18        db = client["main_db"]
19        settings = db.settings.find_one({"org": org_id})
20        client.close()
21        return settings
22
23    # ... 10 more methods, each creating its own MongoClient

Thirteen methods. Thirteen MongoClient() instantiations per authorization check.

Each MongoClient creates a connection pool, background SDAM monitoring threads, and internal caches. Even with client.close(), the cleanup isn’t instant - Python’s garbage collector has to reclaim the thread stacks, socket buffers, and internal data structures.

And if an exception happens between MongoClient() and client.close(), the client never gets closed at all. The connection pool lives on, holding threads and sockets until the GC eventually (maybe) collects them.

At 50 requests/second, each triggering an auth check, we were creating 650 MongoClient instances per second. Most were closed, but the transient memory pressure and leaked exceptions meant we were always carrying a few hundred orphaned connection pools.

The fix

One shared client with a connection pool:

 1class AuthorizationPrecompute:
 2    _client = None
 3
 4    @classmethod
 5    def _get_client(cls):
 6        if cls._client is None:
 7            cls._client = MongoClient(
 8                MONGO_URI,
 9                maxPoolSize=20,
10                serverSelectionTimeoutMS=5000
11            )
12        return cls._client
13
14    def get_user_permissions(self, user_id):
15        db = self._get_client()["main_db"]
16        return list(db.permissions.find({"user": user_id}))
17
18    def get_role_policies(self, role_id):
19        db = self._get_client()["main_db"]
20        return list(db.policies.find({"role": role_id}))

Connection overhead went from ~50ms per call (TCP handshake + auth each time) to ~5ms (reused connection from pool).

The lesson

Each MongoClient is an entire connection pool, not a single connection. Creating one per function call is like starting a new database server for every query. Always share a single client instance (or at most one per thread/worker). And if you can’t guarantee close() will be called (i.e., exceptions exist), you definitely need a shared instance.

Leak #3: Cache entries that never expire

The authorization layer also had an in-memory cache for user access contexts:

 1class UserAccessContextCache:
 2    def __init__(self, ttl_seconds=300):
 3        self._cache = {}
 4        self._ttl = ttl_seconds
 5
 6    def get(self, key):
 7        entry = self._cache.get(key)
 8        if entry and time.time() - entry["created"] < self._ttl:
 9            return entry["value"]
10        return None
11
12    def set(self, key, value):
13        self._cache[key] = {
14            "value": value,
15            "created": time.time()
16        }

See the problem?

When get() finds an expired entry, it just returns None. It doesn’t delete the entry. The stale data sits in the dict forever.

And set() only adds entries. Nothing ever removes them.

Over time, this cache grows without bound. Every unique user who hits the service gets a cache entry that is never cleaned up. The access context objects themselves can be large - they contain permission trees, role hierarchies, and org settings.

With 200MB of stale cache entries after a day of operation, this was a significant contributor to the memory ramp.

The fix

Add proactive cleanup:

 1class UserAccessContextCache:
 2    def __init__(self, ttl_seconds=300, cleanup_interval=60):
 3        self._cache = {}
 4        self._ttl = ttl_seconds
 5        self._cleanup_interval = cleanup_interval
 6        self._last_cleanup = time.time()
 7
 8    def _cleanup_expired(self):
 9        now = time.time()
10        if now - self._last_cleanup < self._cleanup_interval:
11            return
12        self._last_cleanup = now
13        expired_keys = [
14            k for k, v in self._cache.items()
15            if now - v["created"] >= self._ttl
16        ]
17        for k in expired_keys:
18            del self._cache[k]
19
20    def get(self, key):
21        self._cleanup_expired()
22        entry = self._cache.get(key)
23        if entry and time.time() - entry["created"] < self._ttl:
24            return entry["value"]
25        # Delete on read if expired
26        if key in self._cache:
27            del self._cache[key]
28        return None
29
30    def set(self, key, value):
31        self._cleanup_expired()
32        self._cache[key] = {
33            "value": value,
34            "created": time.time()
35        }

Cleanup runs at most once per minute and removes all entries older than TTL.

The lesson

Every in-memory cache needs an eviction strategy. If there’s no mechanism to remove old entries, you have a memory leak with extra steps. “TTL on read” is not enough - if users stop requesting a key, that entry stays forever.

Use cachetools.TTLCache or similar if you don’t want to roll your own. Better yet, if the cache is big enough to matter, put it in Redis where you can set EXPIRE and forget about it.

Leak #4: PyMuPDF documents that never closed

Service B was the PDF processing service. It used PyMuPDF (fitz) to render pages and extract text. The endpoints looked like this:

 1@app.post("/parse_pdf")
 2async def parse_pdf(file: UploadFile):
 3    content = await file.read()
 4    doc = fitz.open(stream=content, filetype="pdf")
 5
 6    pages = []
 7    for page_num in range(len(doc)):
 8        page = doc[page_num]
 9        text = page.get_text()
10        pix = page.get_pixmap(dpi=300)
11        image = pix.tobytes("png")
12        pages.append({"text": text, "image": image})
13
14    return {"pages": pages}

No doc.close(). No context manager. No try/finally.

PyMuPDF is a C library wrapper. When you call fitz.open(), it allocates memory in C-land for the document structure, page data, fonts, images, and rendering buffers. Python’s garbage collector can eventually reclaim the Python wrapper object, but the C-side memory is only freed when doc.close() is explicitly called.

While the wrapper theoretically has a __del__ method, relying on it in a high-throughput async loop is dangerous. Python’s GC is lazy and may not run fast enough to keep up with the rate of C-memory allocation, leading to OOMs before the GC even realizes it needs to clean up.

For a 50-page PDF at 300 DPI, the pixmap rendering alone can consume 200–500MB of memory. If an exception occurs mid-processing (corrupt page, unsupported font), the doc handle leaks entirely.

The fix

Two changes: proper resource cleanup, and parallel OCR with controlled concurrency.

 1@app.post("/parse_pdf")
 2async def parse_pdf(file: UploadFile):
 3    content = await file.read()
 4    doc = fitz.open(stream=content, filetype="pdf")
 5    try:
 6        pages = []
 7        for page_num in range(len(doc)):
 8            page = doc[page_num]
 9            text = page.get_text()
10            pix = page.get_pixmap(dpi=300)
11            image = pix.tobytes("png")
12            # Explicitly free the pixmap
13            pix = None
14            pages.append({"text": text, "image": image})
15        return {"pages": pages}
16    finally:
17        doc.close()

This was applied to all four endpoints in the service: /parse_pdf, /process_clauses, /process_clauses_sync, and /extract_links.

We also added parallel strip processing for OCR (controlled by feature flags):

 1import os
 2from concurrent.futures import ThreadPoolExecutor
 3
 4ENABLE_PARALLEL_OCR = os.getenv("ENABLE_PARALLEL_OCR", "false").lower() == "true"
 5STRIP_OCR_WORKERS = int(os.getenv("STRIP_OCR_WORKERS", "8"))
 6
 7def process_page_strips(page, strips):
 8    if not ENABLE_PARALLEL_OCR:
 9        return [ocr_strip(page, s) for s in strips]
10
11    with ThreadPoolExecutor(max_workers=STRIP_OCR_WORKERS) as pool:
12        futures = [pool.submit(ocr_strip, page, s) for s in strips]
13        return [f.result() for f in futures]

The deployment

We didn’t flip everything on at once:

1. Deploy with ENABLE_PARALLEL_OCR=false - just the doc.close() fixes
2. Monitor memory for 24 hours to confirm the leak is fixed
3. Bump resource limits from 4Gi to 6Gi to accommodate parallel processing headroom
4. Enable parallel OCR: ENABLE_PARALLEL_OCR=true

Memory dropped from a steady 4GB to a stable 1.8GB. That’s a 55% reduction from just closing document handles properly.

The lesson

C extension libraries manage their own memory. Python’s GC has no visibility into fitz, numpy, Pillow, or any other C-backed library. If the library provides a close(), use it. If it supports context managers, use with. And always wrap C-heavy processing in try/finally.

The combined impact

Here’s the before and after across both services:

Service A went from daily OOMKills to zero restarts. Service B went from crashing every 8–10 hours to running for weeks without a restart.

How I found them

No fancy tooling. Here’s the actual process:

1. Look at the memory graph. Linear growth = accumulation leak. Sawtooth that doesn’t return to baseline = handle/resource leak.

2. Read the code. Seriously. I read every function in the hot paths and asked “what gets allocated here that doesn’t get freed?”

3. grep for patterns. Once I found one unawaited coroutine, I searched for the pattern across the entire codebase:

1# Find async functions called without await
2rg "^\s+[a-z_]+\(" --type py | grep -v "await " | grep -v "def " | grep -v "#"

Not perfect, but it flagged the 13 MongoClient instantiations immediately.

4. Check warnings. Enabled RuntimeWarning filtering and got 6 unawaited coroutine warnings within the first minute of startup.

5. Kubernetes events. kubectl describe pod shows OOMKilled with the exact memory at death. Compare that to your resource limits and you know how fast it’s growing.

1kubectl get events --field-selector reason=OOMKilling \
2  --sort-by='.lastTimestamp' -n your-namespace

A checklist for your services

If you’re running Python services on Kubernetes and seeing memory growth, check these in order:

• Unawaited coroutines. Search for RuntimeWarning: coroutine.*was never awaited in your logs. Better yet, make it an error.
• MongoClient instantiation. grep -rn "MongoClient(" your_code/ - if it appears in more than one place (especially inside functions, not module level), you probably have a leak.
• In-memory caches without eviction. Any dict that grows with usage and has no del or size cap is a leak. TTLCache, LRU, or maxsize - pick one.
• C extension resources. PyMuPDF, Pillow, lxml, pdfplumber - anything that wraps C needs explicit close() or with statements.
• Large objects in exception handlers. If your except block captures the full traceback and logs the local variables, those locals (which may include entire documents) are pinned until the log entry is flushed.
• Background tasks holding request data. If asyncio.create_task() captures a closure over request objects, those objects live until the task completes.

Final thought

None of these bugs produced an error. No stack trace. No warning (well, one did, but it was buried). The services ran fine - they just got slower and slower until Kubernetes killed them, then they restarted and the cycle began again.

Memory leaks in garbage-collected languages are not the “forgot to call free()” kind. They’re the “I’m holding a reference I didn’t know about” kind. The GC is doing its job perfectly - it’s keeping alive exactly what you told it to.

You just told it wrong.

A coworker said: “I ran task generation, closed the tab, and the status never moved.”

My first reaction was “no way.” The backend is doing the work, not the browser.

Turns out… yeah. It was right. And once you see why, you stop trusting “code after the stream” forever.

This post is about a very specific trap:

• you stream progress with SSE from a FastAPI endpoint
• you put important side effects after the streaming loop (DB status updates, poller kickoff, Slack notifications)
• a client disconnect cancels the stream
• your “after the loop” code never runs

It doesn’t always fail. It fails just enough to make you doubt your own system.

What the system looked like (so you don’t think this is a demo problem)

We had a step-based pipeline. Each step was triggered by an HTTP call from the frontend and did a few things:

1. Receive a request like POST /process-timeline-done (or similar)
2. Start a long-ish worker (in our case an AI agent, but it could be anything)
3. Stream progress back to the UI via Server‑Sent Events
4. Update state in MongoDB (advance a status field)
5. Submit an external job
6. Start a long poller to watch that job (hours sometimes)
7. Send a Slack notification

Only (3) needs a live HTTP connection. The rest should be able to finish even if the user refreshes or closes the tab.

We had code split into things like:

The bug was that our implementation accidentally made a bunch of “No” things depend on HTTP anyway.

The bug (the exact pattern)

This is the shape of the broken code:

1# ❌ Broken: critical work is after streaming.
2async def process_step():
3    async for update in agent():
4        yield sse(update)  # disconnect can cancel/raise here
5
6    # If the client disconnects, this may never run.
7    await db.update_status("tasks_done")
8    asyncio.create_task(start_polling(job_id))
9    await slack.notify("done")

It feels reasonable because it matches how we think:

“Stream updates while running… then when the loop ends, do the completion stuff.”

But in a streaming response, the loop ending is not guaranteed. The client can just leave.

What actually happens when the client disconnects

Let’s make it concrete with a timeline.

Normal happy path:

Browser                  FastAPI/Starlette           MongoDB / External
   |                           |                          |
   | POST /process-step        |                          |
   |-------------------------->|                          |
   | 200 OK (SSE stream)       |                          |
   |<--------------------------|                          |
   | data: update #1           |                          |
   |<--------------------------|                          |
   | data: update #2           |                          |
   |<--------------------------|                          |
   | ...                       |                          |
   | data: final               |                          |
   |<--------------------------|                          |
   | stream ends               |                          |
   |                           | update status            |
   |                           |------------------------->|
   |                           | start poller             |
   |                           |------------------------->|
   |                           | Slack notify             |
   |                           |------------------------->|

Disconnect path:

Browser                  FastAPI/Starlette           MongoDB / External
   |                           |                          |
   | POST /process-step        |                          |
   |-------------------------->|                          |
   | 200 OK (SSE stream)       |                          |
   |<--------------------------|                          |
   | data: update #1           |                          |
   |<--------------------------|                          |
   | data: update #2           |                          |
   |<--------------------------|                          |
   | [tab closed]              |                          |
   X                           |                          |
                               | stream task canceled     |
                               | (right on a yield/send)  |
                               |                          |
                               | ❌ status update skipped  |
                               | ❌ poller never started   |
                               | ❌ Slack never sent       |

The punchline is simple: SSE streaming is not “just a loop.” It’s a loop being consumed by a response writer that is allowed to stop at any time.

Under the hood (why Starlette cancels your generator)

This is not the exact Starlette source code, but it matches the design.

Streaming responses typically run two things concurrently:

• a task that iterates your generator and sends chunks (http.response.body)
• a task that listens for http.disconnect

When the disconnect happens, the server cancels the streaming task. That cancellation can hit while the framework is trying to send your last chunk… which was produced by your generator… which means your generator gets torn down mid-flight.

Two practical details that matter a lot:

1. Cancellation shows up as asyncio.CancelledError (or sometimes as “write failed” type errors depending on stack).
2. In modern Python, asyncio.CancelledError is a BaseException, not an Exception. So except Exception: won’t catch it.

That second one is why “but I wrapped it in try/except” doesn’t save you.

A minimal repro you can run in two minutes

Here’s a tiny app that demonstrates the exact failure mode. The print() after the loop is your “update MongoDB status” in disguise.

 1import asyncio
 2from fastapi import FastAPI
 3from starlette.responses import StreamingResponse
 4
 5app = FastAPI()
 6
 7async def gen():
 8    for i in range(1000):
 9        yield f"data: {i}\n\n".encode()
10        await asyncio.sleep(0.2)
11    print("stream ended; now do side effects")
12
13@app.get("/sse")
14async def sse():
15    return StreamingResponse(gen(), media_type="text/event-stream")

Run it, then:

1curl -N http://localhost:8000/sse

Wait for a few numbers, hit ctrl+c in the curl terminal.

You will not see the “now do side effects” line.

That’s the bug.

The real fix: stop making your job depend on the HTTP stream

Once you accept “the client can vanish at any time”, the design gets boring (in a good way):

• The job runs to completion in a background task.
• Streaming is just a best-effort view of progress.

I like to think in terms of ownership:

• The worker owns side effects (DB writes, poller, Slack).
• The request handler owns only streaming bytes to the client.

What I tried first (and why it didn’t help): BackgroundTasks

FastAPI has BackgroundTasks. It’s useful, but it’s not a magic “disconnect-proof” button.

The catch is simple: background tasks are executed after the response is done.

For a normal JSON response, “response is done” means “we returned a body, the server wrote it, done.”

For an SSE response, “response is done” means “the stream ended.”

Which is… exactly the thing we can’t rely on.

If the client disconnects and the streaming response gets canceled, your “run this after the response” work is still hanging off the same lifecycle. It’s not the separation you actually want.

“Can’t I just move the status update into finally?”

This is another tempting fix:

1async def gen():
2    try:
3        async for update in agent():
4            yield sse(update)
5    finally:
6        await db.update_status("tasks_done")

Sometimes it works. Sometimes it doesn’t. It depends on where cancellation lands.

Cancellation can interrupt awaits inside finally too. So you end up with a fun situation where you thought you made it reliable, but you just changed the probability of it failing.

If you really want to run cleanup under cancellation, you need to start thinking about shielding and cancel scopes. At that point you’re already halfway to the “worker owns completion” design anyway, so I’d rather just do it properly.

The queue pattern (with one important correction)

The common implementation is “worker pushes updates into a queue, streamer drains it.”

That’s fine, but there’s a nasty gotcha:

If the client disconnects and the streamer stops draining, a bounded queue can block your worker forever.

So don’t do this in your worker:

1await queue.put(update)  # ❌ can hang if consumer is gone

If you do, you just recreated the original bug: completion is again tied to streaming, just indirectly.

What I do instead:

• Keep the queue bounded (to protect memory)
• Put updates in a non-blocking way and drop when full
• Stop queueing progress once the client is gone

Here’s a practical sketch:

 1import asyncio
 2import json
 3import logging
 4
 5logger = logging.getLogger(__name__)
 6
 7def sse(data: dict) -> bytes:
 8    return f"data: {json.dumps(data)}\n\n".encode("utf-8")
 9
10def try_put(queue: asyncio.Queue, item) -> None:
11    try:
12        queue.put_nowait(item)
13    except asyncio.QueueFull:
14        # progress is optional; completion isn't
15        pass
16
17def log_task_result(task: asyncio.Task) -> None:
18    try:
19        task.result()
20    except asyncio.CancelledError:
21        pass
22    except Exception:
23        logger.exception("background worker crashed")
24
25async def run_to_completion(queue: asyncio.Queue, client_gone: asyncio.Event):
26    job_id = None
27    try:
28        async for update in agent():
29            job_id = job_id or update.get("job_id")
30            if not client_gone.is_set():
31                try_put(queue, update)
32
33        # ✅ These must run even if client disconnects.
34        await db.update_status("tasks_done")
35        asyncio.create_task(start_polling(job_id))
36        await slack.notify("done")
37    except Exception as e:
38        if not client_gone.is_set():
39            try_put(queue, {"final": True, "success": False, "error": str(e)})
40        raise
41    finally:
42        if not client_gone.is_set():
43            try_put(queue, None)  # sentinel for streamer
44
45async def stream(queue: asyncio.Queue, client_gone: asyncio.Event):
46    try:
47        while True:
48            item = await queue.get()
49            if item is None:
50                break
51            yield sse(item)
52    except asyncio.CancelledError:
53        # disconnected: stop streaming, signal worker to stop enqueueing
54        client_gone.set()
55        raise
56    finally:
57        client_gone.set()
58
59async def handler():
60    queue = asyncio.Queue(maxsize=200)
61    client_gone = asyncio.Event()
62
63    task = asyncio.create_task(run_to_completion(queue, client_gone))
64    task.add_done_callback(log_task_result)
65
66    return StreamingResponse(stream(queue, client_gone), media_type="text/event-stream")

Is it perfect? No. But it has the right shape:

• worker can finish without a consumer
• progress streaming is best-effort
• failures are logged instead of silently disappearing

If you want something more durable (client reconnects and can replay progress), you need to store progress somewhere real (Redis/pubsub, DB, etc.). In-memory queues won’t do that. But that’s a different problem.

The architecture I like better (when this gets serious)

The queue pattern is fine when:

• you mostly care about “don’t lose completion when the client disconnects”
• you don’t care about reconnecting and replaying a progress history

But once you ship this to real users, you’ll hit two common problems:

1. Browsers reconnect SSE automatically (especially EventSource), and you don’t want a reconnect to re-run the job.
2. People refresh and expect to see progress again (or at least not lose the final result).

So the more robust shape is:

1. POST /steps/{step}/start → returns a run_id
2. Worker runs independent of any SSE connection and persists state (status + maybe progress)
3. GET /runs/{run_id}/events → SSE stream that is purely “subscribe”, not “start”

Now reconnecting is safe. You might lose some live updates, but you can always read the current status from MongoDB, and you can keep polling running even if nobody is watching.

If you want a really clean UX, you can also persist event ids and support Last-Event-ID, but that’s optional. The big win is separating “start work” from “watch work.”

SSE details that are boring but matter

SSE is “simple” in the same way DNS is “simple.”

It works great until you put it behind real infrastructure.

Event formatting is strict

Each event ends with a blank line. If you forget the blank line, the browser buffers and you’ll think streaming is broken.

Simplest possible event:

1data: {"message":"hi"}

(Yes, that extra newline is part of it.)

Keepalives

If your worker can go quiet for a while, send a comment every ~15–30 seconds so proxies don’t decide the connection is “idle”:

1yield b": keep-alive\n\n"

Nginx buffering

If your stream arrives in one big chunk at the end, it’s usually Nginx buffering.

You don’t fix buffering with Python. You fix it with config.

At minimum, you want buffering off for the SSE location and timeouts that won’t murder long streams.

Something like this (not gospel, but you get the idea):

1location /runs/ {
2  proxy_http_version 1.1;
3  proxy_set_header Connection "";
4  proxy_buffering off;
5  proxy_cache off;
6  proxy_read_timeout 3600s;
7  add_header X-Accel-Buffering no;
8}

The other gotchas (the stuff that actually bites later)

Don’t use request-scoped dependencies in your background worker

If you have Depends(get_db) where get_db is a yield dependency, it gets cleaned up when the request is done.

If your worker is still running, it may now be holding a dead session/client.

Solution: the worker should acquire its own long-lived clients (Mongo, HTTP client, etc.) that are not tied to the request lifecycle.

Treat status updates like a state machine

If you do timeline_done → tasks_done → final_done, don’t just write tasks_done unconditionally.

Do “only update if current status is timeline_done” (compare-and-set). Otherwise retries and races will eventually mess up your state.

Don’t confuse “survives client disconnect” with “durable”

This pattern survives a client disconnect. It does not survive:

• the process dying
• the pod being evicted
• a deploy that SIGTERMs the worker mid-step

If the job truly must finish no matter what, use a real job system. SSE should subscribe to a job id, not be the job.

Decide if disconnect should cancel the job (explicitly)

Some people will ask for “closing the tab cancels the job.”

That’s a separate feature. Don’t accidentally implement it via “oops, the client disconnected so the worker got canceled.”

If you want cancellation, do it explicitly:

• create a job id
• store cancel intent
• have the worker check it and stop cleanly

Client disconnect is not a reliable cancel signal. It’s just the network being the network.

A boring checklist (the stuff I verify before I trust the fix)

• Start a step, then close the tab mid-stream → status still advances later
• Start a step, refresh rapidly → no duplicate runs / no broken state transitions
• Simulate a proxy timeout → completion still happens
• Kill the SSE client (curl -N ... then ctrl+c) → completion still happens
• Worker crashes mid-way → status reflects failure, and it doesn’t get stuck “in progress” forever

Performance notes (because someone will ask)

The queue approach is usually cheap, but it’s not free.

• Memory: queue_size * avg_event_size per in-flight stream. Keep event payloads small.
• CPU: queue ops are basically noise; the expensive bits are your DB/network calls.
• DB load: if you add read-after-write verification (I sometimes do), it’s extra reads. Worth it if your system lies to you otherwise.

The bigger risk isn’t CPU. It’s “we launched 500 background tasks because 500 people clicked the button.” Put a cap somewhere.

Deployment strategy (how I roll it out without getting yelled at)

1. Staging: disconnect tests, refresh tests, and “kill the client” tests.
2. Canary: small percentage of traffic. Watch for stuck statuses.
3. Full rollout: watch completion rate for a day. If it drops, roll back fast.

How I tested it (and how I’d recommend you test it)

Nothing fancy. I just did the thing that used to break it:

• start the step
• wait for a couple SSE updates
• close the tab / refresh / kill the connection
• watch MongoDB and server logs

What I wanted to see is boring:

• status advances anyway
• poller starts anyway
• Slack fires anyway

Once you watch the system finish without a client attached, you stop trusting “after the stream” code forever.

And honestly you should. It’s a trap.

The Problem: Cloudflare 521 Errors

If you’re running Nginx behind Cloudflare and seeing 521 errors, you likely have a mismatch between your Cloudflare SSL mode and what your origin server supports.

Common scenario:

• Nginx is only listening on port 80 (HTTP)
• Cloudflare SSL mode is set to “Full” or “Full (Strict)”
• Cloudflare tries to connect to your origin on port 443
• Origin doesn’t respond → 521 Error

Understanding Cloudflare SSL/TLS Modes

Cloudflare offers four SSL modes. Understanding these is crucial:

Key differences:

• Flexible: Quick fix if your origin can’t do HTTPS, but traffic between Cloudflare and your server is unencrypted — vulnerable to MITM attacks on that leg.

• Full: Encrypts both legs, but accepts any certificate (even self-signed, expired, or attacker-provided). Provides encryption but not authentication.

• Full (Strict): The gold standard. Cloudflare validates that your origin certificate is either:

  • A valid public CA certificate
  • A Cloudflare Origin Certificate

  This prevents MITM attacks where an attacker presents a fake certificate.

Architecture Overview

Our target architecture:

┌──────────────────────────────────────────────────────────────────────┐
│                          INTERNET                                    │
└──────────────────────────┬───────────────────────────────────────────┘
                           │ HTTPS (443)
                           ▼
┌──────────────────────────────────────────────────────────────────────┐
│                     CLOUDFLARE EDGE                                  │
│              (SSL Termination + CDN + WAF)                           │
└──────────────────────────┬───────────────────────────────────────────┘
                           │ HTTPS (443) - Origin Certificate
                           ▼
┌──────────────────────────────────────────────────────────────────────┐
│                    YOUR EC2 INSTANCE                                 │
│  ┌─────────────────────────────────────────────────────────────────┐ │
│  │                    DOCKER NETWORK                               │ │
│  │                                                                 │ │
│  │   ┌─────────────┐    HTTP    ┌─────────────────────────────┐    │ │
│  │   │   NGINX     │───────────▶│     APPLICATION CONTAINERS  │    │ │
│  │   │  (SSL/443)  │            │  - Frontend (Next.js:3000)  │    │ │
│  │   │  (80→443)   │            │  - Backend (FastAPI:8000)   │    │ │
│  │   └─────────────┘            │  - Cloud APIs               │    │ │
│  │         ▲                    └─────────────────────────────┘    │ │
│  │         │                                                       │ │
│  └─────────┼───────────────────────────────────────────────────────┘ │
│            │                                                         │
│     Ports 80, 443                                                    │
└──────────────────────────────────────────────────────────────────────┘

Traffic flow:

1. Browser connects to Cloudflare over HTTPS (public SSL certificate)
2. Cloudflare connects to your origin over HTTPS (Origin Certificate)
3. Nginx terminates SSL and proxies to backend containers over HTTP
4. Backend containers communicate on internal Docker network (isolated)

Prerequisites

Before starting, ensure you have:

• A domain proxied through Cloudflare (orange cloud enabled)
• Docker and Docker Compose installed on your server
• An existing Nginx configuration (we’ll modify it)
• Access to Cloudflare dashboard
• Firewall Rules: Ensure ports 80 and 443 are open in your cloud provider’s firewall (AWS Security Group, etc.)

Step 1: Generate a Cloudflare Origin Certificate

Cloudflare Origin Certificates are free, trusted only by Cloudflare, and can be valid for up to 15 years.

1. Log into Cloudflare Dashboard
2. Select your domain
3. Navigate to SSL/TLS → Origin Server
4. Click Create Certificate
5. Configure:
   • Private key type: RSA (2048)
   • Hostnames: *.yourdomain.com, yourdomain.com
   • Validity: 15 years (recommended)
6. Click Create

⚠️ IMPORTANT: Copy both the Origin Certificate (PEM) and Private Key immediately. The private key is shown only once!

Save them to files:

 1# Create SSL directory
 2mkdir -p ssl
 3
 4# Paste the certificate
 5cat > ssl/origin.pem << 'EOF'
 6-----BEGIN CERTIFICATE-----
 7YOUR_CERTIFICATE_HERE
 8-----END CERTIFICATE-----
 9EOF
10
11# Paste the private key
12cat > ssl/origin.key << 'EOF'
13-----BEGIN PRIVATE KEY-----
14YOUR_PRIVATE_KEY_HERE
15-----END PRIVATE KEY-----
16EOF
17
18# Set secure permissions
19chmod 600 ssl/origin.key
20chmod 644 ssl/origin.pem

Add to .gitignore to never commit these:

1echo "ssl/*" >> .gitignore

Step 2: Configure Nginx for HTTPS

Here’s a complete Nginx configuration with SSL, security headers, rate limiting, and reverse proxy setup.

nginx/nginx-main.conf - Main HTTP block configuration:

 1# Main nginx configuration with security settings
 2user nginx;
 3worker_processes auto;
 4error_log /var/log/nginx/error.log warn;
 5pid /var/run/nginx.pid;
 6
 7events {
 8    worker_connections 1024;
 9    use epoll;
10    multi_accept on;
11}
12
13http {
14    include /etc/nginx/mime.types;
15    default_type application/octet-stream;
16
17    # Logging format with security information
18    log_format security '$real_ip - $remote_user [$time_local] "$request" '
19                       '$status $body_bytes_sent "$http_referer" '
20                       '"$http_user_agent" "$http_x_forwarded_for" '
21                       'rt=$request_time uct="$upstream_connect_time" '
22                       'uht="$upstream_header_time" urt="$upstream_response_time"';
23
24    access_log /var/log/nginx/access.log security;
25
26    # Pro Tip: use $real_ip in log_format so you see actual visitor IPs, not Cloudflare's.
27
28    # Basic settings
29    sendfile on;
30    tcp_nopush on;
31    tcp_nodelay on;
32    keepalive_timeout 65;
33    types_hash_max_size 2048;
34    server_tokens off;  # Hide nginx version
35
36    # Gzip compression
37    gzip on;
38    gzip_vary on;
39    gzip_proxied any;
40    gzip_comp_level 6;
41    gzip_types text/plain text/css text/xml text/javascript 
42               application/json application/javascript application/xml+rss;
43
44    # Rate limiting zones
45    limit_req_zone $binary_remote_addr zone=general_limit:10m rate=50r/s;
46    limit_req_zone $binary_remote_addr zone=api_limit:10m rate=30r/s;
47    limit_req_zone $binary_remote_addr zone=burst_limit:10m rate=200r/m;
48    limit_conn_zone $binary_remote_addr zone=conn_limit:10m;
49
50    include /etc/nginx/conf.d/*.conf;
51}

nginx/nginx.conf - Server block configuration with SSL:

  1# Cloudflare IP handling - extract real client IP
  2map $http_cf_connecting_ip $real_ip {
  3    "" $remote_addr;
  4    default $http_cf_connecting_ip;
  5}
  6
  7# Cloudflare protocol handling
  8map $http_x_forwarded_proto $forwarded_proto {
  9    "" $scheme;
 10    default $http_x_forwarded_proto;
 11}
 12
 13# HTTP Server (Port 80) - Redirect to HTTPS
 14server {
 15    listen 80;
 16    server_name _;
 17
 18    # Health check (keep on port 80 for internal monitoring)
 19    location /health {
 20        access_log off;
 21        return 200 "healthy\n";
 22        add_header Content-Type text/plain;
 23    }
 24
 25    # Redirect all other traffic to HTTPS
 26    location / {
 27        return 301 https://$host$request_uri;
 28    }
 29}
 30
 31# HTTPS Server (Port 443)
 32server {
 33    listen 443 ssl http2;
 34    server_name _;
 35
 36    # SSL Configuration - Cloudflare Origin Certificate
 37    ssl_certificate /etc/nginx/ssl/origin.pem;
 38    ssl_certificate_key /etc/nginx/ssl/origin.key;
 39    
 40    # Modern SSL settings
 41    ssl_protocols TLSv1.2 TLSv1.3;
 42    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384;
 43    ssl_prefer_server_ciphers off;
 44    ssl_session_timeout 1d;
 45    ssl_session_cache shared:SSL:10m;
 46    ssl_session_tickets off;
 47
 48    # Connection and request limits
 49    limit_conn conn_limit 100;
 50    client_max_body_size 50M;
 51    client_body_buffer_size 4M;
 52
 53    # Timeout limits (prevent slowloris attacks)
 54    client_body_timeout 10s;
 55    client_header_timeout 10s;
 56    send_timeout 10s;
 57
 58    # Security headers
 59    add_header X-Frame-Options "SAMEORIGIN" always;
 60    add_header X-Content-Type-Options "nosniff" always;
 61    add_header X-XSS-Protection "1; mode=block" always;
 62    add_header Referrer-Policy "strict-origin-when-cross-origin" always;
 63    add_header Permissions-Policy "geolocation=(), microphone=(), camera=()" always;
 64    add_header Content-Security-Policy "default-src 'self'; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-inline'; img-src 'self' data: https:; font-src 'self' data:; connect-src 'self' https:; frame-src 'self';" always;
 65
 66    # Frontend - Next.js
 67    location / {
 68        limit_req zone=general_limit burst=100 nodelay;
 69        
 70        proxy_pass http://frontend:3000;
 71        proxy_http_version 1.1;
 72        proxy_set_header Upgrade $http_upgrade;
 73        proxy_set_header Connection 'upgrade';
 74        proxy_set_header Host $host;
 75        proxy_set_header X-Real-IP $real_ip;
 76        proxy_set_header X-Forwarded-For $real_ip;
 77        proxy_set_header X-Forwarded-Proto $forwarded_proto;
 78        proxy_cache_bypass $http_upgrade;
 79    }
 80
 81    # Backend API
 82    location /api {
 83        limit_req zone=api_limit burst=50 nodelay;
 84        
 85        proxy_pass http://backend:8000;
 86        proxy_http_version 1.1;
 87        proxy_set_header Host $host;
 88        proxy_set_header X-Real-IP $real_ip;
 89        proxy_set_header X-Forwarded-For $real_ip;
 90        proxy_set_header X-Forwarded-Proto $forwarded_proto;
 91        
 92        # Increased timeouts for long-running requests
 93        proxy_connect_timeout 300s;
 94        proxy_send_timeout 300s;
 95        proxy_read_timeout 300s;
 96    }
 97
 98    # Health check
 99    location /health {
100        access_log off;
101        return 200 "healthy\n";
102        add_header Content-Type text/plain;
103    }
104}

Docker DNS Caveat: Nginx caches DNS records at startup. If your backend container restarts and gets a new IP, Nginx might start returning 502 errors. To prevent this, use Docker’s internal resolver resolver 127.0.0.11 valid=30s; and variables for proxy_pass targets.

Step 3: Update Dockerfile

Update your Nginx Dockerfile to expose both ports:

nginx/Dockerfile:

 1FROM nginx:alpine
 2
 3# Copy configurations
 4COPY nginx-main.conf /etc/nginx/nginx.conf
 5COPY nginx.conf /etc/nginx/conf.d/default.conf
 6
 7# Expose HTTP and HTTPS ports
 8EXPOSE 80 443
 9
10CMD ["nginx", "-g", "daemon off;"]

Step 4: Update Docker Compose

Mount the SSL certificates and expose port 443:

docker-compose.yml (nginx service section):

 1services:
 2  nginx:
 3    build:
 4      context: ./nginx
 5      dockerfile: Dockerfile
 6    container_name: my-nginx
 7    restart: unless-stopped
 8    ports:
 9      - "80:80"
10      - "443:443"
11    depends_on:
12      - frontend
13      - backend
14    networks:
15      - app-network
16    volumes:
17      - ./ssl:/etc/nginx/ssl:ro  # Read-only mount
18    healthcheck:
19      test: ["CMD", "curl", "-f", "http://localhost/health"]
20      interval: 30s
21      timeout: 10s
22      retries: 3
23
24networks:
25  app-network:
26    driver: bridge

Step 5: Deploy

Rebuild and deploy only the nginx container (zero downtime for other services):

1# Build the new nginx image
2docker compose build nginx
3
4# Recreate nginx container without touching dependencies
5docker compose up -d --no-deps nginx
6
7# Verify nginx is running
8docker compose logs nginx --tail=20

If you have a deploy script like prod.sh, you might need:

1# If container name conflicts exist
2docker stop my-nginx && docker rm my-nginx
3./prod.sh -d nginx

Step 6: Enable Full (Strict) Mode in Cloudflare

Now that your origin supports HTTPS with a valid Origin Certificate:

1. Go to Cloudflare Dashboard → SSL/TLS → Overview
2. Change encryption mode from Flexible or Full to Full (Strict)

This change takes effect immediately.

Step 7: Verify

Test the connection:

1curl -I https://yourdomain.com

Expected output shows HTTP/2 200 with Cloudflare and security headers:

HTTP/2 200 
server: cloudflare
x-frame-options: SAMEORIGIN
x-content-type-options: nosniff
content-security-policy: default-src 'self'...
cf-ray: xxxxxxxx-XXX

Troubleshooting

521 Error (Web server is down)

• Origin not listening on port 443
• Firewall blocking port 443
• Nginx failed to start (check certificate paths)

1# Check nginx logs
2docker compose logs nginx
3
4# Verify nginx config syntax
5docker compose exec nginx nginx -t

525 Error (SSL handshake failed)

• Certificate/key mismatch
• Invalid certificate format
• Wrong file permissions

1# Verify certificate and key match
2openssl x509 -noout -modulus -in ssl/origin.pem | openssl md5
3openssl rsa -noout -modulus -in ssl/origin.key | openssl md5
4# Both should output the same MD5 hash

526 Error (Invalid SSL certificate)

• Only occurs in Full (Strict) mode
• Origin certificate not trusted by Cloudflare
• Certificate expired or for wrong hostname

1# Check certificate details
2openssl x509 -in ssl/origin.pem -text -noout | grep -A2 "Subject:"
3openssl x509 -in ssl/origin.pem -text -noout | grep -A2 "Validity"

Security Considerations

What This Setup Protects Against

Additional Hardening (Optional)

1. Authenticated Origin Pulls: Mutual TLS where Cloudflare presents a client certificate

   1ssl_client_certificate /etc/nginx/ssl/cloudflare-origin-pull.pem;
   2ssl_verify_client on;

2. Restrict Origin to Cloudflare IPs Only: Configure AWS Security Groups or iptables to only accept connections from Cloudflare IP ranges

3. Enable HSTS in Cloudflare: SSL/TLS → Edge Certificates → Always Use HTTPS + HSTS

Quick Reference

Trade-offs and Limitations

1. Vendor Lock-in: Cloudflare Origin Certificates are not trusted by major browsers, only by Cloudflare. This means you cannot bypass Cloudflare and connect directly to your origin server IP in a browser without getting severe security warnings. If you ever move away from Cloudflare, you will need to replace these certificates with standard ones (e.g., Let’s Encrypt).

2. Local Development: Since these certificates are for specific public domains, you can’t easily use them for localhost development. A common pattern is to have a separate docker-compose.override.yml for local development that overrides the Nginx config to listen on HTTP only or uses mkcert for locally trusted self-signed certificates.

Conclusion

With this setup, you now have:

✅ End-to-end encryption from browser to origin
✅ MITM protection on all network segments
✅ Zero-cost SSL with 15-year Cloudflare Origin Certificate
✅ Rate limiting and security headers
✅ Docker-based deployment for easy management

The key insight is that Full (Strict) mode is what provides actual security. Without it, Cloudflare will accept any certificate from your origin, including one presented by an attacker performing a MITM attack.

Further Reading

• Cloudflare Origin CA certificates
• Cloudflare SSL/TLS encryption modes
• Authenticated Origin Pulls
• Nginx SSL configuration best practices