The whole stack runs on a budget VPS – 1 vCPU, 2.9GB RAM, 30GB disk, $22/year. All three services are running with room to spare.

Initial Setup

SSH key auth first, password auth off:

1# local: generate key if you don't have one
2ssh-keygen -t ed25519 -C "your_email@example.com"
3# paste ~/.ssh/id_ed25519.pub into your provider's SSH key field
4
5# on the server: disable password login
6sed -i 's/^PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
7systemctl restart sshd

Test key login before disabling password auth. If you lock yourself out, you’re recovering via the provider’s console.

Swap – critical on 3GB RAM:

1fallocate -l 2G /swapfile
2chmod 600 /swapfile
3mkswap /swapfile
4swapon /swapfile
5echo '/swapfile none swap sw 0 0' >> /etc/fstab

Without this, CouchDB and Conduit together will occasionally OOM under load.

Firewall:

1ufw allow ssh
2ufw allow 80/tcp
3ufw allow 443/tcp
4ufw allow 3478        # TURN (UDP+TCP)
5ufw allow 5349        # TURN TLS
6ufw allow 49152:65535/udp  # TURN relay range
7ufw enable

The TURN ports are for Matrix voice/video – skip them if you’re not setting up coturn.

Docker:

1curl -fsSL https://get.docker.com | sh

Service 1: Obsidian LiveSync (CouchDB)

LiveSync uses CouchDB as the sync backend. Install it directly on the host rather than in Docker – it’s simpler to configure and easier to troubleshoot.

1apt install -y couchdb

The installer prompts for admin credentials. Set them and keep them somewhere safe.

Configure /opt/couchdb/etc/local.ini:

 1[chttpd]
 2bind_address = 127.0.0.1
 3
 4[couch_httpd_auth]
 5require_valid_user = true
 6
 7[httpd]
 8enable_cors = true
 9
10[cors]
11origins = app://obsidian.md, capacitor://localhost, http://localhost
12credentials = true
13headers = accept, authorization, content-type, origin, referer
14methods = GET, PUT, POST, HEAD, DELETE
15max_age = 3600

1systemctl restart couchdb
2curl -X PUT http://admin:PASSWORD@localhost:5984/obsidian-sync

nginx config:

 1server {
 2    listen 80;
 3    server_name sync.your.domain;
 4    location / {
 5        proxy_pass http://127.0.0.1:5984;
 6        proxy_set_header Host $host;
 7        proxy_set_header X-Real-IP $remote_addr;
 8        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 9    }
10}

Get a cert: certbot --nginx -d sync.your.domain

In Obsidian: install the LiveSync community plugin, set the CouchDB URI to your domain, enter credentials, set a database name and an E2E passphrase. Enable sync. The passphrase encrypts vault content before it leaves your device – CouchDB never sees plaintext.

Service 2: Private Matrix Chat (Conduit + coturn)

Conduit is a Matrix homeserver written in Rust – much lighter than Synapse, which would be too heavy for 3GB RAM. I run it with federation disabled – this is a personal instance, not connected to the public Matrix network.

/opt/conduit/docker-compose.yml:

 1services:
 2  conduit:
 3    image: matrixconduit/matrix-conduit:latest
 4    restart: unless-stopped
 5    volumes:
 6      - conduit_data:/var/lib/matrix-conduit
 7    environment:
 8      CONDUIT_SERVER_NAME: "chat.your.domain"
 9      CONDUIT_DATABASE_BACKEND: "rocksdb"
10      CONDUIT_ALLOW_REGISTRATION: "true"      # flip to false after accounts created
11      CONDUIT_ALLOW_FEDERATION: "false"
12      CONDUIT_MAX_REQUEST_SIZE: "20000000"
13      CONDUIT_LOG: "warn"
14    ports:
15      - "127.0.0.1:6167:6167"
16
17volumes:
18  conduit_data:

1cd /opt/conduit && docker compose up -d

For voice and video calls, Matrix uses WebRTC – which needs a TURN server to traverse NAT. coturn handles this:

1apt install -y coturn
2openssl rand -hex 32   # generate a static auth secret, keep it

/etc/turnserver.conf:

listening-port=3478
tls-listening-port=5349
fingerprint
use-auth-secret
static-auth-secret=YOUR_SECRET
realm=chat.your.domain
cert=/etc/letsencrypt/live/chat.your.domain/fullchain.pem
pkey=/etc/letsencrypt/live/chat.your.domain/privkey.pem
min-port=49152
max-port=65535

nginx config:

 1server {
 2    listen 80;
 3    server_name chat.your.domain;
 4    location / {
 5        proxy_pass http://127.0.0.1:6167;
 6        proxy_set_header Host $host;
 7        proxy_set_header X-Real-IP $remote_addr;
 8        proxy_set_header Upgrade $http_upgrade;
 9        proxy_set_header Connection "upgrade";
10    }
11}

Once Conduit is up: go to Element Web, set homeserver to your domain, register your accounts while ALLOW_REGISTRATION is true, then flip it to false and restart the container. Anyone who finds your homeserver URL can register while it’s open.

Service 3: Calibre-Web

Calibre-Web gives you a browser-based library for EPUB, PDF, and MOBI files. The Docker image is from linuxserver.io.

Critical step before starting the container: Calibre-Web needs a properly initialized metadata.db. Not an empty file, not a directory – a valid Calibre library database. It must be created by calibredb.

1apt install -y calibre   # ~300MB, takes a few minutes
2mkdir -p /opt/calibre/books
3calibredb --with-library /opt/calibre/books list   # initializes metadata.db
4chown -R 1000:1000 /opt/calibre/books              # container runs as uid 1000
5chmod -R 755 /opt/calibre/books

Do this before starting the container. If you start it first and hit “invalid path” or “DB not writable”, this is why.

/opt/calibre/docker-compose.yml:

 1services:
 2  calibre-web:
 3    image: lscr.io/linuxserver/calibre-web:latest
 4    restart: unless-stopped
 5    environment:
 6      - PUID=1000
 7      - PGID=1000
 8      - TZ=Asia/Kolkata
 9    volumes:
10      - calibre_config:/config
11      - /opt/calibre/books:/books
12    ports:
13      - "127.0.0.1:8083:8083"
14
15volumes:
16  calibre_config:

nginx config:

1server {
2    listen 80;
3    server_name books.your.domain;
4    location / {
5        proxy_pass http://127.0.0.1:8083;
6        proxy_set_header Host $host;
7        proxy_set_header X-Real-IP $remote_addr;
8    }
9}

Default login is admin / admin123. Change it immediately. Set library path to /books.

Cloudflare in Front

All three subdomains go through Cloudflare. DNS A records point to the VPS IP, proxy status set to orange cloud (proxied) for all three. SSL mode set to Full – not Full Strict, since the Let’s Encrypt cert on origin is valid enough and Full Strict adds complexity for no real benefit here.

certbot handles the Let’s Encrypt certs on the origin:

1certbot --nginx -d sync.your.domain
2certbot --nginx -d chat.your.domain
3certbot --nginx -d books.your.domain

Resource Usage

All three services running simultaneously:

Swap barely touched. Disk at ~6GB of 30GB. Plenty of headroom for adding services.

Pitfalls

The whole thing took an afternoon to set up. $22/year for services I fully control and can inspect end to end.

This post covers what I set up, what broke, and what actually works now.

Starting point

The provider gives you a root password on first boot. First thing I did was generate an ed25519 SSH key pair locally, paste the public key into the VPS control panel, add a Host personal entry to ~/.ssh/config, and verify key auth works before touching anything else. Then disabling password login:

1sed -i 's/^PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
2systemctl restart sshd

Don’t close your existing SSH session before confirming you can open a new one with key auth. Classic footgun.

Before any services: add swap. 2.9GB of RAM is workable but tight once Docker gets involved.

1fallocate -l 2G /swapfile
2chmod 600 /swapfile
3mkswap /swapfile
4swapon /swapfile
5echo '/swapfile none swap sw 0 0' >> /etc/fstab

Then update packages, set up UFW to allow SSH, HTTP, HTTPS, and the TURN port range (3478, 5349, 49152:65535), and drop everything else.

After all that, the box looked like this:

OS: Ubuntu 24.04 LTS
RAM: 2.9GB (470MB used, 2.4GB free)
Disk: 30GB (2.4GB used, 26GB free)
Swap: 2GB (none used yet)
CPU: 1 core

Plenty of room to work with.

Service 1: Obsidian LiveSync

CouchDB as the backend for Obsidian’s LiveSync community plugin. The plugin syncs your vault across devices using a CouchDB instance you control. End-to-end encrypted via a passphrase set in the plugin – CouchDB stores ciphertext only.

Install CouchDB natively (not Docker – LiveSync requires specific CouchDB config that’s simpler to manage directly):

1apt install -y couchdb

The installer asks for admin credentials. Set them and remember them. Then configure CouchDB to accept external connections by editing /opt/couchdb/etc/local.ini:

 1[chttpd]
 2bind_address = 127.0.0.1  ; nginx will proxy, no need to expose directly
 3
 4[couch_httpd_auth]
 5require_valid_user = true
 6
 7[httpd]
 8enable_cors = true
 9
10[cors]
11origins = app://obsidian.md, capacitor://localhost, http://localhost
12credentials = true
13headers = accept, authorization, content-type, origin, referer
14methods = GET, PUT, POST, HEAD, DELETE
15max_age = 3600

Then create the sync database:

1curl -X PUT http://admin:password@localhost:5984/obsidian-sync

nginx proxies the sync subdomain to localhost:5984. Cloudflare sits in front with Full SSL mode. In the Obsidian LiveSync plugin settings, point it at your CouchDB subdomain, enter the credentials, set your E2E passphrase, and enable sync.

It just works after that. Notes push from Mac, pull on iPhone, no iCloud dependency.

Service 2: Private Matrix Chat

Conduit is a Matrix homeserver written in Rust. It’s small (62MB RAM), single-binary, and handles everything a private two-person server needs. Element Web as the client – works in any browser, no app required.

Docker Compose setup:

 1services:
 2  conduit:
 3    image: matrixconduit/matrix-conduit:latest
 4    restart: unless-stopped
 5    volumes:
 6      - conduit_data:/var/lib/matrix-conduit
 7    environment:
 8      CONDUIT_SERVER_NAME: "your.chat.domain"
 9      CONDUIT_DATABASE_BACKEND: "rocksdb"
10      CONDUIT_ALLOW_REGISTRATION: "true"
11      CONDUIT_ALLOW_FEDERATION: "false"
12      CONDUIT_MAX_REQUEST_SIZE: "20000000"
13      CONDUIT_TRUSTED_SERVERS: '["matrix.org"]'
14      CONDUIT_LOG: "warn"
15    ports:
16      - "127.0.0.1:6167:6167"
17
18volumes:
19  conduit_data:

ALLOW_FEDERATION: false is important for a private server. No traffic leaves your machine to federate with matrix.org or anyone else.

For Element Web, the easiest route is the hosted version at app.element.io – just point it at your homeserver URL when logging in. If you want it self-hosted too, serve the static build from nginx.

For voice and video calls, WebRTC needs a TURN server or calls fail behind NAT. coturn handles this:

1apt install -y coturn

Configure /etc/turnserver.conf:

listening-port=3478
tls-listening-port=5349
fingerprint
use-auth-secret
static-auth-secret=<generate with: openssl rand -hex 32>
realm=your.chat.domain
cert=/etc/letsencrypt/live/your.chat.domain/fullchain.pem
pkey=/etc/letsencrypt/live/your.chat.domain/privkey.pem

Add the TURN server details in Conduit’s config, then tell Element where to find it in the Synapse well-known file. After that, voice and video calls work reliably regardless of NAT.

Create user accounts via the Conduit admin API or by registering through Element while ALLOW_REGISTRATION: true. After both accounts are created, set ALLOW_REGISTRATION: false so no new accounts can be made.

Service 3: Personal Book Library

Calibre-Web in Docker. Serves EPUB, PDF, MOBI with a clean browser interface. Can read books directly in the browser without downloading.

The setup hit two non-obvious problems worth documenting.

Problem 1: Calibre-Web needs a real metadata.db

The container starts and asks for a library path. I pointed it at /books (mapped to /opt/calibre/books on the host). It said “invalid path.”

The issue is that Calibre-Web expects a valid Calibre library database (metadata.db) to already exist at that path. An empty directory is not enough. The database needs to have the correct schema – not just the file.

The fix is to install Calibre on the host and use calibredb to initialize the library properly:

1apt install -y calibre  # takes a while, ~300MB
2calibredb --with-library /opt/calibre/books list

The calibredb list call creates a properly initialized metadata.db if one doesn’t exist. After this, /books becomes a valid library path.

Problem 2: Permissions

Even after the database exists, Calibre-Web may refuse to write to it. The linuxserver/calibre-web Docker image runs as user 1000 internally. If the files on the host are owned by root, the container can’t write.

1chown -R 1000:1000 /opt/calibre/books
2chmod -R 755 /opt/calibre/books

After both fixes, the setup page accepted /books and the library came up clean.

Docker Compose:

 1services:
 2  calibre-web:
 3    image: lscr.io/linuxserver/calibre-web:latest
 4    restart: unless-stopped
 5    environment:
 6      - PUID=1000
 7      - PGID=1000
 8      - TZ=Asia/Kolkata
 9    volumes:
10      - calibre_config:/config
11      - /opt/calibre/books:/books
12    ports:
13      - "127.0.0.1:8083:8083"
14
15volumes:
16  calibre_config:

Default login after first start: admin / admin123. Change it immediately.

nginx and Cloudflare

All three services sit behind nginx. Each subdomain gets a server block that proxies to the local port:

1server {
2    listen 80;
3    server_name your.sync.domain;
4    location / {
5        proxy_pass http://127.0.0.1:5984;
6        proxy_set_header Host $host;
7        proxy_set_header X-Real-IP $remote_addr;
8    }
9}

Cloudflare handles DNS and acts as the TLS terminator in Full mode. Let’s Encrypt issues certs on each subdomain via certbot. The certs auto-renew – I won’t have to touch them.

Credentials stored in pass on the Mac, encrypted with a GPG key. No plaintext passwords in config files or notes.

Resource usage after all three services

Service          RAM
---------        ----
Calibre-Web      159MB
Conduit          62MB
CouchDB          ~80MB
nginx + system   ~453MB
---------        ------
Total            754MB / 2.9GB  (26% used)
Swap used        780KB
Disk used        ~6GB / 30GB

74% of RAM still free. The 2GB swap sits untouched. There’s room for more things.

What’s next

The daily note has some ideas: run Bifrost on the VPS and wire up a blog-writing pipeline, use the unlimited bandwidth for a VPN, set up a small agent. The RAM budget will tighten but there’s headroom to experiment.

For $22/year, the box is already earning its keep.

Tools: Ubuntu 24.04, Docker, nginx, Cloudflare, Let’s Encrypt, CouchDB, Conduit (Matrix), Element Web, Calibre-Web, coturn, UFW, pass + GPG, certbot.

References

• LandChad.net – practical self-hosting guides, covers nginx, certbot, and many services: https://landchad.net
• Conduit (Matrix homeserver in Rust): https://conduit.rs
• Obsidian LiveSync plugin: https://github.com/vrtmrz/obsidian-livesync
• Calibre-Web: https://github.com/janeczku/calibre-web
• coturn TURN server: https://github.com/coturn/coturn

This post covers three CI/CD problems I solved with Docker Buildx, SHA-based image tagging, and dynamic workflow generation — and one thing I broke along the way.

Why latest is a lie

Container registries store images by tag. Tags are mutable pointers. When you docker push myapp:latest, you’re moving a pointer — the old image behind latest is now unreferenced. If two services share the tag latest or two people push to the same tag concurrently, you get a race condition on a mutable pointer. This is exactly the class of bug that version control systems solved decades ago.

The fundamental problem: image tags have no inherent ordering, no history, and no integrity guarantee. They’re a name, not a version.

The solution is to use a content-addressable identifier you already have: the Git commit SHA.

SHA-based image tagging

Every commit has a unique SHA-1 hash. Using it as the image tag creates an injective mapping from source code state → deployed artifact:

commit a1b2c3d → image registry/service:a1b2c3d → K8s pod spec

The implementation in GitHub Actions:

1- name: Derive image tag from commit
2  run: |
3    # Short SHA is sufficient — collision probability
4    # across 2^28 commits is negligible for any single repo
5    echo "IMAGE_TAG=${GITHUB_SHA::7}" >> $GITHUB_ENV

This gives you two things that latest never can:

1. Bijective traceability. Given a running pod, you can recover the exact source:

1$ kubectl get pod $POD -o jsonpath='{.spec.containers[0].image}'
2123456789.dkr.ecr.us-east-1.amazonaws.com/myservice:a1b2c3d
3
4$ git show a1b2c3d --stat

2. Immutable deployment records. The tag a1b2c3d either exists or it doesn’t. It can’t silently change meaning.

Propagating the tag to Kubernetes manifests

A tagged image is useless if the deployment spec still references latest. We close the loop by having CI update the manifest and commit the change:

 1- name: Patch deployment manifest
 2  run: |
 3    sed -i "s|image: $REGISTRY/$SERVICE:.*|image: $REGISTRY/$SERVICE:${IMAGE_TAG}|" \
 4      k8s/$ENVIRONMENT/${SERVICE}-deployment.yaml
 5
 6- name: Commit manifest update
 7  run: |
 8    git config user.name "ci-bot"
 9    git config user.email "ci-bot@noreply"
10    git add k8s/
11    git diff --cached --quiet || \
12      git commit -m "deploy($ENVIRONMENT): $SERVICE → $IMAGE_TAG"
13    git push

Now git log k8s/staging/ is your deployment history. Every deployment is a commit. Every rollback is git revert. This is the minimal viable GitOps pattern — your Git repo is the single source of truth for cluster state.

The git diff --cached --quiet guard prevents empty commits when nothing actually changed (e.g., the service wasn’t rebuilt).

Docker Buildx: layer caching in CI

The default docker build in CI has no cache. Every run downloads the base image, reinstalls all dependencies, copies all source files. For a Python service with 200+ packages, this is 8–12 minutes of work repeated identically on every push.

Docker Buildx provides two things that fix this: persistent builder instances and pluggable cache backends.

How Docker layer caching works

Docker images are a stack of layers. Each Dockerfile instruction creates a layer. Docker caches layers by their input hash — if the instruction and all preceding layers are identical, the cached layer is reused.

Layer 4: COPY . .                    ← changes every commit
Layer 3: RUN pip install -r req.txt  ← changes weekly
Layer 2: RUN apt-get install gcc     ← changes monthly
Layer 1: FROM python:3.11-slim       ← changes rarely

Cache invalidation is top-down: if Layer 2 changes, Layers 3 and 4 are invalidated too, even if requirements.txt hasn’t changed. This means Dockerfile instruction ordering directly determines cache efficiency.

The optimal ordering principle: sort instructions by change frequency, ascending. System deps → language deps → application source.

 1FROM python:3.11-slim AS base
 2WORKDIR /app
 3
 4# Layer 1: System dependencies (changes ~monthly)
 5RUN apt-get update && apt-get install -y --no-install-recommends \
 6    gcc libpq-dev && rm -rf /var/lib/apt/lists/*
 7
 8# Layer 2: Python dependencies (changes ~weekly)
 9COPY requirements.txt .
10RUN pip install --no-cache-dir -r requirements.txt
11
12# Layer 3: Application source (changes every commit)
13COPY . .
14
15CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

GitHub Actions cache backend

The key Buildx configuration:

 1- name: Set up Docker Buildx
 2  uses: docker/setup-buildx-action@v3
 3
 4- name: Build and push
 5  uses: docker/build-push-action@v5
 6  with:
 7    context: services/${{ matrix.service }}
 8    push: true
 9    tags: |
10      ${{ env.REGISTRY }}/${{ matrix.service }}:${{ env.IMAGE_TAG }}
11    cache-from: type=gha
12    cache-to: type=gha,mode=max

type=gha uses GitHub Actions’ built-in cache as the layer storage backend. This is important because CI runners are ephemeral — without an external cache backend, every run starts cold.

mode=max is critical and often missed. By default, Buildx only exports layers from the final stage. mode=max exports all intermediate layers, including those from multi-stage build stages. Without it, your pip install layer from a builder stage won’t be cached.

Under the hood, Buildx serializes each layer as a content-addressed blob and stores it via the GitHub Actions cache API. On subsequent runs, it checks the cache by layer digest before executing the instruction.

Cache efficiency in practice

The difference is dramatic:

The dominant cost axis shifts from build time to cache lookup time. For a source-only change, the only layer that rebuilds is COPY . ., which takes seconds.

Multi-stage builds and image size

If your final image includes build tools (gcc, python3-dev, header files), you’re shipping dead weight. Multi-stage builds separate the build environment from the runtime environment:

 1# Stage 1: Build (includes compilers, headers)
 2FROM python:3.11-slim AS builder
 3WORKDIR /build
 4RUN apt-get update && apt-get install -y gcc libffi-dev
 5COPY requirements.txt .
 6RUN pip install --no-cache-dir --prefix=/install -r requirements.txt
 7
 8# Stage 2: Runtime (minimal)
 9FROM python:3.11-slim
10WORKDIR /app
11COPY --from=builder /install /usr/local
12COPY . .
13CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

The --prefix=/install flag tells pip to install into a separate directory tree, which we then COPY --from=builder into the clean runtime image. The final image has no gcc, no header files, no pip cache.

Impact: image sizes dropped from 1.2GB to ~450MB. This matters for pull times during rollouts — a 450MB image pulls in ~15 seconds vs ~45 seconds for 1.2GB. During a rolling update across 6 replicas, that’s 3 minutes saved per deployment.

Dynamic service discovery in CI

Hardcoding a service list in your workflow is a maintenance hazard:

1# ❌ This will drift
2strategy:
3  matrix:
4    service: [auth, api, worker, processor]

When someone deletes processor/ and forgets to update the workflow, CI breaks. When someone adds a new service and forgets to add it, it never gets built. Both happen eventually.

Filesystem-driven matrix generation

Instead of declaring services, discover them:

 1jobs:
 2  discover:
 3    runs-on: ubuntu-latest
 4    outputs:
 5      services: ${{ steps.find.outputs.services }}
 6    steps:
 7      - uses: actions/checkout@v4
 8      - id: find
 9        run: |
10          services=$(find services/ -name Dockerfile -maxdepth 2 \
11            | xargs -I{} dirname {} \
12            | xargs -I{} basename {} \
13            | jq -R -s -c 'split("\n") | map(select(length > 0))')
14          echo "services=$services" >> $GITHUB_OUTPUT
15
16  build:
17    needs: discover
18    strategy:
19      matrix:
20        service: ${{ fromJson(needs.discover.outputs.services) }}

The contract becomes: a directory under services/ with a Dockerfile is a deployable service. No other registration needed. Add a service → add a directory with a Dockerfile. Remove a service → remove the directory.

Change detection: only build what’s affected

Even with caching, there’s no reason to build 8 services when 1 file changed. The git diff between the previous and current commit tells you what changed:

 1- uses: actions/checkout@v4
 2  with:
 3    fetch-depth: 0  # Required for accurate git diff
 4
 5- id: changes
 6  run: |
 7    changed=$(git diff --name-only ${{ github.event.before }} ${{ github.sha }})
 8    if echo "$changed" | grep -q "^services/${{ matrix.service }}/"; then
 9      echo "build=true" >> $GITHUB_OUTPUT
10    elif echo "$changed" | grep -q "^libs/shared/"; then
11      echo "build=true" >> $GITHUB_OUTPUT
12    else
13      echo "build=false" >> $GITHUB_OUTPUT
14    fi
15
16- name: Build and push
17  if: steps.changes.outputs.build == 'true'
18  uses: docker/build-push-action@v5
19  # ...

The fetch-depth: 0 is essential. GitHub Actions performs a shallow clone by default (fetch-depth: 1), which means git diff doesn’t have enough history and will treat everything as “new.” We discovered this when our changelogs started showing every file in the repo as changed.

The libs/shared/ check handles transitive dependencies — if a shared library changes, all services that could depend on it get rebuilt.

Edge cases that will bite you

ECR lifecycle policies

SHA tags accumulate. At 5 pushes/day × 8 services × ~800MB average image size, that’s 32GB/day of new images. Without cleanup, you’ll hit storage budget limits within weeks.

 1{
 2  "rules": [
 3    {
 4      "rulePriority": 1,
 5      "description": "Keep last 20 tagged images",
 6      "selection": {
 7        "tagStatus": "tagged",
 8        "countType": "imageCountMoreThan",
 9        "countNumber": 20
10      },
11      "action": { "type": "expire" }
12    },
13    {
14      "rulePriority": 2,
15      "description": "Expire untagged images after 24h",
16      "selection": {
17        "tagStatus": "untagged",
18        "countType": "sinceImagePushed",
19        "countUnit": "days",
20        "countNumber": 1
21      },
22      "action": { "type": "expire" }
23    }
24  ]
25}

20 tagged images gives you ~4 days of rollback runway at 5 pushes/day, which is plenty. Untagged images (intermediate layers from multi-stage builds) expire after 24 hours.

Concurrent manifest commits

If two workflow runs try to commit manifest changes simultaneously, the second git push fails. A retry loop with rebase handles this:

 1- name: Push manifest update
 2  run: |
 3    for attempt in 1 2 3; do
 4      git pull --rebase origin main
 5      git add k8s/
 6      git diff --cached --quiet && exit 0
 7      git commit -m "deploy: $SERVICE → $IMAGE_TAG"
 8      git push && exit 0
 9      echo "Push failed (attempt $attempt), retrying..."
10      sleep $((attempt * 2))
11    done
12    echo "::error::Manifest push failed after 3 attempts"
13    exit 1

The exponential backoff (sleep $((attempt * 2))) avoids thundering herd on concurrent pushes. In practice, this retries successfully on the first attempt 99% of the time.

Build context bloat

Docker sends the entire build context directory to the daemon before building. In a monorepo, that context can be enormous if you’re not careful. Each service needs a .dockerignore:

**/__pycache__
*.pyc
.git
.env
tests/
docs/
*.md
node_modules/

Without this, we were sending ~500MB of context for services that only needed ~20MB of source. The context upload alone added 2 minutes to every build.

Silent auth failures

ECR login can silently succeed without actually authenticating (e.g., when secrets aren’t available in fork PRs). The build proceeds, spends 10 minutes compiling, and only fails at docker push.

Add an explicit access check early:

1- name: Verify registry access
2  run: |
3    aws ecr describe-repositories \
4      --repository-names "${{ matrix.service }}" > /dev/null 2>&1 \
5    || { echo "::error::ECR access failed for ${{ matrix.service }}"; exit 1; }

Fail at second 5, not minute 12.

The pipeline end-to-end

git push origin main
        │
        ▼
┌──────────────┐
│   Discover   │  find services/ -name Dockerfile
│   Services   │  → ["auth", "api", "worker"]
└──────┬───────┘
       │
       ▼
┌─────────────────────────────────────────────────┐
│  Per-service matrix job (parallel)              │
│                                                 │
│  1. Checkout (fetch-depth: 0)                   │
│  2. git diff → should this service build?       │
│     └── no → skip remaining steps               │
│  3. Verify ECR access                           │
│  4. Set up Buildx                               │
│  5. Build + push (SHA tag, GHA layer cache)     │
│  6. sed manifest with new SHA                   │
│  7. git commit + push manifest (retry loop)     │
└──────┬──────────────────────────────────────────┘
       │
       ▼
┌──────────────┐
│  GitOps sync │  ArgoCD / Flux / kubectl apply
│  picks up    │  detects manifest diff → rolling update
└──────────────┘

Typical wall-clock time for a single-service source change: ~90 seconds from push to new pod running.

Key metrics

Checklist

• Tag with commit SHA, not latest. Mutable tags are mutable bugs.
• Order Dockerfile by change frequency. System deps → language deps → source. Cache invalidation is top-down.
• Use cache-from: type=gha with mode=max. Without mode=max, intermediate stage layers aren’t cached.
• fetch-depth: 0 on checkout. Shallow clones break git diff and changelog generation.
• Discover services from the filesystem. Hardcoded lists drift. Dockerfiles are the contract.
• Skip unchanged services. git diff + path matching. Don’t forget shared library paths as rebuild triggers.
• Set ECR lifecycle policies. 20 tagged images + 24h untagged expiry is a reasonable starting point.
• .dockerignore per service. Build context size ∝ upload time. Ship source, not tests.
• Verify registry auth early. Don’t spend 10 minutes building an image you can’t push.
• Retry manifest pushes. Concurrent CI runs will race on git push. Rebase + retry handles it.