Workstation WSL Proxy — Docker Image Optimisation, Build Cache, Full Deploy Workflow, and Shipping It with AI Assistance

Companion blog: Faster WSLProxy Deploys. Product: Workstation WSL Proxy · https://wslproxy.org/ · promotion patterns: https://ringpromoter.com/.

1. Lead

Workstation WSL Proxy (open-source product name WSLProxy) is an OpenResty-based API gateway and reverse proxy with live rules, WAF, cache, and multi-POP operations. Architecture notes live in the repo developer guide. This technote is about something narrower and more painful: how bare-metal edges used to recompile OpenResty on every serious deploy, how the project now ships a prebuilt tree via Docker Hub, how that artefact feeds the full delivery pipeline, and how an operator-in-the-loop AI coding agent changed the economics of finishing the work.

Two images, two jobs:

• Prebuilt OpenResty tree image (bwalia/wslproxy-openresty) — distribution format for Ansible extract onto POPs.
• App / fat Docker image (root Dockerfile) — Compose/dev/Hub runtime for the gateway process itself.

Do not conflate them. Layer-cache tricks apply to both; the install consumer differs.

2. Architecture dual diagram

Left (build and cache): push under infra/openresty-prebuilt/** → build-openresty-prebuilt.yml → Buildx + GHA layer cache → Docker Hub → Ansible pull/extract/rsync.

Right (promote and deploy): deploy-wslproxy-delivery-pipeline.yml → int → smoke → test → prod → deploy-environment.yml (secrets, npm/.next caches when needed, Ansible tags from DEPLOY_MODE).

flowchart LR
  subgraph LHS[Images and cache]
    A[Git push prebuilt paths] --> B[build-openresty-prebuilt.yml]
    B --> C[Buildx + GHA cache]
    C --> D[Hub bwalia/wslproxy-openresty]
    D --> E[Ansible extract + rsync]
  end
  subgraph RHS[Full deploy]
    F[delivery-pipeline] --> G[int smoke test prod]
    G --> H[deploy-environment.yml]
    H --> I[DEPLOY_MODE tags]
  end
  D -.-> H

(Rendered figure above is the canonical LHS/RHS view; the Mermaid sketch mirrors it for copy-paste into other docs.)

3. Why on-host compile hurts

Legacy path: openresty.sh.j2 on every target. A DEPLOY_MODE=full or build run pays ./configure + make + luarocks + opm on host CPU, with failure modes that look like “flaky CI”: missing tools on PATH, GCC quirks, CDN 410s. The pain is survivable, so it stays unfixed. Prebuilt mode flips the cost: compile in a controlled builder once; POPs only need matching runtime libraries and a careful rsync.

On-host compile also couples every POP to download mirrors and build-tool packages. A host that drifted from the golden set (missing ldconfig, older GCC, different PCRE package names) fails differently from its neighbour. CI on GitHub Actions with a pinned BASE_IMAGE collapses that variance into one controlled builder. The POP becomes a consumer of an artefact, not a miniature compile farm.

Contrast is the teaching device: leave openresty.sh.j2 in the tree as the legacy escape hatch when someone overrides install mode, but make prebuilt the default so routine pipelines stop paying compile tax.

4. Why teams do not ship this

Backlog economics. Features and incidents beat “make deploys less miserable.” A thirty-minute full job that mostly recompiles nginx does not page anyone — it just drains patience. Ticket systems reward visible customer outcomes; pipeline hygiene is invisible until someone times a release train.

Cross-cutting blast radius. One change set touches Dockerfile design, Hub auth, GHA Buildx cache, Ansible extract excludes, pipeline mode defaults, Next.js caches, and glibc coupling. That is several specialised domains. Hard to staff as a “small PR.” Reviewers need Docker, Ansible, and Actions fluency at once.

Opaque failures. Configure PATH, 410 CDN URLs, and cache misses that look random kill momentum after one burn. Without a fast diagnose→patch→re-run loop, the initiative dies after a single green-hope failure.

Review fear. Reviewers must trust that extract will not clobber live conf/html/logs. Deferral is rational without a soak plan and explicit excludes in the PR description.

Those four forces explain why the pattern is well-known in industry talk tracks and still rare on mid-size fleets. The WSLProxy work did not discover a new algorithm — it finished a backlog item that usually loses to urgency.

5. AI-assisted delivery loop (WSLProxy)

Keep this credible. No sci‑fi. No claim that AI merged to production, rotated secrets, or invented OpenResty.

Honest limits: AI does not replace knowing glibc must match the host, or that rsync excludes protect live config. Humans merge to main/release, hold Hub credentials, and choose DEPLOY_MODE on real POPs. The win is time-to-first-working-pipeline and stamina through CI breakages.

Public artefacts from that loop (evidence of iteration, not autonomy):

• build-openresty-prebuilt.yml
• Actions workflow runs
• Warm/cached success after fixes: run 35500698371 (~3m with cache)

Themes visible on main: prebuilt image + Next caches; ldconfig PATH fix; AIStor mc URL fix; promotion of working trees toward release. Cite files and runs — not private chat transcripts or agent IDs.

6. Prebuilt Dockerfile walkthrough

Source: infra/openresty-prebuilt/Dockerfile.

Stage 1 (builder). ARG BASE_IMAGE=debian:13. Install build deps including libc-bin so ldconfig exists. Set:

ENV PATH=/usr/local/openresty/luajit/bin:/usr/local/openresty/bin:/usr/local/bin:/usr/sbin:/sbin:/usr/bin:/bin

Compile OpenResty with flags aligned to the legacy template (--with-http_v2_module, stream realip, stub status, slice, and the rest mirrored from openresty.sh.j2). Build luarocks against system Lua 5.1 (not LuaJIT — the 65536 constant limit breaks luarocks manifests). Install rocks into the OpenResty LuaJIT tree: hard-fail on required lua-resty-jwt 0.2.3; optional rocks warn and continue so a sockproc/auto-ssl hiccup under a newer GCC does not fail the entire artefact when the role historically used ignore_errors. Pull opm packages and copy prometheus / upstream healthcheck libs from shallow git clones. Install MinIO client late via the AIStor URL so layer cache survives URL tweaks.

Stage 2 (runtime). Matching base + runtime libs only (libpcre2, libssl3, zlib1g, CA certs); COPY --from=builder /usr/local/openresty. Ship the tree, not the compiler toolchain. Use --no-install-recommends and clear apt lists to keep layers lean. Symlink openresty and print openresty -V as a build-time sanity check.

Why multi-stage matters for extract. Ansible only needs the OpenResty prefix. Keeping the builder toolchain out of the runtime stage reduces image size and clarifies that the Hub tag is a packaging vehicle. Controllers still need Docker to pull and docker cp; targets need runtime libraries and a matching glibc, not a full build environment.

6b. Mapping compile flags to the legacy script

The prebuilt Dockerfile comments state the intent explicitly: mirror openresty.sh.j2, deploy_deps.yml rock lists, and cdn-dependencies.sh.j2 opm/lualib steps. When you change a configure flag in one place, change it in both or you will invent “works in Docker, fails on POP” drift. Pin OPENRESTY_VERSION and critical rocks. Prefer failing closed on JWT; fail open on optional auto-ssl bits that already had soft failures in Ansible. That policy is product judgement, not something an agent should invent unsupervised.

7. GitHub Actions: build-openresty-prebuilt.yml

Workflow: build-openresty-prebuilt.yml.

• Triggers: push to main on infra/openresty-prebuilt/** and the workflow file; plus workflow_dispatch with openresty_version and base_image inputs.
• Auth: Docker Hub via DOCKER_USER / DOCKER_PASSWD.
• Tags: bwalia/wslproxy-openresty:<version> and :latest.
• Platform: linux/amd64 today — not multi-arch unless extended.
• Cache: cache-from: type=gha, cache-to: type=gha,mode=max.
• Provenance / SBOM: set false by choice — a trade-off teams sometimes make for Hub push simplicity; do not moralise, document why.

cache-from: type=gha
cache-to: type=gha,mode=max
provenance: false
sbom: false
platforms: linux/amd64

mode=max exports more intermediate layers than the default exporter mode, which is what you want when late stages churn (for example mc URL fixes) while early compile layers should stay warm. Concurrency is grouped and cancels in-progress builds so a rapid series of fixes does not pile up stale runners. Timeout is generous (90 minutes) because a cold OpenResty compile is heavy; warm rebuilds should be far shorter when the cache hits.

Same GHA cache pattern appears on CLI and control-plane image builds (build-wslproxy-cli.yml, deploy-control-plane-k3s1.yml). Steal the pattern; keep secrets naming consistent across workflows so operators are not hunting three Hub credential shapes.

8. Ansible consumer: openresty_prebuilt.yml

Default: openresty_install_mode: prebuilt in role defaults. Tasks: openresty_prebuilt.yml. Express helper: openresty-express-install.sh. Role entry orchestration remains in tasks/main.yml.

Flow on the controller (needs Docker):

1. Assert Docker is available on the controller.
2. docker pull the prebuilt image (platform pinned).
3. Ensure runtime libs on the target (match Dockerfile runtime stage; tolerate PCRE package name differences).
4. docker create → docker cp /usr/local/openresty into staging via tar stream.
5. rsync -a --delete with excludes for /nginx/conf/, /nginx/html/, /nginx/logs/.
6. Symlink openresty and optionally mc; print nginx -v.

rsync -a --delete \
  --exclude='/nginx/conf/' \
  --exclude='/nginx/html/' \
  --exclude='/nginx/logs/' \
  "$STAGING/openresty/" "$PREFIX/"

Those excludes are non-negotiable. Reviewers should treat a PR that removes them as a production incident waiting to happen. Docker here is a packaging format for bare metal, not only a runtime. That mental model is the whole point of the LHS.

cid="$(docker create --platform "$PLATFORM" "$IMAGE")"
docker cp "$cid:$PREFIX" - | tar -x -C "$STAGING"
docker rm -f "$cid" >/dev/null
test -x "$STAGING/openresty/nginx/sbin/nginx"

The extract step is deliberately boring: create a stopped container, stream the prefix out as a tar, assert the nginx binary is executable, then rsync. No entrypoint runs on the POP. If extract succeeds but the binary will not start, suspect glibc/base mismatch before you blame Ansible. If rsync succeeds but sites disappear, you dropped an exclude — restore from backup and treat the change as a severity-one ops defect.

Local versus remote install variants in the task file keep the same excludes; only the transport differs. Passwordless sudo on remote targets matches the express-install expectations. Controllers without Docker cannot use this path — that is an intentional dependency, not an oversight.

9. glibc / BASE_IMAGE coupling

A compiled binary is libc-specific. Default BASE_IMAGE=debian:13 matches current int/prod Debian hosts in this fleet. Rebuild with --build-arg BASE_IMAGE=debian:12 (workflow dispatch) when targets differ. Wrong glibc → extracted nginx will not run cleanly. Arch must match too (linux/amd64 today). This is a human judgement call an agent can remind you of but cannot “fix” with a clever flag on a mismatched POP.

Operational practice: tag images with both OpenResty version and, when you maintain multiple bases, a base suffix or separate repository tags. Document which POP runs which base in the inventory, not only in chat history. Soak a non-prod host after every base bump before touching customer-facing edges.

10. Root Dockerfile lessons (app image)

Root Dockerfile installs MinIO client before COPY ./api so Hub/layer cache survives code churn. Classic /client/mc paths return 410; use:

https://dl.min.io/aistor/mc/release/linux-${ARCH}/mc

Teaching point: heavy binaries before volatile COPY is classic Docker hygiene — independent of the prebuilt extract story. The app image remains the right vehicle for Compose and containerised gateway processes; the prebuilt tree image remains the right vehicle for Ansible bare-metal POPs. Sharing the AIStor URL between them is deliberate consistency, not accidental duplication.

11. RHS full deploy

deploy-wslproxy-delivery-pipeline.yml stages promotion (int → smoke → test → prod). Inputs such as DEPLOY_MODE, branch, and target host drive reusable deploy-environment.yml, which applies secrets (SOPS / vault_or_sops as configured), runs Ansible with tags derived from mode, and when mode contains dashboard or equals full:

• actions/setup-node with cache: npm against openresty-admin-next/package-lock.json
• actions/cache for .next/cache, standalone, and static outputs keyed on lockfile + source hashes

path: |
  openresty-admin-next/.next/cache
  openresty-admin-next/.next/standalone
  openresty-admin-next/.next/static
key: nextjs-${{ runner.os }}-${{ hashFiles('openresty-admin-next/package-lock.json') }}-...

DEPLOY_MODE teaching points (how deploy-environment.yml maps modes today):

Prefer code for routine day-2. Reserve build/full for OpenResty or OS dependency changes. Delivery-pipeline job defaults can still fall back to full in places — operators must override consciously rather than assume every promotion is cheap.

Rules can change without nginx reload; config_status: true still drives conf activation and the reboot flag for server blocks (see CLAUDE.md). For multi-app promotion discipline across rings, see Ring Promoter — health-gated hops, not “skip to prod because the image built.”

Two-layer timeout and ingress lessons from the same codebase apply when k3s sits behind the Ansible POP: raising timeouts only on the outer edge does not help if the ingress controller keeps a lower read timeout. That is adjacent to image cache work but belongs in the operator mental model whenever “full deploy” includes both layers.

11b. What the delivery pipeline is not

It is not a substitute for choosing the right mode. A green pipeline that always runs full will still burn compile or extract time you do not need. It is not multi-cloud magic — hosts and connection modes are configured explicitly. It is not unsupervised AI promotion. Humans still decide when a POP is ready and whether Hub credentials and SOPS keys are in place.

12. Operator playbook

Before declaring victory on a POP: confirm the image tag you intended is what Ansible pulled, confirm excludes left live site roots intact, and confirm health returns 200 on the admin listener your environment actually uses (Docker admin ports differ from Ansible prod templates — check the guide, do not assume 8080).

A practical soak checklist after the first prebuilt install on a new POP: (1) openresty -V matches openresty_expected_version; (2) existing virtual hosts still serve content from untouched html; (3) live nginx/conf was not replaced by image defaults; (4) admin health and a known API route still pass; (5) a subsequent DEPLOY_MODE=code completes without touching the OpenResty install path. Only then promote the same image tag further along the delivery pipeline.

13. Pitfalls checklist

• PATH / ldconfig — agent can triage from configure logs; human confirms base image choice.
• MinIO 410 — agent patches URL; human confirms Hub push succeeded.
• Wrong glibc — human judgement; soak on a non-prod POP first.
• Overwriting conf/html/logs — never drop the rsync excludes; review carefully.
• Treating every push as full — defeats the point; default day-2 to code.
• Assuming multi-arch — workflow is amd64-only today.
• Confusing app image with prebuilt tree — different Dockerfiles; different consumers.
• Claiming unmeasured timings — keep qualitative unless you paste a run URL.

14. Playbook for other teams

Use an agent (Cursor / Claude / Copilot-class — pick your shop standard) to draft the prebuilt path and cache wiring. You own merge, secrets, and first POP soak. Keep the LHS/RHS split in the design doc so reviewers see when the image rebuilds versus when a code deploy suffices. Pair edge promotion with a ring/health-gated promoter if you run many apps — ringpromoter.com is one realisation of that idea.

Suggested sequence for a foreign stack:

1. Measure one full deploy and name the dominant phase (compile, UI, secrets, rsync).
2. Decide whether the binary tree can be a Hub artefact with extract-not-run semantics.
3. Scaffold multi-stage Dockerfile + Buildx GHA cache + path-filtered workflow.
4. Write the install consumer with explicit excludes for live config/site/log trees.
5. Change day-2 defaults to the fast path; reserve full/build for artefact bumps.
6. Document glibc/base coupling and an operator verification checklist.
7. Only then write the marketing narrative — claims follow green Actions runs.

Vendor-neutral advice: any assistant that can read CI logs and edit multiple files helps. None of them should hold production merge authority without a human. Product home for this edge: https://wslproxy.org/; Workstation product page: /en/wsl-proxy.

14b. Separating myths from the WSLProxy record

• Myth: “Never rebuild.” Record: rebuild when compile inputs change; reuse layers via Buildx/GHA.
• Myth: “AI wrote everything unsupervised.” Record: operator+agent loop; humans merge and soak.
• Myth: “Multi-arch everywhere.” Record: linux/amd64 in the prebuilt workflow today.
• Myth: “App image equals prebuilt tree.” Record: different Dockerfiles, different consumers.
• Myth: “Invented 10× forever.” Record: qualitative wins plus one citeable warm run (~3m).

15. Appendix — link table

Measured timings: qualitative except the warm Buildx example (~3m, run 35500698371). Do not invent percentage or minute savings elsewhere.

Draft for Workstation CMS — publish live only after an explicit publish ask. Brand on workstation.co.uk: Workstation WSL Proxy; product deep links use WSLProxy / wslproxy.org / GitHub.

16. Closing

If you remember nothing else: put OpenResty compile on the left as a cached Hub artefact, put promotion and DEPLOY_MODE on the right, and refuse to run full for Lua typos. Use an agent to finish the cross-cutting scaffolding and CI log loop; keep merge, secrets, glibc matching, and first POP soak in human hands. Start from the public files linked above, verify against origin/main, then decide whether your fleet’s day-2 path can look the same.

Further reading on the Workstation site: the benefits-led blog companion, the product page, and related edge/CI pieces that point at wslproxy.org and ringpromoter.com. When you copy the pattern into another organisation, keep British-English operator docs consistent with how you name modes and tags, and refuse to publish timing claims until a green Actions run backs them with a public run URL.