GitLab Runners on Kubernetes: hyperscale cost optimisation with the Kubernetes executor

Technical deep dive: GitLab Runners on Kubernetes, Operator install, Helm values, autoscaling and the FinOps cost model. For the short version, read Stop paying for idle CI.

Most teams pay for continuous integration as if every night and every weekend were still Monday morning. Servers sit warm. Disk stays allocated. Nobody is merging code — and the bill does not notice. That sentence is not a slogan; it is the default shape of a fixed runner fleet.

This long article is the technical companion to a hands-on walkthrough of GitLab Server and GitLab Runner on Kubernetes: how the Kubernetes executor turns each CI job into an ephemeral pod, how that enables dynamic provision and de-provision of capacity, and how that design unlocks huge cost savings by reclaiming CPU, RAM, and storage when demand collapses overnight and at weekends.

Watch both sessions first, then use this page as the written architecture and cost model.

1. Runners & pipelines — autoscaling GitLab Runner workloads using Kubernetes.

Watch: Complete GitLab Runners and CI/CD on Kubernetes · Plain-language version: Stop paying for idle CI

2. GitLab Server on HA Kubernetes — deploy GitLab with the Operator in a few minutes.

Watch: Deploy GitLab in HA Kubernetes cluster using the Operator · Docs: GitLab Operator installation

1. Thesis

GitLab on Kubernetes + the Kubernetes executor changes the unit of CI capacity from “a machine we own all month” to “pods (and nodes) we rent for the job”. The runner manager stays small and always on. Job pods appear for a build and disappear afterwards. With cluster autoscaling, worker nodes follow the same curve. That is how you protect Monday morning throughput and stop paying Monday morning prices for Tuesday at 03:00 and for quiet Saturdays.

In one line — the same line as the layman guide: provision for the peak, de-provision for the trough.

2. The expensive habit, stated as an architecture smell

Traditional CI often looks like a permanent fleet: a fixed set of build machines that stay on “just in case”. That works when traffic is steady. It wastes money when demand is spiky.

• Daytime: developers push branches, open merge requests, run tests — demand spikes.
• Evenings and nights: activity drops; machines still burn CPU and RAM.
• Weekends: often near-idle — yet storage volumes and cloud nodes may still be reserved and billed.

You are not paying for “builds done”. You are paying for capacity that exists whether anyone uses it or not. Fixed fleets also accumulate sticky state: “works on runner 3”, drifted packages, caches that never expire. Elastic pods do not solve discipline by themselves, but they make clean-slate the default.

3. System map

Analogy from the plain guide still holds at this layer: GitLab Runner is the worker; Kubernetes is the dispatcher that calls workers only when parcels (jobs) arrive, then sends them home.

4. Installing GitLab on the cluster (operator path)

The Day 1 training path installs GitLab with the official operator so the server shares the same Kubernetes foundation as the runners. The companion video Deploy GitLab in HA Kubernetes cluster using the Operator walks through that install in a few minutes. Prerequisites: a cluster with ingress, Helm, kubectl, and cert-manager when the chart manages TLS.

helm repo add gitlab https://charts.gitlab.io
helm repo update

helm install gitlab-operator gitlab/gitlab-operator \
  --create-namespace \
  --namespace gitlab-system

kubectl -n gitlab-system get deployment gitlab-controller-manager

# Apply the GitLab custom resource (pin chart version in the manifest)
kubectl -n gitlab-system apply -f deploy_gitlab_server.yaml

Sample CR shape (customise domain and issuer email for your environment):

apiVersion: apps.gitlab.com/v1beta1
kind: GitLab
metadata:
  name: gitlab
spec:
  chart:
    version: "9.2.2"
    values:
      global:
        hosts:
          domain: "example.com"
        ingress:
          configureCertmanager: true
      certmanager-issuer:
        email: you@example.com

First install commonly takes on the order of 10–15 minutes while webservice, Sidekiq, data stores, and related components settle. Retrieve the initial root password from the cluster secret:

kubectl get secret gitlab-gitlab-initial-root-password -n gitlab-system \
  -o jsonpath='{.data.password}' | base64 -d

Reference: GitLab Operator installation (Helm tab on that page). Chart overview: docs.gitlab.com/charts · install guide: charts installation.

5. GitLab Runner with the Kubernetes executor

A GitLab Runner is the application that works with GitLab CI/CD to run jobs in a pipeline. With the Kubernetes executor, the manager does not compile on itself. For each job it:

1. Creates a pod (plus helper / service containers as configured).
2. Streams logs back to GitLab.
3. Collects artefacts and exit status.
4. Deletes the pod — returning CPU and RAM to the cluster.

That create/run/delete loop is the first cost lever. Finished work should not leave a warm machine dedicated overnight.

5.1 Helm install pattern

Training scripts create a namespace, pass a registration token, and may install more than one release (for example alpine-based and ubuntu-based runners) so jobs can select different environments. Official procedure: Install GitLab Runner with the Helm chart; values reference: Configure the GitLab Runner Helm chart.

helm upgrade --install --namespace gitlab-runner gitlab-runner-alpine \
  -f config-runner-alpine.yaml \
  --set serviceAccount.create=true \
  gitlab/gitlab-runner \
  --create-namespace

Obtain tokens from the GitLab UI (instance / group / project runners) or from operator-managed secrets. Prefer modern runner authentication tokens; registration tokens are deprecated toward GitLab 18. Never commit live tokens — lab scripts sometimes contain placeholders that must be replaced locally.

5.2 Values that matter for scale and isolation

From the Workstation templates (gitlab-runner-values-template.yaml / generated config-runner-*.yaml):

gitlabUrl: http://gitlab-webservice-default.gitlab-system.svc.cluster.local:8080/
concurrent: 2
rbac:
  create: true
  clusterWideAccess: true
runners:
  privileged: true
  config: |
    [[runners]]
      environment = ["GIT_SSL_NO_VERIFY=1"]
      [runners.kubernetes]
        namespace = "gitlab-runner"
        image = "alpine/git:latest"
        service_account = "gitlab-runner"
        privileged = true

Concurrency without CPU only lengthens the queue. Pair concurrent with real node headroom or node autoscaling.

6. Autoscaling at two (really four) levels

“Autoscaling GitLab Runners on Kubernetes” is not one switch. Stack these levers:

1. Job pod elasticity — built into the Kubernetes executor. Zero jobs → zero job pods. This alone removes most per-job waste on a shared cluster.
2. Runner concurrent and multiple runner Deployments — separate alpine vs ubuntu (or CPU vs larger builders) so specialised capacity is not held idle for the wrong mix.
3. Cluster / node autoscaling — pending pods add nodes; empty nodes leave after the evening rush.
4. Optional schedule-aware policy — lower concurrent caps or scale node pools on a calendar for nights/weekends as belt-and-braces on top of natural demand collapse.

Hyperscale is the same story with sharper amplitude: fan out for deep queues, then aggressively de-provision so the quiet half of the week does not resemble the busy half on the invoice.

6.1 A note on “idle” in the GitLab Runner sense

Some executors keep a small number of idle machines warm to absorb the next job without cold-start latency. On Kubernetes, cold start is usually creating a pod (seconds), not booting a VM (minutes). Prefer letting pods go to zero and accepting a short schedule delay over keeping a night-shift of empty nodes “just in case”. If you must warm a tiny floor, size that floor for on-call reality — not for Monday’s merge storm.

6.2 Dynamic provision and de-provision end-to-end

Trace one pipeline job:

1. Developer pushes; GitLab creates a job in the pending queue.
2. Runner manager claims the job and asks the API for a pod.
3. If the cluster is short on CPU/RAM, the autoscaler adds a node (provision).
4. Pod runs the script, uploads artefacts, exits.
5. Pod is deleted; resources return to the pool (de-provision at job level).
6. When many nodes sit empty past the scale-down threshold — typical after evening and all weekend — nodes leave (de-provision at infrastructure level).

That full loop is what the video means by GitLab Runners at hyperscale for cost optimisation: not a single fancy flag, but job lifecycle + cluster elasticity working together.

7. Cost model: where huge dollar savings appear

Be precise. Autoscaling does not make builds free. It stops you paying peak rates for off-peak hours. Savings come from matching spend to real demand — the four points from the plain guide, expanded for operators.

7.1 Scale up only when people (or bots) are working

When a busy morning lands dozens of pipelines, Kubernetes schedules many job pods in parallel up to concurrency and cluster limits. Throughput appears when merge pressure is real.

7.2 Scale down hard when demand disappears

When jobs finish, pods go away. With cluster autoscaling (or a right-sized node pool), unused cloud nodes leave the pool. Nights and weekends — typically the quietest windows for product teams — are the first capacity you should stop paying for.

7.3 Stop wasting CPU, RAM, and storage

Over a month, quiet hours dominate the clock. Reclaiming nights and weekends is where dollar savings compound.

7.4 Pay for outcomes, not for furniture

You still pay for GitLab and for a thin control plane (runner manager + cluster essentials). What you stop paying for is a warehouse of empty chairs.

7.5 Worked intuition (not a fabricated benchmark)

Suppose daytime sustainably needs the equivalent of 20 busy executor slots, but nights and weekends need 2. A fixed fleet sized for 20 pays for 20 around the clock. An elastic pool pays for ~20 during the day and ~2 (plus control plane) when quiet. For many product teams, quiet hours are a large fraction of the month — so reclaiming them is how savings become huge without inventing a marketing percentage. Finance multiplies local node-hour prices; architecture’s job is to make empty hours actually empty in the billing account.

Huge cost savings show up when Monday’s peak no longer dictates Tuesday-at-3am’s bill — and when Saturday’s silence finally looks quiet on the invoice too.

7.6 What you still pay for on purpose

• GitLab Server components — size for users and CI coordination, not for the compile farm.
• Runner manager Deployment(s) — small versus job farms.
• Deliberate caches (dependency proxy, PVCs) — helpful, but sized; caches are not free storage.
• Privileged DinD — convenient and heavier; consider Kaniko/BuildKit when density and isolation matter more than familiarity.

8. CI/CD pipelines in practice

With runners online, .gitlab-ci.yml defines stages and jobs. GitLab schedules; the Kubernetes executor materialises pods. Keep pipelines honest about cost:

• Fail fast on lint/unit before heavy image builds.
• Use rules / workflow so draft branches do not fan out full matrices.
• Prefer shared caches with clear keys over unbounded per-VM disk growth.
• Tag jobs to the right runner pool (alpine vs ubuntu, small vs large builders).

stages: [lint, test, build]

workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

lint:
  stage: lint
  image: alpine:3.19
  script:
    - echo "cheap checks first"

test:
  stage: test
  tags: [kubernetes, alpine]
  script:
    - echo "unit tests"

build:
  stage: build
  tags: [kubernetes, dind]
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  script:
    - echo "heavy image build only on default branch"

The economic idea is the same as the platform idea: do not schedule expensive work when a cheap check would have failed, and do not hold specialised runners warm for jobs that never arrive on Saturday.

9. Security and tenancy notes

Ephemeral pods improve hygiene (less sticky state) but privileged DinD and cluster-wide RBAC can widen blast radius. Practical defaults for most teams:

• Run job pods in a dedicated namespace with ResourceQuotas and LimitRanges.
• Prefer non-privileged builds where possible; isolate DinD runners behind specific tags.
• Avoid mounting long-lived cloud credentials into every job; inject short-lived tokens per pipeline.
• Treat runner registration tokens as secrets — rotate when people leave or labs are shared.
• If multiple teams share a cluster, separate runner managers (and optionally node pools) so one noisy neighbour cannot keep the whole farm hot — and keep the cost story attributable per team.

10. Observability that proves the savings

You cannot claim night-and-weekend savings without looking. Track at least:

If pod counts drop at night but the node bill does not, scale-down settings — not GitLab — are the next knob.

11. Fixed VMs vs Kubernetes executors (decision table)

Choose fixed VMs only when you truly have flat 24×7 demand or constraints that forbid a cluster. Everyone else with spiky human-driven pipelines is paying the furniture tax.

12. What you are not giving up

Autoscaling does not mean “no control”. You still set concurrency, allowed images, DinD privileges, network policy, and registration scope. The point is to keep those controls while letting unused capacity evaporate — the same reassurance as in the plain-language piece, now enforced with RBAC, namespaces, and separate runner releases.

13. Operational checklist

1. Pin operator/chart versions; upgrade deliberately.
2. Rotate runner tokens; scrub lab tokens from any script before publishing.
3. Separate namespaces: GitLab system vs runner jobs.
4. Watch pending pods — rising pending means concurrency or node capacity is short at peak.
5. Watch empty node hours — nodes lingering empty overnight mean scale-down delay or pool policy needs tuning.
6. Limit privileged runners to jobs that need them.
7. Back up GitLab independently of runner elasticity.
8. Review the bill by hour-of-week once elastic CI is live — nights and weekends should visibly drop.

14. Who this is for

• Platform engineers and SREs implementing GitLab CI on Kubernetes.
• DevOps leads replacing standing runner VMs with executor pods.
• Engineering and finance partners who need CI to behave like a usage-based utility, not furniture.

15. Official GitLab documentation

These pages are the canonical references for the Helm deployments and scaling behaviour described above. Prefer them over blog posts when versions change.

GitLab Server on Kubernetes

• Install the GitLab Operator — Operator + Helm install path used in the training walkthrough.
• GitLab Helm charts — umbrella chart documentation.
• Install GitLab with the Helm chart — when you deploy GitLab via charts rather than Operator.
• gitlab-runner subchart (GitLab chart) — runner bundled beside GitLab in the main chart (distinct from the standalone runner chart).

GitLab Runner Helm chart

• Install GitLab Runner with the Helm chart — helm repo add gitlab, install/upgrade, OCI chart notes.
• Configure the GitLab Runner Helm chart — concurrent, configuration templates, RBAC, caches, image choices.
• gitlab-runner chart values.yaml — full default values (including optional HPA stubs).

Kubernetes executor and autoscaling

• Kubernetes executor — job pods, helpers, services (e.g. DinD), resource requests, RBAC verbs.
• Configure autoscaling (Kubernetes executor / chart) — pause pods, [runners.kubernetes.autoscaler], period policies (idle by night/weekend).
• Autoscaling GitLab Runner — general autoscaling model (also covers distributed caching).
• Plan and manage a GitLab Runner fleet — fleet-level scaling and operations.
• Advanced configuration (config.toml) — global concurrent and executor sections embedded in Helm runners.config.

CI jobs, builds, and runner settings in GitLab

• Configure runners in GitLab — registration, tags, run-untagged, protected runners.
• Use Docker to build images in CI — DinD and alternatives (relevant when privileged / Docker service is enabled in runner values).

For night-and-weekend cost control, combine: (1) Kubernetes executor job pods that go to zero, (2) chart concurrent and autoscaler policies with off-peak idle_count = 0, and (3) cluster/node autoscaling so empty nodes leave the pool — as described in sections 6–7.

16. Mapping back to the video and repo

The YouTube session — Complete GitLab in ~20 minutes. Autoscaling GitLab Runner workloads using Kubernetes — demonstrates GitLab on Kubernetes, runners via Helm, and the hyperscale cost story: dynamically provision and de-provision infrastructure so you are not paying peak rates for idle nights and weekends. Use the course build prompt gitlab-runners-k3s-hyperscale-course.prompt.md in this repo, with the official docs above as the versioned source of truth. For the non-technical framing of the same ideas, see Stop paying for idle CI.

Closing

Kubernetes does not merely host GitLab Runners. Used properly, it changes the economics of CI: provision on demand, de-provision on idle, reclaim CPU/RAM/storage when humans go home. That is the hyperscale pattern — available to any team willing to trade a standing fleet for a queue of pods.

Start with the video, stand up operator + runner Helm in a lab, turn on node autoscaling, then check whether Saturday finally looks quiet on the invoice.

Published by Workstation.