NVKE and NVKR Technote: Building a Managed GPU Kubernetes Service from Bare Metal, with Reproducible, Verifiable Builds

A practitioner's guide to NVIDIA Kubernetes Engine (NVKE) and NVIDIA Kubernetes Recipes (NVKR), the two open-source projects NVIDIA released in public preview at KubeCon North America on 10 November 2026. It covers the architecture, every component and pinned version, the NVKR build and verification lifecycle, the full workflow from racked servers to a customer GPU cluster, the security model, day-two operations, the preview's known limitations with a hardening checklist for each, how NVKE compares with other ways of running GPU Kubernetes, and a staged evaluation plan.

This technote is the companion to our business-level post, NVIDIA NVKE and NVKR: why GPU Kubernetes is becoming the AI factory's operating system. Everything below is drawn from NVIDIA's announcement walkthrough, the NVKE documentation and the NVKE and NVKR repositories as of 14 November 2026. Commands are NVIDIA's unless marked as a Workstation example. Both projects change quickly during the preview, so always check the current documentation for flags and fields.

1. The problem NVKE and NVKR divide between them

An AI factory (NVIDIA's term for a data centre built to produce AI at scale) that wants to sell or share GPU capacity as Kubernetes has two distinct problems:

1. Running a managed Kubernetes service. API-driven provisioning, tenant isolation, upgrades, node replacement and recovery, backed by service commitments.
2. Qualifying the software. Knowing which combination of operating system, kernel, Kubernetes, network plug-in, agents and GPU runtime works on which GPU generation, and being able to rebuild exactly that combination later.

NVKE takes the first problem and NVKR the second. They are deliberately separable: NVKE has no runtime dependency on NVKR, and NVKR can build artifacts for other Kubernetes management systems. The factory keeps control of its own software, operations, service and customer experience.

2. Components at a glance

3. Architecture

The control flow is easiest to read from the operator's side. The operator talks only to the NVKE API, through nvkectl. NVKE translates a cluster request into Cluster API objects; CAPNICo turns those into NICo instance requests; NICo picks servers of the requested instance type in the requested VPC prefix and boots them with the iPXE script from the chosen machine image. Each server boots the NVKR-built image, its SPIRE agent attests to the management cluster's SPIRE server using the server's TPM, and only then does the node receive the credentials it needs to run kubeadm and join. Customers see none of this: they receive a kubeconfig for a cluster that looks like any other conformant Kubernetes 1.36 cluster.

4. Versions pinned in the preview

NVKE's platform reference lists the validated combination below. "Pinned" is the point: NVKR builds from these exact inputs and records them, so a node can be traced back to its sources.

5. NVKR: Definition, Run, Release, Verify

NVKR is a build and release system rather than a script collection. Its public lifecycle has three stages:

1. Define the deliverables. A Definition selects named outputs (executables, container images, Helm charts, deployment bundles or node images), their components, platforms and formats. Catalogs supply pinned sources, artifacts and tools. An Environment supplies execution, registry and signing settings.
2. Inspect and run. NVKR derives a Plan and dependency graphs that you can review before anything executes, then runs it locally or generates GitHub Actions workflows. Each Run records its required outputs, the artifacts it actually observed, its evidence and its failures.
3. Publish and verify. A Release is a digest-pinned OCI registry reference that connects the outputs to their SBOMs and provenance. Consumers choose a Verification Policy (the workflow identity and issuer they accept) and authenticate the claims before using the artifacts.

5.1 First build

The quickstart builds the NVKE CLI, the controller container image and its Helm chart, with dependency SBOMs. It needs Git 2.38 or later, Bash and a running Docker; the launcher checks prerequisites, selects the pinned Go compiler (asking before installing it) and builds NVKR itself from source for your CPU architecture:

git clone https://github.com/dsx-ai-factory/nvkr.git &&
cd nvkr &&
bash scripts/nvkr-quickstart.sh

Wait for status: complete and observed outputs: 3. The native CLI is bin/nvkr. Nothing is published or deployed, and a local build is unsigned. To build the node image as well, NVIDIA's walkthrough runs a second script:

NVKE_OS_ARCH=amd64 scripts/nvke-os-local.sh node-image-01

That build compiles the NVKE node agent and OVN-Kubernetes from source, adds checksum-pinned upstream releases of the Kubernetes 1.36 packages and the SPIRE agent, boots a virtual machine from the Ubuntu installer, installs the agents and stages the Kubernetes and networking packages on disk. The result is a QCOW2 image. The build host needs Docker and KVM support.

Beyond the quickstart, the NVKR tutorial Definition selects the NVKE CLI, management-plane image and Helm bundle, and the combined NVKE candidate Definition expands that to container and node images for both amd64 and arm64.

5.2 What verification does and does not prove

NVKR is careful about its claims, and your release process should be too. A signature authenticates who made a statement about an artifact; a digest binds that statement to specific bytes. Signing does not claim ownership of the software inside, and a valid signature alone does not make an artifact ready to deploy.

Verification needs Linux, pinned tools and an explicit policy (macOS users verify inside Docker). At preview, NVIDIA states that no NVKR Release is qualified, that supported verification does not establish exact-byte reproduction, freedom from vulnerabilities or deployment readiness, and that node-image SBOMs record producer and material subjects rather than inspecting the installed disk contents. Treat NVKR evidence as one input to your own release gate, alongside vulnerability scanning and hardware validation.

6. Workflow: from racked servers to a customer cluster

6.1 Prerequisites

• Three CPU servers for the management cluster, plus a separate Linux bootstrap host on the site network with containerd, nerdctl, kind, Helm and kubectl.
• Three CPU servers for each workload cluster's control plane.
• A pool of GPU servers for worker nodes.
• A NICo installation with enrolled capacity.
• A Linux build host with Docker and KVM support.
• An HTTP server for the node image, a container registry for the controller image, and internet access for the open-source images and Helm charts used during installation.
• Your own SPIRE certificate authority. Management-cluster nodes built from the node image need your CA files at bootstrap. Without them SPIRE uses a self-signed CA that no node can trust in advance, so nodes cannot attest and cannot join.

6.2 Build and stage

Build as in section 5.1, then push the controller image to your registry, copy nvkectl to the bootstrap host and put the QCOW2 image on the HTTP server. NVIDIA notes that a local image build is not a qualified release: the build and smoke test do not validate a real NICo boot, disk selection or complete first-boot configuration. Provisioning from a QCOW2 also needs a site-specific EFI imager and iPXE script that downloads the image and writes it to disk, validated together on your hardware.

6.3 Bootstrap the management cluster

nvkectl bootstrap site -f site-a.yaml \
  --nvke-controller-image registry.example.com/nvke/nvke-controller:<tag>

site-a.yaml holds the inputs NICo needs to provision three management servers. The command starts a temporary single-node kind cluster on the bootstrap host, which hosts the NVKE software needed for set-up. The NVKE controller asks NICo for three servers, NICo boots them from the node image, and when all three have joined, the bootstrap deletes the temporary cluster. This is the familiar Cluster API "bootstrap and pivot" pattern.

Keep the bootstrap kubeconfig and the etcd encryption key file in secure storage immediately. The key file is the only copy of the management cluster's etcd encryption key that is not on a node, and there is no backup or restore procedure yet.

6.4 Install the management plane and connect nvkectl

Install the NVKE management plane on the management cluster, then run nvkectl config set-site to point the CLI at the NVKE API. For an install without a release description, the controller and profile images must be staged on the management nodes and named in the stack configuration.

Before anyone else can reach it, secure the API (section 7.2): after installation it listens on port 30080 of every management node, over plain HTTP, with authentication off.

6.5 Publish a release set

Customers never handle operating system images or cloud-init. They choose a named release that you publish. A release set has three parts:

1. Machine image: the iPXE script NICo runs to boot a server.
2. Cluster release: a machine image plus the cloud-init configuration that prepares a control-plane node, recording the Kubernetes version it installs.
3. Node pool release: the equivalent image and configuration for worker nodes.

nvkectl create machineimage -f machineimage.yaml
nvkectl create clusterrelease -f clusterrelease.yaml
nvkectl create nodepoolrelease -f nodepoolrelease.yaml

Treat release sets like a product catalogue. Name them so a customer can tell what they get (NVIDIA's example uses k8s-1.36.3-amd64), publish separate releases per CPU architecture and GPU generation, and never change a published release in place: publish a new one and move customers deliberately.

6.6 Create the customer cluster

A cluster request repeats the NICo inputs from the site specification and adds a Kubernetes version. The control plane and each node pool have their own instance types, VPC prefixes and release. NVIDIA's example:

name: customer-a
controlPlaneKubernetesVersion: 1.36.3
etcdEncrypted: true
controlPlaneVIP: 10.0.20.10
nico: {orgID: <org-id>, siteID: <site-id>, vpcID: <vpc-id>}
controlPlaneConfig:
  desiredCount: 3
  nico: {instanceTypePreference: [<cpu-instance-type>], vpcPrefixIDs: [<prefix-id>], release: k8s-1.36.3-amd64}
nodePools:
  - name: gpu
    nodeNamePrefix: gpu
    desiredNodeCount: 4
    nico: {instanceTypeID: <gpu-instance-type>, vpcPrefixIDs: [<prefix-id>], release: k8s-1.36.3-amd64}

nvkectl create cluster -f customer-a.yaml

The cluster stays Provisioning until the first control-plane node has run kubeadm init and NVKE has installed OVN-Kubernetes and NodeLocal DNSCache; it then becomes Provisioned. Note etcdEncrypted: true: encryption of etcd data is a per-cluster choice, and we recommend making it the default in your request templates.

6.7 Hand over access

nvkectl get kubeconfig customer-a --output-file customer-a.kubeconfig
kubeconfig written to customer-a.kubeconfig

This is an administrator kubeconfig, and at preview NVKE cannot revoke it: it stays valid until its client certificate expires, because NVKE does not rotate a workload cluster's certificate authority. Hand it only to the customer's cluster administrators, over a secure channel, and have them create narrower credentials inside the cluster straight away. A Workstation example using standard Kubernetes RBAC:

# Workstation example: run once with the admin kubeconfig, then store it offline
export KUBECONFIG=customer-a.kubeconfig
kubectl create namespace ml-team
kubectl -n ml-team create serviceaccount deployer
kubectl -n ml-team create rolebinding deployer-edit \
  --clusterrole=edit --serviceaccount=ml-team:deployer
# Short-lived token for CI or a developer, instead of sharing the admin file
kubectl -n ml-team create token deployer --duration=8h

6.8 Make it a GPU cluster

NVKE delivers conformant Kubernetes on GPU servers. The GPU-accelerated runtime is added inside each workload cluster with NVIDIA's AI Cluster Runtime (AICR), node packages with NodeWright, and break-fix automation with NVSentinel. NVKR can build AICR artifacts too. Once the runtime advertises GPUs, a smoke test is a pod that requests one (Workstation example; choose a CUDA image tag that matches your driver):

apiVersion: v1
kind: Pod
metadata:
  name: gpu-smoke
spec:
  restartPolicy: Never
  containers:
    - name: cuda
      image: nvcr.io/nvidia/cuda:<tag>
      command: ["nvidia-smi"]
      resources:
        limits:
          nvidia.com/gpu: 1

7. Security model

7.1 Node identity

Every node presents an attested SPIFFE identity, backed by its TPM through SPIRE, before it receives join credentials. Management-cluster nodes trust your SPIRE CA (supplied at bootstrap); workload-cluster nodes receive the management cluster's SPIRE trust bundle in their user data. This closes a classic bare-metal gap: a server that merely reaches the network cannot join a customer's cluster. The quickstart's virtual TPM requirement exists for the same reason.

7.2 The API: plain HTTP, no authentication, no authorisation

Out of the box the NVKE API listens on port 30080 of every management node without TLS, and anyone on a network you admit can call every endpoint, including create and delete. Even with authentication enabled, NVKE does no authorisation: every authenticated caller can manage every workload cluster. Our minimum before admitting any caller:

1. Follow NVKE's procedure to admit only the networks that need the API, and enforce the same at your network edge with firewall rules or switch ACLs on the management VLAN.
2. Turn on API authentication. Once it is on, generate an OIDC kubeconfig with nvkectl generate oidc-kubeconfig; with it, a few commands that read the management cluster's Kubernetes API directly (get kubeconfig, get providers, and the machine image and release commands) must be run with kubectl and the management cluster's admin kubeconfig instead.
3. Put your own TLS endpoint in front of the API, because tokens otherwise cross the network unencrypted, and keep the hop behind it on an isolated management network. A Workstation example with nginx on an admin bastion:

# Workstation example: TLS in front of the NVKE API (preview serves plain HTTP)
upstream nvke_api {
    server 10.0.10.11:30080;
    server 10.0.10.12:30080;
    server 10.0.10.13:30080;
}
server {
    listen 443 ssl;
    server_name nvke-api.example.internal;
    ssl_certificate     /etc/ssl/nvke-api/fullchain.pem;
    ssl_certificate_key /etc/ssl/nvke-api/privkey.pem;
    allow 10.20.0.0/24;   # operator and automation subnet only
    deny  all;
    location / {
        proxy_pass http://nvke_api;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

4. Issue NVKE API tokens only to people and systems that may manage every cluster, and keep that list short. Put any per-tenant self-service in your own portal, which enforces authorisation and calls NVKE on the tenant's behalf.

7.3 Data at rest

Enable etcd encryption per cluster. Do not rely on the full-disk-encryption (FDE) node image variant: each image build generates one volume key that every node built from it inherits, the bootstrap passphrase is public and remains in the GRUB configuration, and anyone with the image and passphrase can recover the key. NVIDIA states FDE images are for evaluation only.

8. Day-two operations

Because NVKE marks but does not drain, the safe recycle pattern for GPU nodes is the usual one: cordon, wait for or checkpoint long-running jobs, drain, delete, and let the pool replace the node from the new release. Automate it in NVSentinel or your own controller, with a budget of one node per pool at a time for training pools.

9. Telemetry

The NVKE telemetry pipeline (NKTP) is an OpenTelemetry-based Helm chart that collects logs, metrics and traces and exports them over OTLP. Its default backend is for evaluation only: it keeps data in memory and gives anonymous users admin access to Grafana. Point it at your own backend before anyone relies on it. In our NVIDIA observability reference design we send OTLP to a Collector gateway that fans out to Grafana Mimir, Tempo and Grafana for metrics and traces and to the Elastic stack for logs, and add DCGM for GPU health and UFM for the InfiniBand fabric. NKTP slots in as one more OTLP source.

Signals worth alerting on from day one: management-plane controller errors, clusters stuck in Provisioning, nodes that fail attestation, nodes marked NeedsRecycle for longer than your recycle window, NICo capacity per instance type, and certificate expiry on handed-over kubeconfigs.

10. Preview limitations and what we do about each

11. How NVKE compares with other approaches

NVKE is not the only way to run Kubernetes on GPU servers. A fair comparison, at the level of design rather than features that change month to month:

The approaches are not exclusive. NVKR's model of pinned, signed node images is worth adopting whatever installs your clusters, and NVIDIA lists several distribution and platform vendors integrating NVKE components into their own products.

12. A staged evaluation plan

1. Week 1: local quickstart. A Linux host or VM (amd64 or arm64) with at least 6 CPUs, 12 GiB free memory and 100 GiB free disk, Docker and virtual TPM support; on macOS, NVKE's local-site guide runs inside a Colima VM. Follow it through workload-cluster creation and confirm nodes report Ready. Goal: the team understands the objects (site, release set, cluster, node pool) and the API.
2. Weeks 2 to 3: NVKR in your pipeline. Run the tutorial Definition, then the combined NVKE candidate; publish to a private registry; write a Verification Policy and verify authenticity and artifact-publication in CI. Goal: you can rebuild and verify every artifact you would deploy.
3. Weeks 3 to 6: hardware lab. Three management servers, three control-plane servers and two to four GPU servers on NICo. Validate image and imager on your hardware, bootstrap, secure the API as in section 7, publish two release sets and create two tenant clusters. Add AICR, run GPU smoke tests and a multi-node training job, recycle a node, upgrade a control plane. Goal: a written gap list against your service commitments.
4. After the preview: pilot. Re-run the lab on the generally available release, close the gaps, then pilot with one internal tenant before any external customer.

Exit criteria worth writing down before you start: time from request to Provisioned, node attestation failures, recycle time per GPU node, upgrade disruption to running jobs, and how many manual steps remain in each runbook.

13. Skills this asks of platform and DevOps teams

• Cluster API and bare-metal provisioning: machines, bootstrap and pivot, iPXE and imaging, VPCs and instance types.
• Supply-chain security: pinned inputs, SBOMs, provenance, signature policies and what each actually proves.
• Workload identity: SPIFFE/SPIRE, TPM attestation, certificate authorities and trust bundles.
• Multi-tenant Kubernetes: isolation, RBAC, credential life cycle, etcd encryption.
• GPU operations: node pools per GPU generation, runtime and driver qualification, safe recycling of nodes running long jobs.
• Observability: OpenTelemetry pipelines, GPU and fabric telemetry, alerting on provisioning and attestation failures.

14. How Workstation can help

We design, supply and operate GPU infrastructure in the UK. For teams looking at NVKE and NVKR we can plan the evaluation, size and supply lab and production hardware, design the management network and API hardening, build your NVKR verification gate into CI, connect NKTP, DCGM and fabric telemetry to an OpenTelemetry stack, and write the runbooks for recycling and upgrades. If you are not ready to run your own racks, our GPU private cloud page compares build options, and gpuleasing.co.uk compares leasing with buying. Talk to us or ask our AI Sales Engineer to size a cluster.

Notes and sources

• NVIDIA Technical Blog, Manage Kubernetes Clusters for GPU Infrastructure with NVIDIA Kubernetes Engine, 10 November 2026.
• NVKE documentation, including the preview status, supported platforms and API authentication pages.
• GitHub: dsx-ai-factory/nvke, dsx-ai-factory/nvkr, cluster-api-provider-nico and their public roadmaps.
• Both projects are in public preview. APIs, configuration and behaviour can change between releases, and NVIDIA does not yet recommend them for production workloads. Versions and limitations are as documented on 14 November 2026.
• Examples marked "Workstation example" are our own and are not part of NVIDIA's documentation.

Acknowledgement of rights: NVIDIA, the NVIDIA logo, DSX and CUDA are trademarks and/or registered trademarks of NVIDIA Corporation in the United States and other countries. Kubernetes is a registered trademark of The Linux Foundation. Other product and company names mentioned are the property of their respective owners. All such rights are acknowledged and reserved to their owners.