Linux, tuned for AI

Choosing a distribution

There is no single best Linux for AI. The right choice depends on where it runs, who supports it and what else it has to fit into. This is how we usually decide.

Kernel & OS optimisations for AI workloads

GPUs only run fast when the host keeps them fed. These are the settings we check on every AI system, then tune against the customer's own workload. None of them is universal: we measure before and after.

A starting point (tested per workload, never copied blind)

# /etc/sysctl.d/90-ai.conf
kernel.numa_balancing = 0
vm.swappiness = 10
vm.max_map_count = 1048576
net.core.rmem_max = 268435456
net.core.wmem_max = 268435456

# kernel command line (GRUB)
iommu=pt transparent_hugepage=madvise

# services
systemctl enable --now nvidia-persistenced
systemctl enable --now nvidia-fabricmanager   # NVSwitch (HGX) systems
cpupower frequency-set -g performance

Kubernetes & Docker for AI

Most of the AI systems we build run in containers, from a single workstation running Docker to multi-node GPU clusters. The building blocks are the same on every distribution.

Native LLM APIs and how they scale

We serve models behind OpenAI-compatible APIs, so applications don't care which model or engine sits behind them. Scaling then happens in four directions.

1. 1

   Inside one GPU

   Quantisation (FP8, NVFP4 on Blackwell, AWQ), paged attention and continuous batching in vLLM, SGLang or TensorRT-LLM fit more users onto every GPU.

2. 2

   Across GPUs in a node

   Tensor parallelism over NVLink serves models too large for one GPU, with the lowest latency.

3. 3

   Across nodes

   Pipeline parallelism with Ray or Kubernetes LeaderWorkerSet for the largest models, and disaggregated prefill and decode (NVIDIA Dynamo, llm-d) at cluster scale.

4. 4

   More replicas

   Horizontal scaling behind a gateway, autoscaled on queue depth or tokens per second with KEDA or the HPA, with KV-cache-aware routing and scale to zero via KServe.

Engines we deploy: vLLM, SGLang, TensorRT-LLM with Triton Inference Server, NVIDIA NIM, and Ollama or llama.cpp for desk-side and edge systems. Many LoRA adapters can share one base model on the same GPUs.

CI/CD for LLMs, agents and data pipelines

AI changes more often than ordinary software: new model versions, new prompts, new data. We give each of the three its own pipeline, with the same discipline: everything versioned, tested, signed and promoted through rings.

CI runs on self-hosted runners inside your network, including GPU runners for evaluation, so models and data never leave it.

How we secure Linux for AI

An AI server holds your most valuable data and some of your most expensive hardware. We secure it in layers, so that no single mistake exposes either.