Shop is live

We build Workstation AI machines for websites, AI agents, and LLM training — configure and buy online.

Get Trade Prices
AI Products
AI LabsOpenAI AgentsClaude AgentsGrok BotWorkstation AI native CRMMarketingAll Products
AI Solutions
Applied AIAI AgentsGenerative AIAI AutomationAI UpskillingAI SME PackagesEnterprise AI LabAI WorkstationsNVIDIA Hardware & SoftwareNVIDIA Accelerated ComputingPrivate AIGPU Private CloudGPU ClustersEdge AIAI by Industry
AI Services
Platform ModernisationDigital EngineeringData Foundations & AIAutonomous OperationsAI ConsultancyDevOps AutomationCyber SecuritySoftware DevelopmentAgent BuildingMLOps SetupGPU LeasingGPU Private Cloud
About Us
CompanyOur StoryPartnersCustomer Stories
AI Data Centres
AI Data CentresGPU Private CloudGPU ClustersGPU Leasing
AI InsightsJev seriesBenchmark LabModel selection
Technical Notes
Documentation
Linux OSUbuntuRed Hat Enterprise LinuxSUSE LinuxAlpine LinuxDebian, Rocky & AlmaLinux
AI Cloud PlatformsAWS & Amazon BedrockGoogle Cloud & Vertex AIMicrosoft Azure & AI FoundryGPU clouds & marketplaces
Private AI Data CentresFor law firmsFor accounting firmsOn-premises platformsFully secured cloud
ToolsRing PromoterJobshout
News
Blog
Contact UsSign in

AI workstations, AI Multi Agentic Software, GPU infrastructure, and intelligent agent solutions for modern businesses.

Contact Us

AI Solutions

Applied AIAI AgentsGenerative AIAI AutomationAI UpskillingAI SME PackagesEnterprise AI LabAI by Industry

Products

All ProductsWorkstation AI native CRM & ERPMarketingOpenAI AgentsRing PromoterJobshout

Company

About UsCompanyAI Data CentresWhy WorkstationPartnersCustomer StoriesPricingContactTenthMatrix

Resources

AI InsightsTechnical NotesDocumentationLinux OSAI Cloud PlatformsPrivate AI Data CentresBlogSearchSitemap
UK Head Office
77-79 Marlowes, Hemel Hempstead HP1 1LFDirections - Take Junction 20 off M25 Outer LondonCompany No: 11641870Mon - Fri: 9:00 AM - 6:00 PM GMT
+44 7515 356 146
Belgium Office
Workstation SRL, Rue Vanderkindere 34, 1180 Uccle, BrusselsBE 0751.518.683Mon - Fri: 9:00 AM - 6:00 PM CET
+32 492 45 67 46
India Office
#159 Sector 9, Pocket 1, DDA Flats, 110077 Dwarka, New Delhi
+91 98881 98841
Taiwan Office
3F, No. 2-4, Aisi Rd., Ren'ai Dist., Keelung City 200, Taiwan基隆市仁愛區愛四路2之4號3樓Mon - Fri: 9:00 AM - 6:00 PM Taiwan time (GMT+8)
+886 926 892 701+32 492 45 67 46
Estonia Office
Aleksander Puškini tn 30-25, 20306 Narva, EstoniaMon - Fri: 9:00 AM - 6:00 PM Estonia time (EET)
+32 492 45 67 46

© 2026 Workstation AI. All rights reserved.

PrivacyData DeletionCookiesTerms of ServiceWebsite Sitemap

NVIDIA, the NVIDIA logo, RTX, RTX PRO, Blackwell, CUDA, NVLink, MGX, Omniverse, Spectrum-X, Quantum, BlueField, DGX and HGX are trademarks and/or registered trademarks of NVIDIA Corporation in the US and other countries. AMD, the AMD Arrow logo, EPYC, Instinct, Ryzen and Threadripper are trademarks of Advanced Micro Devices, Inc. Dell and PowerEdge are trademarks of Dell Inc. or its subsidiaries. Intel and Xeon are trademarks of Intel Corporation or its subsidiaries. AMAX, AceleMax and LiquidMax are trademarks of AMAX Engineering Corporation. All other product names, logos and brands are the property of their respective owners. We use them only to identify the products we describe, supply or support; their use does not mean their owners endorse us.

Loading blog...

Home / Blogs
Engineering

Balancing Safety and Speed in Kubernetes Rolling Updates

JobShout AI Blog Writer Agent (opens in a new window)November 18, 20268 min read
Cover

Optimizing deployments in Kubernetes is not merely about making things faster; it is about engineering a trade-off between safety margins and rollout velocity. In cloud-native environments, the goal is to design, develop, and deploy dependable, dynamic services that meet the demanding requirements of the industry [1]. However, achieving zero-downtime deployments while maintaining high throughput requires a deep understanding of how Kubernetes interacts with underlying infrastructure, load balancers, and build systems. This article explores the technical challenges of rolling updates, the performance optimizations embedded in recent Kubernetes versions, and the tools that help developers navigate the complexity of fast, reliable deployments.

The Challenge of Rolling Updates

A rolling update is designed to update a service's replicas without interrupting service availability. However, the process involves a complex interplay between the Kubernetes API server, the kubelet on the nodes, and the external load balancer. A critical challenge arises during the termination phase of an old pod.

When an application pod is scheduled for termination, the AWS Load Balancer (or similar LBs) continues to send new requests to the target for several seconds after the application has received the termination signal [2]. This behavior creates a race condition: the application may be in the process of shutting down or flushing connections, yet it is still receiving new traffic.

Consequently, determining the amount of time to wait for the endpoint change to fully propagate to the target group is a balancing act between safety margin and rollout speed [2]. If the wait is too short, you risk dropping in-flight requests. If it is too long, your deployment pipeline slows down significantly, increasing feedback loops for developers.

Diagram

Note: The sequence above illustrates the race condition where the Load Balancer may continue routing traffic to a pod that has already received a termination signal [2].

To mitigate this, engineers often attempt to use a sidecar container with a preStop lifecycle hook to introduce a deliberate delay before the container terminates. However, this approach has limitations. Specifically, adding a sidecar container with a preStop lifecycle hook does not work for distroless base images, which lack the necessary shell or init system to execute these commands [2]. This forces teams to rely on other mechanisms, such as graceful shutdown handling within the application code itself, or to accept the risk inherent in the load balancer's propagation delay.

Kubernetes Performance and Scalability

The ability to perform rapid rolling updates is underpinned by the core performance of the Kubernetes cluster. Since Kubernetes 1.1, significant performance tweaks and enhancements have made it possible to serve 1 million requests per second [1]. This baseline performance is crucial because rolling updates often involve scaling up new pods before terminating old ones, placing temporary load on the control plane.

In Kubernetes 1.2, density was further improved from a pods-per-node perspective, allowing up to 100 pods on a single node, compared to 30 in the 1.1 release [3]. This increase in pod density means that a single node can handle more concurrent workloads, which is beneficial during high-churn deployment windows.

Two specific optimizations in Kubernetes 1.2 are particularly relevant to the stability of rolling updates:

  1. PLEG (Pod Lifecycle Event Generator) Optimization: The PLEG component is responsible for polling the state of containers. In previous versions, this polling could be expensive. The introduction of concurrent polling elimination in Kubernetes 1.2 allowed the system to lower the steady-state CPU usage of the Kubelet and the container runtime by 4x [3]. Lower CPU contention on the node ensures that the kubelet can respond quickly to update signals and termination requests, reducing the time between "intended state" and "actual state."
  2. JSON Encoding/Decoding Speedups: Communication between the Kubernetes API server and the kubelets relies heavily on JSON serialization. The adoption of the ugorji codec library provided a 200% improvement in encoding and decoding JSON when using generated serializers [3]. This faster data transfer means that status updates and watch events propagate more quickly through the cluster, which is essential for coordinating the phased rollout of pods across many nodes.
Component Optimization in K8s 1.2 Impact on Rolling Updates
PLEG Eliminated concurrent polling Reduced Kubelet CPU usage by 4x, allowing faster reaction to pod lifecycle events [3]
JSON Codec Use of ugorji codec library 200% improvement in JSON encode/decode speed, accelerating API communications [3]
Pod Density Increased max pods per node Supports up to 100 pods/node (vs 30 in 1.1), allowing higher churn rates on single nodes [3]

The Role of AWS Load Balancers

As discussed in the challenges of rolling updates, the interaction between Kubernetes and cloud-native load balancers is a primary source of deployment friction. The AWS Load Balancer behavior is not a bug, but a design choice that prioritizes connection availability over immediate state synchronization.

The fact that the AWS Load Balancer keeps sending new requests to the target for several seconds after the application is sent the termination signal [2] means that preStop hooks must be carefully calibrated. For standard images, a sidecar container can be used to sleep for a specific duration before the main container stops, effectively holding the port open while the LB deregisters the target.

However, this solution is not universal. For distroless images, which are minimal and immutable, this sidecar approach fails because the necessary tools to execute the delay are missing [2]. In these cases, the application must implement its own graceful shutdown logic, such as:

  • Stopping accepting new requests.
  • Waiting for active requests to complete.
  • Exiting the process.

The application must exit before the termination grace period expires, but after the load balancer has removed it from the target group. Since the exact propagation time varies, developers must often add a buffer to the terminationGracePeriodSeconds setting. This buffer is the "balancing act" between safety and speed [2].

Tilt: A Faster Way to Update Kubernetes

While optimizing the cluster and load balancer behavior reduces downtime, it does not necessarily speed up the feedback loop for developers. Traditional deployment workflows involve building Docker images, pushing them to a registry, and then instructing Kubernetes to pull the new image. This process can take minutes, even for small changes.

Tilt addresses this by making Kubernetes updates fast—really fast. Instead of minutes, Tilt updates Kubernetes in seconds [4]. This is achieved through two key mechanisms:

1. Incremental Builds

In a standard Docker workflow, creating a new layer with the generated/compiled artifacts happens for every update [4]. Even if you only changed one file, Docker will create a new layer and start a build from scratch. Tilt's incremental build API makes it easy to use the build cache on subsequent builds, with realistic builds improving from 30 seconds to 1 second [4].

2. File Syncing via Synclet

Rather than always replacing the entire pod, Tilt can reuse existing pods. A "Synclet" runs on the same node as the pod. When you update files, the Synclet adds them to the existing pod and restarts the container [4]. This bypasses the image pull and container creation overhead entirely, providing a near-instant update experience that is "cloud-as-fast-as-laptop" fast [4].

Approach Mechanism Typical Update Time Use Case
Traditional Docker New layer, full build, image pull Minutes Production releases, full integration tests
Tilt Incremental build, file sync to pod Seconds Development, rapid iteration, local testing [4]

Note: Tilt reuses existing pods by adding updated files to the existing pod and restarting the container, which significantly reduces overhead compared to standard image updates [4].

Best Practices for Rolling Updates

To balance safety and speed effectively, developers should adopt a multi-layered strategy:

  1. Understand LB Propagation Delays: Acknowledge that AWS Load Balancers (and similar services) continue routing traffic for several seconds after termination signals [2]. Do not rely on preStop hooks alone if using distroless images, as these hooks may not function as expected [2].
  2. Leverage Kubernetes Performance Gains: Ensure your cluster is running on a version that benefits from PLEG optimizations and faster JSON serialization [3]. This ensures the control plane can handle the churn of rolling updates without becoming a bottleneck.
  3. Optimize the Development Loop: Use tools like Tilt for rapid iteration to reduce the cognitive load of waiting for builds [4]. By using incremental builds and file syncing [4], you can verify that your rolling update configuration works correctly in seconds, not minutes.
  4. Monitor Service States: Use tools that can track the state of services, such as Gitopper, which can track states like OK, FREEZE, ROLLBACK, BROKEN, and DIFF [5]. This allows you to roll back a service to a specific commit if a rolling update fails [5], providing a safety net when pushing for speed.

By combining an understanding of the underlying infrastructure behaviors (like LB propagation and Kubelet performance) with modern development tooling (like Tilt), teams can achieve deployments that are both fast and safe. The goal is not to eliminate downtime at all costs, but to manage the trade-offs intelligently, ensuring that the "balancing act" [2] results in reliable, high-velocity software delivery.

References

  1. One million requests per second: Dependable and dynamic distributed systems at scale | Kubernetes — blog.kubernetes.io (11 November 2015)
  2. Zero-Downtime Kubernetes Deployments on AWS with EKS | Glasskube — glasskube.dev (10 March 2025)
  3. 1000 nodes and beyond: updates to Kubernetes performance and scalability in 1.2 | Kubernetes — blog.kubernetes.io (28 March 2016)
  4. How Tilt updates Kubernetes in Seconds, not Minutes — medium.com (7 January 2019)
  5. miekg/gitopper — github.com (27 August 2024)