Ahmet Alp Balkan

Kubernetes List API performance and reliability

Ahmet Alp Balkan — Wed, 09 Jul 2025 10:39:11 -0700

At my current employer, we use Kubernetes to run hundreds of thousands of bare metal servers, spread over hundreds of Kubernetes clusters. We use Kubernetes beyond officially supported/tested scale limits by running more than 5,000 nodes and over a hundred thousand of pods in a single cluster.¹ In these large scale setups, expensive “list” calls on the Kubernetes API are the achilles heel of the control plane reliability and scalability. In this article, I’ll explain which list call patterns pose the most risk, and how recent and upcoming Kubernetes versions are improving the list API performance.

From Metal To Apps: our Kubecon EU 2025 talk

Ahmet Alp Balkan — Thu, 03 Apr 2025 11:45:00 +0100

We presented our LinkedIn compute infrastructure team’s journey moving LinkedIn’s large 500,000+ bare metal servers running thousands of microservices and a lot of stateful workloads to a Kubernetes based platform. In this session, we talk about LinkedIn’s scale, how we automate bare metal server management and maintenance from the ground up, built Kubernetes node and cluster management layers for our needs, and how we’re building workload platforms for stateless, stateful and batch workloads.

LinkedIn on the Kubernetes Podcast

Ahmet Alp Balkan — Mon, 10 Mar 2025 20:19:29 +0000

Ronak and I were Abdel’s guests at the Kubernetes Podcast by Google ahead of our KubeCon talk in London next month. We talked about our work building the next generation of the compute infrastructure at LinkedIn with Kubernetes, the challenges we faced and our journey dealing with the scale and complexity so far.

Every pod eviction in Kubernetes, explained

Ahmet Alp Balkan — Thu, 27 Feb 2025 20:19:29 +0000

Anyone who is running Kubernetes in a large-scale production setting cares about having a predictable Pod lifecycle. Having unknown actors that can terminate your Pods is a scary thought, especially when you’re running stateful workloads or care about availability in general.

There are so many ways Kubernetes terminates workloads, each with a non-trivial (and not always predictable) machinery, and there’s no page that lists out all eviction modes in one place. This article will dig into Kubernetes internals to walk you through all the eviction paths that can terminate your Pods, and why “kubelet restarts don’t impact running workloads” isn’t always true, and finally I’ll leave you with a cheatsheet at the end.

So you wanna write Kubernetes controllers?

Ahmet Alp Balkan — Wed, 22 Jan 2025 21:26:45 +0000

Any company using Kubernetes eventually starts looking into developing their custom controllers. After all, what’s not to like about being able to provision resources with declarative configuration: Control loops are fun, and Kubebuilder makes it extremely easy to get started with writing Kubernetes controllers. Next thing you know, customers in production are relying on the buggy controller you developed without understanding how to design idiomatic APIs and building reliable controllers.

Low barrier to entry combined with good intentions and the “illusion of working implementation¹” is not a recipe for success while developing production-grade controllers. I’ve seen the real-world consequences of controllers developed without adequate understanding of Kubernetes and the controller machinery at multiple large companies. We went back to the drawing board and rewritten nascent controller implementations a few times to observe which mistakes people new to controller development make.

Notes on OpenAI Kubernetes outage

Ahmet Alp Balkan — Mon, 18 Nov 2024 17:01:00 +0000

Last week, OpenAI has suffered a several hours long outage and published a detailed postmortem about it. Highly recommend reading it. These technical reports are usually a gold mine for all large-scale Kubernetes users, as we all go through similar set of reliability issues running Kubernetes in production.

Tale of a Kubernetes node-feature-discovery incident

Ahmet Alp Balkan — Fri, 15 Nov 2024 00:00:00 +0000

This is the analysis of a low severity incident that took place in the Kubernetes clusters at the company I work at that taught me a lot about how to think about the off-the-shelf components we bring from the ecosystem into the critical path and operate at a scale much larger than these components are intended.

Kubernetes CRD generation pitfalls

Ahmet Alp Balkan — Tue, 10 Sep 2024 09:57:53 -0700

A quick code search query reveals at least 7,000 Kubernetes Custom Resource Definitions in the open source corpus,¹ most of which are likely generated with controller-gen —a tool that turns Go structs with comments-based markers into Kubernetes CRD manifests, which end up being custom APIs served by the Kubernetes API server.

At LinkedIn, we develop our fair share of custom Kubernetes APIs and controllers to run workloads or manage infrastructure. In doing so, we rely on the custom resource machinery and controller-gen heavily to generate our CRDs.

Why Kubernetes secrets take so long to update?

Ahmet Alp Balkan — Wed, 28 Dec 2022 13:45:00 +0000

I’ve recently done a Twitter poll and only 20% of the participants accurately predicted that it takes Kubernetes 60-90 seconds to propagate changes to Secrets and ConfigMaps on the mounted volumes. So I want to take you on a journey in the codebase on how the mechanics of these volume types work and why it takes so long. Before going on this journey, I would answer the poll “nearly instantly” (like the majority 40% did).

Pitfalls reloading files from Kubernetes Secret & ConfigMap volumes

Ahmet Alp Balkan — Thu, 22 Sep 2022 16:00:37 +0000

Files on Kubernetes Secret and ConfigMap volumes work in peculiar and undocumented ways when it comes to watching changes to these files with the inotify(7) syscall. Your typical file watch that works outside Kubernetes might not work as you expect when you run the same progam on Kubernetes.

On a normal filesystem, you start a watch on a file on disk with a library and expect to get an event like IN_MODIFY (file modified) or IN_CLOSE_WRITE (file opened for writing closed) when the file is changed. But these filesystem events never happen for files on Kubernetes Secret/ConfigMap volumes.