Uber has introduced Normalized Compute Units (NCUs) to make CPU allocation more consistent across its highly heterogeneous server fleet. The approach, implemented in Uber's Odin stateful orchestration platform, accounts for differences in processor performance rather than treating every CPU core as equivalent.
Odin manages around 5 million CPU cores across on-premises data centres and public clouds, including multiple generations of x86 hardware and ARM processors. Uber found that single-threaded performance can vary by almost a factor of two between the fastest and slowest cores. Consequently, the same workload can require significantly different numbers of physical CPU cores depending on where it runs.
The problem creates two competing risks. Allocating resources according to the slowest hardware can waste capacity on faster machines, while allocating according to average usage can leave workloads under-provisioned when they move to slower servers. For stateful workloads, where moving traffic to another machine is often not possible, the latter can become a reliability problem.
Uber's solution is to separate the amount of compute performance a workload requires from the physical cores used to provide it.
Odin assigns each server type a performance factor, using an older server generation as a baseline of 100. Newer hardware can have considerably higher values; Uber reports factors approaching 190 for some servers. These factors are initially determined during hardware qualification and then refined using production workload data.
CPU usage is converted into normalized units using these factors. Odin can then determine that a workload requires a particular amount of compute capacity and translate that requirement into a different number of physical cores depending on the server hosting it.
A workload on faster hardware may therefore receive fewer cores, while the same workload on slower hardware receives more. The objective is not to make the number of cores identical, but to make the computational capacity more comparable.
Uber also found that performance factors differ between technologies. Its measurements for Kafka, for example, showed substantially different performance between ARM, older on-premises hardware, and newer x86 systems. The company therefore updates factors using production measurements rather than relying exclusively on synthetic benchmarks.
For stateless services, differences between servers can often be addressed through load balancing. Traffic can be directed towards less-loaded or faster machines.
Stateful systems do not have the same flexibility. A request may need to reach the server containing a particular shard or piece of data. Moving the request elsewhere does not solve the underlying resource problem.
In the post, Uber describes workloads where a leader consumes more than ten times the CPU of its followers. If that leader moves from a fast server to a slower one, its CPU consumption can increase substantially, potentially causing resource pressure even though the workload itself has not changed.
Uber has rolled the approach out across approximately 1.4 million CPU cores of automatically scaled Odin workloads. For Kafka, CPU utilisation initially varied by around eight percentage points between server types, with the hottest hardware running at approximately 50% higher utilisation than the coldest.
After introducing performance-aware allocation, the difference fell to around three percentage points, with the hottest hardware only about 10% higher than the slowest. Uber also reduced total Kafka CPU allocation by approximately 8%, with much of the reduction occurring on faster hardware where the savings are potentially more valuable.
The approach differs from the CPU abstraction used by Kubernetes, where CPU requests are expressed as cores or millicores. Kubernetes deliberately treats a CPU unit as an absolute quantity of CPU time, regardless of the size of the underlying machine. That abstraction works well for portable workload definitions, but it does not inherently express the relative performance of different processor generations.
Uber's NCU model addresses a somewhat different problem: making the resource itself performance-aware. Rather than primarily deciding how many workloads should fit on a machine or how much CPU a workload should receive, it asks how much computational capacity a CPU allocation actually represents.