How Hyperscale Computing Works: What It Takes to Run AI at Massive Scale

AI deployed at scale is the ultimate test for hyperscalers as well as the industry’s largest cloud providers. To run efficiently, CPUs, accelerators, memory, networking, power, packaging and silicon all need to work together to turn enormous amounts of computation into useful results. The clearest way to understand the challenge is to separate the layers, from system right down to silicon.

Hyperscale infrastructure is designed to expand efficiently and reach enormous scale. At a very basic level, it distributes the load and increases performance by adding more servers to the total system, and it doesn’t treat each new server as an isolated machine. Instead, hyperscale operators design across servers, racks, and data centers so compute, memory, storage and networking can be deployed in vast numbers and run as one coordinated system. This coordination is especially important for large-scale AI workloads and deployments.

One prompt is not always a single request. An AI agent may reason with a model, retrieve information, call an application, execute code and query a database, returning to the model several times before it finishes a task. Bigger tasks can require many processors working together, with access to immense quantities of data, over connections fast enough to keep expensive hardware from sitting idle.

That is why the conversations for AI and inference have shifted beyond “How fast is the processor?” or “How many GPUs is enough?” Performant AI infrastructure is a systems engineering problem, and the efficiency of the whole system determines your overall performance.

There is no universal “AI processor.” Until relatively recently, the debate about the best processor for AI was pretty one dimensional. GPUs and other specialized accelerators reigned supreme with good reason: Training increasingly large AI models required enormous amounts of parallel computation, and running inference on the largest models also requires enormous parallel computation, which is what GPUs are designed for. But the landscape is changing, and a new mix of critical AI workloads is emerging. It looks less like large model inference and more like agentic AI.

CPUs play a much broader role than accelerators: they run operating systems, applications and databases, retrieve and prepare data, manage storage and networking, enforce security, and orchestrate the work around the accelerators. Where this matters most is in this new landscape of agentic AI. If an accelerator finishes the model math in record time, but it sits and waits for retrieval, or for an application or another service to pick up the baton, the accelerator’s speed becomes irrelevant. If the measure of success is how fast the user’s task is completed, no one part can carry the team. This is a group project where the entire system is graded as one.

Because different AI workloads have different compute needs, heterogeneous systems can match each task to the architecture best suited to run it, using CPUs, GPUs and other accelerators together rather than forcing every workload onto a single type of processor. The goal isn't CPU or GPU. It's to send the right workloads to the right hardware and keep every resource productive, which has been the entire premise of hyperscale computing for decades.

Processors have become extraordinarily fast, but computation requires data, and the data has to arrive quickly enough to keep them working. Imagine a very fast factory whose assembly line keeps stopping because materials haven't arrived. Speeding up the machines won't increase output. The same principle applies to computing: a CPU or accelerator that waits on memory wastes some of its theoretical capacity. There are four areas where memory really matters:

• Capacity: How much data can the system keep available?
• Bandwidth: How quickly can that data be delivered?
• Latency: How long does the processor wait to receive it?
• Placement: Where does memory sit relative to the processors using it?

As models grow and more AI applications run at once, pressure builds on all four. Technologies like Compute Express Link (CXL) help by enabling more flexible connections between processors, accelerators and memory, including memory expansion and pooling.

Within a data center, information moves constantly: between processors and accelerators, between processors and memory, between chiplets inside a single package, and between servers across a rack. All of those links have different needs. Some information only needs to move a few millimeters. In other cases, it travels a much longer distance. Some connections demand the lowest possible latency and others enormous aggregate bandwidth.

A range of technologies work together across the system to move data quickly and efficiently, helping compute, memory and accelerators function as one coordinated system rather than allowing the network to become a bottleneck. Different layers of interconnect technology work together with the singular goal of minimizing bottlenecks system-wide. PCI Express connects processors with devices and accelerators. CXL extends that foundation for coherent memory expansion. UCIe connects chiplets within an advanced package. It’s also why the industry keeps pushing toward faster Ethernet and faster underlying signaling, such as 224G-class SerDes (serializer/deserializer) technology, which converts parallel data inside a chip into high-speed serial signals and supports the move toward 800 gigabit and 1.6 terabit networking.

The individual numbers matter less than the reason they keep rising: more AI compute creates more data that needs to move, and more data movement requires faster connections.

At hyperscale, every watt matters. Processors, memory and networking all consume power. Moving data consumes energy. All of it generates heat, so cooling the equipment consumes more energy still. Multiply that across a multitude of racks and data centers, and small tradeoffs in efficiency can become significant differences in power demand and operating cost.

A rack full of processors is only valuable if the facility can power and cool it, so engineers have to optimize performance, utilization, power efficiency, cooling and density at the same time. So, the question changes from “How much compute can we fit in a data center?” to “How much useful work can we deliver with the power we have available?”

Operators will often ask how much useful work a rack delivers within a lower power envelope. In plainer terms: “Forget how fast the car can travel. How far can we go with the fuel we have in the tank?” This is why performance per watt is a design consideration from the data center all the way down to the transistor.

The power problem goes beyond the data center. Power efficiency is a consideration across every layer of silicon and semiconductor technology. Better transistors can deliver more computation with less energy. Process innovations like RibbonFET gate-all-around transistors and PowerVia backside power delivery improve performance per watt in different ways. RibbonFET improves electrostatic control and supports more efficient performance scaling, while PowerVia improves transistor control and power delivery by moving power routing to the back of the die, freeing up front-side real-estate for signal routing and reducing power-delivery congestion.

Advanced packaging innovations also allow for greater transistor density, so architects can overcome the physical limits of how large a single die can practically be. Building every function into one enormous piece of silicon doesn’t always make sense.

The alternative is disaggregation, where functions are built as separate pieces of silicon, called chiplets, and combined in the same package. That only works if the pieces can exchange data extremely quickly and efficiently, which is where packaging technologies like Embedded Multi-die Interconnect Bridge (EMIB) and Foveros Direct 3D come in, connecting multiple dies at high densities.

Intel's next generation server processor, codenamed Diamond Rapids, demonstrates this approach with unified memory fabric, built with Foveros Direct 3D packaging and UCIe-S die-to-die interconnect. This shift, where the package itself is a critical part of the computing architecture, is necessary for delivering scalable, power-efficient compute.

The best hardware and software work together to maximize performance depending on the environment and myriad shifting workload constraints. Hardware provides the capabilities, but it’s software that determines how those capabilities are used and makes the system sing. Model-serving software, operating systems, frameworks, orchestration, telemetry and policies coordinate the resources across increasingly complex and mixed environments. While a closed ecosystem can offer simplification in a highly complex environment, homogenous solutions can restrict efficiency and scale when workloads shift and new technologies emerge.

Open standards let developers and operators work through familiar frameworks while the underlying system evolves, which is critical when the workload is as world-changing and fast-paced as artificial intelligence. These standards also let architects choose CPUs, accelerators, memory and networking to suit the workload, rather than forcing it into a single proprietary architecture. Heterogeneous infrastructure can help ensure hyperscale computing is deployable, manageable and economically useful long term.

All of this changes how AI performance should be measured. TOPS, FLOPS, tokens per second, core counts and model parameters remain useful, but none of them show how well a system performs.

For a real application, architects also weigh end-to-end task latency, throughput at the required service level, utilization, data-movement overhead, power and cooling, security and governance, and total cost. For agentic applications, cost per completed task is a useful system-level lens. A single task can involve many operations before an outcome is produced, which changes the cost analysis of what it takes to deliver completed work.

No single component can deliver AI at scale. As AI systems grow larger and become embedded in more applications, the industry will increasingly have to optimize not just for how fast an individual component can run, but for how effectively the entire infrastructure can turn available resources, including compute, memory, bandwidth and power, into useful work. This is what makes hyperscale computing a systems engineering challenge, and it’s why compute, memory, networking, power, packaging, software and silicon need to advance together.

For more information about scaling with hyperscale infrastructure and in the cloud: