Agentic AI Shifts Data Center Workloads Toward CPUs

Agentic AI changes the balance of data center computing by adding work that conventional language models largely leave to the host system. An agent calls models, evaluates their responses, chooses what to do next and may call more models or services. That orchestration puts more pressure on CPUs, even as GPUs continue to handle most neural network computation.

Industry estimates cited by the article put the CPU-to-GPU ratio for conventional large language model (LLM) systems near 1:4, with agentic workloads potentially pushing it toward 1:1. The precise mix will depend on the system and its tasks, but the direction matters: teams planning AI capacity may need to account for CPU availability alongside accelerators. More CPU work can also raise demand and put upward pressure on prices.

Agents are part of a broader shift in AI architectures. Reasoning models, mixture-of-experts (MoE) designs and reservoir computing address different constraints, from response quality to computation and power use. Most still depend on GPUs or other accelerators for their central calculations. Agentic systems stand out because they add a layer of branching, sequencing and tool use around those models.

From model execution to orchestration

Commercial AI has relied on several kinds of neural networks. Convolutional neural networks (CNNs) became useful for analyzing image data, while recurrent neural networks (RNNs) handled sequences such as audio and language. LLMs have since expanded the role of language models: they represent input as tokens, and tokenization can be applied to many kinds of data, not just ordinary text.

LLMs have an advantage over RNNs when a task benefits from retaining a long context. That context can help a model choose words that fit what came earlier. RNNs, however, can process incoming audio frames as they arrive, without waiting to build a similarly long context. They can also need fewer resources. The choice between these approaches depends on whether a task values broad context, fast incremental processing or both.

Early LLMs were trained to predict the next likely token. That objective can produce fluent text without ensuring that the result is factually sound. Retrieval-augmented generation (RAG) addresses part of this problem by giving a model access to relevant information, often represented through embeddings. The model can then draw on retrieved material while generating an answer, rather than relying only on patterns learned during training.

Agentic AI adds another layer. An agent isn't itself a replacement for an LLM; it coordinates models and other tools. It can send a prompt to a model, inspect the result, select a subsequent action and repeat the cycle. That distinction turns a single model response into a sequence of work. An agent might, for example, take a travel request with a destination, duration and budget, then call services to arrange lodging, transport and activities. The model supplies part of the capability; the agent organizes the process and acts on its output.

That requires more than generating tokens. Agents must track progress and apply conditional logic: if a response has a particular result, take one route; otherwise, take another. Ed Doe, CEO and co-founder of Delos Data, said emerging research suggests agentic inference could eventually exceed other inference workloads combined. That's a forecast, not a settled measure of current demand, but it reflects the potential for agents to multiply model calls per request.

Rob Knoth, product management director in the digital and signoff group at Cadence, compared a well-trained agent to a team member who can be assigned an outcome and work through the steps with limited supervision. The practical value depends on how reliably the agent can handle those steps, and on how much oversight the task requires.

Satadal Bhattacharjee, Arm's global head of cloud and AI infrastructure silicon at Arm, described agents as systems that can call application programming interfaces (APIs) to carry out a multi-part request. He also pointed to scripts, including JSON, as a way to tune and sequence agent behavior. That programmability makes agents more than a conversational interface, but it also creates a need to manage the logic connecting each action.

Why CPUs are gaining work

In a conventional LLM server, the CPU often prepares work for the GPU and manages communication, including transfers over PCI. This host role can be relatively light, allowing one CPU to support several GPUs. The balance changes when software must make frequent decisions between model calls. CPUs are built to handle branching and general-purpose control flow, while GPUs are designed to carry out many similar calculations in parallel.

Agents therefore shift some of the work from accelerator setup to CPU execution. Ron Lowman, strategic marketing manager for IP at Synopsys, said the earlier focus was on how many accelerators could be connected to each CPU; agentic workloads increase CPU demand and change that ratio. Bhattacharjee similarly connected rising CPU demand to the need for more compute delivered with attention to power efficiency. Yong Meng Sua, CTO at QCI, said demand for agents has contributed to CPUs becoming much more expensive. Doe's shorthand: the CPU is back, though its role in the data center has changed.

The difference is not simply that an agent uses a CPU instead of a GPU. An agent still relies on models for tasks such as interpreting input or generating a response. The CPU handles more of the surrounding decision process, while accelerators continue to execute much of the neural network computation. That means agentic systems can increase demand for both kinds of processor, with the CPU taking a larger share of the coordination work.

Each request can also trigger multiple model calls. A person may submit one instruction, but the agent could break it into subtasks, query a model for each, use the results to choose later actions and check whether the task is complete. The number and kind of calls will depend on the agent's design. This extra activity has consequences for capacity planning: systems need to support the model inference and the control logic that surrounds it.

Reasoning models add computation

Agent orchestration addresses how a system decides what to do with model outputs. Reasoning models tackle a different question: how a model can spend more inference effort working through a problem before returning an answer. Depending on the system, it may generate additional intermediate tokens or provide an account of its reasoning process.

Bhattacharjee described reasoning as a way to combine knowledge of a topic with conversation context and information about the person asking. The goal is to produce a response informed by a wider set of considerations than the literal wording of a prompt. Reasoning models can also be improved through training feedback when a step in their process is wrong.

These additional calculations generally run on GPUs or other AI accelerators. Bhattacharjee characterized reasoning as part of model inference: producing extra tokens still involves neural network computation, which suits accelerators. Examples in commercial use include OpenAI's o-series models and DeepSeek-R1. Their demands therefore add to accelerator workloads, even while agent orchestration increases the CPU's role.

There is a separate question of how to establish that a model's answer is correct. Neural network weights are learned through training, which starts from an initial assignment and updates the values over many passes. Different starting weights can lead to different models that still perform the task. Since the computation is distributed across a large number of values, tracing a particular answer back to a clear, human-readable chain of causes can be difficult.

That difficulty matters in applications where teams need confidence in a result or a clear way to assess failure. Reasoning features may expose some of the steps behind an answer, and training can improve performance when errors are identified. But the existence of an explanation does not by itself settle whether a response is correct. For builders, evaluating behavior on relevant tasks remains a practical part of deployment.

MoE models route work to specialists

LLMs can consume substantial resources because a dense model applies its parameters throughout inference. Small language models (SLMs) offer one way to narrow the task: they focus on a particular subject, are trained for that specialty and operate as self-contained models. That can make them lighter to run, but their narrower scope limits what they can handle without help from another system.

MoE models take a different approach. Rather than using one set of parameters for every token, they replace some dense feed-forward network layers with multiple separate networks called experts. A learned router selects which expert or experts should process each token. As Bhattacharjee explained, the experts have distinct weights, and the routing function is part of the model. It generally runs on the accelerator alongside the rest of the model.

The router is trained along with the experts. During training, it learns which experts tend to perform well for particular tokens, then uses that information to route tokens during inference. The experts are typically trained jointly rather than assigned manually to subjects. A model may learn specialized behavior without its creators explicitly designating one expert for a particular field.

Processing commonly begins with a prefill phase, which parses the prompt and establishes its context. The model then routes relevant tokens through selected experts. For example, if a prompt concerns healthcare, an expert that has developed useful capabilities for that subject may be selected. The key-value cache (KV cache), which stores context accumulated during inference, grows as the process continues and informs later steps. It must be made available to the selected expert.

Priyank Shukla, director of product management for interface IP at Synopsys, pointed to moving the KV cache between the initial model and the relevant expert as a system consideration. That exchange, along with routing and memory access, means sparse activation doesn't make the rest of the infrastructure irrelevant.

MoE models can have a large total parameter count because they contain multiple experts. Yet only a subset is activated for each token, so not every parameter needs to participate in every inference operation. This sparse activation can increase a model's total capacity without requiring all of its parameters to run for every token. In comparison, an SLM processes the full input with its smaller, focused model; an MoE routes parts of an input through selected experts within a larger model.

The efficiency tradeoff depends on more than the number of active parameters. Bhattacharjee said an MoE can require less computation per token than a dense model with a comparable total parameter count, while routing, memory access and communication between devices add overhead. A system's design determines whether the reduced computation translates into a practical power or performance gain. Teams should consider how experts are placed and how data moves between them, alongside model size.

Reservoir computing takes a different route

Reservoir computing is a more specialized approach for sequential data. Most of the model is left random, and training focuses on the final layer, which maps the reservoir's output to the desired result. That simpler training process can suit tasks where rapid adaptation and low power matter more than broad, general-purpose capability.

Because the reservoir itself doesn't need to be trained in the usual way, much of it can be hard-coded into silicon. Only the final layer needs trained weights, reducing the amount of learned data that must be fetched. This approach has potential uses in speech and audio recognition, sensor processing, robot motor control and time-series prediction.

QCI is preparing a photonic reservoir computer for launch, according to Prajnesh Kumar, the company's quantum technology lead. Photonics can offer high bandwidth, and wavelength-division multiplexing (WDM) could allow multiple tasks to run at once on different wavelengths. Kumar illustrated the idea with ten wavelengths carrying ten tasks, provided they can use the same kernel without interfering with one another.

Kumar also discussed possible applications in trajectory planning for vertical takeoff vehicles and analysis of brain signals to classify epilepsy against other conditions. These examples point to the kind of targeted workload where specialized hardware may be useful. The product's launch and practical performance will show how well the approach translates from a promising mechanism to a deployable system.

A more varied compute mix

No single architecture is replacing all the others. CNNs remain useful for image tasks, while LLMs and agents are prominent in general-purpose AI. MoE designs offer a way to distribute model capacity among specialists, reasoning models spend more inference effort on selected problems, and reservoir computing targets temporal inputs with a different training and hardware profile.

For infrastructure teams, the near-term planning point is the split between model execution and the work around it. GPUs and other accelerators remain central to neural network operations. Agents add CPU-intensive sequencing and can turn one user request into many inference calls, so the previous CPU-to-GPU balance may no longer fit every deployment. MoE routing and cache movement add their own memory and communication costs, while specialized models can trade breadth for efficiency.

The next useful evidence will come from real deployments: how much CPU capacity agents consume, how often they call models, and whether sparse expert activation reduces total system cost after routing and data movement are included. Those measurements will determine whether the shift toward more CPUs becomes a standard design pattern or remains specific to certain workloads.

Fig. 1: An SLM condenses a specialized model, while an MoE routes tokens to selected experts. Source: Bryon Moyer/Semiconductor Engineering

Reference
1. Michael te Vrugt, “An introduction to reservoir computing,” book chapter to appear in: Artificial Intelligence and Intelligent Matter, Springer, Cham, arXiv:2412.13212v1

Related article
Agentic AI Is Changing Data Center Architectures
Standalone GPUs are being replaced by heterogeneous SoCs and chiplets that combine CPUs, GPUs, and NPUs to reduce memory bottlenecks and latency.

The post Agentic AI Alters The CPU-GPU Balance appeared first on Semiconductor Engineering.