The New Landscape of AI Compute

Where is AI compute headed? Today, discussion of AI compute tends to focus on massive data centers, high-end graphics processing units (GPUs) and ever-larger training clusters. But will the future look the same?

Leading technology companies are investing enormous sums in new AI data centers. Meta has expanded its investment in the Hyperion campus in Louisiana to more than $50 billion and its compute capacity to 5 gigawatts (GW). The project will become Meta’s largest AI data center and one of the world’s largest AI infrastructure projects.

Training larger models requires more chips, faster networks, stronger power supplies and more sophisticated cooling systems. Large-scale data center campuses seem to have become the defining infrastructure of the AI age.

But will AI compute only become more concentrated and larger in scale? In our view, not necessarily. Even as large training centers continue to expand, another shift is underway: AI compute is moving into distributed regional data centers, corporate server rooms, factories, cars, robots and other intelligent devices.

The future of AI compute will not simply be centralized or distributed. Two trends will coexist: the compute used to pretrain frontier models will remain relatively concentrated, while inference will become increasingly differentiated by task.

This does not mean the cloud will become less important. Frontier and complex inference will continue to run extensively in the cloud; private deployments within enterprises will continue to increase; and more inference will move to the edge and onto devices. Different computing tasks will run at different layers of infrastructure.

This shift will not only reshape AI infrastructure and create new demand for chips. It will also change how companies organize themselves, run their processes and make decisions.

Why frontier pretraining will remain concentrated

Training a frontier model is a highly complex systems-engineering undertaking.

Training a large model typically requires thousands or tens of thousands of GPUs or other AI processors to work on the same task. The chips must communicate continuously at low latency. Any communications bottleneck can reduce the efficiency of the entire cluster.

Training capacity depends on more than the performance of individual chips. The system must also connect large numbers of chips efficiently. NVIDIA uses NVLink and NVLink Switch to improve communication among GPUs. Google’s tensor processing unit (TPU) systems similarly use dedicated high-speed networks to organize thousands of chips into unified compute clusters.

What these systems have in common is that they concentrate large amounts of compute and make different chips operate as a single unit. This concentration occurs primarily in frontier-model pretraining, which depends on large-scale, high-bandwidth, low-latency coordination among chips. Large AI data centers will therefore continue to handle frontier-model pretraining and other highly synchronized computing workloads.

But concentrated compute does not mean model development will also become concentrated in the hands of a few companies. As open-source models advance, more companies and research teams can build on foundation models.

Companies in sectors such as finance, manufacturing and pharmaceuticals often have deep domain knowledge and valuable proprietary data. They may choose to further fine-tune, continue training or adapt models to their own businesses. These tasks require far less compute than frontier-model pretraining and can be supported by cloud platforms, specialist providers or the companies themselves.

Most consumer goods, retail and general services companies, by contrast, will not need to train models further. What matters is using existing general-purpose models and connecting them with proprietary knowledge, data and business tools.

But training is only part of AI.

Inference is becoming more differentiated

Inference is what happens when a trained model is put to work — answering questions, running agents, recognizing images, analyzing on-site information or helping companies make decisions.

Unlike frontier training, many inference tasks can run independently without thousands of chips working in parallel. Depending on where users, data and applications reside, these tasks may run in the cloud, inside the enterprise, at the edge or on devices.

In recent years, long-horizon agents, context engineering and multi-step tool use have increased the compute required for some complex inference tasks. They must process longer contexts, invoke more models and tools and maintain state over extended periods. For the foreseeable future, most complex inference will therefore remain in the cloud. Cloud data centers can pool compute across users and tasks, while high hardware utilization, mature software and professional operations help reduce unit inference costs.

On the other hand, on-site sensing, real-time prediction and some inference tasks related to device operations are better suited to distributed deployment. As AI enters more real-world settings, two forces pushing inference outward from the cloud are becoming clearer.

The first is private deployment within the enterprise. Certain data in healthcare, manufacturing, finance and government are highly sensitive and subject to regulatory restrictions, making them difficult to move into public clouds. For these organizations, the reasons are straightforward: data security, privacy, regulatory requirements and control. They may need to run AI in environments they can control themselves.

Companies can run AI in their own data centers, private clouds or dedicated compute nodes, keeping sensitive data and critical operations within controlled environments. Azure Local, Microsoft’s distributed infrastructure solution, allows companies to run cloud services and AI models on local infrastructure. This enables them to perform inference and analysis while keeping certain sensitive data on-premises.

The second force is edge and on-device computing. Both bring compute closer to where data are generated and business takes place, but they operate in different locations. Edge computing runs mainly on servers or compute nodes near operating sites, such as factories, campuses or carrier networks. On-device computing runs directly in cars, robots, cameras and other intelligent devices.

The main drivers of edge and on-device computing are that both data generation and execution occur on site, while some tasks require faster response times, continuous network connectivity and the ability to operate offline.

Physical distance directly affects an AI system’s response time. Sending data from a site to the cloud and returning a result requires multiple stages of network transmission and processing. The farther data must travel, the greater the latency and the more uncertainty the system faces.

A brief delay rarely matters for general knowledge queries or back-office analysis. But some critical situations demand real-time response. For example, vehicle braking and emergency shutdowns of industrial equipment require systems to react within extremely short windows.

For safety-critical and real-time control tasks, the core control loop must therefore remain on site. The cloud can handle less time-sensitive work, such as model training, version updates and data analysis. Edge or on-device AI can support sensing, prediction and decision-making. But it still needs to work together with validated real-time control systems.

In other words, the workloads best suited to edge and on-device deployment do not necessarily involve complex inference by large models. They are often tasks closer to the physical world, such as perception, on-site prediction and decision support.

An individual device may not require much compute, but the number of cars, robots and other intelligent devices could be enormous. Over time, devices may therefore become an important source of growth in distributed inference. Companies such as Amazon Web Services (AWS) have already begun deploying some real-time inference capabilities to factories, operating sites and devices, with certain nodes able to continue operating offline.

“An individual device may not require much compute, but the number of cars, robots and other intelligent devices could be enormous.”

A multilayered AI infrastructure is taking shape

AI infrastructure will likely evolve into a multilayered system extending from the cloud to the edge and then to devices. Large AI data centers will handle frontier-model pretraining and complex inference; edge nodes will provide real-time inference and data processing near operating sites; and devices such as cars and robots will perform sensing, prediction and decision support directly. Final control of safety-critical tasks will remain with validated real-time control systems on site.

At the same time, companies with stringent data security, privacy and compliance requirements will run some AI privately in their own data centers, private clouds or edge nodes.

Energy conditions will also shape this multilayered system. The International Energy Agency expects global data-center electricity consumption to nearly double to around 945 terawatt-hours (TWh) by 2030.

Once built, large compute centers are geographically fixed, making power supply a major constraint on further expansion. Power supply capacity, transmission and distribution conditions, and grid-connection conditions directly determine a center’s room for expansion and ultimate capacity. By contrast, the location of edge nodes depends more on operating sites, real-time requirements and network conditions. On-device compute in cars, robots and other intelligent equipment must move with the device and cannot simply be relocated in response to energy costs.

Therefore, distributed computing will not solve the energy problem by itself. What it changes is where different types of compute take place. Large-scale training and complex inference will remain mainly in compute centers with stable power and infrastructure; more on-site inference will move to the edge and onto devices as business needs dictate. Some companies will also adopt private deployments to maintain data security and control.

The new compute landscape will drive demand for new chips

Distributed inference does not mean moving large training GPUs into small server rooms or intelligent devices.

Training GPUs are designed for maximum computing power, large-scale parallelism and high-speed interconnects. Chips deployed at the edge or on devices, however, face tight constraints on power, space, cooling and cost. These settings call for high energy efficiency, low latency, large local memory and hardware that is compact and easy to deploy.

Even if the basic architecture of AI models gradually converges, inference chips will still offer considerable room for customization. Different applications have different requirements for model size, numerical precision, memory bandwidth, response time, power consumption and cost.

General-purpose GPUs offer flexibility across models and tasks, but that generality comes at an efficiency cost. Some inference workloads are relatively stable and clearly defined. In these cases, chips can be optimized for a specific model, type of computation or device environment, improving energy efficiency and reducing operating costs.

This does not mean general-purpose GPUs will be displaced. Instead, the AI chip market will develop a clearer division of labor. Large GPUs will continue to handle model training and some complex inference; cloud inference chips will serve workloads at scale; edge chips will run inference near operating sites; and on-device chips will increasingly support real-time sensing, prediction and decision assistance inside the device itself. GPUs, neural processing units (NPUs) and specialized accelerators will serve different workloads. New architectures such as compute-in-memory (CIM) may also gain ground in some inference applications.

We recently explored this trend with Houmo.AI. Its MOMAGIC®50 chip is designed for large-model inference at the edge and on devices. Built on a compute-in-memory architecture, it reduces the repeated movement of model parameters between memory and compute units to improve inference efficiency.

Such chips are unlikely to replace large GPUs entirely, but they could expand the boundaries of the AI semiconductor market. Competition will not be determined only by who has the most powerful training chip. It will also depend on who can strike the best balance among performance, cost, power consumption and ease of deployment across different inference scenarios.

Of course, new chips and compute architectures will create commercial value only if they perform in real operating environments, where they must be tested for performance, cost, reliability and deployment complexity. China could become an important test bed.

On the one hand, China offers a vast range of manufacturing, automotive, robotics, logistics, energy and urban infrastructure settings. These industries generate substantial demand for AI that can respond in real time, process data locally and control costs.

On the other hand, China has a complete industrial system extending from chips, servers, carrier networks and industrial parks to equipment manufacturers, automakers, robotics companies and a large base of industrial users. This allows new chips, models and solutions to enter real operating environments quickly, be tested in deployment and be refined through coordination across the value chain.

Policy is reinforcing the trend. In January 2026, eight Chinese government departments, including the Ministry of Industry and Information Technology, issued the Implementation Opinions on the Special Action for “AI + Manufacturing.” Alongside training chips, the document calls for the development of on-device inference chips and edge-computing servers, as well as a “cloud-edge-device” model system.

In such an environment, companies need to validate not just the performance of a particular chip, but whether chips, models, software, networks, data and business processes can work together as a complete solution.

A new technology can first enter a small number of applications, be refined through repeated use, and then scale to more settings through telecommunications operators, equipment manufacturers and industry partners. The result is a cycle of experimentation, refinement and scaling. China could become not only a test bed for new AI chips, but also an important proving ground for bringing new AI infrastructure and solutions to scale.

“China could become not only a test bed for new AI chips, but also an important proving ground for bringing new AI infrastructure and solutions to scale.”

The new compute landscape will reshape organizations

The changing architecture of compute will have profound implications for companies.

Many companies today comprise headquarters staff, regional teams and frontline employees. Headquarters set strategy and standards, allocate resources and build shared capabilities. Frontline employees, closer to customers, equipment and day-to-day operations, must understand local conditions and respond more quickly.

As AI enters more business scenarios, companies are no longer managing only back-end models and tools. Many agents today are built on general-purpose foundation models and connected to corporate knowledge bases, software tools and execution interfaces. Some agents are beginning to take on clearly defined responsibilities. Within set limits, they can access company knowledge and tools and continuously perform specific tasks. In this sense, they are increasingly becoming “digital employees.”

As companies deploy more digital employees, a new division of labor may emerge. It could begin to resemble the way human organizations work. Some will run on central platforms, drawing on common corporate knowledge, rules and capabilities. Others will operate in specific regional, factory or retail settings, using local data and systems to perform tasks closer to the frontline.

Managing digital employees will increasingly resemble managing people. Companies must decide what data they can access, what knowledge they can use, what permissions they hold and what tasks they may complete autonomously.

“The real challenge is not buying more chips. It is organizing compute, data, digital employees and human employees into an integrated system that can keep learning, collaborate effectively and operate at scale.”

From this perspective, what companies will ultimately manage is not merely AI, models or compute, but a new organization composed of human and digital employees.

This shift first requires companies to abandon the idea that one deployment model can solve every problem. Complex inference can continue to benefit from cloud scale; sensitive data and regulated operations can be handled in private deployments; tasks that must remain close to operations can move to the edge; and settings that require data to be processed directly on equipment can use on-device computing. The deployment methods differ, but the principle is the same: each AI application should be deployed in the environment best suited to the actual business need.

This approach also offers large enterprises a more practical path to AI transformation. They do not have to begin with a company-wide overhaul. They can start with one role, one team, one production line or one specific business problem and build a functioning closed loop on a limited scale.

BMW Group’s Factory Genius offers a concrete example. The AI assistant is designed to support maintenance of factory equipment. When production equipment fails, maintenance staff can ask it questions directly. The system searches equipment manuals, quality data, internal fault reports, planning documents and daily updated shift logs, then quickly provides recommendations for the specific problem, helping workers identify and resolve faults faster.

Factory Genius began as a pilot at BMW Group Plant Dingolfing. Plants in Spartanburg in the United States and Rosslyn in South Africa pursued similar efforts. The AI team at BMW’s Munich headquarters then consolidated needs and experience from different plants into a company-wide application and made it available to more facilities through an internal platform.

BMW’s approach shows how companies can begin with a specific business problem and application, validate what works, and then have headquarters integrate the experience, build shared capabilities and gradually roll them out to other business units.

Starting small, of course, does not mean allowing each unit to go its own way.

Enterprise AI infrastructure will probably not come from a single cloud platform, hardware system or supplier. Companies will use public clouds and private environments simultaneously and deploy edge nodes and devices that run AI locally in some operations. Their models and chips may also come from multiple suppliers.

But using multiple suppliers does not mean endlessly expanding the technology stack. Headquarters should establish a common foundation, standards and rules, while business units experiment with and apply AI according to actual needs. Companies must preserve strategic flexibility and system resilience without allowing different units to build isolated AI systems.

They should pursue “centralized governance, distributed applications and a controlled technology stack.” Companies should not allow technology to become fragmented first and rely on central governance to fix the problem later.

Like human employees, digital employees must be trained, evaluated and continuously updated. Companies need to define their knowledge sources, data and permission boundaries, the division of responsibilities between digital and human employees, and how actual performance will be used to correct behavior and turn frontline experience into shared organizational capability.

The deployment of AI infrastructure determines where digital employees run and how they access data and systems in different business settings. As digital employees enter those settings, they will in turn affect how information flows, where decisions are made and how work is divided between human and digital employees.

Technology architecture, organizational structure and business processes will gradually merge into a single system. As digital employees accumulate capabilities, their company-specific knowledge, memory and ways of collaborating may become a new organizational asset.

In this sense, AI is not only changing how companies operate. It is also reshaping the underlying logic of corporate value creation.

AI competition is entering a new stage

AI competition will not be decided only by who owns the largest training cluster.

Large data centers will continue to push the frontiers of model training and complex inference. At the same time, different inference tasks will run in the cloud, at the edge or on devices, depending on data, business scenarios and real-time requirements. Companies that require greater data security and control will run some AI in private environments. Different types of chips will develop a clearer division of labor accordingly.

The deeper change is taking place inside the enterprise. As digital employees enter more business scenarios, companies must rethink the division of labor between them and human employees.

The real challenge is not buying more chips. It is organizing compute, data, digital employees and human employees into an integrated system that can keep learning, collaborate effectively and operate at scale.

Companies need more than AI technology. They need a new systems mindset. What we are witnessing is not only a new landscape for AI compute, but the beginning of a new era in organizational design.

This article was contributed by Gao Feng Advisory Company.