AI Chips Explained: GPU, TPU, NPU, and Custom Accelerators

An AI chip is a processor designed or adapted to perform the matrix and tensor calculations used by machine-learning models more efficiently than a general-purpose processor can.

The term does not describe one single product. It covers several processor families with different priorities:

• • GPUs emphasize parallel performance and flexibility.
  • TPUs and other custom accelerators optimize specific AI workloads at cloud scale.
  • NPUs prioritize efficient AI inference on phones, laptops, and other edge devices.
  • CPUs coordinate the system and still handle tasks that do not benefit from massive parallelism.
  • Unconventional designs, including language-processing and wafer-scale processors, target particular speed, memory, or scaling problems.

The right chip depends on the workload, model size, memory requirements, software ecosystem, latency target, power budget, and total cost. There is no accelerator that is best at everything.

This article is for general educational purposes. Product availability, specifications, and software support can change.

Why It Matters

AI models are largely software, but their practical limits are set by hardware. Chips determine how long a model takes to train, how quickly it responds, how much electricity it uses, and whether it can run locally or must use a data center.

The distinctions also explain several major technology trends. NVIDIA’s GPU ecosystem remains central to large-scale AI. Google, Amazon, Microsoft, and Meta are developing custom silicon to improve control and economics. Meanwhile, NPUs are moving more inference onto consumer devices.

For buyers and developers, the processor name is only the starting point. Memory capacity, interconnects, compilers, supported frameworks, and access to enough hardware can matter as much as headline compute performance.

AI Chip Types at a Glance

These categories overlap. A modern system may use CPUs, GPUs, and dedicated AI engines together, and a single chip package can contain several types of compute blocks.

What Makes an AI Chip Different?

Neural networks repeatedly perform large numbers of multiply-and-accumulate operations. These calculations can often run in parallel, making them a poor match for a CPU working through a smaller number of complex instruction streams.

AI accelerators improve efficiency by combining several ideas:

• • Many parallel compute units
  • Specialized matrix or tensor engines
  • Support for lower-precision number formats
  • High-bandwidth memory close to the processor
  • Fast links between multiple accelerators
  • Software that maps model operations onto the hardware

Specialization creates speed and efficiency, but it also creates trade-offs. A chip designed narrowly for inference may be excellent at serving a model and less suitable for training or general scientific computing.

Where the CPU Still Fits

The CPU has not disappeared from AI systems. It runs the operating system, prepares data, schedules work, manages storage and networking, and handles instructions that do not map efficiently onto an accelerator.

A balanced system matters. An expensive accelerator can sit underused if the CPU, storage system, memory pipeline, or network cannot deliver data quickly enough. In large clusters, the supporting infrastructure is part of AI performance—not an afterthought.

GPU: The Flexible Default for Large-Scale AI

A graphics processing unit was originally developed for graphics, where many similar calculations must be performed at the same time. That parallel structure also works well for neural networks.

GPUs became the default AI accelerator for three main reasons:

• • Flexibility: They can support training, inference, simulation, rendering, and many scientific workloads.
  • Scalability: Multiple GPUs can be connected within servers and across large clusters.
  • Software maturity: NVIDIA’s CUDA platform, AMD’s ROCm ecosystem, and broad framework support give developers established tools, libraries, and optimized kernels.

NVIDIA’s advantage is therefore not only the chip. CUDA and its surrounding libraries have accumulated years of developer adoption and optimization. Moving a production workload to another accelerator can require new compilers, rewritten kernels, different deployment tools, and extensive performance testing.

The main GPU trade-offs are cost, power consumption, cooling requirements, and limited supply for the most sought-after data-center products.

TPU: Google’s Purpose-Built Tensor Processor

A tensor processing unit is Google’s custom AI accelerator. TPUs are application-specific integrated circuits designed around the tensor operations used in machine learning.

Google offers TPUs through Google Cloud rather than as general-purpose cards for customers to install in their own servers. The current available generation, TPU7x or Ironwood, is designed for large-scale training and inference and can be connected in pods containing thousands of chips. Google has also announced eighth-generation TPU products for training and inference, but lists them as coming soon as of this update.

TPUs work best when a team is comfortable with Google Cloud and the TPU software stack. JAX, TensorFlow, and PyTorch/XLA can target TPUs, but code that depends heavily on CUDA-specific libraries or custom GPU kernels may require meaningful adaptation.

The TPU advantage is tight co-design across chips, networking, systems, compilers, and Google’s cloud infrastructure. The trade-off is less portability outside that environment.

NPU: Efficient AI on Phones and PCs

A neural processing unit is a low-power accelerator intended mainly for inference on a local device. It can handle supported AI tasks without continuously sending data to a cloud server.

Common uses include:

• • Camera and photo enhancement
  • Speech recognition and transcription
  • Background effects and noise reduction
  • Face and object detection
  • Small language or vision models
  • Accessibility and personalization features

Examples include Apple’s Neural Engine, Qualcomm’s Hexagon NPU, Intel’s NPU in Core Ultra processors, and AMD’s XDNA-based Ryzen AI engines.

The benefit is efficiency. An NPU can run sustained AI features with less battery impact than routing every task through a CPU or high-performance GPU. Local processing can also reduce latency and keep some data on the device.

An NPU is not simply a smaller data-center GPU. It usually supports a narrower set of operations, number formats, and model sizes. Actual performance depends on whether an application has been optimized for that specific NPU and its runtime.

Custom Accelerators: Why Cloud Companies Build Their Own

At hyperscale, even a small improvement in performance per dollar or performance per watt can materially change data-center costs. That is why major cloud and internet companies design accelerators around their most common workloads.

“Custom” does not always mean inaccessible. Google TPUs and AWS Trainium are available as cloud services, while Microsoft and Meta primarily use their accelerators inside their own infrastructure.

These companies still deploy third-party GPUs. Custom chips provide another option for selected workloads; they do not automatically replace general-purpose accelerators across an entire fleet.

Two Unconventional Designs

Groq’s Language Processing Unit

Groq’s LPU architecture is designed for predictable, low-latency inference. It relies heavily on fast on-chip SRAM and compiler-controlled, deterministic scheduling. Instead of leaving many runtime decisions to the hardware, the compiler plans where operations and data movements occur.

That approach can deliver very fast token generation for supported models. Its trade-offs include a more specialized software path and the need to distribute large models across the on-chip memory of multiple processors.

Cerebras’ Wafer-Scale Engine

Cerebras takes a different approach: it builds a processor that occupies almost an entire silicon wafer. The company reports that its WSE-3 contains 4 trillion transistors, 900,000 AI-optimized cores, and 44GB of on-chip SRAM, with 125 petaflops of peak AI compute.

The design reduces the distance that data must travel between many separate chips. It is engineered for large AI training and inference systems, not as a drop-in replacement for a desktop or conventional server GPU.

Both architectures show that “AI chip” is a design category, not one fixed blueprint.

Training and Inference Need Different Things

Training creates or updates a model. It processes large datasets, calculates gradients, and changes model weights over many iterations. Training usually rewards high throughput, large memory capacity, fast communication between accelerators, and mature distributed-computing software.

Inference uses an already trained model to produce an answer, classification, image, or prediction. It may prioritize:

• • Low response time for interactive applications
  • High throughput for large numbers of requests
  • Low power for on-device use
  • Low cost per generated token or prediction
  • Predictable performance under changing demand

The distinction is not absolute. GPUs and TPUs can perform both jobs, and custom accelerators increasingly target both. But a benchmark for training does not automatically predict inference performance, and vice versa.

Memory Often Matters More Than Headline Compute

A processor cannot run a model efficiently if the model and its working data do not fit within the available memory.

For model weights alone, the theoretical storage requirement is approximately:

• • 4 bytes per parameter at 32-bit precision
  • 2 bytes per parameter at 16-bit precision
  • 1 byte per parameter at 8-bit precision
  • 0.5 byte per parameter at 4-bit precision

That means a 7-billion-parameter model needs about 14GB for 16-bit weights or 3.5GB for 4-bit weights. A 70-billion-parameter model needs about 140GB or 35GB, respectively.

Those figures are minimums, not complete system requirements. Inference also needs memory for the key-value cache, runtime overhead, temporary calculations, and longer contexts. Training requires much more because it stores activations, gradients, optimizer states, and other intermediate data.

Memory bandwidth matters too. A chip can have enormous theoretical compute capacity and still wait for data if memory cannot feed its compute units quickly enough.

The Software Stack Can Decide the Winner

Hardware performance is only useful when software can reach it. Each platform has its own compilers, runtimes, libraries, monitoring tools, and deployment patterns.

Examples include:

• • NVIDIA GPUs: CUDA and CUDA-X libraries
  • AMD GPUs: ROCm
  • Google TPUs: JAX, TensorFlow, and PyTorch/XLA
  • AWS Trainium: the Neuron SDK
  • Apple devices: Core ML and related Apple frameworks
  • Windows and Android NPUs: ONNX Runtime and vendor-specific execution providers or SDKs

Open frameworks can reduce the amount of application code that changes, but they do not eliminate platform-specific optimization. Custom kernels, memory layouts, distributed training strategies, and operational tooling may still need to be redesigned.

This is why the lowest benchmark price is not always the lowest migration cost.

How to Choose the Right AI Hardware

Start with the workload instead of the brand.

• • Define the job. Is the system training a model, serving one, or running a small model locally?
  • Measure the model. Check parameter count, precision, context length, batch size, and memory overhead.
  • Set the performance target. Decide whether latency, throughput, time to train, or power efficiency matters most.
  • Check software compatibility. Identify required frameworks, libraries, custom kernels, and deployment tools.
  • Evaluate scale and availability. A fast chip is not useful if the required quantity or cloud capacity is unavailable.
  • Compare total cost. Include memory, networking, power, cooling, engineering work, and operational support—not only the accelerator price.

For most organizations, benchmarking the actual model on two or more realistic configurations is more useful than comparing vendor peak-performance claims.

Common Misconceptions About AI Chips

“GPUs are the only serious AI processors.”

GPUs remain the most flexible and broadly supported accelerators, but TPUs, Trainium, MTIA, Maia, NPUs, LPUs, and wafer-scale systems are real production technologies built for different workloads.

“A custom chip is automatically faster than a GPU.”

Only for the workload and software path it was designed to accelerate. General-purpose GPUs may still win when flexibility, developer tools, or rapidly changing models matter more.

“TOPS or FLOPS tells you which chip is best.”

Peak operations per second measure only one part of performance. Precision, memory capacity, bandwidth, interconnects, sparsity assumptions, utilization, software quality, and workload shape can change the real result.

“An NPU makes every application faster.”

Only applications that can send supported model operations to the NPU benefit. Other work may continue running on the CPU or GPU.

“Quantization reduces model quality by a fixed amount.”

The effect depends on the model, task, quantization method, and calibration. Lower precision can reduce memory use and increase speed, but the quality trade-off must be tested on the intended workload.

What Happens Next

AI hardware is becoming more specialized at both ends of the market. Cloud providers are expanding custom accelerators for large-scale training and inference, while device makers are increasing local AI capacity in phones and PCs.

The likely result is not one winner. It is a more heterogeneous stack in which CPUs, GPUs, NPUs, and custom accelerators work together. Software portability, memory systems, networking, energy efficiency, and supply will shape adoption as much as the processor architecture itself.

Morning Glance’s AI & Automation coverage follows the companies, infrastructure investments, and policy decisions shaping that market.

The Bottom Line

GPUs are the flexible default for demanding AI workloads. TPUs and other custom accelerators trade some portability for efficiency at scale. NPUs bring supported inference onto power-constrained devices. CPUs coordinate the system, while specialized designs such as Groq’s LPU and Cerebras’ wafer-scale processor attack narrower bottlenecks.

Do not choose an AI chip from a single benchmark or acronym. Match the hardware to the model, memory requirement, software stack, latency target, power budget, availability, and total cost.

Frequently Asked Questions

What is the difference between a GPU and a TPU?

A GPU is a broadly programmable parallel processor used for AI, graphics, simulation, and other workloads. A TPU is Google’s purpose-built accelerator for tensor-heavy machine learning and is offered through Google Cloud.

Is an NPU better than a GPU?

Not generally. An NPU is usually better suited to efficient, supported inference on a phone or PC. A GPU is more flexible and can handle larger or more varied workloads, but typically uses more power.

Do AI models still need CPUs?

Yes. CPUs manage the operating system, data preparation, storage, networking, scheduling, and tasks that do not map efficiently to an accelerator.

What is the difference between training and inference?

Training adjusts a model’s weights using data. Inference uses the trained model to generate a prediction or response. Training usually needs more memory and sustained compute, while inference may emphasize latency, throughput, or power efficiency.

How much memory does a large language model need?

For weights alone, a rough minimum is 2 bytes per parameter at 16-bit precision or 0.5 byte at 4-bit precision. Runtime overhead, context length, batch size, and software increase the actual requirement.

Why do cloud companies build custom AI chips?

They can optimize hardware and software around high-volume internal workloads, improve performance per dollar or watt, reduce dependence on one supplier, and offer differentiated cloud services.

Can software move easily between AI chips?

Sometimes at a basic level, especially through common frameworks. Production performance often requires platform-specific compilers, kernels, memory strategies, and deployment tools, so migration can still be substantial.

Official Sources

• • NVIDIA: CUDA platform
  • Google Cloud: TPU overview and Ironwood documentation
  • AWS: Trainium and Project Rainier
  • Microsoft: Maia 200
  • Meta: MTIA accelerator program
  • Apple Machine Learning Research: Neural Engine
  • Groq: LPU architecture
  • Cerebras: WSE-3 specifications

Source note: Product details were checked against official company documentation available on Aug 25, 2026. Vendor performance claims are identified as company-reported where relevant.

Get the Morning Glance briefing — clear AI infrastructure context without the noise.

Report an error