NPU (Neural Processing Unit): what it is, how it works and what it does

Definition

An NPU (Neural Processing Unit) is a hardware block engineered to efficiently execute the mathematical operations inherent to neural networks, particularly model inference (prediction). Unlike a CPU or GPU, which are general-purpose processors, the NPU optimises its architecture for the recurring computation patterns found in advanced processing, such as matrix multiplications and convolution operations.

Operating principles

The NPU reduces the energy cost and response time of advanced processing tasks by:

• Organising its compute units around matrix operations (MAC, multiply-accumulate), which are the core of dense and convolutional layers.
• Including local working memory to minimise accesses to main memory during inference.
• Supporting low-precision data formats (e.g. INT8 or FP16) that reduce bandwidth and power consumption without significantly degrading model accuracy.

Architecture and system placement

In a modern SoC (system-on-chip), the NPU coexists with the CPU, GPU, and in some cases a DSP. The operating system or advanced processing framework routes workloads to the most suitable accelerator. The NPU does not replace the other processors; it complements their function for tasks where its compute topology is more efficient.

Typical applications

• Real-time image recognition and enhancement (smartphone cameras).
• On-device voice assistants and natural-language processing.
• Video segmentation, object detection, and image stabilisation.
• Inference of lightweight advanced processing models directly on the terminal, without cloud dependency.

Concrete example

A documented example is the MediaTek NPU 655, a neural accelerator integrated into MediaTek's mobile platform. Its presence in a product's technical specifications indicates that the manufacturer has equipped the chip with a dedicated block for advanced processing inference, differentiating it from solutions that delegate all advanced processing tasks to the GPU or to remote servers.

How to interpret its presence in a product

When encountering an NPU reference in a device's specifications, it is worth considering:

1. Local advanced processing integration: the device can run advanced processing models offline, improving privacy and reducing latency.
2. Energy consumption: for sustained advanced processing workloads, the NPU is typically more efficient than the GPU, extending battery life.
3. Limited capacity: the NPU is optimised for moderately sized models; training tasks or very large models still require server infrastructure.

Scope and limitations

The NPU is not a universal processor: its performance depends on the size and architecture of the model running on it. It does not replace the CPU for control logic or the GPU for graphics rendering. Furthermore, compatibility with specific advanced processing frameworks (TensorFlow Lite, ONNX Runtime, etc.) varies by SoC manufacturer and the software support provided.

The NPU represents hardware specialisation for the era of edge advanced processing: it enables tasks that previously required the cloud to be resolved locally, with lower latency and lower power consumption.

Aliases and alternative names

• Neural Processing Unit
• Neural accelerator
• advanced processing Engine / advanced processing Processor (depending on manufacturer)
• Neural Engine (terminology used by some manufacturers)