Embedded AI: where should the intelligence sit — on the sensor, the gateway or the cloud?
2026's microcontrollers and NPUs make local inference realistic, even on small devices. The criteria for deciding what runs where.
For years, "smart IoT" meant: the sensor measures, the cloud thinks. In 2026, hardware is challenging that split. Ultra-low-power microcontrollers now include accelerators able to run compact learning models ("TinyML") directly on the device, and NPUs of a few watts process video streams at the edge (Promwad, Texas Instruments).
Three levels, three roles
- On the sensor or microcontroller: simple anomaly detection, signal classification (vibration, current, sound), conditional wake-up. Constraints: very limited memory and energy, compact quantized models.
- On the gateway or edge computer: aggregating several sensors, camera vision, heavier models. Industry analyses cite NPUs of a few TOPS for a few watts, enough for real-time image classification.
- In the cloud or a central server: training, retraining, fleet analysis, dashboards, large models.
Five criteria for deciding
Latency. A response that must land within a few tens of milliseconds — stopping a machine, rejecting a part — can't depend on a network round trip.
Connectivity. An isolated site, a basement, a low-bandwidth link (LoRa, sub-GHz) calls for deciding locally and reporting only the essentials.
Energy. On battery, transmitting costs more than computing: filtering at the source extends autonomy.
Privacy. Keeping raw images or sound on the device and sending only a reduced result lowers exposure of personal data.
Fleet cost. Sending raw data from thousands of devices costs connectivity and storage; local inference shifts the cost to hardware, paid once.
Pitfalls
- Model lifecycle. A deployed model drifts: machines age, conditions change. Plan remote updates (signed OTA updates) and drift measurement.
- Training data. A good embedded model is built on data representative of the field, not the lab.
- Security. A device carrying a model and its data becomes a target. Encryption, secure key storage and firmware hardening become prerequisites (see our article on the CRA and the RED directive).
- Oversizing. Many use cases need only a statistical threshold or a small classifier. Choose the simplest solution that holds the required accuracy.
Our view
AI shouldn't be added because it's available, but where it makes the system more useful: detecting earlier, deciding locally, reducing traffic. The right starting point is the business question — what must be detected, how fast, under what operating conditions — not the choice of component.
Sources
Frequently asked questions
What is embedded AI?
Embedded AI means running a machine-learning model directly on the device that measures — sensor, microcontroller, gateway — instead of sending raw data to a remote server for analysis. The device then transmits only a result: an anomaly, a count, a state.
What is the difference between embedded AI, edge AI and TinyML?
All three describe the same direction at different scales. TinyML means very compact models running on a microcontroller at a few milliwatts. Edge AI covers any processing at the edge more broadly, including on a gateway with a few-watt NPU able to handle a video stream.
Should the intelligence sit on the sensor, the gateway or the cloud?
The sensor suits simple, repetitive, very-low-power detection. The gateway suits anything that correlates several sources, processes images, or must decide within tens of milliseconds. The cloud remains the right place for training, long history and multi-site views.
What are the concrete benefits of local processing?
Three: latency, since the decision does not wait for a network round trip; cost, since raw streams are neither transported nor stored; and privacy, since sensitive data — an image, a sound, a health reading — need never leave the site.
What are the limits of embedded AI?
Available memory and compute bound the model size, and updating a model deployed across a fleet of devices is an operational problem in its own right. An embedded model must be designed with its life cycle: versioning, staged rollout, and a way to roll back.
Have an IoT project in mind?
Describe your context, we'll propose a tailored solution.