Edge AI means running an artificial intelligence model directly on the device that produces the data (a camera, sensor, robot, phone or machine) instead of sending that data to a cloud server. You gain latency, offline operation and privacy. You give up computing power and ease of updates. It also has nothing to do with the Microsoft Edge browser, even though Google mixes the two up in its search suggestions.
Here's the order: definition, how it works, hardware for a first prototype, then a five-criteria framework for deciding between cloud and device, product by product.
Edge AI: definition and scope
The model runs where the data is created, and the answer comes back in milliseconds, with or without a network.
Inference leaves the cloud
Red Hat defines edge AI as the use of AI combined with edge computing to process data where it is collected, or close to it. The classic example is image recognition running on the camera itself, with no round trip to a data center. IBM adds that companies adopt it to cut latency, strengthen security and lower costs.
Training is what stays in the cloud. As Red Hat puts it, edge AI "could not exist without cloud computing," because edge devices have neither the compute nor the data volume to train a deep model. The cloud trains, compresses, deploys and monitors; the device runs the model.
Edge AI, edge computing, embedded AI: three terms, one principle
Edge computing refers to computation moved close to the source, whether or not AI is involved. Edge AI is the specific case where that computation is an AI model. Embedded AI emphasizes the hardware: the model lives inside an object, often a microcontroller or a dedicated board. Microsoft Edge is a browser, and "Microsoft Edge AI" refers to the Copilot built into it, which is a different thing entirely.
How edge AI works
You train at the center, run at the edge, and compress in between.
Train at the center, run at the edge
The cycle Red Hat describes has four stages. Training happens in the cloud, on data sent back from devices. Deployment pushes an optimized version of the model to resource-constrained devices. Synchronization means the device processes data locally and sends back what's useful for retraining. Monitoring lets the cloud platform track the fleet, detect drift and plan maintenance.
An article in The Conversation gives the order of magnitude that justifies all this: sending large volumes of IoT data to the cloud adds "a few hundred milliseconds to several seconds" of delay depending on the network and the volume. For a smart meter, that's acceptable. For a robot steering around a person, it isn't.
Compressing a model: quantization, distillation, pruning
Three techniques shrink a model that's too heavy for a board running on a few watts. Quantization replaces floating-point numbers with shorter integers. Pruning removes connections that contribute nothing. Distillation trains a small model to imitate a large one. Red Hat lists the first two among the tools cloud services provide to prepare a model for the edge. Runtimes such as Google's LiteRT (formerly TensorFlow Lite) or ONNX Runtime then execute the compressed model on the board's CPU, GPU or accelerator.
Benefits and limits of edge AI
Five gains documented by vendors, and three costs they rarely mention.
Latency, offline operation, privacy, cost at scale
Red Hat names five benefits: lower energy consumption, lower bandwidth, privacy (sensitive data never leaves the device), security (only what's essential gets transmitted) and scalability. The Conversation stresses privacy with a telling example: in a connected home, data from a meter or the lighting reveals when people are in and out, and processing it locally avoids exposing it.
Cloud inference costs you on every request, forever. On-device inference costs you once, built into the price of the board. For a handful of devices, the cloud stays cheaper. For a fleet, multiply your cost per inference by your daily volume, then by the number of years in operation, and compare the result to the price of a board.
Limited compute, updates, physical security
Compute: a 25-watt board can't run the same model as a server, and Lenovo's own glossary lists the "limitations of Edge AI Computing" alongside its strengths. Updates: a model fixed in the cloud is fixed for everyone, but across ten thousand devices you need a deployment mechanism, versioning and rollbacks. Physical security: the device sits at the customer's site, exposed to theft, teardown and model extraction.
Real-world applications
Four fields where milliseconds and sensitive data make the call.
Industrial vision and robotics
Quality control on a production line, defect detection, meter reading, mobile robot navigation: in all these tasks, the image can't wait for a server. Ultralytics documents how to deploy vision models on NVIDIA Jetson, the most widely used board for this use case. In a robot, edge AI is the perception and reflex layer, while heavy planning can stay elsewhere.
Healthcare, mobility, retail
IBM cites smartphones, traffic updates in autonomous vehicles and connected devices. In healthcare, edge AI processes the image or signal on the device itself to protect patient privacy and shorten response time. In retail, in-store video analytics runs locally for the same reason.
LLMs on the device?
Yes, in scaled-down form. LiteRT supports on-device generation. A compact language model on a phone or a Jetson board can handle simple commands offline. For long-form reasoning, the cloud is still the default.
Hardware and tools to get started
Three tiers of boards, three tools, and a prototype budget.
Jetson, Coral, Raspberry Pi: three tiers
| Board | Manufacturer | Best for |
|---|---|---|
| Jetson Orin Nano Super Developer Kit | NVIDIA | vision, robotics, small generative models; onboard GPU, 67 TOPS claimed; $249, price announced by NVIDIA on December 17, 2024, down from $499 |
| Coral Dev Board | running compressed models on the Edge TPU accelerator, low power consumption | |
| Raspberry Pi | Raspberry Pi | prototypes, sensors, lightweight models on CPU or an accelerator module |
Since the December 2024 price cut, the Jetson Orin Nano has become the default entry point for vision and robotics. NVIDIA announced it on its developer blog along with a 1.7x boost in generative inference performance. Coral and Raspberry Pi remain relevant for smaller models and tighter power budgets.
Edge Impulse, LiteRT, ONNX Runtime
Edge Impulse is an end-to-end platform for collecting sensor data, training, compressing and deploying across the board ecosystem; Qualcomm announced its acquisition in March 2025. LiteRT is Google's runtime, the successor to TensorFlow Lite, which Google says runs on billions of devices. ONNX Runtime is the open runtime that executes a model exported to ONNX format from PyTorch or another framework. Pick a board first, then the runtime its documentation recommends.
What a first prototype costs
A $249 Jetson board, a camera, a pre-trained vision model (Ultralytics provides some) and one engineer-week to build the pipeline from data collection through training, export and deployment. At that point you can measure your first results: frames per second and error rate on your own use case.
When a founder should choose edge AI
Five criteria and three verdicts, applied product by product.
Five decision criteria
- Latency. Does the response need to arrive in under 100 milliseconds? A robot, a vehicle, a machine tool: yes. A dashboard: no.
- Connectivity. Will the device sometimes be offline? A field, a basement, an industrial site, a medical practice: often.
- Privacy. Can the raw data leave the site? Patient images, the inside of a home, a customer's production line: often not.
- Unit cost at scale. How many devices, how many inferences per day, over how many years? Multiply them and compare the total to the price of a board.
- Model updates. How often does your model change? Every week: the cloud is simpler. Every quarter: the edge holds up.
Required, nice to have, unnecessary: three verdicts per product
- Required when criterion 1 or 2 applies. Mobile robots, bedside medical devices, agricultural machinery, standalone security cameras.
- Nice to have when criterion 3 or 4 dominates. In-store video analytics, high-volume industrial sensors, mobile apps that process the user's images.
- Unnecessary when none of the five applies. A B2B SaaS that processes documents, an internal tool, a text assistant: the cloud wins, and the edge only adds complexity.
When the verdict is "nice to have," the lowest-risk move is to start in the cloud with an architecture that lets you migrate inference later: export the model to ONNX or LiteRT format from day one, have the device send its data back, and track your cost per inference. The rest of the Tech hub covers the other layers, and if you're building a hardware product, you can apply to swanbase.
The embedded compute layer of Physical AI
In swanbase's pillar article on Physical AI, the stack of a system that acts in the physical world starts with "sensors and embedded compute." Edge AI is that layer: you run perception and reflexes on the machine, while world models and heavy planning can live elsewhere. French humanoid robots illustrate this well: Calvin perceives and balances on board, in milliseconds.
Edge AI FAQ
What's the difference between edge AI and edge computing?
Edge computing brings computation closer to the data source, whatever that computation is. Edge AI is the case where that computation is inference from an AI model. All edge AI is edge computing, but the reverse isn't true.
Does edge AI replace the cloud?
No. The cloud trains the models, compresses them, deploys them and monitors the devices. The edge runs them. As Red Hat sums it up, edge AI could not exist without the cloud.
Do you need a GPU for edge AI?
Not always. A microcontroller can run an anomaly detection model on a sensor; an accelerator like the Coral Edge TPU is enough for image classification; an onboard GPU like the Jetson becomes useful for high-resolution vision, robotics and small generative models.
