Loading…
Loading…
Edge AI runs model inference near the data instead of in the cloud. The four reasons it is justified, the costs vendors omit, and a per-workload decision framework.
Edge AI is running inference near the data source. Training almost always stays central.
"Edge" spans on-device, on-premise server and regional. Ask which one a vendor means, because the cost, latency and compliance profiles differ materially.
Four justifications: latency, bandwidth, data residency, offline operation. A workload needs at least one strongly, or two moderately.
The costs vendors omit are model compression trade-offs, fleet management, hardware heterogeneity, and the physical environment.
Decide per workload, not per company. Hybrid, with edge inference and central training, is the answer for most enterprises.
If a proposal has no line for fleet management or evaluation, it is a pilot plan rather than a deployment plan.
Edge AI is the execution of machine learning inference on hardware physically close to where data originates, rather than in a centralised cloud or datacentre.
The important word is inference. Almost all edge AI deployments still train their models centrally, where the data, the compute and the tooling are. What moves to the edge is the trained model, which then runs against live local data. Training at the edge exists, in the form of federated learning and on-device personalisation, but it is a specialised case.
The edge is not one place. On-device means inference on the sensor or endpoint itself, with the lowest latency and the tightest compute budget. On-premise edge server means a machine in the same building as the devices, with more compute and still no external network dependency. Near edge or regional means a telco point of presence, which helps latency but does not solve data residency, because the data still leaves the building.
When a vendor says edge, ask which of the three they mean. The word alone tells you nothing about cost, latency or compliance.
Latency. Round-tripping to a cloud region adds network time to every inference. The distinction that matters is between latency-sensitive and latency-critical. A dashboard refreshing two seconds late can live in the cloud. A safety interlock, a robot arm, or an operator watching a live feed to intervene cannot. Be sceptical of latency arguments that do not quote a budget in milliseconds, because "real time" is not a number.
Bandwidth and its cost. Video is the clearest case. A few hundred camera streams produce a bandwidth bill that dominates the project. Edge inference inverts the flow: instead of shipping video to the model, you ship the model to the video, and only results travel. An alert with a thumbnail is orders of magnitude smaller than the footage it came from. This is why video analytics is the most common production edge AI workload and usually the first to pay for itself.
Data residency and privacy. Some data cannot leave a building or a jurisdiction, and no architectural preference overrides that. Under India’s Digital Personal Data Protection Act and comparable regimes elsewhere, organisations carry obligations about how and where personal data is processed. Edge inference means the raw data never crosses the boundary and only derived results do, which is often a cleaner position than protecting a transfer.
Operating without a network. For a recommendation engine, connectivity loss is tolerable. For perimeter security at a remote substation, a mine site or a vessel, it is not. Edge inference continues when the link drops and reconciles when it returns. This is the reason most often discovered late, usually after the first outage.
Constrained compute, and therefore constrained models. Edge hardware has a fraction of the memory and compute of a datacentre GPU, so models must be smaller and some accuracy is traded away. Quantisation reduces numerical precision, typically from 32-bit float to 8-bit integer, cutting memory and increasing speed on hardware with integer acceleration. Pruning removes weights that contribute little. Distillation trains a smaller model to imitate a larger one, giving the best quality per parameter for the most work. Each needs measurement on your data, because a published benchmark on a public dataset tells you a technique works in general, not what it does to your model.
Deployment and fleet management. Updating a cloud model means deploying to one place. Updating a fleet means a rollout across hardware you may not physically control, in sites with variable connectivity, where a failed update can take a device offline. This is the cost most consistently underestimated. A serious deployment needs versioning, staged rollout, rollback, health monitoring and remote diagnostics before it reaches scale. The inference is the easy part; the fleet is the engineering.
Hardware heterogeneity. Cloud inference targets a small set of well-understood instance types. Edge inference targets whatever is on site, which after a few years is rarely uniform. Runtimes such as ONNX Runtime abstract much of this, letting one exported model run across different accelerators, but they do not remove the need to validate performance per target, because an operator that is fast on one accelerator may fall back to CPU on another.
Physical reality. Factory floors are hot and dusty, retail back rooms have unreliable power, roadside cabinets face weather and tampering. Thermal throttling, failed storage and physical access control become part of your AI system’s operating profile.
Apply this per workload, not per company.
Choose edge when the decision window is measured in milliseconds, when raw data volume makes central transport expensive, when the data cannot leave the premises, or when the workload must survive a network outage.
Choose cloud when the model is large or changes frequently, when the workload is bursty and benefits from elastic capacity, when you need the newest accelerators without a hardware refresh, or when the data is already centralised.
Choose hybrid, which is the common answer, when inference belongs at the edge for latency or bandwidth reasons but training, model management and long-term analytics belong centrally. The pattern that holds up in production is straightforward: train centrally, deploy to the edge, run inference locally, send results and selected samples back, retrain on the aggregate, redeploy. That feedback loop is what stops edge models decaying quietly, and it is the part most commonly left out of a first implementation.
Video analytics and surveillance is the clearest case, because bandwidth, latency and privacy all point the same way. Inference runs on site and only events leave. This is the workload our own video platform, Visylix, is built around: it deploys on customer infrastructure including fully air-gapped sites, with analytics running locally rather than calling out to a cloud service.
Industrial quality inspection puts cameras on a line and classifies defects within the cycle time of the line itself. It is latency-critical, and the data volume is impractical to move.
Predictive maintenance runs vibration and acoustic models on machinery, often on very small hardware. TinyML techniques make this viable on microcontrollers with power budgets measured in milliwatts.
Retail operations covers footfall, queue length and shelf monitoring, where transporting continuous video from every store is the cost that kills the project. Autonomous and semi-autonomous systems are not a preference at all: a vehicle cannot wait for a cloud round trip to decide whether to brake.
Large language models at the edge. Small models run on edge hardware today and are genuinely useful for constrained tasks. Frontier-scale models do not, and quantised small models are not equivalent to them. If a vendor implies otherwise, ask which model, at what quantisation, on what hardware, at what tokens per second.
"Edge AI reduces cost" as a blanket claim. It moves cost. You trade cloud compute and egress for hardware, deployment engineering and field maintenance. For high-volume continuous workloads such as video it usually wins. For intermittent low-volume workloads cloud is often cheaper, and buying edge hardware for a model that runs a few hundred times a day is an expensive way to be architecturally fashionable.
Edge as a substitute for data governance. Keeping data on site removes a transfer risk. It does not remove obligations around access, retention, purpose limitation or audit.
Federated learning as a default. It is a real technique with real deployments, but it adds substantial complexity and is justified by specific constraints rather than adopted as a general preference.
Consistent with enterprise AI generally, the model is rarely the expensive part.
Cost concentrates in three places. Data and evaluation, because you need representative data from the actual deployment environment, and edge conditions differ from curated datasets in ways that matter. Integration and fleet engineering, which is the deployment, monitoring and rollback machinery described above. Ongoing operation, because models drift, hardware fails and sites change.
A useful diagnostic when reading a proposal: if it has a large line for model development and almost nothing for fleet management and evaluation, it is describing a pilot rather than a deployment. Pilots on edge AI succeed at a much higher rate than production rollouts, and the gap between them is exactly those two lines.
We start with the workload rather than the architecture. The first question is what decision changes as a result of the inference, and the second is which of the four justifications applies. If none applies strongly, cloud is usually the cheaper and simpler answer, and we will say so.
Where edge is the right call, we size the model and the throughput requirement before selecting hardware, because choosing an accelerator first is how organisations end up with equipment that cannot run the model they eventually need. Accuracy after compression is measured on your own data, not inferred from a public benchmark.
We build the fleet machinery as part of the project rather than after it: versioned models, staged rollout, health monitoring and rollback. That is the difference between a pilot that impresses and a deployment that survives its second year.
Running an AI model on or near the device that collects the data, instead of sending that data to a distant cloud server and waiting for a reply. The model comes to the data rather than the data going to the model.
Location of inference, and the consequences that follow. Edge AI runs locally, giving lower latency, less bandwidth use, and the ability to keep data on site and work offline, at the cost of constrained hardware and fleet management. Cloud AI runs centrally with far more compute, easier updates and elastic scale, at the cost of a network round trip and data leaving the premises.
It depends on volume and continuity. For continuous high-volume data such as video, edge is usually cheaper because you are not paying to transport and store everything centrally. For intermittent or low-volume workloads, cloud is typically cheaper because you avoid buying and maintaining hardware.
Not always. Plenty of useful models run on ordinary CPUs at modest throughput. Dedicated accelerators, NPUs or edge GPUs become necessary as model size and frame rate increase. Establish the workload requirement before selecting hardware, not the other way round.
No. It helps by removing a data transfer, which is often the hardest part to justify, but access control, retention limits, lawful basis, audit logging and breach procedures remain your responsibility regardless of where inference runs.
Small and quantised models can, and are useful for constrained tasks such as classification, extraction and short structured responses. Frontier-scale models cannot run on typical edge hardware. Treat any claim otherwise as a prompt to ask which model, at what quantisation, on what hardware, and at what throughput.
Machine learning on microcontroller-class hardware, with power budgets in the milliwatt range. It suits always-on sensing tasks such as vibration monitoring, keyword spotting and anomaly detection on machinery, where the alternative is no intelligence at all.
Through a managed rollout: versioned models, staged deployment to a subset of devices, health checks, and the ability to roll back. Building that machinery is a prerequisite for scale, not an optimisation to add later.