Edge AI System Architecture: Hardware and Software — Part 2

As we saw in part 1 of this article, embedded systems have been an integral part of our lives for decades, powering a wide range of devices from household appliances to industrial machinery. These systems are designed to perform specific tasks with high reliability and efficiency, often operating within constrained environments.

In this article, we will explore the Edge AI System Architecture in detail — covering both hardware and software dimensions. Further we will compare the architectural differences of these two key systems. Understanding Edge AI System Architecture is essential for any engineering team choosing between a traditional MCU-based product and an AI-capable edge platform. Embien's cross-domain expertise enables the design of embedded and Edge AI architectures tailored to diverse industry requirements.

Hardware Architecture of Edge AI System

The Hardware Architecture of Edge AI System is the physical foundation that distinguishes edge AI platforms from traditional embedded systems. Edge AI systems are designed to bring the power of artificial intelligence closer to the source of data generation, enabling real-time processing and decision-making without relying on cloud-based computing resources. A thorough understanding of the edge ai system computing elements — NPUs, heterogeneous SoCs, and high-bandwidth memory subsystems — is essential before finalising any Hardware Architecture of Edge AI System. The Hardware Architecture of Edge AI System typically consists of the following components:

1. Central Processing Unit (CPU): Similar to embedded systems, edge AI systems employ a CPU to handle general-purpose computing tasks, such as running the operating system, managing peripherals, and executing non-AI workloads.
2. Graphics Processing Unit (GPU): While the Edge AI systems use GPUS for graphics processing as well, they are also used for running AI/ML algorithms are their architecture is suited for parallel processing.
3. AI Accelerator: This specialized hardware component, often in the form of a Tensor Processing Unit (TPU), or dedicated AI chip, is optimized for efficient execution of deep learning and machine learning algorithms, accelerating computationally intensive tasks like image recognition, natural language processing, and predictive analytics.
4. Memory subsystem: Edge AI systems require ample memory resources to store and process large datasets, neural network models, and intermediate computation results. This typically includes high-bandwidth memory (e.g., GDDR or HBM) for the AI accelerator and system memory (e.g., DDR) for the CPU.
5. Peripheral interfaces: Similar to embedded systems, edge AI systems may incorporate various peripheral interfaces, such as USB, Ethernet, or PCIe, to facilitate data transfer and communication with external devices or networks.
6. Sensors and actuators: Depending on the application, edge AI systems may integrate sensors for data acquisition (e.g., cameras, microphones, or environmental sensors) and actuators for taking actions based on the AI-powered decisions (e.g., robotics, control systems, or displays).

The hardware components in an Edge AI System Architecture are carefully selected and optimized to balance performance, power efficiency, and cost, enabling real-time AI processing at the edge of the network.

Software Architecture of Edge AI System

The Software Architecture of Edge AI System orchestrates all the layers above the hardware — from the OS and AI framework through to FOTA and cloud integration pipelines. The Software Architecture of Edge AI System is designed to leverage the specialised hardware components and facilitate the deployment and execution of AI models and applications. Robust Embedded Edge AI deployments depend on a Software Architecture of Edge AI System that supports OTA model updates, CI/CD pipelines, and efficient runtime execution. The DevOps services Embien provides extend this further, enabling containerised model deployment and automated regression testing for edge AI firmware. It typically consists of the following layers:

1. Operating System (OS): Edge AI systems often run a lightweight or real-time operating system, such as Linux variants (e.g., Ubuntu Core, Yocto Project), or specialized AI-focused operating systems like NVIDIA's Jetson JetPack ecosystem. Very rarely RTOS’es are used on small MCUs running tinyML frameworks.
2. AI Framework and Libraries: To simplify the development and deployment of AI models, edge AI systems leverage popular AI frameworks and libraries, such as TensorFlow, PyTorch, or ONNX Runtime, which provide optimized implementations for the underlying hardware accelerators.
3. Application Logic: This layer encompasses the core functionality of the edge AI system, including data preprocessing, model inference, and post-processing tasks, as well as any additional business logic or data processing required by the application.
4. Firmware Over-the-Air (FOTA), Configuration Over-the-Air (COTA), and Model Over-the-Air (MOTA): Similar to embedded systems, edge AI systems often incorporate mechanisms for remote updates and configuration, but with the addition of Model Over-the-Air (MOTA) capabilities for seamless deployment of updated or new AI models.
5. Cloud Integration: While edge AI systems perform local processing, they may still integrate with cloud services for tasks such as model training, data aggregation, or centralized management and monitoring.

The software architecture of an edge AI system is designed to leverage the specialized hardware components, optimize AI model execution, and enable seamless integration with cloud services, ensuring efficient and reliable AI processing at the edge.

Architectural Differences between Embedded System and Edge AI system

While embedded systems and edge AI systems share some similarities in their hardware and software architecture, there are notable differences that distinguish these two architectures:

1. Computational Capabilities: Edge AI systems are designed to handle computationally intensive AI and machine learning workloads, leveraging specialized hardware accelerators like GPUs, TPUs, or dedicated AI chips. In contrast, embedded systems are typically optimized for specific tasks and may not have the same level of computational power for complex AI workloads.
2. Memory Requirements: Edge AI systems often require larger memory capacities to store and process large datasets and neural network models. Embedded systems, on the other hand, typically have more modest memory requirements, focused on efficient utilization of available resources.
3. Software Stack: While both architectures may employ real-time operating systems and middleware components, edge AI systems incorporate specialized AI frameworks, libraries, and model optimization tools to facilitate the deployment and execution of AI models. Embedded systems, on the other hand, are primarily focused on running application-specific logic and managing hardware resources.
4. Data Processing: Edge AI systems are designed to process and analyze data locally, leveraging AI algorithms for tasks such as object detection, speech recognition, or predictive maintenance. Embedded systems, while capable of data processing, are often focused on control and automation tasks rather than advanced data analysis.
5. Cloud Integration: Edge AI systems often integrate with cloud services for tasks such as model training, data aggregation, or centralized management. Embedded systems may also communicate with remote systems or the cloud, but their primary focus is on local operation and control.
6. Power and Thermal Considerations: While both architectures aim for power efficiency, edge AI systems may have higher power consumption and thermal dissipation requirements due to the computational demands of AI workloads. Embedded systems are typically designed with stricter power and thermal constraints, prioritizing energy efficiency and compact form factors.
7. Deployment Environments: Embedded systems are often found in a wide range of applications, from consumer electronics to industrial machinery, where they operate in constrained environments with specific requirements. Edge AI systems, on the other hand, are typically deployed in environments where real-time data processing and analysis are critical, such as smart cities, manufacturing plants, or retail settings.

Conclusion

Edge AI System Architecture — encompassing both the Hardware Architecture of Edge AI System and the Software Architecture of Edge AI System — represents a significant step beyond traditional embedded design, driven by the demands of real-time inference at the source of data. A clear grasp of the edge ai system computing elements that underpin the hardware layer, combined with a software stack built for Embedded Edge AI deployment, is what enables engineering teams to deliver production-ready AI systems that are both performant and maintainable.