CASE STUDY SNAPSHOT
Customer : A European OEM in defense industry spanning multiple product portfolioSize : > 10000
Project vertical : Mission Critical
Challenge : Need to acquire video data from two different camera at minimal glass-to-glass latency for Airborne camera systems
Solution : Develop a NVIDIA Jetson Tegra TX1 COTS module-based system and optimize video path
Services Availed : Device Driver Development, FPGA Design Services, Multimedia on Embedded Systems
Tools and Technologies :
- Frameworks: Linux4Tegra (L4T)
- MCU/MPU: Nvidia Jetson Tegra TX1
- Programming Language: C, CUDA and VHDL
- FPGA: Intel Arria V
INTRODUCTION
Our customer is one of the large European companies in defense research and development manufacturing electronics for aircraft and ground-based systems. This case study explores the case where we supported them to develop NVIDIA Jetson Tegra TX1 based Dual Camera design to support two different cameras and stream it with minimal latency.
CHALLENGE
Typically, in such aerial camera systems, multiple cameras operate at different spectral frequencies and conditions. For example, one such camera may operate at high resolution and suitable for capturing illuminated areas. Another camera might be a FLIR camera - Forward Looking InfraRed, operating in the infra-red spectrum at a lower frame rate and is idle for thermographic vision. In our case of airborne camera systems development, the challenge is to acquire image sensor raw data from two different image sensors with a minimal glass-to-glass latency.
SOLUTION
Further to the requirement understanding of the airborne camera systems, Embien designed and developed the complete FPGA and firmware stack to achieve the low-latency camera data acquisition over NVIDIA Jetson Tegra TX1.
Airborne camera systems development
This section captures the typical flow of video data in airborne camera systems. For this dual camera development, while originally the video input was available over SDI input, the Intel Arria V FPGA in the system converted it to CSI signals. One of the camera paths outputs a YUV422 stream with only valid luminous information in a single CSI lane and another an RGB888 stream over dual CSI lane interface. After data acquisition, image quality needed to be improved/enhanced using complex image processing algorithms in real-time. Then various pattern extraction algorithms were to be run on the processed image and features extracted from them.
Jetson TX1 camera module integration
Having an excellent understanding of underlying hardware, Linux drivers, and end application needs, Embien helped identify the right hardware platform. Since the customer technical team already had experience in NVIDIA technologies, Jetson Tegra TX1, a powerful embedded system-on-module (SoM) with quad-core ARM Cortex-A57, 4GB LPDDR4 and integrated 256-core Maxwell GPUs was chosen as the processing module. Elroy carrier card was selected as TX1 development platform to meet 2 CSI interface requirement and address form factor constraints.
Embien’s Linux driver development team developed device drivers based on the NVIDIA's Linux4Tegra kernel base for the two independent MIPI CSI 2 camera modules. Presented as V4L2 (Video for Linux 2) devices, these cameras were available as standard /dev/video nodes. In line with modern Linux kernel requirements, the driver configuration was done via Device Tree Source (DTS) files. Upon compilation the Device Tree Blobs (DTB) were created and are loaded before running the kernel. Since the actual camera set up was done by an external FPGA connected over PCIe, the drivers are automatically loaded. Any changes in the camera settings were done over PCIe address space. The system takes full advantage of the NVIDIA Jetson Tegra TX1’s VI interfaces for effective CSI data extraction and storage. The overall driver architecture is captured in the below diagram.
Gstreamer for Dual Camera Development
For further ease of use in this dual camera development, a custom GStreamer plug-in was developed to read video data from the drivers. This v4l2src based plug-in was designed such that the acquired data is directly stored in CUDA memory space. As all the complex image processing must be done over NVIDIA embedded GPUs for efficient parallel processing, this mechanism of directly feeding the data to CUDA host memory space prevented dual copies and reduced latency. Overall, we were able to achieve the low latency requirements of the customer. Another achievement was that our driver development team was able to finally deliver this in a single target release cycle within one week working at the customer site.
BENEFITS
Embien’s support for this NVIDIA Jetson Tegra TX1 camera module integration with airborne camera systems helped the customer to
- Realize their final product faster with many COTS components.
- Leverage their knowledge in the CUDA framework without much rework.
- Achieve a glass-to-glass latency of less than 1.5 frames with this dual camera development.
- Store and process data in CUDA memory directly with zero-copy mechanism saving time.
- Use the standard GStreamer interface for data acquisition.
CONCLUSION
It is Embien’s passion to work in cutting edge technologies across domains, solve customer’s core business challenges and support them to realize their designs and products. Just like this development of NVIDIA Jetson Tegra TX1 based Airborne camera systems, as a leading product engineering services company, we have delivered countless solutions to businesses of different sizes - from small technology start-ups to major enterprises.
This case study is only one such instance of our team’s capability and commitment to solve the client’s problems. Get in touch with us today to make your NVIDIA related development activity a grand success.