It is Embien’s passion to work in cutting edge technologies across domains, solve customer’s core business challenges and support them to realize their designs and products. As a leading product engineering services company, we have delivered countless solutions to businesses of different sizes - from small technology start-ups to major enterprises. This case study explores one such case where we supported a large corporate to develop NVIDIA Tegra TX1 based Dual Camera design to support two different cameras and stream it with minimal latency.
Our customer is one of the large European companies in defense research and development manufacturing electronics for aircraft and ground-based systems. Typically, in such aerial camera systems, multiple cameras operate at different spectral frequencies and conditions. For example, one such camera may operate at high resolution and suitable for capturing illuminated areas. Another camera might be a FLIR camera - Forward Looking InfraRed, operating in the infra-red spectrum at a lower frame rate and is idle for thermographic vision. Such infrared thermal imaging cameras are used for low or no illumination conditions. Or there might be a multispectral camera that can operate across varied frequencies. Each of these cameras typically has different data interfaces and acquisition needed to be performed in parallel without compromising the data rate or quality. Image quality needs to be improved further in real-time using image processing techniques.
In our case of airborne camera systems development, the challenge is to acquire image sensor raw data from two different image sensors with a minimal glass-to-glass latency. One of these camera’s output a YUV422 stream with only valid luminous information in a single CSI lane and another an RGB888 stream over dual CSI lane interface. After data acquisition, image quality needs to improved/enhanced using complex image processing algorithms in real-time. Then various pattern extraction algorithms are to be run on the processed image and features extracted from them.
Having an excellent understanding of underlying hardware, Linux drivers, and end application needs, Embien helped identify the right hardware platform. Since the customer technical team already has experience in NVIDIA technologies, Jetson Tegra TX1, a powerful embedded system-on-module (SoM) with quad-core ARM Cortex-A57, 4GB LPDDR4 and integrated 256-core Maxwell GPUs was chosen as the processing module. Elroy carrier card was selected as TX1 development platform to meet 2 CSI interface requirement and address form factor constraints.
Embien developed device drivers based on the NVIDIA's Linux4Tegra kernel base for the two independent MIPI CSI 2 camera modules. Presented as V4L2 (Video for Linux 2) devices, these cameras were available as standard /dev/video nodes. In line with modern Linux kernel requirements, the driver configuration was done via Device Tree Source (DTS) files. Upon compilation the Device Tree Blobs (DTB) were created and are loaded before running the kernel. Since the actual camera set up was done by an external FPGA connected over PCIe, the drivers are automatically loaded. Any changes in the camera settings were done over PCIe address space. The system takes full advantage of the Tegra TX1’s VI interfaces for effective CSI data extraction and storage. The overall driver architecture is captured in the below diagram.
For further ease of use, a custom GStreamer plug-in was developed to read video data from the drivers. This v4l2src based plug-in is designed such that the acquired data is directly stored in CUDA memory space. As all the complex image processing has to be done over NVIDIA embedded GPUs for efficient parallel processing, this mechanism of directly feeding the data to CUDA host memory space prevented dual copies and reduced latency. Overall, we were able to achieve low latency requirement of the customer. Another achievement is that our driver development team was able to do this in a single target release cycle within one week working at the customer site.
Embien’s support for this Jetson TX1 camera module integration helped the customer to realize their final product faster and leverage their knowledge in the CUDA framework without much rework. This is only one such instance of our team’s capability and commitment to solve the client’s problems. Get in touch with us today to make your NVIDIA related development activity a grand success.
Embien has been working in the NVIDIA technologies and Tegra development kits for the past few years with hands-on experience in Jetson TK1, TX1 and TX2 systems. Our team’s expertise in associated technologies spans driver development, boot loader customization, fail-safe recovery mechanism’s, space optimization, CUDA framework developments, GStreamer plugin developments, etc. Engaging with Embien for NVIDIA centered developments will bring down your product realization time significantly and get real-time service around the world.
