GigE Thermal Camera On Embedded Linux With GStreamer Plug-in

CASE STUDY SNAPSHOT

Customer : OEM of custom electro-optic equipment and devices
Size : 200-1000
Project vertical : Defense
Challenge : Need to support a Custom GigE Thermal camera with on embedded Linux
Solution : Customized GStreamer Plug-in development
Services Availed :  Embedded Multimedia
Tools and Technologies :
  • Frameworks : GStreamer
  • Programming Language : C
  • Compiler : GCC

INTRODUCTION

One of the OEM of electro-optical equipment wanted to support a GigE Thermal camera for their embedded Linux based system. Embien, with its rich embedded multimedia expertise, supported the same.

CHALLENGE

The GigE vision compatible camera is supported only over Windows by the vendor where as the system was based on embedded Linux running on ARM Cortex A software.

SOLUTION

Embien analyzed the GigE camera communication and created a custom GStreamer plugin to decode it and displayed it as explained in the below section.

Thermal Cameras

Thermal imaging cameras are electronic devices, designed for detecting heat energy by using infrared radiation emitted from the object in a process that is called thermal imaging. Thermal imaging cameras work by detecting and measuring the infrared radiation emanating from objects. To do so, the camera must first be fitted with a lens that allows IR frequencies to pass through, focusing them on to a special sensor array which can, in turn, detect and read them. The sensor array is constructed as a grid of pixels, each of which reacts to the infrared wavelengths hitting it by converting them into an electronic signal. Those signals are then sent to a processor within the main body of the camera, which converts them using algorithms into a color map of different temperature values. It’s this map which is sent on to be rendered by the display screen.

The common standard today for thermal camera is showing warmer objects with a yellow-orange hue that gets brighter as the object gets hotter. Colder objects are displayed with a blue or purple color.

Application of Thermal Cameras in Embedded Systems

The technology is deployed regularly in scenarios including firefighting, night-time police pursuits, and disaster response search and rescue.

In Military Applications:

Thermal imaging is often used in remotely piloted drones, greatly enhancing general recon capabilities in dangerous or difficult-to-observe conditions, while UAV thermal imaging is also highly beneficial to emergency response units, search and rescue teams, and in tactical military applications.

In Fire and Safety Applications:

Thermal imaging not only helps firefighters to locate survivors in low-visibility conditions where dust, fog, ash and other contaminants occlude visibility - it can also help detect hot-spots, further potential sources of ignition, or indicate the presence of still-burning fires which may be originating from unexpected locations.

In Industrial Applications:

Cameras certified for use in areas subject to explosive gases (e.g. the petrochemical industry); in below-ground applications such as mining; or around high volumes of airborne dust particles found in industries like sugar production and grain-handling.

Thermal Camera Interfaces

While the operating principle of a thermal camera is same on the capture side, based on the camera vendor and the output interface the captured interface is shared in different formats. Some of the thermal camera's output in MIPI-CSI interface, some camera's over IP on Ethernet/WiFi, some on USB etc. The encoding mechanisms employed will also be different like A8 format, Y12 etc. The packaging protocol depends on the transport layer. Embien has rich expertise in working with thermal cameras/FLIR cameras from different manufacturers and integrating to various embedded systems.

In this case, the camera from OZRAY employed the GigE protocol over Ethernet with custom pixel format of encoding.

GigE Vision Protocol

GigE Vision is a global camera interface standard developed originally over the Gigabit Ethernet communication protocol and later to WiFi networks. GigE Vision allows for fast image transfer using low-cost standard cables over very long lengths. With the common GigE Vision protocol support, hardware and software from different vendors can interoperate seamlessly over GigE connections.

GigE Vision has four main elements:

GigE Vision Elements Description
GigE Device Discovery Mechanism Helps GigE vision devices in obtaining IP addresses
GigE Vision Control Protocol (GVCP) Running on the UDP protocol, this standard defines how to control and configure devices. Through this interface, it is possible to specify stream channels and send images and configuration data between cameras and computers
GigE Vision Stream Protocol (GVSP) Over UDP protocol, this defines the data types, and the ways images are transferred as per the GigE standard
GenICam XML description file Schema defined by the European Machine Vision Association's GenICam standard to allow access to camera controls and image streams

GigE is one of the widely used camera interface standards in use especially in industrial automation use cases. Many modern thermal cameras also support GigE Vision protocol.

Gstreamer with the Aravis plugin

GStreamer is a popular multimedia framework being used in embedded Linux applications. It provides a pipe-line based architecture to connect elements in the multimedia path and supports hundreds of plugins for encoding, decoding, parsing, streaming etc. For supporting the GigE protocol, there is a Gstreamer plugin called Aravis. Aravis is a glib/gobject-based library for video acquisition using Genicam cameras. It currently implements the GigE and USB3 protocols used by industrial cameras.

Once the Aravis is installed and running with the GStreamer, it shows the output in the format below. As can be seen the quality of the image shown is not in expectation.

Test Setup

Gstreamer with the Aravis plugin

The test setup and image as shown in the OEM/s windows application is as follows.

Multichannel IO Module

Test Setup

To improve the output, we analyzed the frame format being shared by the camera. We were able to understand the unique pixel format changes. New conversion logic was added and integrated so that the pipeline can be used without any changes. Also, we observed that the gain was not sufficient and hence added a software-based gain logic that can be configured via the GStreamer pipeline.

As it can be seen in the below images, we were able to improve the visualization significantly.

Before & After Conversion

Before and After Conversion

Finally, the customer wanted two new palette modes: RAINBOW and IRON as supported by the Windows application. Embien implemented filters and converted them to enable this kind of visualization.

Gray Scale Mode
Rainbow Mode

Iron Mode

A video of this demo is available on the below link:

BENEFITS

With its expertise in multimedia codecs and GStreamer, Embien is able to quickly support the customer in acquiring thermal camera images over GigE protocol on embedded Linux. Some of the unique benefits we offered to our customers include:

  • Quick enablement of GigE thermal camera on embedded Linux
  • Multiple modes of visualization
  • Gain setting for enhanced image rendering
  • Leverage the power of GStreamer plugin ecosystem

The complete code was made available to the customer for further modification and development.

CONCLUSION

This is one of the success stories where Embien has helped with its embedded multimedia expertise especially involving GStreamer based development.

Want to integrate different kind of thermal cameras to your system design?
Take advantage of your embedded multimedia expertise today