Many embedded systems, especially in industrial, automotive, and mission-critical applications, need to be highly responsive, responding to input or events within a specified timeframe, often with strict timing requirements or running operations with very minimal jitters. Such real-time applications need a different kind of design and underlying architecture. While Linux finds wide use, running real time applications on it is challenging due to the inherent nature of the Linux kernel's scheduling mechanism and other overheads. On the hardware front, there are many ready to use computing modules available. Of them, the Jetson TX1/TX2, Nano and AGX are series of powerful single-board computers developed by NVIDIA, specifically designed for embedded systems and edge computing applications. Jetsons runs on the Linux operating system, using a customized version called 'Linux for Tegra' (L4T) that offers advanced features like CUDA, Deepstream libraries to utilise the GPUs.
Real Time Applications on Linux
While the platform is quite powerful, running real time applications directly is difficult. For such applications, specialised real-time operating systems (RTOS) or bare-metal solutions may be more appropriate, for which the powerful Linux-based libraries are not available. As a middle way, it is possible to use a Hypervisor model where Linux, RTOS, or both can be running on it. While the Jetson ecosystem itself does not have built-in support for virtualization, some third-party hypervisors can potentially be adapted to work with it, such as Xen, KVM, and Jailhouse.
In this article, we will see how, by combining the NVIDIA Jetson platform and Jailhouse, we can enable real-time applications with a high level of isolation and determinism. We will use a specific case of the Jetson TX1 as the use case.
Setting up Jailhouse on Jetson TX1
The Jailhouse hypervisor is open-source partitioning Linux-based hypervisor designed for safety-critical real-time applications. It is designed specifically for x86 and ARM processors. It offers strong isolation between different domains or cells and an effective platform for executing numerous real-time applications on a single hardware platform.
The below diagram explains how the internal architecture changed from prior to enabling the Jailhouse to after it was
Initially, Linux used all the cores. After enabling the hypervisor, it moves Linux to the root-cell and still uses all the CPU cores. When we create a new cell (inmate), the hypervisor calls cpu_down() for some of the CPU cores to offline them. The new cell will be assigned these CPU cores and hardware resources dedicated in the cell configuration file. Let us see how to set up the Jailhouse for Jetson TX1.
- Obtain the Jailhouse hypervisor source code from the official repository or a compatible version.
- We will have to cross-compile the Linux kernel with the necessary Jailhouse configurations for the Jetson TX1 platform. Jailhouse configuration files specific to the Jetson TX1 hardware, define the partitions, memory regions, devices, and CPUs, which are expressed in the form of plain C files (found under configs/ in the sources) compiled into raw binaries.
- The hypervisor requires a contiguous piece of RAM for itself and each additional cell. This currently must
be pre-allocated during boot-up.
$ mem=3968M vmalloc=512M
- Via bootarg remove the console access on serial port from Linux.
$ console=ttyS0,115200n8 earlyprintk=uart8250-32bit,0x70006000 console=tty0
- Cross-compile the Linux kernel with Jailhouse support for the Jetson TX1 platform. Run make with the appropriate configuration to build the kernel and generate the kernel image.
$sudo make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- KDIR=$KERNEL_PATH DESTDIR=$BUILD_DIR_PATH install
Jailhouse.ko kernel module is generated and is ready to enable the hypervisor layer and create cell with demo applications.
Running Jailhouse on Jetson TX1
The below section explains how to run the compiled Jailhouse on the platform
Once Jailhouse is compiled, load the Jailhouse hypervisor module into the Linux kernel.
- $ sudo insmod jailhouse.ko
- $ sudo jailhouse enable configs/arm64/jetson-tx1.cell
This module allows the kernel to work as a Jailhouse hypervisor. A device file entry called /dev/jailhouse is created, and Jailhouse is enabled, and the loader linearly maps this memory into the virtual address space of the kernel.
Enable command successfully launches the hypervisor maps Jailhouse's reserved memory region into kernel space using ioremap() and marks its pages as executable. Jailhouse doesn't depend on Linux to provide services to guests. However, Linux is used to initialize the hypervisor and to control it later.
Creating the Cell:
Create the Jail house cell with the following command:
- sudo jailhouse cell create configs/arm64/jetson-tx1-demo.cell
ailhouse needs to 'shrink' the Linux cell by shifting hardware resources to the new cell to create a new cell. Linux becomes the root cell. The jailhouse user-space tool reads and stores them from files in memory. Each Jailhouse cell has its own configuration file. While it is possible for inmates to share one configuration file (and thus one cell), only one of these inmates will be active at a given time.
Jailhouse provides various demo applications one of which is a UART demo application that will access the primary UART console and print hello from the inmate. First load it and then start to begin executing the configured cells.
- $sudo jailhouse cell load jetson-tx1-demo uart-demo.bin
- $sudo jailhouse cell start jetson-tx1-demo
Fig. UART-demo Application print on console.
Shutting down the cell
Cell destruction is performed by specifying the configuration file of the desired cell.
- $sudo jailhouse cell shutdown jetson-tx1-demo
- $sudo jailhouse cell destroy jetson-tx1-demo
This command will remove the cell's configuration from the hypervisor. Unmap all the memory regions from inmate. To destroy a cell or create an additional one, shut down the running cell first.
New cells can be created and executed any number of times.
Real Time Application on Jetson TX1.
With a fair idea of running Jailhouse cells on TX1, let us do the profiling of Jailhouse inmate performance on TX1 using GPIO ‘s. We have created an inmate demo application which toggles one GPIO pin- gpio36 pin no.12 on Jetson TX1 every 400us. Enable access to it with ROOTSHARED flag enabled in configuration file.
In jetson-tx1-demo.c, configuration file set
Now we compiled and loaded the gpio-demo application to the inmate and start. Simultaneously, we ran a script to toggle another GPIO pin on the root cell. These two GPIO’s are used for benchmarking the performance.
The below waveform captures give an idea of performance differences between the Linux application code and Jailhouse powered RTOS code.
We can see that the inmate application provides uniform response without much deviation while in Linux root cell, period is inconsistent. We can also observe some impact due the cache flushes on the Jetson, but it can be mitigated with proper design.
Thus, we can see that real-time behaviours like latency, determinism, and predictability are achieved with a hypervisor-based approach.
Also, Jailhouse Hypervisor excel at providing the minimal level of isolation, resource management, and multi-tenancy, making them ideal for scenarios to run both real-time applications as well as use the powerful GPU acceleration on the Jetson TX1.
Embien has rich experience in tuning Linux for running realtime applications. Our team has worked with numerous Hypervisors like Jailhouse, KVM, QEMU etc for various applications and platforms like Nvidia Jetson TX2, Jetson Nano, NXP ImX8, Renesas RCar platforms. We also have credentials in working with asymmetric multi-core processing technologies as well as containerization technologies.