CASE STUDY SNAPSHOT
Customer : A multinational defense OEM based out of IsraelSize : > 10000
Project vertical : Mission Critical
Challenge : Avoid re-engineering of product due to memory limitation of the COTS system being used and reduce deployment time
Solution : Performed Tegra TK1 Linux Performance tuning and developed automated deployment system
Services Availed : Embedded Software development, OS Porting and BSP
Tools and Technologies :
- Framework: Linux4Tegra (L4T), JetPack
- MPU: Nvidia Tegra TK1
- Programming Language: C, Scripting
- OS: Linux Kernel
- Tools: Valgrind, Profilers etc.,
INTRODUCTION
Leveraging Commercial Off the Shelf (COTS) modules for the designs, provides a great deal of benefits such as less development complexity, better availability of design resources, proven support ecosystem resulting in quick product realization time, and less cost. But the downside is that we will have to manage with the limitations of the modules, such a limited memory size. In such cases, the memory utilization needed to be reduced, if possible. This page discusses one such instance, where for a defense OEM, Embien helped optimize the overall resource usage by performance tuning on a Tegra TK1 based product and in its automated deployment system development.
CHALLENGE
The SoM that was used for the development had a limited memory and made it difficult to add new features. Also, the building system – pulling from the repository, adding patches, compilation and loading them on to the target etc. were tedious jobs that called for automation.
SOLUTION
Understanding the customer pain points and needs, our embedded software team worked on the development.
Tegra TK1 Linux Performance Tuning
Embien has been developing products with COTS modules for its customers. Generally, the approach was to develop a four-layer carrier board with necessary peripherals and form factor. The COTS module abstracted the complex routing for DDR RAM, PCB assembly, etc.
For one of our designs, we used Abaco Systems mCOM10-K1 a Small Tegra K1 COMExpress Type10 module that was powered by NVIDIA Tegra TK1 SoC. The Tegra TK1 was an embedded SoC with quad-core ARM Cortex-A15 processor and GPU with 192 SM3.2 CUDA cores. Useful for deploying computer vision and deep learning applications, Jetson TK1 ran Ubuntu Linux. With this module as a core, a custom carrier card was developed matching the physical and interfacing requirements. The system had 2GB of DDR3 RAM and 4GB of eMMC Flash.
The project was successfully executed and deployed in the field. Later as a part of the customer’s innovation and evolving market needs, some new features were to be added. The proposed updates needed more memory and processing resources. The customer was contemplating redesigning the product and involved Embien in the decision making to leverage its versatile experience in this area and deep understanding of the system.
Embien analyzed the RAM and Flash memory consumption and profiled it to utmost detail. We identified and removed unused drivers in the L4T kernel considering future application updates as well. After a great deal of deliberations and validations, the unused packages in the Jetpack were removed without affecting the functionalities. For some of the packages, less resource-intensive equivalents were identified and replaced with. Since Tegra K1 ran on Ubuntu, it used Unity GUI which occupied a good percentage of space in file system memory. As the system had a very minimal user interface requirement, we removed it and developed a minimalistic GUI relevant for the end application. We automated the profiling and optimization process and brought down the flash consumption effectively by 1GB which is close to 25% of the total memory and 33% of original consumption. This saved a lot for the customer both in terms of cost and time considering the other way is to redesign the whole product. Other customizations such as custom cronjobs, error reporting mechanisms, on-boot validations etc. were also done.
In addition to Linux performance tuning, Embien also profiled and optimized the business logic implementation. NVIDIA provided tool called Tegra System profiler was used to analyze the overall run time resource usage. The application was redesigned to use the GPU effectively. Other third-party tools such as Valgrind are used to check for run-time memory leaks and validated.
Automated deployment System
Apart from this, Embien also helped the customer in the deployment process. The process was divided into two major activities – development and production. The development process involves pulling in the data from the repository, compiling the sources, and performing extensive testing. Some of the activities done includes,
- Kernel source compilation
- Building application sources
- Flashing Kernel and File System
- Backup and Restore functionality
After this was done and the results were validated, the production process begins. The binaries provided by the development team were used as input and the automated deployment system programed each of the boards with it. Then a different set of validation tests started that checks various performances and metrics for about 24 hours. Once this is done successfully, the suite automatically consolidates the results and marks the board for further integration. If any errors are detected, the test aborts and reports are used to analyze the failure and is fixed.
As an added requirement, Embien also supported the on-field upgrade mechanism. Since the overall system had limited physical access, the only access available was over ethernet. The technician can connect his laptop and initiate the update process after due authentication. There were different levels of updates – application only, the kernel only, both application and kernel, with and without bootloader. The overall process was protected against failures such as invalid images, intermittent power failures, etc. A gold recovery partition was kept that contained a basic working image which would be used in case of critical failures during update. This partition itself could be updated independently. Changes were made at different levels right from u-boot, kernel, root file system and the JetPack programming scripts.
BENEFITS
Our expertise in NVIDIA technologies and Tegra development kits for many years with hands-on experience in Jetson TK1, TX1, TX2, Orin systems helped customers in faster product realization. In this case,
- Production cost was reduced as old system was reused by optimizing memory usage by 33%.
- Performance improvements made to make the application faster.
- Automated deployment and system validation were done to reduce manual errors and speed up time.
- Robust update framework developed for FOTA update.
CONCLUSION
This case study is one of many examples of when Embien helped develop and deploy flexible, future-ready products concerning performance, scalability, adaptability, etc. Our team’s expertise in associated technologies spans driver development, boot loader customization, fail-safe recovery mechanisms, space optimization, CUDA framework developments, GStreamer plugin developments etc. Partner with us to build strong business acumen and cutting-edge professional expertise in your process and address challenges in product development landscape.