TULIPP platform blooms for enhancing image processors

10 Sep 2019

European development project leads to more energy-efficient, less costly embedded image processing systems for monitoring, medicine and more.

Whether it is designed for the car or the drone, developing image processing software for embedded systems is time-consuming and therefore very expensive.

Now the newly-developed TULIPP platform is making it possible to develop energy-efficient, embedded image processing systems more quickly and less expensively. The Fraunhofer Institute for Optronics, System Technologies and Image Exploitation (IOSB) is a key member of the EU consortium, which has simplified this process.

At first glance drones, driver assistance systems and mobile medical diagnostic equipment don't appear to have much in common. But in reality they do: both technologies make increasing use of image processing, for example for detecting pedestrians and obstacles. Image processing can also be used with mobile X-ray equipment to ensure adequate image quality at reduced radiation levels, thus considerably reducing radioactive exposure.

Applications such as these require small, lightweight, energy-efficient image processing components that need to be real-time capable. This is why embedded systems using field-programmable gate arrays are often used.

Field-programmable gate arrays (FPGAs) are logic components the circuit structure of which can be configured by programming using a language such as VHDL. However, the majority of image processing applications are written in higher-level programming languages such as C/C++, and migrating these to embedded systems is highly complex, says the IOSB.

Starter kit

"The TULIPP consortium of eight partners from six countries has created a development platform consisting of design guidelines, a configurable hardware platform and a real-time-capable operating system that supports multicore processors, and a programing tool chain," says Dr.-Ing. Igor Tchouchenkov, group manager at Fraunhofer IOSB.

"A starter kit put on the market by one of our partners in TULIPP provides additional support. The starter kit makes developing such applications much faster and easier. Porting C++ programs to FPGA, which frequently means several months of work for the developer, can be handled within only a few weeks using the TULIPP starter kit."

This means the developer first has to consider, based on the software programmed in C++, which code elements should be distributed to which hardware components and which program steps could be optimized or parallelized. The formulated design guidelines provide help with this task.

Then the starter kit comes into play. It contains the configurable hardware to which the necessary sensors and output devices can be connected, the multiprocessor-capable real-time operating system, and what is called the STHEM toolchain.

The applications in the toolchain make it possible to optimize the C++ program in such a way that it can be ported to the FPGA as easily and quickly as possible. "One special focus of the toolchain is on energy optimization: after all, the aim is to design image processing systems that can be powered by a small battery whenever possible," said Tchouchenkov. "The toolchain makes it possible to individually display and optimize energy consumption for each code function."

Stereo camera

The consortium worked through three specific use cases in order to develop and test the TULIPP platform: the IOSB research team addressed stereo camera based obstacle detection for drones, while other project partners worked on pedestrian detection in the vicinity of a car, and on live quality enhancement of X-ray images taken by mobile C-Arms during surgical operations.

In TULIPP, they ported the corresponding image processing software from C++ to FPGA.

The development team says the results are “impressive”. The processing, which originally took several seconds to analyze a single image on a high-end PC, can now run on the drone in real time, so now approximately 30 images are analyzed per second.

"The speed of pedestrian detection algorithm could be increased by a factor of 100: Now the system can analyze 14 images per second compared to one image every seven seconds,” added Tchouchenkov.

Enhancement of X-ray image quality by applying noise-removing image filters enables the intensity of radiation to be reduced during surgical operations to one quarter of the previous level. At the same time energy consumption could be significantly reduced for all three applications.