UZH hybrid camera helps drivers avoid pedestrians

30 May 2024

Two camera modalities and AI processing detects obstacles more rapidly than current camera systems.

A project at the University of Zurich (UZH) has developed a novel and partly bio-inspired camera said to be 100 times faster at detection of pedestrians and obstacles than current automotive cameras.

As described in Nature, the new platform offers efficiency gains of several orders of magnitude compared with traditional photographic methods for imaging the surroundings of automobiles.

These existing camera systems can alert the driver or activate emergency braking if, for example, a pedestrian steps into the road.

But they "are not yet fast or reliable enough, and will need to improve dramatically if they are to be used in autonomous vehicles where there is no human behind the wheel," commented UZH.

The project's goal was to improve the performance currently associated with these image-based RGB cameras in driver assistance systems, where typical frame rates of 30 to 45 frames per second can be too low for urgent high-speed decisions by drivers. They can also have inherent "blind times" lasting 22 to 33 milliseconds, long enough for an accidental impact with a fast-moving pedestrian to be set in train.

"Most current cameras are frame-based, meaning they take snapshots at regular intervals," commented Daniel Gehrig from the UZH Robotics and Perception Group.

"But if something happens during the 20 or 30 milliseconds between two snapshots, the camera may see it too late. A solution would be increasing the frame rate, but that translates into more data that needs to be processed in real-time and more computational power."

The UZH solution is to combine a frame-based approach with an event-based one. Event cameras capture per-pixel changes in intensity, rather than full frames at fixed intervals, inspired by the way that human eyes perceive changes. They can offer low motion blur, a high dynamic range and microsecond-level resolution.

Safer driving without computational burdens

UZH built a hybrid platform, one part of which is a standard camera collecting 20 images per second - a relatively low frame rate, but which feeds into an AI image processing system trained to recognize cars or pedestrians. An event camera in the same device feeds its data into a separate AI system designed to analyze changes in 3D data over time, providing high-rate object detection and the opportunity to anticipate observations by the standard camera.

In an automotive setting this arrangement allows the event camera to cover the blind time intervals of the image-based sensor while requiring only low operational bandwidth. It can provide additional certifiable snapshots of reality that show objects before they become visible in the next image, or capture object movements that encode the intent or trajectory of traffic participants.

"The result is a visual detector that can detect objects just as quickly as a standard camera taking 5,000 images per second would do, but requires the same bandwidth as a standard 50-frame-per-second camera," said Daniel Gehrig.

In trials, the UZH hybrid platform proved to be one hundred times faster at detection, while reducing the amount of data that must be transmitted between the camera and the onboard computer as well as the computational power needed to process the images, without affecting accuracy.

Crucially, the system could indeed effectively detect cars and pedestrians that enter the field of view between two subsequent frames of the standard camera, increasing safety for both the driver and traffic participants, especially at high speeds. Use in combination with automotive lidar systems could provide further advantages for drivers.

"Hybrid systems like this could be crucial to allow autonomous driving, guaranteeing safety without leading to a substantial growth of data and computational power," commented UZH's Davide Scaramuzza.