Fraunhofer IOSB develops novel Advanced Occupant Monitoring System

01 Sep 2021

To assess self-drive vehicle driver activity based on image data; and to assess how quickly they could take control.

An occupant monitoring system supports assistance systems in the vehicle interior. It provides information about the occupants present. Typically, current systems rely on simple sensors for detecting steering times, steering movements, seat occupancy and gaze detection. This provides rudimentary detection of whether the driver is showing signs of fatigue or is too distracted.

Fraunhofer IOSB’s Advanced Occupant Monitoring System (AOMS) goes significantly further than previous state-of-the-art systems and uses optical sensors in the vehicle interior, as they are becoming increasingly common in modern vehicles. The system detects the driver and all occupants equally.

The AOMS recognizes the 3D body poses of all persons, analyzes their movement behavior and classifies the activity of each individual person detected. This makes it possible not only to detect critical situations such as a driver falling asleep, but also to distinguish between different activities and the associated levels of distraction.

The basis of the AOMS is its real-time detection of body poses in 3D. For all vehicle occupants captured by cameras, the body pose is recognized as a 3D skeletal model using machine learning techniques.

The resulting image of the captured occupants includes the position of the eyes and head, neck, shoulders, elbows, wrists, torso, pelvis, and upper and lower legs – as soon as they are visible in the camera image. The recording does not require biometric data and is therefore particularly privacy-friendly.

The system is capable of using individual 3D cameras or several 2D cameras, from whose perspectives the 3D joint points are reconstructed. The cameras can be mounted in any position as long as the view of the respective persons is sufficient.

Gesture recognition

The positions of the eyes, the elbows as well as the wrists also emerge from the determined body pose skeleton. This makes it possible to interpret both the direction of the forearm and the pointing direction of the eye-hand extension as pointing gestures. Both variants are available as a 3D vector.

Pointing gestures can therefore be mapped directly and with centimeter accuracy to known objects inside or outside the vehicle. Since pointing gesture detection is based on 3D body pose recognition, left arms can also be distinguished from right arms and pointing gestures can be recognized anywhere in the interior - not just at conventional seating positions or in prescribed interaction areas.

In the system’s launch statement, the IOSB commented, “It is to be expected that as the vehicle becomes increasingly automated, the driver and other occupants will also be allowed and offered more freedom for secondary activities in the interior.”

Concept vehicles are already breaking with the classic seating arrangement and extending the driver's range of movement and action, for example, to the entire interior by allowing seats to be turned and moved. Against the background of such freedom of movement, the question arises as to how interaction with services in the interior should be realized. Pointing gestures and voice input are obvious options, but they must offer both the necessary robustness and freedom to allow occupants the room to maneuver in the interior.

Activity detection in vehicle interior

In manual driving situations, the driver ideally focuses his full attention on the road. He drives and steers the car and does not engage in any secondary activity. As the level of vehicle automation increases, however, the driver is freed from his driving responsibility.

He is given the freedom to pursue secondary activities. Partially automated vehicles must take this fact into account when dealing with situations in which driving responsibility is to be handed back to the driver. The driver may be distracted, asleep or even have a medical emergency.

Fraunhofer IOSB's Advanced Occupant Monitoring System detects the activity of all occupants inside the vehicle. It is able to distinguish between up to 35 activities, including drinking, eating, sleeping, reading, making phone calls, and more.

For this purpose, state-of-the-art machine learning processes and fuses the 3D body skeleton recognition of the occupants in combination with object detection and intelligent analysis of the movement behavior of all detected persons. This makes it possible to reliably distinguish whether someone is reaching for a cell phone and making a call or opening a bottle and bringing it to their mouth. The Advanced Occupant Monitoring System thus provides important information on the driver's distracted state.

In addition, the system also provides important information on the prevailing situation in the vehicle interior and the context of a person's actions. This makes it possible, for example, to distinguish unintentional from intentional pointing gestures or to offer innovative assistance functions tailored to the individual needs of the occupants.