Using AI To Respond More Quickly To Traffic Hazards

Researchers at the Technical University of Munich (TUM) and the BMW Group have combined sensor data from vehicles, traffic lights and overhead sign gantries to generate specific recommendations for road users. In less than 100 milliseconds, the AI system creates a digital picture of the situation and derives recommendations from it.

As the BMW test vehicle approaches the junction, the artificial intelligence has already gathered information from all relevant sensors in the vicinity. These include infrastructure sensors mounted on the sign gantries at the junction, as well as cameras, radar and a lidar unit on the roof of the test vehicle. The in-car display shows all road users approaching the junction from the right, left, front and rear, with a positional accuracy of up to ten centimeters. The AI continuously predicts where the cars, bicycles and pedestrians are likely to move next.

The advantage of a digital, real-time representation of traffic is clear: "Drivers can now see into areas that would otherwise be obscured, allowing them to react more quickly to potential hazards or be warned of them," explains Professor Alois Knoll, Head of the Chair of Robotics, Artificial Intelligence and Real-Time Systems at TUM. Modern vehicles can already handle highly complex situations with ease. However, there are limits - for example, when children run out onto the road from behind a parked car or cyclists come around a bend. Or when an accident occurs on the motorway. "Our collaboration with TUM is a good example of how research and industry can benefit from each other and jointly explore technological potential," says Ralph Rasshofer, Cluster Product Owner in the Automated Driving division at the BMW Group.

AI consolidates available sensor data

A highly capable artificial intelligence makes it possible to create a real-time representation of traffic that goes well beyond what vehicle sensors can detect. It evaluates all available sensor data from vehicles, sign gantries and traffic lights, and fuses it based, among other things, on current environmental conditions and database information. It then decides whether the respective measurement data should be incorporated into the continually updated 'digital twin' of the traffic situation. "If a sensor is obscured, dirty, out of calibration or otherwise impaired, it is not taken into account in the overall system," explains Prof. Knoll.

In addition to the enormous volume of data that must be communicated and processed simultaneously, the challenge lies in bringing together the various perspectives. The sensors observe traffic from different angles, measure at different sampling rates, and the vehicle sensors are moving at different speeds. "Nevertheless, by constantly predicting movements, it is possible to combine all the sensor data and integrate it into an 'overall model' that is updated every 100 milliseconds," explains TUM researcher Kuo-Yi Chao.

Construction sites as an application scenario

This technology offers particular potential for future applications at construction sites. Vehicles often have accurate maps on board, but these are not always up to date. Traffic routing frequently changes at road construction sites. Here, for example, a camera mounted on a traffic light or a pole can help. It provides a new perspective on the construction site and makes that information available to the vehicle via the digital twin. The researchers have already filed patents for the re-mapping of construction sites and a new approach to data fusion in distributed systems.