A vision of ‘vision and language’ comprises action: An example from road traffic

摘要

This contribution is based on two previously published approaches one of which automatically extracts vehicle trajectories from image sequences of traffic scenes and associates these trajectories with motion verbs. The second approach exploits machine vision in order to maneuver autonomous road vehicles. The combination of these two approaches provides a link from the evaluation of video signals via an abstract representation at the level of natural language concepts to actuator devices in automatic closed loop control of road vehicles. Building on implemented representations for elementary motion verbs and for elementary road vehicle maneuvers, a grammar to represent a nontrivial subset of more complex driving activities on a highway is formulated. Driving on a highway can thereby be investigated not only at the level of control algorithms, but simultaneously at the level of natural language descriptions.