Visual Feedback

Abstract

In the introduction to this book we have emphasized the role of vision as a sensor for machines to interact with complex, unknown, dynamic environments, and we have given examples of successful application of vision techniques to autonomous driving and helicopter landing. Interaction with a dynamically changing environment requires action based on the current assessment of the situation, as inferred from sensory data. For instance, driving a car on the freeway requires inferring the position of neighboring vehicles as well as the ego-motion within the lane in order to adjust the position of the steering wheel and act on the throttle or the breaks. In order to be able to implement such a “sensing and action loop,” sensory information must be processed causally and in real time. That is, the situation at time t has to be assessed based on images up to time t. If we were to follow the guidelines and the algorithms described so far in designing an automated driving system, we would have first to collect a sufficient number of images, then organize them into multiple-view matrices, and finally iterate reconstruction algorithms. Each of these steps introduces a delay that compromises the sensing and action loop: when we drive a car, we cannot wait until we have collected enough images before processing them to decide that we needed to swerve or stop. Therefore, we need to adjust our focus and develop algorithms that are suitable for causal, real-time processing. Naturally, the study of the geometry of multiple views is fundamental, and we will exploit it to aid the design of causal algorithms.

I hope that posterity will judge me kindly, not only as to the things which I have explained, but also to those which I have intentionally omitted so as to leave to others the pleasure of discovery.

— René Descartes, La Géométrie, 1637