Mixed Realities

In Chapter 1 we introduced the concept of reality continuum(Section 1.4) as a way to classify variants of Virtual Reality. Such classification depends on the proportion of virtual imagery with respect to a real-world scene that is displayed in an application. We can distinguish three main categories: Virtual Reality(VR), in which only virtual images are used; augmented reality(AR), where computer graphics are superimposed over images of the real world; and augmented virtuality (AV), where imaged portions of the real world are integrated within a virtual world. The term mixed realities(MR) is used to refer to either augmented reality or augmented virtuality, since both of them combine elements from real and virtual worlds.

All the VR techniques, algorithms, interaction devices, and so on, that are described in this book can also be applied in both AR and AV applications. In addition, an important component should be incorporated: a tracking mechanism. Tracking is usually done applying computer-vision and image-processing techniques to data acquired using a video camera. The objective of tracking is to detect relevant objects in the real-world; acquire their position, orientation, contour, and other visual features, and use this information to “attach” computer-generated images to real objects or portions of the scene.

If an MR application is to operate in real time, where computer-generated graphics are superimposed on live images of the surrounding world, all the MR processes must be able to keep pace with the captured image frame rate. Knowing what is present in the real world and where is a vital component for the effective operation of many MR systems. This description can be used to aid virtual object placement, occlusion culling, collision detection, or many other visual or simulated effects. Building up and reconstructing this realworld description in real time is the main challenge faced by MR systems. Different tracking algorithms and techniques have been developed to solve this problem.