Virtual Aquarium: Mixed Reality Consisting of 3DCG Animation and Underwater Integral Photography

A large number of people keep fishes in their homes and/or offices because of its soothing effect. Recent progress in computer graphics technology has made it possible to create an aquarium using virtual reality. Virtual fish has many advantages compared to real fish. For instance, keeping virtual fishes remove limitations as to the type of fish kept in the tank. From fishes that live deep in the sea to fictional creatures, one’s imagination would be the limit. Fishes that have perished can also be restored and brought back to life.

Figure 1 shows the concept of our system. Creatures created through three-dimensional computer graphics (3DCG) animation appear to be alive and swimming at the surface and beneath the water.

Open image in new window

Our virtual aquarium may share similar concepts with a virtual water tank where a display is arranged on one side of the water tank. In such cases however, displaying the creature in water can be difficult.

AquaTop Display: A True Immersive Water Display System [1]: a display that projects two-dimensional (2D) images on the surface of the water clouded with bath salts. In contrast, our study enables the display of a stereoscopic image as if it actually exists in the water and not only on the surface.

Three-dimensional (3D) crystal engraving or Bubblegram [2]: 3D designs are generated inside a solid block of glass or transparent plastic by irradiating a laser beam inside the material. The objects appear to exist inside the block. However, once it is created, changing the shape or moving the object becomes impossible. In comparison, our system enables animating the 3DCG image displayed inside the water.

We adopted an integral photography (IP) in this system because IP is indisputably the most ideal system among various 3D display systems developed to date. IP, which was invented by Lippmann in 1908 and has been improved continuously, has an advantage in that it provides both horizontal and vertical parallax without the need to wear stereo glasses.

Currently, popular autostereoscopic systems such as the lenticular system and parallax barrier system provide a parallax only in the horizontal direction. Therefore, viewers can see 3D objects from any position only within a certain field of view. In this respect, IP is a technology that has nearly the same advantages as holography. Moreover, IP is considerably more feasible than holography because silver halide photography and laser technology are not required. Composed of a normal flat panel display (FPD), such as a liquid crystal display or an organic light-emitting diode, as well as a fly’s eye lens, as illustrated in Fig. 2, IP is also electronically rewritable. Although color reproducibility depends on the FPD, the reproducibility is still generally excellent.

Open image in new window

Nevertheless, IP has not been widely used until now. One possible reason is that the extremely high initial cost of fly’s eye lens production when produced using a metal mold. Conventionally, because the lens pitch of the fly’s eye lens is considered an integral multiple of the pixel pitch of FPD, the lens should be custom-made according to the pixel pitch of FPD. However, this step can be very costly.

The invention of the extended fractional view (EFV) method [3−5], which is a new method of synthesizing an IP image, drastically changed this situation. In the EFV method, both an integer and an arbitrary real number are allowed as ratio between the lens and pixel pitches. The physical difference between the pitches is processed by software. Therefore, the initial high cost of customizing a fly’s eye lens is reduced because a comparatively inexpensive ready-made fly’s eye lens can be used in combination with various FPDs.

IP is also effective in expressing the glittering effect of a material, such as gems, because the light emitted from each convex lens of a fly’s eye lens depends on its direction [6−8].

In this paper, an IP was introduced and the creature can be seen in the water.　The IP was laid flat to ensure that observation from the top is possible. Figure 3 shows the composition of our system, which consists of a tablet PC with retina display, a plastic box with water, and fly’s eye lens sunk in the water. In this system, the use of water has two implications.

Open image in new window

First, water is essential for imaginary underwater life. This system is a type of mixed reality in which real water and imaginary creatures exist in the same space. In common mixed reality systems, objects created with CG are synthesized with real space using transparent type head-mounted display (HMD). In this system however, IP is used instead of HMD.

A higher reality is provided because the refraction of the light occurs on the surface of the water as shown in Fig. 3.

Second, an advantage is that the focal length of each convex lens of the fly’s eye lens is enhanced when immersed in water [5]. The refractive index of the fly’s eye lens is at 1.5, and its focal length is increased by about three times the original after being sunk in water. As a result, the amount of popping out of underwater objects is tripled, and a higher stereoscopic effect is obtained (Fig. 4).

Open image in new window

The 3DCG scene consists of two swimming creature models and water surface textures. The scene is created in Autodesk Maya.

5.1 3DCG Models

The imaginary fishes were modeled as shown in Fig. 5. The material for the surface had some transparency and was colored light blue. Bead-like materials of various colors were added to the surface to give the impression of a constellation, similar to the modeling for live fish. The addition of bead-like material was done to add mystery and differentiation with the real fishes. Incandescence was also applied to the surface causing it to emit a faint light that enhances the sense of transparency and existence.

Open image in new window

Next, bones were set in the model fishes as shown in Fig. 6. One long backbone was applied to the model from the head to the tail. The backbone was formed with several small bones and then sectioned. By dividing the backbone into several sections, the fishes could bend from left to right, ensuring smooth movement.

Open image in new window

Lights were also set in the scene. The final appearance of the rendered fish is shown in Fig. 7.

Open image in new window

In this scene, two fish models were created and made to appear to exist at different depths. A contrast in positions is necessary to evoke the sense of depth using IP.

5.2 Background

Two plane 2D images, which were painted the surface of the water, were placed in the scene as shown in Fig. 8. One image was placed in front of the fishes and the other placed at the back.. The back image emits a light reflected on the surface of the water, whereas the front image emits a waving light reflected on the bottom. The sense of depth was emphasized by adding the two layers of images to the fishes in the scene.

Open image in new window

The design of the surface of the water was processed as one 2D image plane by using Adobe Photoshop as shown in Fig. 9. The design was created based on the white light reflected on real water. However, image blurring could occur when the synthesized IP image is viewed through the fly’s eye lens (1 mm pitch). Therefore, when the color contrast is weak, the image becomes difficult to see clearly. Hence, the blue and white contrasts were strengthened to enable the white ripple mark to stand out. The rendered scene from the view of camera is shown in Fig. 10.

Open image in new window

Open image in new window

Animating to show movement is important in this case because it will make the fishes seem alive. This time, the creatures were designed based on real fishes. Swimming fishes repeat similar movements. Therefore their movements have a certain periodicity. This movement is similar to how humans walk, that is, they move mostly by repeating the same movement over and over. By focusing on this point, one cycle of the movements was created and then repeated. This repetition enabled us to save time in calculating for the rendering.

The speed of the animation should not be too fast because it takes time to integrate the views from the right and left eyes and to perceive stereoscopic views. When a movement is too fast, a possibility exists that the amount of popping out will decrease.

One fish was placed in the front and the other at the back in this work. The repetitive animation cycle of the swimming movement of the two fishes, which were depicted as shaking their bodies from left to right, was created and its one round period was set to eight frames. The two fishes shake their bodies from left to right in opposite directions. The eight frames of the movement are shown in Fig. 11. When the eighth frame is played, the loop begins once more from the first frame, and repeats this cycle continuously. These frames are displayed as GIF animation.

Open image in new window