Virtual pottery: a virtual 3D audiovisual interface using natural hand motions
- 443 Downloads
In this paper, we present our approach towards designing and implementing a virtual 3D sound sculpting interface that creates audiovisual results using hand motions in real time. In the interface “Virtual Pottery,” we use the metaphor of pottery creation in order to adopt the natural hand motions to 3D spatial sculpting. Users can create their own pottery pieces by changing the position of their hands in real time, and also generate 3D sound sculptures based on pre-existing rules of music composition. The interface of Virtual Pottery can be categorized by shape design and camera sensing type. This paper describes how we developed the two versions of Virtual Pottery and implemented the technical aspects of the interfaces. Additionally, we investigate the ways of translating hand motions into musical sound. The accuracy of the detection of hand motions is crucial for translating natural hand motions into virtual reality. According to the results of preliminary evaluations, the accuracy of both motion-capture tracking system and portable depth sensing camera is as high as the actual data. We carried out user studies, which took into account information about the two exhibitions along with the various ages of users. Overall, Virtual Pottery serves as a bridge between the virtual environment and traditional art practices, with the consequence that it can lead to the cultivation of the deep potential of virtual musical instruments and future art education programs.
KeywordsAudio sonification Virtual pottery Virtual musical instrument Hand gesture recognition
These works have been supported by TransLab, University of California, Santa Barbara (UCSB), under the guidance of Prof. Marcos Novak.
- 1.Beyer and Meier M (2011) “Music interfaces for novice users: composing music on a public display with hand gestures,” in Proceedings of the International Conference on New Interfaces for Musical Expression (NIME 2011)Google Scholar
- 2.Blake A, Isard M (1994) “3D position, attitude and shape input using video tracking of hands and lips,” in Proceedings of the 21st annual conference on computer graphics and interactive techniques (SIGGRAPH '94). ACM Press, New York, pp 185–192Google Scholar
- 3.BlobScanner. Retrieved from http://code.google.com/p/BlobScanner/
- 5.Da Fontoura Costa L, et al. (2000) Shape analysis and classification: theory and practice. CRC PressGoogle Scholar
- 6.Han Y, Han B (2012) “Virtual Pottery: an interactive audio-visual installation,” in Proceedings of 12th international conferences on New Interfaces on Musical Expression (NIME 2012)Google Scholar
- 7.Holz C, Wilson A (2011) “Data miming: Inferring spatial object descriptions from human gesture,” in Proceedings of the 2011 annual conference on Human factors in computing systems (ACM CHI ’11), pp. 811–820Google Scholar
- 8.Ip HHS, Kwong B, Law KCK (2005) “BodyMusic: A novel framework design for body-driven music composition,” in Proceedings of the 2005 ACM SIGCHI international conference on advances in computer entertainment technology. ACM Press, New York, pp 342–345Google Scholar
- 9.Ip HHS, Kwong B, Law KCK (2005) “Cyber composer: hand gesture-driven intelligent music composition and generation,” in Proceedings of 11th international multimedia modeling conference, pp.46–52Google Scholar
- 10.Kameyama K (1997) “Virtual clay modeling system,” in Proceedings of the ACM Symposium on Virtual Reality Software and Technology, pp 197–200Google Scholar
- 12.Korida K, Nishino H, Utsumiya K (1997) “An interactive 3D interface for a virtual ceramic art work environment,” in Proceedings of International Conference on Virtual Systems and Multimedia (VSMM '97), pp. 227–234. IEEE Computer SocietyGoogle Scholar
- 13.L’Artisan Electronique. Retrieved from http://www.creativeapplications.net/openframeworks/lartisan-electronique-openframeworks/.
- 15.Levin G, Lieberman Z (2005) “Sounds from shapes: audiovisual performance with hand silhouette contours in the manual input sessions,” in Proceedings of the 2005 international conference on New Interfaces for Musical Expression (NIME 2005), pp. 115–120Google Scholar
- 16.McDonnell KT, Qin H, Wlodarczyk RA (2001) “Virtual clay: a real-time sculpting system with haptic toolkits,” in Proceedings of the 2001 symposium of Interactive 3D graphics, pp.179–190, New York, USAGoogle Scholar
- 17.Mulder and Fels S (1998) “Sound sculpting: manipulating sound through virtual sculpting,” in Proceedings of Western Computer Graphics SymposiumGoogle Scholar
- 18.Processing. Retrieved from http://processing.org/
- 19.Rigid Body toolkit. Retrieved from http://www.naturalpoint.com/OptiTrack/products/rigidbody-toolkit/
- 20.UDP library. Retrieved from http://ubaa.net/shared/processing/udp/
- 21.Vinayak, Murugappan S, Piya C, Ramani K (2012) “Handy-Potter: Rapid 3D Shape Exploration through Natural Hand Motions,” in Proceedings of the ASME 2012 International Design Engineering Technical Conferences & Computers and Information in Engineering Conference (IDETC/CIE 2012), August 12–15Google Scholar
- 22.Young D (2002) “The Hyperbow controller: Real-time dynamics measurement of violin performance,” in Proceedings of the 2002 conference on New Instruments for Musical Expression (NIME 2002)Google Scholar