The Visual Computer

, 27:905 | Cite as

Stereo music visualization through manifold harmonics

  • Thomas Lewiner
  • Clarissa Marques
  • João Paixão
  • Scarlett de Botton
  • Allyson Cabral
  • Renata Nascimento
  • Vinícius Mello
  • Adelailson Peixoto
  • Dimas Martinez
  • Thales Vieira
Original Article

Abstract

Music visualizations are nowadays included with virtually any media player. They usually rely on harmonic analysis of each sound channel, which automatically generate parameters for procedural image generation. However, only few music visualizations make use of 3d shapes. This paper proposes to use spectral mesh processing techniques, here manifold harmonics, to produce 3d stereo music visualization. The images are generated from 3d models by deforming an initial shape, mapping the sound frequencies to the mesh harmonics. A symmetry criterion is introduced to enhance the stereo effects on the deformed shape. A concise representation of the frequency mapping is proposed to allow for an animated gallery interface with genetic reproduction. Such galleries let the user quickly navigate between visual effects. Rendering such animated galleries in real time is a challenging task, since it requires computing and rendering the deformed shapes at a very high rate. This paper introduces a direct GPU implementation of manifold harmonics filters, which allows the displaying of the animated galleries.

Keywords

Manifold harmonics Symmetry Sound visualization Stereophony Geometry processing GPU Design galleries 

References

  1. 1.
    Apple: PBORenderToVertexArray: render-to-vertex-array using FBO, PBO and VBO (2006). Developer.apple.com/mac/library/samplecode (2011)
  2. 2.
    Bellard, F.: FFmpeg (2004). www.ffmpeg.org
  3. 3.
    Bordignon, A., Sigaud, L., Tavares, G., Lopes, H., Lewiner, T., Morgado, W.: Arch generated shear bands in granular systems. Physica A 388(11), 2099–2108 (2009) CrossRefGoogle Scholar
  4. 4.
    Breebaart, J., Faller, C.: Spatial Audio Processing. Wiley, New York (2007) CrossRefGoogle Scholar
  5. 5.
    Clough, R.W., Penzien, J.: Dynamics of Structures. McGraw-Hill, New York (1975) MATHGoogle Scholar
  6. 6.
    Comstock, H.: Radio adds third dimension. Popular Sci. pp. 104–106 (1953) Google Scholar
  7. 7.
    de Moura Pinto, F., Freitas, C.M.D.S.: Two-level interaction transfer function design combining boundary emphasis, manual specification and evolutive generation. In: Sibgrapi, pp. 281–288. IEEE Press, New York (2006) Google Scholar
  8. 8.
    Gardner, W.: 3D Audio Using Loudspeakers. Kluwer, Dordrecht (1998) Google Scholar
  9. 9.
    Hernandez, V., Roman, J., Vidal, V.: SLEPc: A scalable and flexible toolkit for the solution of eigenvalue problems. ACM Trans. Math. Softw. 31(3), 362 (2005) MathSciNetCrossRefGoogle Scholar
  10. 10.
    Hiebert, G.: OpenAL programmer’s guide (2005). Connect.creativelabs.com/openal
  11. 11.
    Jenny, H.: Cymatics: A Study of Wave Phenomena & Vibration, 3rd edn. Macromedia (2001) Google Scholar
  12. 12.
    Kessenich, J.: The OpenGL Shading Language v 4.0 (2010). www.opengl.org/documentation/glsl
  13. 13.
    Kubelka, O.: Interactive music visualization. In: Central European Seminar on Computer Graphics (2000) Google Scholar
  14. 14.
    Lage, M., Lewiner, T., Lopes, H., Velho, L.: CHF: a scalable topological data structure for tetrahedral meshes. In: Sibgrapi, pp. 349–356. IEEE Press, New York (2005) Google Scholar
  15. 15.
    Lewiner, T., Vieira, T., Bordignon, A., Cabral, A., Marques, C., Paixão, J., Custódio, L., Lage, M., Andrade, M., Nascimento, R., de Botton, S., Pesco, S., Lopes, H., Mello, V., Peixoto, A., Martinez, D.: Tuning manifold harmonics filters. In: Sibgrapi, pp. 110–117. IEEE Press, New York (2010) Google Scholar
  16. 16.
    Lewiner, T., Vieira, T., Martínez, D., Peixoto, A., Mello, V., Velho, L.: Interactive 3D caricature from harmonic exaggeration. Comput. Graph. 35(3), 586–595 (2011) CrossRefGoogle Scholar
  17. 17.
    Lévy, B., Zhang, H.R.: Spectral mesh processing. In: Siggraph Asia Course Note, pp. 1–47. ACM Press, New York (2009) CrossRefGoogle Scholar
  18. 18.
    Liu, Y., Prabhakaran, B., Guo, X.: A robust spectral approach for blind watermarking of manifold surfaces. In: Multimedia and Security, pp. 43–52. ACM Press, New York (2008) Google Scholar
  19. 19.
    Marks, J., Andalman, B., Beardsley, P., Freeman, W., Gibson, S., Hodgins, J., Kang, T., Mirtich, B., Pfister, H., Ruml, W., et al.: Design galleries: a general approach to setting parameters for computer graphics and animation. In: Siggraph, p. 400. ACM Press, New York (1997) Google Scholar
  20. 20.
    O’Brien, J.F., Shen, C., Gatchalian, C.M.: Synthesizing sounds from rigid-body simulations. In: Symposium on Computer animation, pp. 175–181. ACM Press, New York (2002) Google Scholar
  21. 21.
    Ovsjanikov, M., Sun, J., Guibas, L.: Global intrinsic symmetries of shapes. In: SGP, pp. 1341–1348. Eurographics, Geneva (2008) Google Scholar
  22. 22.
    Patin, F.: Beat detection algorithms (2003). www.gamedev.net/reference/programming/features/beatdetection
  23. 23.
    Pentland, A., Williams, J.: Good vibrations: modal dynamics for graphics and animation. ACM Siggraph 23(3), 207–214 (1989) Google Scholar
  24. 24.
    Rong, G., Cao, Y., Guo, X.: Spectral mesh deformation. Vis. Comput. 24(7), 787–796 (2008) CrossRefGoogle Scholar
  25. 25.
    Taubin, G.: A signal processing approach to fair surface design. In: Siggraph, pp. 351–358 (1995) Google Scholar
  26. 26.
    Vallet, B., Lévy, B.: Spectral geometry processing with manifold harmonics. Comput. Graph. Forum, 27, 251–260 (2008) CrossRefGoogle Scholar
  27. 27.
    Vieira, T., Bordignon, A., Peixoto, A., Tavares, G., Lopes, H., Velho, L., Lewiner, T.: Learning good views through intelligent galleries. Comput. Graph. Forum 28(2), 717–726 (2009). (Eurographics Proceedings) CrossRefGoogle Scholar
  28. 28.
    Wang, K., Luo, M., Bors, A., Denis, F.: Blind and robust mesh watermarking using manifold harmonics. In: ICIP, pp. 3657–3660. IEEE Press, New York (2009) Google Scholar
  29. 29.
    Wu, H.Y., Luo, T., Wang, L., Wang, X.L., Zha, H.: 3D shape retrieval by using manifold harmonics analysis with an augmentedly local feature representation. In: VRCAI, pp. 311–313. ACM Press, New York (2009) CrossRefGoogle Scholar
  30. 30.
    Yinghui, C., Jing, W., Xiaohui, L.: Real-time deformation using modal analysis on graphics hardware. In: Graphite, pp. 173–176. ACM Press, New York (2006) CrossRefGoogle Scholar

Copyright information

© Springer-Verlag 2011

Authors and Affiliations

  • Thomas Lewiner
    • 1
  • Clarissa Marques
    • 1
  • João Paixão
    • 1
  • Scarlett de Botton
    • 1
  • Allyson Cabral
    • 1
  • Renata Nascimento
    • 1
  • Vinícius Mello
    • 3
  • Adelailson Peixoto
    • 2
  • Dimas Martinez
    • 2
  • Thales Vieira
    • 2
  1. 1.Department of MathematicsPUC-RioRio de JaneiroBrazil
  2. 2.Institute of MathematicsUFALMaceióBrazil
  3. 3.Institute of MathematicsUFBASalvadorBrazil

Personalised recommendations