Abstract
This paper introduces a prototype of SketchSynth, a system that enables users to graphically control synthesis using sketches of cross-modal associations between sound and shape. The development is motivated by finding alternatives to technical synthesiser controls to enable a more intuitive realisation of sound ideas. There is strong evidence that humans share cross-modal associations between sound and shapes, and recent studies found similar patterns when humans represent sound graphically. Compared to similar cross-modal mapping architectures, this prototype uses a deep classifier that predicts the character of a sound rather than a specific sound. The prediction is then mapped onto a semantically annotated FM synthesiser dataset. This approach allows for a perceptual evaluation of the mapping model and gives the possibility to be combined with various sound datasets. Two models based on architectures commonly used for sketch recognition were compared, convolutional neural networks (CNNs) and recurrent neural networks (RNNs). In an evaluation study, 62 participants created sketches from prompts and rated the predicted audio output. Both models were able to infer sound characteristics on which they were trained with over 84% accuracy. Participant ratings were significantly higher than the baseline for some prompts, but revealed a potential weak point in the mapping between classifier output and FM synthesiser. The prototype provides the basis for further development that, in the next step, aims to make SketchSynth available online to be explored outside of a study environment.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Adeli, M., Rouat, J., Molotchnikoff, S.: Audiovisual correspondence between musical timbre and visual shapes. Front. Hum. Neurosci. 8, 352 (2014). https://doi.org/10.3389/fnhum.2014.00352
Bottini, R., Barilari, M., Collignon, O.: Sound symbolism in sighted and blind. the role of vision and orthography in sound-shape correspondences. Cognition 185, 62–70 (2019). https://doi.org/10.1016/j.cognition.2019.01.006
Braun, V., Clarke, V.: Using thematic analysis in psychology. Qual. Res. Psychol. 3(2), 77–101 (2006). https://doi.org/10.1191/1478088706qp063oa
Bruford, F., Barthet, M., McDonald, S., Sandler, M.B.: Groove explorer: An intelligent visual interface for drum loop library navigation. In: Proceedings of the ACM IUI Workshops. CEUR-WS.org, Los Angeles, USA (2019)
Ćwiek, A., et al.: The Bouba/Kiki effect is robust across cultures and writing systems. Philosop. Trans. Royal Soc. B: Biol. Sci. 377(1841), 20200390 (2022). https://doi.org/10.1098/rstb.2020.0390
Das, A., Yang, Y., Hospedales, T., Xiang, T., Song, Y.Z.: SketchODE: learning neural sketch representation in continuous time. In: Proceedings of International Conference on Learning Representations. OpenReview.net, virtual (2022)
De Man, B., Reiss, J., Stables, R.: Ten years of automatic mixing. In: Proceedings of the Workshop on Intelligent Music Production. Salford, U.K. (2017)
Deng, L.: The MNIST database of handwritten digit images for machine learning research. IEEE Signal Process. Mag. 29(6), 141–142 (2012). https://doi.org/10.1109/MSP.2012.2211477
Engeln, L., Groh, R.: CoHEARence of audible shapes—a qualitative user study for coherent visual audio design with resynthesized shapes. Pers. Ubiquit. Comput. 25(4), 651–661 (2020). https://doi.org/10.1007/s00779-020-01392-5
Engeln, L., Le, N.L., McGinity, M., Groh, R.: Similarity analysis of visual sketch-based search for sounds. In: Proceedings of Audio Mostly 2021, pp. 101–108. Association for Computing Machinery, Trento, Italy (2021). https://doi.org/10.1145/3478384.3478423
Esling, P., Masuda, N., Chemla-Romeu-Santos, A.: FlowSynth: simplifying complex audio generation through explorable latent spaces with normalizing flows. In: Proceedings of International Joint Conference on Artificial Intelligence, pp. 5273–5275 (2020). https://doi.org/10.24963/ijcai.2020/767
Garber, L., y Ciencia, M.A., Ciccola, T., Amusategui, J.C.: AudioStellar, an open source corpus-based musical instrument for latent sound structure discovery and sonic experimentation. In: Proceedings of International Computer Music Conference, pp. 86–91. Santiago, Chile (2021). https://hdl.handle.net/2027/fulcrum.t435gg568
Giannakis, K.: Sound mosaics: a graphical user interface for sound synthesis based on audio-visual associations, Ph. D. thesis, Middlesex University (2001)
Google: Quick, Draw! (2017). https://quickdraw.withgoogle.com/. Accessed 8 Feb 2023
Grill, T., Flexer, A.: Visualization of Perceptual Qualities in Textural Sounds. In: Proceedings of International Computer Music Conference, pp. 589–596. Michigan Publishing Services, Ljubljana, Slovenia (2012). http://hdl.handle.net/2027/spo.bbp2372.2012.110
Ha, D., Eck, D.: A neural representation of sketch drawings. arXiv preprint arXiv:1704.03477 (2017)
Hayes, B.: FM synth study (2020). https://github.com/ben-hayes/fm-synth-study. Accessed 8 Feb 2023
Hayes, B., Saitis, C.: There’s more to timbre than musical instruments: semantic dimensions of FM sounds. In: Proceedings of International Conference on Timbre. Timbre 2020, Thessaloniki, Greece (2020)
Hayes, B., Saitis, C., Fazekas, G.: Disembodied timbres: a study on semantically prompted fm synthesis. J. Audio Eng. Soc. 70(5), 373–391 (2022). https://doi.org/10.17743/jaes.2022.0006
ISMIR: Homepage (2022). https://ismir.net/. Accessed 8 Feb 2023
iZotope: Mix & Master Bundle Advanced (2023). https://www.izotope.com/en/shop/mix-master-bundle-advanced.html. Accessed 8 Feb 2023
Knees, P., Andersen, K.: Searching for audio by sketching mental images of sound: a brave new idea for audio retrieval in creative music production. In: Proceedings of International Conference on Multimedia Retrieval, pp. 95–102. Association for Computing Machinery, New York, USA (2016). https://doi.org/10.1145/2911996.2912021
Köhler, W.: Gestalt psychology. Liveright (1929)
Küssner, M.B., Tidhar, D., Prior, H.M., Leech-Wilkinson, D.: Musicians are more consistent: gestural cross-modal mappings of pitch, loudness and tempo in real-time. Front. Psychol. 5, 00789 (2014). https://doi.org/10.3389/fpsyg.2014.00789
Löbbers, S., Barthet, M., Fazekas, G.: Sketching sounds: an exploratory study on sound-shape associations. In: Proceedings of International Computer Music Conference, pp. 299–304. Michigan Publishing Services, Santiago, Chile (2021). https://hdl.handle.net/2027/fulcrum.t435gg568
Löbbers, S., Fazekas, G.: Seeing sounds, hearing shapes: a gamified study to evaluate sound-sketches. In: Proceedings International Computer Music Conference, pp. 174–179. Michigan Publishing Services, Limerick, Ireland (2022). https://hdl.handle.net/2027/fulcrum.nk322g689
Löbbers, S., Fazekas, G.: Sketching Sounds Dataset (1.0) [Data set] (2023). https://doi.org/10.5281/zenodo.7590916
Löbbers, S., Fazekas, G.: SketchSynth Dataset (1.0) [Data set] (2023). https://doi.org/10.5281/zenodo.7591067
Martino, G., Marks, L.E.: Synesthesia: strong and weak. Curr. Dir. Psychol. Sci. 10(2), 61–65 (2001)
Mehrabi, A., Dixon, S., Sandler, M.B.: Vocal imitation of synthesised sounds varying in pitch, loudness and spectral centroid. J. Acoust. Soc. Am. 141(2), 783–796 (2017)
Moffat, D., Sandler, M.B.: Approaches in intelligent music production. Arts 8(4), 125 (2019). https://doi.org/10.3390/arts8040125
Ramachandran, V.S., Hubbard, E.M.: Synaesthesia-a window into perception, thought and language. J. Conscious. Stud. 8(12), 3–34 (2001)
Sezgin, T.M.: Feature point detection and curve approximation for early processing of free-hand sketches, Ph. D. thesis, Massachusetts Institute of Technology (2001)
Singh, S., Bromham, G., Sheng, D., Fazekas, G.: Intelligent control method for the dynamic range compressor: a user study. J. Audio Eng. Soc. 69(7/8), 576–585 (2021). https://doi.org/10.17743/jaes.2021.0028
Wolin, A., Eoff, B., Hammond, T.: ShortStraw: a simple and effective corner finder for polylines. In: Proceedings of Eurographics Workshop on Sketch-Based Interfaces and Modeling. The Eurographics Association, Annecy, France (2008). https://doi.org/10.2312/SBM/SBM08/033-040
Xiong, Y., LaViola, J.J.: Revisiting shortStraw: improving corner finding in sketch-based interfaces. In: Proceedings of Eurographics Symposium on Sketch-Based Interfaces and Modeling, pp. 101–108. Association for Computing Machinery, New Orleans USA (2009). https://doi.org/10.2312/SBM/SBM09/101-108
XLN Audio: XO product page (2023). https://www.xlnaudio.com/products/xo. Accessed 8 Feb 2023
Xu, P., et al.: SketchMate: deep hashing for million-scale human sketch retrieval. In: Proceedings of Conference on Computer Vision and Pattern Recognition, pp. 8090–8098. IEEE Computer Society, Salt Lake City, USA (2018)
Zbyszyński, M., Di Donato, B., Visi, F.G., Tanaka, A.: Gesture-timbre space: multidimensional feature mapping using machine learning and concatenative synthesis. In: Kronland-Martinet, R., Ystad, S., Aramaki, M. (eds.) CMMR 2019. LNCS, vol. 12631, pp. 600–622. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-70210-6_39
Acknowledgements
EPSRC and AHRC Centre for Doctoral Training in Media and Arts Technology (EP/L01632X/1).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Löbbers, S., Thorpe, L., Fazekas, G. (2023). SketchSynth: Cross-Modal Control of Sound Synthesis. In: Johnson, C., Rodríguez-Fernández, N., Rebelo, S.M. (eds) Artificial Intelligence in Music, Sound, Art and Design. EvoMUSART 2023. Lecture Notes in Computer Science, vol 13988. Springer, Cham. https://doi.org/10.1007/978-3-031-29956-8_11
Download citation
DOI: https://doi.org/10.1007/978-3-031-29956-8_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-29955-1
Online ISBN: 978-3-031-29956-8
eBook Packages: Computer ScienceComputer Science (R0)