Skip to main content

Advertisement

SpringerLink
Log in

MORE – a multimodal observation and analysis system for social interaction research

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The MORE system is designed for observation and machine-aided analysis of social interaction in real life situations, such as classroom teaching scenarios and business meetings. The system utilizes a multichannel approach to collect data whereby multiple streams of data in a number of different modalities are obtained from each situation. Typically the system collects a 360-degree video and audio feed from multiple microphones set up in the space. The system includes an advanced server backend component that is capable of performing video processing, feature extraction and archiving operations on behalf of the user. The feature extraction services form a key part of the system and rely on advanced signal analysis techniques, such as speech processing, motion activity detection and facial expression recognition in order to speed up the analysis of large data sets. The provided web interface weaves the multiple streams of information together, utilizes the extracted features as metadata on the audio and video data and lets the user dive into analyzing the recorded events. The objective of the system is to facilitate easy navigation of multimodal data and enable the analysis of the recorded situations for the purposes of, for example, behavioral studies, teacher training and business development. A further unique feature of the system is its low setup overhead and high portability as the lightest MORE setup only requires a laptop computer and the selected set of sensors on site.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. http://www.livinginpothill.blogspot.fi/2013/03/we-want-more.html

  2. Videos available at http://www.oulu.fi/fsr/node/23934

  3. http://leaforum.fi/business/facilities-equipment/more-recording-system/

References

  1. Amidon E, Flanders N (1967) The role of the teacher in the classroom, (revised edition). Association for Productive Teaching Inc

  2. Anderson H (1939) The measurement of domination and of socially integrative behavior in teachers’ contacts with children. Child Dev 10:73–89

    Article  Google Scholar 

  3. Anderson H, Brewer H (1945) Studies of teachers’ classroom personalities, I. Dominative and socially integrative behavior of kindergarten teachers. Stanford University Press, Stanford, Cal

    Google Scholar 

  4. Anderson H, Brewer J (1946) Studies of teachers’ class-room personalities, II. Effects of dominative and integrative contacts on children’s classroom behavior. Stanford University Press, Stanford, Cal

    Google Scholar 

  5. Chang CC, Lin CJ (2011) LIBSVM: a library for support vector machines. ACM Trans Intell Syst Technol 2(3):27. doi:10.1145/1961189.1961199. software available at. http://www.csie.ntu.edu.tw/cjlin/libsvm

    Article  Google Scholar 

  6. Derry S, Pea R, Engle R, Erickson F, Gpödman R, Hall R, Koschmann T, Lemle J, Sherin M, Sherin B (2010) Conducting video research in the learning sciences: Guidance on selection, analysis, technology, and etchics. J Learn Sci 19(1):3–53

    Article  Google Scholar 

  7. eXist Solutions GmbH (2000) exist-db: Open source native XML database. Online, http://exist-db.org/, accessed April 25, 2014

  8. Flanders N (1970) Analysing Teaching Behavior. Addison-Wesley Pub. Co. Reading, Mass

  9. Gatica-Perez D (2009) Automatic nonverbal analysis of social interaction in small groups: A review. Image Vis Comput (Vis multimodal anal hum spontaneous behav) 27(12):1775–1787. doi:10.1016/j.imavis.2009.01.004

    Google Scholar 

  10. Goldman R (2007) Orion™, an online digital video analysis tool: Changing our perspectives as an interpretive community, Lawrence Erlbaum Associates, Mahwah, NJ, pp 507–520

  11. Horn BK, Schunck BG (1981) Determining optical flow. Artif Intell 17(1):185–203

    Article  Google Scholar 

  12. Horn E (1914) Distribution of opportunity for participation among the various pupils in classroom recitations. Contributions to Education, 67

  13. International Telecommunication Union (ITU) (2012) P.56: Objective measurement of active speech level. Online, http://www.itu.int/rec/T-REC-P.56, accessed August 18, 2014

  14. IRIS Connect (2012) Classroom observation — lesson observation — teacher professional development. Online, http://www.irisconnect.co.uk/, accessed August 18, 2014

  15. Kiema H, Mäenpää M, Leinonen T, Soini H (2014) Peer group counseling as a tool for promoting managers’ communication skills in industrial and planning organizations. In: COLLA 2014, The Fourth International Conference on Advanced Collaborative Networks, Systems and Applications, pp 28–33, http://www.researchgate.net/publication/263203534_Peer_Group_Counseling_as_a_Tool_for_Promoting_Managers%27_Communication_Skills_in_Industrial_and_Planning_Organizations

  16. Mchenry V (1968) The use of video processes in teacher education. Multi-State Teacher Education Project, Baltimore, MD. & Utah State Board of Education, Salt Lake City. D.C, Washington

    Google Scholar 

  17. Noldus (1989) Software and labs for behavioral research and video tracking., http://www.noldus.com/, accessed April 25, 2014

  18. OpenCV (1999) Open source computer vision library. Online., http://code.opencv.org/, accessed December 10, 2014

  19. Pea R, Lindgren R (2008) Video collaboratories for research and education: An analysis of collaboration design patterns. IEEE Trans Learn Technol 1(4):235–247. doi:10.1109/TLT.2009.5

    Article  Google Scholar 

  20. Pea R, Mills M, Rosen J, Dauber K, Effelsberg W, Hoffert E (2004) The Diver project: Interactive digital video repurposing. IEEE Multimedia:54–61. doi:10.1109/MMUL.2004.1261108

  21. Point Grey Research Inc (2013) Accurate 360° spherical imaging with multiple pre-calibrated sensors. Online., http://www.ptgrey.com/Content/Images/uploaded/misc/Ladybug-Spherical-Cameras-WhitePaper.pdf, accessed February 24, 2015

  22. Powell AB, Francisco JM, Maher CA (2003) An analytical model for studying the development of learners’ mathematical ideas and reasoning using videotape data. J Math Behav 22(4):405–435. doi:10.1016/j.jmathb.2003.09.002

    Article  Google Scholar 

  23. Puckett RC (1928) Making supervision objective. School Rev 36:209–212

    Article  Google Scholar 

  24. Schreer O, Fieldmann I, Weissig C, Kauff P, Schafer R (2013) Ultrahigh-resolution panoramic imaging for format-agnostic video production. Proc IEEE 101(1):99–114

    Article  Google Scholar 

  25. Siipo A, Laukka S, Seppänen T, Toivanen J, Partala J, Väyrynen E, Lehtihalmes M, Mattila P, Miettunen, J, Heikkilä J (2010) The multi-technological analyses of learning in the future classroom: Preliminary observatory apparatus. In: Martens A, Tavangarian D, Urban B, Hambach S (eds) 3rd International eLBa Science Conference. Fraunhofer Verlag, pp 60–66

  26. Soleymani M, Lichtenauer J, Pun T, Pantic M (2012) A multimodal database for affect recognition and implicit tagging. IEEE Trans Affect Comput 3(1):42–55

    Article  Google Scholar 

  27. Stevens R, Cherry G, Fournier J (2002) Video traces: Rich media annotations for teaching and learning. In: Proc. Conference on Computer Supported Collaborative Learning

  28. Sun X, Foote J, Kimber D, Manjunath B (2001) Panoramic video capturing and compressed domain virtual camera control. In: Proc. ACM Int. Conf. Multimedia, pp 329–347. doi:10.1145/500141.500191

  29. Sun X, Foote J, Kimber D (2005) Region of interest extraction and virtual camera control based on panoramic video capturing. IEEE Trans Multimedia 7:981–990

    Article  Google Scholar 

  30. Teachscape (1999) Teachscape homepage. Online., http://www.teachscape.com/, accessed April 25, 2014

  31. Transana (2005) Qualitative analysis of software for video and audio data. Online, http://www.transana.org, accessed January 12, 2014

  32. Väyrynen E, Keränen H, Seppänen T, Toivanen J (2005) Performance of f0tool: A new speech analysis software for analyzing large speech data sets. In: Proc 2nd Baltic Conference on Human Language Technologies, Tallinn, Estonia

  33. Väyrynen E, Kortelainen J, Seppänen T (2013) Classifier-based learning of nonlinear feature manifold for visualization of emotional speech prosody. IEEE Trans Affect Comput 4:47–56. doi:10.1109/T-AFFC.2012.35

    Article  Google Scholar 

  34. Viola P, Jones M (2004) Robust real-time face detection. Int J Comput Vis 57(2):137–154. doi:10.1023/B:VISI.0000013087.49260.fb

    Article  Google Scholar 

  35. Wong WK, Poh YC, Loo CK, Lim WS (2010) Wireless webcam based omnidirectional health care surveillance system. In: Second Int. Conf. Computer Research and Development, Second Int. Conf. Computer Research and Development. doi:10.1109/ICCRD.2010.178

  36. Zhao G, Pietikäinen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell 29(6):915–928. doi:10.1109/TPAMI.2007.1110

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Faculty of Information Technology and Electrical Engineering, University of Oulu, Oulu, Finland

    Anja Keskinarkaus, Magda Laszlo, Ilkka Juuso, Eero Väyrynen & Tapio Seppänen

  2. Center for Machine Vision Research, Faculty of Information Technology and Electrical Engineering, University of Oulu, Oulu, Finland

    Sami Huttunen, Jukka Holappa & Janne Heikkilä

  3. Research Unit of Psychology, Learning Research Laboratory (LearnLab), Faculty of Education, University of Oulu, Oulu, Finland

    Antti Siipo & Seppo Laukka

  4. Logopedics, Faculty of Humanities, University of Oulu, Oulu, Finland

    Matti Lehtihalmes

Authors
  1. Anja Keskinarkaus
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sami Huttunen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Antti Siipo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jukka Holappa
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Magda Laszlo
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ilkka Juuso
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Eero Väyrynen
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Janne Heikkilä
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Matti Lehtihalmes
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Tapio Seppänen
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Seppo Laukka
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sami Huttunen.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Keskinarkaus, A., Huttunen, S., Siipo, A. et al. MORE – a multimodal observation and analysis system for social interaction research. Multimed Tools Appl 75, 6321–6345 (2016). https://doi.org/10.1007/s11042-015-2574-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-015-2574-9

Keywords

Search

Navigation