Advertisement

Behavior Research Methods

, Volume 51, Issue 1, pp 361–383 | Cite as

Quantification of nonverbal synchrony using linear time series analysis methods: Lack of convergent validity and evidence for facets of synchrony

  • Désirée SchoenherrEmail author
  • Jane Paulick
  • Susanne Worrack
  • Bernhard M. Strauss
  • Julian A. Rubel
  • Brian Schwartz
  • Anne-Katharina Deisenhofer
  • Wolfgang Lutz
  • Ulrich Stangier
  • Uwe Altmann
Article

Abstract

Nonverbal synchrony describes coordination of the nonverbal behavior of two interacting partners. Additionally, it seems to be important in human interactions, such as during psychotherapy. Currently, there are several options for the automated determination of synchrony based on linear time series analysis methods (TSAMs). However, investigations into whether the different methods measure the same construct have been missing. In this study, N = 84 patient–therapist dyads were videotaped during psychotherapy sessions. Motion energy analysis was used to assess body movements. We applied seven different TSAMs and recorded multiple output scores (average synchrony, maximum synchrony, and frequency of synchrony; in total, N = 16 scores). Convergent validity was examined using correlations of the output scores and exploratory factor analysis. Additionally, two criterion-based validations were conducted: investigations of concordant validity with a more generalized nonlinear method, and of the predictive validity of the synchrony scores for improvement in interpersonal problems at the end of therapy. We found that the synchrony measures only partially correlated with each other. The factor analysis did not support a common-factor model. A three-factor model with a second-order synchrony variable showed the best fit for eight of the selected synchrony scores. Only some synchrony scores were able to predict improvement at the end of therapy. We concluded that the considered TSAMs do not measure the same synchrony construct, but different facets of synchrony: the strength of synchrony of the total interaction, the strength of synchrony during synchronization intervals, and the frequency of synchrony.

Keywords

Nonverbal behavior Movement synchrony Motion energy analysis Time series analysis Convergent validity 

Notes

Author note

This study was funded by the German Research Foundation (DFG project “Timing of Nonverbal Patient-Therapist-Interaction and Therapeutic Success of Social Phobic Patients (TIMPATHIN),” funding number GZ: STR 306/28-1 and LU 660/8-1) and used video recordings from the SOPHONET treatment study, funded by the German Federal Ministry of Education and Research (BMBF, FKZ 01GV0607). We cordially thank U. Willutzki, S. Herpertz, J. Hoyer, P. Joraschky, W. Hiller, E. Leibing, M. Beutel, and F. Leichsenring for providing video recordings for the project. We thank K. Boyle, a native speaker of English, for proof reading.

References

  1. Allsop, J. S., Vaitkus, T., Marie, D., & Miles, L. K. (2016). Coordination and collective performance: Cooperative goals boost interpersonal synchrony and task outcomes. Frontiers in Psychology, 7, 1462.  https://doi.org/10.3389/fpsyg.2016.01462 CrossRefGoogle Scholar
  2. Altmann, U. (2010). Interrater-Reliabilität = 1 in Videostudien? Automatisierte Erhebung von Nonverbalität in einem Experiment zur Kooperation von Schülern [Automated coding of nonverbal behavior in an experiment on the cooperation of students]. Beiträge zur Erziehungswissenschaftliche Forschung–Nachhaltige Bildung, 5, 261–267.Google Scholar
  3. Altmann, U. (2011). Investigation of movement synchrony using windowed cross-lagged regression. In A. Esposito, A. Vinciarelli, K. Vicsi, C. Pelachaud, & A. Nijholt (Eds.), Analysis of verbal and nonverbal communication and enactment: The processing issues (pp. 335-345). Berlin, Germany: Springer.  https://doi.org/10.1007/978-3-642-25775-9_31 CrossRefGoogle Scholar
  4. Altmann, U. (2013). Synchronisation nonverbalen Verhaltens: Weiterentwicklung und Anwendung zeitreihenanalytischer Identifikationsverfahren. Berlin, Germany: Springer.CrossRefGoogle Scholar
  5. Altmann, U., Schoenherr, D., Paulick, J., Deisenhofer, A.-K., Schwartz, B., Rubel, J., ... Strauss, B. M. (2018). Timing of nonverbal patient–therapist interaction and therapeutic success of social phobic patients. Manuscript in preparation.Google Scholar
  6. Altmann, U., Schoenherr, D., Paulick, J., Knitter, L., Worrack, S., Schiefele, A.-K., ... Strauss, B. M. (submitted). Introduction, practical guide, and validation study for measuring body movements using motion energy analysis. Manuscript submitted for publication.Google Scholar
  7. Bailenson, J. N., & Yee, N. (2005). Digital chameleons: Automatic assimilation of nonverbal gestures in immersive virtual environments. Psychological Science, 16, 814–819.  https://doi.org/10.1111/j.1467-9280.2005.01619.x CrossRefGoogle Scholar
  8. Beck, A. T., Ward, C. H., Mendelson, M., Mock, J., & Erbaugh, J. (1961). An inventory for measuring depression. Archives of General Psychiatry, 4, 561–571.  https://doi.org/10.1001/archpsyc.1961.01710120031004 CrossRefGoogle Scholar
  9. Bernieri, F. J. (1988). Coordinated movement and rapport in teacher–student interactions. Journal of Nonverbal Behavior, 12, 120–138.  https://doi.org/10.1007/BF00986930 CrossRefGoogle Scholar
  10. Bernieri, F. J., & Rosenthal, R. (1991). Interpersonal coordination: Behavior matching and interactional synchrony. In R. S. Feldman & B. Rimé (Eds.), Fundamentals of nonverbal behavior (pp. 401–432). New York: Cambridge University Press.Google Scholar
  11. Bilakhia, S., Petridis, S., Nijholt, A., & Pantic, M. (2015). The MAHNOB Mimicry Database: A database of naturalistic human interactions. Pattern Recognition Letters, 66, 52–61.  https://doi.org/10.1016/j.patrec.2015.03.005 CrossRefGoogle Scholar
  12. Boker, S. M., Rotondo, J. L., Xu, M., & King, K. (2002). Windowed cross-correlation and peak picking for the analysis of variability in the association between behavioral time series. Psychological Methods, 7, 338–355.  https://doi.org/10.1037/1082-989X.7.3.338 CrossRefGoogle Scholar
  13. Bouthillier, D., Julien, D., Dubé, M., Bélanger, I., & Hamelin, M. (2002). Predictive validity of adult attachment measures in relation to emotion regulation behaviors in marital interactions. Journal of Adult Development, 9, 291–305.  https://doi.org/10.1023/A:1020291011587 CrossRefGoogle Scholar
  14. Brown, T. A., Barlow, D. H., & Di Nardo, P. A. (1994). Anxiety disorders interview schedule for DSM-IV (ADIS-IV): Client interview schedule. Albany, NY: Graywind.Google Scholar
  15. Cappella, J. N. (1996). Dynamic coordination of vocal and kinesic behavior in dyadic interaction: Methods, problems, and interpersonal outcomes. In J. H. Watt & C. A. VanLear (Eds.), Dynamic patterns in communication processes (pp. 353–386). Thousand Oaks, CA, US: Sage.Google Scholar
  16. Chartrand, T. L., & Bargh, J. A. (1999). The chameleon effect: The perception–behavior link and social interaction. Journal of Personality and Social Psychology, 76, 893–910.  https://doi.org/10.1037/0022-3514.76.6.893 CrossRefGoogle Scholar
  17. Coco, M. I., & Dale, R. (2014). Cross-recurrence quantification analysis of categorical and continuous time series: An R package. Frontiers in Psychology, 5, 510.  https://doi.org/10.3389/fpsyg.2014.00510 CrossRefGoogle Scholar
  18. Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281–302.  https://doi.org/10.1037/h0040957 CrossRefGoogle Scholar
  19. Davis, T. J., Kay, B. A., Kondepudi, D., & Dixon, J. A. (2016). Spontaneous interentity coordination in a dissipative structure. Ecological Psychology, 28, 23–36.  https://doi.org/10.1080/10407413.2016.1121737 CrossRefGoogle Scholar
  20. Dean, D. J., Samson, A. T., Newberry, R., & Mittal, V. A. (2018). Motion energy analysis reveals altered body movement in youth at risk for psychosis. Schizophrenia Research, 200, 35–41.  https://doi.org/10.1016/j.schres.2017.05.035 CrossRefGoogle Scholar
  21. Dean, R. T., & Dunsmuir, W. T. M. (2016). Dangers and uses of cross-correlation in analyzing time series in perception, performance, movement, and neuroscience: The importance of constructing transfer function autoregressive models. Behavior Research Methods, 48, 783–802.  https://doi.org/10.3758/s13428-015-0611-2 CrossRefGoogle Scholar
  22. Delaherche, E., & Chetouani, M. (2010, October). Multimodal coordination: Exploring relevant features and measures. Paper presented at the 2nd international Workshop on Social Signal Processing, Florence, Italy.Google Scholar
  23. Delaherche, E., Chetouani, M., Mahdhaoui, A., Saint-Georges, C., Viaux, S., & Cohen, D. (2012). Interpersonal synchrony: A survey of evaluation methods across disciplines. IEEE Transactions on Affective Computing, 3, 349–365.  https://doi.org/10.1109/T-AFFC.2012.12 CrossRefGoogle Scholar
  24. Duran, N. D., & Fusaroli, R. (2017). Conversing with a devil’s advocate: Interpersonal coordination in deception and disagreement. PLoS ONE, 12, e0178140.  https://doi.org/10.1371/journal.pone.0178140 CrossRefGoogle Scholar
  25. Dutschke, L. L., Stegmayer, K., Ramseyer, F., Bohlhalter, S., Vanbellingen, T., Strik, W., & Walther, S. (2018). Gesture impairments in schizophrenia are linked to increased movement and prolonged motor planning and execution. Schizophrenia Research, 200, 42–49.  https://doi.org/10.1016/j.schres.2017.07.012 CrossRefGoogle Scholar
  26. Feldman, R. (2007). Parent–infant synchrony and the construction of shared timing: Physiological precursors, developmental outcomes, and risk conditions. Journal of Child Psychology and Psychiatry, 48, 329–354.  https://doi.org/10.1111/j.1469-7610.2006.01701.x CrossRefGoogle Scholar
  27. Fogel, A. (1993). Two principles of communication: Co-regulation and framing. In J. Nadel & L. Camaioni (Eds.), New perspectives in early communicative development (pp. 9–22). London, UK: Routledge.Google Scholar
  28. Galbusera, L., Finn, M. T., & Fuchs, T. (2018). Interactional synchrony and negative symptoms: An outcome study of body-oriented psychotherapy for schizophrenia. Psychotherapy Research, 28, 457–469.  https://doi.org/10.1080/10503307.2016.1216624 CrossRefGoogle Scholar
  29. Gottman, J. M., & Ringland, J. T. (1981). The analysis of dominance and bidirectionality in social development. Child Development, 52, 393–412.  https://doi.org/10.2307/1129157 CrossRefGoogle Scholar
  30. Grammer, K., Honda, M., Juette, A., & Schmitt, A. (1999). Fuzziness of nonverbal courtship communication unblurred by motion energy detection. Journal of Personality and Social Psychology, 77, 487–508.  https://doi.org/10.1037/0022-3514.77.3.487 CrossRefGoogle Scholar
  31. Grammer, K., Kruck, K. B., & Magnusson, M. S. (1998). The courtship dance: Patterns of nonverbal synchronization in opposite-sex encounters. Journal of Nonverbal Behavior, 22, 3–29.  https://doi.org/10.1023/A:1022986608835 CrossRefGoogle Scholar
  32. Harrist, A. W., & Waugh, R. M. (2002). Dyadic synchrony: Its structure and function in children’s development. Developmental Review, 22, 555–592.  https://doi.org/10.1016/S0273-2297(02)00500-2 CrossRefGoogle Scholar
  33. Horowitz, L. M., Strauss, B., Thomas, A., & Kordy, H. (2016). IIP-D Inventar zur Erfassung Interpersonaler Probleme—Deutsche Version. [Inventory for the assessment of interpersonel problems—German version] (3rd ed.). Göttingen, Germany: Hogrefe.Google Scholar
  34. Kato, T., Takahashi, E., Sawada, K., Kobayashi, N., Watanabe, T., & Ishh, T. (1983). A computer analysis of infant movements synchronized with adult speech. Pediatric Research, 17, 625–628.  https://doi.org/10.1203/00006450-198308000-00004 CrossRefGoogle Scholar
  35. Kirchmann, H., Fenner, A., & Strauß, B. (2007). Konvergenz des Erwachsenen-Bindungsprototypen-Ratings (EBPR) mit verschiedenen Selbstbeschreibungsinstrumenten zur Erfassung von Bindungsmerkmalen. PPmP—Psychotherapie· Psychosomatik·Medizinische Psychologie, 57, 334–342.  https://doi.org/10.1055/s-2007-970860 Google Scholar
  36. Kleinbub, K. R., & Ramseyer, F. (2018). rMEA: Synchrony in motion energy analysis (MEA) time-series (R package version 1.0.0.9012). https://cran.r-project.org/web/packages/rMEA/index.html
  37. Kupper, Z., Ramseyer, F., Hoffmann, H., Kalbermatten, S., & Tschacher, W. (2010). Video-based quantification of body movement during social interaction indicates the severity of negative symptoms in patients with schizophrenia. Schizophrenia Research, 121, 90–100.  https://doi.org/10.1016/j.schres.2010.03.032 CrossRefGoogle Scholar
  38. Kupper, Z., Ramseyer, F., Hoffmann, H., & Tschacher, W. (2015). Nonverbal synchrony in social interactions of patients with schizophrenia indicates socio-communicative deficits. PLoS ONE, 10, e0145882.  https://doi.org/10.1371/journal.pone.0145882 CrossRefGoogle Scholar
  39. Leichsenring, F., Salzer, S., Beutel, M. E., Herpertz, S., Hiller, W., Hoyer, J., ... Poehlmann, E. (2013). Psychodynamic therapy and cognitive–behavioral therapy in social anxiety disorder: A multicenter randomized controlled trial. American Journal of Psychiatry, 170, 759–767.  https://doi.org/10.1176/appi.ajp.2013.12081125
  40. Leichsenring, F., Salzer, S., Beutel, M. E., Herpertz, S., Hiller, W., Hoyer, J., ... Poehlmann, E. (2014). Long-term outcome of psychodynamic therapy and cognitive–behavioral therapy in social anxiety disorder. American Journal of Psychiatry, 171, 1074–1082.  https://doi.org/10.1176/appi.ajp.2014.13111514
  41. Louwerse, M. M., Dale, R., Bard, E. G., & Jeuniaux, P. (2012). Behavior matching in multimodal communication is synchronized. Cognitive Science, 36, 1404–1426.  https://doi.org/10.1111/j.1551-6709.2012.01269.x CrossRefGoogle Scholar
  42. Manes, S., Nodop, S., Altmann, U., Gawlytta, R., Dinger, U., Dymel, W., ... Petrowski, B. (2016). Social anxiety as a potential mediator of the association between attachment and depression. Journal of Affective Disorders, 205, 264–268.  https://doi.org/10.1016/j.jad.2016.06.060
  43. Marwan, N., & Kurths, J. (2002). Nonlinear analysis of bivariate data with cross recurrence plots. Physics Letters A, 302, 299–307.  https://doi.org/10.1016/S0375-9601(02)01170-2 CrossRefGoogle Scholar
  44. Marwan, N., Romano, M. C., Thiel, M., & Kurths, J. (2007). Recurrence plots for the analysis of complex systems. Physics Reports, 438, 237–329.  https://doi.org/10.1016/j.physrep.2006.11.001 CrossRefGoogle Scholar
  45. Matlab (2016). The MathWorks, Inc., Natick, Massachusetts, United States.Google Scholar
  46. McGarva, A. R., & Warner, R. M. (2003). Attraction and social coordination: Mutual entrainment of vocal activity rhythms. Journal of Psycholinguistic Research, 32, 335–354.  https://doi.org/10.1023/a:1023547703110 CrossRefGoogle Scholar
  47. Mennin, D. S., Fresco, D. M., Heimberg, R. G., Schneier, F. R., Davies, S. O., & Liebowitz, M. R. (2002). Screening for social anxiety disorder in the clinical setting: Using the Liebowitz Social Anxiety Scale. Journal of Anxiety Disorders, 16, 661–673.  https://doi.org/10.1016/S0887-6185(02)00134-2 CrossRefGoogle Scholar
  48. Moulder, R. G., Boker, S. M., Ramseyer, F., & Tschacher, W. (2018). Determining synchrony between behavioral time series: An application of surrogate data generation for establishing falsifiable null-hypotheses. Psychological Methods. Advance online publication.  https://doi.org/10.1037/met0000172
  49. Nagaoka, C., & Komori, M. (2008). Body movement synchrony in psychotherapeutic counseling: A study using the video-based quantification method. IEICE Transactions on Information and Systems, 91, 1634–1640.  https://doi.org/10.1093/ietisy/e91-d.6.1634 CrossRefGoogle Scholar
  50. Nelson, A., Grahe, J., Ramseyer, F., & Serier, K. (2014). Psychological data from an exploration of the rapport/synchrony interplay using motion energy analysis. Journal of Open Psychology Data, 2, e5.  https://doi.org/10.5334/jopd.ae CrossRefGoogle Scholar
  51. Paulick, J., Deisenhofer, A.-K., Ramseyer, F., Tschacher, W., Boyle, K., Rubel, J., & Lutz, W. (2018). Nonverbal synchrony: A new approach to better understand psychotherapeutic processes and drop-out. Journal of Psychotherapy Integration, 28, 367–384.  https://doi.org/10.1037/int0000099 CrossRefGoogle Scholar
  52. Paxton, A. (2015). Coordination: Theoretical, methodological, and experimental perspectives (Doctoral dissertation). University of California, Merced.Google Scholar
  53. Paxton, A., & Dale, R. (2013a). Argument disrupts interpersonal synchrony. Quarterly Journal of Experimental Psychology, 66, 2092–2102.  https://doi.org/10.1080/17470218.2013.853089 CrossRefGoogle Scholar
  54. Paxton, A., & Dale, R. (2013b). Frame-differencing methods for measuring bodily synchrony in conversation. Behavior Research Methods, 45, 329–343.  https://doi.org/10.3758/s13428-012-0249-2 CrossRefGoogle Scholar
  55. Ramseyer, F. (2011). Nonverbal synchrony in psychotherapy: Embodiment at the level of the dyad. In W. Tschacher & C. Bergomi (Eds.), The implications of embodiment: Cognition and communication (pp. 193–207). Exeter, UK: Andrews.Google Scholar
  56. Ramseyer, F. (2013, June). Synchronized movement in social interaction. Paper presented at the 2013 Inputs–Outputs Conference: An Interdisciplinary Conference on Engagement in HCI and Performance, Brighton, UK.Google Scholar
  57. Ramseyer, F., & Tschacher, W. (2010). Nonverbal synchrony or random coincidence? How to tell the difference. In A. Esposito (Ed.), Development of multimodal interfaces: Active listening and synchrony (pp. 182–196). Berlin, Germany: Springer.CrossRefGoogle Scholar
  58. Ramseyer, F., & Tschacher, W. (2011). Nonverbal synchrony in psychotherapy: Coordinated body movement reflects relationship quality and outcome. Journal of Consulting and Clinical Psychology, 79, 284–295.  https://doi.org/10.1037/a0023419 CrossRefGoogle Scholar
  59. Ramseyer, F., & Tschacher, W. (2016). Movement coordination in psychotherapy: Synchrony of hand movements is associated with session outcome. A single-case study. Nonlinear Dynamics, Psychology, and Life Sciences, 20, 145–166.Google Scholar
  60. Robinson, J. W., Herman, A., & Kaplan, B. J. (1982). Autonomic responses correlate with counselor–client empathy. Journal of Counseling Psychology, 29, 195–198.  https://doi.org/10.1037/0022-0167.29.2.195 CrossRefGoogle Scholar
  61. Roisman, G. I., Holland, A., Fortuna, K., Fraley, R. C., Clausell, E., & Clarke, A. (2007). The Adult Attachment Interview and self-reports of attachment style: An empirical rapprochement. Journal of Personality and Social Psychology, 92, 678–697.  https://doi.org/10.1037/0022-3514.92.4.678 CrossRefGoogle Scholar
  62. Schönbrodt, F. D., & Perugini, M. (2013). At what sample size do correlations stabilize? Journal of Research in Personality, 47, 609–612.  https://doi.org/10.1016/j.jrp.2013.05.009 CrossRefGoogle Scholar
  63. Shockley, K. (2005). Cross recurrence quantification of interpersonal postural activity. In M. A. Riley & G. C. Van Orden (Eds.), Tutorials in contemporary nonlinear methods for the behavioral sciences (pp. 142–177). Arlington, VA: National Science Foundation.Google Scholar
  64. Shockley, K., Richardson, D. C., & Dale, R. (2009). Conversation and coordinative structures. Topics in Cognitive Science, 1, 305–319.  https://doi.org/10.1111/j.1756-8765.2009.01021.x CrossRefGoogle Scholar
  65. Tashman, L. J. (2000). Out-of-sample tests of forecasting accuracy: An analysis and review. International Journal of Forecasting, 16, 437–450.  https://doi.org/10.1016/S0169-2070(00)00065-0 CrossRefGoogle Scholar
  66. Taylor, R. (1990). Interpretation of the correlation coefficient: A basic review. Journal of Diagnostic Medical Sonography, 6, 35–39.  https://doi.org/10.1177/875647939000600106 CrossRefGoogle Scholar
  67. Tronick, E. D., Als, H., & Brazelton, T. B. (1977). Mutuality in mother–infant interaction. Journal of Communication, 27, 74–79.  https://doi.org/10.1111/j.1460-2466.1977.tb01829.x CrossRefGoogle Scholar
  68. Tschacher, W., Rees, G. M., & Ramseyer, F. (2014). Nonverbal synchrony and affect in dyadic interactions. Frontiers in Psychology, 5, 1323.  https://doi.org/10.3389/fpsyg.2014.01323 CrossRefGoogle Scholar
  69. Watanabe, T. (1983). A study of motion–voice synchronization. Bulletin of JSME, 26, 2244–2250.  https://doi.org/10.1299/jsme1958.26.2244 CrossRefGoogle Scholar
  70. Watanabe, T. (1987). A quantitative analyzing method for the synchrony between a speaker’s voice and a listener’s nodding in interpersonal communication. JSME International Journal, 30, 2040–2043.  https://doi.org/10.1299/jsme1987.30.2040 CrossRefGoogle Scholar
  71. Wenzel, A., Graff-Dolezal, J., Macho, M., & Brendle, J. R. (2005). Communication and social skills in socially anxious and nonanxious individuals in the context of romantic relationships. Behaviour Research and Therapy, 43, 505–519.  https://doi.org/10.1016/j.brat.2004.03.010 CrossRefGoogle Scholar
  72. Wiltshire, T. (2015). Team interaction dynamics during collaborative problem solving (Doctoral dissertation). University of Central Florida, Orlando, FL.Google Scholar
  73. Wittchen, H.-U., Wunderlich, U., Gruschwitz, S., & Zaudig, M. (1997). SKID-I: Strukturiertes klinisches Interview für DSM-IV. Göttingen, Germany: Hogrefe.Google Scholar

Copyright information

© Psychonomic Society, Inc. 2018

Authors and Affiliations

  • Désirée Schoenherr
    • 1
    Email author
  • Jane Paulick
    • 2
  • Susanne Worrack
    • 1
  • Bernhard M. Strauss
    • 1
  • Julian A. Rubel
    • 2
  • Brian Schwartz
    • 2
  • Anne-Katharina Deisenhofer
    • 2
  • Wolfgang Lutz
    • 2
  • Ulrich Stangier
    • 3
  • Uwe Altmann
    • 1
  1. 1.Institute of Psychosocial Medicine and PsychotherapyUniversity Hospital JenaJenaGermany
  2. 2.Department of Clinical Psychology and PsychotherapyTrier UniversityTrierGermany
  3. 3.Department of Clinical Psychology and PsychotherapyGoethe University Frankfurt/MainFrankfurt/MainGermany

Personalised recommendations