Skip to main content
SpringerLink
Log in

User Experience (re)Design and Evaluation of a Self-Guided, Mobile Health App for Adolescents with Mild Traumatic Brain Injury

  • Published:
Journal of Formative Design in Learning Aims and scope Submit manuscript

Abstract

Mild traumatic brain injury (mTBI) is a significant cause of morbidity for adolescents. Currently, there is a lack of evidence-based interventions to address common sequelae of mTBI. To address this gap, we designed a program to promote recovery for adolescents following mTBI. Preliminary testing of the Self-Monitoring Activity Regulation and Relaxation Treatment (SMART) program demonstrated good usability but indicated a need for modifications to the program. The SMART application was redesigned with the addition of more interactive and gamified components. Content was also reframed to specifically target and engage adolescents with mTBI. We describe the usability evaluation of the updated SMART application. Children aged 11–18 years diagnosed with mTBI who were 1 to 6 months post mTBI were recruited to participate in a 1-–2-h usability session in which they thought aloud and responded to targeted usability-related questions during their interaction with SMART. After completing the session, participants rated their usability experience using the System Usability Scale (SUS) and rated the overall user-friendliness of the program. Participants’ responses during the session were qualitatively coded and analyzed. Six adolescents participated in a usability session (average age = 13.7 years). On the SUS, participants rated the program as highly usable (M = 85.6, SD = 3.24). They also had overwhelmingly positive feedback regarding the content, design, and structure of the program. Overall, findings suggest that the redesigned SMART program was usable, acceptable, and relevant to adolescents with mTBI. Based on adolescents’ feedback, additional modifications were made before the program undergoes efficacy testing.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  • Al Sayegh, A., Sandford, D., & Carson, A. J. (2010). Psychological approaches to treatment of postconcussion syndrome: a systematic review. Journal of Neurology, Neurosurgery & Psychiatry, 81(10), 1128–1134.

    Article  Google Scholar 

  • Anderson, M., & Jiang, J. (2018). Teens, social media & technology 2018. Pew Research Center, 31, 2018.

    Google Scholar 

  • Andersson, G., & Titov, N. (2014). Advantages and limitations of Internet-based interventions for common mental disorders. World Psychiatry, 13(1), 4–11.

    Article  Google Scholar 

  • Babcock, L., Byczkowski, T., Wade, S. L., Ho, M., Mookerjee, S., & Bazarian, J. J. (n.d.). Predicting postconcussion syndrome after mild traumatic brain injury in children and adolescents who present to the emergency department. JAMA Pediatrics, 2(156–161). https://doi.org/10.1001/jamapediatrics.2013.434.

  • Baker, J. G., Leddy, J. J., Darling, S. R., Shucard, J., Makdissi, M., & Willer, B. S. (2016). Gender differences in recovery from sports-related concussion in adolescents. Clinical Pediatrics, 55(8), 771–775.

    Article  Google Scholar 

  • Bangor, A., Kortum, P., & Miller, J. (2009). Determining what individual SUS scores mean: adding an adjective rating scale (Vol. 4).

  • Bangor, A., Kortum, P. T., & Miller, J. T. (2008). An empirical evaluation of the system usability scale. International Journal of Human–Computer Interaction, 24(6), 574–594. https://doi.org/10.1080/10447310802205776.

    Article  Google Scholar 

  • Bath, E., Tolou-Shams, M., & Farabee, D. (2018). Mobile health (Mhealth): Building the case for adapting emerging technologies for justice-involved youth. Journal of the American Academy of Child & Adolescent Psychiatry, 57(12), 903–905. https://doi.org/10.1016/j.jaac.2018.08.012.

    Article  Google Scholar 

  • Bevan, N., Barnum, C., Cockton, G., Nielsen, J., Spool, J., & Wixon, D. (2003). The magic number 5: is it enough for web testing? CHI’03 Extended Abstracts on Human Factors in Computing Systems, 698–699. https://doi.org/10.1145/765891.765936.

  • Brooke, J. (1986). SUS - a quick and dirty usability scale. Digital Equipment Corporation.

  • Bull, S., Devine, S., Schmiege, S. J., Pickard, L., Campbell, J., & Shlay, J. C. (2016). Text messaging, teen outreach program, and sexual health behavior: a cluster randomized trial. American Journal of Public Health, 106(Suppl 1), S117–S124. https://doi.org/10.2105/AJPH.2016.303363.

    Article  Google Scholar 

  • Cancelliere, C., Kristman, V. L., Cassidy, J. D., Hincapié, C. A., Côté, P., Boyle, E., et al. (2014). Systematic review of return to work after mild traumatic brain injury: results of the international collaboration on mild traumatic brain injury prognosis. Archives of Physical Medicine and Rehabilitation, 95(3), S201–S209. https://doi.org/10.1016/j.apmr.2013.10.010.

    Article  Google Scholar 

  • Cassidy, J. D., Carroll, L., Peloso, P., Borg, J., Von Holst, H., Holm, L., & Coronado, V. (2004). Incidence, risk factors and prevention of mild traumatic brain injury: results of the WHO Collaborating Centre Task. Force on Mild Traumatic Brain Injury. Journal of Rehabilitation Medicine, 36(0), 28–60.

    Google Scholar 

  • Deterding, S., Dixon, D., Khaled, R., & Nacke, L. (2011). From game design elements to gamefulness: defining gamification. Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments, 9–15. https://doi.org/10.1145/2181037.2181040.

  • Dexheimer, J. W., Kurowski, B. G., Anders, S. H., McClanahan, N., Wade, S. L., & Babcock, L. (2017). Usability evaluation of the SMART application for youth with Mtbi. International Journal of Medical Informatics, 97, 163–170.

    Article  Google Scholar 

  • Emery, C. A., Barlow, K. M., Brooks, B. L., Max, J. E., Villavicencio-Requis, A., Gnanakumar, V., et al. (2016). A systematic review of psychiatric, psychological, and behavioural outcomes following mild traumatic brain injury in children and adolescents. The Canadian Journal of Psychiatry, 61(5), 259–269. https://doi.org/10.1177/0706743716643741.

    Article  Google Scholar 

  • Eysenbach, G. (2001). What is e-health? Journal of Medical Internet Research, 3(2), e20. https://doi.org/10.2196/jmir.3.2.e20.

    Article  Google Scholar 

  • Glang, A. E., Koester, M. C., Beaver, S., Clay, J., & McLaughlin, K. (2010). Online training in sports concussion for youth sports coaches. International Journal of Sports Science & Coaching, 5(1), 1–11.

    Article  Google Scholar 

  • Glang, A. E., Koester, M. C., Chesnutt, J. C., Gioia, G. A., McAvoy, K., Marshall, S., & Gau, J. M. (2015). The effectiveness of a web-based resource in improving postconcussion management in high schools. Journal of Adolescent Health, 56(1), 91–97.

    Article  Google Scholar 

  • Gravel, J., D’Angelo, A., Carrière, B., Crevier, L., Beauchamp, M. H., Chauny, J.-M., & Chaillet, N. (2013). Interventions provided in the acute phase for mild traumatic brain injury: a systematic review. Systematic Reviews, 2(1), 10–1186.

    Article  Google Scholar 

  • International Organization for Standardization (2011) Systems and software engineering – Systems and software Quality Requirements and Evaluation (SQuaRE) – System and software quality models. (n.d.). Retrieved from https://www.iso.org/standard/35733.html

  • Johnson, D., Deterding, S., Kuhn, K.-A., Staneva, A., Stoyanov, S., & Hides, L. (2016). Gamification for health and wellbeing: a systematic review of the literature. Internet Interventions, 6, 89–106. https://doi.org/10.1016/j.invent.2016.10.002.

    Article  Google Scholar 

  • Johnson, W. L., Rickel, J., Stiles, R., & Munro, A. (1998). Integrating pedagogical agents into virtual environments. Presence, 7(6), 523–546.

    Article  Google Scholar 

  • Kontos, A. P., Elbin, R. J., Schatz, P., Covassin, T., Henry, L., Pardini, J., & Collins, M. W. (2012). A revised factor structure for the post-concussion symptom scale: baseline and postconcussion factors. The American Journal of Sports Medicine, 40(10), 2375–2384. https://doi.org/10.1177/0363546512455400.

    Article  Google Scholar 

  • Kraus, J. F., Hsu, P., Schafer, K., & Afifi, A. A. (2014). Sustained outcomes following mild traumatic brain injury: results of a five-emergency department longitudinal study. Brain Injury, 28(10), 1248–1256. https://doi.org/10.3109/02699052.2014.916420.

    Article  Google Scholar 

  • Kurowski, B. G., Wade, S. L., Dexheimer, J. W., Dyas, J., Zhang, N., & Babcock, L. (2016). Feasibility and potential benefits of a web-based intervention delivered acutely after mild traumatic brain onjury in adolescents: a pilot study. Journal of Head Trauma Rehabilitation, 31(6), 369–378.

    Article  Google Scholar 

  • Langlois, J. A., Rutland-Brown, W., & Thomas, K. E. (2004). Traumatic Brain Injury In the United States. Retrieved from http://www.ncdsv.org/images/CDC_TBIintheUSEDVisitsHospitalizationsAndDeaths_2006.pdf

  • Lenhart, A., Duggan, M., Perrin, A., Stepler, R., Rainie, H., & Parker, K. (2015). Teens, Social Media & Technology Overview 2015. Pew Research Center [Internet & American Life Project].

  • Lovell, M. R., Iverson, G. L., Collins, M. W., Podell, K., Johnston, K. M., Pardini, D., et al. (2006). Measurement of symptoms following sports-related concussion: reliability and normative data for the post-concussion scale. Applied Neuropsychology, 13(3), 166–174.

    Article  Google Scholar 

  • Lumba-Brown, A., Yeates, K. O., Sarmiento, K., Breiding, M. J., Haegerich, T. M., Gioia, G. A., et al. (2018). Centers for disease control and prevention guideline on the diagnosis and management of mild traumatic brain injury among Children. JAMA Pediatrics, 172(11), e182853.

    Article  Google Scholar 

  • McCrory, P., Johnston, K., Meeuwisse, W., Aubry, M., Cantu, R., Dvorak, J., et al. (2005). Summary and agreement statement. In Of the 2nd international conference on concussion in sport, Prague, 39(4) (pp. 196–204). https://doi.org/10.1136/bjsm.2005.018614.

    Chapter  Google Scholar 

  • Nezu, A. M., & D’Zurilla, T. J. (2006). Problem-solving therapy: a positive approach to clinical intervention. Springer Publishing Company.

  • Nielsen, J. (1995). Severity ratings for usability problems. Papers and Essays, 54, 1–2.

    Google Scholar 

  • Nielsen, J. (2000). Why you only need to test with 5 users. Retrieved from https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/.

  • Nowacki, R., van Eldik, N., Eikens, M., Roijen, R., Haga, N., Schott, D., & Wennekes, M. (2017). Evaluation of a follow-up program for mild traumatic brain injury in schoolchildren. European Journal of Paediatric Neurology, 21(2), 382–387.

    Article  Google Scholar 

  • O’Brien, K. H. M., Wyman Battalen, A., Sellers, C. M., Spirito, A., Yen, S., Maneta, E., … Braciszewski, J. M. (2019). An mHealth approach to extend a brief intervention for adolescent alcohol use and suicidal behavior: qualitative analyses of adolescent and parent feedback. Journal of Technology in Human Services.https://doi.org/10.1080/15228835.2018.1561347.

  • Oh, H., Rizo, C., Enkin, M., & Jadad, A. (2005). What is eHealth: a systematic review of published definitions. Journal of Medical Internet Research, 7(1), e1. https://doi.org/10.2196/jmir.7.1.e1.

    Article  Google Scholar 

  • Organization, W. H. (2010). Telemedicine: opportunities and developments in member states. Report on the second global survey on eHealth. World Health Organization.

  • Papoutsis, J., Stargatt, R., & Catroppa, C. (2014). Long-term executive functioning outcomes for complicated and uncomplicated mild traumatic brain injury sustained in early childhood. Developmental Neuropsychology, 39(8), 638–645. https://doi.org/10.1080/87565641.2014.979926.

    Article  Google Scholar 

  • Ponsford, J., Willmott, C., Rothwell, A., Cameron, P., Ayton, G., Nelms, R., & Ng, K. (2001). Impact of early intervention on outcome after mild traumatic brain injury in children. Pediatrics, 108(6), 1297–1303.

    Article  Google Scholar 

  • Røe, C., Sveen, U., Alvsåker, K., & Bautz-Holter, E. (2009). Post-concussion symptoms after mild traumatic brain injury: influence of demographic factors and injury severity in a 1-year cohort study. Disability and Rehabilitation: An International, Multidisciplinary Journal, 31(15), 1235–1243. https://doi.org/10.1080/09638280802532720.

    Article  Google Scholar 

  • Sauro, J., & Lewis, J. R. (2012). Standardized usability questionnaires. In J. Sauro & J. R. Lewis (Eds.), Quantifying the User Experience (Vol. Vol. 8). Boston: Morgan Kaufmann.

    Google Scholar 

  • Scheenen, M. E., Spikman, J. M., de Koning, M. E., van der Horn, H. J., Roks, G., Hageman, G., & van der Naalt, J. (2017). Patients “at risk” of suffering from persistent complaints after mild traumatic brain injury: the role of coping, mood disorders, and post-traumatic stress. Journal of Neurotrauma, 34(1), 31–37. https://doi.org/10.1089/neu.2015.4381.

    Article  Google Scholar 

  • Schwartz, L. A., Psihogios, A. M., Henry-Moss, D., Daniel, L. C., Ver Hoeve, E. S., Velazquez-Martin, B., et al. (2019). Iterative development of a tailored Mhealth intervention for adolescent and young adult survivors of childhood Cancer. Clinical Practice in Pediatric Psychology, 7(1), 31–43. https://doi.org/10.1037/cpp0000272.

    Article  Google Scholar 

  • Silverberg, N. D., & Iverson, G. L. (2013). Is rest after concussion “the best medicine?”: Recommendations for activity resumption following concussion in athletes, civilians, and military service members. The Journal of Head Trauma Rehabilitation, 28(4), 250–259. https://doi.org/10.1097/HTR.0b013e31825ad658.

    Article  Google Scholar 

  • Taylor, C. A., Bell, J. M., Breiding, M. J., & Xu, L. (2017). Traumatic brain injury–related emergency department visits, hospitalizations, and deaths—United States, 2007 and 2013. MMWR. Surveillance Summaries, 66(9), 1–16.

    Article  Google Scholar 

  • Wade, S. L., Walz, N. C., Carey, J., McMullen, K. M., Cass, J., Mark, E., & Yeates, K. O. (2011). Effect on behavior problems of teen online problem-solving for adolescent traumatic brain injury. Pediatrics, 128(4), e947–e953. https://doi.org/10.1542/peds.2010-3721.

    Article  Google Scholar 

  • Wade, S. L., Walz, N. C., Carey, J., McMullen, K. M., Cass, J., Mark, E., & Yeates, K. O. (2012). A randomized trial of teen online problem solving: efficacy in improving caregiver outcomes after brain injury. Health Psychology: Official Journal of the Division of Health Psychology, American Psychological Association, 31(6), 767–776. https://doi.org/10.1037/a0028440.

    Article  Google Scholar 

  • Wade, S. L., Wolfe, C. R., Brown, T. M., & Pestian, J. P. (2005). Can a Web-based family problem-solving intervention work for children with traumatic brain injury? Rehabilitation Psychology, 50(4), 337–345. https://doi.org/10.1037/0090-5550.50.4.337.

    Article  Google Scholar 

  • Wade, S. L., Taylor, H. G., Yeates, K. O., Kirkwood, M., Zang, H., McNally, K., & Zhang, N. (2018). Online problem solving for adolescent brain injury: a randomized trial of 2 approaches. Journal of Developmental and Behavioral Pediatrics, 39(2), 154–162. https://doi.org/10.1097/DBP.0000000000000519.

    Article  Google Scholar 

  • Woodrome, S. E., Yeates, K. O., Taylor, H. G., Rusin, J., Bangert, B., Dietrich, A., & Wright, M. (2011). Coping strategies as a predictor of post-concussive symptoms in children with mild traumatic brain injury versus mild orthopedic injury. Journal of the International Neuropsychological Society, 17(2), 317–326. https://doi.org/10.1017/S1355617710001700.

    Article  Google Scholar 

  • World Health Organization. (2011a). mHealth: new horizons for health through mobile technologies. Retrieved from http://www.who.int/goe/publications/goe_mhealth_web.pdf

  • World Health Organization. (2011b). Second global survey on eHealth. Retrieved from http://www.who.int/goe/publications/goe_mhealth_web.pdf.

  • Worthen-Chaudhari, L., McGonigal, J., Logan, K., Bockbrader, M. A., Yeates, K. O., & Mysiw, W. J. (2017). Reducing concussion symptoms among teenage youth: evaluation of a mobile health app. Brain Injury, 31(10), 1279–1286.

    Article  Google Scholar 

  • Ybarra, M. L., Prescott, T., Mustanski, B., Parsons, J., & Bull, S. S. (2019). Feasibility, acceptability, and process indicators for Guy2Guy, an mHealth HIV prevention program for sexual minority adolescent boys. The Journal of Adolescent Health: Official Publication of the Society for Adolescent Medicine, 65(3), 417–422. https://doi.org/10.1016/j.jadohealth.2019.04.025.

    Article  Google Scholar 

  • Zemek, R., Barrowman, N., Freedman, S. B., Gravel, J., Gagnon, I., McGahern, C., et al. (2016). Clinical risk score for persistent postconcussion symptoms among children with acute concussion in the ED. JAMA, 315(10), 1014. https://doi.org/10.1001/jama.2016.1203.

    Article  Google Scholar 

  • Zhang, N., Kaizar, E. E., Narad, M. E., Kurowski, B. G., Yeates, K. O., Taylor, H. G., & Wade, S. L. (2019). Examination of injury, host, and social-environmental moderators of online family problem solving treatment efficacy for pediatric traumatic brain injury using an individual participant data meta-analytic approach. Journal of Neurotrauma, 36(7), 1147–1155. https://doi.org/10.1089/neu.2018.5885.

    Article  Google Scholar 

Download references

Acknowledgments

We would like to thank Stephanie Boyd, Noura Barzai, Aimee Miley, and Jessica Aguilar for their contributions to the study and manuscript.

Funding

This work was funded by the National Institute of Child Health and Human Development [R21HD087844].

Author information

Authors and Affiliations

  1. University of Florida, Gainesville, USA

    Matthew Schmidt

  2. Cincinnati Children’s Hospital Medical Center, Cincinnati, USA

    Allison P. Fisher, Lynn Babcock, Brad G. Kurowski & Shari L. Wade

  3. Wright State University, Dayton, USA

    Joshua Sensenbaugh & Brandt Ling

  4. University of Cincinnati, Cincinnati, USA

    Carmen Rietta

Authors
  1. Matthew Schmidt
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Allison P. Fisher
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Joshua Sensenbaugh
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Brandt Ling
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Carmen Rietta
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Lynn Babcock
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Brad G. Kurowski
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Shari L. Wade
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Conceptualization, Matthew Schmidt, Lynn Babcock, Brad Kurowski, and Shari Wade; methodology, Matthew Schmidt, Lynn Babcock, Brad Kurowski, and Shari Wade; software, Matthew Schmidt; validation, Matthew Schmidt, Allison Fisher; formal analysis, Matthew Schmidt, Allison Fisher, Brandt Ling, Joshua Sensenbaugh, and Carmen Rietta; investigation, Matthew Schmidt, Carmen Rietta, Lynn Babcock, and Brad Kurowski; resources, Matthew Schmidt and Shari Wade; data curation, Allison Fisher, Brandt Ling, Joshua Sensenbaugh, and Carmen Rietta; writing—original draft preparation, Allison Fisher, Matthew Schmidt, Brandt Ling, and Joshua Sensenbaugh; writing—review and editing, all authors; visualization, Matthew Schmidt; supervision, Matthew Schmidt and Shari Wade; project administration, Shari Wade and Lynn Babcock; funding acquisition, Shari Wade, Lynn Babcock, Matthew Schmidt, and Brad Kurowski.

Corresponding author

Correspondence to Matthew Schmidt.

Ethics declarations

This project was approved by our Institutional Review Board.

Conflict of Interest

The authors declare no conflict of interest.

Disclaimer

The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Schmidt, M., Fisher, A.P., Sensenbaugh, J. et al. User Experience (re)Design and Evaluation of a Self-Guided, Mobile Health App for Adolescents with Mild Traumatic Brain Injury. J Form Des Learn 4, 51–64 (2020). https://doi.org/10.1007/s41686-019-00038-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s41686-019-00038-x

Keywords

Search

Navigation