Empirical Software Engineering

, Volume 22, Issue 1, pp 134–174 | Cite as

Extracting and analyzing time-series HCI data from screen-captured task videos

  • Lingfeng Bao
  • Jing Li
  • Zhenchang Xing
  • Xinyu Wang
  • Xin Xia
  • Bo Zhou


Recent years have witnessed the increasing emphasis on human aspects in software engineering research and practices. Our survey of existing studies on human aspects in software engineering shows that screen-captured videos have been widely used to record developers’ behavior and study software engineering practices. The screen-captured videos provide direct information about which software tools the developers interact with and which content they access or generate during the task. Such Human-Computer Interaction (HCI) data can help researchers and practitioners understand and improve software engineering practices from human perspective. However, extracting time-series HCI data from screen-captured task videos requires manual transcribing and coding of videos, which is tedious and error-prone. In this paper we report a formative study to understand the challenges in manually transcribing screen-captured videos into time-series HCI data. We then present a computer-vision based video scraping technique to automatically extract time-series HCI data from screen-captured videos. We also present a case study of our scvRipper tool that implements the video scraping technique using 29-hours of task videos of 20 developers in two development tasks. The case study not only evaluates the runtime performance and robustness of the tool, but also performs a detailed quantitative analysis of the tool’s ability to extract time-series HCI data from screen-captured task videos. We also study the developer’s micro-level behavior patterns in software development from the quantitative analysis.


Screen-captured video Video scraping HCI data Online search behavior 


  1. Ammar N, Abi-Antoun M (2012) Empirical evaluation of diagrams of the run-time structure for coding tasks. In: Proceedings of the WCRE, pp 367–376Google Scholar
  2. Bandura A (1986) Social foundations of thought and action: a social cognitive theory, vol 1, p 617Google Scholar
  3. Banovic N, Grossman T, Matejka J, Fitzmaurice G (2012) Waken: reverse engineering usage information and interface structure from software videos. In: Proceedings of the UIST, pp 83–92Google Scholar
  4. Bao L, Ye D, Xing Z, Xia X (2015a) ActivitySpace: a remembrance framework to support interapplication information needs. In: Proceedings 30th IEEE/ACM international conference on automated software engineeringGoogle Scholar
  5. Bao L, Li J, Xing Z, Wang X, Zhou B (2015b) Reverse engineering time-series interaction data from screen-captured videos. In: Proceedings of the SANER, pp 399–408Google Scholar
  6. Bao L, Li J, Xing Z, Wang X, Zhou B (2015c) scvRipper: video scraping tool for modeling developers behavior using interaction data. In: Proceedings of the ICSE, pp 673–676Google Scholar
  7. Bao L, Xing Z, Wang X, Zhou B (2015d) Tracking and analyzing cross-cutting activities in developers’ daily work. In: Proceedings of the 30th IEEE/ACM international conference on automated software engineeringGoogle Scholar
  8. Barabasi AL (2005) The origin of bursts and heavy tails in human dynamics. Nature 435(7039):207–211CrossRefGoogle Scholar
  9. Bateman S, Teevan J, White RW (2012) The search dashboard: how reflection and comparison impact search behavior. In: Proceedings of the CHI, 1785Google Scholar
  10. Bay H, Ess A, Tuytelaars T, Van Gool L (2008) Speeded-up robust features (surf). Comp Vision Image Underst 110(3):346–359CrossRefGoogle Scholar
  11. Bordes A, Gabrilovich E (2014) Constructing and mining web-scale knowledge graphs: KDD 2014 tutorial. In: Proceedings of the KDD, p 1967Google Scholar
  12. Brade K, Guzdial M, Steckel M, Soloway E (1992) Whorf: A visualization tool for software maintenance. In: Proceedings 1992 IEEE workshop on visual languages, pp 148–154Google Scholar
  13. Brandt J, Guo PJ, Lewenstein J, Dontcheva M, Klemmer SR, Francisco S (2009) Two studies of opportunistic programming: interleaving web foraging, learning, and writing code. In: Proceedings of the CHI, pp 1589–1598Google Scholar
  14. Brandt J, Dontcheva M, Weskamp M, Klemmer SR, Francisco S (2010) Example-centric programming: integrating web search into the development environment. In: Proceedings of the CHI, pp 513–522Google Scholar
  15. Brown JS, Collins A, Duguid P (1989) Situated cognition and the culture of learningGoogle Scholar
  16. Canny J (1986) A computational approach to edge detection. IEEE Trans Pattern Anal Mach Intell:679–698Google Scholar
  17. Chang T-H, Yeh T, Miller R (2011) Associating the visual representation of user interfaces with their internal structures and metadata. In: Proceedings of the UIST, pp 245–256Google Scholar
  18. Chang T-H, Yeh T, Miller RC (2010) GUI testing using computer vision. In: Proceeding of the CHI, pp 1535–1544Google Scholar
  19. Colin Cameron A, Windmeijer FA (1997) An r-squared measure of goodness of fit for some common nonlinear regression models. J Econ 77(2):329–342MathSciNetCrossRefMATHGoogle Scholar
  20. Corritore CL, Wiedenbeck S (2000) Direction and scope of comprehension-related activities by procedural and object-oriented programmers: An empirical study. In: Proceedings of the IWPC. IEEE, pp 139– 148Google Scholar
  21. Corritore CL, Wiedenbeck S (2001) An exploratory study of program comprehension strategies of procedural and object-oriented programmers. Int J Hum-Comput St 54(1):1–23CrossRefMATHGoogle Scholar
  22. Dekel U, Herbsleb JD (2009) Reading the documentation of invoked API functions in program comprehension, pp 168–177Google Scholar
  23. Dewan P, Agarwal P, Shroff G, Hegde R (2009) Distributed side-by-side programming. In: Proceedings of the 2009 ICSE workshop on cooperative and human aspects on software engineering, pp 48–55Google Scholar
  24. Dixon M, Fogarty J (2010) Prefab: implementing advanced behaviors using pixel-based reverse engineering of interface structure. In: Proceedings of the CHI, pp 1525–1534Google Scholar
  25. Duala-Ekoko E, Robillard MP (2012) Asking and answering questions about unfamiliar APIs: an exploratory study. In: Proceedings of the ICSE, pp 266–276Google Scholar
  26. Ester M, Kriegel H-P, Sander J, Xu X (1996) A density-based algorithm for discovering clusters in large spatial databases with noise. In: Proceedings of the KDD, vol 96, pp 226–231Google Scholar
  27. Forsyth DA, Ponce J (2002) Computer vision: a modern approach. Prentice Hall Professional Technical ReferenceGoogle Scholar
  28. Fritz T, Shepherd DC, Kevic K, Snipes W, Bräunlich C (2014) Developers’ code context models for change tasks. In: Proceedings of the FSE, pp 7–18Google Scholar
  29. Gonzalez RC, Woods RE (2002) Digital image processing. Prentice hall Upper Saddle River, NJGoogle Scholar
  30. Greiler M, van Deursen A, Storey M (2012) Test confessions: a study of testing practices for plug-in systems, pp 244–254Google Scholar
  31. Guha R, Guha R, McCool R, McCool R, Miller E, Miller E (2003) Semantic search. In: Proceedings of the WWW, pp 700–709Google Scholar
  32. Hartmann B, Macdougall D, Brandt J, Klemmer SR (2010) What would other programmers do? Suggesting solutions to error messages. In: Proceedings of the CHI, pp 1019–1028Google Scholar
  33. Hilbert DM, Redmiles DF (2000) Extracting usability information from user interface events. ACM Comput Surv 32(4):384–421CrossRefGoogle Scholar
  34. Hundhausen CD, Brown JL, Farley S, Skarpas D (2006) A methodology for analyzing the temporal evolution of novice programs based on semantic components. In: Proceedings of the ACM international computing education research workshop, pp 59–71Google Scholar
  35. Hurst A, Hudson SE, Mankoff J (2010) Automatically identifying targets users interact with during real world tasks. In: Proceedings of the IUI. ACM, pp 11–20Google Scholar
  36. Kim JH, Gunn DV, Schuh E, Phillips B, Pagulayan RJ, Wixon D (2008) Tracking real-time user experience (TRUE): a comprehensive instrumentation solution for complex systems. In: Proceedings of the CHI, pp 443–452Google Scholar
  37. Kimble C, Hildreth PM, Bourdon I (2008) Communities of practice: creating learning environments for educators, vol 1. Information Age PublisherGoogle Scholar
  38. Ko AJ, Myers BA (2004) Designing the whyline: a debugging interface for asking questions about program behavior. In: Proceedings of the CHI, pp 151–158Google Scholar
  39. Ko AJ, Myers BA (2005) A framework and methodology for studying the causes of software errors in programming systems. J Visual Lang Comput 16(1):41–84CrossRefGoogle Scholar
  40. Ko AJ, Aung HH, Myers BA (2005a) Design requirements for more flexible structured editors from a study of programmers’ text editing. In: CHI’05 extended abstracts on human factors in computing systems. ACM, pp 1557–1560Google Scholar
  41. Ko AJ, Aung HH, Myers BA (2005b) Eliciting design requirements for maintenance-oriented IDEs: a detailed study of corrective and perfective maintenance tasks. In: Proceedings of the ICSE, pp 126–135Google Scholar
  42. Ko AJ, Myers BA, Coblenz MJ, Aung HH (2006) An exploratory study of how developers seek, relate, and collect relevant information during software maintenance tasks. IEEE Trans Softw Eng 32(12):971–987CrossRefGoogle Scholar
  43. Koru AG, Ozok A, Norcio AF (2005) The effect of human memory organization on code reviews under different single and pair code reviewing scenarios. ACM SIGSOFT Software Engineering Notes 30:1–3Google Scholar
  44. Kushman N, Katabi D (2010) Enabling configuration-independent automation by non-expert users. In: Proceedings of the ninth USENIX symposium on operating systems design and implementation, pp 223–236Google Scholar
  45. Lawrance J, Bellamy R, Burnett M, Rector K (2008) Using information scent to model the dynamic foraging behavior of programmers in maintenance tasks. In: Proceedings of the CHI. ACM, pp 1323–1332Google Scholar
  46. Lawrance J, Bogart C, Burnett M, Bellamy R, Rector K, Fleming SD (2013) How programmers debug, revisited: an information foraging theory perspective. IEEE Trans Softw Eng 39(2):197–215CrossRefGoogle Scholar
  47. Lawrence J, Clarke S, Burnett M, Rothermel G (2005) How well do professional developers test with code coverage visualizations? An empirical study. In: Proceedings of the VL/HCC, pp 53–60Google Scholar
  48. Leary MR (1991) Introduction to behavioral research methods. Wadsworth Publishing CompanyGoogle Scholar
  49. Li H, Xing Z, Peng X, Zhao W (2013) What help do developers seek, when and how?. In: Proceedings of the WCRE, pp 142–151Google Scholar
  50. Lowe DG (1999) Object recognition from local scale-invariant features. In: Proceedings of the ICCV, vol 2, pp 1150–1157Google Scholar
  51. Matejka J, Li W (2009) CommunityCommands: command recommendations for software applications. In: Proceedings of the UIST, pp 193–202Google Scholar
  52. Matejka J, Grossman T, Fitzmaurice G (2011) Ambient help. In: Proceeding of the CHI, pp 2751–2760Google Scholar
  53. Mavrikis M, Grawemeyer B, Hansen A, Gutierrez-Santos S (2014) Exploring the potential of speech recognition to support problem solving and reflection. In: Open learning and teaching in educational communities. Springer, Berlin, pp 263–276Google Scholar
  54. Muja M, Lowe DG (2009) Fast approximate nearest neighbors with automatic algorithm configuration. In: VISAPP (1), pp 331–340Google Scholar
  55. Murphy-Hill ER, Zimmermann T, Nagappan N (2014) Cowboys, ankle sprains, and keepers of quality: how is video game development different from software development?. In: Proceeding of the ICSE, pp 1–11Google Scholar
  56. Nie Z, Zhang Y (2005) Object-level ranking: bringing order to web objects. In: Proceeding of the WWW, pp 567–574Google Scholar
  57. Nie Z, Ma Y, Shi S, Wen J-r, Ma W-y (2007) Web Object Retrieval. In: Proceeding of the WWW, pp 81–90Google Scholar
  58. Parzen E (1962) On estimation of a probability density function and mode. Ann Math Stat 33(3):1065–1076MathSciNetCrossRefMATHGoogle Scholar
  59. Piorkowski D, Fleming SD, Scaffidi C, John L, Bogart C, John BE, Burnett M, Bellamy R (2011) Modeling programmer navigation: a head-to-head empirical evaluation of predictive models. In: Proceeding of the VL/HCC, pp 109–116Google Scholar
  60. Ponzanelli L, Bacchelli A, Lanza M (2013) Seahawk: Stack overflow in the IDE. In: Proceeding of the ICSE, pp 1295–1298Google Scholar
  61. Rabiner L, Juang BH (1986) An introduction to hidden Markov models. IEEE ASSP Mag 3(1):4–16CrossRefGoogle Scholar
  62. Rhodes B (1996) Remembrance agent: a continuously running automated information retrieval system. In: The proceedings of the first international conference on the practical application of intelligent agents and multi agent technology, pp 122–125Google Scholar
  63. Robillard MP, Coelho W, Murphy GC (2004) How effective developers investigate source code: an exploratory study. IEEE Trans Softw Eng 30(12):889–903CrossRefGoogle Scholar
  64. Rosten E, Drummond T (2006) Machine learning for high-speed corner detection. In: Computer Vision–ECCV 2006. Springer, Berlin, pp 430–443CrossRefGoogle Scholar
  65. Sarma A, Maccherone L, Wagstrom P, Herbsleb J (2009) Tesseract: interactive visual exploration of socio-technical relationships in software development. In: Proceeding of the ICSE, pp 23–33Google Scholar
  66. Sawadsky N, Murphy GC (2011) Fishtail: from task context to source code examples. In: Proceeding of the 1st workshop on Developing tools as plug-ins - TOPI, p 48Google Scholar
  67. Shi J, Malik J (2000) Normalized cuts and image segmentation. IEEE Trans Pattern Anal Mach Intell 22(8):888–905CrossRefGoogle Scholar
  68. Sillito J, De Voider K, Fisher B, Murphy G (2005) Managing software change tasks: an exploratory study. In: International Symposium on Empirical Software Engineering, IEEE, p 10Google Scholar
  69. Silverman BW (1986) Density estimation for statistics and data analysis, vol 26. CRC pressGoogle Scholar
  70. Sinha SN, Frahm J-M, Pollefeys M, Genc Y (2006) GPU-based video feature tracking and matching. In: EDGE, workshop on edge computing using new commodity architectures, vol 278, p 4321Google Scholar
  71. Vakilian M, Chen N, Negara S, Rajkumar BA, Bailey BP, Johnson RE (2012) Use, disuse, and misuse of automated refactorings. In: Proceeding of the ICSE, pp 233–243Google Scholar
  72. von Mayrhauser A, Vans AM (1997) Program understanding behavior during debugging of large scale software. In: Empirical Studies of Programmers, 7th Workshop, ACM. ACM, pp 157–179Google Scholar
  73. Wang J, Peng X, Xing Z, Zhao W (2011) An exploratory study of feature location process: Distinct phases, recurring patterns, and elementary actions. In: Proceeding of the ICSM, pp 213–222Google Scholar
  74. Wang J, Peng X, Xing Z, Zhao W (2013) Improving feature location practice with multi-faceted interactive exploration. In: Proceeding of the ICSE, pp 762–771Google Scholar
  75. Weisstein EW (2011) Least squares fitting–exponential. MathWorld-A Wolfram Web Resource. http://mathworld.wolfram.com/LeastSquaresFittingExponential.html
  76. Whittaker JA, Poore JH (1993) Markov analysis of software specificationsGoogle Scholar
  77. Wu D-C, Tsai W-H (2000) Spatial-domain image hiding using image differencing. Proc ICCVISP 147(1):29–37Google Scholar
  78. Yang J, McAuley J, Leskovec J, LePendu P, Shah N (2014) Finding progression stages in time-evolving event sequences. In: Proceeding of the WWW, pp 783–794Google Scholar
  79. Yeh T, Chang T-H, Miller RC (2009) Sikuli: using GUI screenshots for search and automation. In: Proceeding of the UIST, pp 183–192Google Scholar
  80. Zhang Q, Chen Y, Zhang Y, Xu Y (2008) SIFT implementation and optimization for multi-core systems. In: Proceeding of the IPDPS, pp 1–8Google Scholar
  81. Zhu J, Nie Z, Liu X, Zhang B, Wen J-R (2009) StatSnowball: a statistical approach to extracting entity relationships. In: Proceeding of the WWW, p 101Google Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  • Lingfeng Bao
    • 1
  • Jing Li
    • 2
  • Zhenchang Xing
    • 2
  • Xinyu Wang
    • 1
  • Xin Xia
    • 1
  • Bo Zhou
    • 1
  1. 1.College of Computer ScienceZhejiang UniversityHangzhouChina
  2. 2.School of Computer EngineeringNanyang Technological UniversitySingaporeSingapore

Personalised recommendations