Skip to main content

Advertisement

SpringerLink
Log in

Evaluating interactive data systems

Survey and case studies

  • Special Issue Paper
  • Published:
The VLDB Journal Aims and scope Submit manuscript

Abstract

Interactive query interfaces have become a popular tool for ad hoc data analysis and exploration. Compared with traditional systems that are optimized for throughput or batched performance, these systems focus more on user-centric interactivity. This poses a new class of performance challenges to the backend, which are further exacerbated by the advent of new interaction modes (e.g., touch, gesture) and query interface paradigms (e.g., sliders, maps). There is, thus, a need to clearly articulate the evaluation space for interactive systems. In this paper, we extensively survey the literature to guide the development and evaluation of interactive data systems. We highlight unique characteristics of interactive workloads, discuss confounding factors when conducting user studies, and catalog popular metrics for evaluation. We further delineate certain behaviors not captured by these metrics and propose complementary ones to provide a complete picture of interactivity. We demonstrate how to analyze and employ user behavior for system enhancements through three case studies. Our survey and case studies motivate the need for behavior-driven evaluation and optimizations when building interactive interfaces.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21

Similar content being viewed by others

References

  1. Airbnb: Vacation Rentals, Homes, Experiences and Places. https://www.airbnb.com/

  2. AMP Benchmarks. https://amplab.cs.berkeley.edu/benchmark/

  3. Crossfilter Library. http://square.github.io/crossfilter/

  4. Delta. https://developer.mozilla.org/en-US/docs/Web/Events/mousewheel

  5. F 015 Luxury in Motion Concept Car: Interaction with the Vehicle Through Gestures, Eye Tracking and High-Res Touch-Screens. https://www.mbusa.com/mercedes/future/model/model-All_New_F015_Luxury

  6. GlobalInterpreterLock. https://wiki.python.org/moin/GlobalInterpreterLock

  7. IMDb. http://www.imdb.com/

  8. LDBC: The Graph and RDF Benchmark Reference. http://ldbcouncil.org/benchmarks

  9. MemSQL: The Fastest In-Memory Database. http://www.MemSQL.com/

  10. Mutation Events. https://developer.mozilla.org/en-US/docs/Web/Guide/Events/Mutation_events

  11. Number of smartphones sold to end users worldwide from 2007 to 2020 (in million units). https://www.statista.com/statistics/263437/global-smartphone-sales-to-end-users-since-2007/

  12. OMDb API—The Open Movie Database. http://www.omdbapi.com/

  13. PostgreSQL: The world’s most advanced open source database. https://www.postgresql.org/

  14. Power BI. https://powerbi.microsoft.com

  15. ScrollTop. https://developer.mozilla.org/en-US/docs/Web/API/Element/scrollTop

  16. Tableau. http://www.tableau.com/

  17. Tableau Vizable. https://vizable.tableau.com/

  18. TEDCAS uses the Myo armband to give surgeons gesture control. http://blog.thalmic.com/myo-armband-surgery/

  19. TPC Benchmark. http://www.tpc.org/

  20. UCI Repository of Machine Learning Databases. https://archive.ics.uci.edu/ml/datasets.html

  21. Use Multi-Touch gestures on your Mac. https://support.apple.com/en-us/HT204895

  22. Worldwide tablet shipments from 2nd quarter 2010 to 3rd quarter 2018 (in million units). https://www.statista.com/statistics/272070/global-tablet-shipments-by-quarter/

  23. Abouzied, A., Hellerstein, J., Silberschatz, A.: Dataplay: Interactive tweaking and example-driven correction of graphical database queries. In: Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology, UIST ’12, New York, NY, USA, pp. 207–218. ACM (2012)

  24. Agarwal, S., Mozafari, B., Panda, A., Milner, H., Madden, S., Stoica, I.: Blinkdb: Queries with bounded errors and bounded response times on very large data. In: Proceedings of the 8th ACM European Conference on Computer Systems, EuroSys’13, New York, NY, USA, pp. 29–42. ACM (2013)

  25. Al-Megren, S.: A predictive fingerstroke-level model for smartwatch interaction. Multimodal Technol. Interact. 2(3), 38 (2018)

    Google Scholar 

  26. Al-Megren, S., Altamimi, W., Al-Khalifa, H. S.: Blind flm: An enhanced keystroke-level model for visually impaired smartphone interaction. In: IFIP Conference on Human–Computer Interaction, pp. 155–172. Springer, New York (2017)

    Google Scholar 

  27. Albert, W., Tullis, T.: Measuring the user experience: collecting, analyzing, and presenting usability metrics. Newnes, Oxford (2013)

    Google Scholar 

  28. Armstrong, T. G., Ponnekanti, V., Borthakur, D., Callaghan, M.: Linkbench: A database benchmark based on the facebook social graph. In: Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data, SIGMOD ’13, New York, NY, USA, pp. 1185–1196. ACM (2013)

  29. Bakke, E., Karger, D., Miller, R.: A spreadsheet-based user interface for managing plural relationships in structured data. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI’11, New York, NY, USA, pp. 2541–2550. ACM (2011)

  30. Bakke, E., Karger, D.R.: Expressive query construction through direct manipulation of nested relational results. In: Proceedings of the 2016 International Conference on Management of Data, pp. 1377–1392. ACM, (2016)

  31. Bangor, A., Kortum, P.T., Miller, J.T.: An empirical evaluation of the system usability scale. Int. J. Hum. Comput. Interact. 24(6), 574–594 (2008)

    Google Scholar 

  32. Barnum, C .M.: Usability Testing Essentials: Ready, Set... Test!. Elsevier, Amsterdam (2010)

    Google Scholar 

  33. Basole, R.C., Clear, T., Hu, M., Mehrotra, H., Stasko, J.: Understanding interfirm relationships in business ecosystems with interactive visualization. IEEE Trans. Vis. Comput. Graph. 19(12), 2526–2535 (2013)

    Google Scholar 

  34. Basu Roy, S., Wang, H., Das, G., Nambiar, U., Mohania, M.: Minimum-effort driven dynamic faceted search in structured databases. In: Proceedings of the 17th ACM Conference on Information and Knowledge Management, CIKM’08, New York, NY, USA, pp. 13–22. ACM (2008)

  35. Battle, L., Chang, R., Heer, J., Stonebraker, M.: Position statement: the case for a visualization performance benchmark. In: 2017 IEEE Workshop on Data Systems for Interactive Analysis (DSIA), pp. 1–5. (2017)

  36. Battle, L., Chang, R., Stonebraker, M.: Dynamic prefetching of data tiles for interactive visualization. pp. 1363–1375, (2016)

  37. Bell, D.A., Deluca, L.S., Levinson, D.J., Salem, R.: Browser interaction for lazy loading operations, US Patent App. 14/570,430 (2014)

  38. Beltran, J.F., Huang, Z., Abouzied, A., Nandi, A.: Don’t just swipe left, tell me why: enhancing gesture-based feedback with reason bins. In: Proceedings of the 22nd International Conference on Intelligent User Interfaces, pp. 469–480. ACM, (2017)

  39. Bendre, M., Venkataraman, V., Zhou, X., Chang, K., Parameswaran, A.: Towards a holistic integration of spreadsheets with databases: a scalable storage engine for presentational data management. In: 2018 IEEE 34th International Conference on Data Engineering (ICDE), pp. 113–124. IEEE, (2018)

  40. Benson, B.: Cognitive bias cheat sheet. Better Hum. (2016)

  41. Bernard, J., Wilhelm, N., Krüger, B., May, T., Schreck, T., Kohlhammer, J.: Motionexplorer: exploratory search in human motion capture data based on hierarchical aggregation. IEEE Trans. Visual. Comput. Graph. 19(12), 2257–2266 (2013)

    Google Scholar 

  42. Bi, X., Li, Y., Zhai, S.: Ffitts law: modeling finger touch with fitts’ law. In: SIGCHI, (2013)

  43. Binnig, C., Fekete, A., Nandi, A.: Hilda’16: Proceedings of the workshop on human-in-the-loop data analytics. New York, NY, USA, ACM (2016)

  44. Biswas, A., Dutta, S., Shen, H.-W., Woodring, J.: An information-aware framework for exploring multivariate data sets. IEEE Trans. Vis. Comput. Graph. 19(12), 2683–2692 (2013)

    Google Scholar 

  45. Brooke, J., et al.: Sus-a quick and dirty usability scale. Usability Eval. Ind. 189(194), 4–7 (1996)

    Google Scholar 

  46. Burley, C., Nandi, A.: Arquery: Hallucinating analytics over real-world data using augmented reality. In: CIDR, (2019)

  47. Cao, N., Gotz, D., Sun, J., Qu, H.: Dicon: interactive visual analysis of multidimensional clusters. IEEE Trans. Vis. Comput. Graph. 17(12), 2581–2590 (2011)

    Google Scholar 

  48. Card, S.K.: The psychology of human–computer interaction. CRC Press, New York (2017)

    Google Scholar 

  49. Chan, S.-M., Xiao, L., Gerth, J., Hanrahan, P.: Maintaining interactivity while exploring massive time series. In: 2008 IEEE Symposium on Visual Analytics Science and Technology, pp. 59–66, (2008)

  50. Chau, P., Vreeken, J., van Leeuwen, M., Shahaf, D., Faloutsos, C.: Proceedings of the ACM SIGKDD workshop on interactive data exploration and analytics. In: ACM SIGKDD 2016 Full-Day Workshop on Interactive Data Exploration and Analytics. IDEA’16, (2016)

  51. Chaudhuri, S., Motwani, R., Narasayya, V.: Random sampling for histogram construction: How much is enough? In: Proceedings of the 1998 ACM SIGMOD International Conference on Management of Data, SIGMOD’98, New York, NY, USA, pp. 436–447. ACM (1998)

  52. Chen, K.: Data-driven techniques for improving data collection in low-resource environments. Ph.D. thesis, UC Berkeley, (2011)

  53. Chen, Y., Alspaugh, S., Borthakur, D., Katz, R.: Energy efficiency for large-scale mapreduce workloads with significant interactive analysis. In: Proceedings of the 7th ACM European Conference on Computer Systems, EuroSys’12, New York, NY, USA, pp. 43–56. ACM (2012)

  54. Chen, Y., Alspaugh, S., Katz, R.: Interactive analytical processing in big data systems: a cross-industry study of mapreduce workloads. Proc. VLDB Endow. 5(12), 1802–1813 (2012)

    Google Scholar 

  55. Cooper, B. F., Silberstein, A., Tam, E., Ramakrishnan, R., Sears, R.: Benchmarking cloud serving systems with YCSB. In: Proceedings of the 1st ACM Symposium on Cloud Computing, SoCC’10, New York, NY, USA, pp. 143–154. ACM (2010)

  56. Correll, M., Li, M., Kindlmann, G., Scheidegger, C.: Looks good to me: Visualizations as sanity checks. IEEE Trans. Vis. Comput. Graph. (2018)

  57. Demiralp, Ç., Haas, P. J., Parthasarathy, S., Pedapati, T.: Foresight: Rapid Data Exploration Through Guideposts. arXiv preprint arXiv:1709.10513 (2017)

    MathSciNet  Google Scholar 

  58. DeWitt, D.J.: The Wisconsin benchmark: past, present, and future, (1993)

  59. Dimara, E., Bailly, G., Bezerianos, A., Franconeri, S.: Mitigating the attraction effect with visualizations. IEEE Trans. Vis. Comput. Graph. 25(1), 850–860 (2018)

    Google Scholar 

  60. Dimara, E., Franconeri, S., Plaisant, C., Bezerianos, A., Dragicevic, P.: A task-based taxonomy of cognitive biases for information visualization. IEEE Trans. Vis. Comput. Graph. (2018)

  61. Dimitriadou, K., Papaemmanouil, O., Diao, Y.: Explore-by-example: an automatic query steering framework for interactive data exploration. In: Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data, SIGMOD’14, New York, NY, USA, pp. 517–528. ACM (2014)

  62. Doshi, P.R., Rundensteiner, E.A., Ward, M.O.: Prefetching for visual data exploration. Eighth Int. Conf. Database Syst. Adv. Appl. 2003, 195–202 (2003)

    Google Scholar 

  63. Drucker, S.M., Fisher, D., Sadana, R., Herron, J. et al.: Touchviz: a case study comparing two interfaces for data analytics on tablets. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 2301–2310. ACM, (2013)

  64. Ebenstein, R., Kamat, N., Nandi, A.: Fluxquery: An execution framework for highly interactive query workloads. In: Proceedings of the 2016 International Conference on Management of Data, pp. 1333–1345. ACM, (2016)

  65. Eichmann, P., Binnig, C., Kraska, T., Zgraggen, E.: Idebench: A benchmark for interactive data exploration. CoRR, arXiv:1804.02593 (2018)

  66. Eichmann, P., Zgraggen, E., Zhao, Z., Binnig, C., Kraska, T.: Towards a benchmark for interactive data exploration. IEEE Data Eng. Bull. 39, 50–61 (2016)

    Google Scholar 

  67. El Batran, K., Dunlop, M.D.: Enhancing KLM (keystroke-level model) to fit touch screen mobile devices. In: Proceedings of the 16th International Conference on Human–Computer Interaction with Mobile Devices and Services

  68. Faith, J.: Targeted projection pursuit for interactive exploration of high- dimensional data sets. In: 11th International Conference on Information Visualization, IV’07, pp. 286–292, (2007)

  69. Faulkner, L.: Beyond the five-user assumption: benefits of increased sample sizes in usability testing. Behav. Res. Methods Instrum. Comput. 35(3), 379–383 (2003)

    Google Scholar 

  70. Fekete, J.-D.: Progressivis: A toolkit for steerable progressive analytics and visualization. In: 1st Workshop on Data Systems for Interactive Analysis, p. 5, (2015)

  71. Fekete, J.-D., Plaisant, C.: Interactive information visualization of a million items. In: Proceedings of the IEEE Symposium on Information Visualization (InfoVis’02), INFOVIS’02, IEEE Computer Society, p. 117, Washington, DC, USA (2002)

  72. Feng, S., Huber, A., Glavic, B., Kennedy, O.: Uncertainty annotated databases-a lightweight approach for approximating certain answers. In: Proceedings of the 2019 International Conference on Management of Data, pp. 1313–1330. ACM, (2019)

  73. Ferreira, N., Poco, J., Vo, H.T., Freire, J., Silva, C.T.: Visual exploration of big spatio-temporal urban data: a study of new york city taxi trips. IEEE Trans. Vis. Comput. Graph. 19(12), 2149–2158 (2013)

    Google Scholar 

  74. Fisher, D., Popov, I., Drucker, S., schraefel, M.: Trust me, I’m partially right: incremental visualization lets analysts explore large datasets faster. In: ACM Conference on Human Factors in Computing Systems, (2012)

  75. Fitts, P.M.: The information capacity of the human motor system in controlling the amplitude of movement. J. Exp. Psychol. 47(6), 381 (1954)

    Google Scholar 

  76. Gibbons, P. B., Matias, Y., Poosala, V.:. Fast incremental maintenance of approximate histograms, vol. 27, New York, NY, USA, pp. 261–298. ACM (2002)

  77. Gray, J.: Benchmark Handbook: For Database and Transaction Processing Systems. Morgan Kaufmann Publishers Inc., San Francisco (1992)

    MATH  Google Scholar 

  78. Gujarathi, N.R., Shah, A.A.: Parameterized computed scrolling for navigation of structured data, (2015)

  79. Gunopulos, D., Kollios, G., Tsotras, V.J., Domeniconi, C.: Approximating multi-dimensional aggregate range queries over real attributes. In: Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data, SIGMOD’00, New York, NY, USA, pp. 463–474. ACM (2000)

  80. Halim, F., Idreos, S., Karras, P., Yap, R.H.C.: Stochastic database cracking: Towards robust adaptive indexing in main-memory column-stores. Proc. VLDB Endow. 5(6), 502–513 (2012)

    Google Scholar 

  81. Haselton, M.G., Nettle, D., Murray, D.R.: The evolution of cognitive bias. In: The Handbook of Evolutionary Psychology, pp. 1–20, (2015)

  82. Hebert, C., Ridgway, J., Vekhter, B., Brown, E.C., Weber, S.G., Robicsek, A.: Demonstration of the weighted-incidence syndromic combination antibiogram: an empiric prescribing decision aid. Infect. Control Hosp. Epidemiol. 33(4), 381–388 (2012)

    Google Scholar 

  83. Heer, J., Agrawala, M., Willett, W.: Generalized selection via interactive query relaxation. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI’08, New York, NY, USA, pp. 959–968. ACM (2008)

  84. Heer, J., Bostock, M.: Crowdsourcing graphical perception: using mechanical turk to assess visualization design. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 3–212. ACM, (2010)

  85. Hellerstein, J.M., Haas, P.J., Wang, H.J.: Online aggregation. In: Acm Sigmod Record, vol. 6, pp. 1–182. ACM, (1997)

  86. Hirte, S., Seifert, A., Baumann, S., Klan, D., Sattler, K.-U.: Data3—a kinect interface for OLAP using complex event processing. In: Proceedings of the 2012 IEEE 28th International Conference on Data Engineering, ICDE’12, IEEE Computer Society , Washington, DC, USA, pp. 1297–1300 (2012)

  87. Holleis, P., Otto, F., Hussmann, H., Schmidt, A.: Keystroke-level model for advanced mobile phone interaction. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1505–1514. ACM, (2007)

  88. Hu, K., Orghian, D., Hidalgo, C.: DIVE: a mixed-initiative system supporting integrated data exploration workflows. In: Proceedings of the Workshop on Human-In-the-Loop Data Analytics, p. 5. ACM, (2018)

  89. Idreos, S.: Database cracking: towards auto-tuning database kernels. CWI and University of Amsterdam, (2010)

  90. Idreos, S., Manegold, S., Kuno, H., Graefe, G.: Merging what’s cracked, cracking what’s merged: adaptive indexing in main-memory column-stores. Proc. VLDB Endow. 4(9), 586–597 (2011)

    Google Scholar 

  91. Idreos, S., Papaemmanouil, O., Chaudhuri, S.: Overview of data exploration techniques. In: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, SIGMOD’15, New York, NY, USA, pp. 277–281. ACM (2015)

  92. Igarashi, T., Hinckley, K.: Speed-dependent automatic zooming for browsing large documents. In: Proceedings of the 13th Annual ACM Symposium on User Interface Software and Technology, UIST’00, New York, NY, USA, pp. 139–148. ACM (2000)

  93. Jagadish, H.V., Chapman, A., Elkiss, A., Jayapandian, M., Li, Y., Nandi, A., Yu, C.: Making database systems usable. In: Proceedings of the 2007 ACM SIGMOD International Conference on Management of Data, SIGMOD’07, New York, NY, USA, pp. 13–24. ACM (2007)

  94. Javed, W., Ghani, S., Elmqvist, N.: Gravnav: Using a gravity model for multi-scale navigation. In: Proceedings of the International Working Conference on Advanced Visual Interfaces, AVI’12, New York, NY, USA, pp. 217–224. ACM (2012)

  95. Jiang, L., Nandi, A.: Snaptoquery: Providing interactive feedback during exploratory query specification. Proc. VLDB Endow. 8(11), 1250–1261 (2015)

    Google Scholar 

  96. John, B.E., Kieras, D.E.: Using GOMS for user interface design and evaluation: Which technique? TOCHI 3(4), 287–319 (1996)

    Google Scholar 

  97. Jota, R., Ng, A., Dietz, P., Wigdor, D.: How fast is fast enough?: a study of the effects of latency in direct-touch pointing tasks. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 2291–2300. ACM, (2013)

  98. Kamat, N., Jayachandran, P., Tunga, K., Nandi, A.: Distributed and interactive cube exploration. In 2014 IEEE 30th International Conference on Data Engineering, pp. 472–483, (2014)

  99. Kamat, N., Nandi, A.: Infiniviz: Interactive visual exploration using progressive bin refinement. arXiv preprint arXiv:1710.01854 (2017)

  100. Kamat, N., Nandi, A.: A session-based approach to fast-but-approximate interactive data cube exploration. ACM Trans. Knowl. Discov Data (TKDD) 12(1), 9 (2018)

    Google Scholar 

  101. Kandel, S., Paepcke, A., Hellerstein, J., Heer, J.: Wrangler: Interactive visual specification of data transformation scripts. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 3363–3372. ACM, (2011)

  102. Kandel, S., Paepcke, A., Hellerstein, J., Heer, J.: Wrangler: Interactive visual specification of data transformation scripts. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI’11, New York, NY, USA, pp. 3363–3372. ACM (2011)

  103. Kang, J., Naughton, J.F., Viglas, S.D.: Evaluating window joins over unbounded streams. In: Proceedings 19th International Conference on Data Engineering (Cat. No.03CH37405), pp. 341–352, (2003)

  104. Kashyap, A., Hristidis, V., Petropoulos, M.: Facetor: Cost-driven exploration of faceted query results. In: Proceedings of the 19th ACM International Conference on Information and Knowledge Management, CIKM’10, New York, NY, USA, pp. 719–728. ACM (2010)

  105. Kashyap, A., Hristidis, V., Petropoulos, M., Tavoulari, S.: Effective navigation of query results based on concept hierarchies. IEEE Trans. Knowl. Data Eng. 23(4), 540–553 (2011)

    Google Scholar 

  106. Kaul, M., Yang, B., Jensen, C.S.: Building accurate 3D spatial networks to enable next generation intelligent transportation systems. In 2013 IEEE 14th International Conference on Mobile Data Management, vol. 1, pp. 137–146, (2013)

  107. Keim, D.A.: Information visualization and visual data mining. IEEE Trans. Visual. Comput. Graph. 8(1), 1–8 (2002)

    MathSciNet  Google Scholar 

  108. Kennedy, O., Ajay, J., Challen, G., Ziarek, L.: Pocket data: the need for TPC-mobile. In: Nambiar, R., Poess, M. (eds.) Performance Evaluation and Benchmarking: Traditional to Big Data to Internet of Things, pp. 8–25. Springer, Cham (2016)

    Google Scholar 

  109. Key, A., Howe, B., Perry, D., Aragon, C.: Vizdeck: self-organizing dashboards for visual analytics. In: Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, SIGMOD’12, New York, NY, USA, pp. 681–684. ACM (2012)

  110. Khan, M., Xu, L., Nandi, A., Hellerstein, J.M.: Data tweening: incremental visualization of data transforms. Proc. VLDB Endow. 10(6), 661–672 (2017)

    Google Scholar 

  111. Kim, A., Blais, E., Parameswaran, A., Indyk, P., Madden, S., Rubinfeld, R.: Rapid sampling for visualizations with ordering guarantees. Proc. VLDB Endow. 8(5), 521–532 (2015)

    Google Scholar 

  112. Kim, J.-M., Kim, J.-S.: Androbench: benchmarking the storage performance of android-based mobile devices. In: Sambath, S., Zhu, E. (eds.) Frontiers in Computer Education, pp. 667–674. Springer, Berlin (2012)

    Google Scholar 

  113. Kosara, R., Bendix, F., Hauser, H.: Parallel sets: interactive exploration and visual analysis of categorical data. IEEE Trans. Vis. Comput. Graph. 12(4), 558–568 (2006)

    Google Scholar 

  114. Kraiss, A., Weikum, G.: Integrated document caching and prefetching in storage hierarchies based on markov-chain predictions. VLDB J. 7(3), 141–162 (1998)

    Google Scholar 

  115. Kraska, T.: Northstar: an interactive data science system. Proc. VLDB Endow. 11(12), 2150–2164 (2018)

    Google Scholar 

  116. Kullback, S.: Information Theory and Statistics. Courier Corporation, Chelmsford (1997)

    MATH  Google Scholar 

  117. Lam, H., Bertini, E., Isenberg, P., Plaisant, C., Carpendale, S.: Empirical studies in information visualization: seven scenarios. IEEE Trans. Vis. Comput. Graph. 18(9), 1520–1536 (2012)

    Google Scholar 

  118. Lazar, J., Feng, J.H., Hochheiser, H.: Research Methods in Human–Computer Interaction. Wiley, Hoboken (2010)

    Google Scholar 

  119. Lee, A., Song, K., Ryu, H.B., Kim, J., Kwon, G.: Fingerstroke time estimates for touchscreen-based mobile gaming interaction. Hum. Mov. Sci. 44, 211–224 (2015)

    Google Scholar 

  120. Lee, D.H., Kim, J.S., Kim, S.D., Kim, K.C., Yoo-Sung, K., Park, J.: Adaptation of a neighbor selection markov chain for prefetching tiled web gis data. In: Yakhno, T. (ed.) Advances in Information Systems, pp. 213–222. Springer, Berlin (2002)

    Google Scholar 

  121. Li, Y., Yang, H., Jagadish, H.V.: Nalix: a generic natural language search environment for xml data. ACM Trans. Database Syst. 32(4), 30 (2007)

    Google Scholar 

  122. Liarou, E., Idreos, S.: dbtouch in action database kernels for touch-based data exploration. In 2014 IEEE 30th International Conference on Data Engineering, pp. 1262–1265, (2014)

  123. Lins, L., Klosowski, J.T., Scheidegger, C.: Nanocubes for real-time exploration of spatiotemporal datasets. IEEE Trans. Vis. Comput. Graph. 19(12), 2456–2465 (2013)

    Google Scholar 

  124. Liu, B., Jagadish, H.: A spreadsheet algebra for a direct data manipulation query interface. In: 2009 IEEE 25th International Conference on Data Engineering, pp. 417–428, (2009)

  125. Liu, F., Kamat, N., Blanas, S., Nandi, A.: To ship or not to (function) ship (extended version). In: 2018 IEEE High Performance Extreme Computing Conference (HPEC). IEEE, (2018)

  126. Liu, Z., Heer, J.: The effects of interactive latency on exploratory visual analysis. IEEE Trans. Vis. Comput. Graph. 20(12), 2122–2131 (2014)

    Google Scholar 

  127. Liu, Z., Jiang, B., Heer, J.: immens: real-time visual querying of big data. Comput. Graph. Forum 32(3–4), 421–430 (2013)

    Google Scholar 

  128. Mackinlay, J., Hanrahan, P., Stolte, C.: Show me: automatic presentation for visual analysis. IEEE Trans. Vis. Comput. Graph. 13(6), 1137–1144 (2007)

    Google Scholar 

  129. Martin, A.R., Ward, M.O.: High dimensional brushing for interactive exploration of multivariate data. In: Proceedings of the 6th Conference on Visualization’95, VIS’95, IEEE Computer Society, Washington, DC, USA, p. 271 (1995)

  130. McLachlan, P., Munzner, T., Koutsofios, E., North, S.: Liverac: Interactive visual exploration of system management time-series data. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI’08, New York, NY, USA, pp. 1483–1492. ACM (2008)

  131. Mohan, C.: Caching technologies for web applications. In: VLDB, vol. 1, p. 726, (2001)

  132. Moritz, D., Fisher, D., Ding, B., Wang, C.: Trust, but verify: Optimistic visualizations of approximate queries for exploring big data. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pp. 2904–2915. ACM, (2017)

  133. Moritz, D., Howe, B., Heer, J.: Falcon: Balancing interactive latency and resolution sensitivity for scalable linked visualizations. (2019)

  134. Mostak, T.: An overview of MAPD (massively parallel database). White paper, Massachusetts Institute of Technology (2013)

  135. Moussavi, F.: Hybrid inertial and touch sensing input device, US Patent App. 12/192,889 (2010)

  136. Munzner, T.: A nested model for visualization design and validation. IEEE Trans. Vis. Comput. Graph. 15(6), 921–928 (2009)

    Google Scholar 

  137. Nandi, A., Jiang, L., Mandel, M.: Gestural query specification. Proc. VLDB Endow. 7(4), 289–300 (2013)

    Google Scholar 

  138. Nelson, W.T., Roe, M.M., Bolia, R.S., Morley, R.M.: Assessing simulator sickness in a see-through HMD: Effects of time delay, time on task, and task complexity. Technical report, air force research lab Wright-Patterson AFB OH (2000)

  139. Nourbakhsh, N., Wang, Y., Chen, F., Calvo, R.A.: Using galvanic skin response for cognitive load measurement in arithmetic and reading tasks. In: Proceedings of the 24th Australian Computer-Human Interaction Conference, OzCHI’12, New York, NY, USA, pp. 420–423. ACM (2012)

  140. Omidvar-Tehrani, B., Nandi, A., Meyer, N., Flanagan, D., Young, S.: Dv8: Interactive analysis of aviation data. In: 2017 IEEE 33rd International Conference on Data Engineering (ICDE), pp. 1411–1412. IEEE, (2017)

  141. Padilla, L.: A case for cognitive models in visualization research. In: Proceedings of the Seventh Workshop on Beyond Time and Errors on Novel Evaluation Methods for Visualization, pp. 143–151, (2018)

  142. Patterson, E.S., Dewart, C.M., Stevenson, K., Mbodj, A., Lustberg, M., Hade, E.M., Hebert, C.: A mixed methods approach to tailoring evidence-based guidance for antibiotic stewardship to one medical system. In: Proceedings of the International Symposium on Human Factors and Ergonomics in Health Care, vol. 7, pp. 224–231. SAGE Publications Sage India: New Delhi (2018)

    Google Scholar 

  143. Pavlovych, A., Gutwin, C.: Assessing target acquisition and tracking performance for complex moving targets in the presence of latency and jitter. In: Proceedings of Graphics Interface 2012, pp. 109–116. Canadian Information Processing Society, (2012)

  144. Plaisant, C.: The challenge of information visualization evaluation. In: Proceedings of the Working Conference on Advanced Visual Interfaces, AVI’04, New York, NY, USA, pp. 109–116. ACM (2004)

  145. Poosala, V., Haas, P.J., Ioannidis, Y.E., Shekita, E.J.: Improved histograms for selectivity estimation of range predicates. In: Proceedings of the 1996 ACM SIGMOD International Conference on Management of Data, SIGMOD’96, New York, NY, USA, pp. 294–305. ACM (1996)

  146. Psallidas, F., Wu, E.: Smoke: fine-grained lineage at interactive speed. Proc. VLDB Endow. 11(6), 719–732 (2018)

    Google Scholar 

  147. Quezada, A., Juárez-Ramírez, R., Jiménez, S., Ramírez-Noriega, A., Inzunza, S., Munoz, R.: Assessing the target? Size and drag distance in mobile applications for users with autism. In: World Conference on Information Systems and Technologies, pp. 1219–1228. Springer, New York, (2018)

    Google Scholar 

  148. Rahman, P., Hebert, C., Nandi, A.: ICARUS: minimizing human effort in iterative data completion. PVLDB, 11(13), Preprint available at go.osu.edu/icarus (2018)

  149. Rahman, P., Nandi, A.: Transformer: A database-driven approach to constrained generating forms for constrained interaction. In: Proceedings of the 24th International Conference on Intelligent User Interfaces. ACM, Preprint available at cse.osu.edu/rahman.92 (2019)

  150. Rahman, S., Aliakbarpour, M., Kong, H.K., Blais, E., Karahalios, K., Parameswaran, A., Rubinfield, R.: I’ve seen enough: incrementally improving visualizations to support rapid decision making. Proc. VLDB Endow. 10(11), 1262–1273 (2017)

    Google Scholar 

  151. Rosa, G.M., Elizondo, M.L.: Use of a gesture user interface as a touchless image navigation system in dental surgery: case series report. Imaging Sci. Dent. 44(2), 155–160 (2014)

    Google Scholar 

  152. Rzeszotarski, J.M., Kittur, A.: Kinetica: Naturalistic multi-touch data visualization. In: Proceedings of the 32Nd Annual ACM Conference on Human Factors in Computing Systems, CHI’14, New York, NY, USA, pp. 897–906. ACM (2014)

  153. Saadé, R.G., Otrakji, C.A.: First impressions last a lifetime: effect of interface type on disorientation and cognitive load. Comput. Hum. Behav. 23(1), 525–535 (2007)

    Google Scholar 

  154. Satyanarayan, A., Heer, J.: Lyra: An interactive visualization design environment. In: Proceedings of the 16th Eurographics Conference on Visualization, EuroVis’14, pp. 351–360. Aire-la-Ville, Switzerland, Eurographics Association (2014)

  155. Seo, J., Shneiderman, B.: A rank-by-feature framework for interactive exploration of multidimensional data. Inf. Vis. 4(2), 96–113 (2005)

    Google Scholar 

  156. Shneiderman, B.: The eyes have it: A task by data type taxonomy for information visualizations. In: Proceedings of the 1996 IEEE Symposium on Visual Languages, VL’96, IEEE Computer Society , p. 336, Washington, DC, USA (1996)

  157. Shneiderman, B., Williamson, C., Ahlberg, C.: Dynamic queries: database searching by direct manipulation. pp. 669–670, (1992)

  158. Siddiqui, T., Kim, A., Lee, J., Karahalios, K., Parameswaran, A.: Effortless data exploration with zenvisage: an expressive and interactive visual analytics system. Proc. VLDB Endow. 10(4), 457–468 (2016)

    Google Scholar 

  159. Sidirourgos, L., Kersten, M.L., Boncz, P.A. et al.: Sciborq: Scientific data management with bounds on runtime and quality. In: CIDR, (2011)

  160. Singh, M., Nandi, A., Jagadish, H.V.: Skimmer: Rapid scrolling of relational query results. In: Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, SIGMOD’12, New York, NY, USA, pp. 181–192. ACM (2012)

  161. St Amant, R., Horton, T.E., Ritter, F.E.: Model-based evaluation of cell phone menu interaction. In: CHI, (2004)

  162. Sweller, J.: Cognitive load during problem solving: effects on learning. Cognit. Sci. 12(2), 257–285 (1988)

    Google Scholar 

  163. Tan, P.: BMW demonstrates future iDrive with Touchscreen, Gesture and Tablet Control. CES 2015, (2015)

  164. Tauheed, F., Heinis, T., Schürmann, F., Markram, H., Ailamaki, A.: Scout: prefetching for latent structure following queries. Proc. VLDB Endow. 5(11), 1531–1542 (2012)

    Google Scholar 

  165. Valdez, A.C., Ziefle, M., Sedlmair, M.: A framework for studying biases in visualization research. (2017)

  166. Van Dillen, L.F., Heslenfeld, D.J., Koole, S.L.: Tuning down the emotional brain: an fmri study of the effects of cognitive load on the processing of affective images. Neuroimage 45(4), 1212–1219 (2009)

    Google Scholar 

  167. Vartak, M., Madden, S., Parameswaran, A., Polyzotis, N.: Seedb: automatically generating query visualizations. Proc. VLDB Endow. 7(13), 1581–1584 (2014)

    Google Scholar 

  168. Viglas, S.D., Naughton, J.F., Burger, J.: Maximizing the output rate of multi-way join queries over streaming information sources. In: Proceedings of the 29th International Conference on Very Large Data Bases—Volume 29, VLDB’03, VLDB Endowment, pp. 285–296. (2003)

    Google Scholar 

  169. Vitter, J.S., Wang, M., Iyer, B.: Data cube approximation and histograms via wavelets. In: Proceedings of the Seventh International Conference on Information and Knowledge Management, CIKM’98, New York, NY, USA, pp. 96–104. ACM (1998)

  170. Wall, E., Agnihotri, M., Matzen, L., Divis, K., Haass, M., Endert, A., Stasko, J.: A heuristic approach to value-driven evaluation of visualizations. IEEE Trans. Vis. Comput. Graph. 25(1), 491–500 (2018)

    Google Scholar 

  171. Wang, L., Zhan, J., Luo, C., Zhu, Y., Yang, Q., He, Y., Gao, W., Jia, Z., Shi, Y., Zhang, S., Zheng, C., Lu, G., Zhan, K., Li, X., Qiu, B.: Bigdatabench: a big data benchmark suite from internet services. In: 2014 IEEE 20th International Symposium on High Performance Computer Architecture (HPCA), pp. 488–499, (2014)

  172. Weaver, C.: Multidimensional visual analysis using cross-filtered views. In: 2008 IEEE Symposium on Visual Analytics Science and Technology, pp. 163–170, (2008)

  173. Wei, J., Shen, Z., Sundaresan, N., Ma, K.-L.: Visual cluster exploration of web clickstream data. In: 2012 IEEE Conference on Visual Analytics Science and Technology (VAST), pp. 3–12, (2012)

  174. Wilkinson, L.: The Grammar of Graphics. Springer, Berlin (2006)

    MATH  Google Scholar 

  175. Willett, W., Heer, J., Agrawala, M.: Scented widgets: improving navigation cues with embedded visualizations. IEEE Trans. Vis. Comput. Graph. 13(6), 1129–1136 (2007)

    Google Scholar 

  176. Wongsuphasawat, K., Moritz, D., Anand, A., Mackinlay, J., Howe, B., Heer, J.: Voyager: exploratory analysis via faceted browsing of visualization recommendations. IEEE Trans. Vis. Comput. Graph. 22(1), 649–658 (2016)

    Google Scholar 

  177. Wongsuphasawat, K., Qu, Z., Moritz, D., Chang, R., Ouk, F., Anand, A., Mackinlay, J., Howe, B., Heer, J.: Voyager 2: Augmenting visual analysis with partial view specifications. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pp. 2648–2659. ACM, (2017)

  178. Wongsuphasawat, K., Smilkov, D., Wexler, J., Wilson, J., Mané, D., Fritz, D., Krishnan, D., Viégas, F.B., Wattenberg, M.: Visualizing dataflow graphs of deep learning models in tensorflow. IEEE Trans. Vis. Comput. Graph. 24(1), 1–12 (2018)

    Google Scholar 

  179. Woodring, J., Shen, H.-W.: Multiscale time activity data exploration via temporal clustering visualization spreadsheet. IEEE Trans. Vis. Comput. Graph. 15(1), 123–137 (2009)

    Google Scholar 

  180. Wu, E., Jiang, L., Xu, L., Nandi, A.: Graphical perception in animated bar charts. arXiv preprint arXiv:1604.00080 (2016)

  181. Wu, E., Jiang, L., Xu, L., Nandi, A.: Graphical perception in animated bar charts. volume arXiv:1604.00080 (2016)

  182. Wu, Y., Chang, R., Wu, E., Hellerstein, J.: Programming with timespans in interactive visualizations. arXiv preprint arXiv:1907.00075 (2019)

  183. Yakout, M., Elmagarmid, A.K., Neville, J., Ouzzani, M., Ilyas, I.F.: Guided data repair. Proc. VLDB Endow. 4(5), 279–289 (2011)

    Google Scholar 

  184. Yang, D., Guo, Z., Rundensteiner, E.A., Ward, M.O.: Clues: A unified framework supporting interactive exploration of density-based clusters in streams. In: Proceedings of the 20th ACM International Conference on Information and Knowledge Management, CIKM’11, New York, NY, USA, pp. 815–824. ACM (2011)

  185. Yang, D., Rundensteiner, E.A., Ward, M.O.: Analysis guided visual exploration of multivariate data. In 2007 IEEE Symposium on Visual Analytics Science and Technology, pp. 83–90, (2007)

  186. Yang, J., Patro, A., Huang, S., Mehta, N., Ward, M.O., Rundensteiner, E.A.: Value and relation display for interactive exploration of high dimensional datasets. In: IEEE Symposium on Information Visualization, pp. 73–80, (2004)

  187. Yang, J., Peng, W., Ward, M.O., Rundensteiner, E.A.: Interactive hierarchical dimension ordering, spacing and filtering for exploration of high dimensional datasets. In: Proceedings of the Ninth Annual IEEE Conference on Information Visualization, INFOVIS’03, IEEE Computer Society, Washington, DC, USA, pp. 105–112. (2003)

  188. Yeşilmurat, S., İşler, V.: Retrospective adaptive prefetching for interactive web GIS applications. GeoInformatica 16(3), 435–466 (2012)

    Google Scholar 

  189. Yi, J.S., Kang, Y ah, Stasko, J.: Toward a deeper understanding of the role of interaction in information visualization. IEEE Trans. Vis. Comput. Graph. 13(6), 1224–1231 (2007)

    Google Scholar 

  190. Yuan, X., Ren, D., Wang, Z., Guo, C.: Dimension projection matrix/tree: interactive subspace visual exploration and analysis of high dimensional data. IEEE Trans. Vis. Comput. Graph. 19(12), 2625–2633 (2013)

    Google Scholar 

  191. Yuan, Y., Wang, G., Chen, L., Wang, H.: Graph similarity search on large uncertain graph databases. VLDB J. Int. J. Very Large Data Bases 24(2), 271–296 (2015)

    Google Scholar 

  192. Yuan, Y., Wang, G., Wang, H., Chen, L.: Efficient subgraph search over large uncertain graphs. Proc. VLDB Endow. 4(11), 876–886 (2011)

    Google Scholar 

  193. Yuan, Y., Wang, G., Xu, J.Y., Chen, L.: Efficient distributed subgraph similarity matching. VLDB J. Int. J. Very Large Data Bases 24(3), 369–394 (2015)

    Google Scholar 

  194. Zarifis, K., Papakonstantinou, Y.: ViDeTTe Interactive Notebooks. In: Proceedings of the Workshop on Human-In-the-Loop Data Analytics, p. 2. ACM, (2018)

  195. Zgraggen, E., Zeleznik, R., Drucker, S.M.: Panoramicdata: data analysis through pen & touch. IEEE Trans. Vis. Comput. Graph. 20(12), 2112–2121 (2014)

    Google Scholar 

  196. Zhang, Z., McDonnell, K. T., Mueller, K.: A network-based interface for the exploration of high-dimensional data spaces. In: Proceedings of the 2012 IEEE Pacific Visualization Symposium, PACIFICVIS’12, IEEE Computer Society, Washington, DC, USA, pp. 17–24, (2012)

Download references

Author information

Authors and Affiliations

  1. Computer Science and Engineering, The Ohio State University, 2015 Neil Ave, Columbus, OH, 43210, USA

    Protiva Rahman & Arnab Nandi

  2. Twitter, Inc., 1355 Market St, San Francisco, CA, 94103, USA

    Lilong Jiang

Authors
  1. Protiva Rahman
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lilong Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Arnab Nandi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Protiva Rahman.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rahman, P., Jiang, L. & Nandi, A. Evaluating interactive data systems. The VLDB Journal 29, 119–146 (2020). https://doi.org/10.1007/s00778-019-00589-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00778-019-00589-2

Keywords

Search

Navigation