Advertisement

Evaluating interactive data systems

Survey and case studies
  • Protiva RahmanEmail author
  • Lilong Jiang
  • Arnab Nandi
Special Issue Paper

Abstract

Interactive query interfaces have become a popular tool for ad hoc data analysis and exploration. Compared with traditional systems that are optimized for throughput or batched performance, these systems focus more on user-centric interactivity. This poses a new class of performance challenges to the backend, which are further exacerbated by the advent of new interaction modes (e.g., touch, gesture) and query interface paradigms (e.g., sliders, maps). There is, thus, a need to clearly articulate the evaluation space for interactive systems. In this paper, we extensively survey the literature to guide the development and evaluation of interactive data systems. We highlight unique characteristics of interactive workloads, discuss confounding factors when conducting user studies, and catalog popular metrics for evaluation. We further delineate certain behaviors not captured by these metrics and propose complementary ones to provide a complete picture of interactivity. We demonstrate how to analyze and employ user behavior for system enhancements through three case studies. Our survey and case studies motivate the need for behavior-driven evaluation and optimizations when building interactive interfaces.

Keywords

Evaluation Benchmark Survey Interactive systems Gestural devices Touch-screen Human-in-the-loop 

Notes

References

  1. 1.
    Airbnb: Vacation Rentals, Homes, Experiences and Places. https://www.airbnb.com/
  2. 2.
  3. 3.
  4. 4.
  5. 5.
    F 015 Luxury in Motion Concept Car: Interaction with the Vehicle Through Gestures, Eye Tracking and High-Res Touch-Screens. https://www.mbusa.com/mercedes/future/model/model-All_New_F015_Luxury
  6. 6.
  7. 7.
  8. 8.
    LDBC: The Graph and RDF Benchmark Reference. http://ldbcouncil.org/benchmarks
  9. 9.
    MemSQL: The Fastest In-Memory Database. http://www.MemSQL.com/
  10. 10.
  11. 11.
    Number of smartphones sold to end users worldwide from 2007 to 2020 (in million units). https://www.statista.com/statistics/263437/global-smartphone-sales-to-end-users-since-2007/
  12. 12.
    OMDb API—The Open Movie Database. http://www.omdbapi.com/
  13. 13.
    PostgreSQL: The world’s most advanced open source database. https://www.postgresql.org/
  14. 14.
  15. 15.
  16. 16.
  17. 17.
  18. 18.
    TEDCAS uses the Myo armband to give surgeons gesture control. http://blog.thalmic.com/myo-armband-surgery/
  19. 19.
    TPC Benchmark. http://www.tpc.org/
  20. 20.
    UCI Repository of Machine Learning Databases. https://archive.ics.uci.edu/ml/datasets.html
  21. 21.
    Use Multi-Touch gestures on your Mac. https://support.apple.com/en-us/HT204895
  22. 22.
    Worldwide tablet shipments from 2nd quarter 2010 to 3rd quarter 2018 (in million units). https://www.statista.com/statistics/272070/global-tablet-shipments-by-quarter/
  23. 23.
    Abouzied, A., Hellerstein, J., Silberschatz, A.: Dataplay: Interactive tweaking and example-driven correction of graphical database queries. In: Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology, UIST ’12, New York, NY, USA, pp. 207–218. ACM (2012)Google Scholar
  24. 24.
    Agarwal, S., Mozafari, B., Panda, A., Milner, H., Madden, S., Stoica, I.: Blinkdb: Queries with bounded errors and bounded response times on very large data. In: Proceedings of the 8th ACM European Conference on Computer Systems, EuroSys’13, New York, NY, USA, pp. 29–42. ACM (2013)Google Scholar
  25. 25.
    Al-Megren, S.: A predictive fingerstroke-level model for smartwatch interaction. Multimodal Technol. Interact. 2(3), 38 (2018)CrossRefGoogle Scholar
  26. 26.
    Al-Megren, S., Altamimi, W., Al-Khalifa, H. S.: Blind flm: An enhanced keystroke-level model for visually impaired smartphone interaction. In: IFIP Conference on Human–Computer Interaction, pp. 155–172. Springer, New York (2017)CrossRefGoogle Scholar
  27. 27.
    Albert, W., Tullis, T.: Measuring the user experience: collecting, analyzing, and presenting usability metrics. Newnes, Oxford (2013)Google Scholar
  28. 28.
    Armstrong, T. G., Ponnekanti, V., Borthakur, D., Callaghan, M.: Linkbench: A database benchmark based on the facebook social graph. In: Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data, SIGMOD ’13, New York, NY, USA, pp. 1185–1196. ACM (2013)Google Scholar
  29. 29.
    Bakke, E., Karger, D., Miller, R.: A spreadsheet-based user interface for managing plural relationships in structured data. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI’11, New York, NY, USA, pp. 2541–2550. ACM (2011)Google Scholar
  30. 30.
    Bakke, E., Karger, D.R.: Expressive query construction through direct manipulation of nested relational results. In: Proceedings of the 2016 International Conference on Management of Data, pp. 1377–1392. ACM, (2016)Google Scholar
  31. 31.
    Bangor, A., Kortum, P.T., Miller, J.T.: An empirical evaluation of the system usability scale. Int. J. Hum. Comput. Interact. 24(6), 574–594 (2008)CrossRefGoogle Scholar
  32. 32.
    Barnum, C .M.: Usability Testing Essentials: Ready, Set... Test!. Elsevier, Amsterdam (2010)Google Scholar
  33. 33.
    Basole, R.C., Clear, T., Hu, M., Mehrotra, H., Stasko, J.: Understanding interfirm relationships in business ecosystems with interactive visualization. IEEE Trans. Vis. Comput. Graph. 19(12), 2526–2535 (2013)CrossRefGoogle Scholar
  34. 34.
    Basu Roy, S., Wang, H., Das, G., Nambiar, U., Mohania, M.: Minimum-effort driven dynamic faceted search in structured databases. In: Proceedings of the 17th ACM Conference on Information and Knowledge Management, CIKM’08, New York, NY, USA, pp. 13–22. ACM (2008)Google Scholar
  35. 35.
    Battle, L., Chang, R., Heer, J., Stonebraker, M.: Position statement: the case for a visualization performance benchmark. In: 2017 IEEE Workshop on Data Systems for Interactive Analysis (DSIA), pp. 1–5. (2017)Google Scholar
  36. 36.
    Battle, L., Chang, R., Stonebraker, M.: Dynamic prefetching of data tiles for interactive visualization. pp. 1363–1375, (2016)Google Scholar
  37. 37.
    Bell, D.A., Deluca, L.S., Levinson, D.J., Salem, R.: Browser interaction for lazy loading operations, US Patent App. 14/570,430 (2014)Google Scholar
  38. 38.
    Beltran, J.F., Huang, Z., Abouzied, A., Nandi, A.: Don’t just swipe left, tell me why: enhancing gesture-based feedback with reason bins. In: Proceedings of the 22nd International Conference on Intelligent User Interfaces, pp. 469–480. ACM, (2017)Google Scholar
  39. 39.
    Bendre, M., Venkataraman, V., Zhou, X., Chang, K., Parameswaran, A.: Towards a holistic integration of spreadsheets with databases: a scalable storage engine for presentational data management. In: 2018 IEEE 34th International Conference on Data Engineering (ICDE), pp. 113–124. IEEE, (2018)Google Scholar
  40. 40.
    Benson, B.: Cognitive bias cheat sheet. Better Hum. (2016)Google Scholar
  41. 41.
    Bernard, J., Wilhelm, N., Krüger, B., May, T., Schreck, T., Kohlhammer, J.: Motionexplorer: exploratory search in human motion capture data based on hierarchical aggregation. IEEE Trans. Visual. Comput. Graph. 19(12), 2257–2266 (2013)CrossRefGoogle Scholar
  42. 42.
    Bi, X., Li, Y., Zhai, S.: Ffitts law: modeling finger touch with fitts’ law. In: SIGCHI, (2013)Google Scholar
  43. 43.
    Binnig, C., Fekete, A., Nandi, A.: Hilda’16: Proceedings of the workshop on human-in-the-loop data analytics. New York, NY, USA, ACM (2016)Google Scholar
  44. 44.
    Biswas, A., Dutta, S., Shen, H.-W., Woodring, J.: An information-aware framework for exploring multivariate data sets. IEEE Trans. Vis. Comput. Graph. 19(12), 2683–2692 (2013)CrossRefGoogle Scholar
  45. 45.
    Brooke, J., et al.: Sus-a quick and dirty usability scale. Usability Eval. Ind. 189(194), 4–7 (1996)Google Scholar
  46. 46.
    Burley, C., Nandi, A.: Arquery: Hallucinating analytics over real-world data using augmented reality. In: CIDR, (2019)Google Scholar
  47. 47.
    Cao, N., Gotz, D., Sun, J., Qu, H.: Dicon: interactive visual analysis of multidimensional clusters. IEEE Trans. Vis. Comput. Graph. 17(12), 2581–2590 (2011)CrossRefGoogle Scholar
  48. 48.
    Card, S.K.: The psychology of human–computer interaction. CRC Press, New York (2017)Google Scholar
  49. 49.
    Chan, S.-M., Xiao, L., Gerth, J., Hanrahan, P.: Maintaining interactivity while exploring massive time series. In: 2008 IEEE Symposium on Visual Analytics Science and Technology, pp. 59–66, (2008)Google Scholar
  50. 50.
    Chau, P., Vreeken, J., van Leeuwen, M., Shahaf, D., Faloutsos, C.: Proceedings of the ACM SIGKDD workshop on interactive data exploration and analytics. In: ACM SIGKDD 2016 Full-Day Workshop on Interactive Data Exploration and Analytics. IDEA’16, (2016)Google Scholar
  51. 51.
    Chaudhuri, S., Motwani, R., Narasayya, V.: Random sampling for histogram construction: How much is enough? In: Proceedings of the 1998 ACM SIGMOD International Conference on Management of Data, SIGMOD’98, New York, NY, USA, pp. 436–447. ACM (1998)Google Scholar
  52. 52.
    Chen, K.: Data-driven techniques for improving data collection in low-resource environments. Ph.D. thesis, UC Berkeley, (2011)Google Scholar
  53. 53.
    Chen, Y., Alspaugh, S., Borthakur, D., Katz, R.: Energy efficiency for large-scale mapreduce workloads with significant interactive analysis. In: Proceedings of the 7th ACM European Conference on Computer Systems, EuroSys’12, New York, NY, USA, pp. 43–56. ACM (2012)Google Scholar
  54. 54.
    Chen, Y., Alspaugh, S., Katz, R.: Interactive analytical processing in big data systems: a cross-industry study of mapreduce workloads. Proc. VLDB Endow. 5(12), 1802–1813 (2012)CrossRefGoogle Scholar
  55. 55.
    Cooper, B. F., Silberstein, A., Tam, E., Ramakrishnan, R., Sears, R.: Benchmarking cloud serving systems with YCSB. In: Proceedings of the 1st ACM Symposium on Cloud Computing, SoCC’10, New York, NY, USA, pp. 143–154. ACM (2010)Google Scholar
  56. 56.
    Correll, M., Li, M., Kindlmann, G., Scheidegger, C.: Looks good to me: Visualizations as sanity checks. IEEE Trans. Vis. Comput. Graph. (2018)Google Scholar
  57. 57.
    Demiralp, Ç., Haas, P. J., Parthasarathy, S., Pedapati, T.: Foresight: Rapid Data Exploration Through Guideposts. arXiv preprint arXiv:1709.10513 (2017)MathSciNetCrossRefGoogle Scholar
  58. 58.
    DeWitt, D.J.: The Wisconsin benchmark: past, present, and future, (1993)Google Scholar
  59. 59.
    Dimara, E., Bailly, G., Bezerianos, A., Franconeri, S.: Mitigating the attraction effect with visualizations. IEEE Trans. Vis. Comput. Graph. 25(1), 850–860 (2018)CrossRefGoogle Scholar
  60. 60.
    Dimara, E., Franconeri, S., Plaisant, C., Bezerianos, A., Dragicevic, P.: A task-based taxonomy of cognitive biases for information visualization. IEEE Trans. Vis. Comput. Graph. (2018)Google Scholar
  61. 61.
    Dimitriadou, K., Papaemmanouil, O., Diao, Y.: Explore-by-example: an automatic query steering framework for interactive data exploration. In: Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data, SIGMOD’14, New York, NY, USA, pp. 517–528. ACM (2014)Google Scholar
  62. 62.
    Doshi, P.R., Rundensteiner, E.A., Ward, M.O.: Prefetching for visual data exploration. Eighth Int. Conf. Database Syst. Adv. Appl. 2003, 195–202 (2003)CrossRefGoogle Scholar
  63. 63.
    Drucker, S.M., Fisher, D., Sadana, R., Herron, J. et al.: Touchviz: a case study comparing two interfaces for data analytics on tablets. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 2301–2310. ACM, (2013)Google Scholar
  64. 64.
    Ebenstein, R., Kamat, N., Nandi, A.: Fluxquery: An execution framework for highly interactive query workloads. In: Proceedings of the 2016 International Conference on Management of Data, pp. 1333–1345. ACM, (2016)Google Scholar
  65. 65.
    Eichmann, P., Binnig, C., Kraska, T., Zgraggen, E.: Idebench: A benchmark for interactive data exploration. CoRR, arXiv:1804.02593 (2018)
  66. 66.
    Eichmann, P., Zgraggen, E., Zhao, Z., Binnig, C., Kraska, T.: Towards a benchmark for interactive data exploration. IEEE Data Eng. Bull. 39, 50–61 (2016)Google Scholar
  67. 67.
    El Batran, K., Dunlop, M.D.: Enhancing KLM (keystroke-level model) to fit touch screen mobile devices. In: Proceedings of the 16th International Conference on Human–Computer Interaction with Mobile Devices and ServicesGoogle Scholar
  68. 68.
    Faith, J.: Targeted projection pursuit for interactive exploration of high- dimensional data sets. In: 11th International Conference on Information Visualization, IV’07, pp. 286–292, (2007)Google Scholar
  69. 69.
    Faulkner, L.: Beyond the five-user assumption: benefits of increased sample sizes in usability testing. Behav. Res. Methods Instrum. Comput. 35(3), 379–383 (2003)CrossRefGoogle Scholar
  70. 70.
    Fekete, J.-D.: Progressivis: A toolkit for steerable progressive analytics and visualization. In: 1st Workshop on Data Systems for Interactive Analysis, p. 5, (2015)Google Scholar
  71. 71.
    Fekete, J.-D., Plaisant, C.: Interactive information visualization of a million items. In: Proceedings of the IEEE Symposium on Information Visualization (InfoVis’02), INFOVIS’02, IEEE Computer Society, p. 117, Washington, DC, USA (2002)Google Scholar
  72. 72.
    Feng, S., Huber, A., Glavic, B., Kennedy, O.: Uncertainty annotated databases-a lightweight approach for approximating certain answers. In: Proceedings of the 2019 International Conference on Management of Data, pp. 1313–1330. ACM, (2019)Google Scholar
  73. 73.
    Ferreira, N., Poco, J., Vo, H.T., Freire, J., Silva, C.T.: Visual exploration of big spatio-temporal urban data: a study of new york city taxi trips. IEEE Trans. Vis. Comput. Graph. 19(12), 2149–2158 (2013)CrossRefGoogle Scholar
  74. 74.
    Fisher, D., Popov, I., Drucker, S., schraefel, M.: Trust me, I’m partially right: incremental visualization lets analysts explore large datasets faster. In: ACM Conference on Human Factors in Computing Systems, (2012)Google Scholar
  75. 75.
    Fitts, P.M.: The information capacity of the human motor system in controlling the amplitude of movement. J. Exp. Psychol. 47(6), 381 (1954)CrossRefGoogle Scholar
  76. 76.
    Gibbons, P. B., Matias, Y., Poosala, V.:. Fast incremental maintenance of approximate histograms, vol. 27, New York, NY, USA, pp. 261–298. ACM (2002)Google Scholar
  77. 77.
    Gray, J.: Benchmark Handbook: For Database and Transaction Processing Systems. Morgan Kaufmann Publishers Inc., San Francisco (1992)zbMATHGoogle Scholar
  78. 78.
    Gujarathi, N.R., Shah, A.A.: Parameterized computed scrolling for navigation of structured data, (2015)Google Scholar
  79. 79.
    Gunopulos, D., Kollios, G., Tsotras, V.J., Domeniconi, C.: Approximating multi-dimensional aggregate range queries over real attributes. In: Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data, SIGMOD’00, New York, NY, USA, pp. 463–474. ACM (2000)Google Scholar
  80. 80.
    Halim, F., Idreos, S., Karras, P., Yap, R.H.C.: Stochastic database cracking: Towards robust adaptive indexing in main-memory column-stores. Proc. VLDB Endow. 5(6), 502–513 (2012)CrossRefGoogle Scholar
  81. 81.
    Haselton, M.G., Nettle, D., Murray, D.R.: The evolution of cognitive bias. In: The Handbook of Evolutionary Psychology, pp. 1–20, (2015)Google Scholar
  82. 82.
    Hebert, C., Ridgway, J., Vekhter, B., Brown, E.C., Weber, S.G., Robicsek, A.: Demonstration of the weighted-incidence syndromic combination antibiogram: an empiric prescribing decision aid. Infect. Control Hosp. Epidemiol. 33(4), 381–388 (2012)CrossRefGoogle Scholar
  83. 83.
    Heer, J., Agrawala, M., Willett, W.: Generalized selection via interactive query relaxation. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI’08, New York, NY, USA, pp. 959–968. ACM (2008)Google Scholar
  84. 84.
    Heer, J., Bostock, M.: Crowdsourcing graphical perception: using mechanical turk to assess visualization design. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 3–212. ACM, (2010)Google Scholar
  85. 85.
    Hellerstein, J.M., Haas, P.J., Wang, H.J.: Online aggregation. In: Acm Sigmod Record, vol. 6, pp. 1–182. ACM, (1997)Google Scholar
  86. 86.
    Hirte, S., Seifert, A., Baumann, S., Klan, D., Sattler, K.-U.: Data3—a kinect interface for OLAP using complex event processing. In: Proceedings of the 2012 IEEE 28th International Conference on Data Engineering, ICDE’12, IEEE Computer Society , Washington, DC, USA, pp. 1297–1300 (2012)Google Scholar
  87. 87.
    Holleis, P., Otto, F., Hussmann, H., Schmidt, A.: Keystroke-level model for advanced mobile phone interaction. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1505–1514. ACM, (2007)Google Scholar
  88. 88.
    Hu, K., Orghian, D., Hidalgo, C.: DIVE: a mixed-initiative system supporting integrated data exploration workflows. In: Proceedings of the Workshop on Human-In-the-Loop Data Analytics, p. 5. ACM, (2018)Google Scholar
  89. 89.
    Idreos, S.: Database cracking: towards auto-tuning database kernels. CWI and University of Amsterdam, (2010)Google Scholar
  90. 90.
    Idreos, S., Manegold, S., Kuno, H., Graefe, G.: Merging what’s cracked, cracking what’s merged: adaptive indexing in main-memory column-stores. Proc. VLDB Endow. 4(9), 586–597 (2011)CrossRefGoogle Scholar
  91. 91.
    Idreos, S., Papaemmanouil, O., Chaudhuri, S.: Overview of data exploration techniques. In: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, SIGMOD’15, New York, NY, USA, pp. 277–281. ACM (2015)Google Scholar
  92. 92.
    Igarashi, T., Hinckley, K.: Speed-dependent automatic zooming for browsing large documents. In: Proceedings of the 13th Annual ACM Symposium on User Interface Software and Technology, UIST’00, New York, NY, USA, pp. 139–148. ACM (2000)Google Scholar
  93. 93.
    Jagadish, H.V., Chapman, A., Elkiss, A., Jayapandian, M., Li, Y., Nandi, A., Yu, C.: Making database systems usable. In: Proceedings of the 2007 ACM SIGMOD International Conference on Management of Data, SIGMOD’07, New York, NY, USA, pp. 13–24. ACM (2007)Google Scholar
  94. 94.
    Javed, W., Ghani, S., Elmqvist, N.: Gravnav: Using a gravity model for multi-scale navigation. In: Proceedings of the International Working Conference on Advanced Visual Interfaces, AVI’12, New York, NY, USA, pp. 217–224. ACM (2012)Google Scholar
  95. 95.
    Jiang, L., Nandi, A.: Snaptoquery: Providing interactive feedback during exploratory query specification. Proc. VLDB Endow. 8(11), 1250–1261 (2015)CrossRefGoogle Scholar
  96. 96.
    John, B.E., Kieras, D.E.: Using GOMS for user interface design and evaluation: Which technique? TOCHI 3(4), 287–319 (1996)CrossRefGoogle Scholar
  97. 97.
    Jota, R., Ng, A., Dietz, P., Wigdor, D.: How fast is fast enough?: a study of the effects of latency in direct-touch pointing tasks. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 2291–2300. ACM, (2013)Google Scholar
  98. 98.
    Kamat, N., Jayachandran, P., Tunga, K., Nandi, A.: Distributed and interactive cube exploration. In 2014 IEEE 30th International Conference on Data Engineering, pp. 472–483, (2014)Google Scholar
  99. 99.
    Kamat, N., Nandi, A.: Infiniviz: Interactive visual exploration using progressive bin refinement. arXiv preprint arXiv:1710.01854 (2017)
  100. 100.
    Kamat, N., Nandi, A.: A session-based approach to fast-but-approximate interactive data cube exploration. ACM Trans. Knowl. Discov Data (TKDD) 12(1), 9 (2018)Google Scholar
  101. 101.
    Kandel, S., Paepcke, A., Hellerstein, J., Heer, J.: Wrangler: Interactive visual specification of data transformation scripts. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 3363–3372. ACM, (2011)Google Scholar
  102. 102.
    Kandel, S., Paepcke, A., Hellerstein, J., Heer, J.: Wrangler: Interactive visual specification of data transformation scripts. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI’11, New York, NY, USA, pp. 3363–3372. ACM (2011)Google Scholar
  103. 103.
    Kang, J., Naughton, J.F., Viglas, S.D.: Evaluating window joins over unbounded streams. In: Proceedings 19th International Conference on Data Engineering (Cat. No.03CH37405), pp. 341–352, (2003)Google Scholar
  104. 104.
    Kashyap, A., Hristidis, V., Petropoulos, M.: Facetor: Cost-driven exploration of faceted query results. In: Proceedings of the 19th ACM International Conference on Information and Knowledge Management, CIKM’10, New York, NY, USA, pp. 719–728. ACM (2010)Google Scholar
  105. 105.
    Kashyap, A., Hristidis, V., Petropoulos, M., Tavoulari, S.: Effective navigation of query results based on concept hierarchies. IEEE Trans. Knowl. Data Eng. 23(4), 540–553 (2011)CrossRefGoogle Scholar
  106. 106.
    Kaul, M., Yang, B., Jensen, C.S.: Building accurate 3D spatial networks to enable next generation intelligent transportation systems. In 2013 IEEE 14th International Conference on Mobile Data Management, vol. 1, pp. 137–146, (2013)Google Scholar
  107. 107.
    Keim, D.A.: Information visualization and visual data mining. IEEE Trans. Visual. Comput. Graph. 8(1), 1–8 (2002)MathSciNetCrossRefGoogle Scholar
  108. 108.
    Kennedy, O., Ajay, J., Challen, G., Ziarek, L.: Pocket data: the need for TPC-mobile. In: Nambiar, R., Poess, M. (eds.) Performance Evaluation and Benchmarking: Traditional to Big Data to Internet of Things, pp. 8–25. Springer, Cham (2016)CrossRefGoogle Scholar
  109. 109.
    Key, A., Howe, B., Perry, D., Aragon, C.: Vizdeck: self-organizing dashboards for visual analytics. In: Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, SIGMOD’12, New York, NY, USA, pp. 681–684. ACM (2012)Google Scholar
  110. 110.
    Khan, M., Xu, L., Nandi, A., Hellerstein, J.M.: Data tweening: incremental visualization of data transforms. Proc. VLDB Endow. 10(6), 661–672 (2017)CrossRefGoogle Scholar
  111. 111.
    Kim, A., Blais, E., Parameswaran, A., Indyk, P., Madden, S., Rubinfeld, R.: Rapid sampling for visualizations with ordering guarantees. Proc. VLDB Endow. 8(5), 521–532 (2015)CrossRefGoogle Scholar
  112. 112.
    Kim, J.-M., Kim, J.-S.: Androbench: benchmarking the storage performance of android-based mobile devices. In: Sambath, S., Zhu, E. (eds.) Frontiers in Computer Education, pp. 667–674. Springer, Berlin (2012)CrossRefGoogle Scholar
  113. 113.
    Kosara, R., Bendix, F., Hauser, H.: Parallel sets: interactive exploration and visual analysis of categorical data. IEEE Trans. Vis. Comput. Graph. 12(4), 558–568 (2006)CrossRefGoogle Scholar
  114. 114.
    Kraiss, A., Weikum, G.: Integrated document caching and prefetching in storage hierarchies based on markov-chain predictions. VLDB J. 7(3), 141–162 (1998)CrossRefGoogle Scholar
  115. 115.
    Kraska, T.: Northstar: an interactive data science system. Proc. VLDB Endow. 11(12), 2150–2164 (2018)CrossRefGoogle Scholar
  116. 116.
    Kullback, S.: Information Theory and Statistics. Courier Corporation, Chelmsford (1997)zbMATHGoogle Scholar
  117. 117.
    Lam, H., Bertini, E., Isenberg, P., Plaisant, C., Carpendale, S.: Empirical studies in information visualization: seven scenarios. IEEE Trans. Vis. Comput. Graph. 18(9), 1520–1536 (2012)CrossRefGoogle Scholar
  118. 118.
    Lazar, J., Feng, J.H., Hochheiser, H.: Research Methods in Human–Computer Interaction. Wiley, Hoboken (2010)Google Scholar
  119. 119.
    Lee, A., Song, K., Ryu, H.B., Kim, J., Kwon, G.: Fingerstroke time estimates for touchscreen-based mobile gaming interaction. Hum. Mov. Sci. 44, 211–224 (2015)CrossRefGoogle Scholar
  120. 120.
    Lee, D.H., Kim, J.S., Kim, S.D., Kim, K.C., Yoo-Sung, K., Park, J.: Adaptation of a neighbor selection markov chain for prefetching tiled web gis data. In: Yakhno, T. (ed.) Advances in Information Systems, pp. 213–222. Springer, Berlin (2002)CrossRefGoogle Scholar
  121. 121.
    Li, Y., Yang, H., Jagadish, H.V.: Nalix: a generic natural language search environment for xml data. ACM Trans. Database Syst. 32(4), 30 (2007)CrossRefGoogle Scholar
  122. 122.
    Liarou, E., Idreos, S.: dbtouch in action database kernels for touch-based data exploration. In 2014 IEEE 30th International Conference on Data Engineering, pp. 1262–1265, (2014)Google Scholar
  123. 123.
    Lins, L., Klosowski, J.T., Scheidegger, C.: Nanocubes for real-time exploration of spatiotemporal datasets. IEEE Trans. Vis. Comput. Graph. 19(12), 2456–2465 (2013)CrossRefGoogle Scholar
  124. 124.
    Liu, B., Jagadish, H.: A spreadsheet algebra for a direct data manipulation query interface. In: 2009 IEEE 25th International Conference on Data Engineering, pp. 417–428, (2009)Google Scholar
  125. 125.
    Liu, F., Kamat, N., Blanas, S., Nandi, A.: To ship or not to (function) ship (extended version). In: 2018 IEEE High Performance Extreme Computing Conference (HPEC). IEEE, (2018)Google Scholar
  126. 126.
    Liu, Z., Heer, J.: The effects of interactive latency on exploratory visual analysis. IEEE Trans. Vis. Comput. Graph. 20(12), 2122–2131 (2014)CrossRefGoogle Scholar
  127. 127.
    Liu, Z., Jiang, B., Heer, J.: immens: real-time visual querying of big data. Comput. Graph. Forum 32(3–4), 421–430 (2013)CrossRefGoogle Scholar
  128. 128.
    Mackinlay, J., Hanrahan, P., Stolte, C.: Show me: automatic presentation for visual analysis. IEEE Trans. Vis. Comput. Graph. 13(6), 1137–1144 (2007)CrossRefGoogle Scholar
  129. 129.
    Martin, A.R., Ward, M.O.: High dimensional brushing for interactive exploration of multivariate data. In: Proceedings of the 6th Conference on Visualization’95, VIS’95, IEEE Computer Society, Washington, DC, USA, p. 271 (1995)Google Scholar
  130. 130.
    McLachlan, P., Munzner, T., Koutsofios, E., North, S.: Liverac: Interactive visual exploration of system management time-series data. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI’08, New York, NY, USA, pp. 1483–1492. ACM (2008)Google Scholar
  131. 131.
    Mohan, C.: Caching technologies for web applications. In: VLDB, vol. 1, p. 726, (2001)Google Scholar
  132. 132.
    Moritz, D., Fisher, D., Ding, B., Wang, C.: Trust, but verify: Optimistic visualizations of approximate queries for exploring big data. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pp. 2904–2915. ACM, (2017)Google Scholar
  133. 133.
    Moritz, D., Howe, B., Heer, J.: Falcon: Balancing interactive latency and resolution sensitivity for scalable linked visualizations. (2019)Google Scholar
  134. 134.
    Mostak, T.: An overview of MAPD (massively parallel database). White paper, Massachusetts Institute of Technology (2013)Google Scholar
  135. 135.
    Moussavi, F.: Hybrid inertial and touch sensing input device, US Patent App. 12/192,889 (2010)Google Scholar
  136. 136.
    Munzner, T.: A nested model for visualization design and validation. IEEE Trans. Vis. Comput. Graph. 15(6), 921–928 (2009)CrossRefGoogle Scholar
  137. 137.
    Nandi, A., Jiang, L., Mandel, M.: Gestural query specification. Proc. VLDB Endow. 7(4), 289–300 (2013)CrossRefGoogle Scholar
  138. 138.
    Nelson, W.T., Roe, M.M., Bolia, R.S., Morley, R.M.: Assessing simulator sickness in a see-through HMD: Effects of time delay, time on task, and task complexity. Technical report, air force research lab Wright-Patterson AFB OH (2000)Google Scholar
  139. 139.
    Nourbakhsh, N., Wang, Y., Chen, F., Calvo, R.A.: Using galvanic skin response for cognitive load measurement in arithmetic and reading tasks. In: Proceedings of the 24th Australian Computer-Human Interaction Conference, OzCHI’12, New York, NY, USA, pp. 420–423. ACM (2012)Google Scholar
  140. 140.
    Omidvar-Tehrani, B., Nandi, A., Meyer, N., Flanagan, D., Young, S.: Dv8: Interactive analysis of aviation data. In: 2017 IEEE 33rd International Conference on Data Engineering (ICDE), pp. 1411–1412. IEEE, (2017)Google Scholar
  141. 141.
    Padilla, L.: A case for cognitive models in visualization research. In: Proceedings of the Seventh Workshop on Beyond Time and Errors on Novel Evaluation Methods for Visualization, pp. 143–151, (2018)Google Scholar
  142. 142.
    Patterson, E.S., Dewart, C.M., Stevenson, K., Mbodj, A., Lustberg, M., Hade, E.M., Hebert, C.: A mixed methods approach to tailoring evidence-based guidance for antibiotic stewardship to one medical system. In: Proceedings of the International Symposium on Human Factors and Ergonomics in Health Care, vol. 7, pp. 224–231. SAGE Publications Sage India: New Delhi (2018)CrossRefGoogle Scholar
  143. 143.
    Pavlovych, A., Gutwin, C.: Assessing target acquisition and tracking performance for complex moving targets in the presence of latency and jitter. In: Proceedings of Graphics Interface 2012, pp. 109–116. Canadian Information Processing Society, (2012)Google Scholar
  144. 144.
    Plaisant, C.: The challenge of information visualization evaluation. In: Proceedings of the Working Conference on Advanced Visual Interfaces, AVI’04, New York, NY, USA, pp. 109–116. ACM (2004)Google Scholar
  145. 145.
    Poosala, V., Haas, P.J., Ioannidis, Y.E., Shekita, E.J.: Improved histograms for selectivity estimation of range predicates. In: Proceedings of the 1996 ACM SIGMOD International Conference on Management of Data, SIGMOD’96, New York, NY, USA, pp. 294–305. ACM (1996)Google Scholar
  146. 146.
    Psallidas, F., Wu, E.: Smoke: fine-grained lineage at interactive speed. Proc. VLDB Endow. 11(6), 719–732 (2018)Google Scholar
  147. 147.
    Quezada, A., Juárez-Ramírez, R., Jiménez, S., Ramírez-Noriega, A., Inzunza, S., Munoz, R.: Assessing the target? Size and drag distance in mobile applications for users with autism. In: World Conference on Information Systems and Technologies, pp. 1219–1228. Springer, New York, (2018)Google Scholar
  148. 148.
    Rahman, P., Hebert, C., Nandi, A.: ICARUS: minimizing human effort in iterative data completion. PVLDB, 11(13), Preprint available at go.osu.edu/icarus (2018)Google Scholar
  149. 149.
    Rahman, P., Nandi, A.: Transformer: A database-driven approach to constrained generating forms for constrained interaction. In: Proceedings of the 24th International Conference on Intelligent User Interfaces. ACM, Preprint available at cse.osu.edu/rahman.92 (2019)Google Scholar
  150. 150.
    Rahman, S., Aliakbarpour, M., Kong, H.K., Blais, E., Karahalios, K., Parameswaran, A., Rubinfield, R.: I’ve seen enough: incrementally improving visualizations to support rapid decision making. Proc. VLDB Endow. 10(11), 1262–1273 (2017)CrossRefGoogle Scholar
  151. 151.
    Rosa, G.M., Elizondo, M.L.: Use of a gesture user interface as a touchless image navigation system in dental surgery: case series report. Imaging Sci. Dent. 44(2), 155–160 (2014)CrossRefGoogle Scholar
  152. 152.
    Rzeszotarski, J.M., Kittur, A.: Kinetica: Naturalistic multi-touch data visualization. In: Proceedings of the 32Nd Annual ACM Conference on Human Factors in Computing Systems, CHI’14, New York, NY, USA, pp. 897–906. ACM (2014)Google Scholar
  153. 153.
    Saadé, R.G., Otrakji, C.A.: First impressions last a lifetime: effect of interface type on disorientation and cognitive load. Comput. Hum. Behav. 23(1), 525–535 (2007)CrossRefGoogle Scholar
  154. 154.
    Satyanarayan, A., Heer, J.: Lyra: An interactive visualization design environment. In: Proceedings of the 16th Eurographics Conference on Visualization, EuroVis’14, pp. 351–360. Aire-la-Ville, Switzerland, Eurographics Association (2014)Google Scholar
  155. 155.
    Seo, J., Shneiderman, B.: A rank-by-feature framework for interactive exploration of multidimensional data. Inf. Vis. 4(2), 96–113 (2005)CrossRefGoogle Scholar
  156. 156.
    Shneiderman, B.: The eyes have it: A task by data type taxonomy for information visualizations. In: Proceedings of the 1996 IEEE Symposium on Visual Languages, VL’96, IEEE Computer Society , p. 336, Washington, DC, USA (1996)Google Scholar
  157. 157.
    Shneiderman, B., Williamson, C., Ahlberg, C.: Dynamic queries: database searching by direct manipulation. pp. 669–670, (1992)Google Scholar
  158. 158.
    Siddiqui, T., Kim, A., Lee, J., Karahalios, K., Parameswaran, A.: Effortless data exploration with zenvisage: an expressive and interactive visual analytics system. Proc. VLDB Endow. 10(4), 457–468 (2016)CrossRefGoogle Scholar
  159. 159.
    Sidirourgos, L., Kersten, M.L., Boncz, P.A. et al.: Sciborq: Scientific data management with bounds on runtime and quality. In: CIDR, (2011)Google Scholar
  160. 160.
    Singh, M., Nandi, A., Jagadish, H.V.: Skimmer: Rapid scrolling of relational query results. In: Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, SIGMOD’12, New York, NY, USA, pp. 181–192. ACM (2012)Google Scholar
  161. 161.
    St Amant, R., Horton, T.E., Ritter, F.E.: Model-based evaluation of cell phone menu interaction. In: CHI, (2004)Google Scholar
  162. 162.
    Sweller, J.: Cognitive load during problem solving: effects on learning. Cognit. Sci. 12(2), 257–285 (1988)CrossRefGoogle Scholar
  163. 163.
    Tan, P.: BMW demonstrates future iDrive with Touchscreen, Gesture and Tablet Control. CES 2015, (2015)Google Scholar
  164. 164.
    Tauheed, F., Heinis, T., Schürmann, F., Markram, H., Ailamaki, A.: Scout: prefetching for latent structure following queries. Proc. VLDB Endow. 5(11), 1531–1542 (2012)CrossRefGoogle Scholar
  165. 165.
    Valdez, A.C., Ziefle, M., Sedlmair, M.: A framework for studying biases in visualization research. (2017)Google Scholar
  166. 166.
    Van Dillen, L.F., Heslenfeld, D.J., Koole, S.L.: Tuning down the emotional brain: an fmri study of the effects of cognitive load on the processing of affective images. Neuroimage 45(4), 1212–1219 (2009)CrossRefGoogle Scholar
  167. 167.
    Vartak, M., Madden, S., Parameswaran, A., Polyzotis, N.: Seedb: automatically generating query visualizations. Proc. VLDB Endow. 7(13), 1581–1584 (2014)CrossRefGoogle Scholar
  168. 168.
    Viglas, S.D., Naughton, J.F., Burger, J.: Maximizing the output rate of multi-way join queries over streaming information sources. In: Proceedings of the 29th International Conference on Very Large Data Bases—Volume 29, VLDB’03, VLDB Endowment, pp. 285–296. (2003)CrossRefGoogle Scholar
  169. 169.
    Vitter, J.S., Wang, M., Iyer, B.: Data cube approximation and histograms via wavelets. In: Proceedings of the Seventh International Conference on Information and Knowledge Management, CIKM’98, New York, NY, USA, pp. 96–104. ACM (1998)Google Scholar
  170. 170.
    Wall, E., Agnihotri, M., Matzen, L., Divis, K., Haass, M., Endert, A., Stasko, J.: A heuristic approach to value-driven evaluation of visualizations. IEEE Trans. Vis. Comput. Graph. 25(1), 491–500 (2018)CrossRefGoogle Scholar
  171. 171.
    Wang, L., Zhan, J., Luo, C., Zhu, Y., Yang, Q., He, Y., Gao, W., Jia, Z., Shi, Y., Zhang, S., Zheng, C., Lu, G., Zhan, K., Li, X., Qiu, B.: Bigdatabench: a big data benchmark suite from internet services. In: 2014 IEEE 20th International Symposium on High Performance Computer Architecture (HPCA), pp. 488–499, (2014)Google Scholar
  172. 172.
    Weaver, C.: Multidimensional visual analysis using cross-filtered views. In: 2008 IEEE Symposium on Visual Analytics Science and Technology, pp. 163–170, (2008)Google Scholar
  173. 173.
    Wei, J., Shen, Z., Sundaresan, N., Ma, K.-L.: Visual cluster exploration of web clickstream data. In: 2012 IEEE Conference on Visual Analytics Science and Technology (VAST), pp. 3–12, (2012)Google Scholar
  174. 174.
    Wilkinson, L.: The Grammar of Graphics. Springer, Berlin (2006)zbMATHGoogle Scholar
  175. 175.
    Willett, W., Heer, J., Agrawala, M.: Scented widgets: improving navigation cues with embedded visualizations. IEEE Trans. Vis. Comput. Graph. 13(6), 1129–1136 (2007)CrossRefGoogle Scholar
  176. 176.
    Wongsuphasawat, K., Moritz, D., Anand, A., Mackinlay, J., Howe, B., Heer, J.: Voyager: exploratory analysis via faceted browsing of visualization recommendations. IEEE Trans. Vis. Comput. Graph. 22(1), 649–658 (2016)CrossRefGoogle Scholar
  177. 177.
    Wongsuphasawat, K., Qu, Z., Moritz, D., Chang, R., Ouk, F., Anand, A., Mackinlay, J., Howe, B., Heer, J.: Voyager 2: Augmenting visual analysis with partial view specifications. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pp. 2648–2659. ACM, (2017)Google Scholar
  178. 178.
    Wongsuphasawat, K., Smilkov, D., Wexler, J., Wilson, J., Mané, D., Fritz, D., Krishnan, D., Viégas, F.B., Wattenberg, M.: Visualizing dataflow graphs of deep learning models in tensorflow. IEEE Trans. Vis. Comput. Graph. 24(1), 1–12 (2018)CrossRefGoogle Scholar
  179. 179.
    Woodring, J., Shen, H.-W.: Multiscale time activity data exploration via temporal clustering visualization spreadsheet. IEEE Trans. Vis. Comput. Graph. 15(1), 123–137 (2009)CrossRefGoogle Scholar
  180. 180.
    Wu, E., Jiang, L., Xu, L., Nandi, A.: Graphical perception in animated bar charts. arXiv preprint arXiv:1604.00080 (2016)
  181. 181.
    Wu, E., Jiang, L., Xu, L., Nandi, A.: Graphical perception in animated bar charts. volume arXiv:1604.00080 (2016)
  182. 182.
    Wu, Y., Chang, R., Wu, E., Hellerstein, J.: Programming with timespans in interactive visualizations. arXiv preprint arXiv:1907.00075 (2019)
  183. 183.
    Yakout, M., Elmagarmid, A.K., Neville, J., Ouzzani, M., Ilyas, I.F.: Guided data repair. Proc. VLDB Endow. 4(5), 279–289 (2011)CrossRefGoogle Scholar
  184. 184.
    Yang, D., Guo, Z., Rundensteiner, E.A., Ward, M.O.: Clues: A unified framework supporting interactive exploration of density-based clusters in streams. In: Proceedings of the 20th ACM International Conference on Information and Knowledge Management, CIKM’11, New York, NY, USA, pp. 815–824. ACM (2011)Google Scholar
  185. 185.
    Yang, D., Rundensteiner, E.A., Ward, M.O.: Analysis guided visual exploration of multivariate data. In 2007 IEEE Symposium on Visual Analytics Science and Technology, pp. 83–90, (2007)Google Scholar
  186. 186.
    Yang, J., Patro, A., Huang, S., Mehta, N., Ward, M.O., Rundensteiner, E.A.: Value and relation display for interactive exploration of high dimensional datasets. In: IEEE Symposium on Information Visualization, pp. 73–80, (2004)Google Scholar
  187. 187.
    Yang, J., Peng, W., Ward, M.O., Rundensteiner, E.A.: Interactive hierarchical dimension ordering, spacing and filtering for exploration of high dimensional datasets. In: Proceedings of the Ninth Annual IEEE Conference on Information Visualization, INFOVIS’03, IEEE Computer Society, Washington, DC, USA, pp. 105–112. (2003)Google Scholar
  188. 188.
    Yeşilmurat, S., İşler, V.: Retrospective adaptive prefetching for interactive web GIS applications. GeoInformatica 16(3), 435–466 (2012)CrossRefGoogle Scholar
  189. 189.
    Yi, J.S., Kang, Y ah, Stasko, J.: Toward a deeper understanding of the role of interaction in information visualization. IEEE Trans. Vis. Comput. Graph. 13(6), 1224–1231 (2007)CrossRefGoogle Scholar
  190. 190.
    Yuan, X., Ren, D., Wang, Z., Guo, C.: Dimension projection matrix/tree: interactive subspace visual exploration and analysis of high dimensional data. IEEE Trans. Vis. Comput. Graph. 19(12), 2625–2633 (2013)CrossRefGoogle Scholar
  191. 191.
    Yuan, Y., Wang, G., Chen, L., Wang, H.: Graph similarity search on large uncertain graph databases. VLDB J. Int. J. Very Large Data Bases 24(2), 271–296 (2015)CrossRefGoogle Scholar
  192. 192.
    Yuan, Y., Wang, G., Wang, H., Chen, L.: Efficient subgraph search over large uncertain graphs. Proc. VLDB Endow. 4(11), 876–886 (2011)Google Scholar
  193. 193.
    Yuan, Y., Wang, G., Xu, J.Y., Chen, L.: Efficient distributed subgraph similarity matching. VLDB J. Int. J. Very Large Data Bases 24(3), 369–394 (2015)CrossRefGoogle Scholar
  194. 194.
    Zarifis, K., Papakonstantinou, Y.: ViDeTTe Interactive Notebooks. In: Proceedings of the Workshop on Human-In-the-Loop Data Analytics, p. 2. ACM, (2018)Google Scholar
  195. 195.
    Zgraggen, E., Zeleznik, R., Drucker, S.M.: Panoramicdata: data analysis through pen & touch. IEEE Trans. Vis. Comput. Graph. 20(12), 2112–2121 (2014)CrossRefGoogle Scholar
  196. 196.
    Zhang, Z., McDonnell, K. T., Mueller, K.: A network-based interface for the exploration of high-dimensional data spaces. In: Proceedings of the 2012 IEEE Pacific Visualization Symposium, PACIFICVIS’12, IEEE Computer Society, Washington, DC, USA, pp. 17–24, (2012)Google Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Computer Science and EngineeringThe Ohio State UniversityColumbusUSA
  2. 2.Twitter, Inc.San FranciscoUSA

Personalised recommendations