Mixer: Mixed-Initiative Data Retrieval and Integration by Example

  • Steven Gardiner
  • Anthony Tomasic
  • John Zimmerman
  • Rafae Aziz
  • Kathryn Rivard
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6946)


Office administrators are frequently asked to create ad hoc reports based on web accessible data. The web contains the desired data but does not allow efficient access in the way the administrator needs, prompting a tedious and labor-intensive task of retrieving and integrating the required data. Mixer is a programming-by-demonstration (PBD) tool empowering administrators to construct ad hoc reports from diverse web sources without tedious piecemeal labor. Mixer’s design builds on the exploration into end user conceptualization of data retrieval tasks from our previous Wizard-of-Oz study [39], and incorporates insights from mixed-initiative researchers into collaboration between end users and software agents. This paper justifies the design decisions that drive Mixer, focusing on general lessons for designers of programming-by-demonstration systems targeting nonprogrammers. We evaluate Mixer by performing a user study showing that administrators are able to accomplish programming tasks without needing to understand programming concepts for data retrieval and integration.


programming by demonstration end user programming mixed initiative data integration 

Supplementary material

Electronic Supplementary material (20,271 KB)


  1. 1.
    Mixed-Initiative Interaction. IEEE Intelligent Systems 14, 14–23 (1999) Google Scholar
  2. 2.
    Allen, J., Chambers, N., Ferguson, G., Galescu, L., Jung, H., Taysom, W.: PLOW: A Collaborative Task Learning Agent. In: AAAI, pp. 22–26 (2007) Google Scholar
  3. 3.
    Beemer, B., Gregg, D.: Mashups: A Literature Review and Classification Framework. Future Internet 1(1), 59–87 (2009)CrossRefGoogle Scholar
  4. 4.
    Bigham, J.P., Kaminsky, R.S., Nichols, J.: Mining web interactions to automatically create mash-ups. In: UIST, pp. 203–212 (2009)Google Scholar
  5. 5.
    Bolin, M.: End-User Programming for the Web, Masters Thesis. MIT (2005) Google Scholar
  6. 6.
    Boodman, A.: Greasemonkey,
  7. 7.
    Clark, J., DeRose, S.: XML Path Language (XPath) Version 1.0, W3C (1999) Google Scholar
  8. 8.
    Cypher, A.: Eager: Programming Repetitive Tasks by Demonstration. In: Watch What I Do: Programming by Demonstration, pp. 205–217. MIT Press, Cambridge (1993)Google Scholar
  9. 9.
    Cypher, A.: End User Programming on the Web. In: No Code Required: Giving Users Tools to Transform the Web, pp. 3–22. Morgan Kaufmann, San Francisco (2010)Google Scholar
  10. 10.
    Dennett, D.C.: The Intentional Stance. The MIT Press, Cambridge (1987)Google Scholar
  11. 11.
    Dontcheva, M., Drucker, S.M., Salesin, D., Cohen, M.F.: Relations, cards, and search templates: user-guided web data integration and layout. In: UIST, pp. 61–70 (2007) Google Scholar
  12. 12.
    Dontcheva, M., Drucker, S.M., Wade, G., Salesin, D., Cohen, M.F.: Summarizing personal web browsing sessions. In: UIST, pp. 115–124 (2006) Google Scholar
  13. 13.
    Ennals, R., Brewer, E., Garofalakis, M., Shadle, M., Gandhi, P.: Intel Mash Maker: join the web. SIGMOD Rec. 36(4), 27–33 (2007)CrossRefGoogle Scholar
  14. 14.
    Horvitz, E.: Reflections on Challenges and Promises of Mixed-Initiative Interaction. AI Magazine 28(2) (2007)Google Scholar
  15. 15.
    Huynh, D., Miller, R., Karger, D.: Potluck: Data Mash-Up Tool for Casual Users. In: Aberer, K., Choi, K.-S., Noy, N., Allemang, D., Lee, K.-I., Nixon, L.J.B., Golbeck, J., Mika, P., Maynard, D., Mizoguchi, R., Schreiber, G., Cudré-Mauroux, P. (eds.) The Semantic Web. LNCS, vol. 4825, pp. 903–910. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  16. 16.
    Kandel, S., Paepcke, A., Hellerstein, J., Heer, J.: Wrangler: Interactive Visual Specification of Data Transformation Scripts. In: CHI (2011) Google Scholar
  17. 17.
    Kawanaka, S., Borodin, Y., Bigham, J. P., Lunn, D., Takagi, H., Asakawa, C.: Accessibility commons: a metadata infrastructure for web accessibility. In: Assets, pp. 153–160 (2008) Google Scholar
  18. 18.
    Kosbie, D.S., Myers, B.A.: PBD Invocation Techniques: A Review and Proposal. In: Watch What I Do: Programming by Demonstration, pp. 415–422. MIT Press, Cambridge (1993)Google Scholar
  19. 19.
    Lau, T., Wolfman, S.A., Domingos, P., Weld, D.S.: Programming by Demonstration Using Version Space Algebra. Mach. Learn. 53(1-2), 111–156 (2003)zbMATHCrossRefGoogle Scholar
  20. 20.
    Leshed, G., Haber, E.M., Matthews, T., Lau, T.A.: CoScripter: automating & sharing how-to knowledge in the enterprise. In: CHI, pp. 1719–1728 (2008)Google Scholar
  21. 21.
    Lewis, C., Olson, G.: Can principles of cognition lower the barriers to programming? In: Empirical Studies of Programmers: Second Workshop, pp. 248–263. Ablex, Norwood (1987)Google Scholar
  22. 22.
    Lin, J., Wong, J., Nichols, J., Cypher, A., Lau, T.A.: End-user programming of mashups with vegemite. In: IUI, pp. 97–106 (2009) Google Scholar
  23. 23.
    Malone, T.W., Grant, K.R., Lai, K.-Y., Rao, R., Rosenblitt, D.: Semistructured Messages Are Surprisingly Useful for Computer-Supported Coordination. ACM Trans. Inf. Syst. 5(2), 115–131 (1987)CrossRefGoogle Scholar
  24. 24.
    Maulsby, D., Witten, I.H.: Metamouse: An Instructible Agent for PBD. In: Watch What I Do: Programming by Demonstration, pp. 155–181. MIT Press, Cambridge (1993)Google Scholar
  25. 25.
    Modugno, F., Myers, B.: Graphical Representation and Feedback in a PBD System. In: Watch What I Do: Programming by Demonstration, pp. 415–422. MIT Press, Cambridge (1993)Google Scholar
  26. 26.
    Myers, B.A.: Peridot: Creating User Interfaces by Demonstration. In: Watch What I Do: Programming by Demonstration, pp. 125–154. MIT Press, Cambridge (1993)Google Scholar
  27. 27.
    Nardi, B.A.: A small matter of programming: perspectives on end user computing. MIT Press, Cambridge (1993)Google Scholar
  28. 28.
    Nardi, B.A., Miller, J.R.: The Spreadsheet Interface: A Basis for End User Programming. HP Laboratories (1990) Google Scholar
  29. 29.
    Norman, D.A.: Cognitive Engineering. In: User Centered System Design: New Perspectives on Human-Computer Interaction, pp. 31–61. Lawrence Erlbaum Associates, Mahwah (1986)Google Scholar
  30. 30.
    Raman, V., Hellerstein, J.M.: Potter’s Wheel: An Interactive Data Cleaning System. The VLDB Journal, 381–390 (2001)Google Scholar
  31. 31.
    Rode, J.A., Toye, E.F., Blackwell, A.F.: The fuzzy felt ethnography–understanding the programming patterns of domestic appliances. Personal and Ubiquitous Computing 8(3), 161–176 (2004)CrossRefGoogle Scholar
  32. 32.
    Sundström, P., Taylor, A.S., Grufberg, K., Wirström, N., Belenguer, J.S., Lundén, M.: Inspirational Bits: Towards a shared understanding of the digital material. In: CHI (2011) Google Scholar
  33. 33.
    Tecuci, G., Boicu, M., Cox, M.: Seven Aspects of Mixed-Initiative Reasoning: An Introduction to this Special Issue on Mixed-Initiative Assistants. AI Magazine 28(2) (2007)Google Scholar
  34. 34.
    Tomasic, A., Zimmerman, J., Hargraves, I., McMullen, R.: User Constructed Data Integration via Mixed-Initiative Design. In: Interaction Challenges for Intelligent Assistants, pp. 122–123 (2007) Google Scholar
  35. 35.
    Toomim, M., Drucker, S.M., Dontcheva, M., Rahimi, A., Thomson, B., Landay, J.A.: Attaching UI enhancements to websites with end users. In: CHI, pp. 1859–1868 (2009)Google Scholar
  36. 36.
    Tuchinda, R., Szekely, P., Knoblock, C.A.: Building Mashups by example. In: IUI, pp. 139–148 (2008)Google Scholar
  37. 37.
    Venkatesh, V., Bala, H.: Technology Acceptance Model 3 and a Research Agenda on Interventions. Decision Sciences 39(2), 273–315 (2008)CrossRefGoogle Scholar
  38. 38.
    Wang, G., Yang, S., Han, Y.: Mashroom: end-user mashup programming using nested tables. In: WWW, pp. 861–870 (2009) Google Scholar
  39. 39.
    Zimmerman, J., Rivard, K., Hargraves, I., Tomasic, A., Mohnkern, K.: User-created Forms as an Effective Method of Human-agent Communication. In: CHI 2009, pp. 1869–1878 (2009) Google Scholar
  40. 40.
    Zimmerman, J., Tomasic, A., Simmons, I., Hargraves, I., Mohnkern, K., Cornwell, J., McGuire, R.M.: VIO: a mixed-initiative approach to learning and automating procedural update tasks. In: CHI, pp. 1445–1454 (2007)Google Scholar
  41. 41.
    Zloof, M.M.: QBE/OBE: A Language for Office and Business Automation. IEEE Computer 14(5), 13–22 (1981)Google Scholar

Copyright information

© IFIP International Federation for Information Processing 2011

Authors and Affiliations

  • Steven Gardiner
    • 1
  • Anthony Tomasic
    • 1
  • John Zimmerman
    • 1
  • Rafae Aziz
    • 1
  • Kathryn Rivard
    • 1
  1. 1.Carnegie Mellon School of Computer SciencePittsburghUSA

Personalised recommendations