Grab ‘n’ Drop: User Configurable Toolglasses
Abstract
We introduce the grab ‘n’ drop toolglass, an extension of the toolglass bi-manual interaction technique. It enables users to create and configure their own toolglasses from existing user interfaces that were not designed for toolglasses. Users compose their own toolglass interactions at runtime from an application’s user interface elements, bringing interaction closer to the objects of interest in a workspace. Through a proof-of-concept implementation for Mac OS X, we show how grab ‘n’ drop capabilities could be added to existing applications at the toolkit level, without modifying application source code or UI design. Finally, we evaluate the power and flexibility of this approach by applying it to a variety of applications. We further identify limitations and risks associated with this approach and propose changes to existing toolkits to foster such user-reconfigurable interaction.
Keywords
User interfaces Toolglasses Instrumental interaction PolymorphismReferences
- 1.Apple Computer Inc.: AppleScript Language Guide. Addison-Wesley Longman Publishing Co., Inc., Boston (1994)Google Scholar
- 2.Beaudouin-Lafon, M.: Instrumental interaction: an interaction model for designing post-WIMP user interfaces. In: CHI 2000: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 446–453. ACM, New York (2000)Google Scholar
- 3.Beaudouin-Lafon, M., Mackay, W.E.: Reification, polymorphism and reuse: three principles for designing visual interfaces. In: AVI 2000: Proceedings of the Working Conference on Advanced Visual Interfaces, pp. 102–109. ACM Press (2000)Google Scholar
- 4.Bier, E.A., Stone, M.C., Pier, K., Buxton, W., DeRose, T.D.: Toolglass and magic lenses: the see-through interface. In: SIGGRAPH 1993: Proceedings of the 20th Annual Conference on Computer Graphics and Interactive Techniques, pp. 73–80. ACM, New York (1993)Google Scholar
- 5.Coutaz, J.: PAC: an object oriented model for implementing user interfaces. SIGCHI Bull. 19(2), 37–41 (1987). http://doi.acm.org/10.1145/36111.1045592 CrossRefGoogle Scholar
- 6.Cypher, A., Dontcheva, M., Lau, T., Nichols, J.: No Code Required: Giving Users Tools to Transform the Web. Morgan Kaufmann Publishers Inc., San Francisco (2010)Google Scholar
- 7.Cypher, A., Halbert, D.C., Kurlander, D., Lieberman, H., Maulsby, D., Myers, B.A., Turransky, A. (eds.): Watch What I Do: Programming by Demonstration. MIT Press, Cambridge (1993)Google Scholar
- 8.Dixon, M., Fogarty, J.: Prefab: implementing advanced behaviors using pixel-based reverse engineering of interface structure. In: CHI 2010: Proceedings of the 28th International Conference on Human Factors in Computing Systems, pp. 1525–1534. ACM, New York (2010)Google Scholar
- 9.Dorn, B., Tew, A.E., Guzdial, M.: Introductory computing construct use in an end-user programming community. In: Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing, VLHCC 2007, pp. 27–32 (2007). http://dx.doi.org/10.1109/VLHCC.2007.33
- 10.Eagan, J.R., Beaudouin-Lafon, M., Mackay, W.E.: Cracking the cocoa nut: user interface programming at runtime. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, UIST 2011, pp. 225–234. ACM, New York (2011). http://doi.acm.org/10.1145/2047196.2047226
- 11.Fishkin, K., Stone, M.C.: Enhanced dynamic queries via movable filters. In: CHI 1995: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 415–420. ACM Press/Addison-Wesley Publishing Co., New York (1995)Google Scholar
- 12.Gantt, M., Nardi, B.A.: Gardeners and gurus: patterns of cooperation among cad users. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 107–117. ACM, New York (1992)Google Scholar
- 13.Hudson, S.E., Rodenstein, R., Smith, I.: Debugging lenses: a new class of transparent tools for user interface debugging. In: UIST 1997: Proceedings of the 10th Annual ACM Symposium on User Interface Software and Technology, pp. 179–187. ACM, New York (1997)Google Scholar
- 14.Hutchings, D.R., Stasko, J.: Quantifying the performance effect of window snipping in multiple-monitor environments. In: Baranauskas, C., Palanque, P., Abascal, J., Barbosa, S.D.J. (eds.) INTERACT 2007. LNCS, vol. 4663, pp. 461–474. Springer, Heidelberg (2007). doi: 10.1007/978-3-540-74800-7_42 CrossRefGoogle Scholar
- 15.Kato, J., Sakamoto, D., Igarashi, T.: Picode: inline photos representing posture data in source code. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2013, pp. 3097–3100. ACM, New York (2013). http://doi.acm.org/10.1145/2470654.2466422
- 16.Klokmose, C.N., Zander, P.-O.: Rethinking laboratory notebooks. In: Lewkowicz, M., Hassanaly, P., Wulf, V., Rohde, M. (eds.) Proceedings of COOP 2010, pp. 119–139. Springer, London (2010). doi: 10.1007/978-1-84996-211-7_8 CrossRefGoogle Scholar
- 17.Mackay, W.E.: Patterns of sharing customizable software. In: Proceedings of the 1990 ACM Conference on Computer-Supported Cooperative Work, pp. 209–221. ACM Press, New York (1990)Google Scholar
- 18.Mackay, W.E.: Users and Customizable Software: A Co-Adaptive Phenomenon. Ph.D. thesis, Massechusetts Institute of Technology (1990)Google Scholar
- 19.Mackay, W.E.: Triggers and barriers to customizing software. In: CHI 1991: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 153–160. ACM Press (1991)Google Scholar
- 20.Mackay, W.E.: Which interaction technique works when?: floating palettes, marking menus and toolglasses support different task strategies. In: AVI 2002: Proceedings of the Working Conference on Advanced Visual Interfaces, pp. 203–208. ACM, New York (2002)Google Scholar
- 21.MacLean, A., Carter, K., Lövstrand, L., Moran, T.: User-tailorable systems: pressing the issues with buttons. In: CHI 1990: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 175–182. ACM Press, New York (1990)Google Scholar
- 22.Nardi, B.A.: A Small Matter of Programming: Perspectives on End User Computing. MIT Press, Cambridge (1993)Google Scholar
- 23.Ponsard, A., McGrenere, J.: Anchored customization: anchoring settings to the application interface to afford customization. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI 2016. pp. 4154–4165. ACM, New York (2016). http://doi.acm.org/10.1145/2858036.2858129
- 24.Reenskaug, T.: Models—views—controllers. Technical report, Xerox PARC, December 1979Google Scholar
- 25.Robertson, G.G., Henderson Jr., D.A., Card, S.K.: Buttons as first class objects on an X desktop. In: Proceedings of the 4th Annual ACM Symposium on User Interface Software and Technology, pp. 35–44. ACM Press, New York (1991)Google Scholar
- 26.Stuerzlinger, W., Chapuis, O., Phillips, D., Roussel, N.: User interface façades: towards fully adaptable user interfaces. In: UIST 2006: Proceedings of the 19th Annual ACM Symposium on User Interface Software and Technology, pp. 309–318. ACM, New York (2006)Google Scholar
- 27.Tan, D.S., Meyers, B., Czerwinski, M.: WinCuts: manipulating arbitrary window regions for more effective use of screen space. In: CHI 2004 Extended Abstracts on Human Factors in Computing Systems, pp. 1525–1528. ACM, New York (2004)Google Scholar
- 28.Yeh, T., Chang, T.H., Miller, R.C.: Sikuli: using GUI screenshots for search and automation. In: UIST 2009: Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology, pp. 183–192. ACM, New York (2009)Google Scholar