A Middleware for Seamless Use of Multiple Displays

  • Satoshi Sakurai
  • Yuichi Itoh
  • Yoshifumi Kitamura
  • Miguel A. Nacenta
  • Tokuo Yamaguchi
  • Sriram Subramanian
  • Fumio Kishino
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5136)


Current multi-display environments (MDEs) can be composed of displays with different characteristics (e.g. resolution, size) located in any position and at different angles. These heterogeneous arrangements present specific interface problems: it is difficult to provide meaningful transitions of cursors between displays; it is difficult for users to visualize information that is presented on oblique surfaces; and it is difficult to spread visual information over multiple displays. In this paper we present a middleware architecture designed to support a new kind of perspective-aware GUI that solves the aforementioned problems. Our interaction architecture combines distributed input and position tracking data to generate perspective-corrected output in each of the displays, allowing groups of users to manipulate existing applications from current operating systems across a large number of displays. To test our design we implemented a complex MDE prototype and measured different aspects of its performance.


3D interactions graphical user interface server-client VNC 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Baudisch, P., Cutrell, E., Hinckley, K., Gruen, R.: Mouse ether: accelerating the acquisition of targets across multi-monitor displays. In: Conference on Human Factors in Computing Systems, pp. 1379–1382 (2004)Google Scholar
  2. 2.
    Baudisch, P., Rosenholtz, R.: Halo: a technique for visualizing off-screen objects. In: Conference on Human Factors in Computing Systems, pp. 481–488 (2003)Google Scholar
  3. 3.
    Biehl, J.T., Bailey, B.P.: ARIS: an interface for application relocation in an interactive space. In: Graphics Interface, pp. 107–116 (2004)Google Scholar
  4. 4.
    Booth, K.S., Fisher, B.D., Lin, C.J.R., Argue, R.: The “mighty mouse” multi-screen collaboration tool. In: 15th annual Symposium on User Interface Software and Technology, pp. 209–212 (2002)Google Scholar
  5. 5.
    Borkowski, S., Letessier, J., Crowley, J.L.: Spatial control of interactive surfaces in an augmented environment. In: 9th IFIP Working Conference on Engineering for Human-Computer Interaction, pp. 228–244 (2004)Google Scholar
  6. 6.
    Brumitt, B., Meyers, B., Krumm, J., Kern, A., Shafer, S.A.: EasyLiving: technologies for intelligent environments. In: 2nd international symposium on Handheld and Ubiquitous Computing, pp. 12–29 (2000)Google Scholar
  7. 7.
    Endres, C., Butz, A., MacWilliams, A.: A survey of software infrastructures and frameworks for ubiquitous computing. Mobile Information Systems Journal, 41–80 (2005)Google Scholar
  8. 8.
    Johanson, B., Hutchins, G., Winograd, T., Stone, M.: PointRight: experience with flexible input redirection in interactive workspaces. In: 15th annual Symposium on User Interface Software and Technology, pp. 227–234 (2002)Google Scholar
  9. 9.
    Loenen, E., Bergman, T., Buil, V., Gelder, K., Groten, M., Hollemans, G., Hoonhout, J., Lashina, T., Wijdeven, S.: Entertaible: a solution for social gaming experiences. In: Workshop on Tangible Play: Research and Design for Tangible and Tabletop Games (in International Conference on Intelligent User Interfaces), pp. 16–19 (2007)Google Scholar
  10. 10.
    Nacenta, M.A., Sallam, S., Champoux, B., Subramanian, S., Gutwin, C.: Perspective cursor: perspective-based interaction for multi-display environments. In: Conference on Human Factors in Computing Systems, pp. 289–298 (2006)Google Scholar
  11. 11.
    Nacenta, M.A., Sakurai, S., Yamaguchi, T., Miki, Y., Itoh, Y., Kitamura, Y., Subramanian, S., Gutwin, C.: E-conic: a perspective-aware interface for multi-display environments. In: 20th annual Symposium on User Interface Software and Technology, pp. 279–288 (2007)Google Scholar
  12. 12.
    Rekimoto, J., Saitoh, M.: Augmented surfaces: a spatially continuous work space for hybrid computing environments. In: Conference on Human Factors in Computing Systems, pp. 378–385 (1998)Google Scholar
  13. 13.
    Richardson, T., Stafford-Fraser, Q., Wood, K.R., Hopper, A.: Virtual network computing. IEEE Internet Computing 2(1), 33–38 (1998)CrossRefGoogle Scholar
  14. 14.
    Román, M., Hess, C., Cerqueira, R., Ranganathan, A., Campbell, R.H., Nahrstedt, K.: A middleware infrastructure for active spaces. IEEE Pervasive Computing 1(4), 74–83 (2002)CrossRefGoogle Scholar
  15. 15.
    Schneiderman, B.: Designing the user interface, 3rd edn. Addison-Wesley, Reading (1998)Google Scholar
  16. 16.
    So, R.H.Y., Griffin, M.J.: Effects of lags on human-operator transfer functions with head-coupled systems. Aviation, Space, and Environmental Medicine 66(6), 550–556 (1995)Google Scholar
  17. 17.
    Tan, D.S., Meyers, B., Czerwinski, M.: WinCuts: manipulating arbitrary window regions for more effective use of screen space. In: Conference on Human Factors in Computing Systems, pp. 1525–1528 (2004)Google Scholar
  18. 18.
    Tandler, P.: Software infrastructure for ubiquitous computing environments: supporting synchronous collaboration with heterogeneous devices. In: Ubiquitous Computing, pp. 96–115 (2001)Google Scholar
  19. 19.
    Wigdor, D., Shen, C., Forlines, C., Balakrishnan, R.: Perception of elementary graphical elements in tabletop and multi-surface environments. In: Conference on Human Factors in Computing Systems, pp. 473–482 (2007)Google Scholar
  20. 20.
    Distributed Multihead X Project.
  21. 21.
    Microsoft DirectX Developer Center,
  22. 22.
  23. 23.
  24. 24.

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Satoshi Sakurai
    • 1
  • Yuichi Itoh
    • 1
  • Yoshifumi Kitamura
    • 1
  • Miguel A. Nacenta
    • 2
  • Tokuo Yamaguchi
    • 1
  • Sriram Subramanian
    • 3
  • Fumio Kishino
    • 1
  1. 1.Graduate School of Information Science and TechnologyOsaka UniversitySuitaJapan
  2. 2.Department of Computer ScienceUniversity of SaskatchewanSaskatoonCanada
  3. 3.Department of Computer ScienceUniversity of BristolBristolUnited Kingdom

Personalised recommendations