Shared Façades: Surface-Embedded Layout Management for Ad Hoc Collaboration Using Head-Worn Displays

  • Barrett EnsEmail author
  • Eyal Ofek
  • Neil Bruce
  • Pourang Irani


Collaboration is a necessary, everyday human activity, yet computing environments specifically designed to support collaborative tasks have typically been aimed toward groups of experts in extensive, purpose-built environments. The cost constraints and design complexities of fully-networked, multi-display environments have left everyday computer users in the lurch. However, the age of ubiquitous networking and wearable technologies has been accompanied by functional head-worn displays (HWDs), which are now capable of creating rich, interactive environments by overlaying virtual content onto real-world objects and surfaces. These immersive interfaces can be leveraged to transform the abundance of ordinary surfaces in our built environment into ad hoc collaborative multi-display environments. This paper introduces an approach for distributing virtual information displays for multiple users. We first describe a method for producing spatially-constant virtual window layouts in the context of single users. This method applies a random walk algorithm to balance multiple constraints, such as spatial constancy of displayed information, visual saliency of the background, surface-fit, occlusion and relative position of multiple windows, to produce layouts that remain consistent across multiple environments while respecting the local geometric features of the surroundings. We then describe how this method can be generalized to include additional constraints from multiple users. For example, the algorithm can take the relative poses of two or more users into account, to prevent information from being occluded by objects in the environment from the perspective of each participant. In this paper, we however focus on describing how to make the content spatially-constant for one user, and discuss how it scales from one to multiple closely confined users. We provide an initial validation of this approach including quantitative and qualitative data in a user study. We evaluate weighting schemes with contrasting emphasis on spatial constancy and visual saliency, to determine how easily a user can locate spatially-situated information within the restricted viewing field of current head-worn display technology. Results show that our balanced constraint weighting schema produces better results than schemas that consider spatial constancy or visual saliency alone, when applied to models of two real-world test environments. Finally, we discuss our plans for future work, which will apply our window layout method in collaborative environments, to assist wearable technology users to engage in ad hoc collaboration with everyday analytic tasks.


Head-worn displays Window manager Spatial constancy Visual saliency Surface detection 


  1. 1.
    Agarawala A, Balakrishnan R (2006) Keepin’ it real: Pushing the desktop metaphor with physics, piles and the pen. In: Proceedings of the CHI’06, ACM (2006), pp 1283–1292Google Scholar
  2. 2.
    Bae S, Cho YCP, Park S, Irick KM, Jin Y, Narayanan V (2011) An FPGA implementation of information theoretic visual-saliency system and its optimization. In: Proceedings of the FCCM ‘11, IEEE (2011), pp 41–48Google Scholar
  3. 3.
    Bell BA, Feiner SK (2000) Dynamic space management for user interfaces. In: Proceedings of the UIST ’00, ACM (2000), pp 239–248Google Scholar
  4. 4.
    Bell B, Feiner S, Höllerer T (2001) View management for virtual and augmented reality. In: Proceedings of the UIST ’01, ACM (2001), pp 101–110Google Scholar
  5. 5.
    Billinghurst M, Bowskill J, Jessop M, Morphett J (1998) A wearable spatial conferencing space. In: Proceedings of the ISWC ‘98, IEEE (1998), 76–83Google Scholar
  6. 6.
    Billinghurst M, Kato H (2002) Collaborative augmented reality. Commun ACM 45(7):64–70CrossRefGoogle Scholar
  7. 7.
    Bouguet JY Camera calibration toolbox for Matlab:
  8. 8.
    Bruce N, Tsotsos J (2005) Saliency based on information maximization. In: Proceedings of the NIPS ’05 (2005), pp 155, 162Google Scholar
  9. 9.
    Cao X, Balakrishnan R (2006) Interacting with dynamically defined information spaces using a handheld projector and a pen. In: Proceedings of the UIST ’06, ACM (2006), pp 225–234Google Scholar
  10. 10.
    Cerf M, Frady EP, Koch C (2009) Faces and text attract gaze independent of the task: Experimental data and computer model. J Vis 9(12):1–15CrossRefGoogle Scholar
  11. 11.
    Di Verdi S, Nurmi D, Höllerer T (2003) ARWin—A desktop augmented reality window manager. In: Proceedings of the ISMAR ’03, IEEE (2003), pp 123–124Google Scholar
  12. 12.
    Dourish P, Bellotti V (1992) Awareness and coordination in shared workspaces. In: Proceedings of the CSCW ‘92. ACM (1992), pp 107–114Google Scholar
  13. 13.
    Ens B, Finnegan R, Irani P The Personal Cockpit: a spatial interface for effective task switching on head-worn displays. In: Proceedings of the CHI ’14, ACM, pp 3171–3180Google Scholar
  14. 14.
    Ens B, Ofek E, Bruce N, Irani P (2015) Spatial constancy of surface-embedded layouts across multiple environments. In: Proceedings of the SUI ’15, ACM, pp 65–68Google Scholar
  15. 15.
    Feiner S, MacIntyre B, Haupt M, Solomon E (1993) Windows on the world: 2D windows for 3D augmented reality. In: Proceedings of the UIST ’93, ACM, pp 145–155Google Scholar
  16. 16.
    Fisher SS, McGreevy M, Humphries J, Robinett W (1986) Virtual environment display system. In: Proceedings of the I3D ’86, ACM, pp 77–87Google Scholar
  17. 17.
    Fitzmaurice GW (1993) Situated information spaces and spatially aware computers. Commun ACM 36, 7:39–49Google Scholar
  18. 18.
    Gabbard JL, Swan JE II, Hix D (2002) The effects of text drawing styles, background textures, and natural lighting on text legibility in outdoor augmented reality. Presence: Teleoperat Virtual Environ 15(1):16–32CrossRefGoogle Scholar
  19. 19.
    Gal R, Shapira L, Ofek E, Kohli P (2014) FLARE: fast layout for augmented reality applications. In: Proceedings of the ISMAR ’14, ACMGoogle Scholar
  20. 20.
    Garth B, Shoemaker D, Inkpen KM (2001) Single display privacyware: Augmenting public displays with private information. In: Proceedings of the CHI ‘10 ACM, pp 522–529Google Scholar
  21. 21.
    Grasset R, Langlotz T, Kalkofen D, Tatzgern M, Schmalstieg D (2012) Image-driven view management for augmented reality browsers. In: Proceedings of the ISMAR ‘12, IEEE, pp 177–186Google Scholar
  22. 22.
    Gupta D (2004) An empirical study of the effects of context-switch, object distance, and focus depth on human performance in augmented reality. Virginia Polytechnic Institute and State UniversityGoogle Scholar
  23. 23.
    Hastings WK (1970) Monte Carlo sampling methods using markov chains and their applications. Biometrika 57(1):97–109MathSciNetCrossRefzbMATHGoogle Scholar
  24. 24.
    Hincapié-Ramos JD, Ivanchuk L, Sridharan SK, Irani P (2014) SmartColor: Real-time color correction and contrast for optical see-through head-mounted displays. In: Proceedings of the ISMAR ’14. ACMGoogle Scholar
  25. 25.
    Izadi S, Kim D, Hilliges O, Molyneaux D, Newcombe R, Kohli P, Shotton J, Hodges S, Freeman D, Davison A, Fitzgibbon A (2011) KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera. In: Proceedings of the UIST ‘11, ACM, pp 559–568Google Scholar
  26. 26.
    Jones BR, Benko H, Ofek E, Wilson AD (2013) IllumiRoom: peripheral projected illusions for interactive experiences. In: Proceedings of the CHI ’13, ACM, pp 869–878Google Scholar
  27. 27.
    Klein G, Murray D (2009) Parallel tracking and mapping on a camera phone. In: Proceedings of the ISMAR ’09, IEEE, pp 83–86Google Scholar
  28. 28.
    Kruijff E, Swan J, Feiner S (2010) Perceptual issues in augmented reality revisited. In Mixed and Augmented Reality. In: Proceedings of the ISMAR ‘10, IEEE, pp 3–12Google Scholar
  29. 29.
    Langlotz T, Nguyen T, Schmalstieg D, Grasset R (2014) Next-generation augmented reality browsers: rich, seamless, and adaptive. Proceedings of the IEEE 120, 2, pp 155–169Google Scholar
  30. 30.
    Lee W, Park Y, Lepetit V, Woo W (2011) Video-based in situ tagging on mobile phones. In: TCSVT 21, 10, IEEE, pp 1487–1496Google Scholar
  31. 31.
    Leykin A, Tuceryan M (2004) Automatic determination of text readability over textured backgrounds for augmented reality systems. In: Proceedings of the ISMAR ’04, IEEE, pp 224–230Google Scholar
  32. 32.
    Maimone A, Lanman D, Rathinavel K, Keller K, Luebke D, Fuchs H (2014) Pinlight displays: Wide field of view augmented reality eyeglasses using defocussed point sources of light. In: TOG 33, 4, ACMGoogle Scholar
  33. 33.
    Merrell P, Schkufza E, Li Z, Agrawala M, Koltun V (2011) Interactive furniture layout using interior design guidelines. In: TOG 30, 4, ACMGoogle Scholar
  34. 34.
    Molyneaux D, Izadi S, Kim D, Hilliges O, Hodges S, Cao X, Butler A, Gellersen H (2012) Interactive environment-aware handheld projectors for pervasive computing spaces. In: Proceedings of the Pervasive ’12, Springer, pp 197–215Google Scholar
  35. 35.
    Nacenta MA, Sakurai S, Yamaguchi T, Miki Y, Itoh Y, Kitamura Y, Subramanian S, Gutwin C (2007) E-conic: a perspective-aware interface for multi-display environments. In: Proceedings of the UIST ’07, ACM, pp 279–288Google Scholar
  36. 36.
    Orlosky J, Kiyokawa K, Takemura H (2013) Dynamic text management for see-through wearable and heads-up display systems. In: Proceedings of the IUI ’13, ACM, pp 363–370Google Scholar
  37. 37.
    Orlosky J, Kiyokawa K, Takemura H (2014) Managing mobile text in head mounted displays: studies on visual preference and text placement. SIGMOBILE Mob Comput Commun Rev 18(2):20–31CrossRefGoogle Scholar
  38. 38.
    Raskar R, van Baar J, Beardsley P, Willwacher T, Rao S, Forlines C (2003) iLamps: Geometrically aware and self-configuring projectors. In: Proceedings of the SIGGRAPH ’03, ACM, pp 809–818Google Scholar
  39. 39.
    Raskar R, Welch G, Cutts M, Lake A, Stesin L, Fuchs H (1998) The office of the future: a unified approach to image-based modelling and spatially immersive displays. In: Proceedings of the SIGGRAPH ’98, ACM, pp 179–188Google Scholar
  40. 40.
    Rekimoto J, Saitoh M (1999) Augmented surfaces: a spatially continuous work space for hybrid computing environments. In: Proceedings of the CHI ’99, ACM, pp 378–385Google Scholar
  41. 41.
    Salas-Moreno R, Glocker B, Kelly P, Davison A (2014) Dense planar SLAM. In: Proceedings of the ISMAR ’14, IEEE (in press)Google Scholar
  42. 42.
    Scarr J, Cockburn A, Gutwin C, Bunt A (2012) Improving command selection with CommandMaps. In: Proceedings of the CHI ’12, ACM, pp 257–266Google Scholar
  43. 43.
    Silberman N, Shapira L, Gal R, Kohli P (2014) A contour completion model for augmenting surface reconstructions. In: Proceedings of the ECCV ’14, ACM, pp 488–503Google Scholar
  44. 44.
    Szalavári Z, Schmalstieg D, Fuhrmann A, Gervautz M (1998) “Studierstube”: An environment for collaboration in augmented reality. Virtual Reality 3(1):37–48CrossRefGoogle Scholar
  45. 45.
    Tak S, Cockburn A, Humm K, Ahlström D, Gutwin G, Scarr J (2009) Improving window switching interfaces. In: Proceedings of the INTERACT ’09, Springer, pp 187–200Google Scholar
  46. 46.
    Tanaka K, Kishino Y, Miyamae M, Terada T, Nishio S (2008) An information layout method for an optical see-through head mounted display focusing on the viewability. In: Proceedings of the ISMAR ’08, IEEE, pp 139–142Google Scholar
  47. 47.
    Vilk J, Molnar D, Ofek E, Rossbach C, Livshits B, Moshchuk A, Wang HJ, Gal R (2014) SurroundWeb: least privilege for immersive “Web Rooms”. Technical Report MSR-TR-2014–25, Microsoft ResearchGoogle Scholar
  48. 48.
    Waldner M, Grasset R, Steinberger M, Schmalstieg D (2011) Display adaptive window management for irregular surfaces. In: Proceedings of the ITS ’11, ACM (2011), 222–231Google Scholar
  49. 49.
    Wilson AD, Benko H (2010) Combining multiple depth cameras and projectors for interactions on, above and between surfaces. In: Proceedings of the UIST ’10 (2010), 273–282Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Barrett Ens
    • 1
    Email author
  • Eyal Ofek
    • 2
  • Neil Bruce
    • 1
  • Pourang Irani
    • 1
  1. 1.Department of Computer ScienceUniversity of ManitobaWinnipegCanada
  2. 2.Microsoft ResearchRedmondUSA

Personalised recommendations