Side-by-Side Human–Computer Design Using a Tangible User Interface

  • Matthew V. LawEmail author
  • Nikhil Dhawan
  • Hyunseung Bang
  • So-Yeon Yoon
  • Daniel Selva
  • Guy Hoffman
Conference paper


We present a digital–physical system to support human–computer collaborative design. The system consists of a sensor-instrumented “sand table” functioning as a tangible space for exploring early-stage design decisions.



This work was supported primarily by the Civil, Mechanical and Manufacturing Innovation Program of the National Science Foundation under NSF Award No. 1635253.


  1. 1.
    Allen JF, Guinn CI, Horvtz E (1999) Mixed-initiative interaction. IEEE Intel Syst Appl 14(5):14–23CrossRefGoogle Scholar
  2. 2.
    Arias E, Eden H, Fischer G, Gorman A, Scharff E (2000) Transcending the individual human mind–creating shared understanding through collaborative design. ACM Trans Computer-Human Int (TOCHI) 7(1):84–113CrossRefGoogle Scholar
  3. 3.
    Arrow KJ (2012) Social choice and individual values, vol 12. Yale University PressGoogle Scholar
  4. 4.
    Babbar-Sebens M, Minsker BS (2012) Interactive genetic algorithm with mixed initiative interaction for multi-criteria ground water monitoring design. Appl Soft Comput J 12(1):182–195CrossRefGoogle Scholar
  5. 5.
    Balling R (1999) Design by shopping: a new paradigm? In: Proceedings of the third world congress of structural and multidisciplinary optimization (WCSMO-3), vol 1, pp 295–297Google Scholar
  6. 6.
    Chen R, Wang X (2008) An empirical study on tangible augmented reality learning space for design skill transfer. Tsinghua Science and Technology 13 Supple (October):13–18Google Scholar
  7. 7.
    Cho SB (2002) Towards creative evolutionary systems with interactive genetic algorithm. Appl Intel 16(2):129–138CrossRefGoogle Scholar
  8. 8.
    Deb K, Pratap A, Agarwal S, Meyarivan T (2002) A fast and elitist multiobjective genetic algorithm: Nsga-ii. IEEE Trans Evol Comput 6(2):182–197CrossRefGoogle Scholar
  9. 9.
    Deb K, Karthik S et al (2007) Dynamic multi-objective optimization and decision-making using modified nsga-ii: a case study on hydro-thermal power scheduling. In: International conference on evolutionary multi-criterion optimization. Springer, pp 803–817Google Scholar
  10. 10.
    Dhanalakshmi S, Kannan S, Mahadevan K, Baskar S (2011) Application of modified nsga-ii algorithm to combined economic and emission dispatch problem. Int J Electr Power Energy Syst 33(4):992–1002CrossRefGoogle Scholar
  11. 11.
    Do-Lenh S, Jermann P, Cuendet S, Zufferey G, Dillenbourg P (2010) Task performance versus learning outcomes: a study of a tangible user interface in the classroom. In: European conference on technology enhanced learning. Springer, pp 78–92Google Scholar
  12. 12.
    Durillo JJ, Nebro AJ (2011) jmetal: A java framework for multi-objective optimization. Adv Eng Softw 42(10):760–771CrossRefGoogle Scholar
  13. 13.
    Egan P, Cagan J (2016) Human and computational approaches for design problem-solving. In: Experimental design research. Springer, pp 187–205Google Scholar
  14. 14.
    Ferguson G, Allen JF et al (1998) Trips: an integrated intelligent problem-solving assistant. In: AAAI/IAAI, pp 567–572Google Scholar
  15. 15.
    Fischer G (2004) Social creativity: turning barriers into opportunities for collaborative design. In: Proceedings of the eighth conference on participatory design: Artful integration: interweaving media, materials and practices-Volume 1, ACM, pp 152–161Google Scholar
  16. 16.
    Gero JS (1998) Conceptual designing as a sequence of situated acts. In: Artificial intelligence in structural engineering. Springer, pp 165–177Google Scholar
  17. 17.
    Grosz BJ (1996) Collaborative systems (aaai-94 presidential address). AI Mag 17(2):67MathSciNetGoogle Scholar
  18. 18.
    Hay L, Duffy AHB, McTeague C, Pidgeon LM, Vuletic T, Grealy M (2017) A systematic review of protocol studies on conceptual design cognition: design as search andexploration. Des Sci 3:e10. arXiv:1011.1669v3Google Scholar
  19. 19.
    Hitomi N, Bang H, Selva D (2017) Extracting and applying knowledge with adaptive knowledge-driven optimization to architect an earth observing satellite system. AIAA Information Systems-AIAA Infotech@ Aerospace, p 0794Google Scholar
  20. 20.
    Ishibuchi H, Masuda H, Tanigaki Y, Nojima Y (2015) Modified distance calculation in generational distance and inverted generational distance. EMO 2:110–125Google Scholar
  21. 21.
    Ishii H, Ratti C, Piper B, Wang Y, Biderman A, Ben-Joseph E (2004) Bringing clay and sand into digital design—continuous tangible user interfaces. BT Technol J 22(4):287–299CrossRefGoogle Scholar
  22. 22.
    Jeyadevi S, Baskar S, Babulal C, Iruthayarajan MW (2011) Solving multiobjective optimal reactive power dispatch using modified nsga-ii. Int J Electr Power Energy Syst 33(2):219–228CrossRefGoogle Scholar
  23. 23.
    Jordà S, Geiger G, Alonso M, Kaltenbrunner M (2007) The reactable: exploring the synergy between live music performance and tabletop tangible interfaces. In: Proceedings of the 1st international conference on Tangible and embedded interaction, ACM, pp 139–146Google Scholar
  24. 24.
    Kaltenbrunner M (2009) Reactivision and tuio: a tangible tabletop toolkit. In: Proceedings of the ACM international conference on interactive tabletops and surfaces, ACM, pp 9–16Google Scholar
  25. 25.
    Kicinger R, Arciszewski T, De Jong K (2005) Evolutionary computation and structural design: A survey of the state-of-the-art. Comput Struct 83(23):1943–1978CrossRefGoogle Scholar
  26. 26.
    Kim HS, Cho SB (2000) Application of interactive genetic algorithm to fashion design. Eng Appl Artif Intell 13(6):635–644CrossRefGoogle Scholar
  27. 27.
    Kim M, Maher M (2005) Comparison of designers using a tangible user interface and graphical user interface and impact on spatial cognition. Proc Human Behav Des 5Google Scholar
  28. 28.
    Kim MJ, Maher ML (2008) The impact of tangible user interfaces on spatial cognition during collaborative design. Des Stud 29(3):222–253CrossRefGoogle Scholar
  29. 29.
    Laugwitz B, Held T, Schrepp M (2008) Construction and evaluation of a user experience questionnaire. In: Symposium of the Austrian HCI and usability engineering group. Springer, pp 63–76Google Scholar
  30. 30.
    Laumanns M, Thiele L, Deb K, Zitzler E (2002) Combining convergence and diversity in evolutionary multiobjective optimization. Evol Comput 10(3):263–282CrossRefGoogle Scholar
  31. 31.
    Liu H, Tang M (2006) Evolutionary design in a multi-agent design environment. Appl Soft Comput J 6(2):207–220MathSciNetCrossRefGoogle Scholar
  32. 32.
    Maher ML, Lee L (2017) Designing for gesture and tangible interaction. Synth Lect Human-Centered Interact 10(2):i–111Google Scholar
  33. 33.
    McCarthy J (2007) What is artificial intelligence. URL:
  34. 34.
    Ozgur A, Johal W, Mondada F, Dillenbourg P (2017) Windfield: learning wind meteorology with handheld haptic robots. In: HRI’17: ACM/IEEE international conference on human-robot interaction proceedings, ACM, EPFL-CONF-224130, pp 156–165Google Scholar
  35. 35.
    Patten J, Ishii H (2000) A comparison of spatial organization strategies in graphical and tangible user interfaces. In: Proceedings of DARE 2000 on designing augmented reality environments, ACM, pp 41–50Google Scholar
  36. 36.
    Petersson K, Kyroudi A, Bourhis J, Ceberg C, Knöös T, Bochud F, Moeckli R (2017) A clinical distance measure for evaluating treatment plan quality difference with pareto fronts in radiotherapy. Phys Imaging Radiat Oncol 3:53–56CrossRefGoogle Scholar
  37. 37.
    Ramchurn SD, Wu F, Jiang W, Fischer JE, Reece S, Roberts S, Rodden T, Greenhalgh C, Jennings NR (2016) Human-agent collaboration for disaster response. Auton Agent Multi-Agent Syst 30(1):82–111CrossRefGoogle Scholar
  38. 38.
    Reed P, Minsker BS, Goldberg DE (2003) Simplifying multiobjective optimization: an automated design methodology for the nondominated sorted genetic algorithm-ii. Water Resour Res 39(7)Google Scholar
  39. 39.
    Selva D (2014a) Experiments in knowledge-intensive system architecting: interactive architecture optimization. In: Aerospace conference, 2014 IEEE, IEEE, pp 1–12Google Scholar
  40. 40.
    Selva D (2014b) Knowledge-intensive global optimization of earth observing system architectures: a climate-centric case study. In: Sensors, systems, and next-generation satellites XVIII, international society for optics and photonics, vol 9241, p 92411SGoogle Scholar
  41. 41.
    Selva D, Cameron BG, Crawley EF (2014) Rule-based system architecting of earth observing systems: earth science decadal survey. J Spacecraft RocketsGoogle Scholar
  42. 42.
    Shen W, Hao Q, Li W (2008) Computer supported collaborative design: retrospective and perspective. Comput Ind 59(9):855–862CrossRefGoogle Scholar
  43. 43.
    Shirado H, Christakis NA (2017) Locally noisy autonomous agents improve global human coordination in network experiments. Nature 545(7654):370–374CrossRefGoogle Scholar
  44. 44.
    Simon HA (1996) The sciences of the artificial. MIT pressGoogle Scholar
  45. 45.
    Smithers T, Conkie A, Doheny J, Logan B, Millington K (1989) Design as intelligent behavior: an ai in design research program. In: Gero JS (ed) Artificial intelligence in designGoogle Scholar
  46. 46.
    Smithwick D, Kirsh D, Sass L (2017) Designerly pick and place: coding physical model making to inform material-based robotic interaction. In: Design computing and cognition’16. Springer, pp 419–436Google Scholar
  47. 47.
    Starcic AI, Zajc M (2011) An interactive tangible user interface application for learning addition concepts_1217 131. 135. Br J Edu Technol 42(6):E131–E135CrossRefGoogle Scholar
  48. 48.
    Thornton C, Du Boulay B (2012) Artificial intelligence through search. Springer Science and Business MediaGoogle Scholar
  49. 49.
    Ullmer B, Ishii H (1997) The metadesk: models and prototypes for tangible user interfaces. In: Proceedings of the 10th annual ACM symposium on user interface software and technology, ACM, pp 223–232Google Scholar
  50. 50.
    Van Veldhuizen DA, Lamont GB (1998) Evolutionary computation and convergence to a pareto front. In: Late breaking papers at the genetic programming 1998 conference, pp 221–228Google Scholar
  51. 51.
    Watson D, Clark LA, Tellegen A (1988) Development and validation of brief measures of positive and negative affect: the panas scales. J Pers Soc Psychol 54(6):1063CrossRefGoogle Scholar
  52. 52.
    Xie L, Antle AN, Motamedi N (2008) Are tangibles more fun? comparing children’s enjoyment and engagement using physical, graphical and tangible user interfaces. In: Proceedings of the 2nd international conference on tangible and embedded interaction, ACM, pp 191–198Google Scholar
  53. 53.
    Zitzler E, Brockhoff D, Thiele L (2007) The hypervolume indicator revisited: on the design of pareto-compliant indicators via weighted integration. In: Evolutionary multi-criterion optimization. Springer, pp 862–876Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Matthew V. Law
    • 1
    Email author
  • Nikhil Dhawan
    • 1
  • Hyunseung Bang
    • 1
  • So-Yeon Yoon
    • 1
  • Daniel Selva
    • 1
  • Guy Hoffman
    • 1
  1. 1.Cornell UniversityIthacaUSA

Personalised recommendations