Skip to main content
SpringerLink
Log in

ML Tools for the Web: A Way for Rapid Prototyping and HCI Research

  • Chapter
  • First Online:
Artificial Intelligence for Human Computer Interaction: A Modern Approach

Part of the book series: Human–Computer Interaction Series ((HCIS))

Abstract

Machine learning (ML) has become a powerful tool with the potential to enable new interactions and user experiences. Although the use of ML in HCI research is growing, the process of prototyping and deploying ML remains challenging. We claim that ML tools designed to be used on the Web are suitable for fast prototyping and HCI research. In this chapter, we review literature, current technologies, and use cases of ML tools for the Web. We also provide a case study, using TensorFlow.js—a major Web ML library, to demonstrate how to prototype with Web ML tools in different prototyping scenarios. At the end, we discuss challenges and future directions of designing tools for fast prototyping and research.

N. Li, J. Mayes and P. Yu—have equal contributions to this chapter.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 199.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 199.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://www.tensorflow.org/.

  2. 2.

    https://pytorch.org/.

  3. 3.

    https://pandas.pydata.org/.

  4. 4.

    https://numpy.org/.

  5. 5.

    https://github.com/tensorflow/tfjs.

  6. 6.

    https://danfo.jsdata.org/.

  7. 7.

    https://www.manning.com/books/deep-learning-with-javascript.

  8. 8.

    https://www.coursera.org/learn/browser-based-models-tensorflow.

  9. 9.

    https://github.com/tensorflow/tfjs.

  10. 10.

    https://github.com/BrainJS/brain.js.

  11. 11.

    https://github.com/mil-tokyo/webdnn.

  12. 12.

    https://github.com/microsoft/onnxjs.

  13. 13.

    https://github.com/PaddlePaddle/Paddle.js.

  14. 14.

    https://www.khronos.org/webgl/.

  15. 15.

    https://webassembly.org/.

  16. 16.

    https://v8.dev/features/simd.

  17. 17.

    https://developers.google.com/web/updates/2018/10/wasm-threads.

  18. 18.

    https://www.tensorflow.org/lite.

  19. 19.

    https://caffe2.ai/.

  20. 20.

    https://github.com/Tencent/ncnn.

  21. 21.

    https://github.com/alibaba/MNN.

  22. 22.

    https://github.com/justadudewhohacks/face-api.js/.

  23. 23.

    https://ml5js.org/.

  24. 24.

    https://runwayml.com/.

  25. 25.

    https://cloud.google.com/automl.

  26. 26.

    https://teachablemachine.withgoogle.com/.

  27. 27.

    https://www.tensorflow.org/js.

  28. 28.

    https://github.com/tensorflow/tfjs-models.

  29. 29.

    https://github.com/tensorflow/tfjs-models/tree/master/body-pix.

  30. 30.

    https://www.youtube.com/watch?v=kFtIddNLcuM.

  31. 31.

    https://github.com/yemount/pose-animator.

  32. 32.

    https://www.youtube.com/watch?v=x1JYnsvvaJs.

  33. 33.

    https://teachablemachine.withgoogle.com/.

  34. 34.

    https://cloud.google.com/automl.

  35. 35.

    This model contains 3.47M parameters, which results in 300 million multiply-accumulate operations in every model execution.

  36. 36.

    This model contains around 15.1M parameters.

  37. 37.

    https://www.tensorflow.org/guide/graph_optimization.

  38. 38.

    https://gpuweb.github.io/gpuweb/.

  39. 39.

    https://www.w3.org/groups/cg/webmachinelearning.

  40. 40.

    https://tensorflow.github.io/tfjs/e2e/benchmarks/local-benchmark/index.html.

  41. 41.

    https://www.tensorflow.org/lite.

  42. 42.

    https://github.com/alibaba/MNN.

  43. 43.

    https://www.tensorflow.org/lite/microcontrollers.

  44. 44.

    https://www.tensorflow.org/lite/inference_with_metadata/task_library/overview.

  45. 45.

    https://danfo.jsdata.org/.

  46. 46.

    https://www.oreilly.com/library/view/learning-tensorflowjs/9781492090786/.

  47. 47.

    https://www.manning.com/books/deep-learning-with-javascript.

  48. 48.

    https://www.apress.com/gp/book/9781484262726.

  49. 49.

    https://www.apress.com/gp/book/9781484264171.

  50. 50.

    https://www.coursera.org/learn/browser-based-models-tensorflow.

  51. 51.

    https://www.pluralsight.com/courses/building-machine-learning-solutions-tensorflow-js-tfjs.

  52. 52.

    https://github.com/tensorflow/tfjs-models.

  53. 53.

    https://tfhub.dev/.

References

  1. Ashktorab Z, Jain M, Vera Liao Q, Weisz JD (2019) Resilient chatbots: repair strategy preferences for conversational breakdowns. In: Proceedings of the 2019 CHI conference on human factors in computing systems (CHI ’19). Association for Computing Machinery, New York, NY, USA, Article Paper 254, 12 pp. ISBN 9781450359702. https://doi.org/10.1145/3290605.3300484

  2. Bisong E (2019) Google AutoML: cloud vision. In: Building machine learning and deep learning models on google cloud platform. Apress, Berkeley, CA, pp 581–598

    Google Scholar 

  3. Bosch N, D’Mello S, Baker R, Ocumpaugh J, Shute V, Ventura M, Wang L, Zhao W (2015) Automatic detection of learning-centered affective states in the wild. In: Proceedings of the 20th international conference on intelligent user interfaces (IUI ’15). Association for Computing Machinery, New York, NY, USA, pp 379–388. ISBN 9781450333061. https://doi.org/10.1145/2678025.2701397

  4. Cai CJ, Guo PJ (2019) Software developers learning machine learning: motivations, hurdles, and desires. In: 2019 IEEE symposium on visual languages and human-centric computing (VL/HCC), pp 25–34. https://doi.org/10.1109/VLHCC.2019.8818751

  5. Carney M, Webster B, Alvarado I, Phillips K, Howell N, Griffith J, Jongejan J, Pitaru A, Chen A (2020) Teachable machine: approachable web-based tool for exploring machine learning classification. In: Extended abstracts of the 2020 CHI conference on human factors in computing systems (CHI EA ’20). Association for Computing Machinery, New York, NY, USA, pp 1–8. ISBN 9781450368193. https://doi.org/10.1145/3334480.3382839

  6. David R, Duke J, Jain A, Janapa Reddi V, Jeffries N, Li J, Kreeger N, Nappier I, Natraj M, Regev S, Rhodes R, Wang T, Warden P (2021) TensorFlow lite micro: embedded machine learning on TinyML systems

    Google Scholar 

  7. Dove G, Halskov K, Forlizzi J, Zimmerman J (2017) UX design innovation: challenges for working with machine learning as a design material. In: Proceedings of the 2017 CHI conference on human factors in computing systems (CHI ’17). Association for Computing Machinery, New York, NY, USA, pp 278–288. ISBN 9781450346559. https://doi.org/10.1145/3025453.3025739

  8. Feurer M, Klein A, Eggensperger K, Springenberg J, Blum M, Hutter F (2015) Efficient and robust automated machine learning. In: Cortes C, Lawrence N, Lee D, Sugiyama M, Garnett R (eds) Advances in neural information processing systems, vol. 28. Curran Associates, Inc. https://proceedings.neurips.cc/paper/2015/file/11d0e6287202fced83f79975ec59a3a6-Paper.pdf

  9. Hall M, Frank E, Holmes G, Pfahringer B, Reutemann P, Witten IH (2009) The WEKA data mining software: an update. 11, 1. ISSN1931-0145. https://doi.org/10.1145/1656274.1656278

  10. Hinton G, Vinyals O, Dean J (2015) Distilling the knowledge in a neural network

    Google Scholar 

  11. Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) MobileNets: efficient convolutional neural networks for mobile vision applications

    Google Scholar 

  12. Inkpen K, Chancellor S, De Choudhury M, Veale M, Baumer EPS (2019) Where is the human? Bridging the gap between AI and HCI (CHI EA ’19). Association for Computing Machinery, New York, NY, USA, pp 1–9. ISBN 9781450359719. https://doi.org/10.1145/3290607.3299002

  13. Jacob B, Kligys S, Chen B, Zhu M, Tang M, Howard A, Adam H, Kalenichenko D (2017) Quantization and training of neural networks for efficient integer-arithmetic-only inference

    Google Scholar 

  14. Jiang X, Wang H, Chen Y, Wu Z, Wang L, Zou B, Yang Y, Cui Z, Cai Y, Yu T, Lv C, Wu Z (2020) MNN: a universal and efficient inference engine

    Google Scholar 

  15. Jun J, Jung M, Kim S-Y, (Kenny) Kim K (2018) Full-body ownership illusion can change our emotion. In: Proceedings of the 2018 CHI conference on human factors in computing systems (CHI ’18). Association for Computing Machinery, New York, NY, USA, Article Paper 601, 11 pp ISBN 9781450356206. https://doi.org/10.1145/3173574.3174175

  16. Kotthoff L, Thornton C, Hoos HH, Hutter F, Kevin L-B (2019) Auto-WEKA: automatic model selection and hyperparameter optimization in WEKA. Automated machine learning. Springer, Cham, pp 81–95

    Google Scholar 

  17. Lee MK, Grgić-Hlača N, Carl Tschantz M, Binns R, Weller A, Carney M, Inkpen K (2020) Human-centered approaches to fair and responsible AI. In: Extended abstracts of the 2020 CHI conference on human factors in computing systems (CHI EA ’20). Association for Computing Machinery, New York, NY, USA, pp 1–8. ISBN 9781450368193. https://doi.org/10.1145/3334480.3375158

  18. Li Y (2010) Protractor: a fast and accurate gesture recognizer. In: Proceedings of the SIGCHI conference on human factors in computing systems (CHI ’10). Association for Computing Machinery, New York, NY, USA, pp 2169–2172. ISBN 9781605589299. https://doi.org/10.1145/1753326.1753654

  19. Li Y, Kumar R, Lasecki WS, Hilliges O (2020) Artificial intelligence for HCI: a modern approach. In: Extended abstracts of the 2020 CHI conference on human factors in computing systems (CHI EA ’20). Association for Computing Machinery, New York, NY, USA, pp 1–8. ISBN 9781450368193. https://doi.org/10.1145/3334480.3375147

  20. Ma Y, Xiang D, Zheng S, Tian D, Liu X (2019) Moving deep learning into web browser: how far can we go?

    Google Scholar 

  21. Maynes-Aminzade D, Winograd T, Igarashi T (2007) Eyepatch: prototyping camera-based interaction through examples. In: Proceedings of the 20th annual ACM symposium on user interface software and technology (UIST ’07). Association for Computing Machinery, New York, NY, USA, pp 33–42. ISBN 9781595936790. https://doi.org/10.1145/1294211.1294219

  22. Munteanu C, Jones M, Oviatt S, Brewster S, Penn G, Whittaker S, Rajput N, Nanavati A (2013) We need to talk: HCI and the delicate topic of spoken language interaction. In: CHI ’13 extended abstracts on human factors in computing systems (CHI EA ’13). Association for Computing Machinery, New York, NY, USA, pp 2459–2464. ISBN 9781450319522. https://doi.org/10.1145/2468356.2468803

  23. Myers BA, Ko AJ, Burnett MM (2006) Invited research overview: end-user programming. In: CHI ’06 extended abstracts on human factors in computing systems (CHI EA ’06). Association for Computing Machinery, New York, NY, USA, pp 75–80. ISBN 1595932984. https://doi.org/10.1145/1125451.1125472

  24. Patel K, Bancroft N, Drucker SM, Fogarty J, Ko AJ, Landay J (2010) Gestalt: integrated support for implementation and analysis in machine learning. In: Proceedings of the 23nd annual ACM symposium on user interface software and technology (UIST ’10). Association for Computing Machinery, New York, NY, USA, pp 37–46. ISBN 9781450302715. https://doi.org/10.1145/1866029.1866038

  25. Patel K, Fogarty J, Landay JA, Harrison B (2008) Examining difficulties software developers encounter in the adoption of statistical machine learning. In: Proceedings of the 23rd national conference on artificial intelligence - volume 3 (AAAI’08). AAAI Press, pp 1563–1566. ISBN 9781577353683

    Google Scholar 

  26. Pece F, Steptoe W, Wanner F, Julier S, Weyrich T, Kautz J, Steed A (2013) Panoinserts: mobile spatial teleconferencing. In: Proceedings of the SIGCHI conference on human factors in computing systems (CHI ’13). Association for Computing Machinery, New York, NY, USA, pp 1319–1328. ISBN 9781450318990. https://doi.org/10.1145/2470654.2466173

  27. Rick SR, Bhaskaran S, Sun Y, McEwen S, Weibel N (2019) NeuroPose: geriatric rehabilitation in the home using a webcam and pose estimation. In: Proceedings of the 24th international conference on intelligent user interfaces: companion (IUI ’19). Association for Computing Machinery, New York, NY, USA, pp 105–106. ISBN 9781450366731. https://doi.org/10.1145/3308557.3308682

  28. Rubin Z, Kurniawan S, Gotfrid T, Pugliese A (2016) Motivating individuals with spastic cerebral palsy to speak using mobile speech recognition. In: Proceedings of the 18th international ACM SIGACCESS conference on computers and accessibility (ASSETS ’16). Association for Computing Machinery, New York, NY, USA, pp 325–326. ISBN 9781450341240. https://doi.org/10.1145/2982142.2982203

  29. Sewell W, Komogortsev O (2010) Real-time eye gaze tracking with an unmodified commodity webcam employing a neural network. In: CHI ’10 extended abstracts on human factors in computing systems (CHI EA ’10). Association for Computing Machinery, New York, NY, USA, pp 3739–3744. ISBN 9781605589305. https://doi.org/10.1145/1753846.1754048

  30. Smilkov D, Thorat N, Assogba Y, Yuan A, Kreeger N, Yu P, Zhang K, Cai S, Nielsen E, Soergel D et al (2019) Tensorflow. js: machine learning for the web and beyond. arXiv:1901.05350

  31. Song Y, Demirdjian D, Davis R (2012) Continuous body and hand gesture recognition for natural human-computer interaction. ACM Trans Interact Intell Syst 2, 1, Article 5, 28 pp. ISSN2160-6455. https://doi.org/10.1145/2133366.2133371

  32. Stackoverflow (2020) 2020 StackOverflow developer survey. https://insights.stackoverflow.com/survey/2020#technology-programming-scripting-and-markup-languages-professional-developers

  33. Thieme A, Belgrave D, Doherty G (2020) Machine learning in mental health: a systematic review of the HCI literature to support the development of effective and implementable ML systems. ACM Trans Comput-Hum Interact 27, 5, Article 34, 53 pp. ISSN1073-0516. https://doi.org/10.1145/3398069

  34. Wang R, Paris S, Popoviundefined J (2011) 6D hands: markerless hand-tracking for computer aided design. In: Proceedings of the 24th annual ACM symposium on user interface software and technology (UIST ’11). Association for Computing Machinery, New York, NY, USA, pp 549–558. ISBN 9781450307161. https://doi.org/10.1145/2047196.2047269

  35. Xu A, Liu Z, Guo Y, Sinha V, Akkiraju R (2017) A new chatbot for customer service on social media. In: Proceedings of the 2017 CHI conference on human factors in computing systems (CHI ’17). Association for Computing Machinery, New York, NY, USA, pp 3506–3510. ISBN 9781450346559. https://doi.org/10.1145/3025453.3025496

  36. Yamashita N, Ishida T (2006) Effects of machine translation on collaborative work. In: Proceedings of the 2006 20th anniversary conference on computer supported cooperative work (CSCW ’06). Association for Computing Machinery, New York, NY, USA, pp 515–524. ISBN 1595932496. https://doi.org/10.1145/1180875.1180955

  37. Yang Q, Suh J, Chen N-C, Ramos G (2018) Grounding interactive machine learning tool design in how non-experts actually build models. In: Proceedings of the 2018 designing interactive systems conference (DIS ’18). Association for Computing Machinery, New York, NY, USA, pp 573–584. ISBN 9781450351980. https://doi.org/10.1145/3196709.3196729

  38. Yuan A, Li Y (2020) Modeling human visual search performance on realistic webpages using analytical and deep learning methods. In: Proceedings of the 2020 CHI conference on human factors in computing systems (CHI ’20). Association for Computing Machinery, New York, NY, USA, pp 1–12. ISBN 9781450367080. https://doi.org/10.1145/3313831.3376870

Download references

Acknowledgements

We would like to thank Sandeep Gupta and Daniel Smilkov for their contributions and valuable feedback.

Author information

Authors and Affiliations

  1. Google, TensorFlow.js Team, Mountain View, CA, USA

    Na Li, Jason Mayes & Ping Yu

Authors
  1. Na Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jason Mayes
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ping Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Na Li .

Editor information

Editors and Affiliations

  1. Google Research (United States), Mountain View, CA, USA

    Yang Li

  2. Advanced Interactive Technologies Lab, ETH Zurich, Zurich, Switzerland

    Otmar Hilliges

Rights and permissions

Reprints and permissions

Copyright information

© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Li, N., Mayes, J., Yu, P. (2021). ML Tools for the Web: A Way for Rapid Prototyping and HCI Research. In: Li, Y., Hilliges, O. (eds) Artificial Intelligence for Human Computer Interaction: A Modern Approach. Human–Computer Interaction Series. Springer, Cham. https://doi.org/10.1007/978-3-030-82681-9_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-82681-9_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-82680-2

  • Online ISBN: 978-3-030-82681-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation