Skip to main content

Advertisement

SpringerLink
Log in

Human-in-the-loop machine learning with applications for population health

  • Regular Paper
  • Published:
CCF Transactions on Pervasive Computing and Interaction Aims and scope Submit manuscript

Abstract

Though technical advance of artificial intelligence and machine learning has enabled many promising intelligent systems, many computing tasks are still not able to be fully accomplished by machine intelligence. Motivated by the complementary nature of human and machine intelligence, an emerging trend is to involve humans in the loop of machine learning and decision-making. In this paper, we provide a macro–micro review of human-in-the-loop machine learning. We first describe major machine learning challenges which can be addressed by human intervention in the loop. Then we examine closely the latest research and findings of introducing humans into each step of the lifecycle of machine learning. Next, a case study of our recent application study in human-in-the-loop machine learning for population health is introduced. Finally, we analyze current research gaps and point out future research directions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. https://democracy.cityoflondon.gov.uk/ecSDDisplay.aspx?NAME=SD573&ID=573&RPID=0.

  2. https://digital.nhs.uk/data-and-information/publications/statistical/quality-and-outcomes-framework-achievement-prevalence-and-exceptions-data.

References

  • Amershi, S., et al.: Effective end-user interaction with machine learning. In: Twenty-fifth AAAI conference on artificial intelligence (2011).

  • Amershi, S., et al.: Power to the people: the role of humans in interactive machine learning. AI Mag. 35(4), 105–120 (2014)

    Google Scholar 

  • Barbosa, N.M., et al,: Rehumanized crowdsourcing: a labeling framework addressing bias and ethics in machine learning, p. 543. CHI. ACM (2019)

  • Cheng, J., et al.: Flock: hybrid crowd-machine learning classifiers. In: Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing, pp. 600–611 (2015).

  • Chu, X., et al.: Katara: a data cleaning system powered by knowledge bases and crowdsourcing. In: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, pp. 1247–1261 (2015).

  • Das, S., et al.: BEAMES: interactive multi-model steering, selection, and inspection for regression tasks. IEEE Comput Graph Appl 39, 20–32 (2019)

    Article  Google Scholar 

  • Deng, J., et al.: Leveraging the wisdom of the crowd for fine-grained recognition. IEEE Trans. Pattern Anal. Mach. Intell. 38(4), 666–676 (2016)

    Article  Google Scholar 

  • Dudley, J.J., et al.: A review of user interface design for interactive machine learning. ACM Trans. Interact. Intell. Syst. 8(2), 8 (2018)

    Article  Google Scholar 

  • Emmanouilidis, C., Pistofidis, P., Bertoncelj, L., Katsouros, V., Fournaris, A., Koulamas, C., Ruiz-Carcel, C.: Enabling the human in the loop: linked data and knowledge in industrial cyber-physical systems. Annu. Rev. Control. 1(47), 249–265 (2019)

    Article  Google Scholar 

  • Feng, Y., Wang, J., Wang, Y., Helal, S.: Completing missing prevalence rates for multiple chronic diseases by jointly leveraging both intra-and inter-disease population health data correlations. In: Proceedings of the web conference, pp. 183–193. (2021).

  • Guo, B., et al.: FlierMeet: a mobile crowdsensing system for cross-space public information reposting, tagging, and sharing. IEEE Trans. Mob. Comput. 14(10), 2020–2033 (2014)

    Article  Google Scholar 

  • Guo, B., et al.: From crowdsourcing to crowdmining: using implicit human intelligence for better understanding of crowdsourced data. World Wide Web, pp. 1–25 (2019).

  • Honeycutt, D., Nourani, M., Ragan, E.: Soliciting human-in-the-loop user feedback for interactive machine learning reduces user trust and impressions of model accuracy. In: Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, vol. 8, pp. 63–72 (2020).

  • Houlsby, N., Huszár, F., Ghahramani, Z., Lengyel, M.: Bayesian active learning for classification and preference learning. arXiv preprint arXiv:1112.5745 (2011).

  • Howe, J.: The rise of crowdsourcing. Wired Mag. 14(6), 1–4 (2006)

    Google Scholar 

  • Jiang, W., Simon, R.: A comparison of bootstrap methods and an adjusted bootstrap approach for estimating the prediction error in microarray classification. Stat. Med. 26(29), 5320–5334 (2007)

    Article  MathSciNet  Google Scholar 

  • Kamar, E.: Directions in hybrid intelligence: complementing AI systems with human intelligence. In: IJCAI, pp. 4070–4073 (2016).

  • Li, G., et al.: Crowdsourced data management: Overview and challenges. In: Proceedings of the 2017 ACM International Conference on Management of Data, pp. 1711–1716. ACM, (2017).

  • Liu, Y., et al.: Doubly active learning: when active learning meets active crowdsourcing. AAAI (2018).

  • Lu, J., Yang, L., Mac Namee, B., Zhang, Y.: A rationale-centric framework for human-in-the-loop machine learning. arXiv preprint arXiv:2203.12918 (2022).

  • Monarch, R.M.: Human-in-the-Loop Machine Learning: Active Learning and Annotation for Human-Centered AI. Simon and Schuster, New York (2021)

    Google Scholar 

  • Mujica-Mota, R.E., Roberts, M., Abel, G., Elliott, M., Lyratzopoulos, G., Roland, M., Campbell, J.: Common patterns of morbidity and multi-morbidity and their impact on health-related quality of life: evidence from a national survey. Qual. Life Res. 24(4), 909–918 (2015)

    Article  Google Scholar 

  • Nushi, B., et al.: Crowd access path optimization: diversity matters. In: Third AAAI Conference on Human Computation and Crowdsourcing (2015)

  • Olah, C., et al.: The building blocks of interpretability. Distill (2018). https://doi.org/10.23915/distill.00010

    Article  Google Scholar 

  • Ribeiro, M.T., et al.: Why should I trust you? Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135–1144. ACM (2016).

  • Riccardo, G., et al.: A survey of methods for explaining black box models. ACM Comput. Surv. (CSUR) 51(5), 93 (2019)

    Google Scholar 

  • Ruan, W., et al.: Global robustness evaluation of deep neural networks with provable guarantees for the hamming distance. IJCAI (2019).

  • Teso, S., Kersting, K.: Why should i trust interactive learners? Explaining interactive queries of classifiers to users. arXiv preprint, 5 (2018).

  • Tong, Y., et al.: Crowdcleaner: data cleaning for multi-version data on the web via crowdsourcing. In: Data engineering (ICDE), 2014 IEEE 30th international conference on, pp. 1182–1185. IEEE (2014).

  • Venanzi, M., et al.: Time-sensitive Bayesian information aggregation for crowdsourcing systems. J. Artif. Intell. Res. 56, 517–545 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  • Wang J, et al.. Crowd-Assisted Machine Learning: Current Issues and Future Directions. Computer. 2019 13;52(1):46–53.

  • Wu X, Xiao L, Sun Y, Zhang J, Ma T, He L. A survey of human-in-the-loop for machine learning. Future Gener. Comput. Syst. 18 (2022).

  • Xin, D., Ma, L., Liu, J., Macke, S., Song, S., Parameswaran, A.: Helix: accelerating human-in-the-loop machine learning. arXiv preprint arXiv:1808.01095. (2018).

  • Xu, C., et al.: Data cleaning: overview and emerging challenges. In: Proceedings of the 2016 international conference on Management of Data, pp. 2201–2206. ACM (2016).

  • Yosinski, J., et al.: Understanding neural networks through deep visualization. In: ICML Workshop on Deep Learning (2015).

  • Zhang, Q.S., Zhu, S.C.: Visual interpretability for deep learning: a survey. Front. Inf. Technol. Electron. Eng. 19(1), 27–39 (2018)

    Article  Google Scholar 

  • Zhang, C., et al.: Reducing uncertainty of schema matching via crowdsourcing with accuracy rates. IEEE Trans. Knowl. Data Eng. 32, 135–151 (2018)

    Article  Google Scholar 

  • Zhang, R., et al.: Leveraging human guidance for deep reinforcement learning tasks. In: Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI) (2019).

  • Zhou, Z.H.: A brief introduction to weakly supervised learning. Natl. Sci. Rev. 5(1), 44–53 (2017)

    Article  Google Scholar 

  • Zhuang, Y, et al.: Hike: a hybrid human-machine method for entity alignment in large-scale knowledge bases. In: Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pp 1917–1926. ACM (2017).

  • Zou, J.Y., et al.: Crowdsourcing feature discovery via adaptively chosen comparisons. In: Proceedings of the 31st International Conference on Machine Learning, Lille, France (2015)

Download references

Funding

All authors certify that they have no affiliations with or involvement in any organization or entity with any financial interest or non-financial interest in the subject matter or materials discussed in this manuscript.

Author information

Authors and Affiliations

  1. Coventry University, Coventry, UK

    Long Chen & Jiangtao Wang

  2. Northwestern Polytechnical University, Xi’an, China

    Bin Guo

  3. Ulster University, Newtownabbey, UK

    Liming Chen

Authors
  1. Long Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jiangtao Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bin Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Liming Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Long Chen.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, L., Wang, J., Guo, B. et al. Human-in-the-loop machine learning with applications for population health. CCF Trans. Pervasive Comp. Interact. 5, 1–12 (2023). https://doi.org/10.1007/s42486-022-00115-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s42486-022-00115-4

Keywords

Search

Navigation