Dealing with Mislabeling via Interactive Machine Learning
- 21 Downloads
We propose an interactive machine learning framework where the machine questions the user feedback when it realizes it is inconsistent with the knowledge previously accumulated. The key idea is that the machine uses its available knowledge to check the correctness of its own and the user labeling. The proposed architecture and algorithms run through a series of modes with progressively higher confidence and features a conflict resolution component. The proposed solution is tested in a project on university student life where the goal is to recognize tasks like user location and transportation mode from sensor data. The results highlight the unexpected extreme pervasiveness of annotation mistakes and the advantages provided by skeptical learning.
KeywordsInteractive learning Knowledge and learning Managing annotator mistakes
This research has received funding from the European Union’s Horizon 2020 FET Proactive project “WeNet–The Internet of us”, Grant Agreement No: 823783.
- 4.Diligenti M, Gori M, Saccà C (2017) Semantic-based regularization for learning and inference. Artif Intell 244:143–165. https://doi.org/10.1016/j.artint.2015.08.011. http://www.sciencedirect.com/science/article/pii/S0004370215001344. (Combining constraint solving with mining and learning)MathSciNetCrossRefGoogle Scholar
- 5.Ester M, Kriegel HP, Sander J, Xu X et al (1996) A density-based algorithm for discovering clusters in large spatial databases with noise. Kdd 96:226–231Google Scholar
- 6.Folleco A, Khoshgoftaar T.M, Hulse J.V, Bullard L (2008) Identifying learners robust to low quality data. In: 2008 IEEE International Conference on Information Reuse and Integration, pp 190–195. https://doi.org/10.1109/IRI.2008.4583028
- 7.Frénay B, Kabán A, et al (2014) A comprehensive introduction to label noise. In: ESANNGoogle Scholar
- 9.Giunchiglia F, Zeni M, Bignotti E (2018) Personal context recognition via reliable human-machine collaboration. In: Pervasive Computing and Communications Workshops (PerCom Workshops), 2018 IEEE International Conference on, in printGoogle Scholar
- 10.Guo B, Yu Z, Zhou X, Zhang D (2014) From participatory sensing to mobile crowd sensing. In: Pervasive Computing and Communications Workshops (PERCOM Workshops), 2014 IEEE International Conference on, pp 593–598Google Scholar
- 13.Restuccia F, Ghosh N, Bhattacharjee S, Das SK, Melodia T (2017) Quality of information in mobile crowdsensing: survey and research challenges. ACM Trans Sens Netw (TOSN) 13(4):34Google Scholar
- 15.Settles B (2009) Active learning literature survey. Computer Sciences Technical Report 1648, University of Wisconsin–Madison. http://axon.cs.byu.edu/~martinez/classes/778/Papers/settles.activelearning.pdf
- 16.Teso S, Sebastiani R, Passerini A (2017) Structured learning modulo theories. Artif Intell 244:166–187. https://doi.org/10.1016/j.artint.2015.04.002. http://www.sciencedirect.com/science/article/pii/S0004370215000648. (Combining constraint solving with mining and learning)MathSciNetCrossRefGoogle Scholar
- 18.West BT, Sinibaldi J (2013) The quality of paradata: a literature review. Improving surveys with paradata. Wiley Online Library, pp 339–359Google Scholar
- 19.Zeni M, Zaihrayeu I, Giunchiglia F (2014) Multi-device activity logging. In: Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication, pp 299–302Google Scholar