Abstract
Considering the large amount of available content, social media platforms increasingly employ machine learning (ML) systems to curate news. This paper examines how well different explanations help expert users understand why certain news stories are recommended to them. The expert users were journalists, who are trained to judge the relevance of news. Surprisingly, none of the explanations are perceived as helpful. Our investigation provides a first indication of a gap between what is available to explain ML-based curation systems and what users need to understand such systems. We call this the Explanatory Gap in Machine Learning-based Curation Systems.
Keywords
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Ackerman, M.S.: The intellectual challenge of CSCW: the gap between social requirements and technical feasibility. Hum.-Comput. Interact. 15(2), 179–203 (2000). https://doi.org/10.1207/S15327051HCI1523_5
Alvarado, O., Heuer, H., Vanden Abeele, V., Breiter, A., Verbert, K.: Middle-aged video consumers’ beliefs about algorithmic recommendations on YouTube. Proc. ACM Hum.-Comput. Interact. 4(CSCW2) (2020). https://doi.org/10.1145/3415192
Alvarado, O., Waern, A.: Towards algorithmic experience: initial efforts for social media contexts. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI 2018, pp. 286:1–286:12. ACM, New York (2018). https://doi.org/10.1145/3173574.3173860, http://doi.acm.org/10.1145/3173574.3173860
Amershi, S., Cakmak, M., Knox, W.B., Kulesza, T.: Power to the people: the role of humans in interactive machine learning. AI Mag. 35(4), 105–120 (2014)
Ardissono, L., Console, L., Torre, I.: An adaptive system for the personalized access to news. AI Commun. 14(3), 129–147 (2001)
Bakshy, E., Rosenn, I., Marlow, C., Adamic, L.A.: The role of social networks in information diffusion. CoRR abs/1201.4145 (2012). http://arxiv.org/abs/1201.4145
Choe, E.K., Lee, N.B., Lee, B., Pratt, W., Kientz, J.A.: Understanding quantified-selfers’ practices in collecting and exploring personal data. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI 2014, pp. 1143–1152. ACM, New York (2014). https://doi.org/10.1145/2556288.2557372, http://doi.acm.org/10.1145/2556288.2557372
Diakopoulos, N., Koliska, M.: Algorithmic transparency in the news media. Digit. J. 5(7), 809–828 (2017). https://doi.org/10.1080/21670811.2016.1208053
Dix, A., Finlay, J., Abowd, G.D., Beale, R.: Human Computer Interaction, 3rd edn. Pearson Prentice Hall, Harlow (2003)
Djajadiningrat, J.P., Gaver, W.W., Fres, J.W.: Interaction relabelling and extreme characters: methods for exploring aesthetic interactions. In: Proceedings of the 3rd Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques. DIS 2000, pp. 66–71. ACM, New York (2000). https://doi.org/10.1145/347642.347664, http://doi.acm.org/10.1145/347642.347664
Eiband, M., Völkel, S.T., Buschek, D., Cook, S., Hussmann, H.: When people and algorithms meet: user-reported problems in intelligent everyday applications. In: Proceedings of the 24th International Conference on Intelligent User Interfaces. IUI 2019, pp. 96–106. ACM, New York (2019). https://doi.org/10.1145/3301275.3302262, http://doi.acm.org/10.1145/3301275.3302262
Eslami, M., et al.: “I always assumed that I wasn’t really that close to [her]”: reasoning about invisible algorithms in news feeds. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. CHI 2015, pp. 153–162. ACM, New York (2015). https://doi.org/10.1145/2702123.2702556, http://doi.acm.org/10.1145/2702123.2702556
Facebook: Facebook News Feed (2018). https://newsfeed.fb.com/
Knight Foundation: American views: Trust, media and democracy, January 2018. https://knightfoundation.org/reports/american-views-trust-media-and-democracy
Geiger, R.S.: Beyond opening up the black box: investigating the role of algorithmic systems in Wikipedian organizational culture. Big Data Soc. 4(2) (2017). https://doi.org/10.1177/2053951717730735, http://journals.sagepub.com/doi/10.1177/2053951717730735
Gena, C., Grillo, P., Lieto, A., Mattutino, C., Vernero, F.: When personalization is not an option: an in-the-wild study on persuasive news recommendation. Information 10(10) (2019). https://doi.org/10.3390/info10100300, https://www.mdpi.com/2078-2489/10/10/300
Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. The MIT Press, Cambridge (2016)
Gottfried, J., Shearer, E.: News Use Across Social Media Platforms 2016, May 2016. http://www.journalism.org/2016/05/26/news-use-across-social-media-platforms-2016/
Green, B., Chen, Y.: The principles and limits of algorithm-in-the-loop decision making. Proc. ACM Hum.-Comput. Interact. 3(CSCW), 50:1–50:24 (2019). https://doi.org/10.1145/3359152, http://doi.acm.org/10.1145/3359152
Griggio, C.F., McGrenere, J., Mackay, W.: Customizations and expression breakdowns in ecosystems of communication apps. In: CSCW 2019, Austin, Texas (2019)
Hamilton, K., Karahalios, K., Sandvig, C., Eslami, M.: A path to understanding the effects of algorithm awareness. In: CHI 2014 Extended Abstracts on Human Factors in Computing Systems. CHI EA 2014, pp. 631–642. ACM, New York (2014). https://doi.org/10.1145/2559206.2578883, http://doi.acm.org/10.1145/2559206.2578883
Heuer, H., Breiter, A.: More than accuracy: towards trustworthy machine learning interfaces for object recognition. In: Proceedings of the 28th ACM Conference on User Modeling, Adaptation and Personalization. UMAP 2020, pp. 298–302. Association for Computing Machinery, New York (2020). https://doi.org/10.1145/3340631.3394873
Heuer, H., Hoch, H., Breiter, A., Theocharis, Y.: Auditing the biases enacted by YouTube for political topics in Germany. In: Proceedings of Mensch Und Computer 2021. MuC 2021. Association for Computing Machinery, New York (2021). https://doi.org/10.1145/3473856.3473864
Heuer, H., Jarke, J., Breiter, A.: Machine learning in tutorials - universal applicability, underinformed application, and other misconceptions. Big Data Soc. 8(1), 20539517211017590 (2021). https://doi.org/10.1177/20539517211017593
Jannach, D., Resnick, P., Tuzhilin, A., Zanker, M.: Recommender systems - beyond matrix completion. Commun. ACM 59(11), 94–102 (2016). https://doi.org/10.1145/2891406. http://doi.acm.org/10.1145/2891406
Jugovac, M., Jannach, D.: Interacting with recommenders - overview and research directions. ACM Trans. Interact. Intell. Syst. 7(3), 10:1–10:46 (2017). https://doi.org/10.1145/3001837, http://doi.acm.org/10.1145/3001837
Kim, B.: Interactive and interpretable machine learning models for human machine collaboration. Ph.D. thesis, Massachusetts Institute of Technology (2015)
Konstan, J.A., Riedl, J.: Recommender systems: from algorithms to user experience. User Model. User-Adap. Inter. 22(1), 101–123 (2012). https://doi.org/10.1007/s11257-011-9112-x
Maron, M.E.: Automatic indexing: an experimental inquiry. J. ACM 8(3), 404–417 (1961). https://doi.org/10.1145/321075.321084. http://doi.acm.org/10.1145/321075.321084
McNee, S.M., Riedl, J., Konstan, J.A.: Being accurate is not enough: how accuracy metrics have hurt recommender systems. In: CHI 2006 Extended Abstracts on Human Factors in Computing Systems. CHI EA 2006, pp. 1097–1101. ACM, New York (2006). https://doi.org/10.1145/1125451.1125659, http://doi.acm.org/10.1145/1125451.1125659
Millecamp, M., Htun, N.N., Conati, C., Verbert, K.: To explain or not to explain: the effects of personal characteristics when explaining music recommendations. In: Proceedings of the 24th International Conference on Intelligent User Interfaces. IUI 2019, pp. 397–407. ACM, New York (2019). https://doi.org/10.1145/3301275.3302313, http://doi.acm.org/10.1145/3301275.3302313
Müller, A., Guido, S.: Introduction to Machine Learning with Python: A Guide for Data Scientists. O’Reilly Media (2016). https://books.google.de/books?id=vbQlDQAAQBAJ
Newman, N., Fletcher, R., Kalogeropoulos, A., Levy, D.A., Nielsen, R.K.: Reuters Institute Digital News Report 2019 (2019). http://www.digitalnewsreport.org/survey/2019/overview-key-findings-2019/
Ng, A.Y., Jordan, M.I.: On discriminative vs. generative classifiers: a comparison of logistic regression and Naive Bayes. In: Dietterich, T.G., Becker, S., Ghahramani, Z. (eds.) Advances in Neural Information Processing Systems 14, pp. 841–848. MIT Press (2002). http://papers.nips.cc/paper/2020-on-discriminative-vs-generative-classifiers-a-comparison-of-logistic-regression-and-naive-bayes.pdf
Powers, D.M.W.: Evaluation: from precision, recall and F-measure to ROC, informedness, markedness & correlation. J. Mach. Learn. Technol. 2(1), 37–63 (2011)
Rader, E., Cotter, K., Cho, J.: Explanations as mechanisms for supporting algorithmic transparency. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. CHI 2018, pp. 103:1–103:13. ACM, New York (2018). https://doi.org/10.1145/3173574.3173677, http://doi.acm.org/10.1145/3173574.3173677
Rader, E., Gray, R.: Understanding user beliefs about algorithmic curation in the Facebook news feed. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. CHI 2015, pp. 173–182. ACM, New York (2015). https://doi.org/10.1145/2702123.2702174, http://doi.acm.org/10.1145/2702123.2702174
Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should i trust you?”: explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD 2016, pp. 1135–1144. ACM, New York (2016). https://doi.org/10.1145/2939672.2939778, http://doi.acm.org/10.1145/2939672.2939778
Rijsbergen, C.J.V.: Information Retrieval, 2nd edn. Butterworth-Heinemann, Newton (1979)
Selvaraju, R.R., Das, A., Vedantam, R., Cogswell, M., Parikh, D., Batra, D.: Grad-CAM: why did you say that? Visual explanations from deep networks via gradient-based localization. CoRR abs/1610.02391 (2016). http://arxiv.org/abs/1610.02391
Sheidin, J., Lanir, J., Kuflik, T., Bak, P.: Visualizing spatial-temporal evaluation of news stories. In: Proceedings of the 22nd International Conference on Intelligent User Interfaces Companion. IUI 2017 Companion, pp. 65–68. ACM, New York (2017). https://doi.org/10.1145/3030024.3040984, http://doi.acm.org/10.1145/3030024.3040984
Shneiderman, B., Plaisant, C.: Designing the User Interface: Strategies for Effective Human-Computer Interaction, 4th edn. Pearson Addison Wesley, Reading (2004)
Smeulders, A.W.M., Worring, M., Santini, S., Gupta, A., Jain, R.: Content-based image retrieval at the end of the early years. IEEE Trans. Pattern Anal. Mach. Intell. 22(12), 1349–1380 (2000). https://doi.org/10.1109/34.895972
Smith, A.: Many Facebook users don’t understand its news feed (2019). http://www.pewresearch.org/fact-tank/2018/09/05/many-facebook-users-dont-understand-how-the-sites-news-feed-works/
Strobelt, H., Gehrmann, S., Huber, B., Pfister, H., Rush, A.M.: Visual analysis of hidden state dynamics in recurrent neural networks. CoRR abs/1606.07461 (2016). http://arxiv.org/abs/1606.07461
Stumpf, S., et al.: Interacting meaningfully with machine learning systems: three experiments. Int. J. Hum.-Comput. Stud. 67(8), 639–662 (2009). https://doi.org/10.1016/j.ijhcs.2009.03.004. http://www.sciencedirect.com/science/article/pii/S1071581909000457
Tintarev, N., Masthoff, J.: Evaluating the effectiveness of explanations for recommender systems: methodological issues and empirical studies on the impact of personalization. User Model. User-Adap. Inter. 22(4–5), 399–439 (2012). https://doi.org/10.1007/s11257-011-9117-5. https://link.springer.com/article/10.1007/s11257-011-9117-5
Trielli, D., Diakopoulos, N.: Search as news curator: the role of Google in shaping attention to news information. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. CHI 2019, pp. 453:1–453:15. ACM, New York (2019). https://doi.org/10.1145/3290605.3300683, http://doi.acm.org/10.1145/3290605.3300683
Tullio, J., Dey, A.K., Chalecki, J., Fogarty, J.: How it works: a field study of non-technical users interacting with an intelligent system. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 31–40. ACM (2007)
Zhang, H.: The optimality of Naive Bayes. In: Barr, V., Markov, Z. (eds.) Proceedings of the Seventeenth International Florida Artificial Intelligence Research Society Conference (FLAIRS 2004). AAAI Press (2004)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Heuer, H. (2021). The Explanatory Gap in Algorithmic News Curation. In: Bright, J., Giachanou, A., Spaiser, V., Spezzano, F., George, A., Pavliuc, A. (eds) Disinformation in Open Online Media. MISDOOM 2021. Lecture Notes in Computer Science(), vol 12887. Springer, Cham. https://doi.org/10.1007/978-3-030-87031-7_1
Download citation
DOI: https://doi.org/10.1007/978-3-030-87031-7_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-87030-0
Online ISBN: 978-3-030-87031-7
eBook Packages: Computer ScienceComputer Science (R0)