A systematic review and taxonomy of explanations in decision support and recommender systems

Abstract

With the recent advances in the field of artificial intelligence, an increasing number of decision-making tasks are delegated to software systems. A key requirement for the success and adoption of such systems is that users must trust system choices or even fully automated decisions. To achieve this, explanation facilities have been widely investigated as a means of establishing trust in these systems since the early years of expert systems. With today’s increasingly sophisticated machine learning algorithms, new challenges in the context of explanations, accountability, and trust towards such systems constantly arise. In this work, we systematically review the literature on explanations in advice-giving systems. This is a family of systems that includes recommender systems, which is one of the most successful classes of advice-giving software in practice. We investigate the purposes of explanations as well as how they are generated, presented to users, and evaluated. As a result, we derive a novel comprehensive taxonomy of aspects to be considered when designing explanation facilities for current and future decision support systems. The taxonomy includes a variety of different facets, such as explanation objective, responsiveness, content and presentation. Moreover, we identified several challenges that remain unaddressed so far, for example related to fine-grained issues associated with the presentation of explanations and how explanation facilities are evaluated.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Notes

  1. 1.

    In general, advice-giving system is a term that can also include other types of systems, such as conversational agents or autonomous systems. In this paper, however, we use the term to refer only to the four types of systems listed above.

  2. 2.

    There is a large number of surveys that provide an overview of explanations published elsewhere (Tintarev and Masthoff 2007b, 2011; Swartout and Moore 1993; Chandrasekaran et al. 1989; Dhaliwal and Benbasat 1996; Gregor and Benbasat 1999; Moulin et al. 2002; Lacave and Díez 2002, 2004; Sørmo et al. 2005; Nakatsu 2006; Papadimitriou et al. 2012; Scheel et al. 2014).

  3. 3.

    http://www.periodicos.capes.gov.br/.

  4. 4.

    Further details about which inclusion and exclusion criterion was fulfilled by each individual paper, along with the detailed results of our analysis, can be found online at: http://inf.ufrgs.br/prosoft/resources/2017/umuai-sr-explanations.

  5. 5.

    The interested reader can find the detailed analysis of each of the studies on the following link: http://inf.ufrgs.br/prosoft/resources/sr-explanations.

  6. 6.

    Note that more papers were published in computer science in general over time.

  7. 7.

    See http://www.fatml.org for a recent workshop series on fair, accountable, and transparent machine learning approaches.

  8. 8.

    We remind the reader that approaches that simply provide a rule extraction algorithm without detailing how the explanations are provided to the end user are excluded from our review as specified in our inclusion and exclusion criteria.

  9. 9.

    One of the examined studies (Roitman et al. 2010) focused on safety in the context of decisions with critical consequences. We included it in the category effectiveness.

  10. 10.

    Bilgic and Mooney’s work is not included in this review because it was published in a workshop and not part of the databases searched in our review. We discuss the choice of databases and possible research limitations later in the paper.

  11. 11.

    Some authors measured the accuracy of the decisions or ratings of the study participants. Such a measurement is categorised as investigating the effectiveness of the explanations.

  12. 12.

    Higher deltas mean that users tend to overestimate or underestimate suggested alternatives based on explanations.

  13. 13.

    http://www.eugdpr.org/.

  14. 14.

    http://www.aaai.org/Library/library.php.

References

  1. Abu-Hakima, S., Oppacher, F.: Rationale: reasoning by explaining. In: Proceedings of the Fourth International Conference on Data Engineering, pp. 258–265 (1988)

  2. Allgayer, J., Harbusch, K., Kobsa, A., Reddig, C., Reithinger, N., Schmauks, D.: XTRA: a natural-language access system to expert systems. Int. J. Man Mach. Stud. 31(2), 161–195 (1989)

    Article  Google Scholar 

  3. Amer-Yahia, S., Galland, A., Stoyanovich, J., Yu, C.: From Del.Icio.Us to x.Qui.Site: recommendations in social tagging sites. In: Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, SIGMOD ’08, pp. 1323–1326 (2008)

  4. Artioli, E., Avanzolini, G., Martelli, L., Ursino, M.: An expert system based on causal knowledge: validation on post-cardiosurgical patients. Int. J. Bio Med. Comput. 41(1), 19–37 (1996)

    Article  Google Scholar 

  5. Bader, R., Woerndl, W., Karitnig, A., Leitner, G.: Designing an Explanation Interface for Proactive Recommendations in Automotive Scenarios, pp. 92–104. Springer, Berlin (2012)

    Google Scholar 

  6. Balleda, K., Satyanvesh, D., Sampath, N.V.S.S.P., Varma, K.T.N., Baruah, P.K.: Agpest: an efficient rule-based expert system to prevent pest diseases of rice amp; wheat crops. In: 2014 IEEE 8th International Conference on Intelligent Systems and Control (ISCO), pp. 262–268 (2014)

  7. Banavar, G.: Learning to Trust Artificial Intelligence Systems: Accountability, Compliance and Ethics in the Age of Smart Machines. White paper, IBM Global Services (2016)

  8. Barbieri, N., Bonchi, F., Manco, G.: Who to follow and why: Link prediction with explanations. In: Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’14, pp. 1266–1275 (2014)

  9. Basu, A., Dutta, A.: Computer based support of reasoning in the presence of fuzziness. Decis. Support Syst. 2(3), 235–256 (1986)

    Article  Google Scholar 

  10. Basu, A., Ahad, R.: Using a relational database to support explanation in a knowledge-based system. IEEE Trans. Knowl. Data Eng. 4(6), 572–581 (1992)

    Article  Google Scholar 

  11. Basu, A., Majumdar, A.K., Sinha, S.: An expert system approach to control system design and analysis. IEEE Trans. Syst. Man Cybern. 18(5), 685–694 (1988)

    Article  Google Scholar 

  12. Bau, D.Y., Brezillon, P.J.: Model-based diagnosis of power-station control systems. IEEE Expert 7(1), 36–44 (1992)

    Article  Google Scholar 

  13. Bavota, G., Gethers, M., Oliveto, R., Poshyvanyk, D., Lucia, Ad: Improving software modularization via automated analysis of latent topics and dependencies. ACM Trans. Softw. Eng. Methodol. 23(1), 4:1–4:33 (2014)

    Article  Google Scholar 

  14. Bedi, P., Sharma, R.: Trust based recommender system using ant colony for trust computation. Expert Syst. Appl. 39(1), 1183–1190 (2012)

    Article  Google Scholar 

  15. Bedi, P., Agarwal, S.K., Sharma, S., Joshi, H.: Saprs: situation-aware proactive recommender system with explanations. In: 2014 International Conference on Advances in Computing, Communications and Informatics (ICACCI), pp. 277–283 (2014)

  16. Beiley, J., Duban, S.: Explanation and learning in medicine. In: Kibby, M. (ed.) Computer Assisted Learning, pp. 91–97. Pergamon, Amsterdam (1990)

    Google Scholar 

  17. Belahcene, K., Labreuche, C., Maudet, N., Mousseau, V., Ouerdane, W.: Explaining robust additive utility models by sequences of preference swaps. Theory Decis. 82(2), 151–183 (2017)

    MATH  MathSciNet  Article  Google Scholar 

  18. Benaroch, M.: Roles of design knowledge in knowledge-based systems. Int. J. Hum. Comput. Stud. 44(5), 689–721 (1996)

    Article  Google Scholar 

  19. Bielza, C., Gómez, M., Ríos-Insua, S., Fernándezdel Pozo, J.A.: Structural, elicitation and computational issues faced when solving complex decision making problems with influence diagrams. Comput. Oper. Res. 27(78), 725–740 (2000)

    MATH  Article  Google Scholar 

  20. Bilgic, M., Mooney, R.J.: Explaining recommendations: satisfaction vs. promotion. In: Proceedings of Beyond Personalization 2005: A Workshop on the Next Stage of Recommender Systems Research at the 2005 International Conference on Intelligent User Interfaces. San Diego, CA (2005)

  21. Blake, J.N., Kerr, D.V., Gammack, J.G.: Streamlining patient consultations for sleep disorders with a knowledge-based CDSS. Inf. Syst. 56, 109–119 (2016)

    Article  Google Scholar 

  22. Blanco, R., Ceccarelli, D., Lucchese, C., Perego, R., Silvestri, F.: You should read this! let me explain you why: explaining news recommendations to users. In: Proceedings of the 21st ACM International Conference on Information and Knowledge Management, CIKM ’12, pp. 1995–1999 (2012)

  23. Bofeng, Z., Na, W., Gengfeng, W., Sheng, L.: Research on a personalized expert system explanation method based on fuzzy user model. In: Fifth World Congress on Intelligent Control and Automation, vol. 5, pp. 3996–4000 (2004)

  24. Bohanec, M., Zupan, B., Rajkovic̃, V.: Applications of qualitative multi-attribute decision models in health care. Int. J. Med. Inform. 5859, 191–205 (2000)

    Article  Google Scholar 

  25. Bohnenberger, T., Jacobs, O., Jameson, A., Aslan, I.: Decision-Theoretic Planning Meets User Requirements: Enhancements and Studies of an Intelligent Shopping Guide, pp. 279–296. Springer, Berlin (2005)

    Google Scholar 

  26. Borlea, l., Buta, A., Dusa, V., Lustrea, B.: DIASE—expert system fault diagnosis for Timisoara 220 kV substation. In: EUROCON 2005—The International Conference on “Computer as a Tool”, vol. 1, pp. 221–224 (2005)

  27. Bosnić, Z., Vraćar, P., Radović, M.D., Devedžić, G., Filipović, N.D., Kononenko, I.: Mining data from hemodynamic simulations for generating prediction and explanation models. IEEE Trans. Inf. Technol. Biomed. 16(2), 248–254 (2012)

    Article  Google Scholar 

  28. Bostandjiev, S., O’Donovan, J., Höllerer, T.: Tasteweights: a visual interactive hybrid recommender system. In: Proceedings of the Sixth ACM Conference on Recommender Systems, RecSys ’12, pp. 35–42 (2012)

  29. Briguez, C.E., Budn, M.C., Deagustini, C.A., Maguitman, A.G., Capobianco, M., Simari, G.R.: Argument-based mixed recommenders and their application to movie suggestion. Expert Syst. Appl. 41(14), 6467–6482 (2014)

    Article  Google Scholar 

  30. Buchanan, B.G., Shortliffe, E.H. (eds.): Explanations as a topic of AI research. In: Rule-Based Systems, pp. 331–337. Addison-Wesley, Massachusetts (1984)

  31. Buchanan, B.G., Moore, J.D., Forsythe, D.E., Carenini, G., Ohlsson, S., Banks, G.: An intelligent interactive system for delivering individualized information to patients. Artif. Intell. Med. 7(2), 117–154 (1995)

    Article  Google Scholar 

  32. Burattini, E., Gregorio, M.D., Tamburrini, G.: Hybrid expert systems: An approach to combining neural computation and rule-based reasoning. In: Leondes, C.T. (ed.) Expert Systems, pp. 1315–1354. Academic Press, Burlington (2002)

    Google Scholar 

  33. Buschner, S., Schirru, R., Zieschang, H., Junker, P.: Providing recommendations for horizontal career change. In: Proceedings of the 14th International Conference on Knowledge Technologies and Data-driven Business, i-KNOW ’14, pp. 33:1–33:4 (2014)

  34. Bussone, A., Stumpf, S., O’Sullivan, D.: The role of explanations on trust and reliance in clinical decision support systems. In: 2015 International Conference on Healthcare Informatics, pp. 160–169 (2015)

  35. Cagnoni, S., Coppini, G., Livi, R., Poli, R., Scarpelli, P.T., Valli, G.: A neural network expert system for computer-assisted analysis of blood-pressure data. In: Proceedings Computers in Cardiology, pp. 473–476 (1991)

  36. Carenini, G., Moore, J.D.: An empirical study of the influence of user tailoring on evaluative argument effectiveness. In: Proceedings of the 17th International Joint Conference on Artificial Intelligence, IJCAI’01, pp. 1307–1312 (2001)

  37. Carenini, G., Moore, J.D.: Generating and evaluating evaluative arguments. Artif. Intell. 170, 925–952 (2006)

    Article  Google Scholar 

  38. Castro, C., Bose, A., Handschin, E., Hoffmann, W.: Comparison of different screening techniques for the contingency selection function. Int. J. Electr. Power Energy Syst. 18(7), 425–430 (1996)

    Article  Google Scholar 

  39. Chandrasekaran, B., Mittal, S.: Deep versus compiled knowledge approaches to diagnostic problem-solving. Int. J. Hum. Comput. Stud. 51(2), 357–368 (1999)

    Article  Google Scholar 

  40. Chandrasekaran, B., Tanner, M.C., Josephson, J.R.: Explaining control strategies in problem solving. IEEE Expert Intell. Syst. Appl. 4(1), 9-15–19-24 (1989)

    Google Scholar 

  41. Chang, C.C., Hsieh, S.C.: Applying web service technology to build a wireless lan problem diagnosis expert system. In: 2010 International Conference on Computational Aspects of Social Networks, pp. 217–220 (2010)

  42. Charissiadis, A., Karacapilidis, N.: Strengthening the Rationale of Recommendations Through a Hybrid Explanations Building Framework, pp. 311–323. Springer, Berlin (2015)

    Google Scholar 

  43. Chelsom, J.J., Ellis, T.J., Carson, E.R., Cramp, D.G.: Blood gas analysis: a knowledge-based adviser for the interpretation of results. In: Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 1360–1361 (1988)

  44. Chen, L., Wang, F.: Sentiment-enhanced explanation of product recommendations. In: Proceedings of the 23rd International Conference on World Wide Web, WWW ’14 Companion, pp. 239–240 (2014)

  45. Chen, W., Hsu, W., Lee, M.L.: Tagcloud-based explanation with feedback for recommender systems. In: Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’13, pp. 945–948 (2013a)

  46. Chen, Y.C., Lin, Y.S., Shen, Y.C., Lin, S.D.: A modified random walk framework for handling negative ratings and generating explanations. ACM Trans. Intell. Syst. Technol. 4(1), 12:1–12:21 (2013b)

    Article  Google Scholar 

  47. Cheng, S.J., Chen, D.S., Peng, X.L.: An expert system for a thermal power station alarm processing. In: International Conference on Advances in Power System Control, Operation and Management, APSCOM-91, pp. 316–320 (1991)

  48. Chiou, A., Yu, X.: Industrial decision support system (IDSS) in weed control and management strategies: expert advice using descriptive schemata and explanatory capabilities. In: IECON 2007—33rd Annual Conference of the IEEE Industrial Electronics Society, pp. 105–110 (2007)

  49. Chouicha, M., Siller, T.: An expert system approach to liquefaction analysis part 1: development and implementation. Comput. Geotech. 16(1), 1–35 (1994)

    Article  Google Scholar 

  50. Cleger-Tamayo, S., Fernandez-Luna, J.M., Huete, J.F.: Explaining neighborhood-based recommendations. In: Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’12, pp. 1063–1064 (2012)

  51. Davey-Wilson, I.: Development of a prolog-based expert system for groundwater control. Comput. Struct. 40(1), 185–189 (1991)

    MATH  Article  Google Scholar 

  52. David, J.M., Krivine, J.P.: Designing knowledge-based systems within functional architecture: the DIVA experiment. In: Proceedings of the Fifth Conference on Artificial Intelligence Applications, pp. 173–180 (1989)

  53. Davis, K.: DORIS (diagnostic oriented rockwell intelligent system). IEEE Aerosp. Electron. Syst. Mag. 1(7), 18–21 (1986)

    Article  Google Scholar 

  54. de Braal, L., Ezquerra, N., Garcia, E., Cooke, C., Krawczynska, E.: PERFUSE: an interactive knowledge-based system for the interpretation and explanation of cardiac imagery. In: Proceedings of 18th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, vol. 3, pp. 1238–1239 (1996)

  55. Deep, R., Czech, D.R., Dizek, S.G., Kennedy, D.K.: A bit-mapping classifier expert system in warranty selection. In: Proceedings of the IEEE 1988 National Aerospace and Electronics Conference, pp. 1222–1224 (1988)

  56. Dhaliwal, J.S., Benbasat, I.: The use and effects of knowledge-based system explanations: theoretical foundations and a framework for empirical evaluation. Inf. Syst. Res. 7(3), 342–362 (1996)

    Article  Google Scholar 

  57. Diederich, J.: Explanation and artificial neural networks. Int. J. Man Mach. Stud. 37(3), 335–355 (1992)

    MathSciNet  Article  Google Scholar 

  58. Du, G., Ruhe, G.: Two machine-learning techniques for mining solutions of the releaseplanner decision support system. Inf. Sci. 259, 474–489 (2014)

    Article  Google Scholar 

  59. Ehrlich, K., Kirk, S.E., Patterson, J., Rasmussen, J.C., Ross, S.I., Gruen, D.M.: Taking advice from intelligent systems: the double-edged sword of explanations. In: Proceedings of the 16th International Conference on Intelligent User Interfaces, IUI ’11, pp. 125–134 (2011)

  60. Ezquerra, N., de Braal, L., Garcia, E., Cooke, C., Krawczynska, E.: Interactive, knowledge-guided visualization of 3D medical imagery. Future Gener. Comput. Syst. 15(1), 59–73 (1999)

    Article  Google Scholar 

  61. Felfernig, A.: Koba4ms: selling complex products and services using knowledge-based recommender technologies. In: Seventh IEEE International Conference on E-Commerce Technology (CEC’05), pp. 92–100 (2005)

  62. Felfernig, A., Gula, B.: An empirical study on consumer behavior in the interaction with knowledge-based recommender applications. In: The 8th IEEE International Conference on E-Commerce Technology and the 3rd IEEE International Conference on Enterprise Computing, E-Commerce, and E-Services (CEC/EEE’06), pp. 37–37 (2006)

  63. Fong, J., Lam, H.P., Robinson, R., Indulska, J.: Defeasible preferences for intelligible pervasive applications to enhance eldercare. In: IEEE International Conference on Pervasive Computing and Communications Workshops, pp. 572–577 (2012)

  64. Friedrich, G., Zanker, M.: A taxonomy for generating explanations in recommender systems. AI Mag. 32(3), 90–98 (2011)

    Article  Google Scholar 

  65. Gallagher, S., Trainor, J., Murphy, M., Curran, E.: A knowledge based system for competitive bidding. In: Proceedings of the 1995 Second New Zealand International Two-Stream Conference on Artificial Neural Networks and Expert Systems, pp. 314–317 (1995)

  66. Garca, A.J., Chesevar, C.I., Rotstein, N.D., Simari, G.R.: Formalizing dialectical explanation support for argument-based reasoning in knowledge-based systems. Expert Syst. Appl. 40(8), 3233–3247 (2013)

    Article  Google Scholar 

  67. Gedikli, F., Ge, M., Jannach, D.: Understanding Recommendations by Reading the Clouds, pp. 196–208. Springer, Berlin (2011)

    Google Scholar 

  68. Gedikli, F., Jannach, D., Ge, M.: How should i explain? A comparison of different explanation types for recommender systems. Int. J. Hum. Comput. Stud. 72(4), 367–382 (2014)

    Article  Google Scholar 

  69. Giboney, J.S., Brown, S.A., Lowryc, P.B., Nunamaker Jr., J.F.: User acceptance of knowledge-based system recommendations: explanations, arguments, and fit. Decis. Support Syst. 72, 1–10 (2015)

    Article  Google Scholar 

  70. Gkika, S., Lekakos, G.: Investigating the effectiveness of persuasion strategies on recommender systems. In: Proceedings of the 9th International Workshop on Semantic and Social Media Adaptation and Personalization, SMAP ’14, pp. 94–97. IEEE Computer Society, Washington, DC, USA (2014)

  71. Glaser, B.G.: Basics of Grounded Theory Analysis: Emergence vs. Forcing. Sociology Pr, Mill Valley (1992)

    Google Scholar 

  72. Gómez-Vallejo, H.J., Uriel-Latorre, B., Sande-Meijide, M., Villamarín-Bello, B., Pavón, R., Fdez-Riverol, F., Glez-Peña, D.: A case-based reasoning system for aiding detection and classification of nosocomial infections. Decis. Support Syst. 84, 104–116 (2016)

    Article  Google Scholar 

  73. Gönül, M.S., Önkal, D., Lawrence, M.: The effects of structural characteristics of explanations on use of a dss. Decis. Support Syst. 42(3), 1481–1493 (2006)

    Article  Google Scholar 

  74. Goud, R., Hasman, A., Peek, N.: Development of a guideline-based decision support system with explanation facilities for outpatient therapy. Comput. Methods Progr. Biomed 91(2), 145–153 (2008)

    Article  Google Scholar 

  75. Gowri, K., Marsh, C., Bedard, C., Fazio, P.: Knowledge-based assistant for aluminum component design. Comput. Struct. 38(1), 9–20 (1991)

    MATH  Article  Google Scholar 

  76. Grando, M.A., Moss, L., Glasspool, D., Sleeman, D., Sim, M., Gilhooly, C., Kinsella, J.: Argumentation-Logic for Explaining Anomalous Patient Responses to Treatments, pp. 35–44. Springer, Berlin (2011)

  77. Gregor, S.: Explanations from knowledge-based systems and cooperative problem solving. Int. J. Hum. Comput. Stud. 54(1), 81–105 (2001)

    MATH  Article  Google Scholar 

  78. Gregor, S., Benbasat, I.: Explanations from intelligent systems: theoretical foundations and implications for practice. MIS Q. 23(4), 497–530 (1999)

    Article  Google Scholar 

  79. Grierson, D.E., Cameron, G.E.: A knowledge-based expert system for computer automated structural design. Comput. Struct. 30(3), 741–745 (1988)

    Article  Google Scholar 

  80. Guida, G., Zanella, M.: Active operator support: a case study in steel production. In: IEEE International Conference on Systems, Man and Cybernetics. Intelligent Systems for the 21st Century, vol. 4, pp. 3340–3345 (1995)

  81. Guida, G., Mussio, P., Zanella, M.: User interaction in decision support systems: the role of justification. In: 1997 IEEE International Conference on Systems, Man, and Cybernetics. Computational Cybernetics and Simulation, vol. 4, pp. 3215–3220 (1997)

  82. Guy, I., Zwerdling, N., Carmel, D., Ronen, I., Uziel, E., Yogev, S., Ofek-Koifman, S.: Personalized recommendation of social software items based on social relations. In: Proceedings of the Third ACM Conference on Recommender Systems, RecSys ’09, pp. 53–60 (2009)

  83. Guy, I., Zwerdling, N., Ronen, I., Carmel, D., Uziel, E.: Social media recommendation based on people and tags. In: Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’10, pp. 194–201 (2010)

  84. Gvenir, H., Emeksiz, N.: An expert system for the differential diagnosis of erythemato-squamous diseases. Expert Syst. Appl. 18(1), 43–49 (2000)

    Article  Google Scholar 

  85. Hair, D.C., Pickslay, K., Chow, S.: Explanation-based decision support in real time situations. In: Proceedings of the Fourth International Conference on Tools with Artificial Intelligence, TAI ’92, pp. 22–25 (1992)

  86. Hanshi, W., Qiujie, F., Lizhen, L., Wei, S.: A probabilistic rating prediction and explanation inference model for recommender systems. China Commun. 13(2), 79–94 (2016)

    Google Scholar 

  87. Hasling, D.W., Clancey, W.J., Rennels, G.: Strategic explanations for a diagnostic consultation system. Int. J. Man Mach. Stud. 20(1), 3–19 (1984)

    Article  Google Scholar 

  88. Hatzilygeroudis, I., Prentzas, J.: Symbolic-neural rule based reasoning and explanation. Expert Syst. Appl. 42(9), 4595–4609 (2015)

    Article  Google Scholar 

  89. Helms, G.L., Richardson, J.W., Cochran, M.J., Rister, M.: A farm level expert simulation system to aid farmers in selecting among crop insurance strategies. Comput. Electron. Agric. 4(3), 169–190 (1990)

    Article  Google Scholar 

  90. Herlocker, J.L., Konstan, J.A., Riedl, J.: Explaining collaborative filtering recommendations. In: Proceedings of the 2000 ACM Conference on Computer Supported Cooperative Work, CSCW ’00, pp. 241–250 (2000)

  91. Hodgkinson, L., Walker, E.: An expert system for credit evaluation and explanation. J. Comput. Sci. Coll. 19(1), 62–72 (2003)

    Google Scholar 

  92. Holman, J.G., Wolff, A.H.: An expert adviser for oliguria occurring on the intensive care unit. In: Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 1442–1443 (1988)

  93. Horan, J., O’Sullivan, B.: Towards diverse relaxations of over-constrained models. In: 2009 21st IEEE International Conference on Tools with Artificial Intelligence, pp. 198–205 (2009)

  94. Horn, W., Popow, C., Miksch, S., Seyfang, A.: Quicker, more accurate nutrition plans for newborn infants. IEEE Intell. Syst. Appl. 13(1), 65–69 (1998)

    Article  Google Scholar 

  95. Hornung, T., Ziegler, C.N., Franz, S., Przyjaciel-Zablocki, M., Schtzle, A., Lausen, G.: Evaluating hybrid music recommender systems. In: IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT), vol. 1, pp. 57–64 (2013)

  96. Hu, Y., Koren, Y., Volinsky, C.: Collaborative filtering for implicit feedback datasets. In: 2008 Eighth IEEE International Conference on Data Mining, pp. 263–272 (2008)

  97. Hudson, D.L., Cohen, M.E.: Human–computer interaction in a medical decision support system. In: Proceedings of the Twenty-Second Annual Hawaii International Conference on System Sciences. Volume II: Software Track, vol. 2, pp. 429–435 (1989)

  98. Hunt, J., Price, C.: Explaining qualitative diagnosis. Eng. Appl. Artif. Intell. 1(3), 161–169 (1988)

    Article  Google Scholar 

  99. Hussain, S., Abidi, S.S.R.: Ontology driven CPG authoring and execution via a semantic web framework. In: 40th Annual Hawaii International Conference on System Sciences, HICSS 2007, pp. 135–135 (2007)

  100. Hussein, T., Neuhaus, S.: Explanation of spreading activation based recommendations. In: Proceedings of the 1st International Workshop on Semantic Models for Adaptive Interactive Systems, SEMAIS ’10, pp. 24–28 (2010)

  101. Jabri, M.A.: Knowledge-based system design using prolog: the PIAF experience. Knowl. Based Syst. 2(1), 72–79 (1989)

    Article  Google Scholar 

  102. Jaimes, A., Gatica-Perez, D., Sebe, N., Huang, T.S.: Guest editors’ introduction: human-centered computing-toward a human revolution. Computer 40(5), 30–34 (2007)

    Article  Google Scholar 

  103. Jamieson, P.W.: A model for diagnosing and explaining multiple disorders. Comput. Biomed. Res. 24(4), 307–320 (1991)

    Article  Google Scholar 

  104. Janjua, N.K., Hussain, F.K.: Defeasible reasoning based argumentative Web-IDSS for virtual teams (VTs). In: 2011 IEEE/WIC/ACM International Conferences on Web Intelligence and Intelligent Agent Technology, pp. 330–334 (2011)

  105. Jannach, D., Adomavicius, G.: Recommendations with a purpose. In: Proceedings of the 2016 ACM Conference on Recommender Systems, RecSys ’16, pp. 7–10 (2016)

  106. Jannach, D., Zanker, M., Felfernig, A., Friedrich, G.: Recommender Systems: An Introduction. Cambridge University Press, New York (2010)

    Google Scholar 

  107. Jannach, D., Resnick, P., Tuzhilin, A., Zanker, M.: Recommender systems—beyond matrix completion. Commun. ACM 59(11), 94–102 (2016)

    Article  Google Scholar 

  108. Ji, K., Shen, H.: Jointly modeling content, social network and ratings for explainable and cold-start recommendation. Neurocomputing 218, 1–12 (2016)

    Article  Google Scholar 

  109. Joch, J., Dudeck, J.: Decision support for infectious diseasesa working prototype. Int. J. Med. Inform. 64(23), 331–340 (2001)

    Article  Google Scholar 

  110. Jugovac, M., Jannach, D.: Interacting with recommenders—overview and research directions. ACM Trans. Interact. Intell. Syst. 7(3), 46 (2017)

  111. Jung, D., Burns, J.R.: Connectionist approaches to inexact reasoning and learning systems for executive and decision support. Decis. Support Syst. 10(1), 37–66 (1993)

    Article  Google Scholar 

  112. Junker, U.: Quickxplain: preferred explanations and relaxations for over-constrained problems. In: AAAI’04, pp. 167–172. USA (2004)

  113. Kadhim, M.A., Alam, M.A., Kaur, H.: Design and implementation of intelligent agent and diagnosis domain tool for rule-based expert system. In: 2013 International Conference on Machine Intelligence and Research Advancement, pp. 619–622 (2013)

  114. Kagal, L., Pato, J.: Preserving privacy based on semantic policy tools. IEEE Secur. Priv. 8(4), 25–30 (2010)

    Article  Google Scholar 

  115. Karwowski, W., Mulholland, N.O., Ward, T.L., Jagannathan, V.: A fuzzy knowledge base of an expert system for analysis of manual lifting tasks. Fuzzy Sets Syst. 21(3), 363–374 (1987)

    Article  Google Scholar 

  116. Katarya, R., Jain, I., Hasija, H.: An interactive interface for instilling trust and providing diverse recommendations. In: International Conference on Computer and Communication Technology (ICCCT), pp. 17–22 (2014)

  117. Keeney, R.L., Raiffa, H.: Decisions with Multiple Objectives: Preferences and Value Tradeoffs. Wiley Series in Probability and Mathematical Statistics. Wiley, Hoboken (1976)

    Google Scholar 

  118. Kim, B.O., Lee, S.M.: A bond rating expert system for industrial companies. Expert Syst. Appl. 9(1), 63–70 (1995)

    Article  Google Scholar 

  119. Kim, S.K., Park, J.I.: A structural equation modeling approach to generate explanations for induced rules. Expert Syst. Appl. 10(3), 403–416 (1996)

    Article  Google Scholar 

  120. Kitchenham, B., Charters, S.: Guidelines for performing systematic literature reviews in software engineering. Technical Report EBSE-2007-01, School of Computer Science and Mathematics, Keele University (2007)

  121. Kitchenham, B., Brereton, P.: A systematic review of systematic review process research in software engineering. Inf. Softw. Technol. 55(12), 2049–2075 (2013)

    Article  Google Scholar 

  122. Klein, D.A., Shortliffe, E.H.: A framework for explaining decision-theoretic advice. Artif. Intell. 67(2), 201–243 (1994)

    MATH  Article  Google Scholar 

  123. Koussev, T., Weiss, M.P., Reiss, K.: A graphic explanation environment for expert systems. In: Second International Conference on Software Engineering for Real Time Systems, pp. 11–15 (1989)

  124. Labreuche, C.: A general framework for explaining the results of a multi-attribute preference model. Artif. Intell. 175(7), 1410–1448 (2011)

    MATH  MathSciNet  Article  Google Scholar 

  125. Lacave, C., Díez, F.J.: A review of explanation methods for bayesian networks. Knowl. Eng. Rev. 17(2), 107–127 (2002)

    Article  Google Scholar 

  126. Lacave, C., Díez, F.J.: A review of explanation methods for heuristic expert systems. Knowl. Eng. Rev. 19(2), 133–146 (2004)

  127. Lacave, C., Oniśko, A., Díez, F.J.: Use of Elvira’s explanation facility for debugging probabilistic expert systems. Knowl. Based Syst. 19(8), 730–738 (2006)

    Article  Google Scholar 

  128. Lambert, S.C., Ringland, G.A.: Knowledge representations and interfaces in financial expert systems. In: UK IT 1990 Conference, pp. 434–441 (1990)

  129. Langlotz, C.P., Shortliffe, E.H.: Adapting a consultation system to critique user plans. Int. J. Man Mach. Stud. 19(5), 479–496 (1983)

    Article  Google Scholar 

  130. Lee, H.M., Hsu, C.C.: Building expert systems by training with automatic neural network generating ability. In: Proceedings Eighth Conference on Artificial Intelligence for Applications, pp. 197–203 (1992)

  131. Levy, M., Ferrand, P., Chirat, V.: SESAM-DIABETE, an expert system for insulin-requiring diabetic patient education. Comput. Biomed. Res. 22(5), 442–453 (1989)

    Article  Google Scholar 

  132. Li, M., Gregor, S.: Outcomes of effective explanations: empowering citizens through online advice. Decis. Support Syst. 52(1), 119–132 (2011)

    Article  Google Scholar 

  133. Libório, A., Furtado, E., Rocha, I., Furtado, V.: Interface design through knowledge-based systems: an approach centered on explanations from problem-solving models. In: Proceedings of the 4th International Workshop on Task Models and Diagrams, TAMODIA ’05, pp. 127–134 (2005)

  134. Lieberman, H., van Dyke, N., Vivacqua, A.: Let’s browse: a collaborative browsing agent. Knowl. Based Syst. 12(8), 427–431 (1999)

    Article  Google Scholar 

  135. Liu, K.F.R., Lee, J., Chiang, W., Yang, S.J.: Fpnes: fuzzy Petri net based expert system for bridges damage assessment. In: Proceedings Tenth IEEE International Conference on Tools with Artificial Intelligence, pp. 302–309 (1998)

  136. Lopez-Suarez, A., Kamel, M.: Dykor: a method for generating the content of explanations in knowledge systems. Knowl. Based Syst. 7(3), 177–188 (1994)

    Article  Google Scholar 

  137. Machado, R.J., da Rocha, A.F.: Inference, inquiry and explanation in expert systems by means of fuzzy neural networks. In: Proceedings of the Second IEEE International Conference on Fuzzy Systems, vol. 1, pp. 351–356 (1993)

  138. Mahmoud, M., Algadi, N., Ali, A.: Expert system for banking credit decision. In: 2008 International Conference on Computer Science and Information Technology, pp. 813–819 (2008)

  139. Malheiro, N., Vale, Z.A., Ramos, C., Santos, J., Marques, A.: Enabling Client-Server Explanation Facilities in a Real-Time Expert System, pp. 333–342. Springer, Berlin (1999)

    Google Scholar 

  140. Mao, J.Y., Benbasat, I.: The effects of contextualized access to knowledge on judgement. Int. J. Hum. Comput. Stud. 55(5), 787–814 (2001)

    MATH  Article  Google Scholar 

  141. Martincic, C.J.: QUE: an expert system explanation facility that answers “why not” types of questions. J. Comput. Sci. Coll. 19(1), 336–348 (2003)

    Google Scholar 

  142. Marx, P., Hennig-Thurau, T., Marchand, A.: Increasing consumers’ understanding of recommender results: a preference-based hybrid algorithm with strong explanatory power. In: Proceedings of the Fourth ACM Conference on Recommender Systems, RecSys ’10, pp. 297–300 (2010)

  143. Matelli, J.A., Bazzo, E., da Silva, J.C.: An expert system prototype for designing natural gas cogeneration plants. Expert Syst. Appl. 36(4), 8375–8384 (2009)

    Article  Google Scholar 

  144. Matsatsinis, N., Doumpos, M., Zopounidis, C.: Knowledge acquisition and representation for expert systems in the field of financial analysis. Expert Syst. Appl. 12(2), 247–262 (1997)

    Article  Google Scholar 

  145. Maybury, M.T.: Enhancing explanation coherence with rhetorical strategies. In: Proceedings of the Fourth Conference on European Chapter of the Association for Computational Linguistics, EACL ’89, pp. 168–173 (1989)

  146. McCarthy, K., Reilly, J., McGinty, L., Smyth, B.: Experiments in dynamic critiquing. In: Proceedings of the 10th International Conference on Intelligent User Interfaces, IUI ’05, pp. 175–182. ACM (2005)

  147. Mcsherry, D.: Explanation in recommender systems. Artif. Intell. Rev. 24(2), 179–197 (2005)

    MATH  Article  Google Scholar 

  148. Mejia-Lavalle, M.: Outlier detection with innovative explanation facility over a very large financial database. In: 2010 IEEE Electronics, Robotics and Automotive Mechanics Conference, pp. 23–27 (2010)

  149. Mendes, D., Rodrigues, I.P., Baeta, C.: Ontology based clinical practice justification in natural language. Procedia Technol. 9, 1288–1293 (2013)

    Article  Google Scholar 

  150. Metzler, D.P., Martincic, C.J.: QUE: explanation through exploration. Expert Syst. Appl. 15(34), 253–263 (1998)

    Article  Google Scholar 

  151. Mitra, S.: Fuzzy mlp based expert system for medical diagnosis. Fuzzy Sets Syst. 65(2), 285–296 (1994)

    Article  Google Scholar 

  152. Mitra, S., Pal, S.K.: Fuzzy multi-layer perceptron, inferencing and rule generation. IEEE Trans. Neural Netw. 6(1), 51–63 (1995)

    Article  Google Scholar 

  153. Mller-Kolck, U.: Expert system support for the therapeutic management of cerebrovascular disease. Artif. Intell. Med. 2(1), 35–42 (1990)

    Article  Google Scholar 

  154. Mocanu, A.: Envisioning a collaborative smart home solution based on argumentative dialogues. In: Proceedings of the 7th Balkan Conference on Informatics Conference, BCI ’15, pp. 23:1–23:6 (2015)

  155. Moulin, B., Irandoust, H., Bélanger, M., Desbordes, G.: Explanation and argumentation capabilities: towards the creation of more persuasive agents. Artif. Intell. Rev. 17(3), 169–222 (2002)

    MATH  Article  Google Scholar 

  156. Muhammad, K., Lawlor, A., Rafter, R., Smyth, B.: Great Explanations: Opinionated Explanations for Recommendations, pp. 244–258. Springer, Berlin (2015)

    Google Scholar 

  157. Muhammad, K.I., Lawlor, A., Smyth, B.: A live-user study of opinionated explanations for recommender systems. In: Proceedings of the 21st International Conference on Intelligent User Interfaces, IUI ’16, pp. 256–260 (2016)

  158. Murphy, D.S., Phillips, M.E.: The effects of expert system use on entry-level accounting expertise: an experiment. Expert Syst. Appl. 3(1), 129–134 (1991)

    Article  Google Scholar 

  159. Nakatsu, R.T.: Explanatory Power of Intelligent Systems, pp. 123–143. Springer, London (2006)

    Google Scholar 

  160. Nakatsu, R.T., Benbasat, I.: Improving the explanatory power of knowledge-based systems: an investigation of content and interface-based enhancements. Trans. Syst. Man Cybern. Part A 33(3), 344–357 (2003)

    Article  Google Scholar 

  161. Narayanan, T., McGuinness, D.L.: Towards leveraging inference web to support intuitive explanations in recommender systems for automated career counseling. In: First International Conference on Advances in Computer–Human Interaction, pp. 164–169 (2008)

  162. Nart, D.D., Tasso, C.: A personalized concept-driven recommender system for scientific libraries. Procedia Comput. Sci. 38, 84–91 (2014)

    Article  Google Scholar 

  163. Ng, G., Ong, K.: Using a qualitative probabilistic network to explain diagnostic reasoning in an expert system for chest pain diagnosis. Comput. Cardiol. 2000(27), 569–572 (2000)

    Google Scholar 

  164. Nilashi, M., Jannach, D., bin Ibrahim, O., Esfahani, M.D., Ahmadi, H.: Recommendation quality, transparency, and website quality for trust-building in recommendation agents. Electron. Commer. Res. Appl. 19, 70–84 (2016)

    Article  Google Scholar 

  165. Norton, S.W.: An explanation mechanism for bayesian inferencing systems. In: Lemmer, J.F., Kanal, L.N. (eds.) Uncertainty in Artificial Intelligence, Machine Intelligence and Pattern Recognition, vol. 5, pp. 165–173. North-Holland, Amsterdam (1988)

    Google Scholar 

  166. Nunes, I., Miles, S., Luck, M., de Lucena, C.J.P.: Investigating explanations to justify choice. In: Proceedings of the 20th International Conference on User Modeling, Adaptation, and Personalization, UMAP’12, pp. 212–224 (2012a)

  167. Nunes, I., Chen, Y., Miles, S., Luck, M., Lucena, C.: Transparent Provenance Derivation for User Decisions, pp. 111–125. Springer, Berlin (2012b)

    Google Scholar 

  168. Nunes, I., Miles, S., Luck, M., Barbosa, S., Lucena, C.: Pattern-based explanation for automated decisions. In: Proceedings of the Twenty-first European Conference on Artificial Intelligence, ECAI’14, pp. 669–674 (2014)

  169. Nuthall, P., Bishop-Hurley, G.: Expert systems for animal feeding management part i: presentation aspects. Comput. Electron. Agric. 14(1), 9–22 (1996)

    Article  Google Scholar 

  170. O’Donovan, J., Gretarsson, B., Bostandjiev, S., Hollerer, T., Smyth, B.: A visual interface for social information filtering. In: 2009 International Conference on Computational Science and Engineering, vol. 4, pp. 74–81 (2009)

  171. Omran, A.M., Khorshid, M.: Intelligent environmental scanning approach (a case study: the Egyptian wheat crop production). IERI Procedia 7, 28–34 (2014a)

    Article  Google Scholar 

  172. Omran, A.M., Khorshid, M.: An intelligent recommender system for long view of Egypt’s livestock production. AASRI Procedia 6, 103–110 (2014b)

    Article  Google Scholar 

  173. Oramas, S., Espinosa-Anke, L., Sordo, M., Saggion, H., Serra, X.: Information extraction for knowledge base construction in the music domain. Data Knowl. Eng. 106, 70–83 (2016)

    Article  Google Scholar 

  174. Overby, M.A.: Psyxpert: an expert system prototype for aiding psychiatrists in the diagnosis of psychotic disorders. Comput. Biol. Med. 17(6), 383–393 (1987)

    Article  Google Scholar 

  175. Pal, K.: An approach to legal reasoning based on a hybrid decision-support system. Expert Syst. Appl. 17(1), 1–12 (1999)

    MathSciNet  Article  Google Scholar 

  176. Pal, K., Palmer, O.: A decision-support system for business acquisitions. Decis. Support Syst. 27(4), 411–429 (2000)

    Article  Google Scholar 

  177. Papamichail, K., French, S.: Explaining and justifying the advice of a decision support system: a natural language generation approach. Expert Syst. Appl. 24(1), 35–48 (2003)

    Article  Google Scholar 

  178. Papadimitriou, A., Symeonidis, P., Manolopoulos, Y.: A generalized taxonomy of explanations styles for traditional and social recommender systems. Data Min. Knowl. Discov. 24(3), 555–583 (2012)

    Article  Google Scholar 

  179. Pazzani, M., Iyer, R., See, D., Schroeder, E., Tilles, J.: CTSHIV: a knowledge-based system for the management of HIV-infected patients. In: Intelligent Information Systems, 1997. IIS ’97, pp. 7–13 (1997)

  180. Perlin, M., Kanal, E., John, A.: A user interface for visualizing concepts in magnetic resonance imaging. In: Proceedings of the First Conference on Visualization in Biomedical Computing, pp. 260–267 (1990)

  181. Popchev, I.P., Zlatareva, N.P., Sinapova, L.J.: EDDY: an expert system in dysmorphology based on truth-maintenance. In: Images of the Twenty-First Century. Proceedings of the Annual International Engineering in Medicine and Biology Society, pp. 1877–1878 (1989)

  182. Pu, P., Chen, L.: Trust-inspiring explanation interfaces for recommender systems. Knowl. Based Syst. 20(6), 542–556 (2007)

    Article  Google Scholar 

  183. Rahwan, I., Simari, G.R.: Argumentation in Artificial Intelligence, 1st edn. Springer, Berlin (2009)

    Google Scholar 

  184. Ramberg, R.: Construing and testing explanations in a complex domain. Comput. Hum. Behav. 12(1), 29–48 (1996)

    Article  Google Scholar 

  185. Ray, A.K.: Equipment fault diagnosisa neural network approach. Comput. Ind. 16(2), 169–177 (1991)

    Article  Google Scholar 

  186. Reggia, J.A., Perricone, B.T., Nau, D.S., Peng, Y.: Answer justification in diagnostic expert systems—part I: abductive inference and its justification. IEEE Trans. Biomed. Eng. BME–32(4), 263–267 (1985)

    Article  Google Scholar 

  187. Reilly, J., McCarthy, K., McGinty, L., Smyth, B.: Explaining compound critiques. Artif. Intell. Rev. 24(2), 199–220 (2005)

    Article  Google Scholar 

  188. Reyes, A., Ibarguengoytia, P.H., Elizalde, F., Snchez, L., Nava, A.: ASISTO: an integrated intelligent assistant system for power plant operation and training. In: 16th International Conference on Intelligent System Applications to Power Systems, pp. 1–6 (2011)

  189. Richards, D.: The reuse of knowledge: a user-centred approach. Int. J. Hum. Comput. Stud. 52(3), 553–579 (2000)

    Article  Google Scholar 

  190. Ringer, M.J., Quinn, T.M., Merolla, A.: Autonomous power system: intelligent diagnosis and control. Telemat. Inform. 8(4), 365–383 (1991)

    Article  Google Scholar 

  191. Riordan, D., Carden, K.J.: Explanation in ecological systems. In: Proceedings of the 1990 ACM SIGSMALL/PC Symposium on Small Systems, SIGSMALL ’90, pp. 249–254 (1990)

  192. Roitman, H., Messika, Y., Tsimerman, Y., Maman, Y.: Increasing patient safety using explanation-driven personalized content recommendation. In: Proceedings of the 1st ACM International Health Informatics Symposium, IHI ’10, pp. 430–434 (2010)

  193. Rook, F.W., Donnell, M.L.: Human cognition and the expert system interface: mental models and inference explanations. IEEE Trans. Syst. Man Cybern. 23(6), 1649–1661 (1993)

    Article  Google Scholar 

  194. Samarasinghe, S.: Neural Networks for Applied Sciences and Engineering. Auerbach Publications, Boston (2006)

    Google Scholar 

  195. Santoso, N.I., Darken, C., Povh, G., Erdmann, J.: Nuclear plant fault diagnosis using probabilistic reasoning. In: Proceedings of the 1999 IEEE Power Engineering Society Summer Meeting, vol. 2, pp. 714–719 (1999)

  196. Sarkar, A., Bandyopadhyay, S., Jullien, G.A.: Bit-level designer’s assistant-a knowledge based approach to systolic processor design. In: Proceedings of the 33rd Midwest Symposium on Circuits and Systems, pp. 1001–1004 (1990)

  197. Saunders, V.M., Dobbs, V.S.: Explanation generation in expert systems. In: IEEE Conference on Aerospace and Electronics, pp. 1101–1106 (1990)

  198. Schaffer, J., Giridhar, P., Jones, D., Höllerer, T., Abdelzaher, T., O’Donovan, J.: Getting the message? A study of explanation interfaces for microblog data analysis. In: Proceedings of the 20th International Conference on Intelligent User Interfaces, IUI ’15, pp. 345–356 (2015)

  199. Scheel, C., Castellanos, A., Lee, T., De Luca, E.W.: The Reason Why: A Survey of Explanations for Recommender Systems, pp. 67–84. Springer, Berlin (2014)

    Google Scholar 

  200. Schröder, O., Möbus, C., Folckers, J., Thole, H.J.: Supporting the construction of explanation models and diagnostic reasoning in probabilistic domains. In: Proceedings of the 1996 International Conference on Learning Sciences, ICLS ’96, pp. 60–67 (1996)

  201. Shaalan, K., Rafea, M., Rafea, A.: KROL: a knowledge representation object language on top of Prolog. Expert Syst. Appl. 15(1), 33–46 (1998)

    Article  Google Scholar 

  202. Sharma, A., Cosley, D.: Do social explanations work? Studying and modeling the effects of social explanations in recommender systems. In: Proceedings of the 22nd International Conference on World Wide Web, WWW ’13, pp. 1133–1144 (2013)

  203. Sherchan, W., Loke, S.W., Krishnaswamy, S.: Explanation-aware service selection: rationale and reputation. Serv. Oriented Comput. Appl. 2(4), 203–218 (2008)

    Article  Google Scholar 

  204. Shoval, P.: Principles, procedures and rules in an expert system for information retrieval. Inf. Process. Manag. 21(6), 475–487 (1985)

    Article  Google Scholar 

  205. Slagle, J.R.: Applications of a generalized network-based expert system shell-artificial intelligence mini-tutorial. In: Proceedings of the Symposium on the Engineering of Computer-Based Medical, pp. 33–42 (1988)

  206. Slotnick, S.A., Moore, J.D.: Explaining quantitative systems to uninitiated users. Expert Syst. Appl. 8(4), 475–490 (1995)

    Article  Google Scholar 

  207. Song, W., Shi, H., Li, Q.: Study of an explanation mechanism in expert system based on fault tree for safety risk assessment. In: 2nd International Conference on Future Computer and Communication, vol. 2, pp. V2–479–V2–483 (2010)

  208. Sørmo, F., Cassens, J., Aamodt, A.: Explanation in case-based reasoning-perspectives and goals. Artif. Intell. Rev. 24(2), 109–143 (2005)

    MATH  Article  Google Scholar 

  209. Srivastava, R.P.: Automating judgmental decisions using neural networks: a model for processing business loan applications. In: Proceedings of the 1992 ACM Annual Conference on Communications, CSC ’92, pp. 351–357 (1992)

  210. Strachan, S.M., McArthur, S.D.J., Judd, M.D., McDonald, J.R.: Incremental knowledge-based partial discharge diagnosis in oil-filled power transformers. In: Proceedings of the 13th International Conference on Intelligent Systems Application to Power Systems (2005)

  211. Strat, T.M., Lowrance, J.D.: Explaining evidential analyses. Int. J. Approx. Reason. 3(4), 299–353 (1989)

    MATH  Article  Google Scholar 

  212. Štrumbelj, E., Kononenko, I., Šikonja, M.R.: Explaining instance classifications with interactions of subsets of feature values. Data Knowl. Eng. 68(10), 886–904 (2009)

    Article  Google Scholar 

  213. Suermondt, H.J., Cooper, G.F.: An evaluation of explanations of probabilistic inference. Comput. Biomed. Res. 26(3), 242–254 (1993)

    Article  Google Scholar 

  214. Swartout, W.R., Moore, J.D.: Explanation in Second Generation Expert Systems, pp. 543–585. Springer, Berlin (1993)

    Google Scholar 

  215. Swinney, L.: The explanation facility and the explanation effect. Expert Syst. Appl. 9(4), 557–567 (1995)

    Article  Google Scholar 

  216. Symeonidis, P., Nanopoulos, A., Manolopoulos, Y.: Providing justifications in recommender systems. IEEE Trans. Syst. Man Cybern. Part A Syst. Hum. 38(6), 1262–1272 (2008)

    Article  Google Scholar 

  217. Tan, W.K., Tan, C.H., Teo, H.H.: Consumer-based decision aid that explains which to buy: decision confirmation or overconfidence bias? Decis. Support Syst. 53(1), 127–141 (2012)

    Article  Google Scholar 

  218. Tanner, M.C., Keuneke, A.M.: Explanations in knowledge systems: the roles of the task structure and domain functional models. IEEE Expert 6(3), 50–57 (1991)

    Article  Google Scholar 

  219. Terano, T., Suzuki, M., Onoda, T., Uenishi, K., Matsuura, T.: CSES: an approach to integrating graphic, music and voice information into a user-friendly interface. In: International Workshop on Industrial Applications of Machine Intelligence and Vision, pp. 349–354 (1989)

  220. Thirumuruganathan, S., Huber, M.: Building bayesian network based expert systems from rules. In: 2011 IEEE International Conference on Systems, Man, and Cybernetics, pp. 3002–3008 (2011)

  221. Tintarev, N., Masthoff, J.: Effective explanations of recommendations: user-centered design. In: Proceedings of the 2007 ACM Conference on Recommender Systems, RecSys ’07, pp. 153–156 (2007a)

  222. Tintarev, N., Masthoff, J.: A survey of explanations in recommender systems. In: Proceedings of the 2007 IEEE 23rd International Conference on Data Engineering Workshop, pp. 801–810 (2007b)

  223. Tintarev, N., Masthoff, J.: Designing and evaluating explanations for recommender systems. In: Ricci, F., Rokach, L., Shapira, B., Kantor, P.B. (eds.) Recommender Systems Handbook, pp. 479–510. Springer, Berlin (2011)

    Google Scholar 

  224. Tintarev, N., Masthoff, J.: Evaluating the effectiveness of explanations for recommender systems. User Model. User Adapt. Interact. 22(4–5), 399–439 (2012)

    Article  Google Scholar 

  225. Tjahjadi, T., Bowen, D., Bevan, J.R.: 3M: a user modelling interface of an expert system for x-ray topographic image interpretation. Interact. Comput. 2(3), 259–278 (1990)

    Article  Google Scholar 

  226. Tong, L.C.: An explanation facility for a grammar writing system. In: Proceedings of the 13th Conference on Computational Linguistics, COLING ’90, pp. 359–364 (1990)

  227. Tong, X., Ang, J.: Explaining control strategies in second generation expert systems. IEEE Trans. Syst. Man Cybern. 25(11), 1483–1490 (1995)

    Article  Google Scholar 

  228. Toulmin, S.E.: The Uses of Argument. Cambridge University Press, Cambridge (2003)

    Google Scholar 

  229. Tzafestas, S., Konstantinidis, N.: ENGEXP—an integrated environment for the development and application of expert systems in equipment and engine fault diagnosis and repair. Adv. Eng. Softw. 14(1), 3–14 (1992)

    Article  Google Scholar 

  230. van Aarle, E., van den Bercken, J.: The development of a knowledge-based system supporting the diagnosis of reading and spelling problems. Comput. Hum. Behav. 8(23), 183–201 (1992)

    Article  Google Scholar 

  231. Vashisth, P., Chandoliya, D., Yadav, B.K., Bedi, P.: Trust enabled argumentation based recommender system. In: 12th International Conference on Intelligent Systems Design and Applications (ISDA), pp. 137–142 (2012)

  232. Vig, J., Sen, S., Riedl, J.: Tagsplanations: explaining recommendations using tags. In: Proceedings of the 14th International Conference on Intelligent User Interfaces, IUI ’09, pp. 47–56 (2009)

  233. Vogiatzis, D., Karkaletsis, V.: A cognitive framework for robot guides in art collections. Univers. Access Inf. Soc. 10(2), 179–193 (2011)

    Article  Google Scholar 

  234. Wall, R., Cunningham, P., Walsh, P., Byrne, S.: Explaining the output of ensembles in medical decision support on a case by case basis. Artif. Intell. Med. 28(2), 191–206 (2003)

    Article  Google Scholar 

  235. Wang, L., Libert, G., Liu, B.: An expert system for forecasting model selection. In: Proceedings of the First IEEE Conference on Control Applications, pp. 704–709 (1992)

  236. Wang, N., Pynadath, D.V., Hill, S.G.: The impact of pomdp-generated explanations on trust and performance in human-robot teams. In: Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems, AAMAS ’16, pp. 997–1005 (2016a)

  237. Wang, N., Pynadath, D.V., Hill, S.G.: Trust calibration within a human-robot team: comparing automatically generated explanations. In: Proceedings of the 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp. 109–116 (2016b)

  238. Wang, W., Qiu, L., Kim, D., Benbasat, I.: Effects of rational and social appeals of online recommendation agents on cognition- and affect-based trust. Decis. Support Syst. 86(C), 48–60 (2016c)

    Article  Google Scholar 

  239. Washington, E.S., Ali, M.: PISCES: an expert system for coal fired power plant monitoring and diagnostics. In: Proceedings of the 1st International Conference on Industrial and Engineering Applications of Artificial Intelligence and Expert Systems, IEA/AIE ’88, pp. 87–93 (1988)

  240. Wick, M.R., Slagle, J.R.: An explanation facility for today’s expert systems. IEEE Expert 4(1), 26–36 (1989a)

    Article  Google Scholar 

  241. Wick, M.R., Slagle, J.R.: The partitioned support network for expert system justification. IEEE Trans. Syst. Man Cybern. 19(3), 528–535 (1989b)

    Article  Google Scholar 

  242. Widyantoro, D.H., Baizal, Z.K.A.: A framework of conversational recommender system based on user functional requirements. In: 2nd International Conference on Information and Communication Technology (ICoICT), pp. 160–165 (2014)

  243. Wong, K.P., Cheung, H.N.: Expert system for protection current transformer design specification preparation. IEE Proc. C Gener. Transm. Distrib. 136(6), 391–400 (1989)

    Article  Google Scholar 

  244. Yasdi, R.: Design of the exis’s explanation component. Comput. Ind. 13(1), 15–21 (1989)

    Article  Google Scholar 

  245. Ye, L.R.: The value of explanation in expert systems for auditing: an experimental investigation. Expert Syst. Appl. 9(4), 543–556 (1995)

    Article  Google Scholar 

  246. Ye, L.R., Johnson, P.E.: The impact of explanation facilities on user acceptance of expert systems advice. MIS Q. 19, 157–172 (1995)

    Article  Google Scholar 

  247. Yen, J.: Gertis: a dempster-shafer approach to diagnosing hierarchical hypotheses. Commun. ACM 32(5), 573–585 (1989)

    Article  Google Scholar 

  248. Yoon, Y., Guimaraes, T., Swales, G.: Integrating artificial neural networks with rule-based expert systems. Decis. Support Syst. 11(5), 497–507 (1994)

    Article  Google Scholar 

  249. Yu, C., Lakshmanan, L., Amer-Yahia, S.: It takes variety to make a world: diversification in recommender systems. In: Proceedings of the 12th International Conference on Extending Database Technology: Advances in Database Technology, EDBT ’09, pp. 368–378 (2009)

  250. Zain, M.F.M., Islam, M.N., Basri, I.H.: An expert system for mix design of high performance concrete. Adv. Eng. Softw. 36(5), 325–337 (2005)

    MATH  Article  Google Scholar 

  251. Zanker, M.: The influence of knowledgeable explanations on users’ perception of a recommender system. In: Proceedings of the Sixth ACM Conference on Recommender Systems, RecSys ’12, pp. 269–272 (2012)

  252. Zanker, M., Ninaus, D.: Knowledgeable explanations for recommender systems. In: 2010 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology, vol. 1, pp. 657–660 (2010)

  253. Zeleznikow, J., Stranieri, A., Gawler, M.: Project report: split-up—a legal expert system which determines property division upon divorce. Artif. Intell. Law 3(4), 267–275 (1995)

    Article  Google Scholar 

  254. Zhang, Y., Lai, G., Zhang, M., Zhang, Y., Liu, Y., Ma, S.: Explicit factor models for explainable recommendation based on phrase-level sentiment analysis. In: Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR ’14, pp. 83–92 (2014)

Download references

Acknowledgements

The authors would like to thank Michael Jugovac for carefully proofreading this paper. Ingrid Nunes also would like to thank for research grants CNPq ref. 303232/2015-3, CAPES ref. 7619-15-4, and Alexander von Humboldt, ref. BRA 1184533 HFSTCAPES-P.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Ingrid Nunes.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Nunes, I., Jannach, D. A systematic review and taxonomy of explanations in decision support and recommender systems. User Model User-Adap Inter 27, 393–444 (2017). https://doi.org/10.1007/s11257-017-9195-0

Download citation

Keywords

  • Explanation
  • Decision support system
  • Recommender system
  • Expert system
  • Knowledge-based system
  • Systematic review
  • Machine learning
  • Trust
  • Artificial intelligence