Have It Both Ways—From A/B Testing to A&B Testing with Exceptional Model Mining

  • Wouter DuivesteijnEmail author
  • Tara Farzami
  • Thijs Putman
  • Evertjan Peer
  • Hilde J. P. Weerts
  • Jasper N. Adegeest
  • Gerson Foks
  • Mykola Pechenizkiy
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10536)


In traditional A/B testing, we have two variants of the same product, a pool of test subjects, and a measure of success. In a randomized experiment, each test subject is presented with one of the two variants, and the measure of success is aggregated per variant. The variant of the product associated with the most success is retained, while the other variant is discarded. This, however, presumes that the company producing the products only has enough capacity to maintain one of the two product variants. If more capacity is available, then advanced data science techniques can extract more profit for the company from the A/B testing results. Exceptional Model Mining is one such advanced data science technique, which specializes in identifying subgroups that behave differently from the overall population. Using the association model class for EMM, we can find subpopulations that prefer variant A where the general population prefers variant B, and vice versa. This data science technique is applied on data from StudyPortals, a global study choice platform that ran an A/B test on the design of aspects of their website.


A/B testing Exceptional Model Mining Association Online controlled experiments E-commerce Website optimization 


  1. 1.
    Adeyemi, O.: Measures of association for research in educational planning and administration. Res. J. Math. Stat. 3(3), 82–90 (2010)MathSciNetGoogle Scholar
  2. 2.
    Duivesteijn, W., Feelders, A., Knobbe, A.J.: Different slopes for different folks – mining for exceptional regression models with cook’s distance. In: Proceedings of KDD, pp. 868–876 (2012)Google Scholar
  3. 3.
    Duivesteijn, W., Feelders, A.J., Knobbe, A.: Exceptional model mining – supervised descriptive local pattern mining with complex target concepts. Data Min. Knowl. Disc. 30(1), 47–98 (2016)CrossRefGoogle Scholar
  4. 4.
    Hand, D.J.: Pattern detection and discovery. In: Hand, D.J., Adams, N.M., Bolton, R.J. (eds.) Pattern Detection and Discovery. LNCS (LNAI), vol. 2447, pp. 1–12. Springer, Heidelberg (2002). CrossRefGoogle Scholar
  5. 5.
    Jakowski, M., Jaroszewicz, S.: Uplift modeling for clinical trial data. In: Proceedings of ICML 2012 Workshop on Machine Learning for Clinical Data Analysis (2012)Google Scholar
  6. 6.
    Kohavi, R., Longbotham, R.: Online controlled experiments and A/B tests. In: Sammut, C., Webb, G.I. (eds.) Encyclopedia of Machine Learning and Data Mining, pp. 1–8. Springer, New York (2016). Google Scholar
  7. 7.
    Kohavi, R., Longbotham, R., Sommerfield, D., Henne, R.M.: Controlled experiments on the web: survey and practical guide. Data Min. Knowl. Discov. 18(1), 140–181 (2009)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Kleinberg, J., Papadimitrou, C., Raghavan, P.: A microeconomic view of data mining. Data Min. Knowl. Disc. 2(4), 311–324 (1998)CrossRefGoogle Scholar
  9. 9.
    Klösgen, W.: Explora: a multipattern and multistrategy discovery assistant. In: Advances in Knowledge Discovery and Data Mining, pp. 249–271 (1996)Google Scholar
  10. 10.
    Krak, T.E., Feelders, A.: Exceptional model mining with tree-constrained gradient ascent. In: Proceedings of SDM, pp. 487–495 (2015)Google Scholar
  11. 11.
    Lavrač, N., Kavšek, B., Flach, P.A., Todorovski, L.: Subgroup discovery with CN2-SD. J. Mach. Learn. Res. 5, 153–188 (2004)MathSciNetGoogle Scholar
  12. 12.
    van Leeuwen, M.: Maximal exceptions with minimal descriptions. Data Min. Knowl. Discov. 21(2), 259–276 (2010)MathSciNetCrossRefGoogle Scholar
  13. 13.
    Leman, D., Feelders, A., Knobbe, A.: Exceptional model mining. In: Daelemans, W., Goethals, B., Morik, K. (eds.) ECML PKDD 2008. LNCS (LNAI), vol. 5212, pp. 1–16. Springer, Heidelberg (2008). CrossRefGoogle Scholar
  14. 14.
    Lemmerich, F., Becker, M., Atzmueller, M.: Generic pattern trees for exhaustive exceptional model mining. In: Flach, P.A., De Bie, T., Cristianini, N. (eds.) ECML PKDD 2012. LNCS (LNAI), vol. 7524, pp. 277–292. Springer, Heidelberg (2012). CrossRefGoogle Scholar
  15. 15.
    Mannila, H., Toivonen, H.: Levelwise search and borders of theories in knowledge discovery. Data Min. Knowl. Discov. 1(3), 241–258 (1997)CrossRefGoogle Scholar
  16. 16.
    Moens, S., Boley, M.: Instant exceptional model mining using weighted controlled pattern sampling. In: Blockeel, H., van Leeuwen, M., Vinciotti, V. (eds.) IDA 2014. LNCS, vol. 8819, pp. 203–214. Springer, Cham (2014). Google Scholar
  17. 17.
    Morik, K., Boulicaut, J.-F., Siebes, A. (eds.): Local Pattern Detection. Springer, Heidelberg (2005). Google Scholar
  18. 18.
    Rzepakowski, P., Jaroszewicz, S.: Decision trees for uplift modeling with single and multiple treatments. Knowl. Inf. Syst. 32(2), 303–327 (2012)CrossRefGoogle Scholar
  19. 19.
    Rebelo de Sá, C., Duivesteijn, W., Soares, C., Knobbe, A.: Exceptional preferences mining. In: Calders, T., Ceci, M., Malerba, D. (eds.) DS 2016. LNCS (LNAI), vol. 9956, pp. 3–18. Springer, Cham (2016). CrossRefGoogle Scholar
  20. 20.
    Siroker, D., Koomen, P.: A/B Testing: The Most Powerful Way to Turn Clicks Into Customers. Wiley, Hoboken (2013)Google Scholar
  21. 21.
  22. 22.
    Tang, L., Jiang, Y., Li, L., Li, T.: Ensemble contextual bandits for personalized recommendation. In: Proceedings of RecSys, pp. 73–80 (2014)Google Scholar
  23. 23.
    Tang, L., Rosales, R., Singh, A.P., Agarwal, D.: Automatic ad format selection via contextual bandits. In: Proceedings of CIKM, pp. 1587–1594 (2013)Google Scholar
  24. 24.
    Williams, J.J., Li, N., Kim, J., Whitehill, J., Maldonado, S., Pechenizkiy, M., Chu, L., Heffernan, N.: MOOClets: A Framework for Improving Online Education through Experimental Comparison and Personalization of Modules. Working Paper No. 2523265 (2014).
  25. 25.
    Wrobel, S.: An algorithm for multi-relational discovery of subgroups. In: Komorowski, J., Zytkow, J. (eds.) PKDD 1997. LNCS, vol. 1263, pp. 78–87. Springer, Heidelberg (1997). CrossRefGoogle Scholar
  26. 26.
    Žliobaitė, I., Pechenizkiy, M.: Learning with actionable attributes: attention - boundary cases! In: Proceedings of ICDM Workshops, pp. 1021–1028 (2010)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Wouter Duivesteijn
    • 1
    Email author
  • Tara Farzami
    • 2
  • Thijs Putman
    • 2
  • Evertjan Peer
    • 1
  • Hilde J. P. Weerts
    • 1
  • Jasper N. Adegeest
    • 1
  • Gerson Foks
    • 1
  • Mykola Pechenizkiy
    • 1
  1. 1.Technische Universiteit EindhovenEindhoventhe Netherlands
  2. 2.StudyPortals B.V.Eindhoventhe Netherlands

Personalised recommendations