Machine Learning

, Volume 88, Issue 3, pp 369-398

First online:

How to reverse-engineer quality rankings

  • Allison ChangAffiliated withOperations Research Center, Mass. Institute of Technology Email author 
  • , Cynthia RudinAffiliated withMIT Sloan School of Management, Mass. Institute of Technology
  • , Michael CavarettaAffiliated withFord Motor Company
  • , Robert ThomasAffiliated withFord Motor Company
  • , Gloria ChouAffiliated withFord Motor Company


A good or bad product quality rating can make or break an organization. However, the notion of “quality” is often defined by an independent rating company that does not make the formula for determining the rank of a product publicly available. In order to invest wisely in product development, organizations are starting to use intelligent approaches for determining how funding for product development should be allocated. A critical step in this process is to “reverse-engineer” a rating company’s proprietary model as closely as possible. In this work, we provide a machine learning approach for this task, which optimizes a certain rank statistic that encodes preference information specific to quality rating data. We present experiments on data from a major quality rating company, and provide new methods for evaluating the solution. In addition, we provide an approach to use the reverse-engineered model to achieve a top ranked product in a cost-effective way.


Supervised ranking Quality ratings Discrete optimization Reverse-engineering Applications of machine learning