Applying matrix factorization to consistency-based direct diagnosis

Configuration systems must be able to deal with inconsistencies which can occur in different contexts. Especially in interactive settings, where users specify requirements and a constraint solver has to identify solutions, inconsistencies may more often arise. In inconsistency situations, there is a need of diagnosis methods that support the identification of minimal sets of constraints that have to be adapted or deleted in order to restore consistency. A diagnosis algorithm’s performance can be evaluated in terms of time to find a diagnosis (runtime) and diagnosis quality. Runtime efficiency of diagnosis is especially crucial in real-time scenarios such as production scheduling, robot control, and communication networks. However, there is a trade off between diagnosis quality and the runtime efficiency of diagnostic reasoning. In this article, we deal with solving the quality-runtime performance trade off problem of direct diagnosis. In this context, we propose a novel learning approach based on matrix factorization for constraint ordering. We show that our approach improves runtime performance and diagnosis quality at the same time.

the corresponding conflict sets [5]. When diagnoses have to be provided in real-time, response times should be less than a few seconds [2]. For example, in communication networks, efficient diagnosis is crucial to retain the quality of service. However, in direct diagnosis approaches, there is a clear trade-off between runtime performance of diagnosis calculation and diagnosis quality [6].
To address this challenge, we propose Learned Constraint Ordering (LCO) for direct diagnosis. Our approach learns constraint ordering heuristics from inconsistent historical transactions which include inconsistent user requirements. For example, our method learns that "display quality" is a more important criterion than "price" according to the past purchases and user requirements regarding a set of digital cameras. For this reason, "display quality" is regarded as more important and "price" would be a candidate for change proposal. For example, a diagnosis would recommend to increase the price in order to be able to find an item that satisfies all requirements. Using historically inconsistent transactions, we build a sparse matrix and then employ matrix factorization techniques to estimate future diagnoses. After an offline learning phase, the most similar transaction to the new inconsistent requirement set is found and the corresponding constraint ordering heuristic (which is calculated in the offline phase) is applied to reorder the inconsistent constraints before direct diagnosis is achieved. Thanks to the learned ordering, direct diagnosis algorithms can solve the diagnosis task with a high quality diagnosis result in a shorter runtime compared to direct diagnosis without constraint ordering. We provide a working example to demonstrate the effects of our approach. Finally, based on experimental evaluations, we show that our constraint ordering approach with direct diagnosis is superior to the baseline (direct diagnosis algorithms without constraint ordering) on popular benchmark constraint satisfaction problems.

Preliminaries
In this section, we give an overview of the basic definitions in consistency-based configuration and diagnosis, and introduce a running example. Finally, we explain our evaluation criteria for the determined diagnoses.

Configuration task
The following (simplified) assortment of digital cameras (see Table 1) and a set of inconsistent user requirements (see Table 2) for selecting a digital camera from the camera product table will serve as a working example to demonstrate how our approach works.
Our working example is represented as a configuration task in Table 2 (on the basis of Definition 1). As shown in Table 2, our example configuration consists of a variable set (V ) with 10 variables (which are also listed in the first column of Table 1) and only one knowledge base constraint (c 1 ) which represents the set of available cameras shown in Table 1). User related preferences are defined in the set of user requirements (REQ Lisa ). Definition 1 (Configuration task) A configuration task can be defined as a CSP (V , D, C). V = {v 1 , v 2 , ... , v n } represents a set of finite domain variables. D = {dom(v 1 ), dom(v 2 ), ..., dom(v n )} represents a set of is a set of domain specific constraints (the configuration knowledge base) that restricts the possible combinations of variable values. REQ = {c q+1 , c q+2 , .., c t } is a set of user requirements, which is also represented as constraints. A configuration (S) for a configuration task is a set of assignments S = {v 1 = a 1 , v 2 = a 2 , ..., v n = a n } where a i ∈ dom(v i ) which is consistent with the constraints in C.

Diagnosis task
In an interactive configuration scenario (see Table 1), a configuration may not find a solution due to the fact that some constraints are inconsistent (see Table 2). Such a "no solution could be found" dilemma is caused by at least one conflict between constraints in the knowledge base C KB and the user requirements REQ assuming the knowledge base C KB to be consistent. Note that it could also be the case that C KB itself is inconsistent. Definition 2 introduces a formal representation of a conflict.

Definition 2 (Conflict) A conflict is a set of constraints
If we have an inconsistency in our knowledge base, we can say that C KB ∪ REQ is a conflict set. Definition 3 introduces the concept of minimal conflicts.
Applying matrix factorization to consistency-based direct diagnosis 7025 Definition 3 (Minimal conflict) A minimal conflict CS is a conflict (see Definition 2) where CS only contains constraints which are responsible for the conflict, s.t.
Our example contains three minimal conflict sets. CS 1 = {c 2 , c 3 } because there is no product with c 2 : effectiveResolution = 20.9 Megapixel and c 3 : display= 2.5inches in the product table C KB , CS 2 = {c 2 , c 9 } since there is no product with c 2 : effectiveResolution= 20.9Megapixel and c 9 : zoom= 5.8x, and CS 3 = {c 3 , c 9 } there is no product with c 3 : display = 2.5inches and c 9 : zoom = 5.8x.
In such cases, we can help users to resolve the conflicts with a diagnosis Δ (a set of constraints). Assuming that C KB is consistent, we can say that the knowledge base always will be consistent if we remove REQ. Removing Δ from REQ leads to a consistent knowledge base (see Definition 4).
Definition 4 (REQ diagnosis task) A user requirements diagnosis task is defined as a tuple (C KB , REQ) where REQ is a set of given user requirements and C KB represents the constraints part of the configuration knowledge base. A diagnosis for a REQ diagnosis task The feasibility (consistency) of the user requirements can be checked using a CSP solver [4,9]. If a CSP is consistent, there exists a solution i.e., solve(CSP ) = ∅ where solve(CSP : {V , D, C}) includes a set of assignments S (a configuration/solution).
In Definition 5, we introduce the term minimal diagnosis which helps to reduce the number of constraints within a diagnosis.
Definition 5 (Minimal diagnosis) A minimal diagnosis Δ is a diagnosis (see Definition 4) where there doesn't exist a subset Δ ⊂ Δ which has the diagnosis property.
The REQ diagnosis task triggered by the requirements of the user Lisa (REQ Lisa ) has two corresponding minimal diagnoses. The removal of the set Δ Lisa 1 = {c 2 , c 9 } or Δ Lisa 2 = {c 3 , c 9 } leads to a consistent configuration knowledge base, i.e., REQ Lisa − Δ Lisa 1 ∪ C KB is consistent and REQ Lisa − Δ Lisa 2 ∪ C KB is consistent.

Direct diagnosis
Algorithmic approaches to provide efficient solutions for diagnosis problems are many-fold. Basically, there are two types of approaches which are conflict-directed diagnosis [19] and direct diagnosis [7].
Conflict-directed diagnosis algorithms calculate conflicts which are then used to find diagnoses. Their runtime performance is often not sufficient especially in real-time scenarios. Direct diagnosis algorithms determine diagnoses by executing a series of queries. These queries check the consistency of the constraint set without the need of precalculated conflict sets. FLEXDIAG [7] is a direct diagnosis algorithm which determines one diagnosis at a time which indicates variable assignments of the original configuration that have to be changed such that a reconfiguration conform to the new requirements is possible. FLEXDIAG is constraint-ordering sensitive.

Constraint ordering
The quality of diagnoses and runtime performance of direct diagnosis algorithms are based on the ordering of the constraints in the set of user requirements: the lower the importance of a constraint the lower the index of the constraint. The lower the ordering of a conflicting constraint, the higher the probability that this constraint will be part of the diagnosis [5].
Users typically prefer to keep the important requirements and to change or delete (if needed) the less important ones [8]. The major goal of (model-based) diagnosis tasks is to identify the preferred (leading) diagnoses [3]. For the characterization of a preferred diagnosis we will rely on a total ordering of the given set of constraints in REQ. Such a total ordering can be achieved, for example, by directly asking the customer regarding the preferences, by applying multi attribute utility theory where the determined interest dimensions correspond with the attributes of REQ or by applying the orderings determined by conjoint analysis [16].

Evaluation criteria
We can evaluate the performance of a direct diagnosis algorithm based on runtime performance, diagnosis quality (in terms of minimality), and combined performance (runtime and minimality).
Runtime runtime(Δ) represents the time needed by the diagnostic search to find Δ. This spent time can be measured in milliseconds or in terms of number of consistency checks (#CC) applied till a diagnosis is found. For a more accurate runtime measurement (excluding the operating system's effects on runtime, etc.), the number of consistency checks can be used.
Minimality Diagnosis quality can be measured in terms of the degree of minimality of the constraints in a diagnosis, the cardinality of Δ compared to the cardinality of Δ min . |Δ min | represents the cardinality of a minimal diagnosis. The highest (best) minimality can be 1 according to Formula (1).
Combined Since it is important to satisfy both evaluation criteria runtime performance and minimality at the same time, we also evaluate combined performance based on the Formula (2). combined(Δ) increases when minimality(Δ) increases and/or runtime(Δ) decreases. This means, the direct diagnosis algorithm provides a diagnosis with a high minimality and low runtime while the combined performance is high.
In order to improve the runtime performance of diagnostic search, FLEXDIAG [6] uses a parameter (m) that helps to systematically reduce the number of consistency checks but at the same time the minimality of diagnoses is deteriorated. In FLEXDIAG, the parameter m is used to control diagnosis quality in terms of minimality, accuracy, and the performance of diagnostic search. The higher the value of m the higher the performance of FLEXDIAG and the lower the degree of diagnosis quality.

Related work
The most widely known algorithm for the identification of minimal diagnoses is hitting set directed acyclic graph (HSDAG) [14]. HSDAG is based on conflict-directed hitting set determination and determines diagnoses based on breadth-first search. It computes minimal diagnoses (also minimal cardinality diagnoses) using minimal conflict sets which can be calculated by QUICKXPLAIN [8]. The major disadvantage of applying this approach is the need of predetermining minimal conflicts which can deteriorate diagnostic search performance [5]. Many different approaches to provide efficient solutions for diagnosis problems are proposed. One approach [24] focuses on improvements of HSDAG. Another approach [23] uses pre-determined sets of conflicts based on binary decision diagrams. In diagnosis scenarios instances where the number of minimal diagnoses and their cardinality is high, the determination of diagnoses with standard conflictbased approaches becomes infeasible.
The direct diagnosis algorithm FLEXDIAG [7] utilizes an inverse version of QUICKXPLAIN [8] which finds directly a diagnosis from an inconsistent constraint set. FLEXDIAG is an extension of FASTDIAG which assures diagnosis determination within certain time limits by systematically reducing the number of solver calls needed. The authors claim that this specific interpretation of anytime diagnosis leads to a trade-off between diagnosis quality (evaluated, e.g., in terms of minimality) and the time needed for diagnosis determination. Our proposed constraint ordering approach LCO improves their direct diagnosis approach in terms of diagnosis quality and at the same time in terms of runtime performance.
The approach of [18] determines diagnoses directly. Authors reduce the number of consistency checks by avoiding the computation of minimized conflict sets. Their approach is similar to [5], but introduces a new pruning rule. They compared their approach only with the standard technique (based on QUICKXPLAIN [8] and HSDAG [14]). In their experiments, they show that their direct diagnosis approach outperforms the standard diagnosis approach in terms of runtime. In their approach, authors collect the constraint ordering directly from users (interactively). Compared to the work [18], our approach learns the constraint ordering from historical inconsistent user requirements and their preferred diagnoses.
The importance of constraint ordering in direct diagnosis scenarios has already been mentioned in related work [7,18]. In our approach, we predict the most important constraints for the users based on an important collaborative filtering approach matrix factorization [10] and employing historical transactions. We learn a constraint ordering where the predicted most important constraints have the highest ordering. This is due to the fact that, the implemented direct diagnosis algorithm FLEXDIAG first starts searching a diagnosis among the constraints ranked lowest. Therefore, the highest ordering constraints have low probability to be in the diagnosis set.

Learned Constraint Ordering (LCO) for direct diagnosis
In this paper, our motivation is to solve the qualityruntime performance trade off problem of direct diagnosis. For this purpose, our proposed method learns constraint ordering heuristics based on historical transactions in an offline phase and then in an online phase it employs a direct diagnosis algorithm on the re-ordered constraints of diagnosis tasks (active transactions). In this paper, we demonstrate and evaluate our approach based on the direct diagnosis algorithm FLEXDIAG [7].
Our contributions in this context are the following. We utilize the constraint-ordering sensitivity of direct diagnosis and propose a novel learning approach for constraint ordering.
In order to increase the diagnosis quality and the runtime performance of FLEXDIAG at the same time, we propose matrix factorization based constraint ordering (LCO) for diagnosis of inconsistent constraints (see Algorithm 1). In our approach, we build a sparse matrix (R) by exploiting historical purchase transactions with inconsistent user requirements and preferred minimal diagnoses and a new inconsistent set of user requirements (constraints). The estimated dense matrix (as a result of matrix factorization of the sparse matrix) provides the input in terms of importance estimates for determining a new diagnosis. According to the importance estimates determined by matrix factorization, constraints are reordered in before employing direct diagnosis. In the following, we explain our approach by demonstrating how it works on the example diagnosis task (REQ Lisa ∪ C KB ). In this context, we show the experimental evaluations of our approach on the basis of real-world configuration knowledge bases. In our camera configuration example, Lisa provides her requirements which are inconsistent with the camera product table (no solution found). Therefore, her requirements need to be diagnosed. Using our approach, we first calculate a constraint ordering (an LCO) and then solve the diagnosis task (REQ Lisa ∪ C KB ) using the defined constraint ordering.

Offline phase: learning from historical transactions
Our proposed method needs an offline phase in which various constraint ordering heuristics are learned based on historical transactions. For the offline learning, matrix factorization techniques and historical transactions with inconsistent user requirements are employed (see Table 3).
An example of using a learned heuristics is a solution search starting with the variable "resolution" and the corresponding value "20.9" and continuing with the other values of "resolution" in the learned order. After the variable "resolution" is satisfied, then continue with the next variable with its ordered values until all the variables-values are consistent with the given set of constraints.
In Table 3, for each user, we have an inconsistent set of user requirements (which leads to "no solution"). After a no-solution situation, some of the users (in our case, Alice, Tom, and Joe in Table 3) decided to buy a product (Purchase) which does not completely satisfy their requirements. Consequently, they had to change their initial requirements which is presented as Δ min . These historical transactions which are completed with a purchase are complete historical transactions. The rest of historical transactions, in which users did not complete their transactions with a purchase (e.g. Bob and Ray in Table 3), is called incomplete historical transactions. In this context, we estimate diagnoses of incomplete historical transactions using matrix factorization and we take into account only historical transactions with minimal diagnoses.
Based on the historical transactions in Table 3, we know that, the product Camera 1 is purchased by Alice. This means, Alice has renounced her last two requirements c 10 : weight = 560g and c 11 : price = 469 and purchased the product Camera 1 which has weight = 475g and price = 659. Therefore, Δ Alice = {c 10 , c 11 } is a diagnosis for the requirements diagnosis task (REQ Alice ∪ C KB ). When we eliminate the diagnosis constraints from the inconsistent requirement set, the diagnosed requirement set becomes REQ Alice = {effective Resolution = 20.9 Megapixel, display = 3.5 inches, touch = yes, wifi = yes, nfc = no, gps = yes, videoResolution = UH D − U 4K/3840x2160, zoom = 3.0x}. Based on REQ Alice , the found solution set is {Camera 1 } which includes the products purchased by Alice (Camera 1 ).

The sparse matrix
Matrix factorization based collaborative filtering algorithms [11] introduce a rating matrix R (a.k.a., user-item matrix) which describes preferences of users for the individual items the users have rated. Thereby, R represents an m × n matrix, where m denotes the number of users and n the number of items. The respective element r u,i of the matrix R describes the rating of the item i made by user u. Given the complete set of user ratings, the recommendation task is to predict how the users would rate the items which they have not yet been rated by these users.
In our approach, we build a sparse matrix R (userconstraint matrix) using inconsistent historical transactions as shown in Table 4a where columns represent constraints. Therefore, each row of the sparse matrix R represents a set of user requirements (the left half) and their corresponding diagnoses In our camera configuration example, in the left half of the first row of R, we set the user requirements of Alice. As shown in Table 3, she prefers 3.5 inches displays, i.e. c 3 : display = 3.5 inches. In Table 4, the assigned value of display is normalized using Formula (3) and represented as 1. In the right half of the first row of R, we set the diagnoses probabilities. If a constraint c i exists in Δ user , its corresponding diagnosis probability δ i is 1, otherwise to 0. We know that Δ Alice = {c 10 , c 11 } (see Table 3), consequently we set δ 10 and δ 11 to 1 and the rest to 0.

Matrix factorization
In terms of matrix factorization [12,13], the sparse matrix R is decomposed into an m × k user-feature matrix P and a k × n constraint-feature matrix Q T which both are used to find the estimated dense matrix P Q T . In this context, k is a variable parameter which needs to be adapted to optimize the prediction quality for the test data.
In our example, we apply matrix factorization to the sparse matrix in Table 4a. Then, the estimated matrix is obtained as shown in Table 4b which includes the estimated diagnoses for Bob and Ray.

Online phase: diagnosing active transactions
After calculating the estimated matrix in the offline phase, in the online phase we diagnose active transactions which includes inconsistencies as in our working example. In active transactions, users still did not leave the configuration session and need real-time help to remove inconsistencies in their requirements to decide on a product to purchase. Therefore, the configuration system should provide a relevant diagnosis in a reasonable time (before users leave the system without a purchase).

The most similar historical transaction
We find the most similar historical transaction to the new set of inconsistent requirements using Formula (4) where H T represents a historical transaction, AT represents the active transaction, H T .c i represents the value of each requirement in the estimated dense matrix P Q T , and AT .c i represents the value of each requirement in the active transaction. i represent a constraint index value in the REQ of AT .
In our camera configuration example, the most similar historical transaction to the active transaction of Lisa is the transaction of Ray. Therefore, to diagnose REQ Lisa , we use LCO Ray : {c 2 , c 9 , c 6 , c 7 , c 11 , c 10 , c 3 , c 4 , c 5 , c 8 }. When we only consider user requirements of Lisa: {c 2 , c 3 , c 5 , c 7 , c 9 }, we obtain the constraint ordering for Lisa LCO Lisa : {c 2 , c 9 , c 7 , c 3 , c 5 }.

Direct diagnosis with LCO
After the most similar historical transaction has been found and its constraint ordering has been applied to the active transaction's user requirements, the direct diagnosis algorithm is employed. We employ FLEXDIAG as the direct diagnosis algorithm with its parameter m which is used to control diagnosis quality.
Applying matrix factorization to consistency-based direct diagnosis 7029 Table 4 Matrix factorization estimates a dense matrix P Q T (b) which closely approximates the sparse matrix R In our working example, user constraints of the diagnosis task are reordered using LCO Lisa as {c 2 , c 9 , c 7 , c 3 , c 5 } and a minimal diagnosis Δ = {c 2 , c 9 } is found by FLEXDIAG(m = 1) with performance results (on the basis of evaluation criteria given in Section 2.5): #CC = 4, minimality = 1, and combined = 0.250. However, when we employ the diagnosis algorithm FLEXDIAG(m = 1) on the default order of user constraints {c 2 , c 3 , c 5 , c 7 , c 9 }, the same diagnosis Δ = {c 2 , c 9 } is found with performance results: #CC = 8, minimality = 1, and combined = 0.125. Therefore, using LCO with FLEXDIAG(m = 1), we improve the performance of diagnosis when diagnosing the working example.
In the following, we discuss effects of our approach on a direct diagnosis algorithm FLEXDIAG, based on the evaluation criteria: runtime (based on the number of consistency checks), minimality, and combined. As shown in the search trees of FLEXDIAG with LCO (Tree-2 and Tree-4) have better combined performance compared to the search trees of FLEXDIAG without LCO (Tree-1 and Tree-3). When m = 1, LCO improved the combined performance (see Formula 2) with the ratio 32% (0.166 instead of 0.125). If m = 2, LCO improves the combined performance with the ratio 100% (0.250 instead of 0.125) and the minimality with the ratio 50% whereas the runtime is not improved. Consequently, as also observed in the example search trees, LCO improves runtime performance and minimality of diagnosing.

Settings
We have developed our approach in Java and tested on a computer with an Intel Core i5-5200U, 2.20 GHz processor, 8 GB RAM and 64 bit Windows 7 Operating System and Java Run-time Environment 1.8.0. Constraint satisfaction problems have been solved by Choco3. 1 which is a java library for constraint satisfaction problems with a FlatZinc (the target language of MiniZinc) parser. For matrix factorization, we have used the SVDRecommender of Apache Mahout [17]. 2

Datasets
We have used Minizinc-2016 benchmark problems [20] where each problem includes five data files with file extension ".dzn". 3 1 http://www.choco-solver.org/ 2 Using the latent factor k = 100 and the number of iterations = 1000 3 http://www.minizinc.org/challenge2016/results2016.html In order to obtain historical and active transactions based on these benchmark problems, we randomly generated 5000 sets of inconsistent user requirements (each with N constraints) based on integer variables. We have employed the java-code snippet below which inserts 10 additional constraints to each benchmark problem and always leads an inconsistency: Applying matrix factorization to consistency-based direct diagnosis 7031

Comparative methods
We compare our approach directly with FLEXDIAG [7]. In this article, we do not compare LCO with more traditional diagnosis approaches-for related evaluations we refer the reader to [7] where detailed analyses of FLEXDIAG can be found. We used FLEXDIAG to show the minimality improvements of using LCO when m ≥ 2. As baseline, we evaluated FLEXDIAG-without constraint ordering. #REQ represents the number of constraints in a set of inconsistent requirements.

Results
As discussed throughout this paper, our main objective is to improve the combined performance (runtime performance and diagnosis quality at the same time). We have compared our approach LCO with the baseline no constraint ordering. In both cases, for diagnostic search, FLEXDIAG is used with three different m values 1, 2, and 4.
As shown in Table 5, based on the average value (in the last row), LCO outperforms the baseline in terms of runtime and minimality since with each m value (1, 2, and 4), LCO has lower runtime than the baseline whereas its minimality is higher (or equal) compared to the baseline with each m value (1, 2, and 4).
Based on the results in Table 5, we present two comparisons in Fig. 1 where relations between performance    Applying matrix factorization to consistency-based direct diagnosis 7033  indicators and the number of constraints in the set of user requirements are shown.
In Fig. 1a, we observe that minimality improves (or the parameter m decreases) when runtime increases. As observed, the number of consistency checks (#CC) are at each m (m = 1, 2, and 4) lower when LCO is used. Moreover, at each m (m = 1, 2, and 4), LCO also provides better of equal minimality results.
The combined performance results are shown in Fig. 1b  and c. Deviations in the results of LCO are more visible than the baseline, because LCO has greater performance values. When we zoom into the results of the baseline, we also observe similar deviations due to the variations in the problems. As observed, our approach improves the combined performance significantly.

Conclusions
In this paper, we proposed a novel learning approach for constraint ordering heuristics to solve the quality-runtime performance trade off problem of direct diagnosis. We employed matrix factorization for learning based on historical transactions. Taking the advantage of learning from historical transactions, we calculated possible constraint ordering heuristics in an offline phase for solving new diagnosis tasks in online phase.
In particular, we applied our constraint ordering heuristics to reorder user constraints of diagnosis tasks and employed a direct diagnosis algorithm FLEXDIAG [7] to diagnose the reordered constraints of the diagnosis tasks. The reason to choose this diagnosis algorithm is that the quality-runtime performance trade off problem is much more obvious in FLEXDIAG when its m parameter is increased. However, our approach can be also applicable to other direct diagnosis approaches (e.g. [18]). We compared our approach with a baseline: FLEXDIAG without heuristics. According to our experimental results, our approach (LCO) solves the the quality-runtime performance trade off problem by improving the diagnosis quality (in terms of minimality) and the runtime performance at the same time.
Funding Open access funding provided by Graz University of Technology.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons. org/licenses/by/4.0/.