1 Introduction

Multiple-criteria decision-making (MCDM) methods are crucial for evaluating alternatives based on various criteria and selecting the best option (Chakraborty et al., 2015). The main purpose of all MCDM methods is to evaluate and rank available alternatives. However, different mathematical algorithms used in each method may lead to varying results. The criteria for selecting the appropriate MCDM method for the current situation are unclear (Chakraborty et al., 2015). However, comparative studies of MCDM methods demonstrate that they are consistent within the context of the selected subject and sample (e.g. Uzun and Kazan, 2016) or that one method produces more accurate results than others (e.g. Junior et al., 2014).

AHP (Analytic Hierarchy Process), one of the MCDM decision-making methods, is based on pairwise comparisons. AHP allows for the calculation of the weight of importance for each sub-criterion by conducting pairwise comparisons between the sub-criteria of the established criteria (Saaty, 1977). Conjoint analysis is based on the principle of selecting the most suitable sample from all possible combinations of sub-criteria and ranking the combinations within this sample. AHP and Conjoint analysis are commonly used in various fields, including consumer behavior, product design, purchasing, investment, finance, education, economic policies, engineering, industry, production, and sports (Vaidya and Kumar, 2006). When analysing educational research, it can be concluded that AHP and Conjoint analysis do not have an effective role as in other fields. This study aims to demonstrate an application of these methods, which are not yet widely used in educational research. The study aims to determine whether these two methods can produce effective results in educational research.

The research question for this study applies AHP and Conjoint analysis methods to investigate the relationship between the qualities that mathematics teachers seek in a good mathematical problem and the qualities of the problems they prefer to use in classroom practices. This issue has mostly been explored in the literature using qualitative research methods. The literature review is expected to provide adequate references to evaluate the results of AHP and Conjoint analysis.

1.1 AHP and conjoint analysis

Problem-solving can be defined as the process of determining the difference between the current situation and the desired situation and eliminating the identified deficiency (Anderson et al., 2014). Specific steps should be followed linearly or cyclically for an effective problem-solving process. However, these steps vary according to the field of the problem (such as education, mathematics, economics, or psychology), the context of the problem (such as individual, social, or economic), and the function of the problem (such as teaching, decision-making), all problem-solving processes start with the identification of the problem and end with an appropriate solution or solution suggestions. Polya (2004) defines the problem-solving process in mathematics education in four steps:

  1. 1.

    Understanding the problem.

  2. 2.

    Create a plan.

  3. 3.

    Implement the plan.

  4. 4.

    Evaluating the outcome.

After Polya, problem-solving steps in mathematics education were revised to include linear or cyclical steps within this framework. Anderson et al. (2014), who work in various fields of quantitative analysis methods and statistics, list the problem-solving steps as follows:

  1. 1.

    Defining the problem.

  2. 2.

    Identify alternative solutions.

  3. 3.

    Identify criteria to be used to evaluate alternatives.

  4. 4.

    Evaluating alternatives.

  5. 5.

    Choosing the appropriate alternative.

  6. 6.

    Implementation of the selected alternative.

  7. 7.

    Evaluate results to determine whether a satisfactory solution has been reached.

As can be seen, apart from terminological differences, the problem-solving process involves similar steps. Decision-making is selecting an alternative from a set of available options. From this point of view, the first five steps of the above seven-step problem-solving process can be considered within the scope of decision-making. Therefore, decision-making has a more limited scope than problem-solving (Anderson et al., 2014). It is clear that there should be at least two alternatives for decision-making. However, it may also be necessary to decide by choosing between many options. This situation brings to mind the question of the weight of the criteria affecting the decision. To decide by analyzing the effect of criteria, multi-criteria decision-making problems and various solution techniques for these problems have emerged. One of the methods frequently used in multi-criteria decision-making problems is the AHP developed by Saaty (1977). The basic principle of AHP is to make pairwise comparisons between criteria. This principle requires more thought and, therefore, consistency in criteria selection. However, it should not be forgotten that the consistency of the person making the choice may not be appropriate to the goal to be achieved or realistic within the framework of the purpose of the research (Özdemir, 2002). The last two steps of the problem-solving process include the implementation of the alternatives and evaluating the results at the end of the decision-making process.

AHP is used in many fields, such as consumer behavior, product design, purchasing, investment, finance, education and economic policies, engineering, industry, production, and sports (Subramanian and Ramanathan, 2012; Vaidya and Kumar, 2006). When educational research is examined, there are few studies in which AHP is used (e.g., Korucuk, 2020; Sezen Yüksel and Çıldır, 2009; Yüksel, 2013). Sezen Yüksel and Çıldır (2009) examined the decision-making processes of prospective physics teachers, Yüksel (2013) examined the teaching methods that can be used in effective chemistry education, and Korucuk (2020) examined the satisfaction levels of classroom teachers towards distance education with the AHP. The AHP can be used in many areas of educational research that require the determination of preferences and possible alternatives, such as determining the factors affecting teaching practices, especially in attitude and belief studies, and can produce effective results. Since the AHP is the subject of fields of study such as fuzzy logic, fuzzy sets, and decision-making in mathematics, and since it is an effective method in determining consumer behavior, its use in the fields of business and economics may cause it to be perceived as a method specific to these fields. Since educational researchers are unlikely to encounter or be aware of the method unless they specifically investigate it, it is unsurprising that its use in academic research is low. This study is planned to increase the AHP’s visibility and create an opinion that it is a functional method in educational research.

The processes in the AHP can be listed as follows (Leal, 2020):

  1. 1.

    Explanation of the problem and collection of information,

  2. 2.

    Identification of the criteria and presentation of the hierarchical structure of the decision problem,

  3. 3.

    Pairwise comparison of the criteria and creation of a pairwise comparison matrix,

  4. 4.

    Obtaining priority values (local weights) from the pairwise comparison matrix,

  5. 5.

    Calculation of the consistency ratio of the identified criteria.

To effectively apply the AHP, one of the most critical issues to be considered is the definition of the problem and then the determination of the criteria and sub-criteria. Decision-making with AHP can be done by pairwise comparisons or by ranking the criteria when the number of sub-criteria related to the criteria is high (> 9). A large number of criteria may prevent consistent pairwise comparisons. For this reason, it is recommended that they should be distinctive, have as little in common with each other as possible, and be few. The hierarchical structure of an AHP model can be shown as follows (Saaty, 1987):

figure a

Once the hierarchical structure of the final objective, criteria, and sub-criteria have been determined, the decision-makers are asked to give the pairwise comparison values of the criteria in line with the evaluation scale (Saaty, 1977). The values and superiority levels in the evaluation scale are as follows (Saaty, 1977):

Assigned value

Definition

1

Equally important

3

Weak importance

5

Strong importance

7

Demonstrated importance

9

Absolute importance

2, 4, 6, 8

Intermediate values

Comparison matrices are created from the data obtained due to pairwise comparisons (individual data or geometric mean of all participant data). The importance weight of the sub-criteria of each criterion is determined by applying the necessary analysis steps. In this way, priorities are determined, and alternative scores can be calculated based on the personal evaluations of decision-makers (Ayçin, 2020). Determining priorities and alternatives is important in revealing the preferences of the whole group as well as the individual preferences of the decision-makers. According to the research question, the existing situation can be evaluated by determining the group’s priorities and choices among the given options, or decisions can be made regarding the group’s preference tendency for ranking, grouping, or choosing the best.

Although conjoint, defined as the analysis of relationships, was conceptually introduced by Luce and Tukey (1964), it was first expressed by Johnson in 1974. Conjoint analysis is generally used as a multi-criteria analysis technique to determine consumer behavior characteristics (Louviere, 1994; Şen and Çemrek, 2004; Turanli et al., 2013). The main feature of conjoint analysis, which is used in many fields of study, is that, as in AHP, decision-makers rank the existing situations according to their preferences in terms of importance. As a result, the most or least preferred situation is determined (Louviere, 1994). Considering this basic feature, it is clear that decision-makers in preference-based research will not only be consumers under appropriate conditions. Various studies using conjoint analysis show that this method is suitable for obtaining effective results in different fields (e.g., Dağhan and Seferoğlu, 2012; Susada, 2018; Soutar and Turner, 2002; Şen et al., 2009). Some of these studies have been conducted in the field of education. For example, Dağhan and Seferoğlu (2012) determined graduate students’ preferences for the basic dimensions of distance education. They concluded that the type of technology used is the most important and the least essential dimension is technical support as a result of conjoint analysis. Susada (2018) investigated the preferences of undergraduate students for mathematics courses and found that the most critical factors for students among teaching style, the language used during teaching (English or local language), course time, and teaching methods was course time, followed by teaching style, teaching methods, and teaching language. The researcher also identified the most preferred criteria under each factor through conjoint analysis.

AHP and conjoint analysis are MCDM methods frequently used in various fields. While these methods can be used for similar purposes, conceptual and measurement differences exist. While the AHP represents a systematic procedure for ranking multiple alternatives, conjoint analysis evaluates alternatives presented to individuals. In the AHP, all alternatives and criteria are presented to decision-makers. In contrast, in conjoint analysis, the alternatives presented to decision-makers are consistent combinations of criteria determined by researchers. In the AHP, all criteria are compared in pairwise comparisons, while in conjoint analysis, combinations representing these criteria are compared (Popovic et al., 2018). Although both methods were initially developed for different purposes, using them in the same research can serve similar purposes (Popovic et al., 2018). When these methods are analyzed in terms of consistency and sensitivity, it has been concluded that applying AHP first and then conjoint analysis in the same research gives better results (Mulye, 1998; Helm et al., 2008).

The advantage of MCDM methods over Likert-type scales, frequently used in educational research, is that they give the ratio of importance between two preferences. A person may rate attributes A and B as “very important” on a Likert-type scale, but “Are attributes A and B equally important?” cannot be answered. With the method of pairwise comparisons, it can be determined which of the characteristics A and B is more important for the person (Galbraith and Haines, 2001).

1.2 Characteristics of problems in mathematics teaching

Problems are indispensable to textbooks, classroom teaching, and assessment activities (NCTM, 2000). Although traditional mathematics teaching is dominated by the concept of the subject first and then the problem, the approach of using problem-solving as a teaching method is thought to be more appropriate to the nature of the problem (Van De Walle et al., 2012). Considering that mathematics is learned through real contexts, problems, and situations, starting the lesson with a problem situation is one of the most effective ways of using mathematical problems in teaching (Schroeder and Lester, 1989; as cited in Van De Walle et al., 2012). Mathematical problems should not only be problems that can be solved with specific strategies and belong to the world of mathematics but should also be of a nature that develops students’ abilities to deal with real-life problems and complex systems beyond school (English, 2008; English and Sriraman, 2010; Gainsburg, 2006). In most of the traditional problems used in school mathematics, students are so far removed from thinking that their only goal is to reach the final result by doing operations with the numbers given in the problem. This shows that students do not try to think about the realistic aspects of the solution (Bonotto, 2007; Buhrman, 2017; Palm, 2008). However, in problem-solving, it is necessary to focus on mathematical associations in the real world (Bonotto, 2007). It is not possible to achieve success with the experiences gained in the world of numbers and rules abstracted from real life.

On the other hand, teaching mathematics through situations with no real-life counterparts creates the perception that students will fail if they solve problems with their real-life knowledge (Boaler, 2008; Gravemeijer, 2011). This situation leads to the belief that students cannot solve real-life problems with school mathematics and mathematical problems at school with their real-life knowledge. Some of the misconceptions students have about the nature of mathematics are that there is one and only one correct answer to mathematical problems, that there is only one right way to solve any mathematical problem - usually the last rule that the teacher shows the class, that mathematics is a solitary activity, that the mathematics learned at school has little or nothing to do with the real world (Schoenfeld, 1992, p.359). One of the reasons why students have such false beliefs about mathematics is the teacher’s expectations.

Some studies conducted with teachers and prospective teachers (e.g., Haser, 2006; Kayan and Çakıroğlu, 2008; Haser et al., 2013) show that beliefs about problem-solving emerge in two ways: before and after the undergraduate program. The fact that the examination system in Turkey is based on multiple-choice questions with one correct answer causes students to exhibit an imitative rather than productive behavior in problem-solving. At the same time, it was concluded that their beliefs changed when they received education in accordance with the innovative approach (at the undergraduate level), but they could not completely break away from their previous ideas (Haser, 2006; Haser et al., 2013). Similarly, Lee and Kim (2005) found that although pre-service elementary school teachers needed a new perspective on the nature of the problem, they thought that it would be difficult to use it in their classes due to the deficiencies arising from the education system or difficulties arising from teachers’ lack of self-confidence. The limitations imposed by the examination system, the curriculum, and the textbooks also affect the problem preferences of teachers (Foong and Koay, 1997; Güven et al., 2016; Özmen et al., 2012). In the studies conducted, it was concluded that teachers prefer verbal problems in their lessons because they are more common in textbooks (Özmen et al., 2012); these problems are generally not related to daily life, routine, and do not contain missing data (Özmen et al., 2012). In addition, teachers tend to prefer problems that are easy to solve, explain, and understand because they think they are more persuasive in teaching (Leikin, 2003).

In addition to the extrinsic factors that affect teachers’ preferences for the problems to be used in their lessons, teachers’ views on how they define a problem and what characteristics a good problem should have are also critical since they are intrinsic factors that affect problem preferences. Chapman (2005) asked pre-service teachers (26 people) how they define a problem and asked them to present sample problems. Most pre-service teachers (87%) presented simple verbal problems (e.g., James’s age is twice Laura’s. Since the sum of their ages is 24, how old are James and Laura? When asked why these problems were “good,” they gave reasons such as all the necessary information was provided, it was not difficult, the solution steps were clear, and the unknown could be reached. For 17% of the pre-service teachers, the problems were interesting and challenging, providing opportunities to seek and explore the unknown. These participants evaluated the problems they prepared as interesting, challenging, and thought-provoking, requiring effort, creativity, and disciplined work to reach the answer.

Lee and Kim (2005) found that when pre-service elementary school teachers evaluated non-routine problems, most did not consider problems with no mathematical purpose, had more than one answer, were complex, ambiguous, too easy, or too difficult as good problems. On the other hand, other participants evaluated the same problems as encouraging students to think, supporting their creativity, allowing different solutions related to real life, and being effective in teaching concepts. They stated that these were good problems. This result shows that the educational perspectives of pre-service teachers (as well as teachers) are directly related to how they define the problem. One of the most effective ways to free teachers from the traditional perspective and to change the value that students attribute to mathematics is to change mathematics from being a world of numbers, symbols, and formulas independent from the real world and to make them accept that mathematics is intertwined with the real world is to change the understanding of problem-solving in mathematics education by the post-positivist teaching paradigm that overlaps with the constructivist learning approach. As Hiebert et al. (1997) point out, to enable students to learn mathematics by understanding it, instead of asking them to imitate what is given in the textbooks or the procedure shown by the teacher, they should be presented with problems that they will create their solutions and asked to share the procedure they develop. Such problems make students think about mathematics and help them develop mathematical thinking skills, such as explanation, interpretation, and justification when sharing with their peers.

An educational system that provides students with opportunities for reflection and communication should be based on real problems (Hiebert et al., 1996, 1997). Hiebert et al. (1996) list at least three criteria that good problems should fulfill. First, the problem situation should be a problem for the students. This does not mean it is difficult or frustrating for students to understand. An interesting problem for students means that there is something to explore and make sense of. Second, problems should be appropriate to students’ existing knowledge and skills. Students should be able to start developing methods for solving problems using the knowledge and skills they already have. Third, the problems should lead students to think about mathematics. They should provide students with opportunities to reflect on important mathematical ideas and to extract and understand mathematical values from their experiences. Students should see problems as meaningful activities that are relatively easy (Davydov, 1996). Considering the problem perspective of many researchers who have studied problem solving (e.g., Polya, Schoenfeld, Charles, and Lester), a mathematical activity or problem should be (a) motivating, (b) unsolvable by known procedures, (c) engaging, and (d) allow for multiple solution approaches. This understanding of problem-solving has brought about significant changes, and the practice of finding mathematical solutions to real-world problems has gained increasing momentum (Hamilton, 2007).

When considering the qualities of a good problem, it is important to note that problems which are motivating, applicable to real-life situations, open to multiple solutions, thought-provoking, and support conceptual learning are valuable tools for developing students’ problem-solving skills. Additionally, these types of problems provide an opportunity to learn mathematics in the context of real-life situations (Lingefjärd and Holmquist, 2005). However, these problems are often excluded from curricula and textbooks, and the emphasis on exam results can cause teachers to avoid teaching them, despite their effectiveness in mathematics education (Sahin, 2019). As explained earlier, the AHP is primarily used for decision-making, while conjoint analysis is used to determine preferences (Mulye, 1998; Helm et al., 2004; Popovic et al., 2018). This study aims to determine the characteristics that teachers seek in a good mathematics problem and the features of problems they prefer in classroom practice, using AHP and Conjoint analysis. According to this objective, the research question is as follows:

  1. (1)

    How can AHP and Conjoint analysis be used to evaluate the relationship between the characteristics teachers seek in a good problem and the problems they prefer to use in classroom practices?

It is aimed to contribute to the literature in two areas through the research question. Firstly, to introduce AHP and Conjoint analysis, that are not yet widely used in educational research. Secondly, to examine the consistency between the characteristics sought in a good mathematics problem and the problems preferred in classroom practice. Although the research question relates to the methodology of the study, it is expected to provide an example of how these methods, which have not yet been widely used in the field of mathematics education but are increasingly popular in other fields, can be applied in educational research.

2 Methodology

This quantitative descriptive study examines the relationship between the characteristics of a good mathematics problem and the characteristics of the problems preferred by middle school mathematics teachers in classroom practices.

2.1 Research group

This study was conducted in the 2023–2024 academic year with the participation of 35 secondary school mathematics teachers in various cities of Turkey. The professional experience of the teachers ranged from 3 to 15 years. The only criterion for determining the participants was to have worked at the secondary school level for at least three years and participated in the study voluntarily. In decision-making studies, there is no standard opinion on the number of participants, as the main thing is that the participants make consistent choices in the relevant field. For example, in conjoint analysis, it is generally stated that there should be at least 150 participants, and this number can go up to 1200, but the research can also be conducted with 30 to 60 participants (Orme, 2010); reliable results are also obtained in studies with a sample size < 75 (Behdioğlu and Çilesiz, 2017; McCullough, 2002). As in any research, a large sample is needed to generalize the study’s results to a specific population. However, a small sample is sufficient to obtain statistically significant results in studies on the decision-making process (Akyurt, 2021; Shepherd and Zacharakis, 1999).

When the literature is examined, there is no specific sample size standard to produce effective results with the AHP (Melillo and Pecchia, 2016). Especially in quantitative research, a small sample size can often negatively affect data analysis, results interpretation, and generalizability. However, one of the most important advantages of AHP is that it only sometimes requires a large sample size to obtain statistically significant analyses (Doloi, 2008). Instead, the involvement of appropriately qualified experts is important to obtain accurate results (Saaty, 2008). Since AHP is usually based on expert perception, even a single expert evaluation can have the power to represent a group (Doloi, 2008; Schot and Fischer, 1993). In this study, a large sample was not needed since the participants who would determine the characteristics of a good math problem were teachers. On the other hand, considering the fact that an increase in the sample size would make it difficult to check the consistency of the arbitrarily answered data and might have a significant negative effect on the overall consistency of the study, it was concluded that 35 participants would be sufficient for the sample of this study.

2.2 Data collection

The data of the study were collected in two stages. First, the relevant literature was searched, and the characteristics of mathematical problems were classified under four dimensions: contextual, functional, physical, and analytical (Table 1). Then, a scale was prepared to include the features under these four criteria and to make pairwise comparisons (Appendix 1). There are three sub-criteria in the contextual criterion, four in the functional and physical criteria, and five in the analytical criterion. Since it was a type of scale the teachers had not encountered before, each participant was explained how to score the form through a sample scale. Then, the scale was given with the value definitions of the preferences, and the participants were asked to score the criteria.

Table 1 Characteristics of mathematical problems
Full size table

After the first stage was completed, alternative profiles were created to identify the characteristics of the problems that teachers preferred in classroom practice. The total number of alternatives includes all possible combinations of the criteria, and there are 240 (3 × 4 × 4 × 5) alternatives. In conjoint analysis, full concept, trade, or choice-based techniques are used to determine the minimum number of profiles that best represent these alternatives (Behdioğlu and Çilesiz, 2017). In this study, the orthogonal design method was used to determine the profile cards for conjoint analysis. All criteria and sub-criteria were defined in the SPSS data analysis program and 25 profiles representing all possible combinations were identified (see samples of Profile Cards in Appendix 2). In order to reasonably reduce the difficulty of the application for the participants and to obtain more reliable results, the participant was asked to rate all sub-criteria between 1 and 10 (1: I pay very little attention; 10: I definitely pay attention) while determining the preference levels of the profiles. The ranking of the profiles was determined by the CORPAS (Complex Proportional Assessment) method, taking into account the weight values determined in the AHP. COPRAS is one of the multi-criteria decision-making methods and was introduced to the literature in 1994 (Zavadskas et al., 1994). This method, which considers both qualitative and quantitative criteria, assumes that the degree of importance and utility of the situations under investigation depends directly and proportionally on a system of criteria that adequately describes the alternatives and the values and weights of the criteria. The degree of importance, priority order, and utility of alternatives are determined by proportionally determining how much better one alternative is than another (Kaklauskas et al., 2006). Compared to other methods, the ease of application and calculation process is considered the most important advantage (Ayçin, 2020). These features are why the COPRAS method was preferred in this study.

2.3 Data analysis

In the analysis of the data, the analysis steps of the AHP and conjoint analysis were followed. First, the data obtained from pairwise comparisons were transferred to the spreadsheet editing program (Excel) and analyzed with the AHP. According to the AHP, the data analysis starts with the normalization of the pairwise comparison matrix. For this purpose, priority values are obtained from the pairwise comparison matrix (Saaty, 1980).

$$\text{CI}=(\lambda_\text{maks}-n)/(n-1)$$
(1)

The fit index (CI) is calculated using the function (1) above (𝛌maks is the largest eigenvalue, n is the total number of criteria). Then, the consistency ratio (CR-Consistency Ratio) is calculated.

$$\mathrm{CR}=\mathrm{CI}/\mathrm{RI}$$
(2)

In the above formula (2), RI (Random Index) stands for Random Value Index and represents the average consistency index of the randomly generated matrices of pairwise comparisons. The RI values take the following values determined by Saaty (1980), taking into account the number of criteria compared (n):

Random Index (RI)

n

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

RI

0.00

0.00

0.58

0.90

1.12

1.24

1.32

1.41

1.45

1.49

1.51

1.48

1.56

1.57

1.58

Considering the characteristics that a good mathematical problem should have, since there are three criteria for contextual characteristics, i.e., n = 3, the RI value for this criterion should be taken as = 0.58. In the AHP, the reference value is accepted as 0.10. Therefore, if the consistency ratio (CR) < 0.10, the pairwise comparison matrix is acceptable, and the weight values are valid. In this case, the criteria can be ranked according to their weight values. A consistency ratio more significant than the reference value indicates that the decision maker does not make consistent choices.

The individual results of the teachers were analyzed according to the AHP, and the participants were asked to re-evaluate the inconsistent dimensions. Then, a holistic evaluation was made based on the geometric mean of the results obtained from all participants. Examining the consistency by analyzing the pairwise comparisons of each participant individually made an essential contribution to the study to ensure its reliability.

After the AHP analysis was completed, the weight values of the criteria were considered, and the problem profiles preferred by the participants in the classroom practices were ranked from the most preferred to the least preferred with the COPRAS method. The SPSS program was used for conjoint analysis. The following steps were followed for this analysis, which is not included in the ready-made tests in SPSS:

  1. 1.

    A .sav file with the profiles was created.

  2. 2.

    The data obtained from the participants (profile rankings) were entered into the SPSS program.

  3. 3.

    Then, in the SPSS program, File > New > Syntax steps were used to access the field where the following command will be written:

    • CONJOINT PLAN=”/Users/sedasahin/Desktop/Profilles.sav”

    • /DATA=”/Users/sedasahin/Desktop/Preferences.sav”

    • /SEQUENCE = PREF1 TO PREF25

    • /SUBJECT = ID

    • /FACTORS = Functional (DISCRETE)

    • Contextual (DISCRETE)

    • Physical (DISCRETE)

    • Analytical (DISCRETE)

    • /PRINT = SUMMARYONLY

  4. 4.

    The results were obtained with the Run command after writing the command

The findings obtained from AHP and conjoint analysis are discussed in detail in the next section. The study’s data collection and analysis process is summarized in Fig. 1.

Fig. 1
figure 1

Research implementation process

Full size image

The diagram summarizing the process from the preparation of the data collection tools to the analysis results starts with the determination of the criteria and sub-criteria. The preparation of the pairwise comparisons scale for AHP and the creation of full profile cards and data collection for Conjoint analysis are independent. However, the data analysis process was carried out sequentially with AHP first. The weight values of the criteria obtained from the AHP results were used together with COPRAS to determine the profile card rankings of the participants. Finally, Conjoint analysis was used to determine the characteristics of the problems teachers prefer in classroom practices.

3 Results

The results of the two-stage data collection process were analyzed using Excel and SPSS. Characteristics of problems were analyzed with the AHP, and teachers’ problem preferences in classroom practices were analyzed with conjoint analysis.

3.1 AHP analysis: Characteristics of a “good” problem

The weights for criteria for the pairwise comparison matrices were categorized as contextual, functional, physical, and solution-oriented characteristics.

Table 2 Pairwise comparison matrix and criteria weights of “Good” problem
Full size table

The consistency ratio (CR) of teachers’ prioritization of characteristics for a problem is 0.002 < 0.10, indicating that the matrix is consistent. Therefore, priority vectors for decision options can be calculated for each criterion. Additionally, Table 2 shows that functional characteristics are the most important for a problem to be considered ‘good’ according to teachers, followed by contextual characteristics. In problem-solving, the solution-oriented characteristics are ranked third, while the physical characteristics of the problem are the last sought-after features. The table below shows the ranking of all features according to the teachers.

Table 3 Pairwise comparison matrix and sub-criteria weights
Full size table

Based on the AHP analysis, it was found that teachers consistently evaluated all criteria with a CR < 0.10 in pairwise comparisons. The most important characteristics of a good problem for teachers were determined to be related to daily life, suitable for the student level, remarkable, and having different solution methods. Table 3 shows that the student level is considered the most important functional feature in a good problem, while the short text is the least important physical feature. Maintaining a balanced approach to all sub-criteria is important when evaluating a problem. The results show that the relationship between the weights of the sub-criteria is variable according to the criteria. For example, the sub-criterion of being interdisciplinary among the contextual features was significantly less preferred (0.15) than the other two sub-criteria. When solution-oriented features are analysed, it is seen that the ability to solve the problem with different solution methods ranks first with a high weight value (0.41), while the other four sub-criteria have very close weights.

3.2 Conjoint analysis: Characteristics of mathematics problems preferred in classroom practices

The importance levels of the criteria for conjoint analysis are listed as contextual, functional, physical and solution-oriented features (Table 4).

Table 4 Conjoint analysis - Cramer’s V Statistics
Full size table

Cramer’s V statistics indicate that correlation coefficients range from 0 to 1. As the table above shows, all values of correlations are close to 0. These results indicate that all attributes used in the conjoint measure show low levels of association and, therefore, measure different aspects.

Table 5 shows the correlation coefficients; Pearson’s Rho and Kendall’s Tau are measures of goodness of fit. As it is known, higher values indicate a better fit. Pearson’s Rho (0.806) and Kendall’s Tau (0.591) are both > 0.5, meaning that the conjoint model has high predictive accuracy and internal validity (Kuckartz et al., 2013, p.213).

Table 5 Conjoint analysis - correlations
Full size table

Table 6 shows the mean preference structure of problems used in classroom practice. Analyzing the preference structure or the relative importance accorded (by teachers) to the four salient attributes, the teachers accorded the maximum utility/importance to the attribute “solution-oriented” (with importance as 28.62%). Considering the part-worth functions, the teachers have primarily defined “value” in terms of the problems that cannot be solved by known procedures. Having one correct answer and more than one correct answer as attributes rank high as per teacher preferences.

Table 6 Mean preference structure of problems used in classroom practice
Full size table

The second most important attribute for the problem is physical (importance 26.99%). Within the purview of this attribute, the teachers accorded the highest priority to problems containing numerical data. The fact that the problems are striking and have short text are physical characteristics teachers pay less attention to.

The third most important attribute of the problem is being functional (with importance as 25.17%). Examining the part-worth functions, teachers have predominantly conceptualized ‘value’ about student’s cognitive level.

Contextual characteristics of the problem are the features that teachers pay the slightest attention to in classroom practices (importance 19.22%). Within the purview of this attribute, the teachers accorded the lowest priority to interdisciplinary problems and the highest priority to have real-life context.

3.3 Combining AHP and conjoint analysis: Characteristics of a “good” problem and problems preferred in classroom practices

The results discussed in detail in the previous sections show a difference between the characteristics teachers look for in a good mathematics problem and the problem characteristics they prefer in classroom practices. In this section, the results of AHP and Conjoint Analysis will be summarized. Table 7. shows the weights and importance values of the problem characteristics and the criteria affecting teachers’ problem preferences according to teachers.

Table 7 Characteristics of a “good” problem and problems preferred in classroom practices - Attributes
Full size table

According to Table 7, while the most important features sought in a good problem were functional features, these features ranked third in problem preferences in classroom practices. While the analytical and physical features of the problems ranked the last two among the factors considered in a good problem, the same features were the determining features of the problems preferred in practice. One of the results was that contextual features were the least influential factor in problem preferences, but they ranked second regarding the features sought in a good problem. Table 8. shows how the overall evaluation results of the criteria changed in terms of sub-criteria.

Table 8 Characteristics of a “good” problem and problems preferred in classroom practices - Sub-criteria
Full size table

When ranking the features sought in a good problem and the features of the problems preferred in classroom practices are examined according to the sub-criteria, differences are seen less than in the main criteria. For example, while functional features are the first features sought in a good problem, the importance value ranks third in preferred problems (Table 7). However, the ranking of the functional features is the same in both cases, and the student level (student suitability) is the first sub-criteria to be considered. When analytical features are analyzed, the fact that the problem cannot be solved with the known procedure is the last feature sought in a good problem, while it is the most crucial feature in classroom practices.

The most significant advantage of AHP and Conjoint analysis is that it reveals the weight and importance levels that will enable the relationship between criteria and sub-criteria. While determining the characteristics of the problems, it was possible to determine which feature was more important to the participants than the others with AHP. For example, the difference in the importance weight between the first two ranked physical features of the problem, that the problem is remarkable (0.43) and that it contains numerical data (0.23), and the difference in the importance weight between the first two ranked functional features, that the problem is appropriate for the student level (0.35) and that it supports permanent learning (0.34), are not the same. Although consecutive, they show how much more or less important one of these features is than the other. This indicates that AHP results provide more descriptive information to researchers beyond a simple ranking. The same is true for Conjoint analysis results in determining preferred problems.

The results of AHP and Conjoint analysis show differences in the weights and importance ranking of the sub-criteria as well as the ratios between the sub-criteria. For example, the difference between the weights of the sub-criteria of relevance to daily life and relevance to real life from the contextual features according to the AHP is 0.01, while the difference between the importance levels according to conjoint analysis is 0.564 in the opposite direction. These results show that the difference between these two sub-criteria is more pronounced in terms of the features preferred in practice; they have almost similar importance regarding the features sought in a good problem. However, the sum of the criteria weights in AHP gives 1, while the sum of the importance values in Conjoint analysis gives 0. Due to the difference in algorithms between AHP and Conjoint analysis, a direct comparison of the differences between the numerical values of the criteria only sometimes guarantees a sound interpretation. However, this should not be seen as a disadvantage. For the study, it would be sufficient to compare by considering the rankings so this algorithmic difference between weight and importance values does not make a problem.

As a result of AHP and Conjoint analysis, when the weight/utility values between the sub-criteria are examined proportionally, it is noted that the importance levels between the sub-criteria differ. However, some sub-criteria were found to have similar importance in both analyses. For example, having more than one correct answer (0.16) and having one correct answer (0.15) have almost the same weight according to AHP results. Similarly, in Conjoint analysis, these two sub-criteria are of close importance to each other, with utility values of 0.045 and 0.057, respectively. These results provide important details in terms of how teachers evaluate the features they seek in an ideal problem and the features they pay attention to in classroom practices based on sub-criteria.

4 Conclusions and future scope

This study aims to demonstrate the application of the MCDM methods AHP and Conjoint analysis in educational research. The study investigates the qualities that mathematics teachers seek in a good mathematical problem and the qualities of the problems they prefer to use in classroom practice. In literature, the characteristics of a good mathematics problem and teachers’ problem preferences are mostly analysed using qualitative methods.The results of these studies and the results of this study are parallel. For example, one of the common findings obtained in qualitative studies is that although teachers define a “good” problem with an innovative perspective, they exhibit a more traditionalist attitude toward problem preferences. Similar results were obtained in this study. The AHP method indicated that the functional features of the ideal mathematical problem were more important than other features. This reflects a constructivist perspective. Conjoint analysis revealed that the features of the problems preferred in classroom practices were ranked. It was observed that, in contrast to the ideal problem, analytical features were considered. This result indicates that teachers could not abandon the traditional approach in practice. The data collected for both AHP and Conjoint analysis revealed that teachers engaged in pairwise comparisons by scoring the given features, which is a more nuanced approach than a simple ranking. This focus on two factors at a time and their interaction allows decision makers to provide relative (as opposed to absolute) preference information, as demonstrated by Kostić-Ljubisavljević and Samčović (2024). Consequently, the outcomes obtained through these methods align with the findings of previous research in the field of educational research, indicating that they are a valuable tool for generating meaningful insights.

Considering the results obtained from this study, AHP and Conjoint analyses can be used in future studies not only in the field of mathematics education but also in all educational studies, especially in studies that require the determination of preferences. In the social sciences, affective characteristics such as attitudes, beliefs, perceptions and motivation are predominantly determined by Likert-type scales. These scales typically comprise options indicating the level of agreement between 4 and 6. These options are ranked in a gradual order from ‘best to worst’ or ‘highest to lowest’. Participants may select the same options for more than one feature or statement. In such cases, it is assumed that their level of agreement with the relevant characteristics is the same. To illustrate, in a scale employed to assess students’ attitudes towards mathematics, when the ‘Agree’ option is selected for two statements, such as ‘Mathematics develops creativity’ and ‘Mathematics makes life easier’, it is assumed that the participant attaches equal importance to both statements. The advantage of MCDM methods over Likert-type scales, frequently used in educational research, is that they give the ratio of importance between two preferences.

The results of this study reveal that the characteristics that mathematics teachers seek in an ideal problem and the characteristics of the problems they prefer to use in classroom practice vary. However, the reasons underlying these preferences are not known. This situation is valid for studies conducted with MCDM methods. In future studies, MCDM methods can be supported by qualitative methods and mixed design studies can be conducted. In this way, not only the preferences of the participants but also the reasons for their preferences can be determined.

Considering the results obtained from this study, AHP and Conjoint analyses can be used in future studies not only in the field of mathematics education but also in all educational studies, especially in studies that require the determination of preferences. The advantage of MCDM methods over Likert-type scales, frequently used in educational research, is that they give the ratio of importance between two preferences. In addition, this study is expected to serve as an example of how different multi-criteria comparison methods can be used effectively in educational research, depending on the preferences of the researchers and the content of their studies.

The study followed a method different from that used in the literature to determine the number of participants. Since expert opinions are generally used in the AHP, the consistency levels of pairwise comparisons are high. Therefore, the number of participants (n) in AHP studies can be determined as 19 < n < 400. In Conjoint analysis, although there is yet to be a consensus on the number of participants, it is recommended to work with at least 30 participants. In the AHP, the high number of participants carries the risk of increasing the inconsistency rate. However, in this study, the pairwise comparison results of each participant were analyzed individually, and the data showing significant inconsistencies were excluded from the analysis. Thus, this risk was eliminated. In the AHP results of the geometric averages obtained from an average of 24 participants during the data analysis, it was determined that although the weights of the criteria and sub-criteria changed, there was no significant change in the rankings. This result supports the idea that the sample size for AHP can be small. However, it should be noted that this result is not generalizable. Especially in future studies where different decision-making methods are used together, more effective results can be obtained by repeating the analyses with a randomly selected number of participants. In further rsearch, different numbers of participants can be used in studies where AHP and Conjoint analysis will be used together. For example, Conjoint analysis can be used with a larger sample group by determining the characteristics of a “good” mathematics problem with the pairwise comparison results of a smaller number of teachers or field experts. The most crucial point to consider here is that the small sample for AHP should consist of experts.

This study has shown that AHP and conjoint analysis, the MCDM methods, can be used in educational research. While it is possible to check the consistency of the results in the AHP, this is not the case in the conjoint analysis method. In this respect, it can be said that AHP results are more reliable.

This study’s data collection and analysis methods can be described as a new approach to educational research. For this reason, this study can be a good example of using AHP and Conjoint analysis in education and contributing to mathematics education. These methods can be used alone or in support of each other.