• Yu-Ling Lu
  • Ie-Bin Lian
  • Chi-Jui LienEmail author


To design and to manufacture in science learning are increasingly important in science education. Yet, evaluation techniques in school for evaluating students’ creative products are apparently left behind. With the aim of developing an evaluation method to evaluate creative products in science and technology class, this study constructed a set of criteria with data collected from teachers and students. The analytic hierarchy process (AHP), a multiple criteria decision-making tool for single rater, was selected for the purpose of weighting and evaluating students’ products. However, the traditional AHP used one rater’s pair-wise comparisons; its subjectivity and complexity limit its applications in school. For solving this problem, this study developed an advanced technique, called direct-rating AHP (DR-AHP), to extend the applicability of the traditional AHP. The DR-AHP is used to obtain weights or preferences for criteria/alternatives by a process of directly ranking criteria/alternatives by single/multi rater(s), checking consistency, and developing a rank vector matrix. The DR-AHP was implemented in obtaining criteria weights of a hierarchy framework for creative products evaluation by a group of science educators (N = 13) and field-tested in ranking creative products by another group of science teachers (N = 9). Results showed its superiority in objectivity and efficiency over traditional ways of evaluation. The results also demonstrate how the AHP and DR-AHP are capable of helping evaluators systematically construct criteria and/or to evaluate students’ creative products for classroom instruction as well as during many other activities.


alternative AHP creative products evaluation method products evaluation scientific creativity 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Besemer, S. P., & O’Quin, K. (1999). Confirming the three-factor creative product analysis matrix model in an american sample. Creativity Research Journal, 12(4), 287.Google Scholar
  2. Bodin, L. & Gass, S. (2004). Exercises for teaching the analytic hierarchy process. Informs Transactions on Education, 4(2).Google Scholar
  3. Besemer, S. P., & Treffinger, D. J. (1981). Analysis of creative products: Review and synthesis. Journal of Creative Behavior, 15(3), 158–178.Google Scholar
  4. Bottero, M., Comino, E. & Riggio, V. (2011). Application of the analytic hierarchy process and the analytic network process for the assessment of different wastewater treatment systems. Environmental Modelling and Software, 26(10), 1211–1224. doi: 10.1016/j.envsoft.2011.04.002.CrossRefGoogle Scholar
  5. Cao, D., Leung, L. C. & Law, J. S. (2008). Modifying inconsistent comparison matrix in analytic hierarchy process: A heuristic approach. Decision Support Systems, 44(4), 944–953.CrossRefGoogle Scholar
  6. Dolan, J. G. (2008). Shared decision-making—transferring research into practice: The analytic hierarchy process (AHP). Patient Education & Counseling, 73(3), 418–425.CrossRefGoogle Scholar
  7. Donaway, R. L., Drummey, K. W., & Mather, L. A. (2000). A comparison of rankings produced by summarization evaluation measures. Paper presented at the proceedings of the 2000 NAACL-ANLP workshop on automatic summarization-Volume 4.Google Scholar
  8. Figueira, J., Greco, S. & Ehrgott, M. (Eds.). (2005). The analytic hierarchy and analytic network processes for the measurement of intangible criteria and for decision-making. New York: Springer.Google Scholar
  9. Finan, J. S. & Hurley, W. J. (1997). The analytic hierarchy process: Does adjusting a pairwise comparison matrix to improve the consistency ratio help? Computers and Operations Research, 24(8), 749–755.CrossRefGoogle Scholar
  10. Gargallo, P., Moreno-Jiménez, J. & Salvador, M. (2007). AHP-group decision making: A Bayesian approach based on mixtures for group pattern identification. Group Decision & Negotiation, 16(6), 485–506. doi: 10.1007/s10726-006-9068-0.CrossRefGoogle Scholar
  11. Godinho, P., Costa, J. P., Fialho, J. & Afonso, R. (2011). Some issues about the application of the analytic hierarchy process to R&D project selection. Global Business and Economics Review, 13(1), 26–41. doi: 10.1504/gber.2011.039188.CrossRefGoogle Scholar
  12. Green, D. R. (1981). How probability pays. Mathematics in School, 10(2), 23–24.Google Scholar
  13. Ho, W. (2008). Integrated analytic hierarchy process and its applications—a literature review. European Journal of Operational Research, 186(1), 211–228. doi: 10.1016/j.ejor.2007.01.004.CrossRefGoogle Scholar
  14. Horn, D. & Salvendy, G. (2006). Product creativity: Conceptual model, measurement and characteristics. Theoretical Issues in Ergonomics Science, 7(4), 395–412. doi: 10.1080/14639220500078195.CrossRefGoogle Scholar
  15. Horng, J.-S. & Lin, L. (2009). The development of a scale for evaluating creative culinary products. Creativity Research Journal, 21(1), 54–63. doi: 10.1080/10400410802633491.CrossRefGoogle Scholar
  16. Hsu, S. L. (2004). The effect of context-based creative-thinking instruction on students' learning. Unpublished Master Thesis. National Taipei University of Education, Taipei.Google Scholar
  17. Islam, R. & Abdullah, N. (2006). Management decision making by the analytic hierarchy process: A proposed modification for large scale problems. Journal of International Business and Entrepreneurship Development, 3(1/2), 18–40.CrossRefGoogle Scholar
  18. Joint Committee on Standards for Educational Evaluation (1994). The program evaluation standards: How to assess evaluations of educational program. Thousand Oaks: Sage.Google Scholar
  19. Layton, D. (1988). Revaluing the T in STS. International Journal of Science Education, 10(4), 367–378.CrossRefGoogle Scholar
  20. Lee, D., McCool, J. & Napieralski, L. (2000). Assessing adult learning preferences using the analytic hierarchy process. International Journal of Lifelong Education, 19(6), 548–560. doi: 10.1080/02601370050209078.CrossRefGoogle Scholar
  21. Liberatore, M. J. & Nydick, R. L. (2008). The analytic hierarchy process in medical and health care decision making: A literature review. European Journal of Operational Research, 189(1), 194–207. doi: 10.1016/j.ejor.2007.05.001.CrossRefGoogle Scholar
  22. Lin, H. Y. (2004). To explore the effect of situated STS teaching on elementary students’ scientific creativity. Unpublished Master Thesis, National Taipei University of Education, Taipei.Google Scholar
  23. Ministry of Education (2001). The 1-9 grades science and life technology curriculum standards. Taipei: Ministry of Education.Google Scholar
  24. Moreno-Jiménez, J. M., Aguarón, J. & Escobar, M. T. (2008). The core of consistency in AHP-group decision making. Group Decision and Negotiation, 17(3), 249–265. doi: 10.1007/s10726-007-9072-z.CrossRefGoogle Scholar
  25. National Research Council, N. (2012). Conceptual shifts in the next generation science standards. Accessed 18 April 2013.
  26. National Taiwan Science Education Center. (2012). Products list of elementary life and applied sciences. Accessed 10 July 2013.
  27. Patton, M. Q. (1987). How to use qualitative methods in evaluation. Newbury Park: Sage.Google Scholar
  28. Saaty, T. L. (1980). The analytic hierarchy process. New York: McGraw-Hill.Google Scholar
  29. Saaty, T. L. (1994). Fundamentals of decision making and priority theory with the analytic hierarchy process. Pittsburgh, PA: RWS.Google Scholar
  30. Saaty, T. L. (1997). That is not the analytic hierarchy process: What the AHP is and what it is not. Journal of Multi Criteria Decision Analysis, 6(6), 320–339.CrossRefGoogle Scholar
  31. Spires, E. E. (1991). Using the analytic hierarchy process to analyze multiattribute decisions. Multivariate Behavioral Research, 26(2), 345–361.CrossRefGoogle Scholar
  32. Stake, R., Migotsky, C., Davis, R., Cisneros, E. J., Depaul, G., Dunbar, C. & Chaves, I. (1997). The evolving syntheses of program value. American Journal of Evaluation, 18(1), 89–103.CrossRefGoogle Scholar
  33. Tang, H. W. V. (2011). Optimizing an immersion ESL curriculum using analytic hierarchy process. Evaluation and Program Planning, 34(4), 343–352. doi: 10.1016/j.evalprogplan.2011.04.002.CrossRefGoogle Scholar
  34. Triantaphyllou, E. (1999). Reduction of pairwise comparisons in decision making via a duality approach. Journal of Multi Criteria Decision Analysis, 8(6), 299–310.CrossRefGoogle Scholar
  35. Trochim, W. M. K. (1998). An evaluation of Michael Scriven’s “Minimalist theory: The least theory that practice requires”. American Journal of Evaluation, 19, 243.Google Scholar
  36. Vidal, L. A., Marle, F. & Bocquet, J. C. (2011). Using a Delphi process and the analytic hierarchy process (AHP) to evaluate the complexity of projects. Expert Systems with Applications, 38(5), 5388–5405. doi: 10.1016/j.eswa.2010.10.016.CrossRefGoogle Scholar
  37. Yager, R. E. & Lutz, M. V. (1995). STS to enhance total curriculum. School Science & Mathematics, 95(1), 28.CrossRefGoogle Scholar
  38. Yu, J. R., Hsiao, Y. W. & Sheu, H. J. (2011). A multiplicative approach to derive weights in the interval analytic hierarchy process. International Journal of Fuzzy Systems, 13(3), 225–231.Google Scholar

Copyright information

© National Science Council, Taiwan 2013

Authors and Affiliations

  1. 1.Department of Science EducationNational Taipei University of EducationTaipei CityRepublic of China
  2. 2.Department of MathematicsNational Changhua University of EducationChanghuaRepublic of China

Personalised recommendations