Skip to main content
SpringerLink
Log in

Web-based support for constructing competence maps: design and formative evaluation

  • Development Article
  • Published:
Educational Technology Research and Development Aims and scope Submit manuscript

Abstract

This article describes the design and formative evaluation of a Web-based tool that supports curriculum developers in constructing competence maps. Competence maps describe final attainment levels of educational programs in terms of—interrelated—competencies. Key requirements for the competence-mapping tool were validity and practicality. Validity refers to internal consistency and meaningful links to the external realities represented. Practicality refers to a design approach of evolutionary prototyping, in which feedback from intended users and domain experts is collected throughout the development process. Formative evaluations of four prototypes were conducted. Measures of design, appeal, goal, content, confidence and relevance showed that the tool is practical. The article describes the formative evaluation process and concludes with a description of the modified tool from the perspective of the user and the instructional designer.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Anderson, J. R. (1985). Cognitive psychology and its implications. San Francisco, CA: Freeman.

    Google Scholar 

  • Boon, J., & van der Klink, M. (2001). Beroepsprofielen in het hoger onderwijs: praktisch artikel [Professional and educational profiles in higher education: A practical article]. Tijdschrift voor Onderwijsinnovatie, 1, 17–25.

    Google Scholar 

  • Brinck, T., Gergle, D., & Wood, S. D. (2002). Designing Web sites that work: Usability fro the Web. San Fransisco: Morgan Kaufmann.

    Google Scholar 

  • Carroll, J. M. (Eds.) (1998). Minimalism beyond the Nurnberg funnel. Cambridge, MA: MIT Press.

  • Cluitmans, J. J. (2002). Aan de slag met competenties: competentiegericht leren in HBO en MBO [Get to work with competencies: competence-based learning in higher and middle professional education]. Nuenen, The Netherlands: Onderwijsadviesbureau Dekkers.

    Google Scholar 

  • De Bie, D. (2003). Morgen doen we het beter: Handboek voor de competente onderwijsontwikkelaar [Tomorrow we will do better: Manual for the competent educational designer]. Houten, The Netherlands: Bohn Stafleu Van Loghum.

    Google Scholar 

  • Fletcher, S. (1997). Analysing competence: Tools and techniques for analyzing jobs, roles and functions. London: Kogan Page.

    Google Scholar 

  • Gery, G. (1991). Electronic performance support systems: How and why to remake the workplace through the strategic application of technology. Boston, MA: Weingarten.

    Google Scholar 

  • Gick, M. L., & Holyoak, K. J. (1980). Analogical problem solving. Cognitive Psychology, 12, 306–355.

    Article  Google Scholar 

  • Gustafson, K. (2002). Instructional design tools: A critique and projections for the future. Educational Technology Research and Development, 50, 59–66.

    Article  Google Scholar 

  • Kessels, J. (1999). A relational approach to curriculum design. In J. van den Akker, R. Branch, K. Gustafson, N. Nieveen, & T. J. Plomp (Eds.), Design approaches and tools in education and training (pp. 59–71). Dordrecht, The Netherlands: Kluwer Academic Publishers.

    Google Scholar 

  • McKenney, S., Nieveen, N., & van den Akker, J. (2002). Computer support for curriculum developers: CASCADE. Educational Technology, Research and Development, 50, 25–35.

    Article  Google Scholar 

  • Miles, B. M., & Huberman, A. M. (1984). Qualitative data analysis. Beverly Hills, CA: Sage publications.

    Google Scholar 

  • Mulcahy, D. (2000). Turning the contradictions of competence: Competency-based training and beyond. Journal of Vocational Education and Training, 52, 259–279.

    Article  Google Scholar 

  • Mulder, M., Wesselink, R., Biemans, H., Nieuwenhuis, L., & Poell, R. (Eds.) (2003). Competentiegericht beroepsonderwijs: Gediplomeerd, maar ook bekwaam? [Competence-based professional education: Qualified, but capable as well?]. Houten: Wolters-Noordhoff.

  • Nielsen, J. (1994). Heuristic evaluation. In J. Nielsen, R. L. Mack (Eds.) Usability inspection methods. New York: John Wiley & Sons.

    Google Scholar 

  • Nielsen, J. (2003a). Ten usability heuristics. Retrieved March 10, 2003, from http://www.useit.com/papers/heuristic/heuristic_list.html.

  • Nielsen, J. (2003b). How to conduct a heuristic evaluation. Retrieved March 10, 2003, from http://www.useit.com/papers/heuristic/heuristic_evaluation.html.

  • Nieveen, N. (1999). Prototyping to reach product quality. In J.v.d. Akker, R. M. Branch, K. Gustafson, N. Nieveen, & T. Plomp (Eds.), Design approaches and tools in education and training (pp. 125–135). Dordrecht: Kluwer academic publishers.

    Google Scholar 

  • Nieveen, N., & Gustafson, K. (1999). Characteristics of computer-based tools for education and training development: An introduction. In J.v.d. Akker, R. M. Branch, K. Gustafson, N. Nieveen, & T. Plomp (Eds.), Design approaches and tools in education and training (pp. 155–174). Dordrecht: Kluwer academic publishers.

    Google Scholar 

  • Norman, D. A. (1993). Things that make us smart: Defending human attributes in the age of the machine. Reading, MA: Addison-Wesley.

    Google Scholar 

  • Paas, F., & van Merriënboer, J. J. G. (1994). Variability of worked examples and transfer of geometrical problem solving skills: A cognitive load approach. Journal of Educational Psychology, 86, 122–133.

    Article  Google Scholar 

  • Perkins, D. N. (1992). Technology meets constructivism: Do they make a marriage? In T. M. Duffy, & D. H. Jonassen (Eds.), Constructivism and the technology of instruction: A conversation (pp. 45–55). Hillsdale, NJ: Lawrence Erlbaum.

    Google Scholar 

  • Schlusmans, K., Slotman, R., Nagtegaal, C., & Kinkhorst, G. (1999). Competentiegerichte leeromgevingen [Competence-based learning environments]. Utrecht, The Netherlands: Lemma.

    Google Scholar 

  • Schneiderman, B. (1998). Designing the user interface: Strategies for effective human–computer interaction. Reading, MA: Addison-Wesley.

    Google Scholar 

  • Shadbolt, N., & Burton, M. (1995). Knowledge elicitation: A systematic approach. In J. R. Wilson, & E. Nigel Corlett (Eds.), Evaluation of human work: a practical ergonomics methodology (pp. 406–440). New York: Taylor & Francis.

    Google Scholar 

  • Smith, S. S. (2001). Web-based instruction: A guide for libraries. Chicago: American Library Association.

    Google Scholar 

  • Stevens, G., & Stevens, E. (1995). Designing electronic performance support tools: Improving workplace performance with hypertext, hypermedia and multimedia. Englewood Cliffs, NJ: Educational Technology Publications.

    Google Scholar 

  • Stoof, A., Martens, R. L., van Merriënboer, J. J. G., & Bastiaens, T. J. (2002). The boundary approach of competence: A constructivist aid for understanding and using the concept of competence. Human Resource Development Review, 1, 345–365.

    Article  Google Scholar 

  • Stoyanov, S. (2001). Mapping in the educational and training design. Unpublished doctoral dissertation, University of Twente, The Netherlands.

  • Strauss, A. L., & Corbin, J. (1990). Basics of qualitative research: Grounded theory, procedures and tactics. London: Sage Publications.

    Google Scholar 

  • Sweller, J., van Merriënboer, J. J. G., & Paas, F. (1998). Cognitive architecture and instructional design. Educational Psychology Review, 10, 251–296.

    Article  Google Scholar 

  • Van den Akker, J. (1999). Principles and methods of developmental research. In J. van den Akker, R. M. Branch, K. Gustafson, N. Nieveen, & T. Plomp (Eds.), Design approaches and tools in education and training (pp. 1–14). Dordrecht, The Netherlands: Kluwer Academic Publishers.

    Google Scholar 

  • Van den Akker, J. (2003). Curriculum perspectives: An introduction. In J. van den Akker, W. Kuiper, & U. Hameyer (Eds.), Curriculum landscapes and trends (pp. 1–10). Dordrecht, The Netherlands: Kluwer Academic Publishers. .

    Google Scholar 

  • Van der Klink, M. R., & Boon, J. (2003). Competencies: The triumph of a fuzzy concept. International Journal of Human Resources Development and Management, 3, 125–137.

    Article  Google Scholar 

  • Van der Meij, H. (2003). Minimalism revisited. Document Design, 4, 212–233. .

    Google Scholar 

  • Van der Meij, H., & Carroll, J. M. (1998). Principles and heuristics for designing minimalist instruction. In J. M. Carroll (Ed.), Minimalism beyond the Nurnberg funnel (pp. 19–53). Cambridge, MA: MIT Press.

    Google Scholar 

  • Van Merriënboer, J. J. G., & Martens, R. (2002). Computer-based tools for instructional design. Educational Technology, Research and Development, 50, 5–9.

    Article  Google Scholar 

  • Van Merriënboer, J. J. G., van der Klink, M. R., & Hendriks, M. (2002). Competenties: van complicaties tot compromis [Competencies: From complications to compromise]. Den Haag, The Netherlands: Onderwijsraad.

    Google Scholar 

  • Ward, M., & Sweller, J. (1990). Structuring effective worked examples. Cognition and instruction, 7, 1–39.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Educational Technology Expertise Center, Open University of the Netherlands, P.O. Box 2960, 6401 DL, Heerlen, The Netherlands

    Angela Stoof & Jeroen J. G. van Merriënboer

  2. Leiden University, Leiden, The Netherlands

    Rob L. Martens

Authors
  1. Angela Stoof
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rob L. Martens
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jeroen J. G. van Merriënboer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jeroen J. G. van Merriënboer.

Appendix: Heuristics in the formative evaluations

Appendix: Heuristics in the formative evaluations

Heuristic

Description

Heuristic evaluation form I

1. Navigation

The structure of the Website should be clear. Users have to know where they are and how to come somewhere else. Users have to be able to find the information they are looking for easily.

2. Interface

The interface should be simple and consistent. The structure of the interface and the color use should match with the content and “tone” of the Website. The structure has to support the use of the Website.

3. Usability

Users quickly should know how to use the Website. The Website should be easy to use. Pages should be downloaded quickly. Users should not have to perform to many actions to reach intended information.

4. Correction and prevention of errors

Errors should be prevented. If a users makes a mistake he or she should receive feedback and to be able to undo errors quickly.

5. Locus of control

Users should have the feeling that they decide what is happening rather than the Website.

6. Short term memory

Pages should not be too long or contain too much or redundant information. The Website cannot request that the user remembers information when navigating to another page.

7. Text

The text should be pleasant to read with respect to paragraphs, use of white lines, color, contrast and typeface. The language should be in accordance with the user and it should be clear and direct. Texts should be not too long and should have a beginning, middle section and end.

8. Media

Media (illustration, clips, etc.) should fit the content of the Website and have a clear function.

9. Attractiveness

The Website should look nice.

10. Motivating aspects

The Website should motivate users to use it.

11. Purpose

The purpose of the Website should be clear.

12. Target group

It should be clear who are the intended users of the Website.

Heuristic evaluation form II

1. Clarity method

The method should be clear. The user should quickly know how to use the method. It should be clear what happens in the steps and phases. The tools of the construction kit should be clear.

2. Usability method

The method should be usable. The user should know exactly what to do every time. It should be clear when the user has to go to a next step or phase, and at what time tools should be used.

3. Reliability method

The method should lead to a reliable competence map. When a user makes a competence map, a second one constructed 3 months later should be similar.

4. Validity method

The method should lead to a valid competence map. The competence map should be a good reflection of the competencies practitioners need in a certain domain of profession.

5. Support example

The example should make clear how to use the method. The example should be clear.

6. Confidence

The Website should raise confidence with the user. The user should have the impression that the Website is constructed carefully. De Website should look professional.

7. Relevance

The method should be relevant. The method should fit to the task of constructing a competence map in practice. People who build a competence map should gain from the method.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Stoof, A., Martens, R.L. & van Merriënboer, J.J.G. Web-based support for constructing competence maps: design and formative evaluation. Education Tech Research Dev 55, 347–368 (2007). https://doi.org/10.1007/s11423-006-9014-5

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11423-006-9014-5

Keywords

Search

Navigation