WMU Journal of Maritime Affairs

, Volume 16, Issue 3, pp 455–472 | Cite as

Improving the validity and reliability of authentic assessment in seafarer education and training: a conceptual and practical framework to enhance resulting assessment outcomes

  • Samrat Ghosh
  • Marcus Bowles
  • Dev Ranmuthugala
  • Ben Brooks
IAMU Section Article

Abstract

Past literature on authentic assessment suggests that it provides a far more reliable and valid indicator of outcomes such as higher student engagement, ability to transfer skills to different contexts, multiple evidence of competence, and student performance. This has appeal in seafarer education and training where both students and employers increasingly perceive traditional assessment methods as failing to consistently generate these outcomes. However, this paper argues that improving different aspects of assessment validity and reliability is essentially required to enhance the outcomes identified above. The paper builds on and extends previous work to investigate and develop a conceptual and practical framework that promotes a holistic approach to authentic assessment that provides greater assurances of validity and reliability throughout all stages of assessment within seafarer programs. It also lays the path to future research directions by establishing the agenda to test the practicality of the framework in the authentic assessment of seafarer students and also investigate the impact of students’ perception of increasing authenticity on performance scores in assessment tasks.

Keywords

Authentic assessment Validity Reliability Seafarer Education and training 

References

  1. Allianz Global Corporate, Specialty (AGCS) (2015) Safety and shipping review 2015. ALLIANZ, GermanyGoogle Scholar
  2. Archbald DA (1991) Authentic assessment: principles, practices, and issues. Sch Psychol Q 6:279–293.  https://doi.org/10.1037/h0088821 CrossRefGoogle Scholar
  3. Ashford-Rowe KH (2009) A heuristic framework for the determination of the critical elements in authentic assessment. PhD Dissertation, University of WollongongGoogle Scholar
  4. Bandura A (1977) Self-efficacy: toward a unifying theory of Behavioural change. Psychol Rev 84(2):191–215.  https://doi.org/10.1037//0033-295x.84.2.191 CrossRefGoogle Scholar
  5. Biggs J (1999) What the student does: teaching for enhanced learning. High Educ Res Dev 18(1):57–75.  https://doi.org/10.1080/0729436990180105 CrossRefGoogle Scholar
  6. Biggs J, Tangs C (2010) Applying constructive alignment to outcomes-based teaching and learning. In training material for “quality teaching for learning in higher education” workshop for master trainers. Ministry of Higher Education, Kuala LumpurGoogle Scholar
  7. Bosco AM, Fern S (2014) Embedding of authentic assessment in work-integrated learning curriculum. Asia-Pacific Journal of Cooperative Education 15(4):281–290 http://www.apjce.org/files/APJCE_15_4_281_290.pdf Google Scholar
  8. Boud D, Walker D (1998) Promoting Reflection in Professional Courses: the challenge of context. Stud High Educ 23(2):191–206.  https://doi.org/10.1080/03075079812331380384 CrossRefGoogle Scholar
  9. Bransford JD, Brown AL, Cocking RR (2000) Learning and transfer. How people learn: brain, mind, experience, and school. National Academy Press, Washington, DCGoogle Scholar
  10. Cassidy KE (2009) Using authentic intellectual assessment to determine level of instructional quality of teacher practice of new elementary school teachers based on teacher preparation route. PhD Dissertation, The George Washington UniversityGoogle Scholar
  11. Cousin G (2006) An introduction to threshold concepts. Planet 17:4–5 http://www.ee.ucl.ac.uk/~mflanaga/Cousin%20Planet%2017.pdf CrossRefGoogle Scholar
  12. Cox QN (2009) MET and industry–gaps to be bridged. In: Loginovsky V (ed) MET trends in the XXI century: shipping industry and training institutions in the global environment – area of mutual interests and cooperation. Admiral Makarov State Maritime Academy, Saint-Petersburg, Russia, pp 171–181Google Scholar
  13. Darling-Hammond L, Snyder J (2000) Authentic assessment of teaching in context. Teach Teach Educ 16(5):523–545.  https://doi.org/10.1016/s0742-051x(00)00015-9 CrossRefGoogle Scholar
  14. Dikli S (2003) Assessment at a distance: traditional vs. alternative assessments. The Turkish Online Journal of Educational Technology 2(3):13–19 http://www.tojet.net/articles/v2i3/232.pdf Google Scholar
  15. Diller KR, Phelps SF (2008) Learning outcomes, portfolios, and rubrics, oh my! Authentic Assessment of an Information Literacy Program portal. Libr Acad 8(1):75–89.  https://doi.org/10.1353/pla.2008.0000 CrossRefGoogle Scholar
  16. Donovan MS, Bransford JD, Pellegrino JW (1999) How people learn: bridging research and practice. National Academy Press, Washington, DCGoogle Scholar
  17. Drost EA (2011) Validity and Reliability in Social Science Research. Education Research and Perspectives 38(1):105–123 http://www.erpjournal.net/wp-content/uploads/2012/07/ ERPV38–1.-Drost-E.-2011.-Validity-and-Reliability-in-Social-Science-Research.pdfGoogle Scholar
  18. Dunn L, Parry S, Morgan C (2002) Seeking quality in criterion referenced assessment. In: Learning Communities and Assessment Cultures Conference. EARLI Special Interest Group on Assessment and Evaluation. University of Northumbria, Newcastle upon TyneGoogle Scholar
  19. Emad G, Roth WM (2007) Evaluating the competencies of seafarers: challenges in current practice. In: Pelton T, Reis G, Moore K (eds) Proceedings of the University of Victoria Faculty of education research conference–Connections’07. University of Victoria, Victoria, BC, Canada, pp 71–76Google Scholar
  20. Ertmer PA, Newby TJ (1993) Behaviorism, Cognitivism, Constructivism: Comparing Critical Features from an Instructional Design Perspective. Perform Improv Q 6(4):50–72.  https://doi.org/10.1002/piq.21143 CrossRefGoogle Scholar
  21. Fatonah S, Suyata P, Prasetyo ZK (2013) Developing an authentic assessment model in elementary school science teaching. Journal of Education and Practice 4(13):50–61 www.iiste.org/Journals/index.php/JEP/article/download/6774/6887 Google Scholar
  22. Findlay LAE (2013) A qualitative investigation into student and teacher perceptions of motivation and engagement in the secondary mathematics classroom. Bachelor Dissertation, Avondale College of Higher EducationGoogle Scholar
  23. Fredricks JA, McColskey W (2012) The Measurement of Student Engagement: A Comaprative Analysis of Various Methods and Student Self-report Instruments. In: Christenson RL (ed) Handbook of Research on Student Engagement. Springer Science+Business Media, New YorkGoogle Scholar
  24. Gao X, Grisham-Brown J (2011) The use of authentic assessment to report accountability data on young Children’s language, literacy and pre-math competency. International Education Studies 4(2):41–53.  https://doi.org/10.5539/ies.v4n2p41 CrossRefGoogle Scholar
  25. Ghosh S, Bowles M, Ranmuthugala D, Brooks B (2014) On a lookout beyond STCW: Seeking standards and context for the authentic assessment of seafarers. In: Ranmuthugala D, Lewarn B (eds) IAMU AGA 15 looking ahead: innovation in maritime education. Training and Research. Australian Maritime College, Launceston, Tasmania, pp 77–86Google Scholar
  26. Ghosh S, Bowles M, Ranmuthugala D, Brooks B (2015) Authentic assessment in seafarer education: using literature review to investigate its’ validity and reliability through rubrics. WMU J Marit Aff 15(2):317–336.  https://doi.org/10.1007/s13437-015-0094-0 CrossRefGoogle Scholar
  27. Hart C, Hammer S, Collins P, Chardon T (2011) The real deal: using authentic assessment to promote student engagement in the first and second years of a regional law program. Leg Educ Rev 21(1):97–121Google Scholar
  28. Hensel D, Stanley L (2014) Group simulation for "authentic" assessment in a maternal-child lecture course. J Scholarsh Teach Learn 14(2):61–70.  10.14434/josotl.v14i2.4081 CrossRefGoogle Scholar
  29. Herrington J, Herrington A (1998) Authentic assessment and multimedia: how university students respond to a model of authentic assessment. Higher Education Research and Development Society of Australasia 17(3):305–322.  https://doi.org/10.1080/0729436980170304 CrossRefGoogle Scholar
  30. Johnson YL (2007) The efficacy of authentic assessment versus pencil and paper testing in evaluating student achievement in a basic technology course. PhD Dissertation, Walden UniversityGoogle Scholar
  31. Jonsson A (2008) Educative assessment for/of teacher competency. A study of assessment and learning in the "interactive examination" for student teachers. PhD dissertation, Malmo universityGoogle Scholar
  32. Jonsson A, Svingby G (2007) The use of scoring rubrics: reliability, validity and educational consequences. Educational Research Review 2(2):130–144.  https://doi.org/10.1016/j.edurev.2007.05.2002 CrossRefGoogle Scholar
  33. Koh K, Luke A (2009) Authentic and conventional assessment in Singapore schools: an empirical study of teacher assignments and student work. Assessment in Education: Principles, Policy and Practice 16(3):291–318.  https://doi.org/10.1080/09695940903319703 CrossRefGoogle Scholar
  34. Lang II TR (2012) An examination of the relationship between elementary education teacher candidates’ authentic assessments and performance on the professional education subtests on the Florida teacher certification exam (FTCE). Graduate Dissertation, University of South FloridaGoogle Scholar
  35. Leberman SI (1999) The transfer of learning from the classroom to the workplace: a New Zealand case study. PhD Dissertation, Victoria UniversityGoogle Scholar
  36. Linn RL, Baker EL, Dunbar SB (1991) Complex, performance-based assessment: expectations and validation criteria. Educ Res 20(8):15–21.  https://doi.org/10.3102/0013189x020008015 CrossRefGoogle Scholar
  37. Lister R (2006) Driving learning via criterion-referenced assessment using Bloom’s taxonomy. In: Assessment in science teaching and learning symposium. The University of Sydney, Sydney, Australia, pp 80–88Google Scholar
  38. Lynch R (2003) Authentic, performance-based assessment in ESL/EFL reading instruction. Asian EFL journal:1–28 http://www.asian-efl-journal.com/dec_03_rl.pdf
  39. McAlister B (2001) The authenticity of authentic assessment: What the research says... or doesn't say. In: Custer RL (ed) Using authentic assessment in vocational education. Center on Education and Training for Employment College of Education, Columbus, pp 19–30Google Scholar
  40. McAlpine M (2002) In: U.O.L (ed) Principles of assessment (Bluepaper no. 1). CAA CNTRE, University of Glasgow, United KingdomGoogle Scholar
  41. McCarthy G (2013) Authentic assessment - key to learning. In: Doyle E, Buckley P, Carroll C (eds) Innovative business school teaching-engaging the millennial generation. Routledge, London, pp 81–92Google Scholar
  42. McLaughlin H (2015) Seafarers in the spotlight. Marit Policy Manag 42(2):95–96.  https://doi.org/10.1080/03088839.2015.1006351 CrossRefGoogle Scholar
  43. McLeod S (2013) What is Reliability? SimplyPsychology http://www.simplypsychology.org/reliability.html. Accessed 24 July 2016
  44. Merriam SB, Leahy B (2005) Learning Transfer: A review of the research in adult education and training. PAACE J Lifelong Learn 14(1):1–24Google Scholar
  45. Messick S (1995) Standards of validity and the validity of standards in performance assessment. Educ Meas: Issues and Prac 14(5):5–8.  https://doi.org/10.1111/j.1745-3992.1995.tb00881.x Google Scholar
  46. Messick S (1996) Validity of performance assessments. In: Phillips GW (ed) Technical issues in large-scale performance assessment. National Centre for Education Statistics, WashingtonGoogle Scholar
  47. Mestre J (2002) Transfer of learning: issues and research agenda. Report of a workshop held at the National Science Foundation. University of Massachusetts-Amherst, VirginiaGoogle Scholar
  48. Mitchell GW (2008) Essential soft skills for success in the twenty-first century workforce as perceived by Alabama business/marketing educators. PhD Dissertation, Auburn UniversityGoogle Scholar
  49. Moon TR, Brighton CM, Callahan CM, Robinson A (2005) Development of authentic assessments for the middle school classroom. The Journal of Secondary Gifted Education 16(2/3):119–133 http://files.eric.ed.gov/fulltext/EJ698321.pdf CrossRefGoogle Scholar
  50. Moore JL (2012) Designing for transfer: a threshold concept. The Journal of Faculty Development 26(3):19–24 http://ezproxy.utas.edu.au/login?url=http://search.proquest.com/docview/1143304893?accountid=14245 Google Scholar
  51. Morrison WGS (1997) Competent crews = safer ships; an aid to understanding STCW 95. WMU Publications, MalmoGoogle Scholar
  52. Morrissey PE (2014) Investigating how an authentic task can promote student engagement when learning about Australian history. PhD Dissertation, University of WollongongGoogle Scholar
  53. Moskal BM, Leydens JA (2000) Scoring rubric development: validity and reliability. Practical Assessment, Research & Evaluation 7(10):71–81 http://pareonline.net/getvn.asp?v=7&n=10 Google Scholar
  54. O’Farrell, C (2005) Enhancing student learning through Assessment. http://www.tcd.ie/teaching-learning/academic-development/ assets/pdf/ 250309_assessment _toolkit.pdf
  55. Oh DM, Kim JM, Garcia RE, Krilowicz BL (2005) Valid and reliable authentic assessment of culminating student performance in the biomedical sciences. Adv Physiol Educ 29(2):83–93.  https://doi.org/10.1152/advan.00039.2004 CrossRefGoogle Scholar
  56. Olfos R, Zulantay H (2007) Reliability and validity of authentic assessment in a web based course. Educational Technology & Society 10(4):156–173 http://www.ifets.info/journals/10_4/15.pdf Google Scholar
  57. Pallis AA, Ng ADK (2011) Pursuing maritime education: an empirical study of students' profiles, motivations and expectations. Marit Policy Manag 38(4):369–393.  https://doi.org/10.1080/03088839.2011.588258 CrossRefGoogle Scholar
  58. Pecota SR, Buckley JJ (2009) Training paradigm assisted accidents: are we setting our students up for failure? In: Loginovsky V (ed) MET trends in the XXI century: shipping industry and training institutions in the global environment – area of mutual interests and cooperation. Admiral Makarov State Maritime Academy, Russia, pp 192–204Google Scholar
  59. Prasad R, Nakazawa T, Baldauf M (2010) Professional development of shipboard engineers and the role of collaborative learning. In: International Association of Maritime Universities AGA11. Korea Maritime University, Busan, Korea, pp 165–174Google Scholar
  60. Quartuch MJ (2011) Is authentic enough? Authentic assessment and civic engagement. Master Dissertation, Moravian CollegeGoogle Scholar
  61. Richards Perry GD (2011) Student perceptions of engagement in schools: a Deweyan analysis of authenticity in high school classrooms. PhD Dissertation, Georgia State UniversityGoogle Scholar
  62. Sadler R (2005) Interpretations of criteria-based assessment and grading in higher education. Assessment & Evaluation in Higher Education 30(2):175–194.  https://doi.org/10.1080/0260293042000264262 CrossRefGoogle Scholar
  63. Sampson H, Gekara V, Bloor M (2011) Water-tight or sinking? A consideration of the standards of the contemporary assessment practices underpinning seafarer licence examinations and their implications for employers. Maritime Policy & Management. The flagship journal of international shipping and port research 38(1):81–92.  https://doi.org/10.1080/03088839.2010.533713 Google Scholar
  64. Sator A (2000) An exploration of transfer of learning opportunities in an online co-operative Prepatory curriculum. Master Dissertation, Simon Fraser UniversityGoogle Scholar
  65. Saunders NG, Saunders GA, Batson T (2001) Assessment and the adult learner: does authentic assessment influence learning? In the annual meeting of the mid-western educational research association. Chicago, ILGoogle Scholar
  66. Schell JW (2000) Think about Authentic Learning and Then Authentic Assessment. In: Custer RL (ed) Using Authentic Assessment in Vocational Education. Centre on Education and Training for Employment, ColumbusGoogle Scholar
  67. Scott J (2000) Authentic Assessment Tools. In: Custer RL (ed) Using Authentic Assessment in Vocational Education. Centre on Education and Training for Employment College of Education, ColumbiaGoogle Scholar
  68. Secolsky C (1987) On the direct measurement of face validity: a comment on Nevo. J Educ Meas 24(1):82–83.  https://doi.org/10.1111/j.1745-3984.1987.tb00265.x CrossRefGoogle Scholar
  69. Spady WG (1994) Outcome-based education: critical issues and answers. American Association of School Administrators, Arlington, VAGoogle Scholar
  70. Stevens P (2013) An examination of a Teacher's use of authentic assessment in an urban middle school setting. PhD Dissertation, Ohio UniversityGoogle Scholar
  71. Taylor JM (2011) Interdisciplinary authentic assessment: cognitive expectations and student performance. PhD Dissertation, Pepperdine UniversityGoogle Scholar
  72. Uchida M (2004) Analysis of human error in marine engine management. In: International Association of Maritime Universities AGA 05. Australian Maritime College, Tasmania, Australia, pp 85–93Google Scholar
  73. Varley MA (2008) Teachers' and administrators' perceptions of authentic assessment at a career and technical education centre. PhD Dissertation, Fordham UniversityGoogle Scholar
  74. Wiggins G (1989) A true test: toward more authentic and equitable assessment. The Phi Delta Kappan 70(9):703–713.  https://doi.org/10.1177/003172171109200721 Google Scholar
  75. Wiggins G (1992) Creating tests worth taking. Educ Leadersh 49(8):26–34 http://www.ascd.org/ASCD/pdf/journals/ed_lead/el_199205_wiggins.pdf Google Scholar
  76. Wiggins GP (1998) Educative assessment: designing assessments to inform and improve student performance. Jossey-Bass, CA,San FranciscoGoogle Scholar

Copyright information

© World Maritime University 2017

Authors and Affiliations

  1. 1.National Centre of Ports and Shipping, Australian Maritime CollegeUniversity of TasmaniaHobartAustralia
  2. 2.Future of Professional Work and LearningDeakin UniversityBurwoodAustralia

Personalised recommendations