Skip to main content

Advertisement

SpringerLink
Log in

Improving the validity and reliability of authentic assessment in seafarer education and training: a conceptual and practical framework to enhance resulting assessment outcomes

  • IAMU Section Article
  • Published:
WMU Journal of Maritime Affairs Aims and scope Submit manuscript

Abstract

Past literature on authentic assessment suggests that it provides a far more reliable and valid indicator of outcomes such as higher student engagement, ability to transfer skills to different contexts, multiple evidence of competence, and student performance. This has appeal in seafarer education and training where both students and employers increasingly perceive traditional assessment methods as failing to consistently generate these outcomes. However, this paper argues that improving different aspects of assessment validity and reliability is essentially required to enhance the outcomes identified above. The paper builds on and extends previous work to investigate and develop a conceptual and practical framework that promotes a holistic approach to authentic assessment that provides greater assurances of validity and reliability throughout all stages of assessment within seafarer programs. It also lays the path to future research directions by establishing the agenda to test the practicality of the framework in the authentic assessment of seafarer students and also investigate the impact of students’ perception of increasing authenticity on performance scores in assessment tasks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  • Allianz Global Corporate, Specialty (AGCS) (2015) Safety and shipping review 2015. ALLIANZ, Germany

    Google Scholar 

  • Archbald DA (1991) Authentic assessment: principles, practices, and issues. Sch Psychol Q 6:279–293. https://doi.org/10.1037/h0088821

    Article  Google Scholar 

  • Ashford-Rowe KH (2009) A heuristic framework for the determination of the critical elements in authentic assessment. PhD Dissertation, University of Wollongong

  • Bandura A (1977) Self-efficacy: toward a unifying theory of Behavioural change. Psychol Rev 84(2):191–215. https://doi.org/10.1037//0033-295x.84.2.191

    Article  Google Scholar 

  • Biggs J (1999) What the student does: teaching for enhanced learning. High Educ Res Dev 18(1):57–75. https://doi.org/10.1080/0729436990180105

    Article  Google Scholar 

  • Biggs J, Tangs C (2010) Applying constructive alignment to outcomes-based teaching and learning. In training material for “quality teaching for learning in higher education” workshop for master trainers. Ministry of Higher Education, Kuala Lumpur

    Google Scholar 

  • Bosco AM, Fern S (2014) Embedding of authentic assessment in work-integrated learning curriculum. Asia-Pacific Journal of Cooperative Education 15(4):281–290 http://www.apjce.org/files/APJCE_15_4_281_290.pdf

    Google Scholar 

  • Boud D, Walker D (1998) Promoting Reflection in Professional Courses: the challenge of context. Stud High Educ 23(2):191–206. https://doi.org/10.1080/03075079812331380384

    Article  Google Scholar 

  • Bransford JD, Brown AL, Cocking RR (2000) Learning and transfer. How people learn: brain, mind, experience, and school. National Academy Press, Washington, DC

  • Cassidy KE (2009) Using authentic intellectual assessment to determine level of instructional quality of teacher practice of new elementary school teachers based on teacher preparation route. PhD Dissertation, The George Washington University

  • Cousin G (2006) An introduction to threshold concepts. Planet 17:4–5 http://www.ee.ucl.ac.uk/~mflanaga/Cousin%20Planet%2017.pdf

    Article  Google Scholar 

  • Cox QN (2009) MET and industry–gaps to be bridged. In: Loginovsky V (ed) MET trends in the XXI century: shipping industry and training institutions in the global environment – area of mutual interests and cooperation. Admiral Makarov State Maritime Academy, Saint-Petersburg, Russia, pp 171–181

    Google Scholar 

  • Darling-Hammond L, Snyder J (2000) Authentic assessment of teaching in context. Teach Teach Educ 16(5):523–545. https://doi.org/10.1016/s0742-051x(00)00015-9

    Article  Google Scholar 

  • Dikli S (2003) Assessment at a distance: traditional vs. alternative assessments. The Turkish Online Journal of Educational Technology 2(3):13–19 http://www.tojet.net/articles/v2i3/232.pdf

    Google Scholar 

  • Diller KR, Phelps SF (2008) Learning outcomes, portfolios, and rubrics, oh my! Authentic Assessment of an Information Literacy Program portal. Libr Acad 8(1):75–89. https://doi.org/10.1353/pla.2008.0000

    Article  Google Scholar 

  • Donovan MS, Bransford JD, Pellegrino JW (1999) How people learn: bridging research and practice. National Academy Press, Washington, DC

    Google Scholar 

  • Drost EA (2011) Validity and Reliability in Social Science Research. Education Research and Perspectives 38(1):105–123 http://www.erpjournal.net/wp-content/uploads/2012/07/ ERPV38–1.-Drost-E.-2011.-Validity-and-Reliability-in-Social-Science-Research.pdf

    Google Scholar 

  • Dunn L, Parry S, Morgan C (2002) Seeking quality in criterion referenced assessment. In: Learning Communities and Assessment Cultures Conference. EARLI Special Interest Group on Assessment and Evaluation. University of Northumbria, Newcastle upon Tyne

    Google Scholar 

  • Emad G, Roth WM (2007) Evaluating the competencies of seafarers: challenges in current practice. In: Pelton T, Reis G, Moore K (eds) Proceedings of the University of Victoria Faculty of education research conference–Connections’07. University of Victoria, Victoria, BC, Canada, pp 71–76

    Google Scholar 

  • Ertmer PA, Newby TJ (1993) Behaviorism, Cognitivism, Constructivism: Comparing Critical Features from an Instructional Design Perspective. Perform Improv Q 6(4):50–72. https://doi.org/10.1002/piq.21143

    Article  Google Scholar 

  • Fatonah S, Suyata P, Prasetyo ZK (2013) Developing an authentic assessment model in elementary school science teaching. Journal of Education and Practice 4(13):50–61 www.iiste.org/Journals/index.php/JEP/article/download/6774/6887

    Google Scholar 

  • Findlay LAE (2013) A qualitative investigation into student and teacher perceptions of motivation and engagement in the secondary mathematics classroom. Bachelor Dissertation, Avondale College of Higher Education

  • Fredricks JA, McColskey W (2012) The Measurement of Student Engagement: A Comaprative Analysis of Various Methods and Student Self-report Instruments. In: Christenson RL (ed) Handbook of Research on Student Engagement. Springer Science+Business Media, New York

    Google Scholar 

  • Gao X, Grisham-Brown J (2011) The use of authentic assessment to report accountability data on young Children’s language, literacy and pre-math competency. International Education Studies 4(2):41–53. https://doi.org/10.5539/ies.v4n2p41

    Article  Google Scholar 

  • Ghosh S, Bowles M, Ranmuthugala D, Brooks B (2014) On a lookout beyond STCW: Seeking standards and context for the authentic assessment of seafarers. In: Ranmuthugala D, Lewarn B (eds) IAMU AGA 15 looking ahead: innovation in maritime education. Training and Research. Australian Maritime College, Launceston, Tasmania, pp 77–86

    Google Scholar 

  • Ghosh S, Bowles M, Ranmuthugala D, Brooks B (2015) Authentic assessment in seafarer education: using literature review to investigate its’ validity and reliability through rubrics. WMU J Marit Aff 15(2):317–336. https://doi.org/10.1007/s13437-015-0094-0

    Article  Google Scholar 

  • Hart C, Hammer S, Collins P, Chardon T (2011) The real deal: using authentic assessment to promote student engagement in the first and second years of a regional law program. Leg Educ Rev 21(1):97–121

    Google Scholar 

  • Hensel D, Stanley L (2014) Group simulation for "authentic" assessment in a maternal-child lecture course. J Scholarsh Teach Learn 14(2):61–70. 10.14434/josotl.v14i2.4081

    Article  Google Scholar 

  • Herrington J, Herrington A (1998) Authentic assessment and multimedia: how university students respond to a model of authentic assessment. Higher Education Research and Development Society of Australasia 17(3):305–322. https://doi.org/10.1080/0729436980170304

    Article  Google Scholar 

  • Johnson YL (2007) The efficacy of authentic assessment versus pencil and paper testing in evaluating student achievement in a basic technology course. PhD Dissertation, Walden University

  • Jonsson A (2008) Educative assessment for/of teacher competency. A study of assessment and learning in the "interactive examination" for student teachers. PhD dissertation, Malmo university

  • Jonsson A, Svingby G (2007) The use of scoring rubrics: reliability, validity and educational consequences. Educational Research Review 2(2):130–144. https://doi.org/10.1016/j.edurev.2007.05.2002

    Article  Google Scholar 

  • Koh K, Luke A (2009) Authentic and conventional assessment in Singapore schools: an empirical study of teacher assignments and student work. Assessment in Education: Principles, Policy and Practice 16(3):291–318. https://doi.org/10.1080/09695940903319703

    Article  Google Scholar 

  • Lang II TR (2012) An examination of the relationship between elementary education teacher candidates’ authentic assessments and performance on the professional education subtests on the Florida teacher certification exam (FTCE). Graduate Dissertation, University of South Florida

  • Leberman SI (1999) The transfer of learning from the classroom to the workplace: a New Zealand case study. PhD Dissertation, Victoria University

  • Linn RL, Baker EL, Dunbar SB (1991) Complex, performance-based assessment: expectations and validation criteria. Educ Res 20(8):15–21. https://doi.org/10.3102/0013189x020008015

    Article  Google Scholar 

  • Lister R (2006) Driving learning via criterion-referenced assessment using Bloom’s taxonomy. In: Assessment in science teaching and learning symposium. The University of Sydney, Sydney, Australia, pp 80–88

    Google Scholar 

  • Lynch R (2003) Authentic, performance-based assessment in ESL/EFL reading instruction. Asian EFL journal:1–28 http://www.asian-efl-journal.com/dec_03_rl.pdf

  • McAlister B (2001) The authenticity of authentic assessment: What the research says... or doesn't say. In: Custer RL (ed) Using authentic assessment in vocational education. Center on Education and Training for Employment College of Education, Columbus, pp 19–30

    Google Scholar 

  • McAlpine M (2002) In: U.O.L (ed) Principles of assessment (Bluepaper no. 1). CAA CNTRE, University of Glasgow, United Kingdom

    Google Scholar 

  • McCarthy G (2013) Authentic assessment - key to learning. In: Doyle E, Buckley P, Carroll C (eds) Innovative business school teaching-engaging the millennial generation. Routledge, London, pp 81–92

    Google Scholar 

  • McLaughlin H (2015) Seafarers in the spotlight. Marit Policy Manag 42(2):95–96. https://doi.org/10.1080/03088839.2015.1006351

    Article  Google Scholar 

  • McLeod S (2013) What is Reliability? SimplyPsychology http://www.simplypsychology.org/reliability.html. Accessed 24 July 2016

  • Merriam SB, Leahy B (2005) Learning Transfer: A review of the research in adult education and training. PAACE J Lifelong Learn 14(1):1–24

    Google Scholar 

  • Messick S (1995) Standards of validity and the validity of standards in performance assessment. Educ Meas: Issues and Prac 14(5):5–8. https://doi.org/10.1111/j.1745-3992.1995.tb00881.x

    Google Scholar 

  • Messick S (1996) Validity of performance assessments. In: Phillips GW (ed) Technical issues in large-scale performance assessment. National Centre for Education Statistics, Washington

    Google Scholar 

  • Mestre J (2002) Transfer of learning: issues and research agenda. Report of a workshop held at the National Science Foundation. University of Massachusetts-Amherst, Virginia

  • Mitchell GW (2008) Essential soft skills for success in the twenty-first century workforce as perceived by Alabama business/marketing educators. PhD Dissertation, Auburn University

  • Moon TR, Brighton CM, Callahan CM, Robinson A (2005) Development of authentic assessments for the middle school classroom. The Journal of Secondary Gifted Education 16(2/3):119–133 http://files.eric.ed.gov/fulltext/EJ698321.pdf

    Article  Google Scholar 

  • Moore JL (2012) Designing for transfer: a threshold concept. The Journal of Faculty Development 26(3):19–24 http://ezproxy.utas.edu.au/login?url=http://search.proquest.com/docview/1143304893?accountid=14245

    Google Scholar 

  • Morrison WGS (1997) Competent crews = safer ships; an aid to understanding STCW 95. WMU Publications, Malmo

    Google Scholar 

  • Morrissey PE (2014) Investigating how an authentic task can promote student engagement when learning about Australian history. PhD Dissertation, University of Wollongong

  • Moskal BM, Leydens JA (2000) Scoring rubric development: validity and reliability. Practical Assessment, Research & Evaluation 7(10):71–81 http://pareonline.net/getvn.asp?v=7&n=10

    Google Scholar 

  • O’Farrell, C (2005) Enhancing student learning through Assessment. http://www.tcd.ie/teaching-learning/academic-development/ assets/pdf/ 250309_assessment _toolkit.pdf

  • Oh DM, Kim JM, Garcia RE, Krilowicz BL (2005) Valid and reliable authentic assessment of culminating student performance in the biomedical sciences. Adv Physiol Educ 29(2):83–93. https://doi.org/10.1152/advan.00039.2004

    Article  Google Scholar 

  • Olfos R, Zulantay H (2007) Reliability and validity of authentic assessment in a web based course. Educational Technology & Society 10(4):156–173 http://www.ifets.info/journals/10_4/15.pdf

    Google Scholar 

  • Pallis AA, Ng ADK (2011) Pursuing maritime education: an empirical study of students' profiles, motivations and expectations. Marit Policy Manag 38(4):369–393. https://doi.org/10.1080/03088839.2011.588258

    Article  Google Scholar 

  • Pecota SR, Buckley JJ (2009) Training paradigm assisted accidents: are we setting our students up for failure? In: Loginovsky V (ed) MET trends in the XXI century: shipping industry and training institutions in the global environment – area of mutual interests and cooperation. Admiral Makarov State Maritime Academy, Russia, pp 192–204

    Google Scholar 

  • Prasad R, Nakazawa T, Baldauf M (2010) Professional development of shipboard engineers and the role of collaborative learning. In: International Association of Maritime Universities AGA11. Korea Maritime University, Busan, Korea, pp 165–174

    Google Scholar 

  • Quartuch MJ (2011) Is authentic enough? Authentic assessment and civic engagement. Master Dissertation, Moravian College

  • Richards Perry GD (2011) Student perceptions of engagement in schools: a Deweyan analysis of authenticity in high school classrooms. PhD Dissertation, Georgia State University

  • Sadler R (2005) Interpretations of criteria-based assessment and grading in higher education. Assessment & Evaluation in Higher Education 30(2):175–194. https://doi.org/10.1080/0260293042000264262

    Article  Google Scholar 

  • Sampson H, Gekara V, Bloor M (2011) Water-tight or sinking? A consideration of the standards of the contemporary assessment practices underpinning seafarer licence examinations and their implications for employers. Maritime Policy & Management. The flagship journal of international shipping and port research 38(1):81–92. https://doi.org/10.1080/03088839.2010.533713

    Google Scholar 

  • Sator A (2000) An exploration of transfer of learning opportunities in an online co-operative Prepatory curriculum. Master Dissertation, Simon Fraser University

  • Saunders NG, Saunders GA, Batson T (2001) Assessment and the adult learner: does authentic assessment influence learning? In the annual meeting of the mid-western educational research association. Chicago, IL

  • Schell JW (2000) Think about Authentic Learning and Then Authentic Assessment. In: Custer RL (ed) Using Authentic Assessment in Vocational Education. Centre on Education and Training for Employment, Columbus

    Google Scholar 

  • Scott J (2000) Authentic Assessment Tools. In: Custer RL (ed) Using Authentic Assessment in Vocational Education. Centre on Education and Training for Employment College of Education, Columbia

    Google Scholar 

  • Secolsky C (1987) On the direct measurement of face validity: a comment on Nevo. J Educ Meas 24(1):82–83. https://doi.org/10.1111/j.1745-3984.1987.tb00265.x

    Article  Google Scholar 

  • Spady WG (1994) Outcome-based education: critical issues and answers. American Association of School Administrators, Arlington, VA

    Google Scholar 

  • Stevens P (2013) An examination of a Teacher's use of authentic assessment in an urban middle school setting. PhD Dissertation, Ohio University

  • Taylor JM (2011) Interdisciplinary authentic assessment: cognitive expectations and student performance. PhD Dissertation, Pepperdine University

  • Uchida M (2004) Analysis of human error in marine engine management. In: International Association of Maritime Universities AGA 05. Australian Maritime College, Tasmania, Australia, pp 85–93

    Google Scholar 

  • Varley MA (2008) Teachers' and administrators' perceptions of authentic assessment at a career and technical education centre. PhD Dissertation, Fordham University

  • Wiggins G (1989) A true test: toward more authentic and equitable assessment. The Phi Delta Kappan 70(9):703–713. https://doi.org/10.1177/003172171109200721

    Google Scholar 

  • Wiggins G (1992) Creating tests worth taking. Educ Leadersh 49(8):26–34 http://www.ascd.org/ASCD/pdf/journals/ed_lead/el_199205_wiggins.pdf

    Google Scholar 

  • Wiggins GP (1998) Educative assessment: designing assessments to inform and improve student performance. Jossey-Bass, CA,San Francisco

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. National Centre of Ports and Shipping, Australian Maritime College, University of Tasmania, Hobart, Australia

    Samrat Ghosh, Dev Ranmuthugala & Ben Brooks

  2. Future of Professional Work and Learning, Deakin University, Burwood, VIC, Australia

    Marcus Bowles

Authors
  1. Samrat Ghosh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Marcus Bowles
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dev Ranmuthugala
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ben Brooks
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Samrat Ghosh.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ghosh, S., Bowles, M., Ranmuthugala, D. et al. Improving the validity and reliability of authentic assessment in seafarer education and training: a conceptual and practical framework to enhance resulting assessment outcomes. WMU J Marit Affairs 16, 455–472 (2017). https://doi.org/10.1007/s13437-017-0129-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13437-017-0129-9

Keywords

Search

Navigation