Advertisement

Social Indicators Research

, Volume 130, Issue 1, pp 351–370 | Cite as

Beyond Employment Rate: A Multidimensional Indicator of Higher Education Effectiveness

  • Maria Cristiana MartiniEmail author
  • Luigi Fabbris
Article

Abstract

This paper proposes a multidimensional indicator of higher education effectiveness that aims at going beyond the limits of measuring university effectiveness merely through employment rates. The units of analysis are the study programmes. Eleven indicators related to external effectiveness are selected, and their reliability for and relevance to the representation of the concept of effectiveness are empirically evaluated. The data are drawn from a longitudinal survey administered to graduates of the University of Padua, Italy, from 2008 to 2011. From our analyses, effectiveness appears to be a multidimensional concept composed by professional empowerment, employability and personal fulfilment. The right time for collecting relevant data on educational outcomes varies according to the types of indicators: indicators of professional empowerment assessed 1 year after graduation are most suitable, while for personal fulfilment measurement both short- and long-term evaluation are relevant, and, for employability, data collected 3 years after graduation cannot discriminate among study programmes.

Keywords

Educational effectiveness Composite indicator Indicator relevance Indicator reliability Structural equation modelling University of Padua 

Notes

Acknowledgments

This work was pursued as part of two projects: (1) Prin 2007 (CUP C91J11002460001) ‘Models, indicators and methods for the analysis of the educational effectiveness of a university study programme with the purpose of its accreditation and improvement’, jointly funded by the Ministry of Education and the University of Padua, and (2) a 2008 project of Padua University (CUP CPDA081538) titled ‘Effectiveness indicators of tertiary education and methodological outcomes of the research on University of Padua graduates’, both coordinated by L. Fabbris. The authors share the responsibility of the whole paper; L. Fabbris edited Sects. 1, 2.1 and 5, and M.C. Martini edited all other sections.

References

  1. Anderson, J. C., & Gerbing, D. W. (1988). Structural equation modelling in practice: A review and recommended two-step approach. Psychological Bulletin, 103(3), 411–423.CrossRefGoogle Scholar
  2. Astin, A. (1993). What matters in college. San Francisco: Jossey-Bass.Google Scholar
  3. Australian Government. (2011). Review of the Report on Government Services’ Performance Indicator Framework: Report to the Steering Committee for the Review of Government Service Provision. http://www.pc.gov.au/__data/assets/pdf_file/0006/102957/independent-reference-group-report.pdf.
  4. Becker, G. S. (1994). Human capital: A theoretical and empirical analysis with special reference to education. Chicago, IL: The University of Chicago Press.Google Scholar
  5. Berk, R. A. (2005). Survey of 12 strategies to measure teaching effectiveness. International Journal of Teaching and Learning in Higher Education, 17(1), 48–62.Google Scholar
  6. Biggeri, L., Bini, M., & Grilli, L. (2001). The transition from university to work: a multilevel approach to the analysis of the time to obtain the first job. Journal of the Royal Statistical Society, Series A, 164, 293–305.CrossRefGoogle Scholar
  7. Bird, S. M., Cox, D., Farewell, V. T., Goldstein, H., Holt, T., & Smith, P. C. (2005). Performance indicators: Good, bad, and ugly. Journal of the Royal Statistical Society, Series A, 168(1), 1–27.CrossRefGoogle Scholar
  8. Blöndal, S., Field, S., & Girouard, N. (2002). Investment in Human Capital through Upper-Secondary and Tertiary Education. OECD Economic Studies, No. 34, OECD, Paris.Google Scholar
  9. Boccuzzo, G., & Paggiaro, A. (2012). Facets of graduates’ job satisfaction. In L. Fabbris (Ed.), Indicators of higher education effectiveness (pp. 133–146). Milan: McGraw-Hill Education.Google Scholar
  10. Bockstaller, C., & Girardin, P. (2003). How to validate environmental indicators. Agricultural Systems, 76, 639–653.CrossRefGoogle Scholar
  11. Bratti, M., McKnight, A., Naylor, R., & Smith, J. (2004). Higher education outcomes, graduate employment and university performance indicators. Journal of the Royal Statistical Society, Series A, 167, 475–496.CrossRefGoogle Scholar
  12. Browne, M. W. (1987). Robustness in statistical inference in factor analysis and related model. Biometrika, 74, 375–384.CrossRefGoogle Scholar
  13. Cainarca, G. C., & Sgobbi, F. (2012). The return to education and skills in Italy. International Journal of Manpower, 33(2), 187–205.CrossRefGoogle Scholar
  14. Cammelli, A., & Gasperoni, G. (2012). Higher education external effectiveness indicators with reference to Italian universities. In L. Fabbris (Ed.), Indicators of higher education effectiveness (pp. 171–182). Milan: McGraw-Hill Education.Google Scholar
  15. Chalmers, D. (2008). Defining quality indicators in the context of quality models. Strawberry Hills: Australian Learning and Teaching Council.Google Scholar
  16. Consorzio Interuniversitario AlmaLaurea. (2011). XII Profilo dei laureati italiani. L’istruzione universitaria nell’ultimo decennio: All’esordio dell’European Higher Education Area. Bologna: Il Mulino.Google Scholar
  17. Cowan, J. (1985). Effectiveness and efficiency in higher education. Higher Education, 14(3), 235–239.CrossRefGoogle Scholar
  18. Daly, M. C., Büchel, F., & Duncan, G. J. (2000). Premiums and penalties for over-and undereducation: Cross-time and cross-national comparisons in the United States and Germany. Economics of Education Review, 19(2), 169–178.CrossRefGoogle Scholar
  19. Dey, E. L., Wimsatt, L. A., Rhee, B. S., & Meader, E. W. (1999). Long-term effect of college quality on the occupational status of students. National Center for Postsecondary Improvement, Stanford University, Technical Report Number 5-06.Google Scholar
  20. Draper, D., & Gittoes, M. (2004). Statistical analysis of performance indicators in UK higher education (with discussion). Journal of the Royal Statistical Society, Series A, 167, 449–474.CrossRefGoogle Scholar
  21. EU-RA – European Research Association. (2006). Key education indicators on social inclusion and efficiency: final project report. Brussels: European Commission, Directorate General for Education and Culture.Google Scholar
  22. Fabbris, L. (2010). Il Progetto Agorà dell’Università di Padova. In Fabbris, L. (ed.), Dal Bo’ all’Agorà, il capitale umano investito nel lavoro (pp. V–XLVI). Cleup, Padova.Google Scholar
  23. Fabbris, L. (2012). Concepts, dimensions and indicators for measuring higher education effectiveness. In L. Fabbris (Ed.), Indicators of higher education effectiveness (pp. 1–20). Milan: McGraw-Hill Education.Google Scholar
  24. Fabbris, L., & Favaro, D. (2012). Graduates’ human capital: An outcome in itself or an instrument for achieving outcomes? In L. Fabbris (Ed.), Indicators of higher education effectiveness (pp. 61–74). Milan: McGraw-Hill Education.Google Scholar
  25. Finnie, R., & Usher, A. (2005). Measuring the quality of post-secondary education: concepts, current practices and a strategic PLAN. Canada: CPRN-RCRPP.Google Scholar
  26. Furnham, A., Petrides, K. V., Jackson, C. J., & Cotter, T. (2002). Do personality factors predict job satisfaction? Personality and Individual Differences, 33, 1325–1342.CrossRefGoogle Scholar
  27. Garcìa-Aracil, A., & Palomares-Montero, D. (2009). Examining benchmark indicator systems for the evaluation of higher education institutions. Higher Education, 60(2), 217–234.CrossRefGoogle Scholar
  28. Gibbs, G. (2010). Dimensions of quality. Heslington: The Higher Education Academy.Google Scholar
  29. Groot, W., & Maassen van den Brink, H. (2000). Overeducation in the labour market: A meta-analysis. Economics of Education Review, 19(2), 149–158.CrossRefGoogle Scholar
  30. Hanushek, E. R. (1979). Conceptual and empirical issues in the estimation of educational production functions. Journal of Human Resources, 14, 351–388.CrossRefGoogle Scholar
  31. Hanushek, E. R., & Woessmann, L. (2008). The role of cognitive skills in economic development. Journal of Economic Literature, 46, 607–668.CrossRefGoogle Scholar
  32. Hartog, J. (2000). Over-education and earnings: Where are we, where should we go? Economics of Education Review, 19(2), 131–147.CrossRefGoogle Scholar
  33. Heckman, J. J. (1999). Policies to Foster Human Capital. Discussion paper No. 7288, National Bureau of Economic Research, New York.Google Scholar
  34. Hoelter, D. R. (1983). The analysis of covariance structures: Goodness-of-fit indices. Sociological Methods and Research, 11, 325–344.CrossRefGoogle Scholar
  35. Hooper, D., Coughlan, J., & Mullen, M. R. (2008). Structural equation modelling: Guidelines for determining model fit. The Electronic Journal of Business Research Methods, 6(1), 53–60.Google Scholar
  36. Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1–55.CrossRefGoogle Scholar
  37. Iezzi, D. F., & Mastrangelo, M. (2012). Final-year students’ study satisfaction as a measure of educational effectiveness. In L. Fabbris (Ed.), Indicators of higher education effectiveness (pp. 21–34). Milan: McGraw-Hill Education.Google Scholar
  38. IIEP-UNESCO. (2011). External Quality Assurance: Options for Higher Education Managers, Module 4: Understanding and Assessing Quality. UNESCO-IIEP (International Institute for Educational Planning), Paris. www.iiep.unesco.org.
  39. Istat. (2009). I laureati e lo studio. Inserimento professionale dei laureati. Indagine 2007. Roma: Istat.Google Scholar
  40. Jöreskog, K. G., & Sörbom, D. (Eds.). (1979). Advances in factor analysis and structural equation models. Cambridge: Abt Books.Google Scholar
  41. Jöreskog, K. G., & Sörbom, D. (2004). Lisrel 8.7 for Windows. Lincolnwood, IL: Scientific Software International, Inc.Google Scholar
  42. Jöreskog, K. G., Sörbom, D., Du Toit, S. H. C., & Du Toit, M. (2001). Lisrel 8: New statistical features (Third Printing with Revisions). Lincolnwood, IL: Scientific Software International Inc.Google Scholar
  43. Judge, T. A., Heller, D., & Mount, M. K. (2002). Five-factor model of personality and job satisfaction: A meta-analysis. Journal of Applied Psychology, 87, 530–541.CrossRefGoogle Scholar
  44. Land, K. C. (1975). Theories, models and indicators of social change. International Social Science Journal, XXVII(1), 7–37.Google Scholar
  45. Lockheed, M. E., & Hanushek, E. A. (1994). Concepts of Educational Efficiency and Effectiveness. Human Resources Development and Operations Policy, HRO Working Paper 24.Google Scholar
  46. Lovaglio, P. G., & Vittadini, G. (2007). Human capital growth for university education evaluation. In L. Fabbris (Ed.), Effectiveness of University Education in Italy: Employability, competences, human capital (pp. 357–368). Heidelberg: Physica-Verlag.Google Scholar
  47. Lucarelli, C., Ungaro, P., & Verzicco, L. (2012). Employment rate as a measure of educational return. In L. Fabbris (Ed.), Indicators of higher education effectiveness (pp. 35–48). Milan: McGraw-Hill Education.Google Scholar
  48. Mainardes, E. W., Raposo, M., & Alves, H. (2012). Public university students’ expectations: An empirical study based on the stakeholders theory. Transylvanian Review of Administrative Sciences, 35, 173–196.Google Scholar
  49. Martini, M. C. (2012). The refusal of offered jobs. In L. Fabbris (Ed.), Indicators of higher education effectiveness (pp. 49–60). Milano: McGraw-Hill.Google Scholar
  50. McGuinness, S. (2006). Overeducation in the labour market. Journal of Economic Surveys, 20, 387–418.CrossRefGoogle Scholar
  51. McKee-Ryan, F. M., & Harvey, J. (2011). ‘I have a job, but.’: A review of underemployment. Journal of Management, 37(4), 962–996.CrossRefGoogle Scholar
  52. Mincer, J. (1981). Human Capital and Economic Growth, NBER Working Paper Series, No. 803.Google Scholar
  53. Nardo, M., Saisana, M., Saltelli, A., & Tarantola, S. (2005). Handbook on Constructing Composite Indicators: Methodology and User Guide, OECD Statistics Working Paper, STD/DOC(2005)3. http://www.oecd.org/std/research.
  54. Nguyen, A. N., & Taylor, J. (2003). Transition from school to first job: the influence of educational attainment. Lancaster University Management School, Working paper 2003/009.Google Scholar
  55. Nunally, J. C. (1978). Psychometric theory. New York, NY: McGraw-Hill Book Company.Google Scholar
  56. Palomares-Montero, D., & Garcìa-Aracil, A. (2011). What are the key indicators for evaluating the activities of universities? Research Evaluation, 20(5), 353–363.CrossRefGoogle Scholar
  57. Pascarella, E. T., & Terenzini, P. (2005). How college affects students: A third decade of research (Vol. 2). San Francisco, CA: Jossey-Bass.Google Scholar
  58. Quintini, G. (2011). Over-Qualified or Under-Skilled: A Review of Existing Literature. OECD Social Employment and Migration Working Papers, No. 121, OECD Publishing.Google Scholar
  59. Rindermann, H. (2008). Relevance of education and intelligence at the national level for the economic welfare of people. Intelligence, 36, 127–142.CrossRefGoogle Scholar
  60. Satorra, A., & Bentler, P.M. (1988). Scaling corrections for Chi square statistics in covariance structure analysis. In Proceedings of the Business and Economic Statistics Section of the American Statistical Association (pp. 308–313).Google Scholar
  61. Scheerens, J., & Bosker, R. J. (1997). The foundations of educational effectiveness. Oxford: Elsevier Science Ltd.Google Scholar
  62. Sicherman, N., & Galor, O. (1990). A theory of career mobility. The Journal of Political Economy, 98(1), 169–192.CrossRefGoogle Scholar
  63. Sloane, P. J. (2003). Much ado about nothing? What does the over-education literature really tell us? In F. Büchel, A. de Grip, & A. Mertens (Eds.), Overeducation in Europe: Current issues in theory and policy (pp. 11–48). Chelterham: Edward Elgar.Google Scholar
  64. Spiegelhalter, D. J., Sherlaw-Johnson, C., Bardsley, M., Blunt, I., Wood, C., & Grigg, O. (2012). Statistical methods for healthcare regulation: Rating, screening and surveillance. Journal of the Royal Statistical Society, Series A, 179(1), 1–25.CrossRefGoogle Scholar
  65. St. Aubyn, M., Pina, A., Garcia, F., & Pais, J. (2009). Study on the efficiency and effectiveness of public spending on tertiary education. Brussels: European Economy, Economic Papers 390.Google Scholar
  66. Stern, J. D. (1986). The Educational Indicators Project at the U.S. Department of Education. Washington, DC: U.S. Department of Education.Google Scholar
  67. UN – United Nations. (1975). Towards a system of social and demographic statistics. Studies in methods, Series F, n. 18. New York: Department of Social and Economic Affairs, United Nations.Google Scholar
  68. UNDP – United Nations Development Program. (2010). Results based management handbook: Strengthening RBM harmonization for improved development results. New York: United Nations Development Group.Google Scholar
  69. UNESCO. (1974). Social Indicators: Problems of Definition and of Selection. Reports and Papers in the Social Sciences No. 30, UNESCO, Paris.Google Scholar
  70. Verhaest, D., & Omey, E. (2009). Objective over-education and worker well-being: A shadow price approach. Journal of Economic Psychology, 30, 469–481.CrossRefGoogle Scholar
  71. Zhou, P., & Ang, B. W. (2009). Comparing MCDA aggregation methods in constructing composite indicators using the Shannon–Spearman measure. Social Indicators Research, 94(1), 83–96.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2015

Authors and Affiliations

  1. 1.Communication and Economics DepartmentUniversity of Modena and Reggio EmiliaReggio EmiliaItaly
  2. 2.Statistics DepartmentUniversity of PaduaPaduaItaly

Personalised recommendations