Skip to main content

Advertisement

SpringerLink
Log in

Beyond Employment Rate: A Multidimensional Indicator of Higher Education Effectiveness

  • Published:
Social Indicators Research Aims and scope Submit manuscript

Abstract

This paper proposes a multidimensional indicator of higher education effectiveness that aims at going beyond the limits of measuring university effectiveness merely through employment rates. The units of analysis are the study programmes. Eleven indicators related to external effectiveness are selected, and their reliability for and relevance to the representation of the concept of effectiveness are empirically evaluated. The data are drawn from a longitudinal survey administered to graduates of the University of Padua, Italy, from 2008 to 2011. From our analyses, effectiveness appears to be a multidimensional concept composed by professional empowerment, employability and personal fulfilment. The right time for collecting relevant data on educational outcomes varies according to the types of indicators: indicators of professional empowerment assessed 1 year after graduation are most suitable, while for personal fulfilment measurement both short- and long-term evaluation are relevant, and, for employability, data collected 3 years after graduation cannot discriminate among study programmes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. Relevant is an indicator of what we need to know. This property implies that the indicator is designed to match the research objectives (i.e. ‘design validity’, see Bockstaller and Girardin 2003); from the stakeholders’ and decision-makers’ viewpoints, an indicator’s output is expected to be useful to inform their decisions.

  2. Reliable is an indicator’s measure that can be trusted. This property basically refers to the consistency of outputs in repeated trials under the same essential conditions (‘output validity’).

  3. The question on matching was posed this way to graduates: ‘Is the university degree you achieved specifically required for your job, can your job be performed with similar results by other graduates, or would a high school degree or even a degree lower than high school suffice?’

  4. The question is constructed with the responses to the same question as the previous indicator.

  5. The question on job-education consistency was posed differently in the three survey waves; in waves 1 and 2, graduates answered the question, ‘How much is your occupation consistent with your studies?’ using an ordinal scale (not at all, a little, somewhat, very much), while in wave 3 they used a cardinal scale (‘On a 1 to 10 scale, where 10 is the maximum, how much is your occupation consistent with your studies?’).

  6. The question on professional specialisation adequacy was posed this way to the graduates: ‘Do you feel that the degree of professional specialisation achieved at university is too high, too low, or adequate for (…)?’ The question was specified differently according to the position of graduates: for the employed, it referred to their current job; for the unemployed, to their ideal job.

  7. The question on graduates’ satisfaction of the skills they achieved at university was posed this way: ‘On a 1 to 10 scale, where 10 is the maximum, are the skills you achieved at university adequate (…)?’ The question was specified differently according to the position of graduates: for the employed, it referred to their current job; for the unemployed, to the job they desired.

  8. The question on willingness to repeat their educational experience was posed this way: ‘If you could go back in time, would you wish to attend the same study programme, the same faculty, the same university or none of them?’ The question was further specified by asking graduates what programme they would attend if they stated they would not repeat the experience.

  9. The question on job satisfaction was posed this way: ‘On a 1 to 10 scale, where 10 is the maximum, do you feel satisfied with your current occupation?’

  10. The question on overall satisfaction was posed this way: ‘On a 1 to 5 scale, where 5 is the maximum, do you feel satisfied with your global university experience?’

  11. To analyse and summarise the indicators at the individual level, we initially forced the missing variables to 0 or to a minimum that was equivalent to the statement that people who do not work also do not have a consistent job or do not have a job requiring a university degree, etc. This procedure generated spurious correlations between the indicators that were either very low or artificially high. The artificially high correlations may have been due to the fact that people who do not work are forced to have analogous values on a number of other indicators. Another possibility to compute the composite indicator of effectiveness at the individual level could be to exclude from the analysis all graduates who do not work, but this shortcut would exclude employability as a dimension of higher education effectiveness.

  12. In this analysis, each study programme is treated as a single observation, and then the possible presence of outlying cases is taken into account by the adopted models. However, a robustness control has been performed by adapting the final model (the one in Fig. 4) to the set of study programmes with at least 10 responses (n = 116) and to the set of study programmes with at least 20 responses (n = 98). The model fit is as good as the ones described in the paper, and the parameters are totally comparable. We report the analysis on study programmes with more than 5 responses to guarantee a better representation of all the majors.

  13. This indicator was formulated as the rate of graduates who did not attend a non-university course in the period after university graduation and until the interview, in order to be consistently oriented with the other indicators.

  14. An oblique rotation was applied because subdimensions of a common construct (i.e. effectiveness) are expected to be, at least to some extent, correlated.

  15. The graduates’ specialisation rate was excluded from our analysis because it correlated so strongly with graduates’ satisfaction of their achieved competencies and for their gained job that the measurement model did not fit the data (the modification indices equalled 14.96 and 11.06, respectively).

  16. An RMSEA below 0.06 and SRMR below 0.08, as well as an NFI and CFI above 0.95, indicate a good fit (Hu and Bentler 1999); for the GFI and AGFI, values of 0.90 or greater are recommended (Hooper et al. 2008); a critical N of 200 or larger indicates a satisfactory fit (Hoelter 1983).

  17. The Satorra–Bentler scaled Chi square was 17.23 (p value = 0.98; RMSEA < 0.0001; 90 % RMSEA C.I. = [0.0; 0.0]; probability of close fit = 0.94; SRMR = 0.048; GFI = 0.96; AGFI = 0.93; CFI = 1.00; NFI = 0.97; and Hoelter’s critical N = 398).

  18. All the fit measures listed in footnote 18 for the first-order measurement model without the job satisfaction items were exactly the same for the second-order model (at least at the precision level reported here).

  19. Ten indices, each measured at three time points, and one measured at graduation, as reported in Sect. 2.1 and in Table 1.

  20. The Satorra–Bentler scaled Chi-square was 5.65 (p value = 0.90; RMSEA < 0.0001; 90 % RMSEA C.I. = [0.00; 0.04]; probability of close fit = 0.875; SRMR = 0.021; GFI = 0.98; AGFI = 0.955; CFI = 1.00; NFI = 0.99; Hoelter’s critical N = 561).

References

  • Anderson, J. C., & Gerbing, D. W. (1988). Structural equation modelling in practice: A review and recommended two-step approach. Psychological Bulletin, 103(3), 411–423.

    Article  Google Scholar 

  • Astin, A. (1993). What matters in college. San Francisco: Jossey-Bass.

    Google Scholar 

  • Australian Government. (2011). Review of the Report on Government Services’ Performance Indicator Framework: Report to the Steering Committee for the Review of Government Service Provision. http://www.pc.gov.au/__data/assets/pdf_file/0006/102957/independent-reference-group-report.pdf.

  • Becker, G. S. (1994). Human capital: A theoretical and empirical analysis with special reference to education. Chicago, IL: The University of Chicago Press.

    Google Scholar 

  • Berk, R. A. (2005). Survey of 12 strategies to measure teaching effectiveness. International Journal of Teaching and Learning in Higher Education, 17(1), 48–62.

    Google Scholar 

  • Biggeri, L., Bini, M., & Grilli, L. (2001). The transition from university to work: a multilevel approach to the analysis of the time to obtain the first job. Journal of the Royal Statistical Society, Series A, 164, 293–305.

    Article  Google Scholar 

  • Bird, S. M., Cox, D., Farewell, V. T., Goldstein, H., Holt, T., & Smith, P. C. (2005). Performance indicators: Good, bad, and ugly. Journal of the Royal Statistical Society, Series A, 168(1), 1–27.

    Article  Google Scholar 

  • Blöndal, S., Field, S., & Girouard, N. (2002). Investment in Human Capital through Upper-Secondary and Tertiary Education. OECD Economic Studies, No. 34, OECD, Paris.

  • Boccuzzo, G., & Paggiaro, A. (2012). Facets of graduates’ job satisfaction. In L. Fabbris (Ed.), Indicators of higher education effectiveness (pp. 133–146). Milan: McGraw-Hill Education.

    Google Scholar 

  • Bockstaller, C., & Girardin, P. (2003). How to validate environmental indicators. Agricultural Systems, 76, 639–653.

    Article  Google Scholar 

  • Bratti, M., McKnight, A., Naylor, R., & Smith, J. (2004). Higher education outcomes, graduate employment and university performance indicators. Journal of the Royal Statistical Society, Series A, 167, 475–496.

    Article  Google Scholar 

  • Browne, M. W. (1987). Robustness in statistical inference in factor analysis and related model. Biometrika, 74, 375–384.

    Article  Google Scholar 

  • Cainarca, G. C., & Sgobbi, F. (2012). The return to education and skills in Italy. International Journal of Manpower, 33(2), 187–205.

    Article  Google Scholar 

  • Cammelli, A., & Gasperoni, G. (2012). Higher education external effectiveness indicators with reference to Italian universities. In L. Fabbris (Ed.), Indicators of higher education effectiveness (pp. 171–182). Milan: McGraw-Hill Education.

    Google Scholar 

  • Chalmers, D. (2008). Defining quality indicators in the context of quality models. Strawberry Hills: Australian Learning and Teaching Council.

    Google Scholar 

  • Consorzio Interuniversitario AlmaLaurea. (2011). XII Profilo dei laureati italiani. L’istruzione universitaria nell’ultimo decennio: All’esordio dell’European Higher Education Area. Bologna: Il Mulino.

    Google Scholar 

  • Cowan, J. (1985). Effectiveness and efficiency in higher education. Higher Education, 14(3), 235–239.

    Article  Google Scholar 

  • Daly, M. C., Büchel, F., & Duncan, G. J. (2000). Premiums and penalties for over-and undereducation: Cross-time and cross-national comparisons in the United States and Germany. Economics of Education Review, 19(2), 169–178.

    Article  Google Scholar 

  • Dey, E. L., Wimsatt, L. A., Rhee, B. S., & Meader, E. W. (1999). Long-term effect of college quality on the occupational status of students. National Center for Postsecondary Improvement, Stanford University, Technical Report Number 5-06.

  • Draper, D., & Gittoes, M. (2004). Statistical analysis of performance indicators in UK higher education (with discussion). Journal of the Royal Statistical Society, Series A, 167, 449–474.

    Article  Google Scholar 

  • EU-RA – European Research Association. (2006). Key education indicators on social inclusion and efficiency: final project report. Brussels: European Commission, Directorate General for Education and Culture.

    Google Scholar 

  • Fabbris, L. (2010). Il Progetto Agorà dell’Università di Padova. In Fabbris, L. (ed.), Dal Bo’ all’Agorà, il capitale umano investito nel lavoro (pp. V–XLVI). Cleup, Padova.

  • Fabbris, L. (2012). Concepts, dimensions and indicators for measuring higher education effectiveness. In L. Fabbris (Ed.), Indicators of higher education effectiveness (pp. 1–20). Milan: McGraw-Hill Education.

    Google Scholar 

  • Fabbris, L., & Favaro, D. (2012). Graduates’ human capital: An outcome in itself or an instrument for achieving outcomes? In L. Fabbris (Ed.), Indicators of higher education effectiveness (pp. 61–74). Milan: McGraw-Hill Education.

    Google Scholar 

  • Finnie, R., & Usher, A. (2005). Measuring the quality of post-secondary education: concepts, current practices and a strategic PLAN. Canada: CPRN-RCRPP.

    Google Scholar 

  • Furnham, A., Petrides, K. V., Jackson, C. J., & Cotter, T. (2002). Do personality factors predict job satisfaction? Personality and Individual Differences, 33, 1325–1342.

    Article  Google Scholar 

  • Garcìa-Aracil, A., & Palomares-Montero, D. (2009). Examining benchmark indicator systems for the evaluation of higher education institutions. Higher Education, 60(2), 217–234.

    Article  Google Scholar 

  • Gibbs, G. (2010). Dimensions of quality. Heslington: The Higher Education Academy.

    Google Scholar 

  • Groot, W., & Maassen van den Brink, H. (2000). Overeducation in the labour market: A meta-analysis. Economics of Education Review, 19(2), 149–158.

    Article  Google Scholar 

  • Hanushek, E. R. (1979). Conceptual and empirical issues in the estimation of educational production functions. Journal of Human Resources, 14, 351–388.

    Article  Google Scholar 

  • Hanushek, E. R., & Woessmann, L. (2008). The role of cognitive skills in economic development. Journal of Economic Literature, 46, 607–668.

    Article  Google Scholar 

  • Hartog, J. (2000). Over-education and earnings: Where are we, where should we go? Economics of Education Review, 19(2), 131–147.

    Article  Google Scholar 

  • Heckman, J. J. (1999). Policies to Foster Human Capital. Discussion paper No. 7288, National Bureau of Economic Research, New York.

  • Hoelter, D. R. (1983). The analysis of covariance structures: Goodness-of-fit indices. Sociological Methods and Research, 11, 325–344.

    Article  Google Scholar 

  • Hooper, D., Coughlan, J., & Mullen, M. R. (2008). Structural equation modelling: Guidelines for determining model fit. The Electronic Journal of Business Research Methods, 6(1), 53–60.

    Google Scholar 

  • Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1–55.

    Article  Google Scholar 

  • Iezzi, D. F., & Mastrangelo, M. (2012). Final-year students’ study satisfaction as a measure of educational effectiveness. In L. Fabbris (Ed.), Indicators of higher education effectiveness (pp. 21–34). Milan: McGraw-Hill Education.

    Google Scholar 

  • IIEP-UNESCO. (2011). External Quality Assurance: Options for Higher Education Managers, Module 4: Understanding and Assessing Quality. UNESCO-IIEP (International Institute for Educational Planning), Paris. www.iiep.unesco.org.

  • Istat. (2009). I laureati e lo studio. Inserimento professionale dei laureati. Indagine 2007. Roma: Istat.

    Google Scholar 

  • Jöreskog, K. G., & Sörbom, D. (Eds.). (1979). Advances in factor analysis and structural equation models. Cambridge: Abt Books.

    Google Scholar 

  • Jöreskog, K. G., & Sörbom, D. (2004). Lisrel 8.7 for Windows. Lincolnwood, IL: Scientific Software International, Inc.

    Google Scholar 

  • Jöreskog, K. G., Sörbom, D., Du Toit, S. H. C., & Du Toit, M. (2001). Lisrel 8: New statistical features (Third Printing with Revisions). Lincolnwood, IL: Scientific Software International Inc.

    Google Scholar 

  • Judge, T. A., Heller, D., & Mount, M. K. (2002). Five-factor model of personality and job satisfaction: A meta-analysis. Journal of Applied Psychology, 87, 530–541.

    Article  Google Scholar 

  • Land, K. C. (1975). Theories, models and indicators of social change. International Social Science Journal, XXVII(1), 7–37.

    Google Scholar 

  • Lockheed, M. E., & Hanushek, E. A. (1994). Concepts of Educational Efficiency and Effectiveness. Human Resources Development and Operations Policy, HRO Working Paper 24.

  • Lovaglio, P. G., & Vittadini, G. (2007). Human capital growth for university education evaluation. In L. Fabbris (Ed.), Effectiveness of University Education in Italy: Employability, competences, human capital (pp. 357–368). Heidelberg: Physica-Verlag.

    Google Scholar 

  • Lucarelli, C., Ungaro, P., & Verzicco, L. (2012). Employment rate as a measure of educational return. In L. Fabbris (Ed.), Indicators of higher education effectiveness (pp. 35–48). Milan: McGraw-Hill Education.

    Google Scholar 

  • Mainardes, E. W., Raposo, M., & Alves, H. (2012). Public university students’ expectations: An empirical study based on the stakeholders theory. Transylvanian Review of Administrative Sciences, 35, 173–196.

    Google Scholar 

  • Martini, M. C. (2012). The refusal of offered jobs. In L. Fabbris (Ed.), Indicators of higher education effectiveness (pp. 49–60). Milano: McGraw-Hill.

    Google Scholar 

  • McGuinness, S. (2006). Overeducation in the labour market. Journal of Economic Surveys, 20, 387–418.

    Article  Google Scholar 

  • McKee-Ryan, F. M., & Harvey, J. (2011). ‘I have a job, but.’: A review of underemployment. Journal of Management, 37(4), 962–996.

    Article  Google Scholar 

  • Mincer, J. (1981). Human Capital and Economic Growth, NBER Working Paper Series, No. 803.

  • Nardo, M., Saisana, M., Saltelli, A., & Tarantola, S. (2005). Handbook on Constructing Composite Indicators: Methodology and User Guide, OECD Statistics Working Paper, STD/DOC(2005)3. http://www.oecd.org/std/research.

  • Nguyen, A. N., & Taylor, J. (2003). Transition from school to first job: the influence of educational attainment. Lancaster University Management School, Working paper 2003/009.

  • Nunally, J. C. (1978). Psychometric theory. New York, NY: McGraw-Hill Book Company.

    Google Scholar 

  • Palomares-Montero, D., & Garcìa-Aracil, A. (2011). What are the key indicators for evaluating the activities of universities? Research Evaluation, 20(5), 353–363.

    Article  Google Scholar 

  • Pascarella, E. T., & Terenzini, P. (2005). How college affects students: A third decade of research (Vol. 2). San Francisco, CA: Jossey-Bass.

    Google Scholar 

  • Quintini, G. (2011). Over-Qualified or Under-Skilled: A Review of Existing Literature. OECD Social Employment and Migration Working Papers, No. 121, OECD Publishing.

  • Rindermann, H. (2008). Relevance of education and intelligence at the national level for the economic welfare of people. Intelligence, 36, 127–142.

    Article  Google Scholar 

  • Satorra, A., & Bentler, P.M. (1988). Scaling corrections for Chi square statistics in covariance structure analysis. In Proceedings of the Business and Economic Statistics Section of the American Statistical Association (pp. 308–313).

  • Scheerens, J., & Bosker, R. J. (1997). The foundations of educational effectiveness. Oxford: Elsevier Science Ltd.

    Google Scholar 

  • Sicherman, N., & Galor, O. (1990). A theory of career mobility. The Journal of Political Economy, 98(1), 169–192.

    Article  Google Scholar 

  • Sloane, P. J. (2003). Much ado about nothing? What does the over-education literature really tell us? In F. Büchel, A. de Grip, & A. Mertens (Eds.), Overeducation in Europe: Current issues in theory and policy (pp. 11–48). Chelterham: Edward Elgar.

    Google Scholar 

  • Spiegelhalter, D. J., Sherlaw-Johnson, C., Bardsley, M., Blunt, I., Wood, C., & Grigg, O. (2012). Statistical methods for healthcare regulation: Rating, screening and surveillance. Journal of the Royal Statistical Society, Series A, 179(1), 1–25.

    Article  Google Scholar 

  • St. Aubyn, M., Pina, A., Garcia, F., & Pais, J. (2009). Study on the efficiency and effectiveness of public spending on tertiary education. Brussels: European Economy, Economic Papers 390.

  • Stern, J. D. (1986). The Educational Indicators Project at the U.S. Department of Education. Washington, DC: U.S. Department of Education.

    Google Scholar 

  • UN – United Nations. (1975). Towards a system of social and demographic statistics. Studies in methods, Series F, n. 18. New York: Department of Social and Economic Affairs, United Nations.

    Google Scholar 

  • UNDP – United Nations Development Program. (2010). Results based management handbook: Strengthening RBM harmonization for improved development results. New York: United Nations Development Group.

    Google Scholar 

  • UNESCO. (1974). Social Indicators: Problems of Definition and of Selection. Reports and Papers in the Social Sciences No. 30, UNESCO, Paris.

  • Verhaest, D., & Omey, E. (2009). Objective over-education and worker well-being: A shadow price approach. Journal of Economic Psychology, 30, 469–481.

    Article  Google Scholar 

  • Zhou, P., & Ang, B. W. (2009). Comparing MCDA aggregation methods in constructing composite indicators using the Shannon–Spearman measure. Social Indicators Research, 94(1), 83–96.

    Article  Google Scholar 

Download references

Acknowledgments

This work was pursued as part of two projects: (1) Prin 2007 (CUP C91J11002460001) ‘Models, indicators and methods for the analysis of the educational effectiveness of a university study programme with the purpose of its accreditation and improvement’, jointly funded by the Ministry of Education and the University of Padua, and (2) a 2008 project of Padua University (CUP CPDA081538) titled ‘Effectiveness indicators of tertiary education and methodological outcomes of the research on University of Padua graduates’, both coordinated by L. Fabbris. The authors share the responsibility of the whole paper; L. Fabbris edited Sects. 1, 2.1 and 5, and M.C. Martini edited all other sections.

Author information

Authors and Affiliations

  1. Communication and Economics Department, University of Modena and Reggio Emilia, Viale Allegri 9, Reggio Emilia, Italy

    Maria Cristiana Martini

  2. Statistics Department, University of Padua, Via Battisti 241, Padua, Italy

    Luigi Fabbris

Authors
  1. Maria Cristiana Martini
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Luigi Fabbris
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Maria Cristiana Martini.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Martini, M.C., Fabbris, L. Beyond Employment Rate: A Multidimensional Indicator of Higher Education Effectiveness. Soc Indic Res 130, 351–370 (2017). https://doi.org/10.1007/s11205-015-1179-z

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11205-015-1179-z

Keywords

Search

Navigation