Measuring Student Success: A Value-Added Approach
- 292 Downloads
The notion of what constitutes a ‘quality’ university has been challenged by the 2014 Gallup-Purdue Survey (Great Jobs, Great Lives: The 2014 Gallup-Purdue Index Report, Gallup, Inc., 2014). This survey of 30,000 US university alumni revealed that engagement and feelings of well-being beyond the university and into the workplace have little to do with the prestige of the university and much to do with having caring professors and being afforded opportunities for experiential learning. The Survey has shifted the focus from what university professors value to what students value. Assuming universities are interested in what students think, the issue then becomes one of assessing ‘value added’, and this paper examines one university’s approach to addressing this issue.
KeywordsUniversity quality Value added Workplace engagement Caring professor Experiential learning
Rothwell and Kulkarni (2015) have noted that: ‘the choice of whether or where to attend college/university is amongst the most important decisions individuals and families can make, yet people know little about how institutions of higher learning compare along important dimensions of quality’ (p. 1). This decision revolves around the notion of a quality university, and a key question is whether ‘quality’ is all about academic reputation as defined by professors, which invariably places the likes of MIT, Oxford and Harvard at the top of the quality ratings, or is it something different? For example, what do students think? Is their notion of quality the same as that of professors? It seems from the Gallup-Purdue Survey (2014) that the typical perception of quality which is generally reflected in the various university ranking systems that, in turn, reflect faculty opinions is far less important to students than other factors that are part of the student experience.
2 The Gallup-Purdue Survey
A major challenge to the accepted wisdom on university quality was presented in the above-mentioned Gallup-Purdue Survey that was conducted in 2014 and involved a survey of 30,000 US university alumni. A US news report that came out on publication of the Survey and was entitled ‘Gallup: College Type Has Little To Do with Success’, also headlined with ‘Just having one inspiring professor can double a graduates odds of being engaged at work’ (Bidwell, 2014). In more detail, the major factors which were viewed by respondents as making significant contributions to their level of workplace engagement and their general feeling of well-being after their university experience were (1) having a mentor who encouraged them to pursue their goals and dreams, (2) having professors who cared about them as people, (3) having professors who made them excited about learning, (4) being engaged in a project that took a semester or more to complete, (5) having an internship or a job that allowed them to apply their classroom learning and (6) being active in extracurricular activities. In summary, the Gallup-Purdue Survey (2014) revealed that graduates’ subsequent engagement in the workplace and feelings of well-being are strongly associated with professors who cared about them and the experiential learning opportunities that the college or university offered them.
Interestingly, whether the college or university was public or private, selective or non-selective as well as its academic reputation were all insignificant factors in producing workplace engagement and feelings of well-being. The Gallup-Purdue Survey (2014) challenged the conventional notion of what constitutes a ‘quality’ university and put the spotlight on aspects of the university experience not normally associated with conventional notions of university quality. In fact, the Survey identified factors which fall within the broad category of ‘value added’. The notion of value added, in a university context, may be described as the educational value derived by students resulting from the time they have spent in a higher education institution. It is not easy to measure value added directly, but we can find indicators of its existence.
3 The Lingnan University Example
Lingnan University is one of seven universities in Hong Kong that is funded through the Hong Kong University Grants Committee (UGC). The UGC’s website describes the UGC as follows: ‘Its main function is to offer impartial and expert advice to the Government on the funding and development of higher education in Hong Kong, and to provide assurance to the Government and the community on the standards and cost-effectiveness of the operations and activities of the UGC-funded institutions’ (2017).
As well as being one of the ‘approved’ UGC institutions, Lingnan University has a long and distinguished history. It has its origins in Guangzhou, China, where it was established as a Christian College in the late nineteenth century by the American Presbyterian Church. By the 1930s, it had established itself as a key education institution in Southern China. However, a nationwide higher education programme reform undertaken in late 1952 saw its closure in the Chinese Mainland and its emergence as Lingnan College in Hong Kong in 1967. In the late 1990s, Lingnan College was given self-accrediting status and was renamed Lingnan University.
The University has emerged since then as the only liberal arts university in Hong Kong with a deep commitment to a mix of liberal studies and discipline-based education. The status and value of liberal arts education is not fully acknowledged in Hong Kong, and consequently the University does not attract the highest-profile students in terms of academic attainment in their school years. It also draws on a catchment area where families are not well off. In some senses, admitting students who are not ‘high fliers’ academically and are also from families with moderate income levels poses a challenge but also offers an opportunity in terms of ‘value added’.
4 Lingnan’s Multi-faceted Approach to Measuring Value Added
One of Lingnan’s major platforms is the ability to take middle- to lower-level students both in terms of entry qualifications and household incomes and produce graduates who can compete with the best. Accordingly, the University has developed a multi-faceted approach to measuring the value added to students, and these elements are summarized below.
4.1 The Graduate Exit Survey
The Graduate Exit Survey contains graduates’ self-reported data (1) on the attainment of graduate outcomes in areas such as critical thinking and problem-solving; (2) on the overall teaching and learning environment, e.g. the extent of active learning, the relationship between teachers and students, etc.; and (3) on the overall impression of the University, e.g. hostel life and the quality of internships and exchanges, and (4) poses an open-ended question on the best and worst aspects of the University experience.
4.2 Survey on the Development of Generic Skills
This is a survey conducted at two points in a student’s academic life, first on entry to the University and second on return to the University for graduation, normally a few months into the graduate’s first job after graduation. The attributes that are surveyed are those that are targeted in the co-curricular courses produced by the Student Services Centre such as organizing ability, social skills and the ability to cope with stress. Again, this data comprises students’ self-reports.
4.3 Value-Added Indicators Specific to the University
Earlier, reference was made to the typical profile of Lingnan’s liberal arts students. These are students who have not been the highest of achievers academically and have come from modest income families. Three indicators are viewed as relevant to this profile. The first is a report on student qualifications on admission; the second is government data that compares, by university, the incidence of grant and loan applications amongst admitted students; and the third is the survey of Lingnan University graduates completed by employers, which, amongst other information, compares the work competence of Lingnan graduates to those of the other UGC institutions.
5 The Findings of Lingnan’s Multi-faceted Approach
The Graduate Exit Survey indicates an upward trend year on year with relations between teachers and students consistently cited as one of the best aspects of the Lingnan University experience; in other words, students generally feel they have caring professors. Similarly, the survey on the development of generic skills also indicates, each year, a significant increase in self-ratings of these skills when comparing a student’s ratings on entry to the University, with those a few months into his or her first job.
Data on student qualifications on admission to the University indicates students to be generally at the ‘C’ grade of academic achievement on entry, and the data on the incidence of grant and loan applications shows that, in comparison with students admitted to the other UGC universities, Lingnan has the second highest incidence of students who have required financial aid of some sort. In summary, the University’s claim to admit students with moderate academic achievement and from families with modest incomes is supported by objective data.
However, the most recent employer’s survey conducted in 2014 revealed that 30% of employers surveyed perceived Lingnan graduates to be better work performers than their peers in the other UGC institutions, with 60% perceiving Lingnan graduates to be at least as good as their peers from the other universities. Not unsurprisingly with data of this nature, 10% of employers rated Lingnan graduates as not as good as their peers. Nevertheless, this is a very encouraging finding given the profile of the University’s students and the fact that, in terms of work performance, they are competing against students from universities, four of which are ranked in the top 100 of the QS World Rankings.
The Gallup-Purdue Survey (2014) opened the door to an examination of quality education from the students’ perspective and indicated that caring professors and opportunities for experiential learning were key factors of engagement and feelings of well-being that carried into the workplace. The existence of such studies challenges conventional ideas on what constitutes a quality university, and it is noteworthy that these conventional notions of quality, such as the prestige of the university, were perceived in the Survey as of minimal value by students.
What then is of value in a university education? Questions such as this raise the whole issue of ‘value added’, and this paper has described how one university has attempted to measure this value through a multi-faceted approach. It may be that other universities, not being counted amongst the elite by conventional methods, might want to experiment with measures similar to those described in this paper.
By way of a final comment, based on the criteria identified in the Gallup-Purdue Survey (2014) as being major factors in student and alumni engagement and feelings of well-being, namely, the caring professor and the opportunity for experiential learning, Lingnan University fares very well. Students consistently laud their relationship with their professors thus satisfying the ‘caring professor’ criterion. Lingnan University also satisfies the experiential learning criterion given that the University not only offers opportunities for students to engage in experiential learning, but it also actually requires it because each student has to engage in service-oriented experiential learning through its Office of Service Learning with service learning being a graduation requirement.
Aside from the specifics of Lingnan University, one might also conclude that indicators such as the presence of caring professors and opportunities for experiential learning should play an increasingly significant role in any measure of university quality if the results of the Gallup-Purdue Survey (2014) are to be accepted. Certainly, these factors are worth considering when parents and sons and daughters are making decisions on whether or where to attend college/university.
- Bidwell, A. Gallup: College type has little to do with success. http://www.usnews.com/news/articles/2014/05/06/gallup-purdue-index-measures-the-magic-equation-to-student-success
- Great Jobs Great Lives. (2014): The 2014 Gallup-Purdue Index Report. Gallup.Google Scholar
- Kulkarni, S., & Rothwell, J. (2015). Beyond college rankings: A value-added approach to assessing two- and four-year schools. Washington, DC: Brookings Institute Press.Google Scholar
- University Grants Committee. (2017). http://www.ugc.edu.hk/eng/ugc/about/overview/roles.html