Abstract
This manuscript chronicles the process and products of a redesign for evaluation of the graduate college experience (GCE) which was initiated by a university graduate college, based on its observed need to reconsider and update its measures and methods for assessing graduate students’ experiences. We examined the existing instrumentation and procedures; met with and interviewed staff and stakeholders regarding individual and organizational needs; collected systematic questionnaire data on stakeholder perceptions; and then redesigned, developed, and tested new evaluation instruments, systems, and procedures. The previously paper-based, one-time global exit questionnaire was redesigned into a digitally administered, multi-event assessment series, with content relevant to students’ incremental academic progress. Previously discrete items were expanded into psychometrically coherent variable scales in parallel forms to assess change over time (entry, mid-point, exit, post-graduation). They were also strategically designed as stable and independent enough so administrators could vary the timing and sequence of administration to fit their ongoing needs The team conducted two testing cycles, gathering pertinent information on the redesigned assessment and procedures (N = 2,835). The final redesigned evaluation serves as an exemplar of evaluation that enhances assessment quality including psychometric properties and multiple stakeholder validation, more effectively addresses the organization’s incremental evaluation needs, increases timeliness of data collection, improves reach to and participation of distributed students, and enables longitudinal data collection to provide ongoing trajectory-of-change evaluation and a research data stream. Product and process analysis informs strategies for more effectively and dynamically assessing graduate education.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
These items presented the university’s acronym, replaced here with the generic “[univ]”.
References
Allum, J. R., Bell, N. E., & Sowell, R. S. (2012). Graduate enrollment and degrees: 2001 to 2011. Washington: Council of Graduate Schools.
Austin, J., Cameron, T., Glass, M., Kosko, K., Marsh, F., Abdelmagid, R., & Burge, P. (2009). First semester experiences of professionals transitioning to full-time doctoral study. College Student Affairs Journal, 27(2), 194–214.
Baker, V. L., & Lattuca, L. R. (2010). Developmental networks and learning: toward an interdisciplinary perspective on identity development during doctoral study. Studies in Higher Education, 35(7), 807–827.
Bandilla, W., Bosnjak, M., & Altdorfer, P. (2003). Survey administration effects? A comparison of web-based and traditional written self-administered surveys using the ISSP environment model. Social Science Computer Review, 21, 235–243.
Belcher, M. J. (1996). A survey of current & potential graduate students. Research report 96–04. Boise: Boise State University.
Benishek, L. A., & Chessler, M. (2005). Facilitating the identity development of counseling graduate students as researchers. Journal of Humanistic Counseling Education and Development, 44(1), 16–31.
Bloom, J. L., Cuevas, A. E. P., Evans, C. V., & Hall, J. W. (2007). Graduate students’ perceptions of outstanding graduate advisor characteristics. NACADA Journal, 27(2), 28–35.
Brinkman, S. N., & Hartsell-Gundy, A. A. (2012). Building trust to relieve graduate student research anxiety. Public Services Quarterly, 8(1), 26–39.
Chism, M., Thomas, E. L., Knight, D., Miller, J., Cordell, S., Smith, L., & Richardson, D. (2010). Study of graduate student perceptions at the University of West Alabama. Alabama Counseling Association Journal, 36(1), 49–55.
Cicognani, E., Menezes, I., & Nata, G. (2011). University students’ sense of belonging to the home town: the role of residential mobility. Social Indicators Research, 104(1), 33–45.
Cook, D. A., & Beckman, T. J. (2006). Current concepts in validity and reliability for psychometric instruments: theory and application. The American Journal of Medicine, 119, 116.e7–166.e16.
Coulter, F. W., Goin, R. P., & Gerard, J. M. (2004). Assessing graduate students’ needs: the role of graduate student organizations. Educational Research Quarterly, 28(1), 15–26.
Council of Graduate Schools. (2012). Findings from the 2012 CGS international graduate admissions survey. Phase III: final offers of admission and enrollment. Washington: Council of Graduate Schools.
Davidson-Shivers, G., Inpornjivit, K., & Sellers, K. (2004). Using alumni and student databases for evaluation and planning. College Student Journal, 38(4), 510–520.
Delaney, A. M. (2004). Ideas to enhance higher education’s impact on graduates’ lives: alumni recommendations. Tertiary Education and Management, 10(2), 89–105.
Fagen, A. P., & Suedkamp Wells, K. M. (2004). The 2000 national doctoral program survey: an online study of students’ voices. In D. H. Wulff, A. E. Austin, & Associates (Eds.), Paths to the professoriate: strategies for enriching the preparation of future faculty (pp. 74–91). San Francisco: Jossey-Bass.
Farley, K., McKee, M., & Brooks, M. (2011). The effects of student involvement on graduate student satisfaction: a pilot study. Alabama Counseling Association Journal, 37(1), 33–38.
Fu, Y. (2012). The effectiveness of traditional admissions criteria in predicting college and graduate success for American and international students. Doctoral dissertation, University of Arizona.
Gansemer-Topf, A. M., Ross, L. E., & Johnson, R. M. (2006). Graduate and professional student development and student affairs. New Directions for Student Services, 2006(115), 19–30.
Gardner, S. K., & Barnes, B. J. (2007). Graduate student involvement: socialization for the professional role. Journal of College Student Development, 48(4), 369–387.
Golde, C. M. (2000). Should I stay or should I go? Student descriptions of the doctoral attrition process. The Review of Higher Education, 23(2), 199–227.
Hardré, P. L. (2012a). Scalable design principles for TA development: lessons from research, theory, testing and experience. In G. Gorsuch (Ed.), Working theories for teaching assistant and international teaching assistant development (pp. 3–38). Stillwater: NewForums.
Hardré, P. L. (2012b). Teaching assistant development through a fresh lens: a self-determination theory framework. In G. Gorsuch (Ed.), Working theories for teaching assistant and international teaching assistant development (pp. 113–136). Stillwater: NewForums.
Hardré, P. L., & Burris, A. (2011). What contributes to TA development: differential responses to key design features. Instructional Science, 40(1), 93–118.
Hardré, P. L., & Chen, C. H. (2005). A case study analysis of the role of instructional design in the development of teaching expertise. Performance Improvement Quarterly, 18(1), 34–58.
Hardré, P. L., & Chen, C. H. (2006). Teaching assistants learning, students responding: process, products, and perspectives on instructional design. Journal of Graduate Teaching Assistant Development, 10(1), 25–51.
Hardré, P. L., Crowson, H. M., & Xie, K. (2010a). Differential effects of web-based and paper-based administration of questionnaire research instruments in authentic contexts-of-use. Journal of Educational Computing Research, 42(1), 103–133.
Hardré, P. L., Nanny, M., Refai, H., Ling, C., & Slater, J. (2010b). Engineering a dynamic science learning environment for K-12 teachers. Teacher Education Quarterly, 37(2), 157–178.
Hardré, P. L., Beesley, A., Miller, R., & Pace, T. (2011). Faculty motivation for research: across disciplines in research-extensive universities. Journal of the Professoriate, 5(2), 35–69.
Hardré, P. L., Crowson, H. M., & Xie, K. (2012). Examining contexts-of-use for online and paper-based questionnaire instruments. Educational and Psychological Measurement, 72(6), 1015–1038.
Hegarty, N. (2011). Adult learners as graduate students: underlying motivation in completing graduate programs. Journal of Continuing Higher Education, 59(3), 146–151.
Hephner LaBanc, B. (2010). Student affairs graduate assistantships: an empirical study of the perceptions of graduate students’ competence, learning, and professional development. Doctoral dissertation, Northern Illinois University.
Higher Education Research Institute (HERI) (2012). Faculty satisfaction survey. http://www.heri.ucla.edu/index.php. Accessed 15 June 2013
Hyun, J., Quinn, B. C., Madon, T., & Lustig, S. (2006). Needs assessment and utilization of counseling services. Journal of College Student Development, 47(3), 247–266.
Kanan, H. M., & Baker, A. M. (2006). Student satisfaction with an educational administration preparation program: a comparative perspective. Journal of Educational Administration, 44(2), 159–169.
Kenner, C., & Weinerman, J. (2011). Adult learning theory: applications to non-traditional college students. Journal of College Reading and Learning, 41(2), 87–96.
Lipschultz, J. H., & Hilt, M. L. (1999). Graduate program assessment of student satisfaction: a method for merging university and department outcomes. Journal of the Association for Communication Administration, 28(2), 78–86.
Lovitts, B. E. (2001). Leaving the ivory tower: the causes and consequences of departure from doctoral study. Lanham: Rowman & Littlefield.
Messick, S. (1995). Standards of validity and the validity of standards in performance assessment. Educational Measurement: Issues and Practice, 14(4), 5–8.
Nesheim, B. E., Guentzel, M. J., Gansemer-Topf, A. M., Ross, L. E., & Turrentine, C. G. (2006). If you want to know, ask: assessing the needs and experiences of graduate students. New Directions for Student Services, 2006(115), 5–17.
Offstein, E. H., Larson, M. B., McNeill, A. L., & Mwale, H. M. (2004). Are we doing enough for today’s graduate student? The International Journal of Educational Management, 18(6/7), 396–407.
Patton, M. Q. (2012). Essentials of utilization-focused evaluation. Thousand Oaks: Sage.
Schlossberg, N. K., Waters, E. B., & Goodman, J. (1995). Counseling adults in transition: kinking practice with theory (2nd ed.). New York: Spring.
Schram, L. N., & Allendoerfer, M. G. (2012). Graduate student development through the scholarship of teaching and learning. Journal of Scholarship of Teaching and Learning, 12(1), 8–22.
Smallwood, S. (2004). Doctor dropout. Chronicle of Higher Education, 50 (19), A10. Retrieved from: http://chronicle.com/article/Doctor-Dropout/33786
Stone, C., van Horn, C., & Zukin, C. (2012). Chasing the American Dream: recent college graduate and the great recession. New Brunswick: John J. Heldrich Center for Workforce Development.
US Department of Education, National Center for Education Statistics. (2005). Integrated post-secondary education data system, Fall 2004. Washington: US Department of Education.
Vincente, P., & Reis, E. (2010). Using questionnaire design to fight nonresponse bias in web surveys. Social Science Computer Review, 28(2), 251–267.
Weidman, J. C., Twale, D. J., & Stein, E. L. (2001). Socialization of graduate and professional students in higher education: a perilous passage? San Francisco: Jossey-Bass.
Williams-Tolliver, S. D. (2010). Understanding the experiences of women, graduate student stress, and lack of marital/social support: a mixed method inquiry. Doctoral dissertation, Capella University.
Yarbrough, D. B., Shulha, L. M., Hopson, R. K., & Caruthers, F. A. (2011). The program evaluation standards: a guide for evaluators and evaluation users. Los Angeles: Sage.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Hardré, P.L., Hackett, S. Beyond instrumentation: redesigning measures and methods for evaluating the graduate college experience. Educ Asse Eval Acc 27, 223–251 (2015). https://doi.org/10.1007/s11092-014-9201-6
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11092-014-9201-6