Measuring academic research impact: creating a citation profile using the conceptual framework for implementation fidelity as a case study
- First Online:
The “citation score” remains the most commonly-used measure of academic impact, but is also viewed as practically and conceptually limited. The aim of this case study was to test the feasibility of creating a “citation profile” for a single, frequently-cited methods paper, the author’s own publication on the conceptual framework for implementation fidelity. This was a proof-of-concept study that involved an analysis of the citations of a single publication. This analysis involved identifying all citing publications and recording, not only how many times the key paper was cited within each citing publication, but also within which sections of that publication (e.g. Background, Methods, Results etc.). Level of impact could be categorised as high, moderate or low. The key paper had been cited more than 400 times and had a high impact in 25 % of publications based on citation frequency within publications, i.e. the key paper was cited three or more times; and a low impact in 58 % of citing publications, i.e. the key paper was cited just once. There were 41 “high impact” publications based on location of the citations, of which 35 (85 %) were also categorised as high impact by frequency. These results suggest that it is both possible and straightforward to categorise the level of impact of a key paper based on its “citation profile”, i.e., the frequency with which the paper is cited within citing publications, thus adding depth and value to the citation metric.
KeywordsCitations Scientific influence Implementation fidelity Impact
- Carroll, C., Rick, J., Pilgrim, H., Cameron, J., & Hillage, J. (2010). Workplace involvement improves return to work rates among employees with back pain on long-term sick leave: A systematic review of the effectiveness and cost-effectiveness of interventions. Disability and Rehabilitation, 32, 607–621.CrossRefGoogle Scholar
- Chandler, C. (2014). What is the meaning of impact in relation to research and why does it matter? A view from inside academia. In P. Denicolo (Ed.), Achieving impact in research. New York: Sage.Google Scholar
- Hopewell, S., Clarke, M., Stewart, L., & Tierney, J. (2007). Time to publication for results of clinical trials. Cochrane Database of Systematic Reviews, (2), MR000011. doi:10.1002/14651858.MR000011.pub2.
- Last, J. (1995). A dictionary of epidemiology. Oxford: Oxford University Press.Google Scholar
- National Library of Medicine (NLM). (2016). Medical subject headings (MeSH). Accessed April 29, 2016, from https://www.nlm.nih.gov/mesh/.
- R. E. F. (2014). REF 2014. Accessed April 26, 2016, from http://www.ref.ac.uk/.
- Rice, K. (2011). Change over time in implementation fidelity of a child welfare practice model. PhD dissertion, University of Maryland in Baltimore, USA.Google Scholar
- Rothstein, H., & Hopewell, S. (2009). Grey literature. In H. Cooper, L. Hedges, & J. Valentine (Eds.), The handbook of research synthesis and meta-analysis. New York: Sage.Google Scholar
- Wilsdon, J., Allen, L., Belfiore, E., Campbell, P., Curry, S., Hill, S., Jones, R., & Kain, R. (2015). The metric tide. In Report of the independent review of the role of metrics in research assessment and management, HEFCE.Google Scholar