, Volume 109, Issue 2, pp 1329–1340 | Cite as

Measuring academic research impact: creating a citation profile using the conceptual framework for implementation fidelity as a case study



The “citation score” remains the most commonly-used measure of academic impact, but is also viewed as practically and conceptually limited. The aim of this case study was to test the feasibility of creating a “citation profile” for a single, frequently-cited methods paper, the author’s own publication on the conceptual framework for implementation fidelity. This was a proof-of-concept study that involved an analysis of the citations of a single publication. This analysis involved identifying all citing publications and recording, not only how many times the key paper was cited within each citing publication, but also within which sections of that publication (e.g. Background, Methods, Results etc.). Level of impact could be categorised as high, moderate or low. The key paper had been cited more than 400 times and had a high impact in 25 % of publications based on citation frequency within publications, i.e. the key paper was cited three or more times; and a low impact in 58 % of citing publications, i.e. the key paper was cited just once. There were 41 “high impact” publications based on location of the citations, of which 35 (85 %) were also categorised as high impact by frequency. These results suggest that it is both possible and straightforward to categorise the level of impact of a key paper based on its “citation profile”, i.e., the frequency with which the paper is cited within citing publications, thus adding depth and value to the citation metric.


Citations Scientific influence Implementation fidelity Impact 



This project received a small amount of departmental financial support to source a number of inter-library loans. The author would like to thank Andrew Tattersall for some key references.

Supplementary material

11192_2016_2085_MOESM1_ESM.xlsx (42 kb)
Supplementary material 1 (XLSX 42 kb)


  1. Bjork, B.-C., & Solomon, D. (2015). Article processing charges in OA journals: Relationship between price and quality. Scientometrics, 103, 373–385.CrossRefGoogle Scholar
  2. Booth, A. (2008). Implementing EBLIP: If it works in Edmonton will it work in Newcastle? Health Information and Libraries Journal, 25, 154–157.MathSciNetCrossRefGoogle Scholar
  3. Carroll, C., Patterson, M., Wood, S., Booth, A., Rick, J., & Balain, S. (2007). A conceptual framework for implementation fidelity. Implementation Science, 2, 40.CrossRefGoogle Scholar
  4. Carroll, C., Rick, J., Pilgrim, H., Cameron, J., & Hillage, J. (2010). Workplace involvement improves return to work rates among employees with back pain on long-term sick leave: A systematic review of the effectiveness and cost-effectiveness of interventions. Disability and Rehabilitation, 32, 607–621.CrossRefGoogle Scholar
  5. Chandler, C. (2014). What is the meaning of impact in relation to research and why does it matter? A view from inside academia. In P. Denicolo (Ed.), Achieving impact in research. New York: Sage.Google Scholar
  6. Fairclough, R., & Thelwall, M. (2015). National research impact indicators from Mendeley readers. Journal of Informetrics, 9, 845–859.CrossRefGoogle Scholar
  7. Field, B., Booth, A., Illott, I., & Gerrish, K. (2014). Using the knowledge to action framework in practice: A citation analysis and systematic review. Implementation Science, 9, 172.CrossRefGoogle Scholar
  8. Harzing, A., & Van Der Wal, R. (2008). Google Scholar as a new source for citation analysis. Ethics in Science and Enviornmental Politics, 8, 61–73.CrossRefGoogle Scholar
  9. Hasson, H., Blomberg, S., & Duner, A. (2012). Fidelity and moderating factors in complex interventions: A case study of a continuum of care program for frail elderly people in health and social care. Implementation Science, 7, 23.CrossRefGoogle Scholar
  10. Hopewell, S., Clarke, M., Stewart, L., & Tierney, J. (2007). Time to publication for results of clinical trials. Cochrane Database of Systematic Reviews, (2), MR000011. doi: 10.1002/14651858.MR000011.pub2.
  11. Last, J. (1995). A dictionary of epidemiology. Oxford: Oxford University Press.Google Scholar
  12. Leaviss, J., & Uttley, L. (2015). Psychotherapeutic benefits of compassion-focused therapy: An early systematic review. Psychological Medicine, 45, 927–945.CrossRefGoogle Scholar
  13. Lortie, C., Aarssen, L., Budden, A., & Leimu, R. (2013). Do citations and impact factors relate to the real numbers in publications? A case study of citation rates, impact, and effect sizes in ecology and evolutionary biology. Scientometrics, 94, 675–682.CrossRefGoogle Scholar
  14. National Library of Medicine (NLM). (2016). Medical subject headings (MeSH). Accessed April 29, 2016, from
  15. Ravallion, M., & Wagstaff, A. (2011). On measuring scholarly influence by citations. Scientometrics, 88, 321–337.CrossRefGoogle Scholar
  16. R. E. F. (2014). REF 2014. Accessed April 26, 2016, from
  17. Rice, K. (2011). Change over time in implementation fidelity of a child welfare practice model. PhD dissertion, University of Maryland in Baltimore, USA.Google Scholar
  18. Rothstein, H., & Hopewell, S. (2009). Grey literature. In H. Cooper, L. Hedges, & J. Valentine (Eds.), The handbook of research synthesis and meta-analysis. New York: Sage.Google Scholar
  19. Royle, P., Kandala, N.-B., Barnard, K., & Waugh, N. (2013). Bibliometrics of systematic reviews: Analysis of citation rates and journal impact factors. Systematic Reviews, 2, 74.CrossRefGoogle Scholar
  20. Sarli, C., Dubinsky, E., & Holmes, K. (2010). Beyond citation analysis: A model for assessment of research impact. Journal of the Medical Library Association, 98, 17–23.CrossRefGoogle Scholar
  21. Van Leeuwen, T., Moed, H., Tijssen, R., Visser, N., & Cvan Raan, A. (2001). Language biases in the coverage of the Science Citation Index and its consequences for international comparisons of national research performance. Scientometrics, 51, 335–346.CrossRefGoogle Scholar
  22. Wilsdon, J., Allen, L., Belfiore, E., Campbell, P., Curry, S., Hill, S., Jones, R., & Kain, R. (2015). The metric tide. In Report of the independent review of the role of metrics in research assessment and management, HEFCE.Google Scholar

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2016

Authors and Affiliations

  1. 1.Health Economics and Decision Science (HEDS), School of Health and Related Research (ScHARR)University of SheffieldSheffieldUK

Personalised recommendations