Types of evidence cited in Australian Government publications
Demand on researchers to justify the impact of their work outside academia is increasing. Both increasing research use in policy and measuring current use are multi-faceted problems, though there are many potential benefits to researchers and policymakers alike. This bibliometric study aimed to gain insight into the research and reference practices of Australian policymakers, and investigate how this approach compares to previous interview and survey studies. We analysed 4649 references from 80 government publications from eight departments from 2010 to 2017, including references to 1836 articles from peer-reviewed journals, noting each author, title, year, parent publication, source type and access level. The number and type of evidence sourced varied per publication, with the most common sources being peer-reviewed journal articles, federal government reports, and Australian business information. This differs from previous large-scale qualitative studies which found policymakers are most likely to speak directly to colleagues for information, and far less inclined to seek out academic research. The study also found a possible increased chance for academic research to be cited if it was open access. Despite criticisms of citation analysis, at least in the field of research utilisation we cannot solely rely on interview or survey data, as cited evidence use differs from reported evidence use. Both the characteristics of evidence sources in policy and the effect of open access publishing on research use in policy are clearly worth investigating further, particularly longitudinally, which would require increased accessibility of government publications.
KeywordsBibliometrics Government Policy Impact assessment
- Analysis & Policy Observatory. (2017). Available at: http://apo.org.au/.
- Archambault, E. (2013). The tipping point: Open access comes of age. In ISSI 2013 Proceedings of 14th International Society of Scientometrics and Informetrics Conference (pp. 1165–1680).Google Scholar
- Archambault, E., Amyot, D., Deschamps, P., Nicol, A., Rebout, L., & Roberge, G. (2013). Proportion of open access peer-reviewed papers at the European and World Levels—2004–2011.Google Scholar
- Australian Public Service Commission. (2012). Appendix 2—APS agencies. Available at: http://www.apsc.gov.au/about-the-apsc/parliamentary/state-of-the-service/2011-12-sosr/appendix-2-aps-agencies.
- Clarivate Analytics. (2017). Journal List, Science Citation Index. Available at: http://ip-science.thomsonreuters.com/cgi-bin/jrnlst/jlresults.cgi?PC=K.
- Curtin University. (2017). Measure research impact and quality. Available at: http://libguides.library.curtin.edu.au/research-impact.
- Deakin University. (2016). Demonstrate your research impact. Available at: http://www.deakin.edu.au/library/research/demonstrate-your-research-impact.
- Department of Education, Training & Youth Affairs. (2000). The impact of educational research. Available at: http://hdl.voced.edu.au/10707/45520.
- Department of Industry Innovation and Science. (2017). Measuring impact and engagement of university research. Available at: http://www.innovation.gov.au/page/measuring-impact-and-engagement-university-research.
- Elsevier. (2017). Content. Available at: https://www.elsevier.com/solutions/scopus/content.
- Gargouri, Y., Larivière, V., Gingras, Y., & Harnad, S. (2012). Green and gold open access percentages and growth, by discipline. 17th international conference on science and technology indicators. Montreal.Google Scholar
- King’s College London and Digital Science. (2015). The nature, scale and beneficiaries of research impact: An initial analysis of Research Excellence Framework (REF) 2014 impact case studies. Bristol, United Kingdom: HEFCE. Available online: http://www.hefce.ac.uk/pubs/rereports/Year/2015/analysisREFimpact/.
- McCabe, M. J., Snyder, C. M. (2014). Identifying the effect of open access on citations using a panel of science journals. Economic Inquiry. doi: 10.1111/ecin.12064.
- Miles, M. B., & Huberman, A. M. (1984). Qualitative data analysis: A sourcebook for new methods. Newbury Park: Sage.Google Scholar
- Saha, S. (2003). Impact factor: A valid measure of journal quality? Journal of the Medical Library Association, 91(1), 42–46.Google Scholar
- Universities Australia. (2016). Research Engagement and Impact Assessment Consultation Paper. Available online: https://www.universitiesaustralia.edu.au/Media-and-Events/submissions-and-reports/Research-Engagement-and-Impact-.