Towards societal impact: What methods do researchers use to track societal impact?
This is the third blog in a series (catch-up on Part 1 and Part 2) exploring the results of a survey with more than 9,000 researchers examining societal relevance, as part of a project ϳԹ is undertaking with the Association of Universities in the Netherlands (VSNU). Here we explore how researchers track their societal impact, or in other words, their reach beyond academia.
Written by Mithu Lucraft, Director, Outreach and Open Research
The vast majority of researchers are trying to track the societal impact of their research. Close to a quarter of respondents (23%) to our 2019 survey said they always attempt to track the societal impact of their published work, compared to just 8% who said that they never try to track impact.
Societal impact is, without a doubt, complex to understand and therefore to track. We anticipate that at least some respondents to our survey may not have fully understood the term societal impact, and that researchers may include academic impact as well when referring to the impact they sought to achieve or what they monitor. What we have seen in the first and second blog in this series is that where research relates to the UN’s Sustainable Development Goals, achieving some kind of impact outside of academia is seen as very important to researchers. However what we have also seen is that the top priority for the majority of researchers is to reach (be read and cited by) other researchers within the same subject field. This is their top motivation, and their primary target audience.
When we turn our attention to measuring societal impact, it is therefore not that surprising to see that researchers predominantly use academic impact measures to track societal impact. Nearly three quarters of respondents track citations, and half refer to downloads as measures of societal impact. As noted above, it is likely that at least some respondents did not differentiate between academic impact and impact beyond academia, however we also suspect that for many academics, citations and downloads are consistent, recognised metrics that are easy to access and are generally understood across all disciplines. Citations were perceived as being the most effective method of tracking impact by respondents (36%).
Altmetrics and the future of measuring societal impact
Broader measures of societal impact may be harder to track across disciplines. As we noted in our earlier blogs, beyond the shared interest in reaching other academics, the target audience for researchers beyond academia varies by the subject discipline, the target SDG and intended impact of the research. For example, a social scientist was most likely to be targeting policy makers and the general public with the intended impact of influencing government policy and public opinion, whereas a researcher in engineering or physics was more likely targeting commercial businesses and industry with the intention of developing commercial applications or economic benefits.
A range of other measures across these desired outcomes is therefore required. In our survey, we saw approximately a third of respondents using citations or references in policy documents as a measure of impact (a good indication of reaching policy makers). Over a quarter were using altmetric data, such as social media mentions, to track their impact.
Interestingly “other qualitative methods” was the least popular method of tracking impact. The researchers who undertook the analysis suggest how important the use of case studies is in evaluating societal impact, but also their work underlines the complexity of doing so.
Researchers want to do more to measure societal impact
Over three quarters (76%) of respondents either ‘somewhat agreed’ or ‘strongly agreed’ that measuring societal impact was something they should do more of, with similar segments of researchers feeling more strongly about this than they did with impact generally (India, younger researchers, medicine and social sciences). Only 7% disagreed.
Whilst lack of time is cited as the main barrier to doing more measurement (45%), lack of a comprehensive approach or methodology is also raised by many researchers (43%). As noted above, impact measurement is a complex topic. 2020 marks five years on from the publication of the and the , both of which stress the need for better measurement of research impact. Funders and researchers across the world are increasingly looking to responsible metrics and better indicators, highlighting the need to include a combination of quantitative and qualitative measures, and to take a broad long term view of impact. But there is still some way to go: mapping out impact pathways is hard, and for researchers working in SDGs this is made even harder by the challenges in mapping research against an SDG goal where these are broad and interdisciplinary.
Our project with VSNU is starting to tackle this by creating relevancy mapping for content against the SDGs. This enables researchers to better identify work relating to a specific SDG, and to also see where their work may be relevant to help tackle key global challenges beyond their own research community. Our project will also start to look towards best practices for researchers in maximising their societal relevance. In our next blog we will turn to the availability of support for researchers, focusing on a project we undertook to review support across the Netherlands.
Click here to read more about the project and find further slides and raw data from the survey.
At the heart of this agenda are the 17 Sustainable Development Goals (SDGs). Supporting researchers to achieve that reach is therefore a key part of ϳԹ’s SDG Programme, aiming to connect researchers who are tackling some of these global grand challenges with the practitioners who can build on these insights.
Mithu Lucraft has worked in academic publishing since 2004. A passion for storytelling combined with a lasting commitment to scholarly communications has led her through a variety of Marketing and Communications roles, including at Oxford University Press, Sage Publishing and Palgrave Macmillan. At ϳԹ she is responsible for promoting open books and research data services; institutional engagement with open research; as well as wider researcher content engagement strategy.
