This week we have a guest blog from Prof Mark Reed, a leading expert on research impact and Professor of Socio-Technical Innovation at Newcastle University.

Mark is a recognized international expert in impact research with >100 publications that have been cited over 8000 times. He has won awards for the impact of his work as a research Professor and Research Manager for an international charity. His work has been funded by ESRC, NERC, AHRC and BBSRC, and he regularly collaborates and publishes with scholars ranging from the arts and humanities to physical sciences. He has reviewed research and sat on funding panels for the UK Research Councils (including MRC, BBSRC, ESRC, NERC), EU Horizon 2020 and many national and international research funders.

Mark regularly provides training and advice to Universities, research funders, NGOs and policy-makers internationally, and regularly works with business.

For more information check out the website http://www.fasttrackimpact.com/

---

If your research has had beneficial impacts in the real world, then there’s a good chance you’ll want (or have) to shout about it. This guide will distill the best advice that is currently available about how to write research impact case studies that have real, erm… impact.

 This guide will help you, if you are:

• A researcher writing about the impact of your own work for your funders or the media

• Writing about research from the marketing department of a University or other research organization

• Writing or helping academics write case studies for a Government assessment of research, such as the UK’s Research Excellence Framework (REF2020)

 8 points for writing an effective impact case study

1.  Create a coherent narrative that explains clearly the relationship between the underpinning research and the impact. It may be useful to briefly explain what was original or distinctive about the research that contributed to the impacts. Consider starting from the perspective of the beneficiaries – how did they benefit and why is that benefit so important to them? Be as specific as possible if you want the link between the research and the impact to be credible, including specific details about the names of researchers, their position and dates and locations of the research activity. Choose a strong headline for your case study and ensure your summary starts with a powerful opening sentence that summarises the case study and draws the reader in
2.  Be as clear as possible about exactly what the impact was, adding some sort of precise quantification with numbers wherever possible. Quantitative data and indicators need to be meaningful and contextualised to clearly support the case being made, not used as a substitute for a clear narrative. Avoid generalised or exaggerated statements about your impact.
3.  Clearly identify specifically who has benefited from the work or which groups/organisations have changed something as a result of the research. Bear in mind that this may include ‘intermediary’ organisations as well as your intended ‘end users’ or audiences.
4.  Be concise. A concise case study that pulls out the key points easily for readers has far greater impact than one that is dense and rambling. The UK’s Research Excellence Framework gives people strict page limits, but if you don’t have this constraint, then you could consider imposing some sort of constraint on your case studies.
5.  Keep your language simple and direct. If possible, get advice from a science writer or communications specialist, but try and avoid introducing inaccuracies. Remember your audience, and that your audience is probably not other academics. Readers should not have to have in-depth expert or prior knowledge to be able to understand your case study. It is essential that impact case studies avoid academic jargon, so that they are accessible to all your readers.
6. If you are writing multiple case studies, identify key features of best practice and be consistent about all your case studies covering these aspects. You might for example want to consider a particular list of sub-headings for each case study to follow. However, avoid making all your case studies sound too similar, as though they are all written by the same person, if you want them to also feel authentic.
7. Related to this, make sure you provide detailed, specific and independent evidence to support every claim you make. All material required to make a judgement about the impact needs to be contained within the case study. Avoid anecdotal evidence or evidence that might perceived as such. If possible, link to published evidence that demonstrates the impact of your research. It is possible to commission work to demonstrate impacts yourself, but you will need to consider carefully how this will be published, to ensure that it is perceived to be sufficiently independent and credible. Rather than just listing sources of evidence, explain how each source of evidence supports a specific aspect of the impact that has been claimed.
8.  Bring your case studies to life with quotes that illustrate the impact with greater resonance than could otherwise be done with formal language. If these quotes are from people with high profile and relevant job titles, then this adds significant credibility to your case study, as well as some lived experience. Finding quotes from people years after an impact has occurred can be tricky however, so it is recommended to collect these from people as the impact unfolds, and if nothing else, keep track of contact details so that people can be easily contacted later. Bear in mind that key people who could attest to the impact of your work may retire or pass away before you need to write your case study. Usually it is best to aim for a good balance of quantitative data and quotes to support impacts. However, in case studies where there is no quantitative data to corroborate an impact (e.g. a policy which has been developed but not yet implemented), quotes may be the primary source of corroboration to evidence your impact.

 What do we know about impact in REF2020?

 We don’t yet know the rules and weighting that will be associated with the assessment of research impact in HEFCE’s Research Excellence Framework in 2020 (REF2020), but we do know four important things:

• Impact was worth 20% of an institution’s score in 2014, and most commentators expect this to either remain the same or increase slightly to 25% for REF2020

• Some impact case studies submitted to REF2014 that have continued to generate further impacts during the current REF period are likely to be eligible for re-submission to REF2020. HEFCE are currently consulting on this, to try and get the balance right between incentivising Higher Education Institutes to continue developing and building on existing impacts, whilst having sufficient incentives to develop new impacts

• It is likely that the eligibility criteria and guidelines for submitting impact case studies in 2020 will be broadly similar to the 2014 assessment, although these are currently under consultation too

• Assessment criteria may be expanded (e.g. including impacts on public discourse and attitudes) or become more fine-grained (e.g. considering different elements of reach such as number versus the types or geographical spread of people adopting a new technology or behavior) in response to feedback from REF2014 panels, but they are still likely to address aspects of significance and reach.

 HEFCE commissioned an assessment of the potential for metrics to inform REF2020, which reported in early 2015. It concluded that metrics could not replace case study narratives for the assessment of impact in 2020. Research commissioned by HEFCE about the impact case studies submitted to REF2014 somewhat uncritically accepted the breadth and depth of impacts captured in the assessment, implying that a similar approach in 2020 could yield similarly authoritative “evidence” of impact. This approach to impact may be criticized as biased towards a narrow set of self-reinforcing perceptions of what constitutes valid and significant impacts, created primarily between different sectors of the research community, rather than in the eyes of beneficiaries. However, although research has been proposed to address these issues, if funded, it will not report until after procedures have been set in place for REF2020. It is therefore reasonable to assume that REF2020 impact case studies will be broadly similar to REF2014.

 This is what Universities were required to submit in 2014, which is likely to be similar in 2020:

• Each Unit of Assessment (UOA) had to submit an impact statement (REF3a, Figure 1)

• Each UOA also had to submit between 2 and 7 impact case studies, depending on the number of academics submitted in the Research Outputs section of the assessment (REF3b, Figure 1)

• Underpinning research had to be “excellent” (2* quality or above) and must have been conducted by University members of staff to be eligible.

Underpinning research could include published reports (e.g. final reports to funders) where it was clear that they were of 2* or above quality, but it is highly risky to submit a case study with underpinning research that has not undergone peer-review in an international journal, in case panel members do not deem it to be of sufficient quality for the impact to be eligible. If you are submitting grey literature as the underpinning research, provide copies or links to any assessments that may have been made by the funders as a form of peer-review, and if this isn’t possible then make sure you get it reviewed by a number of experts in the field to ensure the panel is likely to judge it as 2* or above in quality. Depending on how long there is before REF2020, you may be able to turn a report from the grey literature into a peer-reviewed journal article and submit both as underpinning evidence (the original report to show that the research preceded the impact and the peer-reviewed version to demonstrate 2* quality). The underpinning research has to be conducted (not just published) at the submitting institution, so members of staff appointed between now and the 2020 assessment will need to be able to conduct significant new research to further underpin and extend existing impacts if these are to be claimed by their new institution.

 Research has to have made a distinct and material contribution to any impacts that are claimed, such that the impact would not have occurred or would have been significantly reduced without the research. If this link cannot be adequately established, then the case study may be graded as “unclassified”. Research from a number of different individuals, UOAs and institutions may have contributed towards a particular impact, and as a result, there are a number of REF2014 case studies that cover the same impact in different ways, based on the specific impacts related to the research conducted by each of the different submitting institutions. If you are in this position however, it is worth considering whether the impacts that you are able to claim linked to your work will be as far-reaching and significant as those claimed by a competing institution who is submitting to the same UOA. Panels assessing the same impact submitted by different institutions will inevitably compare case studies, and you want to make sure that if this happens, your grade doesn’t suffer as a result.

FIGURE 1: OUTLINE OF INFORMATION REQUIRED FOR IMPACT CASE STUDY AND IMPACT TEMPLATE DOCUMENTS

 Is your impact significant and far-reaching enough for REF2020?

 In 2014, impact case studies were evaluated against two criteria:

• Reach is the extent and diversity of the communities, environments, individuals, organisations or any other beneficiaries that may have been impacted by the research

• Significance is the degree to which the impact has enriched, influence, informed or changed policies, practices, products, opportunities or perceptions of individuals, communities or organisations.

This led to a star rating for each case study from 1-4 stars (Table 1). Assessing the significance and reach of research impact is extremely challenging. Indeed, about a third of REF2014 panel members did not feel that either of these criteria helped them to fairly or reliably assess impact. One of the key problems is the overlap between concepts of significance and reach, as they are highly connected and correlated in most cases. As a result, most panels evaluated both of these criteria together rather than separately. So rather than worrying too much about the difference between these two concepts, the key thing is to make sure that your impact couldn’t be described as insignificant or lacking reach, and to identify evidence that can strengthen claims to either significance or reach (ideally both).

 Impact was defined as including but not being limited to “an effect on, change or benefit to:

• The activity, attitude, awareness, behaviour, capacity, opportunity, performance, policy, practice, process or understanding

• of an audience, beneficiary, community, constituency, organisation or individuals

• in any geographic location whether locally, regionally, nationally or internationally.

This could include “negative impacts” where something harmful, risky, costly or in some other way negative is stopped or prevented as a result of research. Impacts on research or the advancement of academic knowledge within the higher education sector were excluded in REF2014 and are expected to be excluded again in REF2020, to avoid duplication with the assessment of research outputs. Impacts on students, teaching and other activities in HEIs were excluded unless the reach of those impacts extended significantly beyond the submitting institution.

Geographically local impacts could in theory be considered both significant and far-reaching, according to HEFCE guidelines. However in reality, a number of panels reported that it was more difficult for local impacts to score highly.

 These impacts have to have occurred within the census period (in this case from 2015-2020) to be eligible. This means that many impacts that occurred during the previous REF period and were submitted in 2014 will not be eligible in 2020, as the impacts have already occurred. There is a grey area for cumulative impacts, which HEFCE is expected to clarify when guidance for REF2020 is published. If additional impacts have occurred from an impact claimed in 2014, then the additional impacts may be claimed in 2020 e.g. additional lives saved by a drug or safety device as it is licenced in new countries.

Although you would normally have a good idea of the impact your work has had, if you are submitting a case study based on research by someone who has now left the University, it may be necessary to do quite a bit of your own research to capture all the relevant impacts. Normally this would start with a simple Internet search, but this will often extend to interviews with research users, taking a snowball sampling approach, where you ask each interviewee to recommend others you can interview who may have been impacted by the research.

 Some tips for quantifying impact:

• Google key statistics or phrases from your work to see where they have been used in documents that have been put on the web

• Did people pay for advice or did consultancy work arise from the research, and can you track the flows of money?

• What is the readership of a particular newspaper or online media source and how might this translate into money via advertising revenues linked to page views and circulation?

• In public engagement case studies, can you show evidence of your research being widely disseminated and talked about or used? Did you blog your research and how many people read the blogs? Did you tweet it? Did you get any press or broadcast coverage? Did any trade journals or close-to-policy journals or websites cover the work?

• Did you get letters of support from external bodies before you submitted the grant that funded your research? If so, can you go back to these organisations to see how they used your work? Who did they pass your work onto or talk to about your research, and at what decision-making level? Can these people point you to evidence in documents etc. that your work has been used? If not, can they provide testimonials about the impact of your work?

• Did your engagement with stakeholder continue after the project? If so, what further collaborations arose from this engagement that wouldn’t have happened without the original research? Did collaborators use the methods from the research in their own professional practice after the research ended, and what was the effect of those changes in their practice?

TABLE 1: LEVEL DEFINITIONS USED FOR ASSESSING IMPACT IN REF2014

 Box 1 provides a number of different definitions and types of impact. Although most impact case studies tend to focus only on instrumental impacts, for some topics, other types of impact may be important, and may lead you to more concrete impacts that you can write about in a case study.