I spent some of the weekend clearing out some of my grown-up kids’ books and came across some of my son’s “Ancient Egypt” collection. One front cover grabbed my attention. It showed the god Anubis weighing the heart of a dead Pharaoh against that of a feather, with Thoth recording the result. The heart (of the good) weighing less than a feather allowed the dead to proceed to the afterlife. Now that’s what I call metricating an individual’s lifetime achievement! And so back to the blog and the point of the anecdote. As you will gather, my mind is currently occupied with accounting and assessment. I’ve spent the couple of weeks since my last blog reviewing and assessing a whole heap of academia, both for our own School’s two units of REF assessment and as an external reviewer on a two-day assessment visit with BBSRC.
It has prompted me to think; Is all of this assessment and the attempts at measurement by our funders a good or a bad thing? I think it’s fair to say that as with most aspects of modern life, there are positives and negatives. When I embarked on my own scientific career in the 1980s assessment of research was much more rudimentary. Looking back, I can see that there was limited understanding of what “good outcomes” and impact even looked like. In the main we were expected to list and describe what we had done in weighty reports that often gave little indication of the value of all of the work carried out. Expectations have certainly changed. We are required to tick more boxes and to demonstrate clearly that objectives have been met, along with excellence and impact achieved and I know that for many academics that this weighing of your achievements is enormously frustrating. Some feel that there is no longer any creative “wriggle room” from proposal to assessment of results. At the other end of the spectrum some colleagues delight in the game of metrics and the associated game playing involved.
Where ever you sit on the spectrum, from bean counting to game player, we live at a time when everyone really does need to read the instruction manual. That applies whether you’re trying to put together your Billy Bookcase from Ikea or write a grant proposal. If you miss the key point at the beginning then your outcomes are not going to be successful and your bookcase is in danger of collapsing. I’m sure you have all told students over the years to “read the question!” Without that key step, without knowing what the examiner is looking for and which box needs to be ticked, there is little chance of getting the right answer. Indeed, you may find you are trying to answer the wrong question.
The reality is that everything we do is being assessed all the time, both internally and externally, whether we are conscious of this or not. Just one example – your My Impact information can be seen and information collected from it at any time by the university. That is why you need to ensure it is up to date. Expectations and requirements change constantly, and just keeping up with this can feel like hard work, but it is essential. Fifteen years ago we were being measured on the numbers of papers published but now what matters is the impact of a paper – how many times has it been cited? A paper that has no citations may actually be a negative asset. We have to understand the ways in which expectations change, and respond. Otherwise you may simply continue to answer the wrong question.
It’s a challenge, not only for this university but for all – I know that from my work as an external assessor. But it will be the organisations that “read the instructions” which survive and thrive. This applies throughout the research process. When you are putting together a grant proposal it must address relevant problems, and when carrying out research and assembling information, we have to be addressing the real objectives in the grant proposal and demonstrating this unequivocally. Afterwards it’s too late to go back and find that lost screw that should have been inserted at the beginning. Increasingly we are being managed and assessed by people who are not academics but who operate in a business-oriented environment. Their measurements will be more ruthless and exacting than the processes we have experienced to date.
We don’t need to be too intimidated by this, but we do need to understand the world in which we are operating, the criteria against which we are being measured, and ensure we respond appropriately. Small and obvious actions such as keeping your own records up to date do help. Just as important is to understand that all the metrics being collected aren’t just measuring the School or Newcastle University, they are measuring you and me and everyone who works in this organisation, and we will all be judged against the results. It is a collective process so we need to make a collective response.