I am chairing a University working group concerned with identifying the key performance indicators we should be producing and reviewing at programme level. This is in many ways quite a challenge for a number of reasons I shall explore in this post. I should add that I did volunteer for the role because I find it so interesting.

I shall illustrate the issues through an example, retention data, which is currently the subject of review by our internal audit function. When interviewed for the audit, I had to admit with some embarrassment that I did not know how the calculation is done, couldn’t understand the standard programme data reports on retention and had some awareness that there was an official institutionally adjusted benchmark, whilst not knowing what it was. This is partly because retention has not been a great problem at Newcastle; we haven’t worried too much about monitoring it, because we didn’t think much was at stake for us. The advent of £9,000 fees has sharpened the profile of the issue though, as drop out between confirmation in August and entry in September has emerged as an issue, never mind drop out at later stages.

At a practical level we have been doing more and more to support students and to maximise retention through a stronger emphasis on good induction, the introduction of clearer requirements for personal tutorial meetings in the first semester and through introducing student mentoring schemes. In some individual academic units much good work is done in indentifying students at risk and supporting them. My focus here is on the question of key performance indicators.

The principal issues which I see are:

a) how should we measure retention, or drop-out, its converse? Are we just looking at how many of the starters made it through to Stage 2 or how many completed the whole programme? What’s our starting date? It could be confirmation in August, registration or even the magical 1 December when data returns are made. If we focus on progression to Stage 2 of the programme, what about students who transfer to another degree course? They are not lost to the University and may be happier in their new degree programme. What about students whose progress is delayed by academic failure or personal problems, but eventually make it? How should we account for them? Do they represent a failure to retain or an eventual success? What seems a relatively simple issue of measurement now looks much more complex and might lead us to use a number of different indicators of retention. It is also clearly a very difficult thing to measure technically because it requires a system which can track individual students from start to finish and then amalgamate the data.

b) even within an institution it is clear that retention has a subject dimension. Hard technical disciplines seem to lose more students than other disciplines, perhaps because more fail to proceed from Stage 1, perhaps because more are frightened off by the demands of the discipline. Key performance indicators mean nothing without something to compare them to, whether this be historical trend data, targets or subject benchmarks. So what is the relevant comparison? How should we know how well we’re doing?

c) retention also varies between institutions largely because of the quality of the intake. Highly qualified entrants are less likely to drop out that those less well qualified. Oxford and Cambridge don’t have a retention problem. Institutional benchmarks will of course take such factors into account, but cross-sector comparisons are clearly tricky.

d) the most important issue though concerns whether retention is something we should be monitoring at all, or at least whether monitoring it might have unintended consequences. We need to remember that key performance indicators are not neutral technical measures, but measures which drive (and are designed to drive) behaviour. We have seen numerous examples in the NHS of how government targets drive behaviour and not always in beneficial ways. Retention in principle sounds desirable. It is clearly wasteful if students don’t complete their programmes; the HEI concerned will of course lose money. However, in some cases it will be in the student’s best interests to leave the programme because they are unhappy, are ill suited to it or would prefer to be somewhere else. In other cases students may be successfully supported through a wobble to eventual success. Issues about standards also come into play. Whilst it’s clear that we’d want the overwhelming majority of any cohort to be successful, there still has to be a minimum standard of achievement and it is probable that some students won’t make this, either through lack of effort or through lack of ability.

e) retention or its converse, drop-out, is also linked to efforts to widen participation and to mission. HEIs which make great efforts to widen participation take more risks with their applicants and have a wider recruitment funnel. For example the Open University has a higher drop-out rate because it is so open. Continental European universities are often regarded in the UK with disapproval because of their huge drop out rates in the first year which appears so wasteful and inefficient. However, it reflects the openness of HE to those with school leaving qualifications. Is it wrong to give so many the chance to experience HE, even when you know a high proportion won’t make it to later stages? An emphasis on retention could easily discourage risk-taking in admissions and initiatives which widen access to HE.

In summary this one example illustrates:

a) the difficulties of measurement

b) the difficulties of setting appropriate benchmarks for both subjects and institutions

c) the potential impact on practice of performance indicators which may achieve one goal (higher retention), but only at the expense of another (poorer access).

d) that performance indicators can encourage an overly simplistic approach (e.g. assuming that dropping out is always undesirable) which is heavily influenced by financial considerations.

Last week we had a very successful confirmation and clearing week and came close to reaching the very demanding targets set by the University. This is a considerable achievement which shows the dedication of my colleagues in academic schools, to whom I am very grateful.

Here are a few personal thoughts on the process and the implications for the future.

I hope our relative success (we only know anecdotally how others have done) doesn’t blind us to the inherently volatile nature of undergraduate recruitment in this brave new world. We have certainly had to work much harder to achieve this, whereas in the past recruitment just tended to happen. Also the pattern of recruitment is much less predictable and volatile; even now changes are happening to the numbers. Although we have some very strong areas which have exceeded target, we have others which have fallen short. Over time the pattern of provision is likely to shift with all the consequences which are likely to follow from that, doubtless exactly what David Willets wants to see, with provision shifting to reflect demand rather than supply. In many ways this makes perfect sense, but we do need to remember that universities have heavy investments in fixed costs which cannot be reallocated quickly.  We shall need to be quicker to add resources in the successful areas so that the student experience does not suffer, but it is difficult to match resources to demand when the latter is volatile.

These changes require us to adapt in a number of ways.

1. First of all we need to accept that we can no longer predict recruitment levels with pinpoint accuracy, as some still seem to expect. Frankly undergraduate recruitment has always been a tricky business and it’s a miracle we have been able to manage it so well in the past, but in the new regime it’s impossible,
2. The introduction of an open market for the most able students (with ABB or better) has been a game-changer. It provides us with the opportunity to recruit more of the most able students, as there are no longer any limits on recruitment. However, it is a zero sum game; HEIs are all in competition with each other for the same pool of students. So if we have been successful and so has the rest of the Russell Group (so I am told), it must be at someone else’s expense. Competition will become intense. The supply of ABB+ students is also vulnerable to A level reform. It is no longer fashionable to boast about ever improving standards; rather those in power are keener to show rigour. Clearly there may be arguments in favour of this, but it is hard on the students subject to these recalibrations of standards.
3. The flip side for ourselves is that our core quota (the number of students we can recruit regardless of grade) is relatively small. This can make it difficult to meet our widening participation targets or to recruit EU students (almost certainly a deliberate government policy), but it also makes it harder to recruit students in areas where demand is weaker or where selection is based on artistic or musical ability, rather than A level grades. Government insists that we have enough core quota to meet all reasonable demands, but that’s not how it feels. It is also tricky that Government only tells us what core quota we have at the end of the recruitment cycle. We never seem to have quite enough core quota and allocating it in advance to the areas which most need it is also very challenging and would require perfect foresight to do optimally.
4. We shall also have to devote much more resource to undergraduate recruitment at school and institutional levels and this has started to happen. The role of admissions tutor assumes vastly more importance than it once had and is shifting from being a gatekeeper (we used the word “selector”) to being a strategic manager of recruitment. It also needs to shift from being a lone role to a collectively owned enterprise and in the most successful academic units this process is already visible.
5. We are also having to face up to the fact that in the short to medium term our estate constrains the numbers we can accept, and even where there are not hard physical constraints we have to consider how much growth we can cope with how quickly.

A confession – as you can probably tell, I haven’t blogged before and am not used to the medium. My hope in doing this is to be more open with colleagues and others about the challenges we face and to improve communication through using more informal channels. I shall try to keep this up in future weeks.