Meta Research – Considering Research and Researcher Development

20 November 202220 November 2022

Researcher Wellbeing: what are we willing to change?

A study conducted in the mid 2010’s at Ghent University in Belgium provided some hard evidence (1) for what a fair few of us had suspected for many years. That the wellbeing and mental health of PhD students, and most probably all Early Career Researchers, is in a quite perilous state.

Researchers love to research so a series of subsequent surveys, questionnaires, interviews and reports were funded and actioned (2). The results suggested what we might have expected that the PhD students at Ghent were by no means an outlier but were quite typical. With larger surveys and reviews indicating around one in four research students disclosing issues with their mental health (3). A number that is far greater than other groups of a comparable age. What is probably most worrying within those statistics is the pressure not to disclose any issues due to the potential detrimental influence on a researcher’s career prospects is likely to mean many more researchers are keeping their problems to themselves.

Whilst correlates of mental health difficulties have been identified (4) the overall issue would appear to be a systemic and existential problem in what the doctorate and a subsequent research career looks like. When universities attempt to attract research students to enrol, there is little to no mention of the potential negatives that lie ahead.

Once doctoral students start their programmes the mixed messages will become apparent. The inductions from Graduate Schools, Doctoral Colleges or whatever larger structures will explain that their education is about their own professional development and to take time to engage in the many opportunities available. However, the day to day messaging they will receive from their supervisors, research associates, and senior PhD students may be quite different. The all consuming requirement for, data generated, high impact factor papers published and grants in, becomes a frenzied narrative of career dependence.

The sense of competition and requirement to spend more time researching to compete in the race to fulfil the REF (Research Excellence Framework (5)) requirements of the unit and institution quickly trickles down to a doctoral student just hoping to get to grips with their own project. Fundamentally if we really want to see a significant improvement in how researchers feel then there will need to be a radical shift in the way research is set up. (see another blog post https://blogs.ncl.ac.uk/richardhetherington/2021/02/07/2020-a-vision-for-portfolio-careers-in-academic-research/)

Whist we wait for the great leap forwards, the question remains what can we do about the individuals currently caught in the crossfire of expectations for research outputs and personal and professional development? The first thing is to draw no division between the support and guidance that is technical, and is primarily there to provide tools to help with research and, the support that is for the individual to cope well with the research experience. So, when wellbeing services do recognise that the needs of PhD students are very different to undergraduates (PhD students are much less likely to be homesick for instance). Then the support which is provided is clearly signposted without any potential for stigmatisation by making it clear that personal development and support dealing with challenging situations is normalised as is counselling.

Research students should be able to access Cognitive Behavioural Therapies (CBT), talking therapy based workshops, or mindfulness meditation sessions in the same way as they would guidance on academic writing or statistics. Far better we are able to help researchers before a crisis point than wait until they hit to rocks or are standing on the levy.

For those who are heading toward a crisis point because of complexity of their research or the many other factors that may play a part in making life difficult, there needs to be appropriate support and well directed guidance. This is where the support and guidance provided to supervisors is key. This most complex of working relationships needs the recognition of boundaries and knowledge of signposting to adequate and timely resources. Supervisors are often friends and mentors to their students but they are not mental health counsellors and if a student’s problems go beyond their research project they should be given appropriate support from trained professionals.

So what can institutions introduce that might help researchers keep a sense of perspective and avoid the worst of the situations that might cause problems for their mental health. Here is where I link to my somewhat cryptic image, the reason my phone charger (like many, I guess) has broken at this point is because I’m trying to use the device whist it’s on charge, or I’m quickly pulling at it to draw it back into use when it has been charging. Dedicated researchers are generally so engrossed in their work that they are naturally drawn away from their time recharging. When they are not at the desk, bench or PC they are still thinking about the work. There needs to be freedom to escape and recharge. The first thing is not to amplify further their engagement with the research by making external demands of them to do more. Their own pace for a PhD is almost always fast enough (6).

The other thing is to structure the opportunity for support. Mindfulness meditation might not be for everyone but for those it works for, it really does make a massive difference. It helps people to live in their current moment and frees them up for the worries of what has not gone to plan or challenges that might lie in the future. Having Mindfulness sessions available is a real tangible way to help researchers manage the challenges they face. For others, who are happiest when they get something done, they may need something that is achievable to satisfy the need for some instant gratification that can often be lacking in the very long term goals and outputs of research. For those students community activities such as organising events like the North East Postgraduate Conference NEPG (7) could be one option. For other more individual pursuits like gardening or origami (8) may prove to be a source of some satisfaction. The recognition that time for these recharging processes is key and supervisors should encourage students to build these activities into their day. Productivity comes from happiness and being at ease with requirements not the constant pressure of needing to generate more data or outputs.

Finally, whatever the Universities choose to put in place to support their research students, there must be adequate resource for such support to continue through development. Research on any intervention ought to be through practitioner based enquiry with iterative development of good practice. This should not be an opportunity for researchers to further their careers as they observe from afar, assessing the potential influence of one off projects or schemes. There is surely enough compelling evidence that comprehensive support is needed for researchers to manage the challenges of research and the academic world. The question is should not be what is the one answer but how does the sector keep helping those who will need help.

1.https://biblio.ugent.be/publication/8613173/file/8613174.pdf

2.https://www.officeforstudents.org.uk/advice-and-guidance/student-wellbeing-and-protection/student-mental-health/catalyst-fund-supporting-mental-health-and-wellbeing-for-pgr-students/

3.https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/s13643-020-01443-1

4.http://sro.sussex.ac.uk/id/eprint/102260/1/UDOC%20Survey%20Predictors%20of%20PGR%20symptoms%20BJPsychOpen%20Accepted%2011.10.21.pdf

5. https://www.ref.ac.uk/

6. Berg, M., & Seeber, B. K. (2016). The slow professor: Challenging the culture of speed in the academy. University of Toronto Press.

7. https://ne-pg.co.uk/

8. https://www.vitae.ac.uk/events/past-events/vitae-researcher-development-international-conference-2017/Posters2017/Can%20mindfulness%20through%20meditation%20or%20Origami%20be%20used%20to%20support%20resilience%20and%20well%20being%20in%20researchers

1 September 202212 November 2022

Reporting non-significant results

Sorry for the long absence from blog posts. I’m going to try to ease my way back in with a short post that I’d like to pose as a question as much as offering any obvious solution.

I’ve just been in contact with a colleague at another University who was the external marker on a student’s work. We disagreed on the mark awarded but managed to quickly come to an amicable compromise. The work was well presented and clear. The issue was the student had not been absolutely clear about the results.

Whilst I had to agree that implying there might be significance in a finding where the data suggests otherwise is a major problem. However, I couldn’t help but feel some sympathy for the student. The requirement to report positive findings is so pervasive in scientific research that it is quite difficult to expect academics to instil into students the need to report with complete clarity and honesty on their findings.

Until there is a way to report results with neutrality in terms of the significance this will always favour outputs which support the researcher’s hypothesis. From a broader perspective I would suggest that the pre-registration of studies is a good way to battle against this bias. If researchers were recognised for the quality of the ideas that generate their hypotheses rather than whether the hypotheses are supported by the evidence they seek, that would be a great start.

We do however still have a system in which many journal editors are looking for data driven positive evidence that confirms a hypothesis. From an educational standpoint, I think it’s fair to make students aware that recognition has this bias. It still should be clear in their formative stage of producing work for educational assessment that a well explained non-significant finding can be worth the best grade awarded. If we fail to make this clear students will end up undertaking research projects where the result is so heavily anticipated that the research is hardly even worth doing.

6 May 20216 May 2021

Doctoral Training Partnerships & Centres and how they influence the formative development of researchers

By Oonagh McGee and Richy Hetherington

Over the last 10 years there has been a shift in the way the UK Government has funded doctoral research. The Research Councils, RCUK (now UKRI, UK Research and Innovation) started concentrating funding with Doctoral Training Centres at specific locations and building cohorts of research students working at places where there was a significant area of expertise. This model was developed further ensuring that funding for research students was dedicated to higher education institutions (HEIs) with a strong record in research. Doctoral Training Partnerships (DTPs) required institutions to have a research grant income above a certain threshold to apply. Only a select few universities could apply individually, most would be required to partner up in order to fund a cohort of students. Alongside the DTP model, universities were also able to apply for Centres for Doctoral Training, with significant funding offered by UKRI. The majority of this specifically concentrated on EPSRC priority areas, in comparison to other funding councils. The latest call (launched in 2018) saw a combined investment from EPSRC and external partners of £945M. (1) These targeted investments cover a wide range of topics although subjects can be disparate within each centre and each with their own operating models.

The concept that research students should learn their research skills together and develop as a cohort is intrinsically a very positive step. Everything that can be done to foster collaboration in research endeavours must be considered a positive. It could be argued that where students are co-located with their cohorts, or brought together for regular events and training, these centres could provide an antidote to the ‘lonely journey’ often reported by doctoral students.

In a study of doctoral centres in (2019-2020) students (n=209) and staff (n=6) reported on the less tangible benefits of doctoral centres (2) primarily relating to wellbeing and support but also the opportunity to share methods, skills, and work on problems collaboratively.

There are excellent training opportunities for students, in a variety of areas with individual DTPs arranging training in Science Communication, Enterprise & Innovation and Personal & Professional Development more generally. The BBSRC took a very defined focus on the employability of research students with their Professional Internships for PhD Students (PIPS) scheme. The requirement for three month internships away from the research was originally met with some scepticism from academic supervisors but the internships have proved popular with students. The opportunity to demonstrate and recognise research skills in other environments has proved to be a significant positive.

However, for the many benefits this concentration of funding with three and a half or four year fully funded studentships with numerous additional training opportunities included has increased the disparity in experience between research students depending on how they are being supported and their discipline.

Whilst feedback from students (n=209) was generally positive, students also highlighted issues of competition, comparison and, in some cases, extra activities as a distraction from their core research. Additionally, students highlighted that whilst working together could be beneficial in many areas, they felt a research group would have been a more appropriate environment.

One good example of how DTP funding has made an impact more broadly. Is the North East Postgraduate conference (NE-PG.CO.UK). By supporting a conference for their own students to present their work to the broader research student community, the DTPs can also promote an opportunity for all research students in the discipline to submit and share their work, practice their presentation skills and network. It has also proved an important opportunity for researchers to develop skills in organising such an event. Including a broad range of skills that stem from research, such as reviewing abstracts and providing critical but fair feedback. The Student led conference also allows researchers to develop organisational skills that are a quite tangential extension of those acquired through organising a research conference.

This opportunity to maximise and broaden the benefit that can come from the funding that comes with funded training centres and partnerships is a relative scarcity. The perspective of many of the academics involved in organising partnerships or centres seems to be a one of caution not to kill the goose that laid the golden egg. There have been notable cases where research intensive universities did not get the DTP they applied for or did not have their partnership renewed. A clear steer from UKRI that spreading and maximising the support would be greatly welcomed. Brexit and Covid19 are already greatly testing international research students’ desire to come to the UK. If the Universities are to continue to encourage a range of models for research students they need to ensure there is not a two tier (or many tier) system in doctoral education. Whilst the funded centre model has provided opportunities for students to work collaboratively across institutions, there is also the argument that some Centres for Doctoral Training (CDTs) & DTPs can lead to a two-tier system with funding, opportunities and support built around a specific cohort of students which are not afforded to those without this funding .

It has been recognised for some time that the Master and Apprentice model of PhD supervision is becoming outmoded with research being performed by larger interdisciplinary groups. There could be some consideration into the way in which doctorates are assessed. The individual monograph is not entirely in-keeping with modern research methods, nor does it accurately reflect how many students will go on to work within their chosen career . Industries outside of academia have embraced the effectiveness of cross-cutting teams and collaboration. It would appear there is increasing recognition that the complexity of the projects academics researchers face means that there will need to be greater openness collaboration and recognition for all parties involved, this is articulate very well by Professor Muzlifah Haniffa when speaking about how the research team they work in responded to Covid-19 (3). If universities and research councils are genuinely interested in developing individuals for their career post-doctorate a radical rethink may be required.

UKRI have continued to fund large-scale, multidisciplinary grants over the last decade, recognising that in order to solve grand challenges a new approach is required. If we hope to develop research culture away from the application of individual rewards to the lead or last author on papers, and the named individual on grant applications then it maybe be necessary to change the formative stages of building a research career to bring in greater recognition of the contribution that many individuals make together on a project. Is the portfolio PhD the way forward?

Recommendations from the report to the UKCGE Conference 2020 (2)

Could UKRI funding be allocated to enable engagement with non-funded students?
Could we have further funding distributed across institutions?
Ensure student feedback is incorporated into review process
Integrate student wellbeing and satisfaction scales and benchmark with PRES.

6 April 20216 April 2021

Could Artificial Intelligence Free Up Valuable Time for Academics?

A busy academic might be sent many requests the peer review papers each month, those involved in teaching will have student marking to contend with too. They may sit on grant awarding committees or editorial boards for journals or be required to decide whether to accept conference presentations for meetings. This will all be on top of the processes they need to go through in order to store their own data appropriately, ensure their projects have appropriate ethical approval and rectify any changes requested of their own work, as they respond to reviewers and editors comments.

All of these processes are not actually research or teaching, they are what research and teaching require to be robust and carefully monitored. As I tell PhD students early in their studies (1) there is not a governing body for academic research, the whole process is self-policing. Peer review is essential for confidence that research processes are robust. Marking students’ work is also a necessity to ensure robust judgements on students’ grades.

However, is everything being done as efficiently and as objectively as it could be? Assuming appropriate ethical approval has been provided and the research work has been considered worthy of funding from some source then it seems a logical extension that the findings should be published. Even if the results are null the academic community should be made aware of the work. Even an experiment that goes nowhere to support the hypothesis being tested may avoid that hypothesis being tested again when null results are provided.

So assuming all research work should be published there is then a question as to why journals choose to accept or reject articles for publication. In the days of hard copy journals this selective process had a genuine practical relevance. Now with journals being almost entirely viewed online there is no real pressure on the number of articles they have to present. A question about the merit or the noteworthiness of the research would seem be a key aspect. Journals retaining their esteem by selecting research work of a particular standard.

To some extent a judgement on the research question was made before the results are derived, analysed, discussed, written about and published because funders decided whether the research question was worthy of investigation. The concept of pre-registration of studies has been presented on several occasions and is being put into practice, with Prof Marcus Munafò championing this cause (2). So with ways to determine whether research is interesting or worth discussion already in place why should there be a decision after the work has been written.

It might also be expected the work has been conducted ethically, since ethical approval would be required before commencing. It is actually a little ironic that the ethical approval process can be completed without human intervention if no risks are flagged. I would seem reasonable to suggest that studies that are pre-registered would be unbiased in the set-up of the research and the analysis of the data generated. Then the only question that remains for the peer reviewer is to check whether the paper has been written up in a logical way, with good syntax and that is grammatically sound for the readers to interpret. So why not employ Artificial Intelligence to check the readability of submissions and make suggestions on improvements to the text if required. These are methods that are employed by software packages such as Grammarly.

There could also be a case made for the marking of students’ work using a developing computerised algorithm (or AI). It would probably prove an uncomfortable step for many to replace human markers entirely but it would seem like a reasonable approach to replace one marker of a pair with an AI moderator. Running numerous previous years’ exam scripts through a developing algorithm should allow the system to be tuned-in to identify the key factors in giving scores. If the AI and human maker disagree then a second human marker could moderate. There are experiments taking place with school students’ work (3) that could help show how well this might work.

It would be a brave step by any University that instigated marking partially by a computerised algorithm and also it would also be a bold move for a publishers that to review articles. However, if they were to do this they may benefit from freeing up the time of their academics to more creative in the research and teaching processes.

References

2 March 20212 March 2021

Teaching and research recognition for the continuum not the divide

Research Informed Teaching

The concept of research informed teaching has become a popular one with a variety of ways in which research is either presented to or enacted by students¹. The involvement of students in research is particularly important for the British research intensive universities that are looking to differentiate their offering to students paying ever escalating fees for their undergraduate and masters programmes.

However, an undergraduate or Masters students’ expectations and experience is probably quite different from the realities of research for those who conduct the work on a daily basis. Bioscience students are lured to degree programmes by the latest buzz-words of technological advance: stem cells, proteomics, Nanomedicine etc. The early stages of the students’ programmes build these expectations further as academic staff provide lectures on recent advances. The staff delivering this information tend to be either teaching lecturers, who have taken a sideways step away from research or well established academics who are likely to have developed and used these technologies, but now have limited time for the hands-on work. It is often a research group’s post docs, PhDs, staff scientists and technicians who are conducting research work as their day to day activity but those individuals are not so likely to interact with the students.

The interaction between the active researchers and the students is not likely be at the early stages of the degree programme. Only as undergraduate students start to engage in projects will they become involved in the research and with the researchers. Understandably, the students will be keen to get their hands on the equipment and build new skills. The length of time for projects and the desire to fulfil this requirement for practical experience will often mean parachuting the students directly into sample preparation and data collection. Those parts of the research process are only a part of the work and in many areas automation means it is diminishing part.

Delivering a Truer Picture of Research

Technological advances mean large amounts of data are collected by complex machines meaning a researcher’s time is often taken up with planning and data analysis. Recognition that there have been failures in the robustness and reproducibility of research also means researchers are increasingly asked to focus on getting their processes right. Time has to be spent on data management planning, ethical approval and recording the meta-data that accompanies new protocols and datasets. These are the processes that students need to be aware of, every bit as much as the experimental methods. Students should also understand that research can only happen when it is funded and know that much of an academic researcher’s work is obtaining funding to support their projects. Unless we teach all aspects of these processes then the students who do transition to research degrees will not have a fair representation of what life as a researcher or academic is really like.

Teaching Informed Research

In STEM subjects research is scarcely conducted individually, most research is performed in large groups of individuals interconnected in a variety of ways. Although the recognition for first and last author on academic papers might reward the individuals there are usually several people who contribute to the work. The quality of the interaction between those individuals may be critical to the success of a research group and project. Although the transfer of information within groups is far from linear or didactic there are certain expectations that the Group Leaders or Principal Investigators will pass down their vision for the projects, the junior lecturers, research fellows and post docs and PhD student will plan out the research and conducted the procedures with the help support and guidance of staff scientists and technical staff. Those new to the group will be informed by those who are more experienced. This means there is a lot to take in for anyone new to the research group and those looking out for new doctoral students or project students will have a significant task to get those new to the group up-to-steam.

Whatever we call the process: training, continual professional development (CPD) or learning on the job, this is a form of teaching and the standard to which it is performed will have a significant impact on the learner and the productivity of the group. Historically, the expectation has been that those who rise to the top in academic research will be able to impart their knowledge and a new generation of researchers will sink or swim as they learn how the research works. This model becomes less tenable as time passes, it’s unrealistic to expect the Group Leaders to know everything that is required for projects that are increasingly interdisciplinary. It is also unrealistic to hope or expect that everyone will be as good at passing on their skills and knowledge as they are at conducting the work.

Support and guidance for those teaching research

There has been recognition that research students can benefit from training and Development Programmes(2). More general skills development and positive attributes can be delivered effectively outside the research group and researchers will be able to reflect on their own developments and training needs(3). Recognising the need to support academics and researchers with the s guidance they need to be teachers of research has been less obvious. As Professor Stan Taylor suggests we should think of PhD supervision as teaching (4) in doing so we can start to recognise the continuum that occurs between teaching and research and support academics to contribute to both aspects of their role. One area we have recognised as quite deficient was in the formative stages of this passing down of knowledge; Doctoral Students or Early Career Researchers / Post Docs might not be considered the formal supervisors of an undergraduate or Masters’ project students but they do have a significant role in teaching them. While it is commonplace for PhD student lab class demonstrators (Graduate Teaching Assistants) to have to training before teaching, those with an informal role in project supervision do not. We introduced guidance sessions in this area and it has proved popular and helpful not only to PhD students but to staff too.

The potential for positive changes

Engaging undergraduate students with more of the processes in research may help them to get a broader understanding of how research work is undertaken and move their fixation away from the newest techniques. This may also help inform undergraduate students thinking of research masters and doctoral study. Providing this broader engagement with research could involve the PhD students and staff that are active in the day to day work of research. If these PhD students, Early Career Research Staff and Staff Scientists are trained to provide this teaching they will be more experienced and hopefully better in the process of supporting their colleagues and research students as they advance. Changing the expectations of undergraduate and masters’ students so their projects might be longer but encompass a broader view of research processes may help to unburden faculties trying to meet a requirement to offer the technical aspects of research. This broader curriculum for undergraduates may also allow the academics normally involved in teaching a little more time to keep their research interests live. Recognising the range of activity that researchers and teachers get involved in with a greater place for formal recognition of the teaching role that is intrinsic to research (5).

7 February 20218 February 2021

20:20 a vision for portfolio careers in academic research

Restricting the time individuals spend on their own research project to twenty hours a week might allow for greater equality and less burn-out.

There’s no such thing as a job for life anymore, has now become an old adage, we have moved on from that, to the point where a more common discussion is whether work-life balance is ‘a thing’ (1). However, much of what is sold to a new generation of academic researchers, starting out on a PhD is exactly that, the idea of a job for life. They are tacitly offered the prospect that they might follow in the footsteps of their academic mentors or supervisors. They are hijacked into thinking their doctorate, their academic papers, their conference presentations and their successful grant applications will lead to a permanent contract in academic research or higher education. The truth is, for some that will happen. It might mean working really long hours, facing countless rejections, reapplications and resubmissions but ultimately it is possible they will end up with a full time position at a University and never leave the academic world. For the majority though, the reality is they will leave academia and will have diverse and potentially more financially fruitful careers elsewhere. Arguments about what could be considered success we can leave to one side, but I am minded of the former PhD student who told me of their lucrative career in Management Consultancy and how the boss of the company had a Nature paper to their name, but had left academia of their own volition. What I really want to consider is how the careers of those who stay in academia develop . A study of Professors in the Netherlands indicated that their time conducting research was down to 17%, and this article suggests anyone who would like to continue in research should probably look outside of academia (2)

So how do we address the existential crisis in academic careers that seems to be at the route of problems in the research environment? Whether researchers have an open or fixed term contract they are always likely to call for more funding to resolve the issue. Whilst I wouldn’t argue with the idea of increasing funding for research, I’d be wary of suggesting this would be a panacea. More research funding is likely to mean more PhD students and more post doc positions, even if it did mean more open contracts they would quickly be filled and the issues of the promise of a career would not materialise for many.

The inequalities in research can be very obvious and policies to address those issues need to be firmly held up. However, some things relate to societal differences that affect people more broadly. Quite simply the 60+ hour working week many Early Career Researchers feel pressured to adhere to is a much more of a feasible option for someone without the caring responsibilities of children or elderly relatives. It’s also likely to be a good deal easier for those without physical disabilities or mental health issues. Simplistic legislation that demands equality will not be enough to deal with the complex issues that limit what people can do. Whilst some have recognised their own limits and realised that it is only possible to be truly productive for so long (3), there will still be many who are using presenteeism as a proxy for what it takes to be successful.

The answer to several of these issue maybe an acceptance that the development of new research ideas is improved by allowing academics more time to think and with a fairer distribution of the range of other administrative and teaching roles. Requiring a more nurturing environment for those learning from the academics means a greater contribution to research from everyone in the Higher Education system.

Here, I propose limiting the academic research of all research staff to a maximum of 20 hours per week . This might be an entire contract for a working parent. They could then avoid falling behind colleagues with who can devote more time to their research. Of course it would be up to institutions how the other 20 hours were made up. If this was included the support of undergraduate teaching, PhD supervision or technical support for other projects, that would all be good and could help with their own development in the broader sense of their careers. It could also be fairly allocated in order to that researchers could take a strategic approach to their applications for further research funding or to work as a reviewer or editor. As has been identified advancing in academic research tends to lead to growing out of it. Supporting and guiding researchers to longer careers in research with a broader role, encompassing their own development and the development of others, can only be to the long term advantage of the research environment. It may also help early career researchers recognise the importance of the broader aspects of a role in academia. Once more time is required to be devoted outside of the individual’s research project goals then a greater emphasis could be made in assessing their achievements in the broader environment.

With the academic role split more fairly for all on a full time contract covering administrative, teaching, outreach and development roles this could mean broader recognition of those activities. Success across all aspects of the role could then be considered when it came to. Perhaps, this may mean less promotion and fewer Professors, but ultimately if it provided more careers including research with passion and longevity this may create a flatter but happier pyramid of academic careers.

14 January 202114 January 2021

Where ’R’ we headed with data analysis?

A few years ago I was in an internal meeting with presentations from across the University. Someone from outside of my Faculty was making an impassioned argument for the use of the open source programming language R and suggesting all students who require quantitative skills within their discipline should be taught R. Something they said then stayed with me.

“If SPSS is like waiting for the bus, then R is like taking a taxi”

Although intriguing there was something in what they said that made me a little uneasy. It took me a little while to work out what it was, and a good deal longer to write about it but I think I am OK about saying what the issues are; The bus can take you on a route that anyone can follow and assuming you have a pretty good idea where you are going it will get you there, along with whoever wants to follow you to that destination. Taxis in a familiar place a fine but what about when you are going somewhere new? It is really down to the taxi driver to interpret where your destination is, if you are not sure then you could end up in the wrong place and if the taxi has followed a circuitous route, or a route only they know is it realistic to expect anyone else to be able to follow you?

I am not going aiming to suggest everyone should use standardised statistics packages for data analysis, or that everyone should always get the bus. What I would say is there are reasons to be vigilant when following the analysis performed by others and to also be very careful when working with tools as powerful as R with all of its downloadable packages.

It can be quite laborious remaining faithful to the scientific method, which requires you to discount the null hypotheses before accepting your own as the most likely. The using a programming language such as R offers several short cuts and it is good to be aware of what is acceptable and what might be ways that confirm inherent biases.

Free to use

One absolutely undeniably brilliant feature of R is it is free and can be downloaded at https://www.r-project.org/ in a somewhat ironic way this means that anyone can use it to do their research analysis as long as they can afford a half decent computer. The only people who might struggle are those who have system administrators that will not allow packages to be added. It is likely those working on a University machine can access ‘R Studio’ or something similar but they may struggle to download the ‘distributed packages’ that have been created to run in R.

“R is an anarchist utopia” [1]

The many thousands of distributed packages are a great positive but also presents a couple of the inherent risks with R, one is the packages are not guaranteed to work, which could be annoying. What I see as more problematic is choosing the right packages. One of the packages might provide a significant result with your dataset but what checks are in place to make sure the data is being analysed appropriately?

Choosing the right statistical test

As an example there is function, cor.test which can test for correlation between two datasets but it will assume parametric data for variables and how do you know the data is normally distributed? Let us imagine that we are looking for a correlation between the heights of parents and their children. I don’t suppose anyone would make such a simple mistake but if we were to look for the correlation between both the heights of mothers and fathers with the height of sons and daughters the test should be a non-parametric test and require a different function. The data from both comparison variables may appear normally distributed but is actually bimodal and so a non-parametric test would be required. The function won’t know this, only you will. In this example the same mistake could be made just as easily in SPSS or Minitab, as it could in R but I guess as the data or the questions get more complex then the chances of errors increase.

If one was to use the Correlation Test perhaps the shapiro.test should be used first to test whether the data is normally distributed. I guess the issue here is it might take a very large sample to recognise that the distribution is not exactly normal.

Representing data graphically

The ability to make attractive looking graphics is a very useful function in R. This is very helpful for making publication quality graphics. The ability to change the qualities of a set of variables is a good way to emphasise the point you are trying to make with your data. Inherent with the capacity to do this is the risk that you give an impression your argument is somewhat stronger than it really is. There is also the chance to change the graph type or the axis to make a point more convincingly. The peer review of any academic outputs is reasonably likely to pull you up on anything too obvious or flagrant but this is not a clearly defined right and wrong. Of course you can do this with any graphics package as well as you can with R but cycling through a set of alternatives might not be quite as straight forward as it would be with the graphics packages in R. This is also something to consider beyond the academic publication. It’s unlikely the general public would find many of the graphs displayed in academic papers as visually appealing. So the chance to redraw graphs for public consumption is there, with alt-metrics becoming an important feature what researchers do to present their data could be quite critical in getting some broader recognition to their findings. There are some very obvious ways to display data in a misleading way, covered well in this blog (https://venngage.com/blog/misleading-graphs/) [2]

I wouldn’t really expect anyone reading this blog to be displaying information with complete contempt for the relationship to evidence but as I say there is some gradation between helping the reader see key points in the data and exaggerating significance.

Standard Tests and Model Selection

With any quantitative analysis plotting out the data to see how what it looks like is a very good thing to do. It’s possible to see trends and correlations and then you can test to see if there is a fit that goes along with what you might expect. If there is not an obvious fit to go along with your hypothesis you can see how other variables might be at play. Faceting a dataset could help identify a key factor. I guess the troublesome question I have in faceting is, how many different variables should be tried to find a trend or correlation that matches what is expected. If this is just an exploration of datasets in order to generate a hypothesis to be tested then that seems fine, how else do we find out? However, if it’s a finalised dataset and this is an action to find some significance, any significance well there might be something there but run enough tests with enough variables and one it twenty times one will crop up. I think. Splitting up data sets in enough ways could even end up with a Simpson’s Paradox (https://en.wikipedia.org/wiki/Simpson%27s_paradox [3]) type situation where entirely the opposite findings could be retrieved from the same data set.

An advantage of looking for statistical significance with something like a T-test or a linear regression is the relative simplicity of the answer. Comparing two datasets to see if they are different or if they correlate gives something of an all or nothing result, even though it should not really be viewed that way. R appears to be good if you have a more complex set of variables and you want to see which model fits the interactions best. I guess the concern I have with this approach is what if the best model is just the best of a bad bunch. It also appears there are a few methods of testing which model is the best, so selecting which is best might make a difference and is it clear which is best [4].

The complexity and rigour of analysis for research

My impression is the very nature of research is changing, maybe quite slowly but as more questions are answered the next set of queries we look at become ever more complicated and involve more variables. When there are a large number of factors at play generating evidence through the scientific method becomes complicated. If the null hypothesis is going to be disproved all of the confounding variables need to be accounted for. So I can see why looking to alternatives to the scientific method become attractive. I guess the risk is how robust alternative approaches are. The great advantage of trying to disprove the null hypothesis is it’s a pretty rigorous process and as long as the experiments are set up without biases it should be sound. So when we look at alternatives can we be as assured they have the same degree of rigour? My issue with model selection is it could be possible to find solutions to a data set where the model fits but the solutions are correlative, not causal. The same thing could happen with the scientific method approaches but it somehow seems like less of an easy trap to fall into. Sorry for a rather cruel example, lets imagine someone has noted that ‘Spiders with no legs aren’t eating any flies’. In the scientific method checking the dietary consumption of spiders with their legs removed might show us that they do eat less but it doesn’t disprove a null hypothesis that spiders are hungry regardless of how many legs they have. However, if we start comparing the hypothetical models that ‘the fewer legs spiders have the less hungry they are’ and ‘Spiders count the number of flies they should eat each day on their legs’ we might find we select a hypothesis that correlates with our data but does not reflect what is really going on.

Once model comparison has been adopted as an approach to search for evidence then maybe it is better just to concede that dredging the data to find the best model is an inevitability. If you don’t do it someone else may well use your open data and look. If you have missed the most likely model that the data predicts you might need to explain why you went with the one that you did select. If the model the dredging throws up as the most likely doesn’t match your hypothesis, then that seems a bit like the null hypothesis and maybe a new a hypothesis is needed.

Meta-analysis

Collecting data together from various sources to build an evidence base on a particular hypothesis seems to be the most logical and strongest way of supporting that hypothesis. The meta-analysis of various datasets can be used to quantify how strongly you can support your hypothesis. It all seems perfectly reasonable, the only issue I have with this is the independence of the data sets. In my time I’ve been known to make a few bets on the horse racing. It is interesting to see how punters make this type of error when they analyse the chances of a horse winning. The will look at factors like how often a horse has won over different distances, ground conditions, against a certain number of runners, with a particular jockey etc. By the time they come to a conclusion they can be completely convinced the horse is bound to win. What they don’t consider is they have counted the same previous win or good run multiple times. I don’t think the mistakes are as obvious in meta-analysis of scientific research but the principle is the same. Unless the evidence isn’t associated in any way then the factors that are linked need to be considered and weighted appropriately. Here are some papers that describe that issue better than I have. Senn 2009 provides a helpful review on the subject. [5]

References

Nettle D. https://workshops.ncl.ac.uk/fms/rprogramming/
McReady R. https://venngage.com/blog/misleading-graphs/
Colin R. Blyth (June 1972). “On Simpson’s Paradox and the Sure-Thing Principle”. Journal of the American Statistical Association. 67 (338): 364–366.
Brownlee J. https://machinelearningmastery.com/probabilistic-model-selection-measures/

24 November 202024 November 2020

Support and Care in Academia is Female Dominated but Poorly Rewarded

Vitae, the umbrella organisation that considers the professional development of academic researchers in the UK, hosts an annual conference each year it attracts over a thousand delegates, well in a normal year, at least. While across academic disciplines male professors outnumber their female counterparts nearly three to one[i]. The professors that have presented at this event (n=55) were majority female (55%). What is more striking was considering all presenters (n=600), where 70% were female[ii], with the same proportions for those using Dr. as their salutation, this markedly differs from academia as a whole where differences in subject area balance out to a near fifty-fifty split in gender. It seems that the people who have dedicated their careers to supporting the development of researchers, or at least have a keen interest in researcher development are more likely to be female. This is no bad thing in itself but the role comes with less recognition than the academic researchers they are there to support. The majority of those professors presenting have achieved that position in another field and have taken a sideways step to consider the development of colleagues. If working in researcher development or at least being interested in the development of others is not adequately rewarded or considered a key aspect of an academics role then research will ultimately suffer.

Data from Universities in the Netherlands[iii] corroborates HESA data indicating female academics are more likely to have greater teaching responsibilities than their male counterparts, who are engaging more in research activities, I’m doubtful other parts of the world trend differently. It quite clearly isn’t an inability of women to conduct great research but a system that allows more structure and a marginally better work life balance in teaching roles. The freedom for men to work 12 hour days when the job requires is there in society and although most wouldn’t want a work-life balance so skewed, some will take it to make strides at an early point in their career. They can do this in the knowledge that in the future they can tell their PhD students and post-docs this is the way it has to be if they want to progress. The dichotomy between teaching and research does not help this situation. Giving more academics a broader role may help to promote a better transfer of the cutting edge of research to our undergraduate and master’s students but it would also ground the research focussed staff in the structure of the University and broaden the opportunities for a contribution to research.

Amongst PhD students the difference between the typical attitudes of men and women to their own development is palpable. At a recent Public Engagement competition (Three Minute Thesis) the female students out-numbered the male competitors fifteen to one. Had the numbers been reversed and a list of eight men was presented as the finalists it would have likely caused some significant consternation, it certainly would have been dropped as a data-point from any Athena Swan application. The prevalence of women in this competition is however symptomatic of the larger issue. Women do care that their research makes a broader impact, than the impact factor of the journal the work is published in. They are doing the research to make a difference in a world they care about. Their goals are less individually focussed and are more for the broader benefits the work could bring. Working hard to be a success in disseminating research findings more broadly is a ‘nice to have’ in any fellowship or tenure application. Until there is much greater recognition of the overall benefits research brings then we will struggle to see men taking the time to think about the depth of their personal development. Nor will we see women recognised for what they do to disseminate their work, progress their own development or consider the development of the researchers that work with them.

The problem lies in the short term objectives in academic research. Academics who are able to dedicate a significant proportion of their time to generating data, publishing papers and writing grant applications are routinely rewarded. Whilst all of these activities are critical aspects of research success they only take an incidental attitude toward the development of the research students and research staff who are being mentored by that academic. And so perpetuates a selfishness in the academic world. Collaboration and collegiate activity do occur but these symbiotic relationships tend to be ephemeral because ultimately the rewards are given to individuals. With societal structures as they are and the timing of significant moves in academic careers coinciding with a time when starting a family is likely, women are significantly disadvantaged by this system.

Current University fellowships make no formal account of the contribution a researcher has made to the supporting undergraduate, masters or PhD students. There is little consideration of whether this candidate will continue to help others develop. The bottom-line of prestigious publications and grant income far outweigh those longer term goals that may see a paradigm shift to a much broader pool of contributors to research.

[i] https://www.hesa.ac.uk/news/23-01-2020/sb256-higher-education-staff-statistics

[ii] https://www.vitae.ac.uk/events/event-presenters

[iii] https://www.erasmusmagazine.nl/en/2019/03/12/female-academics-teach-more-lectures-and-conduct-less-research/

27 October 202027 October 2020

Open Science: A Spotify for Research?

If you are as old as I am you might remember buying music on vinyl records not because of the warmth of the sound or nostalgia, but because it was the only option. If you wanted to hear something new without purchasing it the local library would allow the loan of vinyl records as long as they were returned without scratches. The transition through various formats has brought us to a point where music aficionados can access most major musical outputs by streaming them directly, anything from classical overtures to death metal, and we are given the choice whether we accept the full output or just the single track we would like to hear. Similarly those of us who have been involved in academic research in a previous millennium will have remembered trips to a university library to leaf through bound copies of a monthly periodical. Whilst the Journals have become electronic and we don’t need to visit the library to access them, the publishers of academic journals are still acting like record labels; researchers who have published previously and have been cited frequently are strongly encouraged to submit their papers. Knowing an academic has produced popular work on a hot topic means they are likely to get the journal more citations and boost the impact factor, good news for both the publisher and the researcher. Starting out, a research career nearly always means teaming up with one of the established artists of research and the choice a prospective PhD student or postdoc makes can be the most critical aspect of their career success.

Open access publishing gives everyone with the internet access to research published through the open access journals. So in some respects open access is like Spotify for research papers but unlike Spotify the researchers themselves, or indirectly their funders, pay to release the papers. Whilst the returns for most of the musicians who put their work out on Spotify are minimal at least it is those who are in receipt of the goods that pay. When national governments are the funders of research they can make open access publishing a requirement and as long as the grants incorporate this funding there shouldn’t be too much trouble. The problem is the majority of research is funded by a variety of other sources (https://royalsociety.org/~/media/policy/projects/eu-uk-funding/uk-membership-of-eu.pdf). Charities might not want to spend more money just to give away the data and findings they have paid for, and industrial sponsors of research are probably quite keen to make sure the research can stay under-wraps until the findings can be exploited for the commercial gain they have invested to receive. The most problematic issue currently is the mixture of types of journal, so even for data and findings that people want to share there is a ‘paid for’ and ‘pay to’ model. Appealing to the good nature of researchers to only publish in Open Access journals seems to be the current approach for the Open Science advocates, while the most morally upstanding and well-funded may see fit to do this, others who may be under significant pressures for their career might not feel so predisposed to act in community spirited altruism.

So what if there was one unified system where all publications could be submitted for free and institutions and individuals could pay a subscription to access all of that information, a Spotify for research? The economies of scale would be likely to mean the subscription charges could be kept to a level that institutions, research groups and maybe, even interested individuals could quite easily afford.

The current subscription model is already suffering an equivalent of home-taping, which incidentally didn’t kill the music industry, it probably didn’t even damage the profits of the major labels; when an academic sends a Twitter message to ask if someone in their network can share a pdf they are likely to get a positive response. This illicit sharing is as likely to end with the researcher asking their library if they can subscribe to the journal in the future as it is to the researcher asking the same associate to send multiple papers or issues of that journal.

There is an issue with Spotify for music, the most popular artists of our time receive a much greater proportion of the subscription revenue than when Record Labels paid artists by advances on their future albums. An artist who sells a 100,000 copies of their album on Spotify might only receive a fraction of what they need to live. However, the researchers don’t need the returns for paper views just that citations are recorded then this issue does not seem so important. Without the significance of the journal’s prestige influencing the number of citations then only the title, abstract and keywords will be needed to bring in the readers. Then it would really be down to the ingenuity and quality of the research that would determine the number of citations. Maybe the ‘Spotify of research’ does not need to stop there, it could become not only a repository for full papers but a place for academics to post their raw data and hypotheses, avoiding researchers hoarding together their findings for a paper of the greatest significance. The ideas and the data could still be cited showing the impact that the work has had. A thoroughly comprehensive system might even look to adapt the patent system so attribution for contribution and reward for ideas rather than resources could be made, but that’s for another time maybe.

The fundamentals of this system are already in place, Scopus and PubMed provide links to all the journal articles but are still just a conduit for the publishers to be paid for the journals that aren’t open access. By demanding a subscription for PubMed and this being used to pay the publishers a transition could be reached. Any journal that did not accept the terms offered would not have their journals on the system. It may require some brinksmanship in the first instance but ultimately less of the money for research would be diverted to a system that is a hangover from a less free information age. You never know there might even be a nostalgic resurgence of print copy journals that researchers can have on shelves in their office or go to a library to leaf through.

Dr Richy Hetherington

Lecturer, Graduate School, Faculty of Medical Sciences, Newcastle University

27 October 2020

Hello world!

Welcome to NCL Blog Service. This is your first post. Edit or delete it, then start writing!