Reporting non-significant results

scatter plot not significant

Sorry for the long absence from blog posts. I’m going to try to ease my way back in with a short post that I’d like to pose as a question as much as offering any obvious solution.

I’ve just been in contact with a colleague at another University who was the external marker on a student’s work. We disagreed on the mark awarded but managed to quickly come to an amicable compromise. The work was well presented and clear. The issue was the student had not been absolutely clear about the results.

Whilst I had to agree that implying there might be significance in a finding where the data suggests otherwise is a major problem. However, I couldn’t help but feel some sympathy for the student. The requirement to report positive findings is so pervasive in scientific research that it is quite difficult to expect academics to instil into students the need to report with complete clarity and honesty on their findings.

Until there is a way to report results with neutrality in terms of the significance this will always favour outputs which support the researcher’s hypothesis. From a broader perspective I would suggest that the pre-registration of studies is a good way to battle against this bias. If researchers were recognised for the quality of the ideas that generate their hypotheses rather than whether the hypotheses are supported by the evidence they seek, that would be a great start.

We do however still have a system in which many journal editors are looking for data driven positive evidence that confirms a hypothesis. From an educational standpoint, I think it’s fair to make students aware that recognition has this bias. It still should be clear in their formative stage of producing work for educational assessment that a well explained non-significant finding can be worth the best grade awarded. If we fail to make this clear students will end up undertaking research projects where the result is so heavily anticipated that the research is hardly even worth doing.

Could Artificial Intelligence Free Up Valuable Time for Academics?

A busy academic might be sent many requests the peer review papers each month, those involved in teaching will have student marking to contend with too. They may sit on grant awarding committees or editorial boards for journals or be required to decide whether to accept conference presentations for meetings. This will all be on top of the processes they need to go through in order to store their own data appropriately, ensure their projects have appropriate ethical approval and rectify any changes requested of their own work, as they respond to reviewers and editors comments.

All of these processes are not actually research or teaching, they are what research and teaching require to be robust and carefully monitored. As I tell PhD students early in their studies (1) there is not a governing body for academic research, the whole process is self-policing. Peer review is essential for confidence that research processes are robust. Marking students’ work is also a necessity to ensure robust judgements on students’ grades.

However, is everything being done as efficiently and as objectively as it could be? Assuming appropriate ethical approval has been provided and the research work has been considered worthy of funding from some source then it seems a logical extension that the findings should be published. Even if the results are null the academic community should be made aware of the work. Even an experiment that goes nowhere to support the hypothesis being tested may avoid that hypothesis being tested again when null results are provided.

So assuming all research work should be published there is then a question as to why journals choose to accept or reject articles for publication. In the days of hard copy journals this selective process had a genuine practical relevance. Now with journals being almost entirely viewed online there is no real pressure on the number of articles they have to present. A question about the merit or the noteworthiness of the research would seem be a key aspect.  Journals retaining their esteem by selecting research work of a particular standard.

To some extent a judgement on the research question was made before the results are derived, analysed, discussed, written about and published because funders decided whether the research question was worthy of investigation. The concept of pre-registration of studies has been presented on several occasions and is being put into practice, with Prof Marcus Munafò championing this cause (2). So with ways to determine whether research is interesting or worth discussion already in place why should there be a decision after the work has been written.

It might also be expected the work has been conducted ethically, since ethical approval would be required before commencing. It is actually a little ironic that the ethical approval process can be completed without human intervention if no risks are flagged. I would seem reasonable to suggest that studies that are pre-registered would be unbiased in the set-up of the research and the analysis of the data generated.  Then the only question that remains for the peer reviewer is to check whether the paper has been written up in a logical way, with good syntax and that is grammatically sound for the readers to interpret. So why not employ Artificial Intelligence to check the readability of submissions and make suggestions on improvements to the text if required. These are methods that are employed by software packages such as Grammarly.

There could also be a case made for the marking of students’ work using a developing computerised algorithm (or AI). It would probably prove an uncomfortable step for many to replace human markers entirely but it would seem like a reasonable approach to replace one marker of a pair with an AI moderator.  Running numerous previous years’ exam scripts through a developing algorithm should allow the system to be tuned-in to identify the key factors in giving scores. If the AI and human maker disagree then a second human marker could moderate. There are experiments taking place with school students’ work (3) that could help show how well this might work.

It would be a brave step by any University that instigated marking partially by a computerised algorithm and also it would also be a bold move for a publishers that to review articles. However, if they were to do this they may benefit from freeing up the time of their academics to more creative in the research and teaching processes.

References

  1. https://workshops.ncl.ac.uk/fms/integrity/
  2. https://academic.oup.com/ntr/article/19/7/773/3106460
  3. https://ofqual.blog.gov.uk/2020/01/09/exploring-the-potential-use-of-ai-in-marking/

20:20 a vision for portfolio careers in academic research

drowning clock

Restricting the time individuals spend on their own research project to twenty hours a week might allow for greater equality and less burn-out.

There’s no such thing as a job for life anymore, has now become an old adage, we have moved on from that, to the point where a more common discussion is whether work-life balance is ‘a thing’ (1). However, much of what is sold to a new generation of academic researchers, starting out on a PhD is exactly that, the idea of a job for life. They are tacitly offered the prospect that they might follow in the footsteps of their academic mentors or supervisors. They are hijacked into thinking their doctorate, their academic papers, their conference presentations and their successful grant applications will lead to a permanent contract in academic research or higher education. The truth is, for some that will happen. It might mean working really long hours, facing countless rejections, reapplications and resubmissions but ultimately it is possible they will end up with a full time position at a University and never leave the academic world. For the majority though, the reality is they will leave academia and will have diverse and potentially more financially fruitful careers elsewhere. Arguments about what could be considered success we can leave to one side, but I am minded of the former PhD student who told me of their lucrative career in Management Consultancy and how the boss of the company had a Nature paper to their name, but had left academia of their own volition. What I really want to consider is how the careers of those who stay in academia develop . A study of Professors in the Netherlands indicated that their time conducting research was down to 17%, and this article suggests anyone who would like to continue in research should probably look outside of academia (2)

So how do we address the existential crisis in academic careers that seems to be at the route of problems in the research environment? Whether researchers have an open or fixed term contract they are always likely to call for more funding to resolve the issue. Whilst I wouldn’t argue with the idea of increasing funding for research, I’d be wary of suggesting this would be a panacea. More research funding is likely to mean more PhD students and more post doc positions, even if it did mean more open contracts they would quickly be filled and the issues of the promise of a career would not materialise for many.

The inequalities in research can be very obvious and policies to address those issues need to be firmly held up. However, some things relate to societal differences that affect people more broadly. Quite simply the 60+ hour working week many Early Career Researchers feel pressured to adhere to  is a much more of a feasible option for someone without the caring responsibilities of children or elderly relatives. It’s also likely to be a good deal easier for those without physical disabilities or mental health issues. Simplistic legislation that demands equality will not be enough to deal with the complex issues that limit what people can do. Whilst some have recognised their own limits and realised that it is only possible to be truly productive for so long (3), there will still be many who are using presenteeism as a proxy for what it takes to be successful.

The answer to several of these issue maybe an acceptance that the development of new research ideas is improved by allowing academics more time to think and with a fairer distribution of the range of other administrative and teaching roles. Requiring a more nurturing environment for those learning from the academics means a greater contribution to research from everyone in the Higher Education system.

Here, I propose limiting the academic research of all research staff to a maximum of 20 hours per week . This might be an entire contract for a working parent. They could then avoid falling behind colleagues with who can devote more time to their research. Of course it would be up to institutions how the other 20 hours were made up. If this was included the support of undergraduate teaching, PhD supervision or technical support for other projects, that would all be good and could help with their own development in the broader sense of their careers. It could also be fairly allocated in order to that researchers could take a strategic approach to their applications for further research funding or to work as a reviewer or editor. As has been identified advancing in academic research tends to lead to growing out of it. Supporting and guiding researchers to longer careers in research with a broader role, encompassing their own development and the development of others, can only be to the long term advantage of the research environment. It may also help early career researchers recognise the importance of the broader aspects of a role in academia. Once more time is required to be devoted outside of the individual’s research project goals then a greater emphasis could be made in assessing their achievements in the broader environment.

With the academic role split more fairly for all on a full time contract covering administrative, teaching, outreach and development roles this could mean broader recognition of those activities. Success across all aspects of the role could then be considered when it came to. Perhaps, this may mean less promotion and fewer Professors, but ultimately if it provided more careers including research with passion and longevity this may create a flatter but happier pyramid of academic careers.

Support and Care in Academia is Female Dominated but Poorly Rewarded

vitae conference 2019

Vitae, the umbrella organisation that considers the professional development of academic researchers in the UK, hosts an annual conference each year it attracts over a thousand delegates, well in a normal year, at least. While across academic disciplines male professors outnumber their female counterparts nearly three to one[i]. The professors that have presented at this event (n=55) were majority female (55%). What is more striking was considering all presenters (n=600), where 70% were female[ii], with the same proportions for those using Dr. as their salutation, this markedly differs from academia as a whole where differences in subject area balance out to a near fifty-fifty split in gender. It seems that the people who have dedicated their careers to supporting the development of researchers, or at least have a keen interest in researcher development are more likely to be female. This is no bad thing in itself but the role comes with less recognition than the academic researchers they are there to support. The majority of those professors presenting have achieved that position in another field and have taken a sideways step to consider the development of colleagues. If working in researcher development or at least being interested in the development of others is not adequately rewarded or considered a key aspect of an academics role then research will ultimately suffer.

Data from Universities in the Netherlands[iii] corroborates HESA data indicating female academics are more likely to have greater teaching responsibilities than their male counterparts, who are engaging more in research activities, I’m doubtful other parts of the world trend differently. It quite clearly isn’t an inability of women to conduct great research but a system that allows more structure and a marginally better work life balance in teaching roles. The freedom for men to work 12 hour days when the job requires is there in society and although most wouldn’t want a work-life balance so skewed, some will take it to make strides at an early point in their career. They can do this in the knowledge that in the future they can tell their PhD students and post-docs this is the way it has to be if they want to progress. The dichotomy between teaching and research does not help this situation. Giving more academics a broader role may help to promote a better transfer of the cutting edge of research to our undergraduate and master’s students but it would also ground the research focussed staff in the structure of the University and broaden the opportunities for a contribution to research.

Amongst PhD students the difference between the typical attitudes of men and women to their own development is palpable. At a recent Public Engagement competition (Three Minute Thesis) the female students out-numbered the male competitors fifteen to one. Had the numbers been reversed and a list of eight men was presented as the finalists it would have likely caused some significant consternation, it certainly would have been dropped as a data-point from any Athena Swan application. The prevalence of women in this competition is however symptomatic of the larger issue. Women do care that their research makes a broader impact, than the impact factor of the journal the work is published in. They are doing the research to make a difference in a world they care about. Their goals are less individually focussed and are more for the broader benefits the work could bring. Working hard to be a success in disseminating research findings more broadly is a ‘nice to have’ in any fellowship or tenure application. Until there is much greater recognition of the overall benefits research brings then we will struggle to see men taking the time to think about the depth of their personal development. Nor will we see women recognised for what they do to disseminate their work, progress their own development or consider the development of the researchers that work with them.

The problem lies in the short term objectives in academic research. Academics who are able to dedicate a significant proportion of their time to generating data, publishing papers and writing grant applications are routinely rewarded. Whilst all of these activities are critical aspects of research success they only take an incidental attitude toward the development of the research students and research staff who are being mentored by that academic. And so perpetuates a selfishness in the academic world. Collaboration and collegiate activity do occur but these symbiotic relationships tend to be ephemeral because ultimately the rewards are given to individuals. With societal structures as they are and the timing of significant moves in academic careers coinciding with a time when starting a family is likely, women are significantly disadvantaged by this system.

Current University fellowships make no formal account of the contribution a researcher has made to the supporting undergraduate, masters or PhD students. There is little consideration of whether this candidate will continue to help others develop. The bottom-line of prestigious publications and grant income far outweigh those longer term goals that may see a paradigm shift to a much broader pool of contributors to research.


[i] https://www.hesa.ac.uk/news/23-01-2020/sb256-higher-education-staff-statistics

[ii] https://www.vitae.ac.uk/events/event-presenters

[iii] https://www.erasmusmagazine.nl/en/2019/03/12/female-academics-teach-more-lectures-and-conduct-less-research/