Tag: Meta-Research

Posted on

Reporting non-significant results

scatter plot not significant

Sorry for the long absence from blog posts. I’m going to try to ease my way back in with a short post that I’d like to pose as a question as much as offering any obvious solution.

I’ve just been in contact with a colleague at another University who was the external marker on a student’s work. We disagreed on the mark awarded but managed to quickly come to an amicable compromise. The work was well presented and clear. The issue was the student had not been absolutely clear about the results.

Whilst I had to agree that implying there might be significance in a finding where the data suggests otherwise is a major problem. However, I couldn’t help but feel some sympathy for the student. The requirement to report positive findings is so pervasive in scientific research that it is quite difficult to expect academics to instil into students the need to report with complete clarity and honesty on their findings.

Until there is a way to report results with neutrality in terms of the significance this will always favour outputs which support the researcher’s hypothesis. From a broader perspective I would suggest that the pre-registration of studies is a good way to battle against this bias. If researchers were recognised for the quality of the ideas that generate their hypotheses rather than whether the hypotheses are supported by the evidence they seek, that would be a great start.

We do however still have a system in which many journal editors are looking for data driven positive evidence that confirms a hypothesis. From an educational standpoint, I think it’s fair to make students aware that recognition has this bias. It still should be clear in their formative stage of producing work for educational assessment that a well explained non-significant finding can be worth the best grade awarded. If we fail to make this clear students will end up undertaking research projects where the result is so heavily anticipated that the research is hardly even worth doing.

Posted on

Support and Care in Academia is Female Dominated but Poorly Rewarded

vitae conference 2019

Vitae, the umbrella organisation that considers the professional development of academic researchers in the UK, hosts an annual conference each year it attracts over a thousand delegates, well in a normal year, at least. While across academic disciplines male professors outnumber their female counterparts nearly three to one[i]. The professors that have presented at this event (n=55) were majority female (55%). What is more striking was considering all presenters (n=600), where 70% were female[ii], with the same proportions for those using Dr. as their salutation, this markedly differs from academia as a whole where differences in subject area balance out to a near fifty-fifty split in gender. It seems that the people who have dedicated their careers to supporting the development of researchers, or at least have a keen interest in researcher development are more likely to be female. This is no bad thing in itself but the role comes with less recognition than the academic researchers they are there to support. The majority of those professors presenting have achieved that position in another field and have taken a sideways step to consider the development of colleagues. If working in researcher development or at least being interested in the development of others is not adequately rewarded or considered a key aspect of an academics role then research will ultimately suffer.

Data from Universities in the Netherlands[iii] corroborates HESA data indicating female academics are more likely to have greater teaching responsibilities than their male counterparts, who are engaging more in research activities, I’m doubtful other parts of the world trend differently. It quite clearly isn’t an inability of women to conduct great research but a system that allows more structure and a marginally better work life balance in teaching roles. The freedom for men to work 12 hour days when the job requires is there in society and although most wouldn’t want a work-life balance so skewed, some will take it to make strides at an early point in their career. They can do this in the knowledge that in the future they can tell their PhD students and post-docs this is the way it has to be if they want to progress. The dichotomy between teaching and research does not help this situation. Giving more academics a broader role may help to promote a better transfer of the cutting edge of research to our undergraduate and master’s students but it would also ground the research focussed staff in the structure of the University and broaden the opportunities for a contribution to research.

Amongst PhD students the difference between the typical attitudes of men and women to their own development is palpable. At a recent Public Engagement competition (Three Minute Thesis) the female students out-numbered the male competitors fifteen to one. Had the numbers been reversed and a list of eight men was presented as the finalists it would have likely caused some significant consternation, it certainly would have been dropped as a data-point from any Athena Swan application. The prevalence of women in this competition is however symptomatic of the larger issue. Women do care that their research makes a broader impact, than the impact factor of the journal the work is published in. They are doing the research to make a difference in a world they care about. Their goals are less individually focussed and are more for the broader benefits the work could bring. Working hard to be a success in disseminating research findings more broadly is a ‘nice to have’ in any fellowship or tenure application. Until there is much greater recognition of the overall benefits research brings then we will struggle to see men taking the time to think about the depth of their personal development. Nor will we see women recognised for what they do to disseminate their work, progress their own development or consider the development of the researchers that work with them.

The problem lies in the short term objectives in academic research. Academics who are able to dedicate a significant proportion of their time to generating data, publishing papers and writing grant applications are routinely rewarded. Whilst all of these activities are critical aspects of research success they only take an incidental attitude toward the development of the research students and research staff who are being mentored by that academic. And so perpetuates a selfishness in the academic world. Collaboration and collegiate activity do occur but these symbiotic relationships tend to be ephemeral because ultimately the rewards are given to individuals. With societal structures as they are and the timing of significant moves in academic careers coinciding with a time when starting a family is likely, women are significantly disadvantaged by this system.

Current University fellowships make no formal account of the contribution a researcher has made to the supporting undergraduate, masters or PhD students. There is little consideration of whether this candidate will continue to help others develop. The bottom-line of prestigious publications and grant income far outweigh those longer term goals that may see a paradigm shift to a much broader pool of contributors to research.

[i] https://www.hesa.ac.uk/news/23-01-2020/sb256-higher-education-staff-statistics

[ii] https://www.vitae.ac.uk/events/event-presenters

[iii] https://www.erasmusmagazine.nl/en/2019/03/12/female-academics-teach-more-lectures-and-conduct-less-research/

Posted on

Open Science: A Spotify for Research?

joke logo

If you are as old as I am you might remember buying music on vinyl records not because of the warmth of the sound or nostalgia, but because it was the only option. If you wanted to hear something new without purchasing it the local library would allow the loan of vinyl records as long as they were returned without scratches. The transition through various formats has brought us to a point where music aficionados can access most major musical outputs by streaming them directly, anything from classical overtures to death metal, and we are given the choice whether we accept the full output or just the single track we would like to hear. Similarly those of us who have been involved in academic research in a previous millennium will have remembered trips to a university library to leaf through bound copies of a monthly periodical. Whilst the Journals have become electronic and we don’t need to visit the library to access them, the publishers of academic journals are still acting like record labels; researchers who have published previously and have been cited frequently are strongly encouraged to submit their papers. Knowing an academic has produced popular work on a hot topic means they are likely to get the journal more citations and boost the impact factor, good news for both the publisher and the researcher. Starting out, a research career nearly always means teaming up with one of the established artists of research and the choice a prospective PhD student or postdoc makes can be the most critical aspect of their career success.

Open access publishing gives everyone with the internet access to research published through the open access journals. So in some respects open access is like Spotify for research papers but unlike Spotify the researchers themselves, or indirectly their funders, pay to release the papers. Whilst the returns for most of the musicians who put their work out on Spotify are minimal at least it is those who are in receipt of the goods that pay. When national governments are the funders of research they can make open access publishing a requirement and as long as the grants incorporate this funding there shouldn’t be too much trouble. The problem is the majority of research is funded by a variety of other sources (https://royalsociety.org/~/media/policy/projects/eu-uk-funding/uk-membership-of-eu.pdf). Charities might not want to spend more money just to give away the data and findings they have paid for, and industrial sponsors of research are probably quite keen to make sure the research can stay under-wraps until the findings can be exploited for the commercial gain they have invested to receive. The most problematic issue currently is the mixture of  types of journal, so even for data and findings that people want to share there is a ‘paid for’ and ‘pay to’ model. Appealing to the good nature of researchers to only publish in Open Access journals seems to be the current approach for the Open Science advocates, while the most morally upstanding and well-funded may see fit to do this, others who may be under significant pressures for their career might not feel so predisposed to act in community spirited altruism.

So what if there was one unified system where all publications could be submitted for free and institutions and individuals could pay a subscription to access all of that information, a Spotify for research? The economies of scale would be likely to mean the subscription charges could be kept to a level that institutions, research groups and maybe, even interested individuals could quite easily afford.

The current subscription model is already suffering an equivalent of home-taping, which incidentally didn’t kill the music industry, it probably didn’t even damage the profits of the major labels; when an academic sends a Twitter message to ask if someone in their network can share a pdf they are likely to get a positive response. This illicit sharing is as likely to end with the researcher asking their library if they can subscribe to the journal in the future as it is to the researcher asking the same associate to send multiple papers or issues of that journal.

There is an issue with Spotify for music, the most popular artists of our time receive a much greater proportion of the subscription revenue than when Record Labels paid artists by advances on their future albums. An artist who sells a 100,000 copies of their album on Spotify might only receive a fraction of what they need to live. However, the researchers don’t need the returns for paper views just that citations are recorded then this issue does not seem so important. Without the significance of the journal’s prestige influencing the number of citations then only the title, abstract and keywords will be needed to bring in the readers. Then it would really be down to the ingenuity and quality of the research that would determine the number of citations. Maybe the ‘Spotify of research’ does not need to stop there, it could become not only a repository for full papers but a place for academics to post their raw data and hypotheses, avoiding researchers hoarding together their findings for a paper of the greatest significance. The ideas and the data could still be cited showing the impact that the work has had. A thoroughly comprehensive system might even look to adapt the patent system so attribution for contribution and reward for ideas rather than resources could be made, but that’s for another time maybe.

The fundamentals of this system are already in place, Scopus and PubMed provide links to all the journal articles but are still just a conduit for the publishers to be paid for the journals that aren’t open access. By demanding a subscription for PubMed and this being used to pay the publishers a transition could be reached. Any journal that did not accept the terms offered would not have their journals on the system. It may require some brinksmanship in the first instance but ultimately less of the money for research would be diverted to a system that is a hangover from a less free information age. You never know there might even be a nostalgic resurgence of print copy journals that researchers can have on shelves in their office or go to a library to leaf through.

Dr Richy Hetherington

Lecturer, Graduate School, Faculty of Medical Sciences, Newcastle University