Reporting non-significant results

scatter plot not significant

Sorry for the long absence from blog posts. I’m going to try to ease my way back in with a short post that I’d like to pose as a question as much as offering any obvious solution.

I’ve just been in contact with a colleague at another University who was the external marker on a student’s work. We disagreed on the mark awarded but managed to quickly come to an amicable compromise. The work was well presented and clear. The issue was the student had not been absolutely clear about the results.

Whilst I had to agree that implying there might be significance in a finding where the data suggests otherwise is a major problem. However, I couldn’t help but feel some sympathy for the student. The requirement to report positive findings is so pervasive in scientific research that it is quite difficult to expect academics to instil into students the need to report with complete clarity and honesty on their findings.

Until there is a way to report results with neutrality in terms of the significance this will always favour outputs which support the researcher’s hypothesis. From a broader perspective I would suggest that the pre-registration of studies is a good way to battle against this bias. If researchers were recognised for the quality of the ideas that generate their hypotheses rather than whether the hypotheses are supported by the evidence they seek, that would be a great start.

We do however still have a system in which many journal editors are looking for data driven positive evidence that confirms a hypothesis. From an educational standpoint, I think it’s fair to make students aware that recognition has this bias. It still should be clear in their formative stage of producing work for educational assessment that a well explained non-significant finding can be worth the best grade awarded. If we fail to make this clear students will end up undertaking research projects where the result is so heavily anticipated that the research is hardly even worth doing.

Open Science: A Spotify for Research?

joke logo

If you are as old as I am you might remember buying music on vinyl records not because of the warmth of the sound or nostalgia, but because it was the only option. If you wanted to hear something new without purchasing it the local library would allow the loan of vinyl records as long as they were returned without scratches. The transition through various formats has brought us to a point where music aficionados can access most major musical outputs by streaming them directly, anything from classical overtures to death metal, and we are given the choice whether we accept the full output or just the single track we would like to hear. Similarly those of us who have been involved in academic research in a previous millennium will have remembered trips to a university library to leaf through bound copies of a monthly periodical. Whilst the Journals have become electronic and we don’t need to visit the library to access them, the publishers of academic journals are still acting like record labels; researchers who have published previously and have been cited frequently are strongly encouraged to submit their papers. Knowing an academic has produced popular work on a hot topic means they are likely to get the journal more citations and boost the impact factor, good news for both the publisher and the researcher. Starting out, a research career nearly always means teaming up with one of the established artists of research and the choice a prospective PhD student or postdoc makes can be the most critical aspect of their career success.

Open access publishing gives everyone with the internet access to research published through the open access journals. So in some respects open access is like Spotify for research papers but unlike Spotify the researchers themselves, or indirectly their funders, pay to release the papers. Whilst the returns for most of the musicians who put their work out on Spotify are minimal at least it is those who are in receipt of the goods that pay. When national governments are the funders of research they can make open access publishing a requirement and as long as the grants incorporate this funding there shouldn’t be too much trouble. The problem is the majority of research is funded by a variety of other sources (https://royalsociety.org/~/media/policy/projects/eu-uk-funding/uk-membership-of-eu.pdf). Charities might not want to spend more money just to give away the data and findings they have paid for, and industrial sponsors of research are probably quite keen to make sure the research can stay under-wraps until the findings can be exploited for the commercial gain they have invested to receive. The most problematic issue currently is the mixture of  types of journal, so even for data and findings that people want to share there is a ‘paid for’ and ‘pay to’ model. Appealing to the good nature of researchers to only publish in Open Access journals seems to be the current approach for the Open Science advocates, while the most morally upstanding and well-funded may see fit to do this, others who may be under significant pressures for their career might not feel so predisposed to act in community spirited altruism.

So what if there was one unified system where all publications could be submitted for free and institutions and individuals could pay a subscription to access all of that information, a Spotify for research? The economies of scale would be likely to mean the subscription charges could be kept to a level that institutions, research groups and maybe, even interested individuals could quite easily afford.

The current subscription model is already suffering an equivalent of home-taping, which incidentally didn’t kill the music industry, it probably didn’t even damage the profits of the major labels; when an academic sends a Twitter message to ask if someone in their network can share a pdf they are likely to get a positive response. This illicit sharing is as likely to end with the researcher asking their library if they can subscribe to the journal in the future as it is to the researcher asking the same associate to send multiple papers or issues of that journal.

There is an issue with Spotify for music, the most popular artists of our time receive a much greater proportion of the subscription revenue than when Record Labels paid artists by advances on their future albums. An artist who sells a 100,000 copies of their album on Spotify might only receive a fraction of what they need to live. However, the researchers don’t need the returns for paper views just that citations are recorded then this issue does not seem so important. Without the significance of the journal’s prestige influencing the number of citations then only the title, abstract and keywords will be needed to bring in the readers. Then it would really be down to the ingenuity and quality of the research that would determine the number of citations. Maybe the ‘Spotify of research’ does not need to stop there, it could become not only a repository for full papers but a place for academics to post their raw data and hypotheses, avoiding researchers hoarding together their findings for a paper of the greatest significance. The ideas and the data could still be cited showing the impact that the work has had. A thoroughly comprehensive system might even look to adapt the patent system so attribution for contribution and reward for ideas rather than resources could be made, but that’s for another time maybe.

The fundamentals of this system are already in place, Scopus and PubMed provide links to all the journal articles but are still just a conduit for the publishers to be paid for the journals that aren’t open access. By demanding a subscription for PubMed and this being used to pay the publishers a transition could be reached. Any journal that did not accept the terms offered would not have their journals on the system. It may require some brinksmanship in the first instance but ultimately less of the money for research would be diverted to a system that is a hangover from a less free information age. You never know there might even be a nostalgic resurgence of print copy journals that researchers can have on shelves in their office or go to a library to leaf through.

Dr Richy Hetherington

Lecturer, Graduate School, Faculty of Medical Sciences, Newcastle University