Sorry not to be in regular blogging mode at the moment. Here’s a video of our evidence session to parliament, where they are running an inquiry into research integrity. I think clinical trials are the best possible way to approach this issue. Lots of things in “research integrity” are hard to capture in hard logical rules,  so you end up with waffly “concordats” and rules that are applied inconsistently. With clinical trials you can make clear rules, you can measure compliance, and you can enforce compliance. There is lots of chat about this in the video below from 17:37 with me, Simon Kolstoe (who did this neat audit using ethics committee data), and Sile Lane from the AllTrials campaign.

It’s a very interesting inquiry overall. In the session before you can see Ivan Oransky from RetractionWatch and others. You can read more about the inquiry overall here, and see lots of written submissions here. I’ve posted my submission below. Cheers!

Written evidence submitted to House of Commons Science and Technology Committee inquiry on Research Integrity, March 2017

Dr Ben Goldacre

I am a medical doctor and academic. I run the Evidence-Based Medicine DataLab at the University of Oxford, with a small team of ten people, where we make tools that turn medical data into actionable insights. I also work on public engagement, wrote the “Bad Science” column in the Guardian for a decade, and have sold over half a million books on problems in science and medicine. I am a co-founder of AllTrials.net, the global campaign for all clinical trial results to be reported. In policy work I conducted an independent external review on evidence-based policy for DfE, co-authored a Cabinet Office paper on randomised trials in policymaking, and have sat on various advisory groups for MoJ, NHS England, and DfE.

1. Executive Summary

Fraud is not the most important issue. The culture of incomplete and inaccurate reporting of research has greater impact on patients and society, and can be addressed. Clinical trials are the “canary in the cage”. Trials are large expensive research projects used to generate knowledge that is then used, in clinical practice, to make vitally important decisions; and yet trials are commonly left unreported, or misreported. This is a waste of money, and distorts the evidence underpinning medical practice. Non-reporting of clinical trials has been left unaddressed for three decades. The government could take simple action: require all trials to be reported; support audit that identifies which trials have been left unreported; and fund research into what will improve standards. Improvements and innovation in this area will generalise to the rest of science.

1. The extent of the research integrity problem

1.1 Fraud is not the most important issue.

Fraud is problematic, and our mechanisms to identify and address it remain poor; however fraud is also relatively infrequent, and universally recognised as unacceptable. More urgent is the widespread culture of incomplete and inaccurate reporting of scientific research; the culture of permissiveness and denialism around this; and the failure of government, public funders, and public institutions to take simple steps to improve the situation.

1.2 Non-reported studies.

The results of clinical trials are used by clinicians, patients, health services and policymakers to make informed choices about which interventions are most effective. However there is extensive and longstanding evidence that the methods and results of clinical trials are routinely left unavailable. To assess non-publication researchers take a list of trials known to have been completed; they then check some time later whether the results have been made available, usually by searching academic journals (where doctors and researchers would generally expect to find trial results), or sometimes by searching in other less well known repositories. A 2014 systematic review1 of all the literature on this question summarises 17 studies following up cohorts of trials approved by ethics committees: overall 46% of trials were published. They also found 22 studies on cohorts of trial listed in public registries: 54% were published. Studies with statistically significant results were three times more likely to be published, which is consistent with previous work2.

Furthermore, it is impossible to get a perfect assessment of how many trials are unreported, because we do not know what trials have been conducted: a 2015 cohort study on all drugs approved in 2012 found that only 57% of trials were registered3; and in our team’s experience of conducting similar research, ethics committees can refuse to disclose their list of trials that have been approved. Furthermore we have repeatedly found evidence that the information entered into trial registries is incomplete and inaccurate.

1.3 Mis-reported studies

Related to the problem of non-reported trials is that of mis-reported trials, and the selective reporting of results from within one study. It is very common for researchers to record a large number of different outcomes during the conduct of a trial (such as blood tests, hospital admissions, symptom rating scales, and so on); and then only report some of the results at the end. This is an important source of bias, and can lead to exaggeration or misrepresentation of the true effects of a treatment.  

There are various systems in place to prevent this practice. I will describe these systems and their failure in some detail, as they represent a clear example of how we have failed to address shortcomings in research, despite extensive discussion, guidelines, and investment in infrastructure. Before carrying out a clinical trial, all measured outcomes are supposed to be pre-specified in a publicly accessible trial protocol, and the trial’s registry entry. The trial report is supposed to contain results for all these pre-specified outcomes, or declare and explain any deviations. The importance of reporting all prespecified outcomes and documenting changes is emphasised in numerous authoritative locations including the International Conference on Harmonisation of Good Clinical Practice (ICH-GCP) 4, and the CONSORT guidelines on trial reporting5 which are widely respected and endorsed by over 500 individually named academic journals6. It is one of the core purposes of trial registration, itself supported by organisations including the WHO, the International Committee of Medical Journal Editors, and various national laws and regulations.

However, despite near universal recognition of the importance of complete outcome reporting, trial reports in academic journals routinely breach these norms. A 2015 systematic review 7 found 27 studies comparing prespecified outcomes against those reported, in cohorts of between 1 and 198 published trials. The median proportion of trials with a discrepancy on primary outcomes alone was 31%.

Because over 500 journals have committed to ensure all trials are correctly reported, and yet the same journals continue to mis-report trials, we ran a project monitoring every trial in five major journals, and sent a correction letter, for publication, on every mis-reported trial. This was not to repeat the finding that trials are misreported, but rather to assess whether the self-correction mechanisms of science are operating correctly. We found that they are not. Most letters were rejected, and the responses from trialists and editors demonstrated that many do not understand the issue of correct outcome reporting, or are not concerned by breaches, despite their public stance to the contrary, and despite all journals in our study being listed as endorsing the CONSORT guidelines, which require complete outcome reporting. I am happy to share more details on this project prior to its final publication, all data is shared online at www.COMPare-trials.org

2. The impact of these ongoing problems

We have collectively failed to address these issues throughout the ecosystem of science and medicine: this includes funders, regulators, researchers, journal editors, professional bodies, and more. The immediate burden is clear. In pure science, research spend is wasted. In applied sciences such as medicine, the impact is more acute. We cannot make informed choices about which treatment works best when the results of clinical trials are routinely and legally withheld from doctors, researchers, and patients. There is also an equally important reputational hit to science. The public are increasingly aware that serious problems have been left unaddressed: that trial results are routinely withheld, that there has been little serious effective effort to fix the issue over decades, that the biggest players in the ecosystem of scientific research are not taking adequate action. Faced with this reality patients and the public will conclude, to a greater or lesser extent, that science is structurally incompetent, or even somewhat corrupt. Our permissiveness and failure to put our own house in order lays fertile ground for quacks, anti-vaccination conspiracy theorists, and climate change denialists. Science needs to earn and retain its reputation for generating accurate knowledge about the world.

3. The effectiveness of controls/regulation (formal and informal), and what further measures if any are needed

Neither soft “culture change” interventions nor legislation have been effective to date. Sharing the methods and results of all trials has been recognised as an ethical and scientific imperative for many years8–10. More recently institutions such as the World Health Organization (WHO)11, the European Commission12, and the US Food and Drug Administration (FDA)13 have called for results disclosure. The first data on non-publication in medical trials was published in 19869; and yet three decades later the data shows that trial results are still routinely left unreported. In the US, the FDA Amendment Act 2007 (FDAAA) requires sponsors to post results within 12 months for a subset of clinical trials. Two cohort studies have now explored compliance with the broad intent of FDAAA and found compliance rates of only one trial in five 14,15; however, the formal implementation of this legislation has been delayed by a decade, and the first trials to be legally covered by it will not complete until 2018. The UK government could address this very simply, with the steps set out below. There are innumerable regulations, edicts, reports, guidelines, and strategy documents around trial reporting. None have been enforced or implemented, and breaches are not documented. This creates a dual burden: it creates the illusion that problems have been fixed, when in reality they are ongoing, and so removes the sense of urgency around improving standards, leaving a false sense of reassurance.

4. What can be done.

We have seen poor progress in addressing these issues. In my view this is driven by a small number of key factors, all of which can be reversed.

4.1 Generate better data on breaches

We can only improve quality by measuring performance. Trial publication is a very simple thing to measure. We conducted an audit of publication of all trials in Oxford and helped identify areas for improvement16.  Our team is also creating a number of global “Trials Tracker” dashboards comparing publication performance by various metrics (although this is currently made needlessly difficult by poor quality information on trials in databases and registries). Audit is a positive process. It identifies systemic problems, and regions where more targeted education or assistance is required. It identifies areas where systems are functioning well, so that others can learn from them. The very process of public audit itself can also help to drive up standards: there is currently almost no accountability for those who do not report their trials, or mis-report their trials, whether they are publicly funded, or funded by industry.

The government could and should simply require all UK research funders to publicly declare all trials they have funded, and whether those trials have published their results, with performance statistics reported, and all individual reported and unreported trials identified. Several funders committed to produce summary statistics on the proportion of trials reported, among those they have funded, at this Committee’s previous inquiry into non-publication of trial results. To my knowledge, four years later, none of these audits have been made publicly available.

The government could also require all ethics committees to generate data on non-reporting with no great increase in administrative burden. At present, all trials in the UK need to be approved by an ethics committee. Currently ethics committees approve research, and then effectively lose contact. The government could simply require all ethics committees to request an end of study report on all approved trials, and publicly disclose whether each study is reported. In related work, we have found that ethics committees are reluctant to share their deliberations on trials with questionable designs: this material should also be routinely publicly available17. Overall, we should have a national programme of audit on all clinical trials conducted in the UK, and report performance on whether trials are reported, broken down by institution, funder, researcher, disease area, and so on18. This could be conducted by institutions, or ideally independently.

It is worth noting that many US universities now have staff nominated to work on ensuring that all trials are registered, with their results reported within 12 months of completion. These staff take responsibility for monitoring compliance, and supporting researchers who require assistance. They are organised under an umbrella organisation: the “Clinical Trials Registration and Results Reporting Taskforce.” The government should require UK universities and funders to replicate this model.

4.2 Create a clear legislative requirement to register and report all trials

The UK government should require that all trials conducted in the UK are registered, and their full methods and results publicly reported. There is new EU legislation requiring this, but only for drug trials, and the UK is leaving the EU. Previously it has been suggested that taking a strong line on this issue would undermine our ability to attract international clinical trials business to the UK. This is mistaken. The UK share of the global clinical trials market is shrinking annually because of cheaper options in low- and middle-income countries, and Eastern Europe. We cannot compete on price. We should not even consider competing on weaker regulation. We can, however, compete on quality. Good quality evidence is about rigour, and the credibility that comes from this rigour. If we make it clear that UK trials are the most reliable in the world – that they produce more reliable evidence than other territories – then we will attract trials, and help drive up standards for science globally.

4.3 Legitimise and Fund Work on Research Integrity

Working on issues around research integrity is currently regarded, in derisory terms, as either a “hobby” outside a researcher’s real work; or as somehow radical, transgressive, or trouble-making. This dismissive attitude reflects negatively on the profession. The situation arises, in part, because there is essentially no funding in the UK to work on problems around research integrity. That is a startling omission for an issue so central to the whole of science: it reflects badly on the discipline and the community, and is surely connected to the very poor progress seen across all these issues. Without funding on research and interventions to improve research integrity, we cannot be surprised that the problems persist.

The Committee could improve this situation by communicating clearly that work on research integrity is a legitimate academic pursuit; and by encouraging government agencies and public institutions to cooperate with efforts to document the extent of the problem. If we are to make concrete progress, the Committee should also request funders to to create specific funding streams on research integrity. This would help to achieve five important goals:

• Provide funding to develop and assess new interventions to improve research reporting and research integrity
• Provide pilot funding to develop infrastructure to improve reporting, such as “Registered Reports”, or new reporting platforms.
• Provide implementation funding for interventions shown to improve reporting such as the guidelines of EQUATOR network, and related pro-active offshoots.
• Support research to better document the extent and character of the problem
• Support independent audit to compare individual institutions and sponsors, and drive up standards by identifying areas requiring the most improvement

Research integrity is a global problem, and work in this area has positive global impact. If we do not fund work on research integrity we must expect all the problems documented by the Committee to persist. This is an active choice by government and funders.

References

1. Schmucker, C. et al. Extent of Non-Publication in Cohorts of Studies Approved by Research Ethics Committees or Included in Trial Registries. PLoS One 9, e114023 (2014).
2. Song, F. et al. Dissemination and publication of research findings: an updated review of related biases. Health Technol. Assess. 14, iii, ix–xi, 1–193 (2010).
3. Miller, J. E., Korn, D. & Ross, J. S. Clinical trial registration, reporting, publication and FDAAA compliance: a cross-sectional analysis and ranking of new drugs approved by the FDA in 2012. BMJ Open 5, e009758 (2015).
4. Good Clinical Practice : ICH. Available at: www.ich.org/products/guidelines/efficacy/efficacy-single/article/good-clinical-practice.html. (Accessed: 1st September 2016)
5. Schulz, K. F., Altman, D. G., Moher, D. & CONSORT Group. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMJ 340, c332 (2010).
6. Consort – Endorsers. Available at: www.consort-statement.org/about-consort/endorsers. (Accessed: 1st September 2016)
7. Jones, C. W., Keil, L. G., Holland, W. C., Caughey, M. C. & Platts-Mills, T. F. Comparison of registered and published outcomes in randomized controlled trials: a systematic review. BMC Med. 13, 282 (2015).
8. Chalmers, I. Underreporting research is scientific misconduct. JAMA 263, 1405–1408 (1990).
9. Simes, R. J. Publication bias: the case for an international registry of clinical trials. J. Clin. Oncol. 4, 1529–1541 (1986).
10. World Medical Association. World Medical Association Declaration of Helsinki: ethical principles for medical research involving human subjects. JAMA 310, 2191–2194 (2013).
11. Moorthy, V. S., Ghassan, K., Vannice, K. S. & Marie-Paule, K. Rationale for WHO’s New Position Calling for Prompt Reporting and Public Disclosure of Interventional Clinical Trial Results. PLoS Med. 12, e1001819 (2015).
12. European Medicines Agency. Posting of clinical trial summary results in European Clinical Trials Database (EudraCT) to become mandatory for sponsors as of 21 July 2014. (2014). Available at: www.ema.europa.eu/ema/index.jsp?curl=pages/news_and_events/news/2014/06/news_detail_002127.jsp&mid=WC0b01ac058004d5c1. (Accessed: 28th October 2016)
13. FDA. Food and Drug Administration Amendments Act (FDAAA) of 2007. (2007). Available at: www.fda.gov/RegulatoryInformation/Legislation/SignificantAmendmentstotheFDCAct/FoodandDrugAdministrationAmendmentsActof2007/ucm095442.htm. (Accessed: 28th October 2016)
14. Anderson, M. L. et al. Compliance with results reporting at ClinicalTrials.gov. N. Engl. J. Med. 372, 1031–1039 (2015).
15. Prayle, A. P., Hurley, M. N. & Smyth, A. R. Compliance with mandatory reporting of clinical trial results on ClinicalTrials.gov: cross sectional study. BMJ 344, d7373 (2012).
16. Tompson, A. C., Petit-Zeman, S., Goldacre, B. & Heneghan, C. J. Getting our house in order: an audit of the registration and publication of clinical trials supported by the National Institute for Health Research Oxford Biomedical Research Centre and the Musculoskeletal Biomedical Research Unit. BMJ Open 6, e009285 (2016).
17. Mendel, J., Goldacre, B., Ernst, E. & Whittle, S. Problems with ethical approval and how to fix them: lessons from three trials in rheumatoid arthritis. BMJ 354, i4626 (2016).
18. Goldacre, B. How to get all trials reported: audit, better data, and individual accountability. PLoS Med. 12, e1001821 (2015).