On 26 November 2021 I wrote to Toby Lasserson, Cochrane’s Deputy Editor-in-Chief, copying in in Gill Leng the Chief Executive Officer of NICE. I received a short response from Cochrane on 9 December. Today, 31 December, I wrote to Gill Leng, outgoing CEO of NICE copying in the incoming CEO of NICE, and other relevant stakeholders.
Dear Professor Leng
cc Dr Samantha Roberts, incoming CEO, NICE; Paul Chrisp Director, Centre for Guidelines, NICE; Judith Brodie, Interim CEO, Cochrane; Ken Stein, Director of the NIHR Evidence Synthesis Programme/support for NICE Technology Appraisal Programme.
I am writing in a personal capacity, not as a representative of my employer.
I have attached the email I sent to Toby Lasserson*, Deputy Editor-in-Chief of Cochrane at the end of November [1]. I copied you in as the email was in response to the news of the Cochrane/NICE collaboration [2]. I received a short response from Cochrane in December (also attached) [3].
One of my requests – to remove the flags falsely indicating the two Cochrane reviews [4,5] had been used to inform the recent NICE guideline for ME/CFS – has now been carried out. None of the other issues raised have been adequately addressed.
I have commented on sections of the response from Cochrane quoted below
As you will be aware, Cochrane continues to support the ongoing review of exercise therapy with a new review team, in conjunction with an Independent Advisory Group… The decision to retain the amended [exercise] review from 2019 will not be reconsidered by the Editor in Chief
As Toby said in news article about the NICE/Cochrane collaboration, “Sharing evidence with NICE is part of a broader approach to help Cochrane to identify priority reviews, and to identify them early” [2]
Why, then, are Cochrane spending time and public money to produce a new review of the evidence for exercise therapy for ME/CFS, when NICE has just produced its own evidence reviews covering all interventions for the condition? In NICE’s comprehensive review, you found that most of the evidence supporting current potential treatment options, including for exercise, was of very-low quality.
Cochrane is keeping its 2019 Exercise review [4] (and the 2008 CBT review [5] (which now has a “do not use for clinical decision making” notice on it) live on the Cochrane Library. The exercise therapy review concludes “Exercise therapy probably has a positive effect on fatigue in adults with CFS compared to usual care or passive therapies”. The conclusion is based on a GRADE rating of “moderate” quality evidence of a “non-zero” effect on a single outcome – fatigue at end of treatment.
Keeping the exercise review on the Cochrane Library has meant that critics of the new NICE guideline for ME/CFS, which include many prominent and influential Cochrane and GRADE working group members, can continue to use it to criticise NICE for “…a disastrous misapplication of GRADE methodology” [6].
If NICE use Cochrane to supply systematic review evidence in the future, you will no longer have oversight or control on how GRADE is applied by Cochrane review teams. Many of those longstanding Cochrane and GRADE working group members, so openly critical of NICE’s review methods, are also in leadership roles within Cochrane and are therefore guiding and teaching review teams on all aspects of conducting systematic reviews, including how to apply GRADE.
NICE have recognised the crucial importance of equipoise when conducting systematic reviews of the evidence by developing robust procedures for separating the review conduct and evidence quality assessment processes from the Guideline Development Committees who make recommendations based on the resulting evidence reports. There is no such mechanism in the Cochrane review process.
The NICE Guideline Committee for ME/CFS, unlike Cochrane, also acknowledge the limitations of evidence from trials which focus on subjective outcomes when neither patients nor therapists can be blinded to treatment allocation [7, 8].
It was a shocking revelation, via Freedom of Information, that a person who is on first name terms with you and also acknowledged to be connected with the Royal College of Psychiatrists, put pressure on you to alter the NICE review evidence reports after recommendations based on those reports had been agreed and signed off by the Guideline Development Committee [9]
As a charity, Cochrane is not subject to Freedom of Information requests. Any pressure from those with an interest in the results of a review applied during production or after publication will be even more difficult to detect. It is only if the pressure is applied to or by a person from a publicly regulated body that it is possible to uncover it via Freedom of Information [10].
Finally, the response from Cochrane to my email concludes that NICE…
…have confirmed that details of the collaborative approach to support the use of both Cochrane reviews and the topic expertise of Cochrane review authors, groups and networks in the development of NICE guidelines will be included the planned update to the NICE Guidelines Manual expected in 2022
How will the update to NICE guidelines manual ensure that evidence reviews supplied by Cochrane, or other evidence synthesis suppliers, have not been influenced by people with vested interests in review findings? I fear that using an unregulated organisation like Cochrane, who are not subject to FOI, and who stonewall reasonable criticism of its reviews, would be a false economy.
Rather than deliver the hoped-for time efficiencies and value for money, it could further obscure corrupt forces at work within the clinical research establishment. This could result in recommendations based on biased or corrupted appraisals of the evidence. This is not in the best interests of patients.
With best wishes
Caroline Struthers
* I address Toby by his first name as I know him personally from when I worked for Cochrane 2008-2014
1. Letter to Toby Lasserson, Deputy Editor-in-Chief, Cochrane, regarding Cochrane/NICE collaboration https://healthycontrol.org/2021/11/26/letter-to-toby-lasserson-cochrane/
2. Interview: Cochrane and NICE collaborate to improve health guidelines https://www.cochrane.org/news/interview-cochrane-and-nice-collaborate-improve-health-guidelines
3 Reply to message addressed to the Deputy Editor in Chief of The Cochrane Library. https://docs.google.com/document/d/1LWHEJtu3idPq5HIiTHSgAha_EY0JlzOWsLqaYHES8ps/edit?usp=sharing
4. Larun L, Brurberg KG, Odgaard-Jensen J, Price JR. Exercise therapy for chronic fatigue syndrome. Cochrane Database of Systematic Reviews 2019, Issue 10. Art. No.: CD003200. DOI: 10.1002/14651858.CD003200.pub8
5. Price JR, Mitchell E, Tidy E, Hunot V. Cognitive behaviour therapy for chronic fatigue syndrome in adults. Cochrane Database of Systematic Reviews 2008, Issue 3. Art. No.: CD001027. DOI: 10.1002/14651858.CD001027.pub2
6. Busse et al. Rapid response to: Updated NICE guidance on chronic fatigue syndrome https://www.bmj.com/content/371/bmj.m4774/rr-7
7. Tack M. Rapid response to: Updated NICE guidance on chronic fatigue syndrome https://www.bmj.com/content/371/bmj.m4774/rr-9
8. Tack M, Tuller DM, Struthers C. Bias caused by reliance on patient-reported outcome measures in non-blinded randomized trials: an in-depth look at exercise therapy for chronic fatigue syndrome. Fatigue: Biomedicine, Health & Behavior. 2020;8:181–92.
9. Text-message lobbying of senior NICE staff by individuals at NHS England and the Royal College of Psychiatrists in days before ME/CFS guideline pause https://domsalisbury.github.io/mecfs/nice-mecfs-guideline-pause/
10. Correspondence between Cochrane and Atle Fretheim, Norwegian Institute of Public Health/Gordon Guyatt, GRADE Working Group, McMaster 24 May-17 June 2019 https://drive.google.com/file/d/1-OWWCxS8IL39NKp3wqvLdcot6cpGQc73/view?usp=sharing
Caroline Struthers
Senior EQUATOR Research Fellow
UK EQUATOR Centre, University of Oxford, United Kingdom
caroline.struthers@csm.ox.ac.uk | www.ndorms.ox.ac.uk/team/caroline-struthers