On 26 November 2021 I wrote to Toby Lasserson, Cochrane’s Deputy Editor-in-Chief, copying in in Gill Leng the Chief Executive Officer of NICE. My letter was prompted by the announcement that NICE and Cochrane have signed collaborative agreement. Among a few other issues, I am seeing reassurance that NICE will manage the risk posed by potentially delegating important decisions about the quality of evidence influencing NICE’s recommendations to an outside, self-regulated organisation.
From: Caroline Struthers
Sent: 26 November 2021 09:54
Subject: Cochrane/NICE collaboration
Attachments: NICE enq ref: EH-319164 :RE: Clinical Knowledge Summary Tiredness in Adults
Dear Toby, (cc Gill Leng)
I am writing in a personal capacity, not as a representative of my employer.
1. Guidelines informed by Cochrane reviews
In the news item (1) about Cochrane’s new collaboration with NICE, you say “Cochrane has always recognised the importance of the evidence it produces for guideline development, and our reviews are flagged in this way on the Cochrane Library. It also means that we know we are producing high priority reviews for stakeholders”
The new ME/CFS guideline (NICE (2021) Myalgic encephalomyelitis (or encephalopathy)/chronic fatigue syndrome: diagnosis and management NG206 (2) was published on 29 October. The two relevant Cochrane reviews, Exercise as treatment for adults with chronic fatigue syndrome (3) and Cognitive behaviour therapy for chronic fatigue syndrome (4) could not be used by NICE to help them develop the guideline because they “…did not include all critical outcomes specified in this review protocol and included study populations where not all participants had ME/CFS” (5). However, both reviews are still flagged as being used in the development of both the new NICE guideline for ME/CFS and also the NICE Clinical Knowledge Summary for tiredness/fatigue in adults (6)
Flags to guidelines informed by the exercise review (7)
Flags to guidelines informed by the CBT review (8)
You will see from the email correspondence with NICE attached, the references to both reviews have now been removed from the Clinical Knowledge Summary.
- Please will you therefore remove the flags indicating these reviews have been used to inform the new 2021 NICE guideline for ME/CFS and the NICE Clinical Knowledge Summary, and let me know when this has been done.
I presume the developers of the other three guidelines flagged in the CBT review have not yet been made aware that there is now a note on the review saying that it should not be used for clinical decision making.
- Please can you confirm when you or the UK Cochrane Centre have contacted the developers to inform them of this, and asked them to amend their guideline accordingly? Then please remove the links to these guidelines from the review. Again, please let me know when this process has been completed.
2. Inappropriate listing of related reviews and protocols, and allocation of topics
The list of related protocols and reviews on both reviews lists around ten reviews of drug and non-drug therapies for mental disorders such as anxiety, depression and PTSD. There also seems to be little overlap between the lists. The list on the CBT review includes the review on Chinese herbal medicine which has been withdrawn and the protocol for the IPD review of exercise which has also been withdrawn. Topics given on the CBT review include Insurance Medicine and the subtopics of mental health. ME/CFS is a condition of unclear aetiology but has been classified by the WHO as a disorder of the nervous system (9) since 1969.
- Please could you correct the list of related reviews and protocols and topics for both these reviews to ensure they are consistent with each other, and remove any link with mental health conditions? Please let me know when this has been done.
3. Confusion and potential harm to patients caused by non-withdrawal of misleading reviews
There are thousands of patients and clinicians without a ME/CFS clinical guideline for their country or health system who will look to Cochrane reviews directly for information and guidance. There are doubtless also other recommendations and guidelines, not flagged by Cochrane explicitly, which rely on their findings; findings which now directly contradict the new NICE guidelines for ME/CFS. There is recent evidence of the confusion caused by these reviews remaining on the Cochrane Library in the stakeholder comments on the NICE guideline draft published in November 2020 (10, 11). For example, one comment from the Royal College of Psychiatrists said “It is unhelpful for clinicians to have evidence from two reputable sources pointing in opposite directions” (p. 1064 consultation-comments-and-responses-4). I recently discovered another Cochrane-branded version of the Exercise review (12) published in 2017, copyright Royal College of Psychiatrists. This publication includes none of the caveats of the 2019 Cochrane review amendment, such as the fact that any evidence of benefit is for a non-zero effect rather than a clinically meaningful one, the limited applicability of the evidence to people with ME/CFS, the lack of evidence of long-term effectiveness, and the lack of information about potential harms which were acknowledged in the statement made by Karla Soares-Weiser (13).
Responses from NICE to stakeholder comments challenging NICE’s decision not to use the Cochrane reviews are repeated numerous times in both comments and responses documents. I counted how many times the three statements (below) were repeated in just one of them (consultation-comments-and-responses-4) (10)
Repeated 8 times: With reference to Larun 2017: This Cochrane review looked at exercise therapy versus passive controls or other active treatments in adults with ‘CFS’. The main reasons for exclusion from evidence review G are as follows: The approach to meta-analysis was different to our approach. All exercise therapies were pooled regardless of the type of exercise therapy delivered, and comparators considered ‘passive’ control arms (treatment as usual, relaxation or flexibility) were also pooled. We did not consider this to be appropriate for the purposes of decision-making for this guideline. Additionally, the following critical outcomes were not assessed (not primary or secondary outcomes for the review): cognitive function, activity levels, return to school/work, exercise performance measures, and mortality. However, all studies included in this Cochrane review were included in our review.
Repeated 11 times: We note that the Cochrane review ‘Exercise therapy for chronic fatigue syndrome’ (Larun et al., 2019) is contested and that it ‘is still based on a research question and a set of methods from 2002, and reflects evidence from studies that applied definitions of ME/CFS from the 1990s’ (https://www.cochrane.org/news/cfs) The review is currently undergoing a full update
Repeated 7 times: Price 2008: This Cochrane review looked at CBT versus usual care or other interventions in adults with ‘CFS’. The main reasons for exclusion are as follows: Studies with mixed populations where at least 90% of participants had a primary diagnosis of CFS were included. The committee agreed it was important that all participants in included reviews were diagnosed with ME/CFS. Additionally, the following critical outcomes were not assessed (not primary or secondary outcomes for the review): cognitive function, pain, sleep quality, activity levels, exercise performance, and mortality. It is also worth noting that Cochrane has stated that this review is no longer current and should not be used for clinical decision making.
- Please will you reconsider your decision and withdraw the reviews on the grounds given in Cochrane’s review withdrawal policy (14) that following the conclusions of the published review could result in harm to patients.
4. Withdrawal or flagging of pre-GRADE Cochrane reviews
Regarding the warning note now on Price 2008, which was noted by NICE in its responses to stakeholders, I presume you are already in the process of, or will soon be starting work on, identifying all Cochrane reviews which also predate the mandatory use of GRADE methodology, and will place an editorial note on them to say they should not be used for clinical decision‐making. However, I would hope most authors who are no longer able to fulfil their obligation to keep their reviews up to date would agree to have their review withdrawn rather than it remain on the library providing out-of-date and potentially misleading information which could harm patients.
- Please let me know if this work has started yet, and if not, when you will start.
5. Inappropriate use of GRADE by Cochrane reviewers
Since publication of the NICE ME/CFS guideline, several prominent and influential GRADE and Cochrane contributors have criticised NICE both for not using the Cochrane reviews, and for using GRADE incorrectly. For example, Peter White, a named advisor on the Cochrane exercise review, said “NICE have banned graded exercise therapy, in spite of it being found to be helpful in a major Cochrane systematic review” (15). Paul Garner, a co-founder and influential member of Cochrane, said on Twitter “Nice (sic) lost the plot with GRADE with ME/CFS, according to the people that actually developed the methods: “a disastrous misapplication of GRADE methodology is the source of the problem” (16). Garner also describes Live Landmark, a Norwegian PhD student and Lighting Process practitioner as “Somebody being honest about the perversion of the GRADE process by Nice (sic) on ME/CFS” (17)
The Cochrane exercise review team used GRADE inappropriately to avoid downgrading certainty of evidence from “moderate” as advised by then Editor-in-Chief David Tovey. In the correspondence obtained via FOI (18), David clearly wished to withdraw the review, rather than keep the certainty rating as moderate. He remarked “I can see three possible reasons for a downgrade: lack of blinding/subjective outcomes, imprecision, and inconsistency, so the conclusion that this is moderate certainty evidence seems indefensible to me, and as we know, I am not alone in this” (page 23). Having refused either to allow withdrawal of the review, or downgrade the certainty of the evidence as advised by Dr Tovey, the review authors added an inconspicuous note (b) under the first summary of findings table (19) explaining why they hadn’t downgraded the certainty of the evidence for inconsistency: “Inconsistency (certainty not downgraded): we chose not to downgrade because all studies gave the same direction and because the observed heterogeneity (80%) was mainly caused by a single outlier. The estimate remains consistent with a non‐zero effect size (SMD −0.44; 95% CI ‐0.63 to ‐0.24) also when the outlier is excluded”. A non-zero effect is clinically meaningless. This questionable use of GRADE was condoned by the incoming Cochrane Editor-in-Chief Karla Soares-Weiser and the GRADE expert acting as arbitrator. It allowed the review authors to keep a statement in the summary of findings table which said that “Exercise therapy probably reduces fatigue after 12‐26 weeks”.
With the new collaboration between NICE and Cochrane (20), how will NICE manage the risk of similar manipulation of GRADE by Cochrane review teams? Will NICE now defer to Cochrane for GRADE assessments?
- Please share any documentation which explains in more detail how the collaboration will work, and how Cochrane and NICE will provide reassurance that this serious risk is acknowledged, and will be managed transparently and effectively.
- https://www.cochrane.org/news/interview-cochrane-and-nice-collaborate-improve-health-guidelines
- Myalgic encephalomyelitis (or encephalopathy)/chronic fatigue syndrome: diagnosis and management https://www.nice.org.uk/guidance/ng206
- Larun L, Brurberg KG, Odgaard-Jensen J, Price JR. Exercise therapy for chronic fatigue syndrome. Cochrane Database of Systematic Reviews 2019, Issue 10. Art. No.: CD003200. DOI: 10.1002/14651858.CD003200.pub8
- Price JR, Mitchell E, Tidy E, Hunot V. Cognitive behaviour therapy for chronic fatigue syndrome in adults. Cochrane Database of Systematic Reviews 2008, Issue 3. Art. No.: CD001027. DOI: 10.1002/14651858.CD001027.pub2
- https://www.nice.org.uk/guidance/ng206/evidence/g-nonpharmacological-management-of-mecfs-pdf-9265183028
- https://cks.nice.org.uk/topics/tiredness-fatigue-in-adults/
- https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD003200.pub8/related-content#guidelines_data
- https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD001027.pub2/related-content#guidelines_data
- 8E49 Postviral fatigue syndrome https://icd.who.int/browse11/l-m/en#/http://id.who.int/icd/entity/569175314
- Consultation on draft guideline – stakeholder comments table 4 https://drive.google.com/file/d/1K36ZGo-Oi34qFTsKmk_IkM6CRPFcvWQ-/view?usp=sharing
- Consultation on draft guideline – stakeholder comments table 5 https://drive.google.com/file/d/1QWmK92iTnXW4VpRZZrTlbdFIMu2PtwGy/view?usp=sharing
- Larun, L., Brurberg, K., Odgaard-Jensen, J., & Price, J. (2017). Exercise therapy for chronic fatigue syndrome. BJPsych Advances, 23(3), 144-144. doi:10.1192/apt.23.3.144
- Publication of Cochrane Review: ‘Exercise therapy for chronic fatigue syndrome https://www.cochrane.org/news/cfs
- Withdrawal of published articles: Cochrane Reviews https://www.cochranelibrary.com/cdsr/editorial-policies#withdrawal
- https://www.sciencemediacentre.org/expert-reaction-to-updated-nice-guideline-on-diagnosis-and-management-of-me-cfs/
- https://twitter.com/PaulGarnerWoof/status/1462734200797384709
- https://twitter.com/PaulGarnerWoof/status/1462095868832665601
- Correspondence between Cochrane and NIPH 24 May-17 June 2019 https://drive.google.com/file/d/1-OWWCxS8IL39NKp3wqvLdcot6cpGQc73/view?usp=sharing
- Summary of findings 1. Exercise therapy versus control for chronic fatigue syndrome https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD003200.pub8/full#CD003200-tbl-0001
- https://www.nice.org.uk/news/article/nice-and-cochrane-sign-collaborative-agreement-to-deliver-living-guideline-recommendations
I look forward to hearing from you.
With best wishes
Caroline
Caroline Struthers
Senior EQUATOR Research Fellow, UK EQUATOR Centre, University of Oxford, United Kingdom
1 comment