After a lot of to-ing and fro-ing and fruitless correspondence with the Cochrane Editorial Unit by me and many others over years, I was asked yesterday to submit a summary of my complaint about this review to the Governing Board of Cochrane. This is it.
Cochrane Review: Exercise Therapy for Chronic Fatigue Syndrome
1. Avoidable ideological and reputational conflicts of interest – including protecting the reputation of Cochrane over protecting patients – has led to the publication of a review which says, wrongly, that Graded Exercise Therapy (GET), is safe and effective for people with ME/CFS. There is in fact no good evidence that it is effective, and some evidence that it’s harmful.
2. The review authors chose to look only subjective outcomes when none of the included studies could be blinded. They took advice on the conduct of the review from Peter White, PI of the main included study, the discredited PACE trial (https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(11)60096-2/fulltext). He is acknowledged and thanked for his advice on the review.
3. Using only subjective outcomes in unblindable trials is terrible methodology. This is even more inexplicable in a review. And avoidable. Some of the included trials, including PACE, did measure objective outcomes, such as employment levels. They didn’t support the positive findings from subjective measures.
4. Peter White and co-authors of the PACE trial developed and use GET in their clinical practice, in accordance with their belief in the biopsychosocial model of treating ME/CFS. They also had financial ties to the insurance industry. The PACE trial was partially funded by the Department for Work and Pensions and happily for them, the “moderately positive” results of the PACE trial on these subjective measures meant they could stop people’s benefits if they refused to have GET. There were many other well-documented flaws in the conduct of PACE (outcome switching, changing measurement boundaries, not declaring conflict of interest in patient information sheets, marketing GET as an effective treatment to study participants….and many more). Cochrane judged it as a “well-conducted trial” in the Cochrane Clinical Answer of treatments for ME/CFS
5. The PACE trial is now being used as a teaching example of how not to do a randomised trial
6. The management of the review by the Common Mental Disorders Group underlines the biased assumption that ME/CFS can be treated/cured using treatments designed around the biopsychosocial model for treating a mental illness. ME/CFS is not a mental illness.
7. A person with ME has been in correspondence with David Tovey and Rachel Churchill since before the review was originally published in 2014 warning of the huge problems with it. There was no apparent involvement of patients in the review, despite offers of input well in advance of publication. As I am in friendly correspondence with this person, I can supply you with all the correspondence and notes from meetings with David Tovey on request.
8. Cochrane Editors ignored the detailed feedback on the huge methodological flaws and conflict of interest influencing the conduct of the review, and the conflicts of the authors of included studies, preferring to bat it back to the review authors several times via the feedback system.
9. One patient (Robert Courtney) who provided detailed, rigorous and irrefutable criticism via the feedback system once the review was published received such a poor response from the author he made an official complaint. This is now being investigated by David Tovey. Robert sadly took his own life earlier this year before knowing that any action was being taken. I am not intending to imply there is a causal link, I am stating a fact. The effort Robert took to assemble the feedback and the complaint cannot be overestimated, and there would have been a huge physical and emotional toll over the years of trying to get an adequate response from Cochrane.
10. I have been corresponding with David Tovey for some time on this and other issues to no avail. The last time he wrote to me in early September he said this issue in particular was a high priority having spoken to the investigative journalist David Tuller, and to the person with ME I mentioned in point 7.
David Tuller has been covering the flawed PACE trial for some time and has also specifically written about the Cochrane review – examples below
Trial By Error: Cochrane Decides to Withdraw Flawed Exercise Review
11. In August 2018 the BMJ wrote an article about PACE https://www.bmj.com/content/362/bmj.k3621 with 14 rapid responses https://www.bmj.com/content/362/bmj.k3621/rapid-responses
12. In October 2018, David Tovey said he was going to withdraw the review pending investigation. But this investigation turned out to be sending it back to the authors again. Inaccurate coverage by Reuters said the withdrawal, which never happened, was in response to patient complaints https://uk.reuters.com/article/us-health-chronicfatigue-dispute/exclusive-science-journal-to-withdraw-chronic-fatigue-review-amid-patient-activist-complaints-idUKKCN1MR2PI . In the article, David denied it was due to patient pressure, but not before various clinicians and medical establishment figures who hadn’t bothered to read the review or look into the history back-lashed against patients calling them anti-science. It seems that David Tovey has again caved to pressure from powerful academics and has not withdrawn the review as he promised. He has put a note on it saying the authors are working on another response to complaints. I think this has been possibly worse than doing nothing at all.
13. Why did the critique of the HPV review prompt an “urgent investigation”, but Mark Vink’s critique of this review didn’t? https://journals.sagepub.com/doi/full/10.1177/2055102918805187 .
14. There was a three hour session in Parliament on research into treatments for ME/CFS in June, and the MP Carol Monaghan stated that when the full truth about the PACE Trials comes out it will be one of the biggest medical scandals of the 21st Century
https://www.parliamentlive.tv/Event/Index/3d63c39d-18f3-4cd7-8509-c91785dced98.
15. Now all the PACE authors, plus a prominent member of Cochrane and proponent of exercise as a useful treatment for CFS, are writing a review of exercise therapy for CFS using individual patient data. This review has not yet been published because of scathing peer and consumer comments on the draft.
https://www.cochrane.org/CD011040/DEPRESSN_exercise-therapy-chronic-fatigue-syndrome-individual-patient-data
This review has had a catastrophic effect on people with ME who are still being forced to have an ineffective and potentially harmful treatment. Imagine if it was a drug or a vaccine with even a fraction of the evidence of harm, combined with such poor evidence of benefit? People assume that Cochrane is independent, has high scientific standards and is vigilant and robust about dealing with conflict of interest. In my view, and certainly in this case, this assumption is incorrect.
This review should be retracted and Cochrane should issue a full unreserved apology to patients.
Thank you for setting out the main problems with the Cochrane exercise review for ME/CFS.
It is problematic to know where to stop listing the myriad problems with this review, but I personally would have included the fact the no studies using the Oxford Criteria definition of CFS should have any place in consideration of relevant treatments.
The Oxford definition is so broad that we have no idea what percentage of subjects actually had ME/CFS.
Thanks Peter. Very true. It’s a fundamental and important point. It is likely that any benefit (subjectively or objectively measured) seen in trials using this criteria would be for people who didn’t have ME/CFS.
The case of clinical criteria is only a side note as there is still a lack of evidence for objective improvements in activity levels, concentration ability, employment etc regardless of criteria used.
Likewise negative effects on these aspects were not adequately monitored in most studies either, though far more participants in the PACE trial were receiving welfare/income benefits after the trial than before, regardless of the intervention arm.
I agree with Peter that this is a substantial issue. In its 2014 US Agency for HealthCare Research and Quality evidence review, AHRQ lumped together all CFS and ME studies and determined that GET was moderately effective.
But AHRQ also noted that the Oxford definition should be retired because it “results in a high risk of including patients who may have an alternate fatiguing illness or whose illness resolves spontaneously with time.”
The 2014 US Institute of Medicine report also noted that based on definitional differences, “the diagnostic criteria for ME have required the presence of specific or different symptoms from those required by the diagnostic criteria for CFS; thus, a diagnosis of CFS is not equivalent to a diagnosis of ME.”
Because of these two points, AHRQ did a 2016 reanalysis of the efficacy of GET after excluding Oxford studies and concluded that there was “insufficient evidence of the effectiveness of GET on any outcome.” AHRQ also concluded that studies of CBT and GET using more specific case definitions that require hallmarks of the disease such as post-exertional malaise were “blatantly missing.”
Cochrane’s GET evidence review is basing recommendations for people who have hallmark symptoms of the disease based on studies in cohorts who may not have. Cochrane’s response to Kindlon on the AHRQ point on Oxford is not adequate (see below). Cochrane needs to do what AHRQ did and reevaluate the evidence after excluding Oxford studies. But right now, its not clear that they will
Mary Dimmock
Cochrane’s response to Kindlon’s point on Oxford.
The availability of relevant trials limits which subgroup analyses are possible to carry out in a systematic review, and hence, we were only able to contrast CDC versus Oxford criteria and found no evidence for a difference.