Below is the text of a formal complaint I made in March this year.
Unlike previous complaints – I’ve been plugging away since 2018 – this one was specifically about the actions of the Editor-in-Chief. It focuses on her decision to allow the authors of the Exercise for CFS review to update the review, instead of agreeing that the review could be withdrawn.
Evidence from correspondence obtained via a freedom of information request shows that the Cochrane Editor-in-Chief failed to ensure recommended clarifications about the lack of important benefit of exercise were made to the updated review. These recommendations were made by the father of evidence-based medicine, Gordon Guyatt. The authors had promised to comply with his recommendations, but they did not.
As a result, the review is still misleading, contradicts the latest recommendations of NICE, but is still being used to inform clinical practice guidelines all over the world, such as the Royal Australian College of GPs
In April, the CEO recognized the need for an independent investigation. But two weeks later, the tone changed and there was going to be no investigation at all, independent or otherwise. Case closed.
Below is the full complaint, with the attachments linked.
Catherine Spencer Chief Executive Officer Cochrane, 11-13 Cavendish Square, London W1G 0AN United Kingdom
23 March 2023
Dear Catherine
Formal complaint about Karla Soares-Weiser
I am writing in a personal capacity and not as a representative of my employer
For simplicity, this complaint will focus on two avoidable and serious errors in the Exercise Therapy for Chronic Fatigue Syndrome Review for which Karla Soares-Weiser is ultimately responsible.
- The risk of bias judgement on selective reporting for White 2012 is still “low risk” when it should have been changed to “high risk”
- The evidence quality rating for fatigue at end of treatment is “moderate” when it should be “low”
These and many other errors mean that following the findings of the review could lead to patient harm which is grounds for withdrawal of the review.
Serious error in a Cochrane Review. Following the conclusions of the published review could result in harm to patients or populations of interest (other than known adverse effects); and/or there are factual errors in describing one or more included studies that risk misinforming implementation or investment decisions about an intervention; and/or the reported treatment effect is inconsistent with the real effect shown in the reported data
I asked for the review to be withdrawn for this reason in 2020. The request was denied because a Cochrane internal committee decided that there were no serious errors in the review.
The attached correspondence obtained via Freedom of Information shows how Karla Soares-Weiser prioritized the wishes of the review authors over the quality and integrity of a Cochrane review, and over patient safety. Her actions have led to a flawed, misleading, and out of date review remaining on the Cochrane Library. This has caused confusion to users of Cochrane evidence because it contradicts the findings of a NICE evidence review, and harm to patients because the findings lead people to believe exercise may be a useful treatment for ME/CFS, and now Long Covid, when there is no good evidence that it is, and there is also evidence (documented by NICE) that it may be harmful.
The review of Exercise Therapy for Chronic Fatigue Syndrome (now known as ME/CFS) was originally published in 2013 and is now in its fifth iteration with many revisions of each version. The latest version was published in October 2019 along with the announcement that a new review with a new protocol was a high priority and work would start on it early in 2020.
None of the several detailed critical comments published on the current review were adequately addressed by the authors. One critic, Robert Courtney, eventually submitted a formal complaint on 12 February 2018. This was investigated by Cochrane, and Cochrane produced a report on 10 April 2018 (attached) upholding the all the points made by Courtney and recommending the actions which needed to be taken.
A key action included in the report was for the authors to change the risk of bias judgement on selective outcome reporting for White 2011 to “high risk”.
Comments on feedback submitted by Robert Courtney PACE Trial selective reporting bias
This judgement [of low risk] is not justifiable and should be changed to `high risk’
Screening result – further action
Authors must address all the points in this report or provide a reasonable explanation for why they are not necessary
On 15 October 2018, then Editor-in-Chief David Tovey and Senior Editor Rachel Churchill wrote to the lead author Lillebeth Larun saying
“We are … temporarily withdrawing your review to allow you and your co-authors time to adequately address the feedback received.
Consequently, your review will shortly be removed from the Cochrane Library.”
The authors protested and forced Tovey to keep the review on the Cochrane Library whilst they made the changes. This took another seven months.
They submitted an amended version of the review in May 2019.
Tovey said once again, that the review was not up to standard and that it should be withdrawn. He had even prepared the withdrawal statement
I have quoted relevant extracts from the correspondence.
David Tovey 24 May 2019:
Thank you for your re-submission…. I am afraid that I am not able to publish this version
I asked my internal team to review the submission, and also three senior colleagues who I believe have the expertise and detachment required to provide an objective evaluation. As the attached documents show, all of these assessors continue to have important concerns
In particular, I remain concerned that the presentation in general, but in particular of the short term results on fatigue, remains too optimistic.
I am very grateful for the work you have done to revise this document and have no wish to cause you further distress, and therefore I suggest that the communications run along these lines:
‘Cochrane and the review author team have been unable to agree on revisions to the review, and therefore, with regret, it will now be withdrawn from the Cochrane Database of Systematic Reviews.
The Cochrane authors are not in agreement with this decision. The review authors are now free to publish this revised version elsewhere. In the future, Cochrane may decide to commission a replication of this review or a separate review on this topic’
I have not seen the internal review and comments from Tovey’s senior colleagues. I imagine as well as the particular concern about the over- optimistic presentation of short-term results on fatigue, Cochrane Editors may also have questioned the fact that the authors did not change the risk of bias judgement on selective outcome reporting for White 2011 from “low risk” to “high risk” and the justification for this, if given, was not adequate
Atle Fretheim, then Director of Research and Innovation at the Norwegian Institute of Public Health, representing the authors, argues against the withdrawal and Tovey replies (my emphasis)
David Tovey 27 May 2019:
When a collection of experienced and dispassionate colleagues are all making, in effect, the same criticisms, it is hard to ignore this. You are right that I said that I didn’t want to fall out over one increment on a GRADE rating, but the rating in question jars with me every time I see it. I can see three possible reasons for a downgrade: lack of blinding/subjective outcomes, imprecision, and inconsistency, so the conclusion that this is moderate certainty evidence seems indefensible to me, and as we know, I am not alone in this. As you know, there are others that would add selective outcome reporting to this list. It seems absolutely clear to me that this is low certainty evidence
29 May 2019:
I certainly consider that the grading of moderate is impossible to defend for the reasons outlined in the previous email
Tovey retired from Cochrane at the end of May 2019. and handed over responsibility for the review to his successor Karla Soares-Weiser.
Andy Oxman, Editor at Cochrane Norway, colleague of Fretheim and the review authors at the Norwegian Institute of Public Health, and member of the GRADE working group, suggested Gordon Guyatt as an independent arbiter. Fretheim agrees to abide by Guyatt’s decision on whether the rating can stay as “moderate” or be downgraded to “low” as Tovey and his Cochrane senior colleagues all wished.
I don’t know if Gordon Guyatt was aware that the risk of bias rating for selective reporting for White 2011 had not been changed to “high risk”. However, he seemed to have no problem with keeping the “moderate” rating if the authors made it clear in the review that the effect size was not clinically important (euphemistically a “non-zero” effect). He emphasized this in two separate emails.
Gordon Guyatt 14 June 2019
…if we agree we rating out (sic) certainty in there being a non-zero effect then there is no reason to rate down for imprecision and the overall certainty would be moderate. Were this the case, all language in abstract and elsewhere should be consistent with that certainty rating and the discussion might usefully include a statement that there is no claim being made that an important effect exists
15 June 2019
…the judgment of moderate is contingent on rating certainty on a non- zero effect.
This implies that the authors do not claim an important effect and should avoid any language that claims an important effect. It would be reasonable to require in limitations that they highlight that they have not established an important effect
However, two days later, there is obviously no clarification in the review text that the authors have not established an important effect. Karla Soares-Weiser writes to Freheim to check whether the judgement is based on a non-zero effect. If the Editor-in-Chief can’t tell from the text, and must ask for clarification, how on earth are readers of the review meant to get this crucially important message which Guyatt had advised should be made crystal clear?
Karla Soares-Weiser 17 June 2019
Could you please confirm whether the judgement is based on rating certainty on a non-zero effect? I am very keen to finalise this process
Atle Fretheim 17 June 2019
Since the authors have already agreed to delete all words that indicate anything about effect size, this means that we are talking about a non-zero effect
Rather than highlighting the lack of important effect, as advised twice by Guyatt, they chose to include one single mention of it in a footnote of the Summary of Findings table saying “The estimate remains consistent with a non‐zero effect size”.
Soares-Weiser, in her keenness to finalize the process, sacrificed clarity, truth and the safety of patients to appease the authors.
The truth is that exercise therapy has no important effect on fatigue either in the short or long term. This truth has been deliberately obfuscated by Cochrane, and the conclusion of the review is that “Exercise therapy probably reduces fatigue after 12‐26 weeks”
The plain language summary also states “Exercise therapy is recommended by treatment guidelines and often used as treatment for people with chronic fatigue syndrome”
The first part of this statement is no longer true, certainly in the UK, as NICE published updated its guidelines in 2021 and exercise is no longer recommended. Using GRADE, NICE reviewers found the quality of the evidence for exercise was “low” or “very low”.
The second half of this statement may well be true. People who think Cochrane provide trustworthy information will still use the findings of the review in good faith, and doctors will continue to prescribe an ineffective and potentially unsafe treatment to people with ME/CFS, and now also to people with Long Covid.
The 2019 Cochrane review is now well over three and a half years out of date and contradicts NICE’s more recent review findings. This has caused confusion and led to prominent people associated with Cochrane to accuse NICE of
It is Cochrane not NICE who have allowed their authors to manipulate and misapply GRADE to maintain the desired “moderate” quality rating on one outcome when Cochrane Editors and other senior advisors had been recommending it be downgraded to “low” since 2018 or before.
Soares-Weiser also allowed the authors to ignore the explicit recommendation of the father of Evidence-Based-Medicine and co- developer of GRADE, Gordon Guyatt, that this “moderate” rating was contingent on highlighting that it was for an unimportant effect.
Please confirm that there will be an independent investigation of my complaint, as outlined in my previous correspondence with you attached.
Please inform me about who has been appointed to conduct the investigation and their credentials as independent.
I would also appreciate an estimate of the time it will take to conduct the investigation.
The promised update review to replace what was acknowledged, even by Cochrane, to be a flawed one was started three and a half years ago and has not even reached protocol stage. There has been no update on progress since 2021 despite promises of monthly updates.
Please withdraw this out of date and misleading review. I am looking forward to hearing from you
Caroline Struthers Senior Research Fellow University of Oxford UK