On 27 September I submitted a rapid response to the BMJ article RoB 2: a revised tool for assessing risk of bias in randomised trials. On the advice of their lawyers, they wouldn’t publish it until I took out references to the PACE Trial and the Cochrane Exercise Review. I have posted the original version below. The link to the published version is here
*********************************************************************************
At the end of 2013 there was a debate held at the Cochrane Methods Symposium in Quebec on whether funding source should be a part of the Cochrane Risk of Bias tool. Lisa Bero argued in favour [1] and Jonathan Sterne against [2]. Sterne’s argument concluded that to add funding source as a standard item would send a negative message and would stop collaborative work with industry to help address problems with their research. It is not only industry research that has serious problems with bias which need to be addressed. Much greater attention should be paid to the way self-interest of all researchers, both inside and outside industry, can bias published research findings.
Both the old and revised version of the RoB tool focus on identifying bias using information published in journal articles, and by liaising with triallists. The bias affecting decisions taken before data collection is not considered. There is also no systematic assessment of bias affecting review authors’ decisions when conducting a review.
Triallists from both industry and academia often have financial success and reputation invested in a favourable result. These conflicts will affect many decisions including the choice of outcomes, and outcome measures, and are likely to flatter a favoured intervention. If things still don’t seem to be going the right way, triallists are given considerable leeway to make post-hoc adjustments, particularly if they are eminent clinicians and/or experienced triallists from prestigious institutions.
Review authors can also have their own conflicts. A Cochrane systematic review Exercise therapy for Chronic Fatigue Syndrome [3] is being revised after extensive criticism, validated by an internal Cochrane report [4]. The authors altered the way results for the review’s primary outcome were presented, changing from a null result at follow-up to a largely positive result. They also chose to stick to the largely subjective outcome measures specified by authors of the previous version. This is despite the fact than none of the included trials could be blinded. The reviewers also incorrectly gave a low risk of bias assessment for selective reporting for one of the included studies, the PACE Trial [5]. The PACE triallists controversially deviated from protocol-specified criteria for primary and secondary outcomes. They dropped entirely an objective measure of participant activity levels after they were informed another research group had had null results with this outcome. It was only after a costly legal battle that results for the trial’s pre-specified outcome measures were released [6, 7].
Unblinded trials are unlikely to yield useful data unless both self-report and objective outcome measures are used [8]. In this case the Cochrane researchers chose to stick with subjective outcomes. This is also selective outcome reporting bias, but it is ignored because the RoB tool assesses risk of bias in included studies only. Unblinded trials using outcomes prone bias can be classed as having a low risk of bias if the assessors decide that it was not “likely that assessment of the outcome was influenced by knowledge of intervention received”. RoB 2 gives increased leeway for judging individual outcomes at low risk of bias, and will further obscure serious problems which affect the reliability of trial data.
The review authors in this case have rejected valid criticism of their risk of bias assessment and many other problems with the review. They have defended what the previous Cochrane Editor-in-Chief called an “over-optimistic” assessment of the intervention, the success of which appears to be tied to their academic reputations. This assessment was achieved as a result of numerous methodological failings, including the incorrect assessment of the PACE Trial as at low risk of bias for selective reporting, and the ill-justified reliance on subjective outcomes.
RoB 2 has an explicitly stated objective of increasing the number of trials/outcomes receiving low risk of bias assessments. Individual outcomes can now be rated as having a low risk of selective reporting bias if protocol deviations occurred prior to data unblinding. There is no provision made for the fact that in unblinded trials, early indications of results can easily be discerned. Considering the problems afflicting important areas of medical research, this is a backward step in Cochrane’s stated mission to identify and summarise only the most reliable research evidence. It seems likely that the use of RoB 2 will further weaken the hand of those wishing to challenge over-optimistic evaluations of the evidence by recommending reviewers ignore important indications that the conduct of included studies was influenced by the pursuit of a positive result.
An important part of systematic reviewers’ role is to expose biased research practices to ensure patients are properly informed of any potential problems which lead to endorsement of treatments that could be ineffective or harmful. RoB 2 seems likely to do a worse job of identifying poor primary research practices, such as relying on subjective outcomes in unblinded trials. If RoB 1 had been correctly applied, the PACE Trial would have been given a high risk of bias assessment for selective reporting. RoB 2 will allow review authors to judge the risk of selective reporting bias as low with even less justification.
Going back to Bero’s argument that information about funding sources should be included in reviews, I would go further. Reviews should include systematically collected objective meta-data about included studies. Information could include funding source, researcher allegiance (e.g. involvement in development of intervention), balance of subjective/objective measures, patient involvement (e.g. in choice of outcomes). This would give readers a more complete overview of previous research including positive (e.g. patient involvement) and negative factors (e.g. inappropriate outcome choice) affecting the risk of bias. This would help illustrate how reputational conflicts of interest can also lead to serious bias towards findings which help bolster academic and clinical careers rather than protect patients from ineffective or harmful treatments.
[1] https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.ED000075/full
[2] https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.ED000076/full
[3] https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD003200.pub7/full
[4] http://www.virology.ws/wp-content/uploads/2019/03/Cochrane-Report-on-Courtney-Complaint-1.pdf
[5] https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(11)60096-2/fulltext
[6] https://www.tandfonline.com/doi/full/10.1080/21641846.2017.1259724
[7] https://bmcpsychology.biomedcentral.com/articles/10.1186/s40359-018-0218-3
[8] https://academic.oup.com/ije/article/43/4/1272/2952051