What Kind of Bias Are Systemic Reviews at Risk for

• Research
• Open Access
• Published: 15 November 2019

Risk of bias tools in systematic reviews of health interventions: an assay of PROSPERO-registered protocols

Systematic Reviews volume 8, Article number:280 (2019) Cite this article

• 31k Accesses

• 53 Citations

• 16 Altmetric

• Metrics details

Abstract

Background

Systematic reviews of wellness interventions are increasingly incorporating evidence exterior of randomized controlled trials (RCT). While non-randomized study (NRS) types may be more prone to bias compared to RCT, the tools used to evaluate take a chance of bias (RoB) in NRS are less straightforward and no gilded standard tool exists. The objective of this study was to evaluate the planned use of RoB tools in systematic reviews of wellness interventions, specifically for reviews that planned to comprise evidence from RCT and/or NRS.

Methods

We evaluated a random sample of not-Cochrane protocols for systematic reviews of interventions registered in PROSPERO betwixt Jan i and Oct 12, 2018. For each protocol, we extracted information on the types of studies to be included (RCT and/or NRS) besides as the name and number of RoB tools planned to be used according to written report design. We then conducted a longitudinal assay of the almost usually reported tools in the random sample. Using keywords and name variants for each tool, we searched PROSPERO records past year since the inception of the database (2011 to December vii, 2018), restricting the keyword search to the "Risk of bias (quality) cess" field.

Results

In full, 471 randomly sampled PROSPERO protocols from 2018 were included in the analysis. Well-nigh two-thirds (63%) of these planned to include NRS, while 37% restricted study design to RCT or quasi-RCT. Over half of the protocols that planned to include NRS listed only a unmarried RoB tool, most frequently the Cochrane RoB Tool. The Newcastle-Ottawa Scale and ROBINS-I were the most commonly reported tools for NRS (39% and 33% respectively) for systematic reviews that planned to apply multiple RoB tools. Looking at trends over time, the planned utilize of the Cochrane RoB Tool and ROBINS-I seems to be increasing.

Conclusions

While RoB tool selection for RCT was consistent, with the Cochrane RoB Tool being the most frequently reported in PROSPERO protocols, RoB tools for NRS varied widely. Results suggest a need for more than education and awareness on the appropriate use of RoB tools for NRS. Given the heterogeneity of report designs comprising NRS, multiple RoB tools tailored to specific designs may be required.

Peer Review reports

Background

With the growing involvement in "real-world prove" obtained from analyzing authoritative health data and the development of sophisticated quasi-experimental report designs [i], regulatory agencies [2], and others who systematically review wellness interventions are increasingly incorporating non-randomized studies (NRS) into their evidence syntheses [3]. As such, methods to appraise the risk of bias, defined as the chance of systematic error in results or inferences [4], of these complex bear witness sources are now coming under closer scrutiny. The choice of risk of bias tools (RoB tools) is not straightforward for reviews of NRS, although methodological tools for assessing the risk of bias in randomized controlled trials (RCT) are more well-established, with the Cochrane Collaboration's RoB Tool [5] now considered the standard [6]. The concluding 2 decades have seen a proliferation of tools developed to evaluate the risk of bias in NRS; a 2012 systematic review identified 74 tools developed for quality appraisal, of which adventure of bias is a component, of non-experimental studies [vii]. However, none of these existing NRS quality appraisal tools are currently accepted as the gold standard [1, 8], and information technology is unclear which tools are the most rigorous and practical.

Quality appraisal for NRS is complicated by the heterogeneity of this category of study pattern. Under this umbrella term are a multitude of designs, including experimental studies (e.thousand., non-randomized controlled clinical trials), quasi-experimental studies (e.g., controlled before-after studies, interrupted fourth dimension serial), and traditional observational studies (e.thousand., accomplice, case-control, cantankerous-exclusive studies). NRS may be at higher risk of bias due to confounding compared to RCT [9]; still, a single checklist may non fairly assess the risks particular to the diverse types of NRS. For case, by studies have found that existing tools are insufficient for the evaluation of the risk of bias in pharmacoepidemiological prophylactic studies [10], natural experimental studies [11], and other quasi-experimental designs [12]. Moreover, if multiple checklists are used in systematic reviews that incorporate multiple study designs, review authors need to consider whether these tools are comparable, particularly in terms of rating evidence within a grading arrangement or when using a cutting-off to determine which studies to include in a systematic review or meta-analysis.

Two studies published in 2018 found a wide variation in the employ of RoB tools for NRS in published systematic reviews [13, xiv]. While the Newcastle-Ottawa Scale was the most frequently used tool for NRS in both studies, it was as well not uncommon for systematic reviews to utilise no RoB tools at all or to inappropriately apply tools intended for RCT. Further, Quigley et al. reviewed methodological recommendations from health applied science assessment bodies and ended that there is no consensus on which tool(due south) should be the standard of practise for appraising bias in NRS [13].

To our cognition, no previous study has assessed the apply of RoB tools past examining pre-published systematic review protocols, which may provide more than detailed methodological information compared to published systematic reviews. Evaluating protocols registered in PROSPERO, an "international prospective annals of systematic reviews," enables us to look frontwards into the future to anticipate emerging trends in RoB tools, as well equally look at historical trends in RoB tool use over time. Given the ongoing development of new RoB tools, certain tools may accept fallen out of favor or gained currency over time.

In the present written report, nosotros conducted a cross-sectional assay of systematic review protocols on health interventions registered in PROSPERO to identify which tools were the nigh ordinarily cited in 2018 to evaluate the risk of bias of RCT and NRS in systematic reviews. We also conducted a retrospective assay of trends in the use of these commonly cited RoB tools in protocols of health interventions registered in PROSPERO since database inception (2011). In the absence of a golden standard, identifying the almost common tools cited for apply would assist researchers position their RoB tool selection in the context of their peers. Knowing how RoB tools are applied in practice could as well inform future tool evolution or place areas where educational interventions on RoB tool use are needed.

Methods

Review of 2018 PROSPERO records

Information source and sample selection

The search for eligible protocols was conducted using PROSPERO's database filters for blazon and method of the review, source of the review, and date of improver to the database [search strategy: (Intervention):RT NOT Cochrane:DB WHERE CD FROM 01/01/2018 TO 12/10/2018]. To be included in this analysis, PROSPERO protocols had to exist for systematic reviews of health interventions. We excluded Cochrane review protocols because they were assumed to use Cochrane methodology and RoB tools. Protocols for rapid reviews were excluded as their approach to quality appraisal may exist different compared to full systematic reviews. Protocols for overviews of reviews (or "umbrella" reviews), reviews of guidelines, qualitative studies, preclinical studies, and economic evaluations were excluded as the take chances of bias cess for these study designs was exterior the scope of this study. Further, we selected protocols from merely the most contempo year available (2018) in guild to determine contemporary practices in the utilize of RoB tools. Retrieved records were screened by i reviewer (K.F.) for inclusion.

All PROSPERO records that met the appointment and database review type limits were downloaded on October 12, 2018. There were 4215 eligible protocols registered in PROSPERO from Jan ane to October 12, 2018. Of these protocols, 500 (approximately 10% of registered protocols) were randomly selected for practicality, equally the aim of this analysis was to place which RoB tools were the well-nigh usually cited in systematic review protocols in 2018 when this analysis was conducted. A simple random sample was created using the random number generator from RANDOM.org.

Data extraction

Data was extracted on the types of studies to exist included from each of the selected systematic review protocols. Protocols were then coded as including RCT (including quasi-RCT), NRS (including non-randomized experimental, quasi-experimental, or observational study designs), or both.

Data was besides extracted on all of the tools the protocol authors planned to employ for risk of bias assessment and, if specified, the study designs that the tools will be used to appraise. Since we wanted to understand what RoB tools authors were choosing to use for quality appraisement, we recorded tools co-ordinate to author intentions and regardless of whether the tools were specifically designed for this purpose. Nosotros recorded the systematic review using "suites of tools" in cases where the RoB tool was comprised of separate checklists for different report designs produced by the same organization, just the exact number of checklists to be used was not specified. For example, the Joanna Briggs Institute (JBI) produces a number of tools for appraising diverse study designs [15]. If authors just refer to JBI tools generally, it is unclear how many tools are beingness employed. We recorded the review using "multi-design tools" in cases where the RoB tool was designed to appraise both RCT and NRS, for case, the Downs and Black checklist [16]. If both RCT and NRS were to be included in the systematic review, we recorded whether the authors planned to use unlike tools for these designs, or whether they used a single tool for both types. If the authors stated that they were following Cochrane guidelines and simply included RCT, we assumed they were using the Cochrane RoB Tool. Data was extracted by ane reviewer (K.F.).

Longitudinal analysis of PROSPERO records

To determine usage trends over time of RoB tools that are in common contemporary use, nosotros searched PROSPERO records for the names of the near oft cited RoB tools identified in the above cross-exclusive analysis to decide how often each tool was mentioned on an annual ground. Nosotros assessed annual trends for tools that were named in 5 or more of the protocols included in the random sample of 2018 PROSPERO records. Tools that were not developed for take chances of bias cess, e.g., reporting guidelines, were excluded. Using keywords and name variants for each tool, we searched PROSPERO records past year since the inception of the database (2011) to December 7, 2018, restricting the keyword search to the "Risk of bias (quality) assessment" field. Searches were express to protocols for reviews of interventions. Cochrane review protocols were excluded, as it was assumed that they followed the risk of bias procedures outlined in the Cochrane Handbook. The number of records retrieved for each tool per year was recorded. We did not further verify the text of the protocol records. Tools were classified by the types of designs they were intended to assess: RCT only, NRS only, multi-design tools, and suites of tools.

Statistical analysis

Descriptive statistics were used to summarize the frequency and proportion of the RoB tools in the random sample and year-by-year assay of PROSPERO records.

Results

Included 2018 PROSPERO records

In total, 471 of the 500 PROSPERO protocols from the 2018 random sample were included in the final analysis. Twenty-5 protocols were excluded afterwards screening for not meeting pre-specified inclusion criteria and another four protocols were excluded for having unclear information on the types of studies to be included (come across catamenia diagram in Boosted file ane). Approximately 2-thirds (63%) of the protocols analyzed planned to include NRS, while the remaining 37% of protocols stated that they would limit the analysis to RCT or quasi-RCT. A small proportion of protocols, two% (x/471), did not anticipate finding any RCT given the nature of the topics, and 1 protocol specifically excluded RCT.

Run a risk of bias tools in PROPOSERO-registered protocols

The number of RoB tools listed in protocols according to the types of study designs included is presented in Table 1. Overall, 10% of protocols did non list whatsoever specific RoB tool. Over half of the protocols that planned to include NRS in addition to RCT listed merely a single RoB tool.

Table 1 Number of risk of bias tools listed by study designs included

Full size table

As shown in Tabular array 2, in protocols that listed just a unmarried RoB tool, the Cochrane RoB Tool was by far the virtually commonly cited tool in systematic review protocols including only RCT (85.ii%), and to a lesser extent, those including both RCT and NRS (35.6%). The Newcastle-Ottawa Calibration and Downs and Blackness were the next nigh common tools planned to be used in reviews including both RCT and NRS when a single RoB tool was planned to be used to assess studies. There was a wider variation in the RoB tools listed in reviews including both RCT and NRS compared with RCT only.

Table 2 PROSPERO protocols with a unmarried risk of bias tool listed

Full size table

Tabular array iii displays the specific tools mentioned in systematic review protocols that planned to include both RCT and NRS and to employ multiple RoB tools. Tools are listed past design, based on the protocol authors' intentions. When multiple RoB tools were planned to be used in a systematic review, the well-nigh commonly listed tool for assessing RCT was the Cochrane RoB Tool. In that location was express use of field of study-specific scales, such as the PEDro Scale for assessing studies of physiotherapy interventions. Few protocols specifically mentioned the revised Cochrane RoB 2 Tool. For NRS, the Newcastle-Ottawa Scale and ROBINS-I were the most frequently listed in reviews using multiple RoB tools.

Table three PROSPERO protocols listing multiple tools by study design

Full size table

A full count of all the RoB tools listed in the random sample of protocols is presented in Additional file 2.

Almanac trends in risk of bias tool employ in PROSPERO protocols

Fifteen specific RoB tools were listed at least five times in the included 2018 PROSPERO records. Of these, two were excluded (Cochrane Handbook and GRADE approach) because they were guidelines non tools designed for risk of bias assessment. We did non differentiate between the two versions of the Cochrane RoB Tool in the temporal trends analysis, since it was not technically possible in the PROPSERO search interface to search for the term "2.0," which would be used to identify the revised version of the tool. For ROBINS-I, keywords for the previous version of the tool, "A Cochrane Risk Of Bias Assessment Tool: for Non-Randomized Studies of Interventions" (ACROBAT-NRSI), were also included. A twelvemonth-by-year search of PROSPERO records was performed on these 12 RoB tools. The full search strategy is provided in Additional file 3.

The number of results in the protocols' "risk of bias" sections for each tool by year is provided in Boosted file 4. Of all the RoB tools, the Cochrane RoB Tool had, by far, the highest frequency of planned usage throughout the unabridged time period, mentioned in over 40% of records every year. Given the generic search terms used for this tool, it is possible these figures are inflated somewhat, but this blueprint of use is similarly seen in the random sample of 2018 PROSPERO records. Apply of the Cochrane RoB Tool also appears to be increasing over time, rising from 40.8 to 59.3% of protocols from 2011 to December 7, 2018 (see Fig. 1a).

Fig. 1

Trends over time for the almost oftentimes cited RoB tools in the included 2018 PROSPERO protocols past type of tool. Per centum of full not-Cochrane systematic review protocols on interventions in PROSPERO, by year, for tools for RCT (a), tools for NRS (b), multi-blueprint tools (c), and suites of tools (d). CASP = Critical Appraisal Skills Program; EPHPP = Effective Public Health Exercise Projection tool; JBI = Joanna Briggs Institute; MINORS = Methodological Index for Non-Randomized Studies; MMAT = Mixed Methods Assessment Tool; NHLBI = National Heart, Lung, and Blood Institute (National Institutes of Health); NOS = Newcastle-Ottawa Scale; NRS = non-randomized studies; PEDro = Physiotherapy Evidence Database; RCT = randomized controlled trial; RoB = run a risk of bias; ROBINS-I = Run a risk Of Bias In Non-randomized Studies - of Interventions. Limits: intervention reviews; exclude: Cochrane protocols; restrict to field: assessment of bias

Full size image

The Newcastle-Ottawa Calibration was the next most common tool of the 12 included for assay and was the most frequently mentioned RoB tool for NRS. Use of the ROBINS-I tools for NRS has increased since it was developed in 2015, ascension to 6.iv% of the total number of not-Cochrane protocols on wellness interventions in 2018 (see Fig. 1b).

Multi-design tools were the least commonly mentioned; all three multi-design tools had less than four% prevalence every year. Of the three multi-blueprint tools searched, the Downs and Black checklist appeared in the highest number of protocols throughout the years reviewed (see Fig. 1c).

Lastly, iii of the 12 tools were suites of tools. It was non possible to tell using the search results which specific checklist inside the suite was being referred to in the PROSPERO protocols. These suites of tools had low frequency of use (< five% of total records) throughout the entire fourth dimension period, with the exception of the JBI Critical Appraisal Tools in 2012 (see Fig. 1d).

Give-and-take

In this study, two-thirds of PROSPERO protocols on health interventions in the 2018 sample intended to include testify from NRS in addition to RCT, while the remaining protocols restricted to RCT only. When protocols were restricted to RCT, the option of RoB tool was highly consistent, with 85.2% planning to use the Cochrane RoB Tool. A few additional protocols (1.9%) planned to use Cochrane RoB 2 Tool, which was first introduced in 2016 as an update to the original Cochrane RoB Tool [20]; however, the uptake of Cochrane RoB 2 Tool may exist underestimated, equally authors may not have specified the version number in their protocol.

In protocols that intended to include both RCT and NRS, the choice of tools was more heterogeneous, consistent in finding with current stance that there is no consensus on the preferred tools for evaluating bias in NRS [3, 8, 12, thirteen]. This finding is also consistent with previous research from Seehra et al., which described quality appraisal tool use in systematic reviews as "varied and inconsistent" [xiv]. Just over half of protocols including both RCT and NRS listed only i tool for take a chance of bias assessment, well-nigh frequently the Cochrane RoB Tool, which was designed to appraise adventure of bias in RCT [36]. In a review of 686 systematic reviews, Quigley et al. found that RoB tools designed for RCT were frequently misapplied to NRS [13]. The choice to use a RoB tool for a study blueprint that it was not intended to be used for might exist made for several reasons, such as the convenience of using ane tool for multiple study designs, misinformation on appropriate RoB tools, or a lack of a gold standard RoB tool available for NRS. Information technology is also possible that authors had not planned on assessing the quality of NRS. For example, Briere et al. observed that many meta-analyses and health technology assessments using real-earth bear witness from NRS did not critically appraise these studies [3], and in Deeks et al.'s review of 511 systematic reviews that included NRS, only a third performed quality cess for NRS [37]. We likewise plant that some protocol authors were not being specific in identifying RoB tools a priori or were inappropriately applying tools to assess risk of bias for NRS. To compound the challenges in appraising the quality of NRS, most of the commonly cited RoB tools for NRS, such as the Newcastle-Ottawa Calibration, ROBINS-I, and MINORS, have non been sufficiently validated [13].

When systematic reviews that intended to include NRS planned to use multiple tools to assess risk of bias, the Newcastle-Ottawa Scale was the most commonly listed RoB tool to assess NRS (39%), followed by ROBINS-I (33%). Although some have pointed out that the Newcastle-Ottawa Scale has several weaknesses, including low inter-rater reliability [38] and "uncertain validity" of some items [39], this calibration appears to be the virtually pop choice of all the NRS tools and is considered easy to utilise [forty]. Both Quigley et al. and Seehra et al. as well institute that the Newcastle-Ottawa Scale was the most frequently used tool to appraise risk of bias in NRS. In the trend assay of commonly listed tools, the Newcastle-Ottawa Scale was the dominant NRS appraisal tool each year, from 2011 to 2018. Still, the ROBINS-I tool (previously ACROBAT-NRSI) appears to exist gaining in popularity in recent years.

Limitations

Because this report was conducted using systematic review protocols, we exercise not know whether the concluding systematic reviews actually used the tools listed in these protocols. The analysis of PROSPERO protocols for the trend analysis relied on keywords and counts from the search results without further verification in the text of protocols, which may have overestimated the use of certain tools, particularly for Cochrane tools and suites of tools. However, keywords were restricted to the risk of bias section of the registered protocol. Equally not all systematic reviews are registered prospectively in PROSPERO, results of this report may not be generalizable to the wider body of systematic reviews on health interventions. Authors who are motivated to register systematic reviews in PROSPERO or publish their protocols in peer-reviewed journals, both of which are recommended past the AMSTAR systematic review quality appraisal tool [41], may be more likely to utilise RoB tools recommended in institutional guidelines, such as the Cochrane Handbook. An additional limitation is that the trend analysis was conducted for simply the nearly normally cited tools planned for use in systematic reviews in 2018. Therefore, this analysis does not capture consummate trends for the planned use of RoB tools over the final eight years in PROSPERO.

Conclusions and implications for exercise

Results of this analysis emphasize that the Cochrane RoB Tool has get the standard for systematic reviews of RCT. Despite the existence of dozens of tools for assessing NRS, relatively few are commonly used in practice, with the Newcastle-Ottawa Scale and ROBINS-I existence the most oftentimes used. There is too evidence that the Cochrane RoB Tool for RCT may be used inappropriately to assess NRS, indicating a need for more pedagogy and sensation on the appropriate utilize of tools for the quality assessment of non-randomized designs.

With a lack of gold standard for assessing risk of bias in NRS, some have called for the evolution of an improved tool that could effectively evaluate different kinds of quasi-experimental studies [12]. Others have suggested using different tools based on the types of study designs that are identified by the review [iii, xiii]. The development of a "meta" quality appraisal tool, such as the 1 created by Public Wellness Ontario [42], which recommends particular tools by written report design, may be a coherent way to address the lack of guidance on gamble of bias assessment for systematic reviews incorporating NRS evidence. Time to come enquiry should focus on the development and validation of tools for specific NRS designs.

Availability of data and materials

The datasets used in the current study are bachelor from the respective writer on reasonable request.

Abbreviations

ACROBAT-NRSI:

A Cochrane Run a risk Of Bias Assessment Tool: for Non-Randomized Studies of Interventions

ADA:

American Dietetic Association

AND:

Academy of Nutrition and Dietetics

CASP:

Critical Appraisement Skills Program

CEBM:

Centre for Testify-based Medicine (Oxford)

EPHPP:

Effective Public Health Exercise Projection

EPOC:

Constructive Practice and Organisation of Care

JBI:

Joanna Briggs Institute

MINORS:

Methodological Index for Non-Randomized Studies

MMAT:

Mixed Methods Cess Tool

NHLBI:

National Heart, Lung, and Blood Institute

Overnice:

National Institute for Health and Care Excellence

NRS:

Non-randomized report

PEDro:

Physiotherapy Evidence Database

QUADAS:

Quality Assessment of Diagnostic Accurateness Studies

RCT:

Randomized controlled trial

RoB:

Risk of bias

ROBINS-I:

Risk of Bias in Non-randomized Studies - of Interventions

STROBE:

Strengthening the Reporting of Observational Studies in Epidemiology

References

1. Reeves BC, Wells GA, Waddington H. Quasi-experimental study designs series-paper 5: a checklist for classifying studies evaluating the effects on health interventions-a taxonomy without labels. J Clin Epidemiol. 2017;89:30–42.

   Article  Google Scholar

2. United States Food and Drug Administration (FDA). Real globe evidence. Argent Leap: FDA; 2019. https://www.fda.gov/ScienceResearch/SpecialTopics/RealWorldEvidence/default.htm. Accessed eighteen Jan 2019

   Google Scholar

3. Briere J-B, Bowrin G, Taieb V, Millier A, Toumi M, Coleman C. Meta-analyses using existent-world data to generate clinical and epidemiological evidence: a systematic literature review of existing recommendations. Curr Med Res Opin. 2018;34(12):2125–30.

   Commodity  Google Scholar

4. Higgins JPT, Altman DG, Sterne JAC, on behalf of the Cochrane Statistical Methods Group and the Cochrane Bias Methods Group. 8.2.i Bias and gamble of bias. In: Higgins JPT, Green Due south, editors. Cochrane handbook for systematic reviews of interventions version five.i.0: The Cochrane Collaboration; 2011. [updated March 2011]. http://handbook-v-1.cochrane.org/. Accessed 17 May 2019.

5. Higgins JPT, Altman DG, Sterne JAC, on behalf of the Cochrane Statistical Methods Group and the Cochrane Bias Methods Grouping. viii.5 The Cochrane Collaboration'southward tool for assessing take chances of bias. In: Higgins JPT, Green S, editors. Cochrane handbook for systematic reviews of interventions version 5.i.0 [updated March 2011]: The Cochrane Collaboration; 2011. http://handbook-5-1.cochrane.org/. Accessed viii Feb 2019.

6. Jørgensen L, Paludan-Müller AS, Laursen DRT, Savović J, Boutron I, Sterne JAC, et al. Evaluation of the Cochrane tool for assessing risk of bias in randomized clinical trials: overview of published comments and assay of user practice in Cochrane and not-Cochrane reviews. Syst Rev. 2016;5:80.

   Article  Google Scholar

7. Jarde A, Losilla J-One thousand, Vives J. Methodological quality assessment tools of non-experimental studies: a systematic review. Ann Psychol. 2012;28(ii):617–28.

   Google Scholar

8. Lang S, Kleijnen J. Quality assessment tools for observational studies: lack of consensus. Int J Evid Based Healthc. 2010;8(4):247.

   Article  Google Scholar

9. Higgins JP, Ramsay C, Reeves BC, Deeks JJ, Shea B, Valentine JC, et al. Issues relating to study blueprint and hazard of bias when including not-randomized studies in systematic reviews on the effects of interventions. Res Synth Methods. 2013;four(i):12–25.

   Article  Google Scholar

10. Neyarapally GA, Hammad TA, Pinheiro SP, Iyasu Southward. Review of quality cess tools for the evaluation of pharmacoepidemiological safety studies. BMJ Open. 2012;2(5):e001362.

    Article  Google Scholar

11. Humphreys DK, Panter J, Ogilvie D. Questioning the application of risk of bias tools in appraising evidence from natural experimental studies: disquisitional reflections on Benton et al., IJBNPA 2016. Int J Behav Nutr Phys Human activity. 2017;14(1):49.

    Article  Google Scholar

12. Waddington H, Aloe AM, Becker BJ, Djimeu EW, Hombrados JG, Tugwell P, et al. Quasi-experimental written report designs serial-paper half-dozen: hazard of bias assessment. J Clin Epidemiol. 2017;89:43–52.

    Commodity  Google Scholar

13. Quigley JM, Thompson JC, Halfpenny NJ, Scott DA. Critical appraisal of nonrandomized studies-a review of recommended and commonly used tools. J Eval Clin Pract. 2019;25(1):44–52.

    Article  Google Scholar

14. Seehra J, Pandis N, Koletsi D, Fleming PS. Use of quality assessment tools in systematic reviews was varied and inconsistent. J Clin Epidemiol. 2016;69:179–184.e5.

    Article  Google Scholar

15. Joanna Briggs Institute (JBI). Critical appraisement tools. South Australia: The University of Adelaide; 2018. http://joannabriggs.org/enquiry/critical-appraisal-tools.html. Accessed xv Oct 2018

    Google Scholar

16. Downs SH, Black N. The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and not-randomised studies of health care interventions. J Epidemiol Community Health. 1998;52(six):377–84.

    CAS  Article  Google Scholar

17. Maher CG, Sherrington C, Herbert RD, Moseley AM, Elkins M. Reliability of the PEDro scale for rating quality of randomized controlled trials. Phys Ther. 2003;83(8):713–21.

    PubMed  PubMed Central  Google Scholar

18. Wells GA, Shea B, O'Connell D, Peterson J, Welch V, Losos Chiliad, et al. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. Ottawa: Ottawa Hospital Enquiry Institute; 2018. http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp. Accessed 15 October 2018

    Google Scholar

19. Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials. 1996;17(1):1–12.

    CAS  Article  Google Scholar

20. Higgins JPT, Sterne JAC, Savović J, Page MJ, Hróbjartsson A, Boutron I, et al. A revised tool for assessing risk of bias in randomized trials. In: Chandler J, McKenzie J, Boutron I, Welch Five (editors). Cochrane Methods. Cochrane Database Syst Rev. 2016;ten(Suppl ane).

21. Sterne JAC, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan Thousand, et al. ROBINS-I: a tool for assessing risk of bias in non-randomized studies of interventions. BMJ. 2016;355:i4919.

22. Balshem H, Helfand Chiliad, Schünemann HJ, Oxman Advertisement, Kunz R, Brozek J, et al. GRADE guidelines: 3. Rating the quality of show. J Clin Epidemiol. 2011;64(4):401–6.

    Article  Google Scholar

23. Pace R, Pluye P, Bartlett G, Macaulay Ac, Salsberg J, Jagosh J, et al. Testing the reliability and efficiency of the pilot mixed methods appraisal tool (MMAT) for systematic mixed studies review. Int J Nurs Stud. 2012;49(ane):47–53.

    Commodity  Google Scholar

24. Higgins JPT, Light-green Due south, editors. Cochrane handbook for systematic reviews of interventions version 5.i.0 [updated March 2011]: The Cochrane Collaboration; 2011. http://handbook-5-1.cochrane.org/. Accessed 8 February 2019

25. Law M, Steward D, Pollock Due north, Letts Fifty, Bosch J, Westmorland M. Disquisitional review course quantitative studies. Hamilton: McMaster University; 1998. https://srs-mcmaster.ca/wp-content/uploads/2015/04/Critical-Review-Form-Quantitative-Studies-English.pdf. Accessed 8 Feb 2019

    Google Scholar

26. McMaster Prove Review and Synthesis Team. Effective Public Health Do Project (EPHPP): quality assessment tool for quantitative studies. Hamilton: McMaster Evidence Review & Synthesis Middle; 2018. https://merst.ca/ephpp/. Accessed 15 December 2018

    Google Scholar

27. Slim K, Nini E, Forestier D, Kwiatkowski F, Panis Y, Chipponi J. Methodological index for non-randomized studies (MINORS): evolution and validation of a new instrument. ANZ J Surg. 2003;73(nine):712–6.

    Article  Google Scholar

28. Center for Evidence-Based Medicine (CEBM). Oxford Eye for Show-based Medicine. Levels of bear witness. Oxford: Nuffield Department of Principal Care Wellness Sciences; 2009. https://world wide web.cebm.net/2009/06/oxford-centre-evidence-based-medicine-levels-prove-march-2009/. Accessed 15 Oct 2018

    Google Scholar

29. University of Nutrition and Dietetics. Quality criteria checklist: primary enquiry. Chicago: University of Nutrition and Dietetics; [date unknown]. https://www.andeal.org/vault/2440/web/files/QCC_3.pdf. Accessed viii February 2019.

30. National Heart, Lung, and Claret Establish (NHLBI). Report quality cess tools. Bethesda: U.S. Department of Health & Human Services; [engagement unknown]. https://www.nhlbi.nih.gov/health-topics/study-quality-assessment-tools. Accessed 15 Dec 2018.

31. Whiting PF, Rutjes AWS, Westwood ME, Mallett S, Deeks JJ, Reitsma JB, et al. QUADAS-2: a revised tool for the quality assessment of diagnostic accurateness studies. Ann Intern Med. 2011;155(8):529–36.

    Article  Google Scholar

32. Critical Appraisal Skills Programme (CASP). CASP checklists. Oxford: CASP; 2018. https://casp-uk.net/casp-tools-checklists/. Accessed xv October 2018

    Google Scholar

33. Effective Practice and Organisation of Intendance (EPOC) Group. EPOC resources for review authors: Cochrane; 2017. https://epoc.cochrane.org/resource/epoc-resources-review-authors. Accessed 25 Jan 2019

34. von Elm Eastward, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP. STROBE initiative. The strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. Int J Surg. 2014;12(12):1495–9.

    Article  Google Scholar

35. Squeamish. The guidelines transmission. Appendix B: methodology checklist: systematic reviews and meta-analyses. London: NICE; 2012. https://www.nice.org.great britain/process/pmg6/resources/the-guidelines-transmission-appendices-bi-2549703709/chapter/appendix-b-methodology-checklist-systematic-reviews-and-meta-analyses. Accessed 15 October 2018

    Google Scholar

36. Reeves BC, Deeks JJ, Higgins JPT, Wells GA, on behalf of the Cochrane Non-Randomised Studies Methods Group. xiii.5.2.3 Tools for assessing methodological quality or risk of bias in non-randomized studies. In: Higgins JPT, Dark-green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]: The Cochrane Collaboration; 2011. http://handbook-5-i.cochrane.org/. Accessed 8 Feb 2019.

37. Deeks JJ, Dinnes J, D'Amico R, Sowden AJ, Sakarovitch C, Song F, et al. Evaluating non-randomised intervention studies. Health Technol Appraise. 2003;7(27):iii–x, ane–173.

    CAS  Article  Google Scholar

38. Hartling L, Milne A, Hamm MP, Vandermeer B, Ansari M, Tsertsvadze A, et al. Testing the Newcastle Ottawa calibration showed low reliability betwixt individual reviewers. J Clin Epidemiol. 2013;66(9):982–93.

    Article  Google Scholar

39. Stang A. Disquisitional evaluation of the Newcastle-Ottawa scale for the assessment of the quality of nonrandomized studies in meta-analyses. Eur J Epidemiol. 2010;25(9):603–5.

    Article  Google Scholar

40. Margulis AV, Pladevall Thousand, Riera-Guardia N, Varas-Lorenzo C, Hazell Fifty, Berkman ND, et al. Quality assessment of observational studies in a drug-safety systematic review, comparison of two tools: the Newcastle-Ottawa scale and the RTI item banking concern. Clin Epidemiol. 2014;half dozen:359–68.

    Article  Google Scholar

41. Shea BJ, Reeves BC, Wells G, Thuku Thousand, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisement tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008.

42. Public Health Ontario. MetaQAT - critical appraisal tool. Toronto: Ontario Agency for Wellness Protection and Promotion; 2018. https://world wide web.publichealthontario.ca/en/ServicesAndTools/Pages/Critical-Appraisement-Tool.aspx. Accessed xv Oct 2018

    Google Scholar

Download references

Acknowledgements

We would like to thank the Health Library of Health Canada and the Public Health Bureau of Canada for conducting a background literature search to inform this study.

Funding

This study was funded by the Public Wellness Agency of Canada.

Author data

Affiliations

1. Centre for Immunization and Respiratory Infectious Diseases, Public Health Agency of Canada, Ottawa, Canada

   Kelly Farrah, Kelsey Immature, Matthew C. Tunis & Linlu Zhao

Contributions

All authors contributed to the study design. KF carried out information collection and assay and drafted the original manuscript. All authors reviewed and revised the draft for critical content. All authors read and approved the last manuscript.

Respective author

Correspondence to Linlu Zhao.

Ethics declarations

Ethics approval and consent to participate

Not applicative.

Consent for publication

Non applicable.

Competing interests

The authors declare that they accept no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Boosted file 1.

Selection of 2018 Sample of PROSPERO Protocols. Sample selection menses diagram.

Additional file 2.

Risk of Bias Tools Intended to be Used in 2018 PROSPERO Sample. Table with full count of all the risk of bias tools listed in the random sample of protocols.

Additional file 3.

PROSPERO Annual Trends in Take chances of Bias Tools Search Strategy. Full search strategy for 12 commonly used take a chance of bias tools from 2011 to Dec seven, 2018 in PROSPERO.

Additional file 4.

Annual Frequency of Common Tools Listed in PROSPERO Protocol Risk of Bias Department. Full data on number of records that mentioned the 12 usually used take a chance of bias tools by year.

Rights and permissions

Open Admission This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted employ, distribution, and reproduction in any medium, provided you give appropriate credit to the original writer(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/one.0/) applies to the data fabricated bachelor in this article, unless otherwise stated.

Reprints and Permissions

About this article

Cite this article

Farrah, Thou., Young, 1000., Tunis, Thousand.C. et al. Risk of bias tools in systematic reviews of health interventions: an analysis of PROSPERO-registered protocols. Syst Rev 8, 280 (2019). https://doi.org/10.1186/s13643-019-1172-8

Download citation

• Received: 27 March 2019

• Accepted: 27 September 2019

• Published: 15 Nov 2019

• DOI : https://doi.org/10.1186/s13643-019-1172-8

Keywords

• Critical appraisal
• Non-randomized studies
• PROSPERO
• Risk of bias
• Systematic reviews

warrenjacessid.blogspot.com

Source: https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/s13643-019-1172-8

Share this post