article · BMJ
The Prediction model Risk Of Bias ASsessment Tool, known as PROBAST, assesses the quality, risk of bias, and applicability of predictive algorithms and studies. Because predictive modelling and artificial intelligence techniques have progressed significantly since 2019, an updated tool named PROBAST+AI has been established. The updated system comprises two parts covering model development and model evaluation. Model development relies on 16 signalling questions to assess quality and applicability. Model evaluation uses 18 signalling questions to assess risk of bias and applicability. Across both sections, the evaluation considers four core domains: participants and data sources, predictors, outcome, and analysis. Applicability ratings focus on the first three domains. PROBAST+AI serves as a replacement for the 2019 framework, offering a standardised method to evaluate healthcare prediction models regardless of whether they employ traditional regression methods or artificial intelligence techniques.
Artificial intelligence models are increasingly used to predict health outcomes, but flawed models can mislead medical decisions. By providing a structured evaluation system, this updated tool helps researchers, clinicians, and policy organisations judge whether healthcare prediction models are reliable, unbiased, and suitable for practical clinical use, regardless of the underlying algorithm used to build them.
The tool is ready for immediate practical application by artificial intelligence companies, model developers, healthcare providers, and regulatory or policy organisations. It enables these stakeholders to assess the quality, bias, and clinical applicability of healthcare algorithms before deployment. While not a commercial product itself, the framework functions as an applied evaluation standard for vetting predictive technologies moving into health systems.
AI-generated from the published abstract. Always read the original work before citing.
The Prediction model Risk Of Bias ASsessment Tool (PROBAST) is used to assess the quality, risk of bias, and applicability of prediction models or algorithms and of prediction model/algorithm studies. Since PROBAST’s introduction in 2019, much progress has been made in the methodology for prediction modelling and in the use of artificial intelligence, including machine learning, techniques. An update to PROBAST-2019 is thus needed. This article describes the development of PROBAST+AI. PROBAST+AI consists of two distinctive parts: model development and model evaluation. For model development, PROBAST+AI users assess quality and applicability using 16 targeted signalling questions. For model evaluation, PROBAST+AI users assess the risk of bias and applicability using 18 targeted signalling questions. Both parts contain four domains: participants and data sources, predictors, outcome, and analysis. Applicability of the prediction model is rated for the participants and data sources, predictors, and outcome domains. PROBAST+AI may replace the original PROBAST tool and allows all key stakeholders (eg, model developers, AI companies, researchers, editors, reviewers, healthcare professionals, guideline developers, and policy organisations) to examine the quality, risk of bias, and applicability of any type of prediction model in the healthcare sector, irrespective of whether regression modelling or AI techniques are used.
This page summarises published work. The authoritative version sits with the publisher.
DOI: 10.1136/bmj-2024-082505
Is something wrong with this record? Report it or request removal.
Discussion
Have you built on this work, tried to replicate it, or seen it applied in practice? Share what you know. Verified researchers and MARATTO™ domain experts can open a discussion, and any member can reply. Contributions are reviewed before they appear.
No discussion yet. Open the first thread.
New to MARATTO™? Create a free account.
