Inter-Rater Reliability Of Assigning A Pediatric Appendicitis Grade: An International Multicenter Study By The Canadian Consortium For Research In Pediatric Surgery (CanCORPS)

Abstract

Background: Appendicitis is typically described, treated, and reported in outcome studies as simple versus perforated, without appropriate risk stratification. Objective: We performed an international study to assess the inter-rater reliability of assigning a Pediatric Appendicitis Grade (PAG) to children undergoing appendectomy. Methods: Pediatric surgery attending staff and trainees from 29 institutions in 8 countries on 6 continents rated 15 edited videos of laparoscopic appendectomy (3 for each PAG) on a web-based platform. Error rates were compared using Fisher exact test. Inter-observer reliability was estimated with the intraclass correlation coefficient (ICC), computed in R with a 2-way random-effects model, reported on a scale of 0 (no correlation) to 1 (perfect agreement) with a 95% confidence interval (CI). Results: Of 274 eligible participants, 121 respondents (44%) completed the study. The proportion of correctly identified cases was 71% overall: PAG 1, 97%; PAG 2, 70%; PAG 3, 73%; PAG 4, 64%; PAG 5, 51%. ICC was 0.81 (95% CI = 0.65–0.94), indicating good reliability among raters. There were no significant subgroup differences in ICC when comparing attending staff (ICC = 0.85, 95% CI = 0.74–0.94) versus trainees (0.84, 95% CI = 0.73–0.93), or when comparing respondents with low (0.84, 95% CI = 0.73–0.93), medium (0.81, 95% CI = 0.70–0.92), or high case volumes (0.80, 95% CI = 0.68–0.91). Conclusion: The PAG can be accurately and reliably assigned by surgeons and trainees in diverse settings and practices. Its routine use can improve prognostication, benchmarking, outcomes reporting, and resource allocation for the most common acute surgical illness in children.