Inter-rater and intra-raters’ variability in evaluating complete dentures insertion procedure in senior undergraduates’ prosthodontics clinics

Abstract

Background: Direct clinical assessment is the mainstay of evaluation in dentistry education. An effective evaluation method in prosthodontics should be equally valid and consistent; however, this is not attained frequently. A limited number of studies have applied an analytic evaluation in prosthodontics. Objective: To compare the intra- and inter-raters’ variability in two evaluation methods: glance and grade (global), and checklist and criteria (analytical). Moreover, to identify the components of the analytical evaluation system and its applicability. Methods: This cross-sectional study was carried out on outpatients attending removable prosthodontics clinics affiliated with King Abdulaziz University (Jeddah, Saudi Arabia) from December 2017 to April 2018. Two prosthodontist examiners evaluated a sample of 35 complete denture cases (20 male, 15 female) twice over a period of five months. Inter-rater and intra-rater agreement were computed using reliability test (interclass correlation coefficient ICC). Data were analyzed in IBM SPSS version 23, using paired-samples t-test, weighted kappa, and Wilcoxon signed-rank test. The level of significance was set at p≤0.05. Results: The intra-rater agreement for the first and second exposures under global and analytical evaluation methods with Examiner A was outstanding with 90.7% and 99.8% agreement respectively. While with Examiner B, global was lower but still in the acceptable range with about 78.1%, and 96.1% for the analytical evaluation. Inter-rater reliability analysis showed high agreement between the two raters in the first exposure of the analytical evaluation with 97.3%, while it was 87.5% in the global evaluation. This trend goes the same with analytical in the second exposure with 93.2%; however, the second exposure under global evaluation failed the cut off, which is only 56.6% agreement. In evaluation of inter-raters agreement, the second exposure of the global method demonstrated inconsistency between the two examiners (p=0.002), while the analytical second exposure demonstrated more homogeneity (p=0.050). Intra-rater variability between first and second exposure in analytical evaluation was (0.711 for the first rater and 0.677 for the second rater). Intra-rater variability between first and second exposure in global evaluation was (<0.001 for the first rater and 0.075 for the second rater). Conclusion: A simple objective and detailed method to evaluate complete denture insertion procedure was developed, and it showed that both intra-rater and inter-rater agreement were excellent for the analytical method that might overcome errors and subjectivity in evaluation that result from the limitations of global method. Results recommend suitability of using the analytical evaluation to improve reliability between raters.