(824) MD vs. ML: Comparing clinician intuition to machine learning to predict mode of delivery

Integration of machine learning (ML) models in healthcare remains rare. One reason is uncertainty regarding these models’ predictive abilities compared to clinical intuition. We previously developed the Partometer, an ML model that passively collects both admission and intrapartum data to calculate accurate, q5 minute predictions of a patient’s probability of vaginal delivery. The objective of this study is to assess if the Partometer is able to predict as well as a patient’s obstetrician the probability of a vaginal delivery.

Study Design:

A prospective cohort study to evaluate calibration of the Partometer compared to clinician intuition. 42 clinicians agreed to participate. Patients admitted to these physicians, laboring and planning a vaginal delivery were the subjects for the prediction model. At 4 hours (a previously tested threshold time), we queried both the Partometer for a prediction of vaginal delivery likelihood (0 = likely cesarean to 1 = likely vaginal delivery) and concurrently asked the patient’s physician (blinded to the Partometer’s prediction).  The calibration of each method’s prediction was then compared to the final outcome by fitting a linear mixed model for the difference in the Brier scores between the two methods with random intercept to account for the predictions made by the same clinician. Calibration results were stratified for age, race and ethnicity, parity, and BMI.

Results:

159 deliveries were included. Approximately half of subjects were nulliparous (55.3%), >35 years old (44%), and a third had BMI >30 (32.7%). The Partometer’s predictions (mean Brier score = 0.11) were as well calibrated as clinicians’ (mean Brier score = 0.12) in determining if the patient would have a vaginal delivery (estimated intercept = 0.01, p=0.343 from the mixed model). The demonstrated equivalence did not differ when stratifying by other predictive variables.

Conclusion:

Our ML model, the Partometer, was able to predict as well as the patient’s clinician the likelihood of a vaginal delivery, laying foundational work for integration of ML into obstetric decision making.