Icon Legend

This session is not in your schedule.

This session is in your schedule. Click again to remove it.

Presentation Icons

Additional registration fee required

Faculty have requested this content not be shared on social media

CME Credit Offered

Poster Icons

Foundation Awardee

No Social Media

Back

Q&A

Education/Simulation

Poster Session 2

(404) Frequently Asked Questions in Prenatal Testing: Comparing Readability Between AI (ChatGPT) and ACOG

Tuesday, February 13, 2024

3:30 PM – 5:00 PM

Location: Prince George's Exhibit Hall CDE, Lower Atrium Level

Primary & Presenting Author(s)

Angelo Cadiente, BS (he/him/his)

Medical Student
Hackensack Meridian School of Medicine
Nutley, NJ, United States

Coauthor(s)

Jamie Chen, BS (she/her/hers)

Medical Student
Hackensack Meridian School of Medicine
Nutley, NJ, United States
Andrew Haddad, MD

Hackensack University Medical Center
Hackensack, NJ, United States
MA

Manuel Alvarez, MD

Hackensack University Medical Center
Hackensack, NJ, United States
Antonia F. Oladipo, MD, MSCI (she/her/hers)

Assistant Professor
Hackensack University Medical Center
Hackensack, NJ, United States

Objective: Gravid patients often seek answers to their pregnancy-related questions online, requiring simple and accurate responses to ensure adequate understanding. We aim to compare the readability between a natural language processing tool (ChatGPT) and the American College of Obstetricians and Gynecologists (ACOG) when responding to frequently asked questions (FAQs) about prenatal testing.

Study Design: ACOG FAQ sheets were pooled under the topic “Prenatal Testing”. Each question was queried in ChatGPT-3.5 (July 20 Version) - with some questions requiring contextualization, prompting “In the context of [topic], [question]?” Readability scores and grade level scores were calculated for each response. With the exception of the Flesch Kincaid Reading Ease score, each measure describes the amount of formal education required to understand a given response. Statistical analysis utilized a two-tailed t-test to compare ACOG and ChatGPT-3.5 responses.

Results: Within the 11 FAQ sheets under “Prenatal Testing”, 182 questions were assessed. In all readability indices, ACOG responses scored significantly better than ChatGPT-generated responses and were statistically significant. These readability indices include: Flesch Kincaid Reading Ease 57.23 (SD 19.49) vs 35.97 (9.07), Flesch Kincaid Grade Level 9.45 (3.88) vs 12.70 (1.60), Gunning Fog Score 12.31 (4.31) vs 16.33 (2.03), Smog Index 9.01 (3.07) vs 12.07 (1.34), Coleman Liau Index 11.72 (3.36) vs 15.11 (1.70), and Automated Readability Index 9.00 (4.64) vs 12.30 (1.84; all p < 0.0001).

Conclusion: ACOG FAQ responses are clearer and more readable when compared to ChatGPT-generated responses. This suggests that fewer years of formal education is required to comprehend the content. Patients can reliably use ACOG verified responses as a source of high-quality, “easy to read” and understandable information. ChatGPT is an accessible resource but caution should be taken when asking pregnancy-related questions. Further studies are needed to assess the quality of natural language processors and its impact on patient’s knowledge.