(557) Evaluating the Validity of ChatGPT Responses on Common Obstetric Issues: Potential Clinical Applications and Implications

We curated 15 common and potentially clinically significant questions that pregnant women are asking. Two native English-speaking women were asked to reframe the questions in their own words, and we employed the ChatGPT language model to generate responses to the questions. To evaluate the accuracy, completeness, and safety of the ChatGPT's generated responses we developed a questionnaire with a scale of one to five. that Obstetrics and Gynecology experts from different countries were invited to rate accordingly. The ratings were analyzed to evaluate the average level of agreement and percentage of positive ratings (four or above) for each criterion.

Results:

Out of the 42 experts invited, 20 responded to the questionnaire. The combined score for all responses yielded a mean rating of four, with 75% of responses receiving a positive rating (four or above). While examining specific criteria, the ChatGPT responses were better for the "accuracy" criterion, with a mean rating of 4.2 and 80% of the questions received a positive rating. The responses scored less for the "completeness" criterion, with a mean rating of 3.8 and 46.7% of questions received a positive rating. For "safety", the mean rating was 3.9 and 53.3% of questions received a positive rating. There was no response with an average negative rating below three.

Conclusion: This study demonstrates promising results regarding potential use of ChatGPT's in providing accurate responses to obstetric clinical questions posed by pregnant women. However, it is crucial to exercise caution when addressing inquiries concerning the safety of the fetus or the mother.