Evaluation of large language model responses to different types of questions based on European Society of Endodontology position statements

Merve Defişet; Merve Yeniçeri Özata; Ayşenur Çam

doi:10.7126/cumudj.1953387

EN TR

Evaluation of large language model responses to different types of questions based on European Society of Endodontology position statements

Abstract

Objectives: This study aimed to compare the performance of DeepSeek-V3, ChatGPT 4.0, ChatGPT 4.5, Gemini Advanced 2.0, and Grok in answering single-correct answer multiple-choice (SCQ), multiple-correct answer multiple-choice (MCQ), binary (true/false) (BQ), and open-ended questions (OEQ) based on the European Society of Endodontology (ESE) position statements. Materials and Methods: A total of 70 questions (24 SCQ/MCQ, 29 BQ, and 17 OEQ) were posed to five LLMs. Responses were scored on a 0–10 scale for comprehensiveness, scientific accuracy, clarity, and relevance. SCQ, MCQ, and BQ formats were additionally evaluated for correctness (1 = correct, 0 = incorrect). Data were analyzed using Pearson’s chi-square test and the Kruskal-Wallis H test (p < 0.05). Results: In the SCQ/MCQ format, Gemini Advanced 2.0 scored significantly higher than Grok. In the BQ format, ChatGPT 4.5 and Gemini Advanced 2.0 achieved significantly higher scores than ChatGPT 4.0 and Grok. In the OEQ format, ChatGPT 4.5 obtained significantly higher scores than Grok (p < 0.05). Accuracy rates were 67.9% for ChatGPT 4.5, 66.0% for Gemini Advanced 2.0, 58.5% for DeepSeek-V3, 52.8% for ChatGPT 4.0, and 49.1% for Grok (p > 0.05). In all models, scientific accuracy scores were significantly lower than relevance scores (p < 0.05). Conclusions: ChatGPT 4.5 and Gemini Advanced 2.0 demonstrated stronger performance than the other models, whereas Grok showed lower response-content performance. However, since LLMs may appear convincing even when generating incorrect clinical information, they should be used only as supplementary tools in endodontic education and preliminary information retrieval. Clinically relevant outputs should be verified by an endodontist or an appropriately qualified clinician.

Keywords

Ethical Statement

This study was based exclusively on AI-generated responses and publicly available professional position statements. No human participants, patient data, animal subjects, or identifiable personal information were used. Therefore, ethics committee approval and informed consent were not applicable.

Thanks

This study was presented as an oral presentation at the 28th International Dental Congress of the Turkish Dental Association, held in Diyarbakır, Türkiye, on September 18–21, 2025

Avrupa Endodonti Derneği Pozisyon Bildirgelerine Dayalı Farklı Soru Türlerine Büyük Dil Modellerinin Verdiği Yanıtların Değerlendirilmesi

Abstract

Amaç: Bu çalışmanın amacı, DeepSeek-V3, ChatGPT 4.0, ChatGPT 4.5, Gemini Advanced 2.0 ve Grok’un Avrupa Endodonti Derneği’nin pozisyon bildirgelerine dayalı olarak hazırlanan tek doğru yanıtlı çoktan seçmeli, birden fazla doğru yanıtlı çoktan seçmeli, ikili doğru/yanlış ve açık uçlu sorulara verdikleri yanıt performanslarını karşılaştırmaktır. Gereç ve Yöntemler: Beş büyük dil modeline toplam 70 soru yöneltildi. Bu soruların 24’ü tek doğru yanıtlı/birden fazla doğru yanıtlı çoktan seçmeli, 29’u ikili doğru/yanlış ve 17’si açık uçlu soru formatındaydı. Yanıtlar kapsamlılık, bilimsel doğruluk, açıklık ve konuya uygunluk açısından 0–10 puanlık bir ölçek kullanılarak değerlendirildi. Tek doğru yanıtlı çoktan seçmeli, birden fazla doğru yanıtlı çoktan seçmeli ve ikili doğru/yanlış soru formatları ayrıca doğru yanıt için 1, yanlış yanıt için 0 olacak şekilde doğruluk açısından değerlendirildi. Veriler Pearson ki-kare testi ve Kruskal-Wallis H testi kullanılarak analiz edildi (p < 0,05). Bulgular: Tek doğru yanıtlı/birden fazla doğru yanıtlı çoktan seçmeli soru formatında Gemini Advanced 2.0, Grok’tan anlamlı düzeyde daha yüksek puan aldı. İkili doğru/yanlış soru formatında ChatGPT 4.5 ve Gemini Advanced 2.0, ChatGPT 4.0 ve Grok’a kıyasla anlamlı düzeyde daha yüksek puan elde etti. Açık uçlu soru formatında ise ChatGPT 4.5, Grok’a kıyasla anlamlı düzeyde daha yüksek puan elde etti (p < 0,05). Doğruluk oranları ChatGPT 4.5 için %67,9, Gemini Advanced 2.0 için %66,0, DeepSeek-V3 için %58,5, ChatGPT 4.0 için %52,8 ve Grok için %49,1 olarak belirlendi (p > 0,05). Tüm modellerde bilimsel doğruluk puanları, konuya uygunluk puanlarından anlamlı düzeyde daha düşüktü (p < 0,05). Sonuçlar: ChatGPT 4.5 ve Gemini Advanced 2.0 diğer modellere kıyasla daha güçlü performans göstermiştir; Grok ise daha düşük yanıt içeriği performansı sergilemiştir. Bununla birlikte, büyük dil modelleri yanlış klinik bilgiler üretmelerine rağmen ikna edici görünebileceğinden, endodonti eğitiminde ve ön bilgi edinme sürecinde yalnızca tamamlayıcı araçlar olarak kullanılmalıdır. Klinik açıdan önemli çıktılar bir endodontist veya uygun nitelikte bir klinisyen tarafından doğrulanmalıdır.

Keywords

Ethical Statement

Bu çalışma yalnızca yapay zekâ tarafından oluşturulan yanıtlar ve kamuya açık profesyonel pozisyon bildirgeleri esas alınarak yürütülmüştür. Çalışmada insan katılımcılar, hasta verileri, hayvan denekler veya kimliği belirlenebilir herhangi bir kişisel bilgi kullanılmamıştır. Bu nedenle etik kurul onayı ve bilgilendirilmiş onam gerekli değildir.

Thanks

Bu çalışma, 18–21 Eylül 2025 tarihlerinde Diyarbakır, Türkiye’de düzenlenen Türk Dişhekimleri Birliği 28. Uluslararası Diş Hekimliği Kongresi’nde sözlü sunum olarak sunulmuştur.

References

1. King MR. The future of AI in medicine: a perspective from a chatbot. Ann Biomed Eng 2023;51:291-295.
2. Dhopte A, Bagde H. Smart smile: revolutionizing dentistry with artificial intelligence. Cureus 2023;15:e41227.
3. Aminoshariae A, Kulild J, Nagendrababu V. Artificial intelligence in endodontics: current applications and future directions. J Endod 2021;47:1352-1357.
4. Rodrigues JA, Krois J, Schwendicke F. Demystifying artificial intelligence and deep learning in dentistry. Braz Oral Res 2021;35:e094.
5. Pompili D, Richa Y, Collins P, Richards H, Hennessey DB. Using artificial intelligence to generate medical literature for urology patients: a comparison of three different large language models. World J Urol 2024;42:455.
6. Rokhshad R, Zhang P, Mohammad-Rahimi H, Pitchika V, Entezari N, Schwendicke F. Accuracy and consistency of chatbots versus clinicians for answering pediatric dentistry questions: a pilot study. J Dent 2024;144:104938.
7. Kipp M. From GPT-3.5 to GPT-4o: a leap in AI’s medical exam performance. Information 2024;15:543.
8. OpenAI. Introducing GPT-4.5. Available from: https://openai.com/index/introducing-gpt-4-5/. Accessed April 1, 2025.

9. Adinath DR, Smiju IS. Manus AI, Gemini, Grok AI, DeepSeek, and ChatGPT: a comparative analysis of advancements in NLP. Available from: https://ssrn.com/abstract=5185131. Accessed March 19, 2025.
10. Imran M, Almusharraf N. Google Gemini as a next generation AI educational tool: a review of emerging educational technology. Smart Learn Environ 2024;11:22.
11. Conroy G, Mallapaty S. How China created AI model DeepSeek and shocked the world. Nature 2025;638:300-301.
12. Şahin MF, Topkaç EC, Doğan Ç, Şeramet S, Özcan R, Akgül M, et al. Still using only ChatGPT? The comparison of five different artificial intelligence chatbots’ answers to the most common questions about kidney stones. J Endourol 2024;38:1172-1177.
13. Ver Berne J, Saadi SB, Politis C, Jacobs R. A deep learning approach for radiological detection and classification of radicular cysts and periapical granulomas. J Dent 2023;135:104581.
14. Karobari MI, Adil AH, Basheer SN, Murugesan S, Savadamoorthi KS, Mustafa M, et al. Evaluation of the diagnostic and prognostic accuracy of artificial intelligence in endodontic dentistry: a comprehensive review of literature. Comput Math Methods Med 2023;2023:7049360.
15. Hiraiwa T, Ariji Y, Fukuda M, Kise Y, Nakata K, Katsumata A, et al. A deep-learning artificial intelligence system for assessment of root morphology of the mandibular first molar on panoramic radiography. Dentomaxillofac Radiol 2019;48:20180218.
16. Fukuda M, Inamoto K, Shibata N, Ariji Y, Kutsuna S, Nakata K, et al. Evaluation of an artificial intelligence system for detecting vertical root fracture on panoramic radiography. Oral Radiol 2020;36:337-343.
17. Aljamani S, Hassona Y, Fansa HA, Saadeh HM, Jamani KD. Evaluating large language models in addressing patient questions on endodontic pain: a comparative analysis of accessible chatbots. J Endod 2025;51:1617-1624.
18. Patil SR, Karobari MI. Exploring artificial intelligence for enhanced endodontic practice: applications, challenges, and future directions. Adv Public Health 2024;2024:8075515.
19. Büyüközer Özkan H, Doğan Çankaya T, Kölüş T. The impact of language variability on artificial intelligence performance in regenerative endodontics. Healthcare 2025;13:1190.
20. European Society of Endodontology. Resources for clinicians. Available from: https://www.e-s-e.eu/for-professionals/resources-for-clinicians/. Accessed March 10, 2025.
21. Giannakopoulos K, Kavadella A, Aaqel Salim A, Stamatopoulos V, Kaklamanos EG. Evaluation of the performance of generative AI large language models ChatGPT, Google Bard, and Microsoft Bing Chat in supporting evidence-based dentistry: comparative mixed methods study. J Med Internet Res 2023;25:e51580.
22. Duncan HF, Kirkevang LL, Peters OA, El-Karim I, Krastl G, Del Fabbro M, et al. Treatment of pulpal and apical disease: the European Society of Endodontology S3-level clinical practice guideline. Int Endod J 2023;56:238-295.
23. Ramlogan S, Raman V, Ramlogan S. A pilot study of the performance of ChatGPT and other large language models on a written final year periodontology exam. BMC Med Educ 2025;25:727.
24. Özcivelek T, Özcan B. Comparative evaluation of responses from DeepSeek-R1, ChatGPT-o1, ChatGPT-4, and Dental GPT chatbots to patient inquiries about dental and maxillofacial prostheses. BMC Oral Health 2025;25:871.
25. Tokgöz Kaplan T, Cankar M. Evidence-based potential of generative artificial intelligence large language models on dental avulsion: ChatGPT versus Gemini. Dent Traumatol 2025;41:178-186.
26. Ahmed WM, Azhari AA, Alfaraj A, Alhamadani A, Zhang M, Lu CT. The quality of AI-generated dental caries multiple choice questions: a comparative analysis of ChatGPT and Google Bard language models. Heliyon 2024;10:e28198.
27. Özden I, Gökyar M, Özden ME, Sazak Öveçoglu H. Assessment of artificial intelligence applications in responding to dental trauma. Dent Traumatol 2024;40:722-729.
28. Rao A, Kim J, Kamineni M, Pang M, Lie W, Dreyer KJ, et al. Evaluating GPT as an adjunct for radiologic decision making: GPT-4 versus GPT-3.5 in a breast imaging pilot. J Am Coll Radiol 2023;20:990-997.
29. Li J, He X, Wang Y, Liu Y, Liu J, Liu M, et al. Clinical decision support of advanced large language models in endodontic disease. J Dent Sci 2026;21:191-197.
30. Sumbal A, Sumbal R, Amir A. Can ChatGPT-3.5 pass a medical exam? A systematic review of ChatGPT's performance in academic testing. J Med Educ Curric Dev 2024;11:23821205241238641.
31. Liu M, Okuhara T, Chang X, Shirabe R, Nishiie Y, Okada H, et al. Performance of ChatGPT across different versions in medical licensing examinations worldwide: systematic review and meta-analysis. J Med Internet Res 2024;26:e60807.
32. Dermata A, Arhakis A, Makrygiannakis MA, Giannakopoulos K, Kaklamanos EG. Evaluating the evidence-based potential of six large language models in paediatric dentistry: a comparative study on generative artificial intelligence. Eur Arch Paediatr Dent 2025;26:527-535.
33. Tsai CY, Hsieh SJ, Huang HH, Deng JH, Huang YY, Cheng PY. Performance of ChatGPT on the Taiwan urology board examination: insights into current strengths and shortcomings. World J Urol 2024;42:250.

Details

Primary Language

English

Subjects

Endodontics

Journal Section

Research Article

Authors

Merve Defişet
0009-0003-6270-8979
Türkiye

Merve Yeniçeri Özata ^*
0000-0003-3619-3118
Türkiye

Ayşenur Çam
0009-0003-6404-9057
Türkiye

Publication Date

September 30, 2026

Submission Date

May 17, 2026

Acceptance Date

August 31, 2026

Published in Issue

Year 2026 Volume: 29 Number: 3

DOI

https://doi.org/10.7126/cumudj.1953387

IZ

https://izlik.org/JA45MC22ME

Cite

RIS / Bibtex

APA

Defişet, M., Yeniçeri Özata, M., & Çam, A. (2026). Evaluation of large language model responses to different types of questions based on European Society of Endodontology position statements. Cumhuriyet Dental Journal, 29(3), 487-496. https://doi.org/10.7126/cumudj.1953387

AMA

1.Defişet M, Yeniçeri Özata M, Çam A. Evaluation of large language model responses to different types of questions based on European Society of Endodontology position statements. Cumhuriyet Dent J. 2026;29(3):487-496. doi:10.7126/cumudj.1953387

Chicago

Defişet, Merve, Merve Yeniçeri Özata, and Ayşenur Çam. 2026. “Evaluation of Large Language Model Responses to Different Types of Questions Based on European Society of Endodontology Position Statements”. Cumhuriyet Dental Journal 29 (3): 487-96. https://doi.org/10.7126/cumudj.1953387.

EndNote

Defişet M, Yeniçeri Özata M, Çam A (September 1, 2026) Evaluation of large language model responses to different types of questions based on European Society of Endodontology position statements. Cumhuriyet Dental Journal 29 3 487–496.

IEEE

[1]M. Defişet, M. Yeniçeri Özata, and A. Çam, “Evaluation of large language model responses to different types of questions based on European Society of Endodontology position statements”, Cumhuriyet Dent J, vol. 29, no. 3, pp. 487–496, Sept. 2026, doi: 10.7126/cumudj.1953387.

ISNAD

Defişet, Merve - Yeniçeri Özata, Merve - Çam, Ayşenur. “Evaluation of Large Language Model Responses to Different Types of Questions Based on European Society of Endodontology Position Statements”. Cumhuriyet Dental Journal 29/3 (September 1, 2026): 487-496. https://doi.org/10.7126/cumudj.1953387.

JAMA

1.Defişet M, Yeniçeri Özata M, Çam A. Evaluation of large language model responses to different types of questions based on European Society of Endodontology position statements. Cumhuriyet Dent J. 2026;29:487–496.

MLA

Defişet, Merve, et al. “Evaluation of Large Language Model Responses to Different Types of Questions Based on European Society of Endodontology Position Statements”. Cumhuriyet Dental Journal, vol. 29, no. 3, Sept. 2026, pp. 487-96, doi:10.7126/cumudj.1953387.

Vancouver

1.Merve Defişet, Merve Yeniçeri Özata, Ayşenur Çam. Evaluation of large language model responses to different types of questions based on European Society of Endodontology position statements. Cumhuriyet Dent J. 2026 Sep. 1;29(3):487-96. doi:10.7126/cumudj.1953387