Comparative Evaluation of Large Language Models’ Accuracy in Answering Multiple-Choice Restorative Dentistry Questions From a National Specialty Examination

Çilem Bulut; Gülben Çolak; Gürkan Çolak

doi:10.7126/cumudj.1794852

TR EN

Diş Hekimliği Eğitiminde Büyük Dil Modellerinin Karşılaştırmalı Analizi: Restoratif Diş Hekimliği Sınav Sorularındaki Doğruluk

Abstract

Amaç: Büyük dil modellerinin (LLM’ler) diş hekimliği eğitimine entegrasyonu hızla artmasına rağmen, alan-spesifik değerlendirmelerdeki gerçek performansları belirsizliğini korumaktadır. Bu çalışma, dört farklı LLM’in (ChatGPT-4.0, Gemini Advanced 1.5 Pro, DeepSeek-V3 ve Perplexity) restoratif diş hekimliği sorularındaki doğruluğunu değerlendirmeyi ve karşılaştırmayı amaçlamaktadır. Gereç ve Yöntem: 2012–2021 yılları arasında yapılan Türk Diş Hekimliği Uzmanlık Sınavı’ndan (DUS) toplam 127 çoktan seçmeli soru toplanmış ve 19 içerik alanına göre kategorize edilmiştir. Her soru, standart talimatlarla Türkçe olarak LLM’lere yöneltilmiştir. Yanıtlar kaydedilmiş ve resmi cevap anahtarlarına göre doğruluğu belirlenmiştir. İstatistiksel farklılıklar uygun testler kullanılarak analiz edilmiştir. Bulgular: En yüksek doğruluk oranı ChatGPT-4.0’da (%93,65) gözlenmiştir. Bunu Gemini (%82,54), DeepSeek (%71,43) ve Perplexity (%65,87) takip etmiştir. ChatGPT ile DeepSeek (p = 0,027) ve Perplexity (p = 0,004) arasında anlamlı fark bulunurken, ChatGPT ile Gemini arasında fark istatistiksel olarak anlamlı bulunmamıştır (p = 0,118). Tüm modeller teorik sorularda daha yüksek doğruluk gösterirken, beyazlatma ve kavite preparasyonu gibi klinik odaklı alanlarda daha düşük performans sergilemiştir. Sonuç: ChatGPT-4.0, değerlendirilen dil modelleri arasında en yüksek genel doğruluğu göstermiştir ve teorik diş hekimliği eğitiminde destekleyici bir araç olarak umut vadetmektedir. Bununla birlikte, klinik alanlardaki sınırlı performansı, hekim gözetiminde dikkatli bir şekilde uygulanması gerektiğini ortaya koymaktadır.

Keywords

Supporting Institution

Bu araştırma kamu, ticari veya kâr amacı gütmeyen sektörlerdeki fon sağlayıcı kurumlardan herhangi bir özel destek almamıştır.

Ethical Statement

Bu çalışma insan katılımcılar, hayvan denekler veya klinik veriler içermemektedir. Bu nedenle Etik Kurul onayı gerekmemiştir. Araştırma yalnızca kamuya açık veriler ve yapay zekâ analiz yöntemleri kullanılarak gerçekleştirilmiştir. Yazarlar bu çalışma ile ilgili herhangi bir etik kaygı bulunmadığını beyan eder.

Thanks

Çalışmanın geliştirilmesi sırasında katkı sağlayan ve geri bildirim veren tüm meslektaşlarımıza teşekkür ederiz.

Comparative Evaluation of Large Language Models’ Accuracy in Answering Multiple-Choice Restorative Dentistry Questions From a National Specialty Examination

Abstract

Objective: Although the integration of large language models (LLMs) into dental education is rapidly increasing, their actual performance in domain-specific assessments remains unclear. This study aimed to evaluate and compare the accuracy of four LLMs (ChatGPT-4.0, Gemini Advanced 1.5 Pro, DeepSeek-V3, and Perplexity) on restorative dentistry questions in a dental specialty examination. Materials and methods: A total of 127 multiple-choice questions from the Turkish Dental Specialty Examination (DUS) conducted between 2012 and 2021 were collected and categorized into 19 content areas. Each question was entered into LLMs in Turkish with standardized instructions. Responses were recorded, and their accuracy was determined according to official answer keys. Statistical differences were analyzed using the appropriate tests. Results: ChatGPT-4.0 had the highest accuracy rate (93.65%), followed by Gemini (82.54%), DeepSeek (71.43%), and Perplexity (65.87%). Significant differences were observed between ChatGPT and DeepSeek (p = 0.027) and Perplexity (p = 0.004), but not between ChatGPT and Gemini (p = 0.118). All models showed higher accuracy in theoretical questions but lower performance in clinically oriented areas such as bleaching and cavity preparation. Conclusion: ChatGPT-4.0 demonstrated the highest overall accuracy among the language models evaluated and shows promise as a supportive tool in theoretical dental education. However, its limited performance in clinical domains underlines the need for careful implementation under physician supervision.

Keywords

Supporting Institution

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Ethical Statement

This study did not involve human participants, animal subjects, or clinical data. Therefore, approval from an Ethics Committee was not required. The research was conducted using publicly available data and artificial intelligence analysis methods only. The authors declare that there are no ethical concerns related to this study.

Thanks

The authors would like to thank all colleagues who contributed to the discussions and feedback during the development of this study.

References

1. Labadze L, Grigolia M, Machaidze L. Role of AI chatbots in education: systematic literature review. Int J Educ Technol High Educ 2023;20:56.
2. AlZu’bi S, Mughaid A, Quiam F, Hendawi S. Exploring the capabilities and limitations of ChatGPT and alternative large language models. AIA 2024;2(1):28–37.
3. Kung TH, Cheatham M, Medenilla A, Sillos C, De Leon L, Elepano C, et al. Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLOS Digit Health 2023;2(2):e0000198.
4. Hadi MU, Al-Tashi Q, Qureshi R, Shah A, Muneer A, Irfan M, et al. Large language models: a comprehensive survey of applications, challenges, limitations, and future prospects. Authorea Preprints 2025.
5. ALI K, Barhom N, Marino FT, Duggal M. The thrills and chills of ChatGPT: Implications for assessments in undergraduate dental education. Preprints 2023.
6. Rao A, Pang M, Kim J, Kamineni M, Lie W, Prasad AK, et al. Assessing the utility of ChatGPT throughout the entire clinical workflow: development and usability study. J Med Internet Res. 2023;25:e48659.
7. Thirunavukarasu AJ, Ting DSJ, Elangovan K, Gutierrez L, Tan TF, Ting DSW. Large language models in medicine. Nat Med 2023;29:1930–1940.
8. van Dis EAM, Bollen J, Zuidema W, van Rooij R, Bockting CL. ChatGPT: five priorities for research. Nature 2023;614(7947):224–226.

9. Sokullu R, Akın Ö, Aslan E, Simonoski O, Marina N, Paunkoska N, et al. Emerging computer technologies 2. İzmir: İzmir Akademi Derneği 2022.
10. Rathje S, Mirea DM, Sucholutsky I, Marjieh R, Robertson CE, Van Bavel JJ. GPT is an effective tool for multilingual psychological text analysis. Proc Natl Acad Sci U S A. 2024;121(34):e2308950121.
11. Yoo MH, Kim J, Song S. Multilingual capabilities of GPT: a study of structural ambiguity. PLoS One. 2025;20(7):e0326943.
12. Saab K, Tu T, Weng W-H, Tanno R, Stutz D, Wulczyn E, et al. Capabilities of Gemini models in medicine. arXiv 2024.
13. Reese JT, Danis D, Caufield JH, Groza T, Casiraghi E, Valentini G, et al. On the limitations of large language models in clinical diagnosis. medRxiv 2024;2023.07.13.23292613.
14. Cross JL, Choma MA, Onofrey JA. Bias in medical AI: implications for clinical decision-making. PLOS Digit Health 2024;3(11):e0000651.
15. Temsah A, Alhasan K, Altamimi I, Jamal A, Al-Eyadhy A, Malki K, et al. DeepSeek in healthcare: revealing opportunities and steering challenges of a new open-source artificial intelligence frontier. Cureus 2025;17(2):e79221.
16. Perplexity AI Portal. Accessed April 17, 2023. https://www.perplexity.ai/.
17. Iannantuono GM, Bracken-Clarke D, Karzai F, Choo-Wosoba H, Gulley JL, Floudas CS. Comparison of large language models in answering immuno-oncology questions: a cross-sectional study. Oncologist 2024;29(5):407-414.
18. Zuccon G, Koopman B, Shaik R. ChatGPT hallucinates when attributing answers. In: Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region 2023.
19. Revilla-León M, Barmak BA, Sailer I, Kois JC, Att W. Performance of an artificial intelligence-based chatbot (ChatGPT) answering the European Certification in implant dentistry exam. Int J Prosthodont 2024;37(2):221-224.
20. Morishita M, Fukuda H, Muraoka K, Nakamura T, Hayashi M, Yoshioka I, et al. Evaluating GPT-4V’s performance in the Japanese national dental examination: a challenge explored. J Dent Sci 2024;19(3):1595–1600.
21. Rokhshad R, Zhang P, Mohammad-Rahimi H, Pitchika V, Entezari N, Schwendicke F. Accuracy and consistency of chatbots versus clinicians for answering pediatric dentistry questions: a pilot study. J Dent 2024;144:104938.
22. Sismanoglu S, Capan BS. Performance of artificial intelligence on Turkish dental specialization exam: can ChatGPT-4.0 and Gemini Advanced achieve comparable results to humans? BMC Med Educ 2025;25(1):214.
23. Çekiç EC, Tavşan O. Evaluating large language models using national endodontic specialty examination questions: are they ready for real-world dentistry? BMC Med Educ 2025;25(1):1308.
24. Nguyen HC, Dang HP, Nguyen TL, Hoang V, Nguyen VA. Accuracy of latest large language models in answering multiple choice questions in dentistry: a comparative study. PLoS One 2025;20(1):e0317423.
25. Au Yeung J, Kraljevic Z, Luintel A, Balston A, Idowu E, Dobson R, et al. AI chatbots not yet ready for clinical use. Front Digit Health 2023;5:1161098.
26. Strong E, DiGiammarino A, Weng Y, Basaviah P, Hosamani P, Kumar A, et al. Performance of ChatGPT on free-response, clinical reasoning exams. medRxiv 2023;29:2023.03.24.23287731.
27. Doddi S, Hibshman T, Salichs O, Bera K, Tippareddy C, Ramaiya N, et al. Assessing appropriate responses to ACR urologic imaging scenarios using ChatGPT and Bard. Curr Probl Diagn Radiol 2024;53(2):226–229.
28. Sarangi PK, Datta S, Swarup MS, Panda S, Nayak DSK, Malik A, et al. Radiologic decision-making for imaging in pulmonary embolism: accuracy and reliability of large language models—Bing, Claude, ChatGPT, and Perplexity. Indian J Radiol Imaging 2024;34(4):653–660.
29. Kavadella A, Dias da Silva MA, Kaklamanos EG, Stamatopoulos V, Giannakopoulos K. Evaluation of ChatGPT’s real-life implementation in undergraduate dental education: mixed methods study. JMIR Med Educ 2024;10:e51344.
30. Mohammad-Rahimi H, Ourang SA, Pourhoseingholi MA, Dianat O, Dummer PMH, Nosrat A. Validity and reliability of artificial intelligence chatbots as public sources of information on endodontics. Int Endod J 2024;57(3):305-314.
31. Chau RCW, Thu KM, Yu OY, Hsung RT, Lo ECM, Lam WYH. Performance of generative artificial intelligence in dental licensing examinations. Int Dent J 2024;74(3):616-621.
32. Gilson A, Safranek CW, Huang T, Socrates V, Chi L, Taylor RA, et al. How does ChatGPT perform on the United States Medical Licensing Examination (USMLE)? Implications for medical education and knowledge assessment. JMIR Med Educ 2023;9:e45312.
33. Bilgin Avşar D, Ertan AA. A comparative study of ChatGPT-3.5 and Gemini’s performance in answering prosthetic dentistry questions in the dentistry specialty exam: a cross-sectional study. Turkiye Klinikleri J Dental Sci 2024;30(4):668–673.
34. Tassoker M. ChatGPT-4 Omni’s superiority in answering multiple-choice oral radiology questions. BMC Oral Health 2025;25(1):173.
35. Yilmaz BE, Gokkurt Yilmaz BN, Ozbey F. Artificial intelligence performance in answering multiple-choice oral pathology questions: a comparative analysis. BMC Oral Health 2025;25(1):573.
36. Claman D, Sezgin E. Artificial intelligence in dental education: opportunities and challenges of large language models and multimodal foundation models. JMIR Med Educ 2024;10:e52346.
37. Pecher B, Spiegel M, Belanec R, Cegin J. Revisiting prompt sensitivity in large language models for text classification: The role of prompt underspecification. arXiv 2026;arXiv:2602.04297.
38. Liu P, Yuan W, Fu J, Jiang Z, Hayashi H, Neubig G. Pre-train, prompt, and predict: a systematic survey of prompting methods in natural language processing. ACM Comput Surv 2023;55(9):1–35.
39. Künzle P, Paris S. Performance of large language artificial intelligence models on solving restorative dentistry and endodontics student assessments. Clin Oral Investig 2024;28(11):575.
40. Huang H, Zheng O, Wang D, Yin J, Wang Z, Ding S, et al. ChatGPT for shaping the future of dentistry: the potential of multi-modal large language model. Int J Oral Sci. 2023;15:29.

Details

Primary Language

English

Subjects

Restorative Dentistry

Journal Section

Research Article

Authors

Çilem Bulut
0000-0001-9436-8399
Türkiye

Gülben Çolak ^*
0000-0003-0786-9609
Türkiye

Gürkan Çolak
0009-0000-8393-0913
Sweden

Publication Date

September 30, 2026

Submission Date

October 1, 2025

Acceptance Date

September 7, 2026

Published in Issue

Year 2026 Volume: 29 Number: 3

DOI

https://doi.org/10.7126/cumudj.1794852

IZ

https://izlik.org/JA48LT77PU

Cite

RIS / Bibtex

APA

Bulut, Ç., Çolak, G., & Çolak, G. (2026). Comparative Evaluation of Large Language Models’ Accuracy in Answering Multiple-Choice Restorative Dentistry Questions From a National Specialty Examination. Cumhuriyet Dental Journal, 29(3), 385-392. https://doi.org/10.7126/cumudj.1794852

AMA

1.Bulut Ç, Çolak G, Çolak G. Comparative Evaluation of Large Language Models’ Accuracy in Answering Multiple-Choice Restorative Dentistry Questions From a National Specialty Examination. Cumhuriyet Dent J. 2026;29(3):385-392. doi:10.7126/cumudj.1794852

Chicago

Bulut, Çilem, Gülben Çolak, and Gürkan Çolak. 2026. “Comparative Evaluation of Large Language Models’ Accuracy in Answering Multiple-Choice Restorative Dentistry Questions From a National Specialty Examination”. Cumhuriyet Dental Journal 29 (3): 385-92. https://doi.org/10.7126/cumudj.1794852.

EndNote

Bulut Ç, Çolak G, Çolak G (September 1, 2026) Comparative Evaluation of Large Language Models’ Accuracy in Answering Multiple-Choice Restorative Dentistry Questions From a National Specialty Examination. Cumhuriyet Dental Journal 29 3 385–392.

IEEE

[1]Ç. Bulut, G. Çolak, and G. Çolak, “Comparative Evaluation of Large Language Models’ Accuracy in Answering Multiple-Choice Restorative Dentistry Questions From a National Specialty Examination”, Cumhuriyet Dent J, vol. 29, no. 3, pp. 385–392, Sept. 2026, doi: 10.7126/cumudj.1794852.

ISNAD

Bulut, Çilem - Çolak, Gülben - Çolak, Gürkan. “Comparative Evaluation of Large Language Models’ Accuracy in Answering Multiple-Choice Restorative Dentistry Questions From a National Specialty Examination”. Cumhuriyet Dental Journal 29/3 (September 1, 2026): 385-392. https://doi.org/10.7126/cumudj.1794852.

JAMA

1.Bulut Ç, Çolak G, Çolak G. Comparative Evaluation of Large Language Models’ Accuracy in Answering Multiple-Choice Restorative Dentistry Questions From a National Specialty Examination. Cumhuriyet Dent J. 2026;29:385–392.

MLA

Bulut, Çilem, et al. “Comparative Evaluation of Large Language Models’ Accuracy in Answering Multiple-Choice Restorative Dentistry Questions From a National Specialty Examination”. Cumhuriyet Dental Journal, vol. 29, no. 3, Sept. 2026, pp. 385-92, doi:10.7126/cumudj.1794852.

Vancouver

1.Çilem Bulut, Gülben Çolak, Gürkan Çolak. Comparative Evaluation of Large Language Models’ Accuracy in Answering Multiple-Choice Restorative Dentistry Questions From a National Specialty Examination. Cumhuriyet Dent J. 2026 Sep. 1;29(3):385-92. doi:10.7126/cumudj.1794852