Abstract
Objectives: This study aims to assess the comprehension and interpretation performance of Chat Generative Pre-Train Omni (GPT-4o) in the context of oral radiology education and practice.
Materials and Methods: Utilizing a set of 99 questions derived from the book "White and Pharoah's Oral Radiology: Principles and Interpretation 8th Edition," this study employed ChatGPT-4o to respond to these questions thrice daily at varying times over 10 days, generating a total of 60 responses for each question. Two oral radiologists independently answered the same questions and verified their answers with the relevant textbook. Responses were compared to those of ChatGPT-4o.
Results: The study revealed that ChatGPT-4o's correct answer rate was 59.4%. Time-based analysis revealed performance differences across specific day periods. Specifically, during noon and evening sessions, the success rate on the first and seventh days was statistically significantly higher (p = 0.003 and p = 0.002, respectively), while morning performance on those days was significantly lower (p < 0.05), indicating that the time and day of the query may influence response accuracy. In contrast, no significant relationship was found between the difficulty level of the questions and the model's accuracy (p > 0.05).
Conclusions: Presently, ChatGPT exhibits inadequacies in its application to oral radiology training and clinical practice. Despite this, expectations for platform improvement and expansion in utility persist, particularly with increased data input and advancements in artificial intelligence.
Ethical Statement
Ethical approval There is no human as a participant in the study and ethical approval for this research was not required.
Supporting Institution
There is no
References
- 1. Rodrigues JA, Krois J, Schwendicke F. Demystifying artificial intelligence and deep learning in dentistry. Braz Oral Res 2021;35:e094.
- 2. Deng L. Artificial intelligence in the rising wave of deep learning: the historical path and futureoutlook. IEEE Signal Process Mag 2018;35(1):180-177.
- 3. Mogali SR. Initial impressions of ChatGPT for anatomy education. Anat Sci Educ 2024;17(2):444-447.
- 4. Abd-Alrazaq A, AlSaad R, Alhuwail D, Ahmed A, Healy PM, Latifi S, Aziz S, Damseh R, Alabed Alrazak S, Sheikh J. Large language models in medical education: opportunities, challenges, and future directions. JMIR Med Educ 2023;9:e48291.
- 5. Cadamuro J, Cabitza F, Debeljak Z, De Bruyne S, Frans G, Perez SM, Ozdemir H, Tolios A, Carobene A, Padoan A. Potentials and pitfalls of ChatGPT and natural-language artificial intelligence models for understanding laboratory medicine test results. Clin Chem Lab Med 2023;61(7):1158-1166.
- 6. Li H, Moon JT, Purkayastha S, Celi LA, Trivedi H, Gichoya JW. Ethics of large language models in medicine and medical research. Lancet Digit Health 2023;5(6):e333-e335.
- 7. Gilson A, Safranek CW, Huang T, Socrates V, Chi L, Taylor RA, Chartash D. How does ChatGPT perform on the United States Medical Licensing Examination? The implications of large language models for medical education and knowledge assessment. JMIR Med Educ 2023;9:e45312.
- 8. Arif TB, Munaf U, Ul-Haque I. The future of medical education and research: is ChatGPT a blessing or blight in disguise? Med Educ Online 2023;28(1):2181052.
- 9. Kung TH, Cheatham M, Medenilla A, Sillos C, De Leon L, Elepaño C, Madriaga M, Aggabao R, Diaz-Candido G, Maningo J, Tseng V. Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLOS Digit Health 2023;2(2):e0000198.
- 10. Yeo YH, Samaan JS, Ng WH, Ting PS, Trivedi H, Vipani A, Ayoub W, Yang JD, Liran O, Spiegel B, Kuo A. Assessing the performance of ChatGPT in answering questions regarding cirrhosis and hepatocellular carcinoma. Clin Mol Hepatol 2023;29(3):721-732.
- 11. Samaan JS, Yeo YH, Rajeev N, Hawley L, Abel S, Ng WH, Srinivasan N, Park J, Burch M, Watson R, Liran O, Samakar K. Assessing the Accuracy of Responses by the Language Model ChatGPT to Questions Regarding Bariatric Surgery. Obes Surg 2023;33(6):1790-1796.
- 12. Fatani B. ChatGPT for future medical and dental research. Cureus 2023;15(4):e37285.
- 13. de Souza LL, Lopes MA, Santos-Silva AR, Vargas PA. The potential of ChatGPT in oral medicine: a new era of patient care? Oral Surg Oral Med Oral Pathol Oral Radiol 2024;137(1):1-2.
- 14. Khurana S, Vaddi A. ChatGPT from the perspective of an academic oral and maxillofacial radiologist. Cureus 2023;15(6):e40053.
- 15. Mago J, Sharma M. The potential usefulness of ChatGPT in oral and maxillofacial radiology. Cureus 2023;15(7):e42133.
- 16. Bhayana R, Krishna S, Bleakney RR. Performance of ChatGPT on a radiology board-style examination: insights into current strengths and limitations. Radiology 2023;307(5):e230582.
- 17. Öztürk HP, Avsever İH, Şenel B, Ayran Ş, Özgedik HS, Baysal N. ChatGPT in oral and maxillofacial radiology education. Res Square 2023;3566948.
- 18. Dave T, Athaluri SA, Singh S. ChatGPT in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations. Front Artif Intell 2023;6:1169595.
- 19. Mallya S, Lam E. White and Pharoah's oral radiology: principles and interpretation. 8th ed. Elsevier Health Sciences 2018.
- 20. Suárez A, Díaz-Flores García V, Algar J, Gómez Sánchez M, Llorente de Pedro M, Freire Y. Unveiling the ChatGPT phenomenon: evaluating the consistency and accuracy of endodontic question answers. Int Endod J 2024;57(1):108-113.
- 21. Antaki F, Touma S, Milad D, El-Khoury J, Duval R. Evaluating the performance of ChatGPT in ophthalmology: an analysis of its successes and shortcomings. Ophthalmol Sci 2023;3(4):100324.
- 22. Bragazzi NL, Szarpak L, Piccotti F. Assessing ChatGPT’s potential in endodontics: preliminary findings from a diagnostic accuracy study. SSRN 2023;4631017.
Oral Radyoloji Eğitimi ve Uygulamasında ChatGPT-4o'nun Başarısının Değerlendirilmesi: Öncü Bir Araştırma
Abstract
Amaç: Bu çalışma, Chat Generative Pre-Train Omni (GPT-4o) platformunun oral radyoloji eğitimi ve pratiği konusunda anlama ve yorumlama performansını değerlendirmeyi amaçlamaktadır.
Gereç ve Yöntemler: Çalışmada, "White and Pharoah's Oral Radiology: Principles and Interpretation 8th Edition" kitabından alınan 99 sorudan oluşan bir set kullanılmıştır. Bu sorular, ChatGPT-4o tarafından günün üç farklı periyodunda, 10 gün boyunca yanıtlanmış ve her soru için toplam 60 yanıt üretilmiştir. Aynı soruları iki ağız, diş ve çene radyolojisi uzmanı da yanıtlamış ve verdikleri cevaplar ilgili ders kitabı ile doğrulanmıştır. ChatGPT-4o'nun cevapları ile uzmanların cevapları karşılaştırılmıştır.
Bulgular: Çalışmada ChatGPT-4o’nun doğru yanıt verme oranı %59,4 olarak bulunmuştur. Zaman temelli analiz, belirli gün ve saat dilimlerinde performans farklılıkları olduğunu ortaya koymuştur. Özellikle öğle ve akşam oturumlarında, birinci ve yedinci günlerdeki başarı oranı istatistiksel olarak anlamlı şekilde daha yüksekken (sırasıyla p = 0,003 ve p = 0,002), aynı günlerdeki sabah performansı anlamlı derecede daha düşüktü (p < 0,05). Bu bulgular, sorgulamanın yapıldığı gün ve saatin yanıt doğruluğunu etkileyebileceğini göstermektedir. Öte yandan, soruların zorluk düzeyi ile modelin doğruluk oranı arasında anlamlı bir ilişki bulunmamıştır (p > 0,05).
Sonuç: Mevcut durumda, ChatGPT oral radyoloji eğitimi ve klinik pratiğinde yetersizlikler göstermiştir. Buna rağmen, platformun daha fazla veri girişi ve yapay zekâdaki ilerlemelerle birlikte gelişmesi ve kullanım alanlarının genişlemesi konusunda beklentiler devam etmektedir.
References
- 1. Rodrigues JA, Krois J, Schwendicke F. Demystifying artificial intelligence and deep learning in dentistry. Braz Oral Res 2021;35:e094.
- 2. Deng L. Artificial intelligence in the rising wave of deep learning: the historical path and futureoutlook. IEEE Signal Process Mag 2018;35(1):180-177.
- 3. Mogali SR. Initial impressions of ChatGPT for anatomy education. Anat Sci Educ 2024;17(2):444-447.
- 4. Abd-Alrazaq A, AlSaad R, Alhuwail D, Ahmed A, Healy PM, Latifi S, Aziz S, Damseh R, Alabed Alrazak S, Sheikh J. Large language models in medical education: opportunities, challenges, and future directions. JMIR Med Educ 2023;9:e48291.
- 5. Cadamuro J, Cabitza F, Debeljak Z, De Bruyne S, Frans G, Perez SM, Ozdemir H, Tolios A, Carobene A, Padoan A. Potentials and pitfalls of ChatGPT and natural-language artificial intelligence models for understanding laboratory medicine test results. Clin Chem Lab Med 2023;61(7):1158-1166.
- 6. Li H, Moon JT, Purkayastha S, Celi LA, Trivedi H, Gichoya JW. Ethics of large language models in medicine and medical research. Lancet Digit Health 2023;5(6):e333-e335.
- 7. Gilson A, Safranek CW, Huang T, Socrates V, Chi L, Taylor RA, Chartash D. How does ChatGPT perform on the United States Medical Licensing Examination? The implications of large language models for medical education and knowledge assessment. JMIR Med Educ 2023;9:e45312.
- 8. Arif TB, Munaf U, Ul-Haque I. The future of medical education and research: is ChatGPT a blessing or blight in disguise? Med Educ Online 2023;28(1):2181052.
- 9. Kung TH, Cheatham M, Medenilla A, Sillos C, De Leon L, Elepaño C, Madriaga M, Aggabao R, Diaz-Candido G, Maningo J, Tseng V. Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLOS Digit Health 2023;2(2):e0000198.
- 10. Yeo YH, Samaan JS, Ng WH, Ting PS, Trivedi H, Vipani A, Ayoub W, Yang JD, Liran O, Spiegel B, Kuo A. Assessing the performance of ChatGPT in answering questions regarding cirrhosis and hepatocellular carcinoma. Clin Mol Hepatol 2023;29(3):721-732.
- 11. Samaan JS, Yeo YH, Rajeev N, Hawley L, Abel S, Ng WH, Srinivasan N, Park J, Burch M, Watson R, Liran O, Samakar K. Assessing the Accuracy of Responses by the Language Model ChatGPT to Questions Regarding Bariatric Surgery. Obes Surg 2023;33(6):1790-1796.
- 12. Fatani B. ChatGPT for future medical and dental research. Cureus 2023;15(4):e37285.
- 13. de Souza LL, Lopes MA, Santos-Silva AR, Vargas PA. The potential of ChatGPT in oral medicine: a new era of patient care? Oral Surg Oral Med Oral Pathol Oral Radiol 2024;137(1):1-2.
- 14. Khurana S, Vaddi A. ChatGPT from the perspective of an academic oral and maxillofacial radiologist. Cureus 2023;15(6):e40053.
- 15. Mago J, Sharma M. The potential usefulness of ChatGPT in oral and maxillofacial radiology. Cureus 2023;15(7):e42133.
- 16. Bhayana R, Krishna S, Bleakney RR. Performance of ChatGPT on a radiology board-style examination: insights into current strengths and limitations. Radiology 2023;307(5):e230582.
- 17. Öztürk HP, Avsever İH, Şenel B, Ayran Ş, Özgedik HS, Baysal N. ChatGPT in oral and maxillofacial radiology education. Res Square 2023;3566948.
- 18. Dave T, Athaluri SA, Singh S. ChatGPT in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations. Front Artif Intell 2023;6:1169595.
- 19. Mallya S, Lam E. White and Pharoah's oral radiology: principles and interpretation. 8th ed. Elsevier Health Sciences 2018.
- 20. Suárez A, Díaz-Flores García V, Algar J, Gómez Sánchez M, Llorente de Pedro M, Freire Y. Unveiling the ChatGPT phenomenon: evaluating the consistency and accuracy of endodontic question answers. Int Endod J 2024;57(1):108-113.
- 21. Antaki F, Touma S, Milad D, El-Khoury J, Duval R. Evaluating the performance of ChatGPT in ophthalmology: an analysis of its successes and shortcomings. Ophthalmol Sci 2023;3(4):100324.
- 22. Bragazzi NL, Szarpak L, Piccotti F. Assessing ChatGPT’s potential in endodontics: preliminary findings from a diagnostic accuracy study. SSRN 2023;4631017.
Details
| Primary Language | English |
|---|---|
| Subjects | Oral and Maxillofacial Radiology, Dentistry (Other) |
| Journal Section | Original Research Articles |
| Authors | |
| Publication Date | June 30, 2025 |
| Submission Date | January 21, 2025 |
| Acceptance Date | April 4, 2025 |
| Published in Issue | Year 2025Volume: 28 Issue: 2 |
Cumhuriyet Dental Journal (Cumhuriyet Dent J, CDJ) is the official publication of Cumhuriyet University Faculty of Dentistry. CDJ is an international journal dedicated to the latest advancement of dentistry. The aim of this journal is to provide a platform for scientists and academicians all over the world to promote, share, and discuss various new issues and developments in different areas of dentistry. First issue of the Journal of Cumhuriyet University Faculty of Dentistry was published in 1998. In 2010, journal's name was changed as Cumhuriyet Dental Journal. Journal’s publication language is English.
CDJ accepts articles in English. Submitting a paper to CDJ is free of charges. In addition, CDJ has not have article processing charges.
Frequency: Four times a year (March, June, September, and December)
IMPORTANT NOTICE
All users of Cumhuriyet Dental Journal should visit to their user's home page through the "https://dergipark.org.tr/tr/user" " or "https://dergipark.org.tr/en/user" links to update their incomplete information shown in blue or yellow warnings and update their e-mail addresses and information to the DergiPark system. Otherwise, the e-mails from the journal will not be seen or fall into the SPAM folder. Please fill in all missing part in the relevant field.
Please visit journal's AUTHOR GUIDELINE to see revised policy and submission rules to be held since 2020.