We use cookies to understand how you use our site and to improve your experience. This includes personalizing content and advertising. To learn more, click here. By continuing to use our site, you accept our use of cookies. Cookie Policy.

Features Partner Sites Information LinkXpress
Sign In
Advertise with Us

Download Mobile App

New Scoring Systems Increase Accuracy of AI-Generated Radiology Reports

By MedImaging International staff writers
Posted on 07 Aug 2023
Print article
Image: Scientists have designed a new way to score accuracy of AI-generated radiology reports (Photo courtesy of Freepik)
Image: Scientists have designed a new way to score accuracy of AI-generated radiology reports (Photo courtesy of Freepik)

Artificial intelligence (AI) tools that efficiently produce detailed narrative reports of CT scans or X-rays can significantly lighten the workload of busy radiologists. These AI reports go beyond simple identification of abnormalities and instead provide complex diagnostic information, detailed descriptions, nuanced findings, and appropriate degrees of uncertainty, similar to how human radiologists describe scan results. While several AI models capable of generating such detailed medical imaging reports have emerged, automated scoring systems meant to assess these tools are proving to be less effective at gauging their performance, according to a new study.

In the study, researchers at Harvard Medical School (Boston, MA, USA) tested various scoring metrics on AI-generated narrative reports and had six human radiologists read these reports. The analysis revealed that automated scoring systems performed poorly compared to human radiologists when it came to evaluating AI-generated reports. These systems misinterpreted and even missed significant clinical errors made by the AI tool. Ensuring the reliability of scoring systems is crucial for AI tools to continue improving and gaining clinicians' trust. However, the metrics tested in the study failed to reliably identify clinical errors in the AI reports, highlighting an urgent need for improvement and the development of high-fidelity scoring systems that accurately monitor tool performance.

In order to create better scoring metrics, the research team designed a new method called RadGraph F1 for evaluating the performance of AI tools generating radiology reports from medical images. Additionally, they created a composite evaluation tool called RadCliQ, which combines multiple metrics to produce a single score that more closely aligns with how a human radiologist would assess an AI model's performance. Using these new scoring tools, the researchers evaluated several state-of-the-art AI models and found a notable gap between their actual scores and the top possible scores.

Going forward, the researchers envision building generalist medical AI models capable of performing various complex tasks, including solving novel problems. Such AI systems could effectively communicate with radiologists and physicians about medical images, assisting in diagnosis and treatment decisions. The team also aims to develop AI assistants that can explain imaging findings directly to patients using everyday language, enhancing patient understanding and engagement. Ultimately, these advancements could revolutionize medical imaging practices, improving efficiency, accuracy, and patient care.

“Accurately evaluating AI systems is the critical first step toward generating radiology reports that are clinically useful and trustworthy,” said study senior author Pranav Rajpurkar, assistant professor of biomedical informatics in the Blavatnik Institute at HMS. “By aligning better with radiologists, our new metrics will accelerate development of AI that integrates seamlessly into the clinical workflow to improve patient care,”

Related Links:
Harvard Medical School 

Gold Member
Solid State Kv/Dose Multi-Sensor
Endoscopic Ultrasound Fine Needle Biopsy Device
Mobile Digital C-arm X-Ray System
X-Ray Detector

Print article


Nuclear Medicine

view channel
Image: Researchers have identified a new imaging biomarker for tumor responses to ICB therapy (Photo courtesy of 123RF)

New PET Biomarker Predicts Success of Immune Checkpoint Blockade Therapy

Immunotherapies, such as immune checkpoint blockade (ICB), have shown promising clinical results in treating melanoma, non-small cell lung cancer, and other tumor types. However, the effectiveness of these... Read more

Imaging IT

view channel
Image: The new Medical Imaging Suite makes healthcare imaging data more accessible, interoperable and useful (Photo courtesy of Google Cloud)

New Google Cloud Medical Imaging Suite Makes Imaging Healthcare Data More Accessible

Medical imaging is a critical tool used to diagnose patients, and there are billions of medical images scanned globally each year. Imaging data accounts for about 90% of all healthcare data1 and, until... Read more
Copyright © 2000-2024 Globetech Media. All rights reserved.