We use cookies to understand how you use our site and to improve your experience. This includes personalizing content and advertising. To learn more, click here. By continuing to use our site, you accept our use of cookies. Cookie Policy.

Features Partner Sites Information LinkXpress
Sign In
Advertise with Us

Download Mobile App

New Scoring Systems Increase Accuracy of AI-Generated Radiology Reports

By MedImaging International staff writers
Posted on 07 Aug 2023
Print article
Image: Scientists have designed a new way to score accuracy of AI-generated radiology reports (Photo courtesy of Freepik)
Image: Scientists have designed a new way to score accuracy of AI-generated radiology reports (Photo courtesy of Freepik)

Artificial intelligence (AI) tools that efficiently produce detailed narrative reports of CT scans or X-rays can significantly lighten the workload of busy radiologists. These AI reports go beyond simple identification of abnormalities and instead provide complex diagnostic information, detailed descriptions, nuanced findings, and appropriate degrees of uncertainty, similar to how human radiologists describe scan results. While several AI models capable of generating such detailed medical imaging reports have emerged, automated scoring systems meant to assess these tools are proving to be less effective at gauging their performance, according to a new study.

In the study, researchers at Harvard Medical School (Boston, MA, USA) tested various scoring metrics on AI-generated narrative reports and had six human radiologists read these reports. The analysis revealed that automated scoring systems performed poorly compared to human radiologists when it came to evaluating AI-generated reports. These systems misinterpreted and even missed significant clinical errors made by the AI tool. Ensuring the reliability of scoring systems is crucial for AI tools to continue improving and gaining clinicians' trust. However, the metrics tested in the study failed to reliably identify clinical errors in the AI reports, highlighting an urgent need for improvement and the development of high-fidelity scoring systems that accurately monitor tool performance.

In order to create better scoring metrics, the research team designed a new method called RadGraph F1 for evaluating the performance of AI tools generating radiology reports from medical images. Additionally, they created a composite evaluation tool called RadCliQ, which combines multiple metrics to produce a single score that more closely aligns with how a human radiologist would assess an AI model's performance. Using these new scoring tools, the researchers evaluated several state-of-the-art AI models and found a notable gap between their actual scores and the top possible scores.

Going forward, the researchers envision building generalist medical AI models capable of performing various complex tasks, including solving novel problems. Such AI systems could effectively communicate with radiologists and physicians about medical images, assisting in diagnosis and treatment decisions. The team also aims to develop AI assistants that can explain imaging findings directly to patients using everyday language, enhancing patient understanding and engagement. Ultimately, these advancements could revolutionize medical imaging practices, improving efficiency, accuracy, and patient care.

“Accurately evaluating AI systems is the critical first step toward generating radiology reports that are clinically useful and trustworthy,” said study senior author Pranav Rajpurkar, assistant professor of biomedical informatics in the Blavatnik Institute at HMS. “By aligning better with radiologists, our new metrics will accelerate development of AI that integrates seamlessly into the clinical workflow to improve patient care,”

Related Links:
Harvard Medical School 

Gold Supplier
128 Slice CT Scanner
Supria 128
Gold Supplier
IMRT Thorax Phantom
CIRS Model 002LFC
Wireless Flat Panel Detector
X-Ray Flat Panel Detector

Print article
Sun Nuclear -    Mirion



view channel
Image: The AI model improves tumor removal accuracy during breast cancer surgery (Photo courtesy of UNC School of Medicine)

AI Model Analyzes Tumors Removed Surgically in Real-Time

During breast cancer surgery, the surgeon removes the tumor, also known as a specimen, along with a bit of the adjacent healthy tissue to ensure all cancerous cells are excised. This specimen is then X-rayed... Read more


view channel
Image: MRI screen-detected breast cancers have been found to be most often invasive cancers (Photo courtesy of 123RF)

MRI Screen-Detected Breast Cancers Are Mostly Invasive

Annual breast MRI screening is advised for patients with a lifetime breast cancer risk exceeding 20%. There exists robust data about the features of mammographic screen-detected breast cancers, although... Read more


view channel
Image: FloPatch is a revolutionary tool that facilitates real-time precision in IV fluid management in sepsis (Photo courtesy of Flosonics)

Wireless, Wearable Doppler Ultrasound Revolutionizes Precision Fluid Management in Sepsis Care

When a patient comes to the hospital with sepsis, administering intravenous (IV) fluids is usually the first course of action. However, too much IV fluid can do more harm than good, causing additional... Read more

Nuclear Medicine

view channel
Image: An AI model can evaluate brain tumors on PET (Photo courtesy of Freepik)

AI Model for PET Imaging Determines Patient Response to Brain Tumor Treatments

The assessment of changes in metabolic tumor volume (MTV) through PET scans using specific radiotracers like F-18 fluoroethyl tyrosine (FET) plays a vital role in evaluating the treatment response in patients... Read more

Imaging IT

view channel
Image: The new Medical Imaging Suite makes healthcare imaging data more accessible, interoperable and useful (Photo courtesy of Google Cloud)

New Google Cloud Medical Imaging Suite Makes Imaging Healthcare Data More Accessible

Medical imaging is a critical tool used to diagnose patients, and there are billions of medical images scanned globally each year. Imaging data accounts for about 90% of all healthcare data1 and, until... Read more
Copyright © 2000-2023 Globetech Media. All rights reserved.