NA-MIC Project Weeks

Back to Projects List

Clinical Information Extraction via Locally Fine-Tuned LLMs

Key Investigators

• Paul Dumont (University of North Carolina at Chapel Hill, USA)
• Alexandre Buisson (University of North Carolina at Chapel Hill, USA)
• Juan Carlos Prieto (University of North Carolina at Chapel Hill, USA)
• Eduardo Caleme (University of North Carolina at Chapel Hill, USA)
• Lucia Cevidanes (University of North Carolina at Chapel Hill, USA)
• Steve Pieper (Isomics, USA)

Project Description

This project develops a privacy-preserving extraction framework using locally deployable open-weight LLMs to structure dense clinical narratives. A primary goal is to harmonize the diverse data and writing styles of different clinicians and doctors. Furthermore, we aim to bypass expensive and non-private cloud APIs by fine-tuning models to extract Common Data Elements (CDEs) from Orthodontic and TMJ progress notes entirely offline

Objective

1. Fine-tune local LLMs (Llama-3.1-8B, Qwen-2.5-7B) to parse variable clinical shorthand into deterministic JSON dictionaries.

2. Benchmark these local models against cloud baselines and human inter-rater ceilings for accuracy and computational efficiency.

3. Deploy these trained models directly into 3D Slicer by developing a custom extension.

Approach and Plan

1. Fine-tune Meta-Llama-3.1-8B for orthodontic notes and Qwen-2.5-7B for TMJ records.

2. Evaluate the extractions using exact match F1-scores.

3. Build a fully offline 3D Slicer extension for secure, on-device data processing.

Progress and Next Steps

Fine-tuned Llama-3.1-8B (LoRA) achieved a 0.740 F1 score on orthodontic notes , and fully fine-tuned Qwen-2.5-7B reached a 0.78 F1 score on TMJ records. Next Steps: Deploy these trained models directly within 3D Slicer by developing a custom extension with a dedicated user interface (UI) for clinical application.

Illustrations

Illustrative example of the pipeline for local clinical data extraction.

Dataset partitioning and subset allocation.

Representative Example of a Progress Note Paired / Labelling

Model benchmarking and matched external comparisons.

3D Slicer user interface

Background and References

NIH funding : R01DE024450