NA-MIC Project Weeks
Back to Projects List
AI-Agent for SlicerAutomatedDentalTools
Key Investigators
- Paul Dumont (University of North Carolina at Chapel Hill, USA)
- Alexandre Buisson (University of North Carolina at Chapel Hill, USA)
- Juan Carlos Prieto (University of North Carolina at Chapel Hill, USA)
- Lucia Cevidanes (University of North Carolina at Chapel Hill, USA)
- Steve Pieper (Isomics, USA)
Project Description
Problem: The Slicer Automated Dental Tools extension provides robust craniofacial analysis, but complex module selection and parameter tuning create a steep learning curve for users.
Solution: This project introduces an AI Agent and chatbot UI integrated into 3D Slicer to streamline the workflow. By allowing drag-and-drop inputs and natural language prompts, users can easily request complex tasks (e.g., segmentation, landmarking, or orientation on CBCT/IOS). The agent autonomously translates these requests into actions, selecting the right tools and automatically configuring the parameters to execute the workflow.
Objective
- Local LLM Integration: Uses Ollama to run a lightweight local model (currently llama3:latest).
- Cross-Encoder Retrieval: the system leverages a Cross-Encoder. It directly scores the semantic relevance between the user’s query and tool use cases to retrieve the top 3 most appropriate modules.
- Autonomous Execution: The LLM analyzes the retrieved context, selects the optimal tool, extracts the required parameters from the user’s text, and automatically triggers the execution within Slicer.
Approach and Plan
- Build the core backend pipeline integrating the Cross-Encoder retrieval model and the local LLM.
- Deploy the pipeline within 3D Slicer and connect it to the user-facing chatbot UI.
- Implement an autonomous feedback loop. The model will verify if the selected Slicer tools executed successfully or encountered an error, and provide real-time feedback to the user.
Progress and Next Steps
Current Progress The backend retrieval system is completely operational. The Cross-Encoder model reliably identifies and selects the appropriate Slicer tool from natural language input.
Next Steps
-
LLM Integration: Implement the logic for the local LLM to parse the user’s prompt, auto-fill the required parameters, and trigger the tool execution.
-
Slicer Deployment: Embed the interactive UI and connect the entire AI pipeline directly within the 3D Slicer environment.
Illustrations
What would the Slicer user interface look like?
Slicer Automated Dental Tools Overview :
Tool demo:
https://github.com/user-attachments/assets/6c0fbb3c-7a7f-4aa8-aa00-e0512f3cf760
Results comparing local LLMs and Cloud-based Models:
Background and References
Github link: https://github.com/DCBIA-OrthoLab/SlicerAutomatedDentalTools This work was supported by the National Institute of Dental and Craniofacial Research (NIDCR) of the National Institutes of Health under Award Number R01DE024450.