NA-MIC Project Weeks

Back to Projects List

Testing Local LLMs for Agentic Tasks via Slicer Skills

Key Investigators

Paul Dumont (University of North Carolina at Chapel Hill, USA)
Alexandre Buisson (University of North Carolina at Chapel Hill, USA)
Juan Carlos Prieto (University of North Carolina at Chapel Hill, USA)
Lucia Cevidanes (University of North Carolina at Chapel Hill, USA)
Steve Pieper (Isomics, USA)

Project Description

This project develops a free, confidential, and fully local alternative to cloud-based LLMs for medical imaging workflows. To bypass expensive and non-private cloud infrastructure, we are building an offline AI Agent for 3D Slicer powered by Ollama. Specifically, we aim to evaluate the capability of these local models to leverage existing “Slicer skills” to execute agentic user tasks via the Slicer API. Finally, we will benchmark the performance, context-awareness, and reliability of these local models against established cloud baselines like Claude.

Objective

Evaluate the capability of local LLMs (deployed via Ollama) to leverage the existing “Slicer skill” to execute agentic user tasks.
Benchmark the performance, context-awareness, and reliability of these local models against established cloud baselines (Claude).

Approach and Plan

Connect a local LLM client to existing Slicer MCP execution servers to enable code execution. Evaluate zero-shot coding accuracy on multi-step Slicer workflows using purely local inference

Progress and Next Steps

Progress: Reviewed current cloud-reliant MCP integrations (slicer-skill, mcp-slicer) and local LLM baselines (SlicerChat).

Illustrations

Results: Result in the 3D Slicer scene after a simple prompt with a local 7B Qwen model:

Using the Slicer MCP tool execute_python, write and run Python code that creates a red cube model and adds it to the 3D scene.
Using the Slicer MCP tool load_sample_data, load the MRHead sample dataset.

Chatbot Interface (Cline)

Background and References

slicerClaw: https://github.com/jumbojing/slicerClaw
mcp-slicer: https://github.com/zhaoyouj/mcp-slicer
slicer-skill: https://github.com/pieper/slicer-skill
SlicerChat: https://arxiv.org/abs/2407.11987

NIH funding NIDCR R01DE024450