The AI in Medicine Program at Mass General Brigham works on self-supervised and foundation models for medical imaging, with a focus on radiotherapy planning and 3D segmentation tasks.

Their work spans MRI, CT, and other volumetric modalities - domains where labeled data is scarce, 3D pipelines are complex, and robust pretrained models are still underdeveloped. 

Medical imaging pipelines at the lab are highly heterogeneous: models must support full 3D volumes, non-standard intensity distributions, variable voxel spacing, and task-specific augmentation strategies. 

Unlike 2D vision, there is no widely adopted pretrained backbone for 3D CT segmentation that works consistently across datasets.

This creates several technical constraints:

• Existing 3D pretrained models tend to be dataset-specific rather than broadly generalizable.
• Most architectures require full model fine-tuning, which is slow and expensive.
• Public SSL repositories weren’t designed for 3D or medical pipelines, making adaptation difficult.
• Augmentations from natural images (e.g., color jitter) are not meaningful for CT.

The team wanted to move beyond task-specific tuning and instead train a DINO-based CT foundation model that could serve multiple oncology use cases, ideally requiring only light downstream adaptation.

With five researchers leading SSL efforts inside a 25-person lab, they needed an implementation that was clean, modular, and easy for several PhD students to use consistently.

To keep experimentation consistent, they standardized on: MONAI for medical-imaging data handling, PyTorch Lightning for workflow orchestration, Lightly SSL for the DINOv2 implementation internal config system (“sparkwheel” for experiment management).

The team first experimented with Meta’s DINO repositories, but the code complexity and frequent upstream changes made collaboration difficult. LightlySSL became the natural choice, especially since the group had already used Lightly in earlier self-supervised projects for about four years, which meant it fit naturally into their workflow and was easy for the team to adopt.

Choosing LightlySSL meant a clear and well-structured DINOv2 workflow that worked seamlessly with MONAI and PyTorch Lightning. The implementation was straightforward to extend to 3D and provided a reproducible setup that fit cleanly into their existing pipeline. It also reduced coordination overhead across PhD students, who could now work from a shared, consistent configuration.

Adapting DINOv2 for CT

To make DINO suitable for volumetric CT, the team introduced CT-appropriate augmentations and extended the pipeline to support 3D inputs. Using Lightly, they were able to:

• Replace color jitter with histogram-shift augmentations appropriate for CT
• Add 3D affine transformations to capture anatomical invariance
• Integrate SimCLR-style medical augmentations validated in earlier projects
• Support volumetric patch extraction aligned with CT geometry
• Incorporate the workflow into their internal experiment-management system

These modifications enabled the model to capture finer radiological structure than their previous masked autoencoder (MAE) based setup.

With LightlySSL, the team built a DINOv2-based CT foundation model that demonstrated promising early results for segmentation and served as a stable baseline across several ongoing studies.

The new workflow enabled them to:

• integrate DINOv2  into a 3D SSL pipeline previously limited to MAE
• improve feature-level representations through volumetric augmentations,
• establish a reproducible training setup shared across PhD researchers,
• produce DINOv2 baselines requested by peer reviewers with minimal overhead,
• support downstream projects such as vision-language models and SAM-style segmentation approaches.

The foundation model now underpins multiple research directions in the lab and accelerates internal experimentation by reducing engineering complexity. Check out the project on Github.