Improving YOLOv8 Instrument Detection Using Active Learning on Surgical Videos

Surgical Data Science Collective (SDSC) is a non-profit research organization committed to enhancing surgical outcomes by harnessing the potential of artificial intelligence. SDSC collaborates with surgeons to produce extensive metrics from surgical videos, offering them valuable insights to enhance their procedures. SDSC also assists surgeons in creating searchable libraries of their surgical videos, ensuring secure backups. By providing surgeons with their state-of-the-art technology, SDSC strives to enhance patient outcomes and advance the field of medicine.

Obtaining and working with high-quality data from surgical videos is a challenge. It is crucial for the success of SDSC projects to effectively curate this data and extract the necessary information for optimal training of the models. However, labeling every frame in hours-long videos with 30 frames per second is impossible. Moreover, this approach would result in a huge number of similar-looking images and an imbalance in the variety of surgical tools labeled. For instance, suction instruments are almost constantly present on the screen.

SDSC has collaborated with Lightly to successfully implement an innovative and dynamic active machine learning solution. This cutting-edge solution is designed to intelligently and actively select the most optimal frames for labeling and retraining purposes. Through this collaboration, SDSC and Lightly have combined their expertise to revolutionize the process of frame selection in surgical videos, making it more efficient and accurate than ever before to find the best subset of frames for labeling and model training.

SDSC has implemented an active machine-automated learning pipeline using Lightly that:

‍

1. Runs an inference on the selected videos with the selected pre-trained model (YOLOv8)
2. Creates all required Lightly files, organizes data for optimal processing in Lightly and uploads all data to an Amazon S3 bucket
3. Starts an EC2 instance (g4dn.2xlarge with 8 vCPUs, 32GB of system memory, one T4 GPU – image: Deep Learning AMI GPU PyTorch 1.13.1 (Amazon Linux 2)) configured to be a Lightly worker (configuration was done following this documentation)
4. Starts the Lightly worker and schedules the Lightly processing run in a Docker container on the EC2 instance 
5. Waits for the Lightly process (active learning component) to be complete 
6. Stops the EC2 instance
7. Sends images to Encord Annotate and creates an Annotation Project ready to be labeled and sent to labelers

Using Lightly, SDSC was able to:

• Process over 2.3 million frames within the first month of use
• 10x their labeling speed