MIRACLE Robotic Left Lateral Sectionectomy 12M (RLLS12M) Dataset

Introduction

We contribute the first robotic left lateral sectionectomy (RLLS) dataset, RLLS12M, with hierarchical annotation, addressing the lack of benchmark datasets for hierarchical surgical workflow analysis. This RLLS12M dataset contains abundant real-world scenarios, providing a new incubator for developing AI algorithms closer to the clinical environment.

Dataset Description

This dataset consists of 50 RLLS videos of 50 patients operated by 5 surgeons in the Chinese PLA General Hospital, including 2,113,510 annotated video frames. We apply a hierarchical surgical interpretation, denoted as “ST<IAO>”. Our annotation is a three-level surgical interpretation with 6 annotation types, including 6 steps (S), 15 tasks (T), 38 triplet activities (<IAO>), 11 instruments (I), 8 actions (A), and 16 objects (O). All frames are collectively annotated by 3 expert surgeons. There is a total of 12,681,060 annotation entities in the proposed RLLS dataset, making it a unique robotic surgery dataset with hierarchical annotations. With annotated frames, our dataset provides a satisfactory group of relations that includes 3 inter-relations from combinations of annotation types and 6 intra-relations from each annotation type.

Fig. 1: The illustration of inter- and intra- relations from a three-level annotation in the RLLS12M dataset.

For example, there is an inter-relation between the task transection of the liver parenchyma and the activity bipolar forceps coagulate liver parenchyma, and there is an activity-level intra-relation of bipolar forceps coagulate liver parenchyma and cautery hook dissects liver parenchyma.

Fig. 2: Summary of the ST⟨IAO⟩ hierarchy of temporal annotations in the RLLS dataset. The blobs with the same color indicate that they belong to the same sub-hierarchy. Note that activity annotation is the fundamental element in our surgery interpretation, which has multiple occurrences across particular groups of tasks, such as AI1.

Data Split

We provide two experimental configurations in RLLS12M:

1. RLLS
   • The RLLS configuration only has RLLS sequences, including 29 videos from 2 expert surgeons for training.
   • The test set consists of in total 17 RLLS videos from the other 3 expert surgeons.
2. RLLS-w-RC
   • RLLS with robotic cholecystectomy, or RLLS-w-RC, contains 4 videos of RLLS with RC from different experts to set up the experiment with leave-one-out-cross-validation (LOOCV).
   • The 4 RLLS-w-RC videos have a partially overlapped domain knowledge to RLLS, which aims to evaluate the robustness of the proposed relation-based framework.

License

The RLLS12M dataset is released under the Creative Commons license CC BY-NC-SA 4.0 LICENSE:

This implies that:

• The dataset cannot be redistributed,
• The dataset is only for research purposes,
• The dataset cannot be used for any commercial purposes,
• The dataset can be transformed (additional annotations, etc.),
• Redistribution or transfer of data or data links is strictly prohibited,

By downloading and using this dataset, you agree to these terms and conditions.

Download

Please email shangzhao@ustc.edu.cn for download links.