1. Automatic Detection and Segmentation of Robot-Assisted Surgical Motions presented by Henry C. Lin Henry C. Lin, Dr. Izhak Shafran, Todd E. Murphy, Dr. David D. Yuh, Dr. Allison M. Okamura, Dr Gregory D. Hager

2. MICCAI 2005 ERC-CISST Johns Hopkins University 2 Authors Henry Lin PhD Student Computer Science Izhak Shafran Research Scientist ECE Todd Murphy MS, 2004 Mechanical Engineering David Yuh Surgeon Cardiac Surgery Allison Okamura Assistant Professor Mechanical Engineering Gregory Hager Professor Computer Science

3. MICCAI 2005 ERC-CISST Johns Hopkins University 3 Can we automatically detect and segment the surgical motions common in both videos? Can we quantitatively and objectively determine which video is an expert surgeon and which is an intermediate surgeon? Motivation Expert Intermediate

4. MICCAI 2005 ERC-CISST Johns Hopkins University 4 Cartesian Position Plots - Left Manipulator --- Pull suture with left hand --- Move to middle with needle Expert Surgeon - trial 4Intermediate Surgeon - trial 22

5. MICCAI 2005 ERC-CISST Johns Hopkins University 5 Previous Work Darzi, et al. Imperial College Surgical Assessment Device (ICSAD) quantified motion information by tracking electromagnetic markers on a trainee’s hands. Rosen, et al. Used force/torque data from laparoscopic trainers to create a hidden Markov model task decomposition specific to each surgeon.

6. MICCAI 2005 ERC-CISST Johns Hopkins University 6 Goals Train LDA-based statistical models with labeled motion data of an expert surgeon and an intermediate surgeon. Be able to accurately parse unlabeled raw motion data into a labeled sequence of surgical gestures in an automatic and efficient way. Ultimately create evaluation metrics to benchmark surgical skill.

7. MICCAI 2005 ERC-CISST Johns Hopkins University 7 Corpus 78 motion variables acquired at 10Hz (we use 72 of them) 4-throw suturing task 15 expert trials 12 intermediate trials each trial roughly 60 seconds in length

8. MICCAI 2005 ERC-CISST Johns Hopkins University 8 Gesture Vocabulary 1. Reach for needle 2. Position needle 3. Insert and push needle through tissue 6. Pull suture with left hand 4. Move to middle with needle (left hand) 8. Orient needle with both hands 7. Pull suture with right hand 5. Move to middle with needle (right hand)

9. MICCAI 2005 ERC-CISST Johns Hopkins University 9 API signals X(1,t) X(78,t) X(t) Local Feature Extraction L(t) Feature Normalization N(t) Linear Discriminant Analysis Y(t) Probabilistic (Bayes) Classifier P(Y(t)|C) Probabilistic Models for Surgical Motions C(t) System Approach API signals X(1,t) X(72,t) X(t) Local Feature Extraction

10. MICCAI 2005 ERC-CISST Johns Hopkins University 10 Local Feature Extractions X(k t ) ++++ X(k t-m+1 )X(k t+m-1 ) ++ X(k t-m )X(k t+m ) |L(k t )| = (2m+1)|X(k t )| Example: m=5, |L| = 792

11. MICCAI 2005 ERC-CISST Johns Hopkins University 11 API signals X(1,t) X(78,t) X(t) Local Feature Extraction L(t) Feature Normalization N(t) Linear Discriminant Analysis Y(t) Probabilistic (Bayes) Classifier P(Y(t)|C) Probabilistic Models for Surgical Motions C(t) System Approach API signals X(1,t) X(72,t) X(t) Local Feature Extraction L(t) Feature Normalization N(t) Linear Discriminant Analysis

12. MICCAI 2005 ERC-CISST Johns Hopkins University 12 Linear Discriminant Analysis x1x1 x2x2 The objective of LDA is to perform dimensionality reduction while preserving as much of the class discriminatory information as possible.

13. MICCAI 2005 ERC-CISST Johns Hopkins University 13 where the linear transformation matrix W is estimated by maximizing the Fisher discriminant. Linear Discriminant Analysis Fisher discriminant - ratio of distance between the classes and the average variance of each class LDA class-labeled motion data expected reduced output dimension reduced dimension motion data

14. MICCAI 2005 ERC-CISST Johns Hopkins University 14 LDA Reduction (6 Labeled Classes, 3 Dimensions) Expert Surgeon

15. MICCAI 2005 ERC-CISST Johns Hopkins University 15 LDA Reduction (6 Labeled Classes, 3 Dimensions) Intermediate Surgeon

16. MICCAI 2005 ERC-CISST Johns Hopkins University 16 Storage Savings of LDA MethodTemporalNeighbors (m)Space required (values) Rawno-432,000 Raw + temporal yes54,752,000 LDA-basedyes518,000 For a 10 minute procedure (6000 input samples)

17. MICCAI 2005 ERC-CISST Johns Hopkins University 17 API signals X(1,t) X(78,t) X(t) Local Feature Extraction L(t) Feature Normalization N(t) Linear Discriminant Analysis Y(t) Probabilistic (Bayes) Classifier P(Y(t)|C) Probabilistic Models for Surgical Motions C(t) System Approach API signals X(1,t) X(78,t) X(t) Local Feature Extraction L(t) Feature Normalization N(t) Linear Discriminant Analysis Y(t) Probabilistic (Bayes) Classifier P(Y(t)|C) Probabilistic Models for Surgical Motions C(t)

18. MICCAI 2005 ERC-CISST Johns Hopkins University 18 Results ‘Leave 2 out’ cross-validation paradigm used. 15 expert trials, 15 rounds Output of 2 test trials were compared against the manually labeled data. The average across the 15 tests was used to measure performance. Training set (13) Test set (2)

19. MICCAI 2005 ERC-CISST Johns Hopkins University 19 Results

20. MICCAI 2005 ERC-CISST Johns Hopkins University 20 Results nNumber of labeled classes LDA output dimensions % correct 16391.26 26491.46 36591.14 45391.06 55491.34 65392.09 75491.92 84391.88

21. MICCAI 2005 ERC-CISST Johns Hopkins University 21 Contributions An automated and space efficient method to accurately parse raw motion-data into a labeled sequence of surgical motions. Results support previous work that there exist quantitative differences in the varying skill levels of surgeons. Linear discriminant analysis is a useful tool for separating surgical motions.

22. MICCAI 2005 ERC-CISST Johns Hopkins University 22 Future Work Currently getting synchronized stereo video and API data. Will allow vision-based segmentation methods to complement our statistical methods. Apply to a larger set of expert surgeons to other representative surgical tasks. Create performance metrics to be used as benchmarks for surgical skill evaluation.

23. MICCAI 2005 ERC-CISST Johns Hopkins University 23 Acknowledgements Minimally Invasive Surgical Training Center at the Johns Hopkins Medical School (MISTC-JHU) - Dr. Randy Brown, Sue Eller Intuitive Surgical Inc. - Chris Hasser, Rajesh Kumar National Science Foundation

24. Automatic Detection and Segmentation of Robot-Assisted Surgical Motions Henry C. Lin, Dr. Izhak Shafran, Todd E. Murphy, Dr. David D. Yuh, Dr. Allison M. Okamura, Dr Gregory D. Hager Thank you! Any questions?