1. www.roboticvision.org roboticvision.org ARC Centre of Excellence for Robotic Vision Vision-guided medical robotics: From laparoscopic diagnosis towards arthroscopic surgery MSc. Andrés F. Mármol Vélez

2. www.roboticvision.org roboticvision.org ARC Centre of Excellence for Robotic Vision Outline Short biography. Medical background. Part I: Liver laparoscopy –Acknowledgement. –Project overview. –Personal contribution. Part II: Knee arthroscopy –Perspectives at QUT. –Proposed roadmap. –Discussion.

3. www.roboticvision.org roboticvision.org ARC Centre of Excellence for Robotic Vision Short biography 2004-2008: BSc. Mechatronics. 2009: University tutoring. 2009-2011: MSc. Mechatronics. 2012-2014: On-site PM at ISRA Vision AG. 2015: QUT-ACRV. LabVIEW, Microcontroler electronics, Autodesk CAD.

4. www.roboticvision.org roboticvision.org ARC Centre of Excellence for Robotic Vision Medical background Laparoscopic vs. open surgery [2] Knee arthroscopy [3] Laparoscope[1]

5. www.roboticvision.org roboticvision.org ARC Centre of Excellence for Robotic Vision Acknowledgement Hamburg University of Technology (TUHH). Vision Systems (2010-2015): –Prof.-Eng Rolf-Rainer Grigat. –(Dr.-Eng) Jan Marek Marcinczak. –MSc. Udo von Öhsen. –MSc. Isabel Patiño. –Olaf Zeuge.

6. www.roboticvision.org roboticvision.org ARC Centre of Excellence for Robotic Vision Project overview Motivation: Clinical diagnosis of cirrhosis using endoscopic images. –Minimally invasive procedure. –Consistent diagnosis. Proposed solution: –Part 1: Sparse reconstruction (SfM). –Part 2: Total variation dense reconstruction.

7. www.roboticvision.org roboticvision.org ARC Centre of Excellence for Robotic Vision Project overview Liver endoscopy [4] Liver cirrhosis [5]

8. www.roboticvision.org roboticvision.org ARC Centre of Excellence for Robotic Vision Part 1: SfM Strategy SfM overview [6]

9. www.roboticvision.org roboticvision.org ARC Centre of Excellence for Robotic Vision Challenges Specular reflections, shadow, motion: –Segmentation, RANSAC/MSAC Endoscopy snapshots[4]

10. www.roboticvision.org roboticvision.org ARC Centre of Excellence for Robotic Vision Challenges SfM on a non-rigid environment: –Stiffness assumption. Reconstruction with reduced baseline: –Master thesis. Motions in endoscopy [7]

11. www.roboticvision.org roboticvision.org ARC Centre of Excellence for Robotic Vision Challenges Evaluation: –Ground truth, hand-eye calibration. Evaluation uncertainties[7]

12. www.roboticvision.org roboticvision.org ARC Centre of Excellence for Robotic Vision Setup Phantom livers. Endoscopes. 5 DoF custom-made robot. Endoscope and phantoms [8]. 5 DoF robot [9]

13. www.roboticvision.org roboticvision.org ARC Centre of Excellence for Robotic Vision Setup Polaris Vicra Optical Tracking System. NDI Polaris tracking system [10].

14. www.roboticvision.org roboticvision.org ARC Centre of Excellence for Robotic Vision Contribution Frame selection: –FME –KME Triangulation uncertainty [11]. Error propagation impact [9].

15. www.roboticvision.org roboticvision.org ARC Centre of Excellence for Robotic Vision Contribution Evaluation: –Robot inverse kinematics. Translation and rotation due to axis 4 motion [9].

16. www.roboticvision.org roboticvision.org ARC Centre of Excellence for Robotic Vision Contribution 3D structure: –Optimal triangulation by Hartley. –ePnP by Lepetit. –SBA by Rabaud. Reconstruction overview [7].

17. www.roboticvision.org roboticvision.org ARC Centre of Excellence for Robotic Vision Results Initial point cloud:Ground truth: Initial 3D reconstruction [7]. Ground truth for evaluation [8].

18. www.roboticvision.org roboticvision.org ARC Centre of Excellence for Robotic Vision Results Final 3D reconstruction [12].

19. www.roboticvision.org roboticvision.org ARC Centre of Excellence for Robotic Vision Perspectives at QUT Prof.-Eng Jonathan Roberts and Prof.-MD Ross Crawford. Knee arthroscopy. Motivation and hypotheses: –Aging population. –Physicians’ learning curve. –Suboptimal leg/arthroscope handling. –Unintuitive angled optics. –Involuntary injuries.

20. www.roboticvision.org roboticvision.org ARC Centre of Excellence for Robotic Vision Challenges Water infusion vs. air insufflation: –Bubbles, reflections, floating tissue. Non-rigid tissue: –Tracking, occlusions. Arthroscopy snapshots[13]

21. www.roboticvision.org roboticvision.org ARC Centre of Excellence for Robotic Vision Challenges Leg handling Tool tracking Real time processing(!) …

22. www.roboticvision.org roboticvision.org ARC Centre of Excellence for Robotic Vision Roadmap

23. www.roboticvision.org roboticvision.org ARC Centre of Excellence for Robotic Vision Thank you for your time Questions? Discussion

24. www.roboticvision.org roboticvision.org ARC Centre of Excellence for Robotic Vision References [1]: Adapted from World Laparoscopy Hospital. http://www.laparoscopyhospital.com/frequently_asked_qu estion_about_laparoscope.html. [Online; accessed 21- May-2015]. [2]: Adapted from iHealthBlogger. http://www.ihealthblogger.com/2013/05/problems-after- gallbladder-removal.html. [Online; accessed 21-May- 2015]. [3]: Adapted from MedlinePlus. http://www.nlm.nih.gov/medlineplus/ency/imagepages/106 3.htm. [Online; accessed 21-May-2015]. [4]: Courtesy of the Universitätsklinikum Hamburg-Eppendorf UKE. Department of internal medicine. [5]: Jeanne Marrazzo David H. Spach, H. Nina Kim. Hepatitis web study. http://depts.washington.edu/hepstudy/definitions/. [Online; accessed 29-Oct-2013].

25. www.roboticvision.org roboticvision.org ARC Centre of Excellence for Robotic Vision References [6]: Jan Marek Marcinczak, Sven Painer, Rolf-Rainer Grigat: Sparse Reconstruction of Liver Cirrhosis from Monocular Mini-Laparoscopic Sequences. SPIE (2015). [7]: Andres Marmol: Implementation and evaluation of optimal 3D reconstruction algorithms in endoscopic videos. TUHH Thesis (2011). [8]: Isabel Patino: Hand-Eye calibration of a laparascope using optical tracking systems. TUHH Thesis (2013). [9]: Udo von Öhsen: Ego-motion estimation in laparoscopic videos. TUHH Thesis (2011). [10]: NDI Medical Website. http://www.ndigital.com/medical/products/polaris-family/. [Online; accessed 14-May-2015].

26. www.roboticvision.org roboticvision.org ARC Centre of Excellence for Robotic Vision References [11]: Hartley & Zisserman: Multiple view geometry in computer vision. Cambridge (2004). [12]: Courtesy of the TUHH Vision Systems group. Dipl.-Eng. Jan Marek Marcinczak. [13]: Courtesy of The Prince Charles Hospital. Prof.-MD Ross Crawford.

27. www.roboticvision.org roboticvision.org ARC Centre of Excellence for Robotic Vision Publications 2015 Jan Marek Marcinczak, Sven Painer, Rolf- Rainer Grigat Sparse Reconstruction of Liver Cirrhosis from Monocular Mini-Laparoscopic Sequences Maren Brumm, Jan Marek Marcinczak, Rolf- Rainer Grigat Improved Confidence Measures for Variational Optical Flow

28. www.roboticvision.org roboticvision.org ARC Centre of Excellence for Robotic Vision Publications 2014 Bidisha Chakraborty, Jan Marek Marcinczak, Rolf-Rainer Grigat Classification of Weak Specular Reflections in Laparoscopic Images Jan Marek Marcinczak, Rolf-Rainer Grigat Total Variation Based 3D Reconstruction from Monocular Laparoscopic Sequences Rahul Sinha, Jan Marek Marcinczak, Rolf- Rainer Grigat Automatic Seed Selection for Segmentation of Liver Cirrhosis in Laparoscopic Sequences

29. www.roboticvision.org roboticvision.org ARC Centre of Excellence for Robotic Vision Publications 2013 Jan Marek Marcinczak, Rolf-Rainer Grigat Closed Contour Specular Reflection Segmentation in Laparoscopic Images Jan Marek Marcinczak, Siddharth Kumar, Rolf-Rainer Grigat Detection of Lymphangiectasia in Laparoscopic Images: a Semi-interactive Approach

30. www.roboticvision.org roboticvision.org ARC Centre of Excellence for Robotic Vision Publications 2012 Andrés F. Marmol-Velez, Jan Marek Marcinczak, Rolf-Rainer Grigat Structure from Motion Based Approaches to 3D Reconstruction in Minimal Invasive Laparoscopy Jan Marek Marcinczak, Udo von Öhsen, Rolf-Rainer Grigat A Robust Motion Estimation System for Minimal Invasive Laparoscopy Udo von Öhsen, Jan Marek Marcinczak, Andrés F. Marmol Velez, Rolf-Rainer Grigat Keyframe Selection for Robust Pose Estimation in Laparoscopic Videos

31. www.roboticvision.org roboticvision.org ARC Centre of Excellence for Robotic Vision Publications 2011 Jan Marek Marcinczak, Ulrike Denzer, Ansgar Lohse, Rolf-Rainer Grigat Quantitative Bestimmung von Leberzirrhose mittels Co-Occurrence Analyse

32. www.roboticvision.org roboticvision.org ARC Centre of Excellence for Robotic Vision