1. HMM Technology in Surgery Timothy Kowalewski BioRobotics Laboratory Blake Hannaford, PhD Jacob Rosen, PhD Support: National Science Foundation, ITR Program John’s Hopkins University, Greg Hager, Allison Okamura, Russell Taylor & UW Center for Videoendoscopic Surgery, Mika Sinanan, et al.

2. OVERVIEW Part I: HMM Background and Experimental Setup HMM Background and Experimental Setup Initial Approach: White Box HMM’s Initial Approach: White Box HMM’s Current Approach: Black Box HMM’s Current Approach: Black Box HMM’s A Step Back: The Bigger Picture A Step Back: The Bigger Picture Research Goals and Future Work Research Goals and Future Work

3. OVERVIEW Part II: HMM Implementation Issues (noise, sufficient training, surgical ‘babbling’) HMM Implementation Issues (noise, sufficient training, surgical ‘babbling’) Lessons in VQ Lessons in VQ Toolkits available Toolkits available Variations on a Theme: different approaches to surgery via HMM’s Variations on a Theme: different approaches to surgery via HMM’s

4. Part I: HMM Background & Experimental Setup Surgery  Signals BioRobotics and HIT Lab Surgical Databasing

5. HMM Intro Computers are ‘deterministic’, so try that… Computers are ‘deterministic’, so try that… -record a waveform and do a file-compare -record a waveform and do a file-compare ? == Problem: we want to detect a ‘word’-- not an event or quantity …we need Abstraction Problem: we want to detect a ‘word’-- not an event or quantity …we need Abstraction

6. HMM Intro: in speech Need to transcend: Need to transcend: --timing of utterances --pitch, tone and volume --rates of speech --rates of speech --accents --accents --noise and ‘variation,’ etc --noise and ‘variation,’ etc in order to identify the ‘words’ in order to identify the ‘words’ Hopefully, adapt to errors, changes in Hopefully, adapt to errors, changes in language, and phonetics. language, and phonetics.

7. …Abstraction… HMM Intro The ‘Black Box’ Approach: The ‘Black Box’ Approach: know input expected output TRAIN: ? some input correct output USE: X

8. HMM Intro: Basics Markov Chains Markov Chains--States--Observations State Transition Matrix (A) State Transition Matrix (A) Initial State Distribution (  ) Initial State Distribution (  ) Of the States themselves! (O) Of the States themselves! (O)

9. Markov Models Markov Chains Markov Chains

10. HIDDEN Markov Models -- States are not observable (or even physically representable) -- Observations are probabilistic functions of state -- Observations are probabilistic functions of state -- State transitions are still probabilistic -- State transitions are still probabilistic Hungry Sleepy Asleep Dead Wake Up stop breathing (.01) Don’t start breathing (100%) stop breathing (.01) eat

11. HMM’s The Balls and Urns Formulation The Balls and Urns Formulation Observation: Color & Source Urn (seq.) [R #3] 1 2 3 RGBRGBRGBRGBRGBRGB

12. HMM’s The Balls and Urns Formulation The Balls and Urns Formulation 1 2 3 RGBRGBRGBRGBRGBRGB Observation: Color ONLY [R ?] HIDDEN

13. HMM’s The Balls and Urns Formulation The Balls and Urns Formulation 1 2 3 R-100% G- 0% B- 0% Observation: Color ONLY [R P(1)=1] R- 0% G- 100% B- 0% R- 0% G- 0% B- 100%

14. HMM’s The Balls and Urns Formulation The Balls and Urns Formulation Totally Distinct (not HMM) Weighted Mix (HMM) Uniform Distribution

15. HMM’s 1. Evaluation: --of ‘fit’ 2. Inference: --of ‘hidden’ state sequence (Synthesis/ AI) 3. Training: --Capturing the Abstract (4.) Comparison: --Another method of evaluation? Infer Seq. O q Evaluate Fit O P “Train” O I.C. Compare ? == 1 2

16. Surgery via Signals “It’s no longer blood and guts, it’s bits and bytes” “It’s no longer blood and guts, it’s bits and bytes” (Col. Richard Satava, MD) Prof. Of Surgery, UW; DARPA

17. The Blue Dragon

18. (Blue Signals)

19. Surgery via Signals “It’s no longer blood and guts, it’s bits and bytes” “It’s no longer blood and guts, it’s bits and bytes” (Col. Richard Satava, MD) Prof. Of Surgery, UW; DARPA

20. VR Simulator Databases Chuck Edmond MD (PI) & Lockheed Martin Over 100 Subjects Over 100 Subjects 400+ Total Trials 400+ Total Trials Over 100 Subjects Over 100 Subjects Commercializing Commercializing Urology / TURP ENT / ESS Robert Sweet MD (PI) & UW HIT Lab

21. ESS Simulator: Tool Position Non-MDJunior Resident Senior ResidentStaff ENT

22. Blue Dragon: Tool Position Expert Novice Left HandRight Hand

23. Blue Dragon: Tool Torques Expert Novice Left HandRight Hand

24. Part I: Initial Approach: White Box HMM’s

25. Surgery - Language Elements

26. (Taxonomy)

27. White Box Training

28. Force/Torque Signatures

29. (State Mesh)

30. (Model in P explanation)

31. (PIP Video)

32. Learning Curve - Markov Model Statistical Distance R1 R2 R3 R4 R5 E

33. Normalized Statistical Distance

34. Normalized Subjective Score (Video Analysis)

35. Normalized Completion Time

36. Normalized Trajectory Length

37. Correlation Between Subjective and Objective Assessment of Surgical Skill

38. Part I: Current Approach: Black Box HMM’s & Vector Quantization

39. White Box vs. Black Box Open/Defined vs. Hidden Open/Defined vs. Hidden Based on Extant Human Knowledge vs. Unobservable or Non-intuitive Assertions Based on Extant Human Knowledge vs. Unobservable or Non-intuitive Assertions Procedure Specific vs. Procedure independent of Cross-Procedural Procedure Specific vs. Procedure independent of Cross-Procedural Precise vs. Unpredictable Precise vs. Unpredictable … Grey Box ? … Grey Box ?

40. HMM’s By Analogy Speech Recognition   Words/Vocabulary  (dictionary)   Grammar/Syntax  (expression)Surgery   Surgical ‘words’  (VQ codebook)   Surgical Syntax  (Trained HMM)

41. VQ – Intro Purpose: Discretization & Abstraction Purpose: Discretization & Abstraction Used Initially in Image Processing / Compression Used Initially in Image Processing / Compression

42. (VQ intro1)

43. (VQ intro2)

44. (VQ intro3)

45. (VQ intro4)

46. (VQ intro5)

47. (VQ intro6)

48. (VQ intro7)

49. (VQ intro 3D)

50. (VQ distortion Curve)

51. Suturing Dictionary: VQ

52. Different Vocabularies across skill levels w/VQ

53. (VQ % word use)

54. HMM’s By Analogy (again) Speech Recognition   Words/Vocabulary  (dictionary)   Grammar/Syntax  (expression)Surgery   Surgical ‘words’  (VQ codebook)   Surgical Syntax  (Trained HMM)

55. …Abstraction… We Want: The ‘Black Box’ Approach: The ‘Black Box’ Approach: know input expected output TRAIN: ? some input correct output USE: X

56. Authoritative Performance Standard Black Box Performance  Surgeon’s Analysis  White Box Results  HMM Parametrs  Human Knowledge VQ Assumption Variable Under Test Performance Standard

57. Variables To Consider: Initial Conditions (Number of Iterations) Initial Conditions (Number of Iterations) Complexity of Model: Number of States Complexity of Model: Number of States Amount of Human Knowledge: State definitions, transition eliminations Amount of Human Knowledge: State definitions, transition eliminations Presence of Data Hybrids Presence of Data Hybrids Presence of Context Presence of Context Direction of Analysis (more or less ?) Direction of Analysis (more or less ?)

58. Sample Trial Using Random Initializations: Demo of # of I.C. Iterations …elegance?

59. Why Bother ? Can Hidden markov modeling offer cross- procedural surgical skill assessment? Can Hidden markov modeling offer cross- procedural surgical skill assessment? Can the technology be optimized and embedded? Can the technology be optimized and embedded?

60. Part I: A Step Back: The Bigger Picture

61. Studying Tools or Studying Surgery? ‘…the trees were in the way’ ‘…the trees were in the way’ Precision vs. Accuracy Precision vs. Accuracy Implementation Previous Progress vs. Lost in Analogy Implementation Previous Progress vs. Lost in Analogy

62. HMM’s are NOT A complete replacement of human surgeons or human accreditation – rather a tool for assessment or surgery A complete replacement of human surgeons or human accreditation – rather a tool for assessment or surgery

63. HMM’s are not: …the stand-alone analysis platform for surgical skill –rather a gray box, i.e., ‘both and” not “either or” …cumulatives, errors, etc. …the stand-alone analysis platform for surgical skill –rather a gray box, i.e., ‘both and” not “either or” …cumulatives, errors, etc. …without limitations. …without limitations.

64. Limitations Meant to assess and develop ONLY the ‘basic’ surgical skill set… ‘only 300 words required for daily speech’ Meant to assess and develop ONLY the ‘basic’ surgical skill set… ‘only 300 words required for daily speech’ Not in Not in

65. Research Goals & Future Work “The OR of the Future”

66. Analysis Goals: Obtain data-rich, diverse surgical database Obtain data-rich, diverse surgical database (BlueDragon, VR TURP, ENT Simulator) Realize standardized VQ for entire database Realize standardized VQ for entire database  Run HMM analysis (8 toolkits so far): white box vs. black box  Find optimal HMM parameters to characterize surgery  Embed the system on a single chip and it implement in hardware

67. Looking Ahead With a surgical language developed and an HMM database trained, embeddable possibilities include… With a surgical language developed and an HMM database trained, embeddable possibilities include… Real-time surgical evaluation/assistanceReal-time surgical evaluation/assistance Standardized skill set developed for surgeonsStandardized skill set developed for surgeons Artificial Intelligence for surgical assistanceArtificial Intelligence for surgical assistance Bandwidth reduction for tele-operationBandwidth reduction for tele-operation

69. Acknowledgments: JHU: Greg Hager NSF ITR (funding), Allison Okamura, Russ Taylor BRL: Blake Hannaford, Jacob Rosen, Jeff Brown http://brl.ee.washington.edu http://brl.ee.washington.edu UW Med/CVES: Mika Sininan, Lily Chang, Rob Sweet, Rick Satava HIT Lab: Rob Sweet, Suzanne Weghorst, Jeff Berkely, Ganesh Sankaranaraynan

70. OVERVIEW Part II: Lessons in VQ Lessons in VQ HMM Implementation Issues (noise, sufficient training, surgical ‘babbling’) HMM Implementation Issues (noise, sufficient training, surgical ‘babbling’) Variations on a Theme: different approaches to surgery via HMM’s Variations on a Theme: different approaches to surgery via HMM’s Toolkits available Toolkits available

71. VQ – Lessons Handedness Handedness Data-scaling Data-scaling Data Type /preservation of ‘importance’ Data Type /preservation of ‘importance’ Data Appending/transforming/conditioning Data Appending/transforming/conditioning Different Algorithms and HMM context Different Algorithms and HMM context (display data streams)

72. (4-D Data Stream, GUI Plots & VQ Code-Book)

73. HMM Implementation Issues The Surgical Babbling problem & Noise The Surgical Babbling problem & Noise O vector choice & Multiple Observations per state O vector choice & Multiple Observations per state Continuous vs. Discrete Continuous vs. Discrete State duration State duration Choice of Data sampling, Decimation/interpolation Choice of Data sampling, Decimation/interpolation Order of Model Order of Model Context Context Lots of variables (Highly Unconstrained) Lots of variables (Highly Unconstrained) Sufficient Training Sufficient Training

74. Continuous vs. Discrete Heads Or Tails Vs Vs Weight, size, etc. 50.32

75. State Duration P(State Duration) vs. Sample Time Std HMM state duration Variable Duration Prob(in A 11 for 100 samples) = A 11 100 << 1

76. Choice of Data sampling Rate, Decimation /interpolation P(State Duration) vs. Sample Time 1 sec Stay in one State for one second … Stay in one State for one second … …State duration, decimation, or interpolation are possible solutions …State duration, decimation, or interpolation are possible solutions

77. Context Sample Application of HMM’s (speech) Sample Application of HMM’s (speech) --Real Life Implementation-- --Real Life Implementation-- --Build in some ‘determinism’… “eyoo” --Train Distinct phonetic units (phonemes) with lots of samples and variability (same abstractive quality) ‘too’ –p1 ‘to’ –p2 ‘two’ –p3 ‘tool’ –p4 Prob. RecognitionContext and Grammar Link ‘too’ + (adverb) ‘to’ + (preposition) ‘two’ + (pl. noun) ‘tool’ (pronoun) + (article / adj) +

78. Context for Speech Sample Application of HMM’s (speech) Sample Application of HMM’s (speech) …easily translates into VQ paradigm …easily translates into VQ paradigm Block Diagram, Continuous Speech Recognition

79. HMM Implementation Conclusion Lots of variables (Highly Unconstrained) Lots of variables (Highly Unconstrained) Hope for Sufficient Training …only have trial and error to base results on. Hope for Sufficient Training …only have trial and error to base results on.

80. Variations on HMM’s in Surgery Multiple Model HMM’s: benefits (where to segment the tasks/models) Multiple Model HMM’s: benefits (where to segment the tasks/models) Adaptive Learning Adaptive Learning # of states (automated) dynamic of States # of states (automated) dynamic of States Modeling Intention/Mental States Modeling Intention/Mental States “Plato, Suture Here” “Plato, Suture Here”

81. Toolkits Available C Code: —HTK 3.x (C-code) Industrial Standard Hybrid models speech-specific —UHMM (C-code) Testing —Past BRL Projects (C) —JPL C-Code —JPL C-Code —Previous Thesis Work —Previous Thesis Work (contains modified HMM for (contains modified HMM for tweaking state durations) tweaking state durations) —GNU Distribution —Grant’s @ JHU MATLAB Code: —MATLAB Statistics 4.1 Has VQ extras and HMM’s Seems stable/robust Discrete only! —UW Toolkit (Matlab, BRL) full spectrum of models tutorial linked + GUI Beta-Version —H2M Toolkit (Matlab, French) GNU Distribution limited spectrum of use

82. To Do: Get HITL simulation people to hook up w/ sutchering sim and Haptics AR Toolkit. (Jeff B, Ganesh, Suzanne, new MechE guy) Get HITL simulation people to hook up w/ sutchering sim and Haptics AR Toolkit. (Jeff B, Ganesh, Suzanne, new MechE guy) Develop possibilities for Joint project(s). Develop possibilities for Joint project(s). Haptics-E info Haptics-E info