12 August 2015 Chair for Computer Aided Medical Procedures & Augmented Reality Department of Computer Science | Technische Universität München Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Automatic Feature Generation for Endoscopic Image Classification Ulrich Klank

CAMP | Department of Computer Science | Technische Universität München | 12 August Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Endoscopic Image Classification  Synchronization of surgeries by phases is essential for Workflow Project 1  Analyzing endoscopic images supports the classification of the current state into a phase. 1 A. Ahmadi, T. Sielhorst, R. Stauder, M. Horn, H. Feussner, N. Navab -Recovery of surgical workflow without explicit models

CAMP | Department of Computer Science | Technische Universität München | 12 August Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Task  How can these images be classified into surgical phases?  Or if they can’t be classified well enough, which features can be added to the rest of the information?

CAMP | Department of Computer Science | Technische Universität München | 12 August Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Definition of a Feature  Feature as representation of discriminative information  Numerical vector calculated out of the image  Reduction of the dimension  Features  Basic: Gradient/Color/Motion Histogram  Integral image 1  Phase concruency 2 Gradient Histogram 1 Viola, Jones- Robust Real-time Object Detection 2 Kovesi - Image Features from Phase Congruency

CAMP | Department of Computer Science | Technische Universität München | 12 August Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Feature Space  With a limit to algorithms with an output smaller or equal than the input we get a number of injective function about: depending on the dimension of the input vector d and the number of different values c which are possible in the input.  Exploration of this space with Genetic Programming 1  A method to explore a high dimensional space  Fitness function influences the resulting features 1 Feature Extraction from Multiple Data Sources Using Genetic Programming - John J. Szymanski,* Steven P. Brumby, Paul Pope, Damian Eads, Diana Esch-Mosher,Mark Galassi, Neal R. Harvey, Hersey D.W. McCulloch, Simon J. Perkins, Reid Porter,James Theiler, A. Cody Young, Jeffrey J. Bloch and Nancy David ( c d ) c d ( c d ¡ 1 ) 2

CAMP | Department of Computer Science | Technische Universität München | 12 August Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Feature Generation by Genetic Programming 12 Mutation Evaluation Code 1 Wolfgang Banzhaf, Peter Nordin, Robert E. Keller, Frank D. Francone - Genetic Programming: An Introduction 2 Koza - Genetic Programming

CAMP | Department of Computer Science | Technische Universität München | 12 August Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Implementation Steps  Definition of a programming language (more)(more)  Designed for recombination and mutation  Designing a Framework to run programs  Translation of basic features in the new language  Evaluation of the fitness of programs  Recombination and Mutation

CAMP | Department of Computer Science | Technische Universität München | 12 August Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu An Example as Proof of Concept  Implementation of a color histogram in the new language  Test on an easy sample, 6 images(10*10*RGB, jpeg) in 3 classes:  Dimension of input here (cd):

CAMP | Department of Computer Science | Technische Universität München | 12 August Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Evaluation of the Example  The Fitness Function is the core of the system

CAMP | Department of Computer Science | Technische Universität München | 12 August Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Evaluation of the Example Comparison of several Programs P i P1P1 P1P1 P2P2 P2P2 …

CAMP | Department of Computer Science | Technische Universität München | 12 August Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Evaluation of the Example A random selection of the labeled input I k several from of each class P1P1 P1P1 P2P2 P2P2 … I1I1 I2I2 I3I3

CAMP | Department of Computer Science | Technische Universität München | 12 August Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Evaluation of the Example Execution of the program with every selected input I k As result there are output vectors: P1P1 P1P1 P2P2 P2P2 … I1I1 I2I2 I3I3 o 11 ; o 12 ¢¢¢ o 23

CAMP | Department of Computer Science | Technische Universität München | 12 August Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Evaluation of the Example Compare each pair of output vectors of one Program, by the distance P1P1 P1P1 P2P2 P2P2 … I1I1 I2I2 I3I3 d i k ; l = P l en m = 0 ( o i k [ m ] ¡ o i l [ m ]) 2 (j o i k [ m ]j + j o i l [ m ]j) 2 o 11 ; o 12 ¢¢¢ o 23

CAMP | Department of Computer Science | Technische Universität München | 12 August Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Evaluation of the Example Select the maximum of these distances, only considering pairs of different classes: P1P1 P1P1 P2P2 P2P2 … I1I1 I2I2 I3I3 o 11 ; o 12 ¢¢¢ o 23 d i k ; l = P l en m = 0 ( o i k [ m ] ¡ o i l [ m ]) 2 (j o i k [ m ]j + j o i l [ m ]j) 2 Now we have two classes k, l which our feature separates argmax kl ( d i kl j k : c l ass 6 = l : c l ass )

CAMP | Department of Computer Science | Technische Universität München | 12 August Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Evaluation of the example Subtract the distances in images of the classes k or l P1P1 P1P1 P2P2 P2P2 … I1I1 I2I2 I3I3 o 11 ; o 12 ¢¢¢ o 23 d i k ; l = P l en m = 0 ( o i k [ m ] ¡ o i l [ m ]) 2 (j o i k [ m ]j + j o i l [ m ]j) 2 f i = d i kl ¡ d i kk + d i ll 2 argmax kl ( d i kl j k : c l ass 6 = l : c l ass )

CAMP | Department of Computer Science | Technische Universität München | 12 August Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu 3.1 Results in this Example New Program

CAMP | Department of Computer Science | Technische Universität München | 12 August Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Fitness in the Example:  Input Program was calculating a Color Histogram (Output length 512 in ~2000 steps), resulting fitness: ~ 1.2 %  The new generated program calculated one number (in 30 steps) and got a fitness of: ~ 98.8 % P1P1 P1P1 P2P2 P2P2

CAMP | Department of Computer Science | Technische Universität München | 12 August Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Evolution Technique used in the Example  Selection of the two best Program and generation several children by these two methods:  One parental program: Mutation  Adding of random commands  Removing a single commands1  Duplicating an existing command  Switching commands  Two parental programs: Breeding  Mixing both, preferring blocks of a specific size  Adding random commands in the gaps between P1P1 P1P1 P 11 P1P1 P1P1 P2P2 P2P2 P12P12 P12P12

CAMP | Department of Computer Science | Technische Universität München | 12 August Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Conclusion  The concept seems to work  Adaptation to workflow project’s data can start

CAMP | Department of Computer Science | Technische Universität München | 12 August Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Steps to adapt it to the Workflow project  Adapt fitness function (more)(more)  Change Genetic rules (more)(more)  Optimize the framework  Generate C code out of the intermediate language

CAMP | Department of Computer Science | Technische Universität München | 12 August Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Any Questions?

CAMP | Department of Computer Science | Technische Universität München | 12 August Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Appendix

CAMP | Department of Computer Science | Technische Universität München | 12 August Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Planned Improvements in the Genetic rules  Probability to survive should depend on the fitness of all other existing programs  Even a program with a low fitness should have a small change to survive  Also the reproduction should not be restricted to the best program Back

CAMP | Department of Computer Science | Technische Universität München | 12 August Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Fitness  Actual fitness expresses:  Linear separation of two classes  Further improvements:  Static Analysis of Code 1  Semantic Checks  to prevent running in infinite loops  to prevent the execution of programs not using the input  …  Other Classifiers to evaluate the output 1 Colin G. Johnson-Deriving genetic programming fitness properties by static analysis Back

CAMP | Department of Computer Science | Technische Universität München | 12 August Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Specification of the language  Assembler style with some high level functions  1 Opcode and 3 Arguments for one command  High level examples:  Reading of the input and its dimensions  For loops  Adding of more functions possible Back

CAMP | Department of Computer Science | Technische Universität München | 12 August Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Scheme for the programming language OPCode1.Parameter2. Parameter3. Parameter 010 For counter to stack 0 From Constant 100 To Constant 65 End of the loop Constant 123 Read pixel Value to stack x From stack 20 y Constant … …

CAMP | Department of Computer Science | Technische Universität München | 12 August Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab.cs.tum.edu Generated Program (without nops) INPUTDIMENSION SAVE INPUTDIMENSION SAVE FOR LOAD LOAD SAVE LOAD POP READINPUT LOAD POP PUSH SAVE PUSH MULTIPLY LOAD PLUS PLUS PLUS JUMP DIVIDE PUSH LOAD LOAD RETURN JUMP FOR