Automatic in vivo Microscopy Video Mining for Leukocytes * Chengcui Zhang, Wei-Bang Chen, Lin Yang, Xin Chen, John K. Johnstone

Background Information What is in vivo microscopy? Images of the cellular and molecular processes in a living organism Why video-mine leukocytes? To Predict Inflammatory response Rolling velocity and magnitude of adhesion of leukocytes are the main predictors Currently analyzed manually Time consuming / Expensive Subjective

Objectives Given a sequence of in vivo images, Track the moving leukocytes Calculate their average velocity Find the magnitude of adherent leukocytes

Challenges Server Noise Background movement Due to movement of the living organism Deformation of leukocytes Change of contrast in different frames

Previous Work [Eden et al.] use local features (e.g. color) for a tracking system Assume that leukocytes roll along the vessel centerline [Acton et al.] Background removal + morphological filter Assumes the shape/size leukocytes does not change

Suggested Approach Three main steps: 1. Frame Alignment To correct the camera/subject movement 2. Detect Moving Leukocytes 3. Detect Adherent Leukocytes After moving leukocytes are removed

Step 1- Frame Alignment 1.1- Detect Camera/Subject Movement Define a (dis)similarity measure between consecutive frames This allows for some tolerance within radius r If S(f t-1, f t ) is larger than a threshold, then f t requires frame alignment

1.2- Frame Matching Generate a number of high dimensional, local scale-invariant features [SIFT] for the frame and its predecessor Use nearest-neighbor to find a match for each feature point Calculate the transformation matrix H, such that Step 1- Frame Alignment For every matched point x and x’

Step 1- Frame Alignment Use Random Sample Consensus (RANSAC) to correct the mismatches

Step 2 - Detecting Moving Leukocytes Approach 1 - Probabilistic Learning For pixel j in the image, let x 1j, x 2j,..., x Nj be the intensity of the pixel over N frames Assume that P(x tj ) has a normal distribution over time with mean x tj If P(x tj ) is smaller than a threshold, then it is a foreground pixel Problem: Difficult to find a threshold

Step 2 - Detecting Moving Leukocytes Approach 1 - Probabilistic Learning Problem: Difficult to find the threshold value Solution: Use One-Class SVM to classify background and foreground pixels

Step 2 - Detecting Moving Leukocytes Approach 2 - Neural Network Train a neural net to learn the predictable pattern of the background pixels Input: [x(t-m), x(t-m+1),..., x(t-1)] A sliding window of the intensity sequence Output: x(t) Prediction for the intensity of the pixel at the next frame If the neural-net prediction and the real pixel intensity are very different, the pixel in the current frame is in foreground

Step 2 - Detecting Moving Leukocytes Approach 2 - Neural Network

Step 2 - Detecting Moving Leukocytes Calculating the leukocytes velocity Find the centroid of each group of connected foreground pixels For each centroid, find the closest centroid in the previous frame If their distance is smaller than a threshold, they are a match Compute the mean velocity

Step 3- Detecting Adherent Leukocytes First, remove the moving leukocytes Three main types of regions left Tissues Vessels Adherent Leukocytes These three have different intensity values

Step 3- Detecting Adherent Leukocytes

Finding the threshold values Fit an 8 th degree polynomial to the histogram curve The real part of the second largest root is the ideal threshold Justification? Problem with false positives and false negatives

Experimental Results Test video of 148 frames Detecting moving leukocytes: 1% false positive for probabilistic learning(?) 49% false positive for neural-net approach 50% recall Detecting Adherent leukocytes 2% false positive 95% recall

Final Remarks Paper is mainly related to Vision The algorithms require many “magic parameters” that need hand tuning Would the current parameters work as well for a new video sequence from a new equipment? Do we want to pursue more video-mining papers?