1. Three-Dimension (3D) Whole-slide Histological Image Analytics
Yanhui Liang
Department of Biomedical Informatics
Department of Computer Science
The Pathology Informatics Summit 2016, May 23-26, 2016

2. Pathology Analytical Imaging
Glass Slides
Scanning
Whole Slide Images
Image Analysis
As we all know, digital pathology images are more and more adopted in research.
Provide rich information about morphological and functional characteristics of biological systems
Have tremendous potential for understanding diseases and supporting diagnosis
Introduction

3. Problems Digital Pathology Image analysis
From raw pixels to 2D and 3D meaningful structures
Data management and analytics
Explore global patterns and spatial relationships
In order to fully use these digital pathology images to help clinicians better understand diseases, we get two big problems. The first one is image analysis. From the pixels in images, we need to extract meaningful structures, both in 2D and 3D, for the following analysis.
After we get these structures or objects, we perform spatial analytics to explore the spatial relationships between the micro-anatomic objects and discover the global spatial patterns as they are essential to understand the disease progression. How the disease propagates.
In practice, we have these two issues for both 2D and 3D pathology imaging.
Introduction

4. 3D Vessel Branch and Nuclei
3D Pathology Imaging
3D pathology imaging enables stacking serial sections, which have significant potential to enhance the study of both normal and disease processes
Compared to 2D objects, provide more accurate shapes, structures, spatial relationships
Provide 3D volumetric models for 3D printing
3D reconstruction of cellular level objects is a critical step
significant potential to enhance the study of both normal and disease processes
from structural changes or spatial relationships
WSIs from serial sections have significant potential to enhance the study of both normal and disease processes
3D reconstruction of cellular level objects is a critical step
A Reconstructed
3D Vessel Branch and Nuclei
Human Liver
3D WSI Volume
Introduction

5. Challenges Explosion of derived data Image analysis Spatial analytics
10 billion pixels and 1 million objects per slice
Tens of millions 3D objects in a 3D volume with hundreds of slices
Image analysis
Multiple types, large-scale
Large variations in shapes, complex structures
Spatial analytics
High computational complexity
Multi-dimensional (3D)
Heavy duty geometric computations
However, in reality, to solve the above two problems are quite challenging. The first challenge is the data size. One WSI may have 10 billion pixels and contains 1 million objects. This is real “big data” we need to handle.
And the geometric computations on these 3D objects have pretty high time and space complexity
The second challenge is to develop a generic image analysis framework. As we have seen in previous slides, in each pathology image, there may have several different types of micro-anatomic objects and their shapes are irregular and totally different. So it’s difficult to develop an efficient segmentation method to extract the boundaries. Besides, when we perform object matching between adjacent slides for 3D reconstruction, due to the large variations in shape and complicated structures, traditional multi-objects tracking doesn’t work well for our case.
The last challenge we have is the high computational complexity in our spatial analytics of the extracted objects from pathology image. As we may extract hundrands of features for each extracted object, each object is stored as multi-dimension data. Also, when try to explore their spatial relationships or spatial patterns, the geometric computations are too expensive to be directly used in real applications. Thus we need scalable and cost-effective systems to perform the spatial analytics
Introduction

6. Our work
Quantitative analysis of whole slide images to derive spatial structures and features in 3D, and perform spatial analytics on derived 3D micro-anatomic objects such as blood vessels and nuclei
A framework for 3D primary blood vessel reconstruction
A system for 3D spatial analytics on 3D blood vessels and nuclei
Motivated by the above problems and challenges, in our paper, we present a ……
We are interested in 3D objects because compared to 2D, 3D present the actual strucctus of micro-annotomic object in biolgocial systems and keep the real spatial relationships.
In this paper, we mainly focus on spatial relationship based queries, including spatial joins: two-way and multi-way joins, window based queries and point-in-polygon queries. A special case of spatial join is polygon overlay or spatial cross-matching queries.
Spatial cross-matching is to identify and compare derived spatial objects belonging to different analyses or measurements. Such query could involve large number of spatial objects to compare and are highly expensive to process.
Introduction

7. 3D Primary Vessel Reconstruction
3D WSI Volume
Image Registration
Vessel Association
Image Segmentation
Vessel Interpolation
3D Vessel Rendering
3D Pathology Image Analysis

8. 3D Vessel Reconstruction
Pathology image volume
A stack of 2D sequential image slides
Image registration
Align all slices into the same coordinate system
Image segmentation
Vessel Directed Fitting Energy (VDFE) within a variational level set framework
Tissue slide, scan deformations
IHC, CD34 as blood vessel biomarker, x y 0.25 micrometers, z 50 micrometers 54 slides
Perform image registration to align all slide sections in the same coordinate system.
As mentioned above, we need to register all slides to the .. Rigid and non-rigid are used in this step. As we can see, after registraion, there are some translations, rotaions and non-rigid transformations in the registerd slide.
Prior information on vessel wall probability : Vessel directed fitting energe
3D Pathology Image Analysis

9. Vessel Segmentation Results
Final Result
Tissue slide, scan deformations
Perform image registration to align all slide sections in the same coordinate system.
As mentioned above, we need to register all slides to the .. Rigid and non-rigid are used in this step. As we can see, after registraion, there are some translations, rotaions and non-rigid transformations in the registerd slide.
3D Pathology Image Analysis

10. 3D Vessel Reconstruction
Vessel association
Four association cases
Two stages: local bi-slide mapping and global vessel structure association
Extension
Bifurcation
Emergence
Disappearance
Tissue slide, scan deformations
Perform image registration to align all slide sections in the same coordinate system.
As mentioned above, we need to register all slides to the .. Rigid and non-rigid are used in this step. As we can see, after registraion, there are some translations, rotaions and non-rigid transformations in the registerd slide.
Prior information on vessel wall probability : Vessel directed fitting energe
3D Pathology Image Analysis

11. 3D Vessel Reconstruction
Vessel interpolation
Make transition more smooth
3D vessel rendering
Extract the iso-surface of the volumes from 2D vessel objects
Original slice t
Original slice t+1
Interpolated slices
Tissue slide, scan deformations
Perform image registration to align all slide sections in the same coordinate system.
As mentioned above, we need to register all slides to the .. Rigid and non-rigid are used in this step. As we can see, after registraion, there are some translations, rotaions and non-rigid transformations in the registerd slide.
Prior information on vessel wall probability : Vessel directed fitting energe
3D Pathology Image Analysis

12. 3D Vessel Rendering 1 2
3D Pathology Image Analysis

13. Database for Spatial Queries and Analytics
To store, manage and analyze the derived 3D cellular objects such as blood vessels and nuclei
For each 3D nucleus, return the distance to its nearest blood vessel
Scalable, efficient and cost-effective system for 3D spatial objects analytics based on Hadoop
3D Data partitioning for parallel processing
Indexing to speed up spatial access
3D query engine to support various spatial queries
3D spatial cross-matching
3D k-Nearest Neighbor (kNN) query
The first query type is spatial cross-matching. It is used for 3D segmentation algorithm validation or evaluation. Given two segmentation results of 3D nuclei, we first extract their minimum bounding box (MBBs) and the whole space info. We then apply data partitioning and obtain the 3D objects from two datasets but belonging the same cuboid. To speed up spatial query, we build local R*-tree index on the second dataset, and use a filter –and – refine approach to perform spatial cross-matching. We first use the MBBs of objects in dataset1 for intersection checking. If their MBBs intersect, we then check if their geometries are intersected or not. This is the geometry refinement step. If required, spatial measurement such as the intersection volume is calculated.
3D NN query has broad applications in various domains. For 3D analytical pathology imaging, pathologists are interested in spatial queries such as “for each 3D cell, return its nearest 3D blood vessel and
the distance”. In this case, we can perform 3D NN query. We propose both voronoi diagram based and R*-tree based methods for NN query, and each of them achieves high performance.
3D Spatial Analytics

14. 3D Spatial Queries and Analytics
3D Input Data
Data Storage on HDFS
3D Data
Partitioning
After we extract the structure of 3D micro-anatomic objects, we perform 3D spatial analytics to analyze the structural relationship between them. Here we mainly focus on 3D blood vessels and 3D nuclei. After we reconstruct blood vessel structures, we simulate its surrounding nuclei for spatial analysis. Here is the architecture of our system. Our system is built on Hadoop. Given the 3D data as input, we store them on HDFS for efficient access. As the input 3D space is huge, we perform data partitioning to partition the space into small cuboids. Then each cuboid is taken as a processing unit for spatial queries on MapReduce. Our spatial query engine supports 3D spatial join and nearest neighbor query. It has two indexing strategies: global indexing for cuboids and local indexing for objects within cuboids. Our system is scalable and cost-effective, and can support 3D spatial queries efficiently.
3D Global Spatial Indexing
3D ODSQUE
3D Spatial Index Builder
3D Spatial Query Processing
3D Boundary Object Handling
3D Spatial
Query Engine
3D Cuboid Spatial Indexing
Hadoop
3D Spatial Analytics

15. 3D Summary 3D pathology image analysis
3D data management and spatial analytics
The first query type is spatial cross-matching. It is used for 3D segmentation algorithm validation or evaluation. Given two segmentation results of 3D nuclei, we first extract their minimum bounding box (MBBs) and the whole space info. We then apply data partitioning and obtain the 3D objects from two datasets but belonging the same cuboid. To speed up spatial query, we build local R*-tree index on the second dataset, and use a filter –and – refine approach to perform spatial cross-matching. We first use the MBBs of objects in dataset1 for intersection checking. If their MBBs intersect, we then check if their geometries are intersected or not. This is the geometry refinement step. If required, spatial measurement such as the intersection volume is calculated.
3D NN query has broad applications in various domains. For 3D analytical pathology imaging, pathologists are interested in spatial queries such as “for each 3D cell, return its nearest 3D blood vessel and
the distance”. In this case, we can perform 3D NN query. We propose both voronoi diagram based and R*-tree based methods for NN query, and each of them achieves high performance. 3D
3D Spatial Analytics

16. Questions?