1. Integrating the Healthcare Enterprise
Image Fusion Integration Profile
Todd Kantchev
Siemens Molecular Imaging
IHE TC and Nuclear Medicine Meeting RSNA

2. IHE Nuclear Medicine – RSNA 2005
Terminology
Image Registration- A process of changing the pixel spatial coordinates of a TDS so that the patient’s anatomical features are aligned with those of the SDS
Rigid Registration- as defined in SRO
Deformable Registration- out of scope
Image Fusion- The process of superimposing (overlaying) two DS so that they can be seen as one combined image with the intention to complement features, present in both.
DS- Data Set- a series of images or set of frames with a specified image position and orientation in the patient’s space, from which a 3-D image volume can be built and displayed for some clinical purpose. They may or may not have same orientation or be uniformly spaced in x-y-z. The requirements to build a volume from the DS are application specific and are outside the scope of this profile.
SDS- Source data Set- a DS which is used as a base to which a TDS can be registered.
TDS- Target Data Set- a DS, which is spatially transformed to the SDS by the process of Image Registration
SRO- Spatial Registration Object- as defined in DICOM PS 3 Section A.39
BSPS- Blending Softcopy Presentation State as defined in DICOM Supplement 100.
IHE Nuclear Medicine – RSNA 2005

3. Fusion Brain CT Brain MR
Registration of medical images usually deals with two or more images or image sets, taken from the same patient anatomical area, for example brain CT and MR
The goal is to have them aligned, superimposed and blended, so that we could see clinical information from both complementing modalities: CT (bone) and MR (soft tissue).
The simplistic approach of aligning, without taking care for the imaging geometry does not work.

4. 3-D Spatial RegistrationX Y Z X Y Z
Source Data Set (SDS)
Two Patient-based Coordinate Systems, defined by two Frames of Reference
Target Data Set (TDS)
Image Orientation (Patient)- x
Image Position (Patient)
Image Orientation (Patient)- y
CT and MR series of images have been taken In 3D space on two acquisition devices at different time, hence their geometry in terms of image position/orientation (x,y) is encoded in DICOM in two different coordinate systems, identified by two Frames of Reference UIDs.
Both coordinate systems have their origin in different points of the patient body.
The goal of registration is to align the patient anatomical region of the Target Data Set (TDS)- chosen for some reason, with the same anatomical region of the Source Data Set (SDS), through translation, rotation, and scaling, to the best match.
This process changes the coordinates of the original TDS, which is recorded as a transformation
Once calculated and stored the transformation can be applied at any time.
Align the patient anatomic region of the Target Data Set (TDS) to match the same anatomical region in the Source Data Set (SDS).

5. Registration=>Fusion (CT-MR)
Translate
Rotate Rigid Registration
Scale
Warp- optional (Deformable Registration)- not in scope
Fuse (display superimposed)
Translation, Rotation, Scaling (or generally affine) operations encompass Rigid Registration.
The result can be complemented with an additional step of “warping” to match local misalignments. This is called Deformable Registration
We use the term Fusion for the combined process of registration followed by superimposing and blending of the image sets for display.

6. IHE Nuclear Medicine – RSNA 2005
Normal Workflow
On Evidence Creator
Load Source Data Set (SDS) and Target Data Set (TDS).
Find best anatomical matching of TDS to the SDS (usually pixel intensity statistical methods are used or manual). The methods are out of scope!
Compute Transformation for the TDS (translate, rotate, scale) and apply
Re-sample the TDS (take new axial slices in the new image orientation). The resolution usually is to the x-y-z resolution of the SDS. Fuse and display.
Store Transformation
On Image Display (possibly via IM)
Load original SDS and TDS
Apply the Transformation to the TDS
Fuse and display
Export Transformation Matrix:
Pros:
No need to store and communicate DTDS- save storage space and bandwidth. (Transformation Matrix (Rigid) is small compared to DTDS)
Cons.
Workstations which do not have capability to apply registration and re-sample image data cannot be used in the workflow. Ex. Basic Review WS
Applying the registration is an additional step after loading the image in the display pipeline- extended time for loading
Export Derived TDS:
The receiving application may be inexpensive 3-D viewing workstation. It may not even have a 3-D viewing capabilities.
Quick DTDS loading (quicker than loading OTDS and re-applying the registration)
Cons:
Re-sampled data may be too big (ex. PET res. to CT res.)
IHE Nuclear Medicine – RSNA 2005

7. Use Case- Brain CT-MR Store-Retrieve-Apply Transformation
Storage
Post Processing Workstation or CAD
Review Workstation
Store CT
Store MR
Retrieve CT, MR
Register MR to CT get T
Fuse CT and MRR get BSPS
Store T
An example workflow for the first method is shown on this diagram.
The workflow is managed by a centralized archive (PACS) and the use of DICOM Query Retrieve Service.
The CT and the MR image data sets are stored on PACS after they are acquired and quality assured by a technologist.
A physician on the Post-processing Workstation (PPWS) uses the Q/R mechanism to access the data at any convenient time later (assuming no electronic work list is in place).
The data sets are registered and fused on the PPWS and the physician is able to perform a diagnosis. Then the Transformation Matrix object and the Blending Softcopy Presentation State are stored on the archive system for future use.
A clinician uses a high-end Review Workstation to retrieve the CT, MR, the transformation matrix and the presentation state, apply the transformation and perform the image fusion. The clinician is then able to review the case.
Store BSPS
Retrieve CT, MR, T, BSPS
Apply T to MR
Fuse CT and MR

8. IHE Nuclear Medicine – RSNA 2005
Alternative Workflow
On Evidence Creator
Load Source Data Set (SDS) and Target Data Set (TDS)
Find best anatomical matching of TDS to the SDS (usually pixel intensity statistical methods are used or manual). The methods are out of scope!
Compute Transformation for the TDS (translate, rotate, scale) and apply
Re-sample the TDS (take new axial slices in the new image orientation). The resolution usually is to the x-y-z resolution of the SDS. Fuse and display.
Store the re-sampled TDS (resolution may differ)
On Image Display (possibly via IM)
Load original SDS and the re-sampled TDS
Apply the Transformation to the TDS
Fuse and display
Export Transformation Matrix:
Pros:
No need to store and communicate DTDS- save storage space and bandwidth. (Transformation Matrix (Rigid) is small compared to DTDS)
Cons.
Workstations which do not have capability to apply registration and re-sample image data cannot be used in the workflow. Ex. Basic Review WS
Applying the registration is an additional step after loading the image in the display pipeline- extended time for loading
Export Derived TDS:
The receiving application may be inexpensive 3-D viewing workstation. It may not even have a 3-D viewing capabilities.
Quick DTDS loading (quicker than loading OTDS and re-applying the registration)
Cons:
Re-sampled data may be too big (ex. PET res. to CT res.)
IHE Nuclear Medicine – RSNA 2005

9. Use Case- Brain CT-MR Store-Retrieve- Display Transformed TDS
Storage
Review Workstation
Store CT
Post Processing Workstation or CAD
Store MR
Retrieve CT, MR
Register MR to CT get MRR
Store MRR
Fuse CT and MRR get BSPS
Store BSPS
An example workflow for the second method is shown on this diagram.
The workflow is managed by a centralized archive (PACS) and the use of DICOM Query Retrieve Service.
The CT and the MR image data sets are stored on PACS after they are acquired and quality assured by a technologist.
A physician on the Post-processing Workstation (PPWS) uses the Q/R mechanism to access the data at any convenient time later (assuming no electronic work list is in place).
The data sets are registered and fused on the PPWS and the physician is able to perform a diagnosis. Then the Transformed image data set (MRR) and the Blending Softcopy Presentation State are stored on the archive system for future use.
A clinician uses a low-end Review Workstation to retrieve the CT, MRR, and the presentation state and perform the image fusion. The clinician is then able to review the case.
Retrieve CT and MRR
Retrieve BSPS
Fuse CT
and MRR

10. Pros and Cons Export Transformation Matrix: Pros:
Save storage space and bandwidth (SRO is small compared with TDS)
Cons.
More expensive (Image Display higher cost)
Slower
Export Derived TDS:
Less expensive (Image Display lower cost)
Quicker
Cons:
Re-sampled data may be too big (ex. PET res. to CT res.)
Export Transformation Matrix:
Pros:
No need to store and communicate DTDS- save storage space and bandwidth. (Transformation Matrix (Rigid) is small compared to DTDS)
Cons.
Workstations which do not have capability to apply registration and re-sample image data cannot be used in the workflow. Ex. Basic Review WS
Applying the registration is an additional step after loading the image in the display pipeline- extended time for loading
Export Derived TDS:
The receiving application may be inexpensive 3-D viewing workstation. It may not even have a 3-D viewing capabilities.
Quick DTDS loading (quicker than loading OTDS and re-applying the registration)
Cons:
Re-sampled data may be too big (ex. PET res. to CT res.)

11. Blending Presentation State SOP Class
DICOM Standard provides a solution for storage of presentation of fused data (Supplement 100)
The SDS would be used as an underlying image data set
The TDS would be used as superimposed data set
The MLUT and VOI LUT transformations are first applied as normal to both data sets
The stored pseudo color will then be applied to the superimposed image data set.
Both data sets will then be blended with the stored blending factor.
The PCS color transformation would normally be Identity in this case (bypassed).
DICOM Standard, Supplement 100

12. IHE Nuclear Medicine – RSNA 2005
Slide provided by David Clunie
Blending for CT-PET
select underlying
select superimposed
IHE Nuclear Medicine – RSNA 2005

13. IHE Nuclear Medicine – RSNA 2005
Slide provided by David Clunie
Blending for CT-PET
select underlying
[register]
select superimposed
IHE Nuclear Medicine – RSNA 2005

14. IHE Nuclear Medicine – RSNA 2005
Slide provided by David Clunie
Blending for CT-PET
select underlying
[register]
select superimposed
resample
IHE Nuclear Medicine – RSNA 2005

15. IHE Nuclear Medicine – RSNA 2005
Slide provided by David Clunie
Blending for CT-PET
select underlying
[register]
within slices
select superimposed
resample
IHE Nuclear Medicine – RSNA 2005

16. IHE Nuclear Medicine – RSNA 2005
Slide provided by David Clunie
Blending for CT-PET
select underlying
[register]
within slices
select superimposed
resample
[between slices]
IHE Nuclear Medicine – RSNA 2005

17. IHE Nuclear Medicine – RSNA 2005
Slide provided by David Clunie
Blending for CT-PET
select underlying
rescale and window
[register]
within slices
select superimposed
resample
[between slices]
IHE Nuclear Medicine – RSNA 2005

18. IHE Nuclear Medicine – RSNA 2005
Slide provided by David Clunie
Blending for CT-PET
select underlying
rescale and window
[register]
within slices
select superimposed
resample
pseudo-color
[between slices]
IHE Nuclear Medicine – RSNA 2005

19. IHE Nuclear Medicine – RSNA 2005
Slide provided by David Clunie
Blending for CT-PET
select underlying
rescale and window
[register]
blend
within slices
select superimposed
resample
pseudo-color
[between slices]
IHE Nuclear Medicine – RSNA 2005

20. Blending Presentation overlay transparency (blending factor)
20%
50%
80%
50%
50%
100%
50%
IHE Nuclear Medicine – RSNA 2005

21. Interoperability for Image Registration
We want to display fused data, generated from an acquisition device or workstation on another vendor’s system.
Two possible solutions:
Export the alignment solution (transformation matrix) and reproduce the transformation on the receiving workstation. (Supplement 73, Spatial Registration Storage SOP Classes - in the Standard since 2003)
Export the transformed TDS and display it on the receiving workstation.
Both methods are complemented by the use of Blending Softcopy Presentation State (BSPS) (Supplement 100: Color Softcopy Presentation State Storage SOP Classes- in the 2005 Standard)
We want to display fused data, generated from one workstation on another vendor’s system.
Two possible methods:
Export the alignment solution (transformation matrix) and reproduce the transformation on the receiving workstation. (Supplement 73, Spatial Registration Storage SOP Classes - in the Standard since 2003)
Export the transformed/re-sampled (derived) TDS and display it on the receiving workstation.
Both methods are complemented by the use of Blending Softcopy Presentation State (BSPS) (Supplement 100: Color Softcopy Presentation State Storage SOP Classes- in the 2005 Standard)

22. The Problem
DICOM Spatial Registration object enables a very broad context and does not guarantee consistency in terms of data referencing, workflow transactions etc.
Blending Softcopy Presentation state object can be used to reference image series for fusion, but referencing can sometimes be ambiguous and vendors may choose not to support it or interpret it differently.
Existing DICOM objects need to be constrained and further clarified in order to avoid misinterpretation and ambiguity. Require optional attributes to be present.
Display capabilities such as multi-planar reformatting, color blending etc. which are essential to reading many types of fused studies, are either not supported by some display devices or are not consistently applied to achieve interoperability
Sending fused data from a workstation to another system for further processing, display, and interpretation is currently loosely defined in standards to achieve the necessary multi-vendor inter-operability.
Vendors are not accurately representing their support for such features

23. The Solution New Profile
No New Actors or Transactions. Additional requirements on existing actors and transactions
Clarifications of Fusion Workflow mapping to SWF
Constraints on Fusion related Data
Requirements/Recommendations on Fusion Clinical Display concepts, facilitating interoperability
Connectathon to help test and confirm

24. Image Fusion Integration Profile
Charge Posting
Scheduled Workflow -
Presentation of Grouped Procedures
Reporting Workflow
Patient Info. Recon-ciliation
Post-Processing Workflow
Image Fusion
Image Fusion
Consistent Present- ation of Images
Evidence Docs
Key Image Notes
Simple Image & Numeric Reports
Access to Radiology Information
Portable Data for Imaging
Basic Security

25. Affected Actors and Transactions
1: Patient Registration
ADT
1: Patient Registration
12: Patient Update
12: Patient Update
2: Placer Order Management
Order Filler
Order Placer
3: Filler Order Management
6: Modality PPS in Progress
4: Procedure Scheduled
7: Modality PPS Completed
11: Image Availability Query
20: Creator PPS in Progress
12: Patient Update
21: Creator PPS Completed
13: Procedure Update
C-FIND
Evidence Creator
Image Display
Storage Commitment
20: Creator PPS in Progress
21: Creator PPS Completed
14: Query Images
10: Storage
Commitment
18: Creator Image Stored
Performed
Procedure Step
Manager
16: Retrieve Images
C-STORE
C-MOVE
Image
Manager
Image
Archive
6: Modality PPS in Progress
7: Modality PPS Completed
20: Creator PPS in Progress
10: Storage
Commitment
21: Creator PPS Completed
8: Modality Image Stored
Storage Commitment
6: Modality PPS in Progress
C-STORE
7: Modality PPS Completed
Acquisition Modality
5: Modality Worklist Provided

26. Acquisition Modality (Modality Images Stored [8])
Shall create image objects as specified (additional attributes mandatory, same Frame of Reference for hybrid modalities, etc.)
Fusion related objects- SRO, SBPS, shall be consistent (referencing, blending etc.)
Registration object shall be stored. The SRO shall be in the same Study as either SRS or TDS. If no additional alignment will be required before presentation, the SRO shall encode IDENTITY transformation.
Blending PS object shall be required if the use of specific blending factor and color are essential to the clinical workflow and interpretation. The BSPS object shall be in the same Study as either SRS or TDS.

27. Evidence Creators- Required Creator Images Stored [18])
Shall be able to co-register and fuse data sets with images from multiple modalities (PET, CT, MR, NM in different combinations, including same modalities).
The user shall be able to specify one of the input datasets to be the primary, so that registration uses that volume as the source (typically CT or MR).
The user shall be able to specify which of the target volumes (typically PET or MR) to register to the source.
Registration object shall be stored. The SRO shall be in the same Study as either SRS or TDS. If the transformed TDS is also stored and no additional alignment will be required before presentation, the SRO shall encode IDENTITY transformation.
Storage Commitment- not required

28. Evidence Creators- Recommended Creator Images Stored [18])
Transformed TDS, if stored, may or may not be re-sampled to the same pixel and slice spacing, as SDS.
The transformed target data set, if exported, shall be in the same frame of reference as the source data set and all other attributes shall be consistent with the new pixel data.
Blending PS object shall be required if the use of specific blending factor and pseudo color are essential to the clinical workflow and interpretation. The BSPS object shall be in the same Study as either SRS or TDS.

29. Image Display- Required (Query Images [14], Retrieve Images [16])
Shall be able to:
Fuse (superimpose) and display at least two previously registered data sets (for example a structural CT and a registered PET scan re-sampled to the same pixel and slice spacing). If Spatial Registration object for the pair is available it shall be used.
Note: The ID may load and display data without an SRO present. However the fact that both DS are in the same Frame of Reference is not enough to guarantee that they can be fused.
Display the fused data either color blended, or as a checker board (grayscale squares of the “checkerboard” are one image set, and the pseudo color squares are the other image set. ). If Blending SP object for the pair is available, it shall be used for initial presentation.
Enable fusion-specific user interaction like transparency of fusion overlay (blending factor); color map selection; Window Width/Level control.
Display MPR (Multi Planar Reconstruction) so that coronal and sagittal planes could be displayed from available transaxial data
Display specific values like Derivation Code Sequence (0008,9215) or Derivation Description (0008, 2111), Series Description (0008,103E)

30. Image Display- Recommended (Query Images [14], Retrieve Images [16])
Should be able to:
Read a SRO and apply transformation, fuse and display two data sets, which may have not been previously re-sampled to the same pixel and slice spacing. The pixel intensity interpolation method associated with this will not be specified, but may be recommended.

31. Image Manager/Archive (Modality Images Stored [8], Creator Images Stored [18] Query Images [14], Retrieve Images [16])
Shall be able to:
Store necessary objects like PET, CT, MR, NM, Spatial Registration, Pseudo Color and Blending SP State, Real World Value Mapping etc.
Support query and retrieval of necessary objects at series level with required specific keys

32. General Framework (from discussions so far)
It will be a general Image Fusion profile (applicable to CT, MR, PET and NM), rather than be specific for PET-CT.
SWF may not be required
Fusion profile is not intended at this stage to be all-encompassing. For example, a reasonable limitation would be that it would deal with paired 3D data sets (no 2D, image stitching etc. ). No elaboration will be made at this stage on multidimensional presentation. Only generic features like 3D fusion/blending.
Explain how SR and BSPS object reference TDS and SDS etc.

33. Open Issues/ Questions
Do we need specific matching and return keys to be specified for query transactions?
Shall we consider MPPS support for EC (Transactions 20,21 for Implicit Post-Processing- see Vol )?
IHE Nuclear Medicine – RSNA 2005

34. IHE Nuclear Medicine – RSNA 2005
Table 1- Attributes
Name
Tag
Acquisition Modality
Evidence Creator
Comment
Related Series Sequence
(0008,1250)
May be R+ for hybrid PET-CT
If so, may help ID to link SDS and TDS in case it cannot read SRO
May extend CID 7210, Part 16. Mind NM which is MF
Series Description
(0008,103E) ??
yes
Return key for ID/IM
Derivation Code Sequence or
Derivation Description
(0008,9215)
(0008, 2111) no
Yes
(if re-sampled)
Matching key for ID/IM???
Request Attribute Sequence
(0040,0275)
Yes (if SWF supported)
yes (if SWF)
Already in SWF
IHE Nuclear Medicine – RSNA 2005

35. IHE Nuclear Medicine – RSNA 2005
Expected Timeline
Public comment preparation: December- January (TCON for Jan 6)
Draft approved on a face to face meeting: Jan 30-Feb
Released for public comment: Feb
Public Comment closed : March
Face to Face Meeting: April , Chicago
Publish for Trial Implementation: April
Connectathon: End of 2006-June 2007 (possibly SNM)
IHE Nuclear Medicine – RSNA 2005