1. Image processing in the compressed domain Assist.Eng. Camelia Florea, Technical University of Cluj-Napoca, ROMANIA SSIP 2009

2. Contents Brief Overview JPEG standard coding The two ways to process JPEG compressed images DCT Coefficients JPEG features space for image segmentation A DCT-based approach for detecting patients‘ identification information Bayesian segmentation of hepatic biopsy color images in the JPEG compressed domain Image enhancement operations in the compressed domain Compressed domain implementation of fuzzy rule-based contrast enhancement

3. Brief Overview Compressed domain image processing algorithms in encoded JPEG image domain - provide a powerful computational alternative to classical. This field is in its beginning - the algorithms reported in the literature are mostly based on linear arithmetic point operations (addition, substraction, multiplication). Advantage: no need to decompress/ recompress the whole image prior to processing/after processing. we compute less data (after quantization many of the DCT coefficients are zero).

4. JPEG standard coding The color space of the image is converted (RGB to YUV) The image is divided into 8x8 blocks Values are scaled symmetrical towards 0 (from [0, 255] to [-128, 127]) Each 8x8 block is processed for compression DCT is applied on each block => obtain the DCT coefficients (DC and AC) DCT coefficients are quantized - small coefficients are quantized to zero zig-zag scan of the DCT blocks RLE (Run Length Encoding) is performed finally - entropy coding

5. The image in the JPEG compressed domain

6. The two ways to process JPEG compressed images Having a JPEG compressed image is more efficiently to process it, without: performing decompression, pixel level processing, and recompression. Processing in the compressed domain is made over the RLE vectors. RLE vector contains data about the variation and mean of luminance/color

7. DCT Coefficients There are many local texture features embedded in the DCT coefficients, reflecting color and texture: the DC coefficient - the average color in a block of pixels, the AC coefficients - the variance of luminance and chrominance Two dimensional DCT basis functions (N = 8).

9. JPEG features space for image segmentation Image segmentation is the technique of partitioning an image into units which are homogeneous with respect to one or more characteristics. This can be done on pixel level for highest accuracy, but also on JPEG block level, in the compressed domain, considering the local color and texture information roughly needed, is completely present in this representation.

10. A DCT-based approach for detecting patients‘ identification information Addressed problem: implementing an algorithm for detecting patient’s data from JPEG ultrasound images, applied directly on DCT coefficients. Advantage: no need to convert medical images/video back to the spatial (uncompressed) domain. the algorithm can detect textual information using the amount of energy, computed using only AC coefficients. HIPAA recommends to healthcare providers: the protection of the confidentiality of their patients’ health data. Medical information, regarding both: patients’ identification and their treatment, can be: transmitted and stored and, are susceptible of being accessed by unauthorized people. Images contains textual data about the patient, data that requires special security measures when disseminating the images. They must be handled by authorized health care professionals only.

11. It is possible to hide or eliminate the patient information without processing the image content itself (pixels) – by using only the DCT coefficients. The DCT - is one of the best filters for feature extraction in the frequency domain – it could be used here. In an ultrasound medical image the areas with very high energy amount – are the regions containing textual information. Areas with patient’s data can then be encrypted, blurred or eliminated. Having a basic knowledge of the ultrasound machine used, + With the same image acquisition conditions, => we can detects (and hide) only the patient identification information in the image and keep the medical information. A DCT-based approach for detecting patients’ identification information

12. System’s architecture

13. Analyzing the grey level variation The block’s energy is computed and analyzed – for each 8×8 block. (E AC - the average of AC coeff. Energy) In ultrasound images are many 8×8 blocks where the local variation of the brightness is small: the background area, and the examination area from ultrasound image. => every such block - do not contain text information and, under no circumstance, information about patients. If the 8×8 blocks exhibit a large variation of the grey levels around the average brightness value, => we have areas with sudden changes of brightness from black to white: text, or cartesian axes, from the ultrasound image.

14. The data hiding algorithm in JPEG compressed domain 1.Compute the E AC. 2.If E AC data from the original image are kept (no processing). 3.If E AC ≥ e thd => the block has a significant content of details,  areas of interest - need to be processed for data selection: -patient’s identification information - will be protected, -examination data - no processing. -where e thd represents the optimal selection threshold between: -the uniform blocks, and -the blocks with a significant number of details.

15. High energy blocks Energy blocks Selected data to hide. Medical image resulted just by keeping only the blocks with high energy. High energy blocks Apply selection rules Post Processing

16. Bayesian segmentation of hepatic biopsy color images in the JPEG compressed domain Addressed problem color image segmentation based on: the color information, and, also on the local texture information, for each 8×8 pixel neighborhoods. Advantage: no need to decompress/recompress the whole image prior to processing/after processing. we compute less data (after quantization many of the DCT coefficients are zero). This reduced dimensionally feature space makes easier the training and implementation of rather complex classifiers, as e.g. the Bayesian classifier with class probabilities modeled by Gaussian mixtures used here.

17. Color in RGB vs. YUV space Many image representation spaces can be used in segmentation process. The YUV representation yields certain advantages over RGB: The YUV is the representation used in the JPEG standard, The YUV provides a clear separation between the luminance representation (Y) and color (U,V), The luminance information Y, the color information U and V exhibits poor correlation The image storage format itself provides the information needed for an accurate identification/segmentation e.g.: segmentation of the hepatic biopsies into tissue vs. microscopic slide and further, of the tissue into healthy tissue vs. hepatic fibrosis.

18. Bayesian classification of pixel block in the discrete cosine transform domain We use the Bayes decision rule to classify a DCT block into microscopic slide, healthy tissue or fibrosis (features space = zig-zag scanned quantized DCT coefficients). A powerful yet simple model for blocks classification is the Multivariate Gaussian model: where: The Bayes decision rules for minimal cost are the following: Typically in the hepatic biopsy there is no obvious reason to assume uneven distribution of the tissue vs. microscopic slide, and neither of fibrosis vs. healthy tissue, therefore, we consider:

19. Gaussian parameters estimation For the classification we need a-priori knowledge of the class statistics: If is square and singular, then its inverse does not exist. → This might be the case when the matrix is sparse, as is the case when using DCT quantized coefficients. In these cases, the computation is based on computing singular value decomposition of (the base of Moore-Penrose pseudo-inverse). Any singular values less than a threshold-value are treated as zero (>0.01). The determinant of matrix is the product of the diagonal singular values.

20. Bayesian segmentation of hepatic biopsy color images in the JPEG compressed domain The training phase of the algorithm The statistical properties of the classes used by the two classifiers are determined using ground-truth images: As a result the mean values and the covariance matrices (as well as their pseudo- inverse) are found for each class. The test phase of the algorithm Each and every 8×8 block from the microscopic compressed image is considered and the blocks are processed for classification. The segmentation of an image is performed as a 2-step classification process: 1 st step - Discrimination between microscopic slide and hepatic tissue 2 nd step - Identify the blocks that exhibit fibrosis among the hepatic tissue blocks

21. Discrimination between microscopic slide and hepatic tissue (1 st step) 1 st step is the discrimination between microscopic slide and hepatic tissue (with or without fibrosis). In this case the luminance information gives sufficient information for segmentation, and the decision rule is: with Y dct [8×8] – the matrix of the DCT coefficients

22. Identify the blocks that exhibit fibrosis among the hepatic tissue blocks (2 nd step) The hepatic biopsies are treated with Sirius stain → the coloration of hepatic fibrosis appears reddish – unlike the healthy tissue, of beige color. This 2-class Bayesian classification is performed at block level, but this time, the Y and V components are used to compute the two class probabilities U is not used since it is a measure of the dominance of blue Color components and luminance information are not correlated → the joint class probabilities are the product of the luminance and color probabilities: where: Y dct - the 8×8 block of the luminance DCT coefficients; V dct - the 8×8 block of the reddish chrominance DCT coefficients.

23. Experimental results Patient DCT algorithm [%] Pixel level algorithm [%] Scores of fibrosis P14.154.471 P25.327.121 P37.056.231 P410.711.22 P514.9514.62 P616.7120.53 P717.84213 Classifications results using our algorithm and pixel level algorithm:

24. Experimental results 1 st classifier2 nd classifier FARFRRFARFRR Average0.750.992.141.81 Worse case FAR1.410.983.721.95 Worse case FRR0.821.931.743.59 False acceptance rate (FAR) and false rejection rate (FRR), for one patient:

25. Image enhancement operations in the compressed domain Mostly linear algorithms developed for the compressed domain: Pointwise image addition/substraction Constant addition/substraction to each spatial position Constant multiplication to each spatial position Pointwise image multiplication Pixel arithmetic can be used to implement a number of operations cross-fade between two images or video sequences image composition - overlaying a forecaster on a weather map implementation of fuzzy rule-based contrast enhancement

26. Alpha-blending between two images

27. Image composition

28. Compressed domain implementation of fuzzy rule-based contrast enhancement implementing a non-linear operator using compressed domain processing – fuzzy rule-based contrast enhancement,Takagi-Sugeno Advantage: no need to decompress/ recompress the whole image prior to processing/after processing. for the 8×8 size blocks processed in the compressed domain, the processing implies a single comparison of the coefficient with the threshold (instead of 64 comparisons needed at pixel level).

29. Description of the fuzzy rule-based contrast enhancement algorithm The fuzzy rule base of the Takagi-Sugeno fuzzy systems comprises the following 3 rules: R1: IF l u is Dark THEN l v is Darker R1: IF l u is Dark THEN l v =l v d R2: IF l u is Gray THEN l v is Midgray  R2: IF l u is Gray THEN l v =l v g R3: IF l u is Bright THEN l v is Brigter, R3: IF l u is Bright THEN l v =l v b Input and output membership functions for fuzzy rule-based contrast enhancement: For any value at the input of our Takagi-Sugeno contrast enhancement fuzzy system, in the output image, the corresponding brightness is obtained by applying the Takagi-Sugeno fuzzy inference, as: Where:,, denote the membership degrees of the currently processed brightness to the input fuzzy sets Dark, Gray and Bright.

30. The adaptive algorithm for contrast enhancement To obtain in the compressed domain the same processing results as the one given by the pixel-level approach - the algorithm must be reformulated as a block level processing. The nonlinear operations, like the thresholding in fuzzy rule-based contrast enhancement algorithm, must be carefully addressed. The DC coefficient gives the average brightness in the block - is used as an estimate for selecting the processing rule for all the pixels in the blocks with small AC energy. In this algorithm an adaptive minimal decompression is used: - full decompression is no longer needed, - but, decompression is used for the block having many details, for an improved accuracy of processing.

31. The fuzzy set parameters selection using the DC histogram in the compressed domain A reasonable choice for the thresholds values, and would be the minimum, the mean and the maximum grey level from the image histogram. Roughly speaking, if the DC coefficients would be the only ones used to reconstruct the pixel level representation (without any AC information), - they would give an approximation of the image, - with some block boundary effects and some loss of details, - but, however still preserving the significant visual information. Therefore, - the histogram built only from the DC coefficients will have approximately the same shape as the grey level histogram. Histogram of DC coefficients, and at pixel level (frog.jpg)

32. Experimental results Input membership function superimposed on the DC histogram of the Y component DC histogram of the Y component after fuzzy contrast enhancement The algorithm is applied only on the luminance component. However, it can be used to enhance color images as well, with no change of the chrominance components

33. Imagee thd EffBlocks [%] MSE frog.jpg313.751.79 woman.jpg74.421.44 Lena.jpg109.790.014 keyboard.jpg37.291.94 Results for different values e thd The Mean Squared Error (MSE ) between the pixel level processed images and the images processed with our algorithm was used as quality performance measure. The efficiency (EffBlocks) of the proposed method formulated above for the compressed domain, is evaluated by examining the number of blocks processed at pixel level as percent from the total number of 8×8 pixels blocks in the image.

34. SVM in the compressed domain