1. WCPM 1 Chang-Tsun Li Department of Computer Science University of Warwick UK Image Clustering Based on Camera Fingerprints

2. WCPM 2 Digital Image Acquisition Process Scene Post- Processing Lens Sensor CFA Interpolation CFA Photo CFA: Colour Filter Array RG GB Bayer CFA RG GB mapping of CFA to sensor pixels RG GB RG GB RG GB RG GB RG GB

3. WCPM 3 What is Camera Fingerprint Lens aberration Sensor pattern noise Colour filter array (CFA) interpolation artefacts Camera response function Quantisation table of JPEG compression Scene Post- Processing Lens Sensor CFA Interpolation CFA Photo

4. WCPM 4 Camera Fingerprint for Multimedia Forensics Multimedia Forensics: The use of “fingerprints” left in images by the imaging devices for source device identification source device linking content integrity verification image classification

5. WCPM 5 What is Sensor Pattern Noise l Sensor Pattern Noise (SPN) is the noise left in the images by the sensors of digital imaging devices such as cameras, camcorders and scanners. l SPN is mainly caused by –manufacturing imperfection and –different sensitivity of pixels to light due to the inhomogeneity of silicon wafers. l Sensors made from the same silicon wafer possess unique SPN because of the non-uniform imperfection. l SPN can differentiate cameras of the same model.

6. WCPM 6 “Traditional” SPN Extraction Method l Lukáš et al’s model for SPN extraction (IEEE TIFS 2006) –I is the original image –I’ is the low-pass filtered version of I by the Weiner filter applied in the wavelet transform domain –n is the extracted SPN SPN is the high-frequency component of the image.

7. WCPM 7 Interference from Scene Details l Scene details, e.g., brick walls, tree leaves, or other kinds of textures, contribute to the high-frequency components of images. a contaminated SPN natural image SPN a clean SPN

8. WCPM 8 SPN Enhancement at Warwick C.-T. Li, "Source Camera Identification Using Enhanced Sensor Pattern Noise," IEEE Trans. on Information Forensics and Security, June 2010 C.-T. Li and Y. Li, "Color-Decoupled Photo Response Non- Uniformity for Digital Image Forensics," IEEE Trans. on Circuits and Systems for Video Technology, 2012 X. Lin and C.-T. Li, "Preprocessing Reference Sensor Pattern Noise via Spectrum Equalization," IEEE Trans. on Information Forensics and Security, 2016. X. Lin and C.-T. Li, "Enhancing Sensor Pattern Noise via Filtering Distortion Removal," IEEE Signal Processing Letter, accepted for publication in 2016

9. WCPM 9 Image Classification/Clustering Scenario: A forensic investigator has a large set of images taken by an unknown number of unknown digital cameras and wishes to cluster those images into a number of classes, each including the images acquired by the same camera. Each data point represents one image Each cluster present one unknown device

10. WCPM 10 Challenges Facing Image Classification l The forensic investigator does not have the cameras that have taken the images to generate reference SNPs for comparison. l No prior knowledge about the number and types of the imaging devices are available. l With a large dataset, exhaustive fingerprint comparison is computationally prohibitive. l Given the shear number of images, analysing each image in its full size is computationally infeasible.

11. WCPM 11 Image Classification – a MRF Approach Step 1. Extract and enhance the fingerprint of each block cropped from the images Step 2. Establish a similarity matrix ρ for a Focus Set of M images Step 3. Train the classifier based on the similarity matrix ρ. For each fingerprint i, which is treated as a random variable 3.1. Assign a unique random class label 3.2. Calculate a reference similarity (i.e., a “soft” threshold) 3.3. Establish a membership committee (neighbourhood) 3.4. Update the class label iteratively based on the information from the membership committee until there are no changes of class labels to any SPN throughout a entire iteration Step 4. Classify the rest of the dataset using the classifier

12. WCPM 12 Establishing Similarity Matrix To establish an M × M similarity matrix ρ, the similarity between any two enhanced SPNs i and j in the Focus Set is calculated using i, j 1234….M 11.00 2 3 4 :::: M

13. WCPM 13 Classifier Training Each fingerprint (SPN) is treated as a random variable. 3.1. Assign a unique random class label to each SPN 3.2. Calculate a reference similarity r Normally intra-class similarities > inter-class similarities. A similarity less than r indicates that the two images are taken by different devices, otherwise by the same device. similarity Similarity  inter-class similarity  intra-class similarity μ 1 r μ 2

14. WCPM 14 3.3. Establish a membership committee For each SPN i, a membership committee C i with c SPN members from the focus set that are most similar to i is established. Classifier Training vv vv vv ×

15. WCPM 15 f i : class label of SPN i C ρ( i, C i ) is the similarities between SPN i and the members of C i, i.e., 3.4. Update the class label iteratively according to p ( f i |ρ( i, C i ), L i ) until there is no change of class label to any SPN in x consecutive iterations Classifier Training p ( f i |ρ( i, C i ), L i ): probability of assigning f i given the conditions r i : reference similarity (“soft” threshold) of I

16. WCPM 16 similarity Similarity  inter-class similarity  intra-class similarity μ 1 r μ 2 The combination of the s(.) and ρ(.) says, a penalty (i.e. positive value) will be incurred  if ρ(i,j) > r and a different label than f i is to be assigned to i or  if ρ(i,j) < r and the same label as f j is to be assigned to i a reward (i.e. negative value) will be given  if ρ(i,j) < r and a different label than f i is to be assigned to i or  if ρ(i,j) > r and the same label as f j is to be assigned to i Objective Function / Cost Function

17. WCPM 17 Image Classification l The centroids of the image clusters provided by the classifier training process at the end of Step 3.4 are used to classify the images. To classify an image x, we compare the similarity of its SPN to the centroid of each identified cluster and classify it to the class with its centroid closest to the image.

18. WCPM 18 Clustering in progress ….. Classifier Training - Simulation Initial label configure: Each pattern is assigned an unique label / colour

19. WCPM 19 Final Classification

20. WCPM 20 Experimental Results c Block Size 256 × 256256 × 512512 × 512 Focus set size (M) 120300120300120300 M-1 M/2 M/3 M/4 M/5 Table 1. Classification error rate. c is the size of the membership committee. c : the size of the membership committee M : the size of the focus set 8.889 8.333 4.000 3. 778 2.333 3. 778 1.333 1.222 1.444 1.556 1.444 Misclassification rates: 1200 images taken by six cameras, each taking 200. Class identification stops when there are no class label changes throughout an iteration. A misclassification rate in the range (1.2 ~ 1.6) is likely to be the best the system can achieve.

21. WCPM 21 Conclusions l Multimedia forensics using “fingerprint” left in the images by the imaging devices has emerged as a new area of research in the last few years. l Sensor pattern noise (SPN) is one of the most promising types of fingerprint. l The “traditional” SPN extraction method is unable to cope with the interference of scene details. l The proposed classifier is feasible, but is unable to classify images without clean SPNs provided by the proposed SPN enhancer.