1. Copyright Protection of Images Based on Large-Scale Image Recognition Koichi Kise, Satoshi Yokota, Akira Shiozaki Osaka Prefecture University

2. Outline  Demo  Image Recognition with Local Descriptors  Our Research Topics  Copyright Protection and Digital Watermarking  Proposed Method  Experiments  Conclusion 2

3. DEMO Large-Scale Image Recognition

4. Outline  Demo  Image Recognition with Local Descriptors  Our Research Topics  Copyright Protection and Digital Watermarking  Proposed Method  Experiments  Conclusion 4

5. Image Recognition with Local Descriptors Database Images are represented by local descriptors (feature vectors) Typically a few hundreds to a few thousands are extracted Image 1 Image 2 Image 3 Image 4 Query 5

6. Database Image 1 Image 2 Image 3 Image 4 Query Image Recognition with Local Descriptors Nearest Neighbor Search 1 2 3 4 no. of votes max. votes 6 Nearest vector in the DB Nearest vector in the DB

7. Pros and Cons  Extraction of descriptors a few seconds / query  Matching By brute-force NN search with the DB of 1,000 images 315 seconds / query PROS CONS Robust to occlusion and variations Locality and stability of descriptors Large number of expensive descriptors 7

8. Outline  Demo  Image Recognition with Local Descriptors  Our Research Topics  Copyright Protection and Digital Watermarking  Proposed Method  Experiments  Conclusion 8

9. Our Research Topics Extraction of Descriptors Matching (with 10,000 images) BaselineSoftware2.3 sec.Brute-force3,000 sec. Existing Technologies ANN (Approximate Nearest Neighbor) 50 ms 60,000 times faster Our MethodsGPU & CUDA0.57 sec.Hash & Cascade0.78 ms 4 times faster60 times faster 9 Application Copyright Protection

10. Outline  Demo  Image Recognition with Local Descriptors  Our Research Topics  Copyright Protection and Digital Watermarking  Proposed Method  Experiments  Conclusion 10

11. Copyright Protection and Digital Watermarking  Digital Watermarking is a technology to embed information (copyright notice) to original digital data.  Embedded information (watermark) is often unnoticeable.  We focus on digital images as original data.  Methods:  Non-blind  Blind 11 Copyright: A Purchaser: B Copyright: A Purchaser: B original watermarked

12. Non-Blind and Blind Watermarking  Non-blind watermarking  Blind watermarking  History  From non-blind to blind watermarking  Non-blind watermarking needs matching of images  spoils the scalability  Blind watermarking has been the focus of interest 12 watermarked image & originalimage Copyright: A Purchaser: B Copyright: A Purchaser: B watermarked image Copyright: A Purchaser: B Copyright: A Purchaser: B

13. Limitations of Blind Watermarking  StirMark  distorts images imperceptibly  severely damages the wartermark 13 OriginalStirMark

14. StirMark: Some Image Transformations 14 corner shift cropping punch / pinch original fraction of pixel displacement

15. Outline  Demo  Image Recognition with Local Descriptors  Our Research Topics  Copyright Protection and Digital Watermarking  Proposed Method  Experiments  Conclusion 15

16. Proposed Method: Ideas  How do we solve the problem of blind watermarking?  Cancel the effect of StirMark, especially geometric transformations  Normalization could be done by comparing with the original image  Blind to non-blind 16 StirMark’ed OriginalStirMark’ed normalization normalized StirMark’ed

17. Proposed Method: Ideas  How to solve the problem of non-blind watermarking?  Matching problem  solved by the efficient and robust image recognition  How about the normalization?  Image recognition with local descriptors  provides matching of descriptors as a side effect  helpful for normalization 17

18. Proposed Method: Ideas  To solve the problem of StirMark  Image normalization is required to detect the watermark  Normalization could be done by comparing with the original image  Non-blind watermarking  Efficient and robust image recognition  overcome the difficulty of matching in non-blind watermarking  precise matching of local descriptors (feature points) is obtained as a side effect of recognition  This helps us to normalize the StirMark’ed image to detect the watermark 18

19. Proposed Method: Processing Steps 19 STEP1: Recognition Image Database query image point matching STEP2: Normalization STEP3: Detection Embedded Watermark

20. Proposed Method: Processing Steps 20 STEP2: Normalization Embedded Watermark STEP3: Detection Local Descriptors: PCA-SIFT Matching: ANN Local Descriptors: PCA-SIFT Matching: ANN STEP1: Recognition Image Database query image point matching

21. Outline  Demo  Image Recognition with Local Descriptors  Our Research Topics  Copyright Protection and Digital Watermarking  Proposed Method  Experiments  Conclusion 21

22. Experiments: Overview  Experiment 1  Robustness against StirMark with default setting (imperceptible change)  Experiment 2  Robustness against perceptible changes  Experiment 3  Robustness against watermarking 22

23. Experiments: Overview  Experiment 1  Robustness against StirMark with default setting (imperceptible change)  Experiment 2  Robustness against perceptible changes  Experiment 3  Robustness against watermarking 23

24. Experiments: Dataset & Task 24 10,000 images 100 Base set WatermarkStirMark query images Input : query image Task: find its original from 10,000 images Input : query image Task: find its original from 10,000 images

25. Experiment 1 25 imperceptible 100 random application with default setting 10,000 images 100 StirMark query images 10,000 images 100 query images watermarkingwithoutwith accuracy100%99.49% time (per query)217ms216ms Watermark with StirMark without 10,000

26. Experiment2  4 image transformations  cropping, fraction of pixel displacement, punch/pinch, corner shift  perceptible 26 perceptible 4 trans. 43 application in total 10,000 images 100 StirMark query images Watermark 10,000 images 100 StirMark query images with without 4,300

27. 97% Cropping: 100 ; Accuracy: 97%

28. Cropping: Original Image

29. 97% Cropping: 70; Accuracy 97% 29

30. 95% Cropping: 50; Accuracy: 95% 30

31. 92% Cropping: 30; Accuracy: 92% 31

32. Cropping 32 With watermarking Without watermarking Accuracy [%] Cropping parameter

33. Punch / Pinch Original Image 33

34. 99% Punch / Pinch: 1; Accuracy: 99% 34

35. 99% Punch / Pinch: 25; Accuracy: 99% 35

36. 97% Punch / Pinch: 50; Accuracy: 97% 36

37. 50% Punch / Pinch: 100; Accuracy: 50% 37

38. Punch / Pinch 38 Accuracy [%] Punch / Pinch Parameter With watermarking Without watermarking

39. Conclusion  A non-blind framework for copyright protection of images has been proposed  Efficient image recognition technology with local descriptors  Fundamental experiments on image recognition  Image recognition is robust to StirMark attacks and watermarking  Future Work  Completion of overall method 39

40. Copyright Protection of Images Based on Large-Scale Image Recognition Thank you for your attention

41. Image Recognition: Local Descriptors  PCA-SIFT (PCA Scale-Invariant Feature Transform)  36 dimensions  Invariant to scale and rotation changes 41

42. Image Recognition: Matching  ANN(Approximate Nearest Neighbor)  Tree structure Feature Space Feature Vector

43. ANN ： Feature Vector of a query

44. ANN Nearest Neighbor Search for

45. ANN NN Search for Approximate Result This cell is not searched

46. ANN more approximation with a smaller circle Wrong Correct Errors caused by approximation NN Search for Approximate

47. NNS accuracy vs. Image recognition rate 47 80 85 90 95 100 0 10 20 30 40 50 60 70 80 90 100 Image recognition rate[%] NNS accuracy[%] ANN dist E2LSH

48. Corner Shift Original Image 48

49. 99% Corner Shift: 1; Accuracy: 99% 49

50. 98% Corner Shift: 100; Accuracy: 98% 50

51. 96% Corner Shift: 150; Accuracy: 96% 51

52. 74% Corner Shift: 225; Accuracy: 74% 52

53. Corner Shift 53 Accuracy [%] Corner Shift Parameter With watermarking Without watermarking

54. Fraction of Pixel Displacement Original Image 54

55. 99% Fraction of Pixel Displacement: 1; Accuracy 99% 55

56. 94% Fraction of Pixel Displacement: 3 Accuracy: 94% 56

57. 81% Fraction of Pixel Displacement: 5; Accuracy: 81% 57

58. 58% Fraction of Pixel Displacement: 8 Accuracy: 58% 58

59. 24% Fraction of Pixel Displacement: 10 Accuracy: 24% 59

60. Fraction of Pixel Displacement 60 Accuracy [%] Fraction Parameter With watermarking Without watermarking