1. 1 LYU0503 Document Image Restoration on Mobile Phone Using Onboard Camera Supervisor: Prof. Michael R.Lyu Group Member: Leung Man Kin, Stephen Ng Ying Kit, Tommy

2. 2 Outline Introduction Introduction Approach Approach Stitching Algorithm Stitching Algorithm Geometric Transformation Geometric Transformation Progress Progress Conclusion Conclusion Future Plan Future Plan

3. 3 Outline Introduction Introduction Approach Approach Stitching Algorithm Stitching Algorithm Geometric Transformation Geometric Transformation Progress Progress Conclusion Conclusion Future Plan Future Plan

4. 4 Introduction Use mobile phone to capture photo of document. Use mobile phone to capture photo of document. The document can be stored in the phone or sent to other people by several means. The document can be stored in the phone or sent to other people by several means.

5. 5 Objective Use camera to take photo of different parts of a document. Use camera to take photo of different parts of a document. Use suitable algorithm to merge the photos back to a whole document. Use suitable algorithm to merge the photos back to a whole document. Transmit the document by several means Transmit the document by several means e.g. Fax, MMS, etc.e.g. Fax, MMS, etc.

6. 6 Motivation Most of phones nowadays have built-in camera. Most of phones nowadays have built-in camera.  Resolution of camera is not enough for document The text on the document would be blurredThe text on the document would be blurred Symbian phone can provide several transmission function. E.g. Fax, MMS Symbian phone can provide several transmission function. E.g. Fax, MMS Fax at any time! Fax at any time!

7. 7 Outline Introduction Introduction Approach Approach Stitching Algorithm Stitching Algorithm Geometric Transformation Geometric Transformation Progress Progress Conclusion Conclusion Future Plan Future Plan

8. 8 Approach Take photos of different part of document Take photos of different part of document Use stitching algorithm to merge the photos into a photo Use stitching algorithm to merge the photos into a photo Stitching algorithm is our major research area of our project Stitching algorithm is our major research area of our project

9. 9 Assumption The document consists mainly of text. The document consists mainly of text. The camera has macro mode. The camera has macro mode. Sharp Close Up.Sharp Close Up.

10. 10 Approach Procedure of our design Procedure of our design Take a photo of full document first.Take a photo of full document first. Then ask the user to take photos of different partsThen ask the user to take photos of different parts Overlay a rough picture on the screen of the phone. Overlay a rough picture on the screen of the phone.

11. 11 Approach A4 document

12. 12 Outline Introduction Introduction Approach Approach Stitching Algorithm Stitching Algorithm Geometric Transformation Geometric Transformation Progress Progress Conclusion Conclusion Future Plan Future Plan

13. 13 Stitching Algorithm Stitching needs to find corresponding points in two images Stitching needs to find corresponding points in two images two different approaches two different approaches Direct alignmentDirect alignment Compare images pixel by pixel Compare images pixel by pixel e.g. e.g. Feature-based registrationFeature-based registration Use some methods to find feature points Use some methods to find feature points Match features in two images Match features in two images

14. 14 Stitching Algorithm Feature-based registration is chosen. Feature-based registration is chosen. Usually, direct alignment is less efficient.Usually, direct alignment is less efficient. Need to consider different scale, rotation, etc.Need to consider different scale, rotation, etc. There are several methods to extract feature. There are several methods to extract feature. E.g. corner detector.E.g. corner detector. Scalable Invariant Feature Transform (SIFT) is used to extract the feature. Scalable Invariant Feature Transform (SIFT) is used to extract the feature. The features are invariant to image scale and rotation.The features are invariant to image scale and rotation.

15. 15 Stitching Algorithm How to get the feature point? How to get the feature point? Identifies key locations in scale space by looking for locations that are maxima or minima of a difference-of-Gaussian function.Identifies key locations in scale space by looking for locations that are maxima or minima of a difference-of-Gaussian function.

16. 16 Stitching Algorithm Varying σ Varying σ Repeat the above process using subsampled image for several times Repeat the above process using subsampled image for several times Decrease the resolution by 2.Decrease the resolution by 2.

17. 17 Stitching Algorithm Find the maxima or minima by comparing 26 neighbors. Find the maxima or minima by comparing 26 neighbors. They are the key feature points. They are the key feature points.

18. 18 Stitching Algorithm Property of the feature keypoints Property of the feature keypoints Achieve partial invariance to local variations like affine or 3D projections.Achieve partial invariance to local variations like affine or 3D projections. Need to filter feature keypoints furtherNeed to filter feature keypoints further Reject keypoints that have low contrast. Reject keypoints that have low contrast.

19. 19 Stitching Algorithm How to store the feature? How to store the feature? For each keypoint, image gradient magnitudes and orientations are sampled to store the feature.For each keypoint, image gradient magnitudes and orientations are sampled to store the feature.

20. 20 Stitching Algorithm Compare the features in two images Compare the features in two images Get the position to stitchGet the position to stitch However, the photo taken may suffer from geometric distortion However, the photo taken may suffer from geometric distortion e.g. Lens distortion, taking photo with different anglese.g. Lens distortion, taking photo with different angles

21. 21 Stitching Algorithm Use the matched feature to calculate the transformation parameters. Use the matched feature to calculate the transformation parameters. Transform images to the correct orientation before merging Transform images to the correct orientation before merging

22. 22 SIFT Stitching Algorithm Extract feature keypoint Matching keypoints in two images Find transformation parameter by matched keypoints Transform images using the parameter Merge all the images

23. 23 Outline Introduction Introduction Approach Approach Stitching Algorithm Stitching Algorithm Geometric Transformation Geometric Transformation Progress Progress Conclusion Conclusion Future Plan Future Plan

24. 24 Lens Correction Transformation

25. 25 Transformation parameters a i Transformation parameters b i Image2 Image3 Image1

26. 26 Take several overlapping images of the document ……

27. 27 High Resolution Document Image Restored!

28. 28 Outline Introduction Introduction Approach Approach Stitching Algorithm Stitching Algorithm Geometric Transformation Geometric Transformation Progress Progress Conclusion Conclusion Future Plan Future Plan

29. 29 Porting to Symbian Phone On the PC, the stitching algorithm is implemented using C# and Matlab. On the PC, the stitching algorithm is implemented using C# and Matlab. We should port the algorithm to the Symbian phone. We should port the algorithm to the Symbian phone. Symbian OS use C++ as the language. Symbian OS use C++ as the language.

30. 30 Porting to Symbian Phone Difficulty: Difficulty: Their syntaxes are not the same.Their syntaxes are not the same. e.g. set / get keyword in C# e.g. set / get keyword in C# Symbian and C# provide different API.Symbian and C# provide different API. Math library, Array library. Math library, Array library. Some of data types are different in Symbian C++.Some of data types are different in Symbian C++.

31. 31 Porting to Symbian Phone Difficulty: Difficulty: The processor on Symbian phone is not powerful.The processor on Symbian phone is not powerful. The memory space on Symbian phone is not large.The memory space on Symbian phone is not large. We may need modify the algorithm so that it would suit the phone. We may need modify the algorithm so that it would suit the phone.

32. 32 Outline Introduction Introduction Approach Approach Stitching Algorithm Stitching Algorithm Geometric Transformation Geometric Transformation Progress Progress Conclusion Conclusion Future Plan Future Plan

33. 33 Conclusion We have studied different stitching algorithms We have studied different stitching algorithms We have implemented the stitching algorithm on PC We have implemented the stitching algorithm on PC We are porting the stitching algorithm on Symbian phone We are porting the stitching algorithm on Symbian phone

34. 34 Outline Introduction Introduction Objective Objective Motivation Motivation Approach Approach Stitching algorithm Stitching algorithm Geometric Transformation Geometric Transformation Progress Progress Conclusion Conclusion Future Plan Future Plan

35. 35 Future Plan Implement the stitching on the Symbian phone. Implement the stitching on the Symbian phone. Modify SIFT or design other algorithms for more efficient performance on mobile phone Modify SIFT or design other algorithms for more efficient performance on mobile phone

36. 36 Future Plan Improve the algorithm of merging. Improve the algorithm of merging. Study super-resolution to make text clearer. Study super-resolution to make text clearer.