1. 1LYU0703 Electronic Advertisement Guide on PS3 1 Huang Hiu Fung 05700512 Wong Chung Hoi05596742 Supervised by Prof. Michael R. Lyu Department of Computer Science and Engineering, CUHK 2007-2008 Final Year Project Presentation (2nd term)

2. 2 Agenda Background Information Project Motivation and Objectives Implementation Result Analysis Performance and Cost Comparison Remote Demo Q&A

3. 3LYU0703 Electronic Advertisement Guide on PS3 3 Background Information Market of Commercial Monitoring Verify Ad. broadcast as stated in contract (No. of broadcast, broadcast time, duration) E.g. Large scale Ad. Campaign of HKD$10,000,000, spend 5% (HKD$ 500,000) for commercial monitoring Current Solution Monitor manually inefficient

4. 4LYU0703 Electronic Advertisement Guide on PS3 4 Project Motivation Hundreds of TV channels Increasing need for TV commercial monitoring PlayStation®3, a cheap parallel machine

5. 5 Project Objectives accurate algorithm for commercial monitoring Parallelize  better performance Generate an Electronic Advertisement Guide (EAG)

6. 6LYU0703 Electronic Advertisement Guide on PS3 6 Developing Environment PlayStation®3 A multi-core machine produced by Sony with Cell Broadband Engine processor Strong Computation Power Linux run on it Open platform for different applications and development

7. 7 Implementation of Commercial Monitoring Algorithm

8. 8 High Level Description of the Solution 1. Converting raw video data into series of Hl3 files 2. Processing Hl3 files to become the final EAG

9. 9 Hl3 Files In form of a 32 x 32 array (1024 integers) Frame capture from analog TV card Digest of a frame 352 x 288 pixels Time information on file name Hl3 2008.03.12.00.10004415.T.1.Hl3

10. 10 Obtaining Hl3 Files from Raw Video Data At first, frames captured in constant frequency For example, 25fps, 1 hr video 3600 seconds X 25 = 90,000 frames High frequency Low frequency

11. 11 Obtaining Hl3 Files from Raw Video Data Frames captured by scene change

12. 12 Processing Hl3 Files to Become EAG Minimum difference algorithm m “Target” files P (p 1, p 2, …, p k ) n “Repository” files Q (q 1, q 2, …, q k ) Match P to Q such that is minimum O (m x n x k)

13. 13 Weaknesses of Minimum Difference Algorithm 1.Many to one matching 2.Out of phrase matching 3.Force matching

14. 14 Longest Common Subsequence (LCS) ACGGT AGCTC LCS = AGT or ACT Dynamic programming i\jAGCTC 000000 0 A011111 1 C011222 2 G012222 3 G012222 4 T012233 5 012345 AGCTC 000000 0 A011111 1 C011222 2 G012222 3 G012222 4 T012233 5 012345

15. 15 LCS on Character Strings to Hl3 Files Alphabet = Hl3 file Subsequence = Adv. 1.One to one matching 2. In phrase matching 3.No force matching

16. 16 Modification of LCS algorithm R frame not match with T frame –T-R combination Out of phrase advertisement –Multiple passes LCS

17. 17 Original LCS algorithm Allows R frame match with T frame Computation = m x n x k i\jRTRTRT 0000000 0 R0 1 1 1 1 1 1 1 T0 1 2 2 2 2 2 2 R0 1 2 2 3 3 3 3 T0 1 2 3 3 3 3 4 R0 1 2 3 4 4 4 5 T0 1 2 3 4 4 5 6 0123456 RTRTRT 0000000 0 R01 1 T0 2 R0 3 T0 4 R0 5 T0 6 0123456

18. 18 T-R Combination

19. 19 T-R Combination Save the comparisons between T and R frame (half the computation) Table size reduce to m/2 x n/2 Computation: m/2 x n/2 x 2k = ½ x m x n x k i\jT-R 0000 0 0111 1 0122 2 0122 3 0123

20. 20 Single Pass LCS Cannot recognize crossed advertisement segments The longer advertisement is recognized

21. 21 Multiple Pass LCS Number of passes depends on relative complexity of 2 videos Around 7-8 passes for 2 1 hour long video Speed up by caching “Equality” comparison result in 1 st pass

22. 22 Program Flow 1.Compute result table 2.Backtrack LCS result 3.Analyze LCS result 4.Synchronize analysis result 5.Propagates result back to input data stream 6.Print output 7.Loop back to 1 until no more new advertisements detected

23. 23 Analyze LCS Result LCS result has no information on start time, end time of advertisement An LCS may contain more than one advertisements or some mismatch frame

24. 24 Printing Out Result

25. 25 Speeding up with PlayStation®3 1.Compute result table  Parallelization, SIMD 2.Backtrack LCS result Double buffering 3.Analyze LCS result Loop unrolling 4.Synchronize analysis result 5.Propagates result back to input data stream 6.Print output 7.Loop back to 1 until no more new  Caching advertisements detected comparison result Step 1:O(m x n x k) Step 2-6: O(m + n)

26. 26 Parallelization on Computing Result Table 12345 23456 34567 45678 56789 When filling the cell (i, j), information from (i-1, j-1), (i, j-1) and (i-1, j)

27. 27 Parallelization on Computing Result Table Example, 3 cores, 5x5 table Not fully utilize in first 2 steps and last 3 steps Example, 6 cores, 4000 x 4000 table Only first 5 and last 5 steps not fully utilize Core 112345 Core 223456 Core 334567 Core 1678910 Core 27891011

28. 28 Caching “Equality” Comparison Result 1 st pass done all the “Equality” comparison between all T-R pairs Memory requirement m/2 x n/2 = ¼ x m x n 1 st pass m/2 x n/2 x 2k = ½ x m x n x k 2 nd pass and onward m/2 x n/2 = ¼ x m x n In our case, k = 1024 (no. of values in Hl3) 2 nd pass and onward are speeded up by 2048X

29. 29 Result Analysis

30. 30 Experiment of Cross-Comparison of 7 Videos “TV Easy( 宣傳易 )” from 7 different days about 5 minutes only Ad. available Aim: To show all Ad. appeared twice within 7 days can be found by cross-comparison times

31. 31 Data Information

32. 32 Flow of Comparison

33. 33 Flow of Comparison

34. 34 One of the Detail Result miss in one day, still found in other day

35. 35 Overall Result

36. 36 Experiment of Comparing 2 One- Hour-Long Videos two one-hour-long videos including commercials, news reports and drama programs Aim: Show the matching rate Show the performance

37. 37 Overall Result

38. 38 False Positive Blank frame Low min difference Example of a False Positive Advertisement

39. 39 Performance and Cost Comparison

40. 40 Performance Comparison on PC and PlayStation®3

41. 41 Performance Comparison on PC and PlayStation®3

42. 42 Performance Comparison on PC and PlayStation®3 24 times faster than using only PPU on PlayStation®3 2.8 times faster than using 1 SPU on PlayStation®3 5.7 times faster than running on a PC

43. 43 Cost Comparison 1 hr compare to 6 hr 6 x 99 sec = 10 mins on PlayStation®3 6 x 562 sec = 56 mins on PC Cost for PS3 $3000 Cost for 5 PCs $20000 Cost for 3 staff $20000 per month Elapsed Time for 1hr compare 1hr PS3PC sec99562

44. 44 Conclusion Implement LCS algorithm for commercial monitoring, better than 1st term approach Parallize on PS3, 5.7 times faster than PC Achieve: accuracy ~= 95% performance ~= 99 sec for 1hr video comparing 1hr video Generate a EAG from the result

45. 45LYU0703 Electronic Advertisement Guide on PS3 45 Q&A

46. 46LYU0703 Electronic Advertisement Guide on PS3 46 The End

47. 47 Supplementary

48. 48 Limitation on Direct Memory Access (DMA) SPU transfer data to and from main memory to local store via DMA At least 16 bytes or multiple of 16 bytes Each element in result table is integer (4bytes) Compute 4 elements at a time Core 11111222233334444 Core 22222333344445555 Core 33333444455556666 Core 15555666677778888 Core 26666777788889999

49. 49 Double buffering Memory Flow Controller (MFC) operate in parallel SPU Fetch data from main memory to local store and compute result at same time Using extra buffer to store data that are pre- fetched

50. 50 SIMD intrinsic function Operate on multiple data at the same time Mostly work on float or double 128 bits register as input 4 float value at a time Speed up 4X

51. 51 Loop unrolling Get rid of computation time for loop counter in a loop Contribute to a lot computation time if little statements within loop Reduce run time from 2mins  1mins 35 secs

52. 52 Analyze LCS result Assign a flag for each LCS unit (T-R pairs) START_FLAG 'S'indicating it is a start point of an advertisement END_FLAG ‘E’indicating it is an end point of an advertisement MIDDLE_FLAG ‘>’indicating it is part of the advertisement SINGLE_FLAG‘A’indicating it is an advertisement itself (Start + End) ISOLATE_FLAG‘X’indicating it does not belong to any advertisement

53. 53 Analyze LCS result Split the LCS result into different segments by using a threshold Recognized a segments as an advertisement if duration of the segments > 5 seconds Assign a character string to the analysis result

54. 54 Synchronizing flags Analysis of LCS on 2 video stream may give different result Synchronizing is needed

55. 55 Synchronizing flags Similar to logical AND operation Recognized as advertisement if both analysis string agree with each other

56. 56 Propagates flags to input data stream

57. 57 Advantage of Comparing to 6 Other Video miss in one day, still found in other day

58. 58 False Negative Comparing 0304 to 0305Comparing 0305 to 0304 False-negative condition cause by using same scenes in 2 Ad.

59. 59 Future Development

60. 60 Supplementary – Future Development Remove TV Promo Logo recognition Use Knowledge of Electronic Program Guide Identifying New Commercials

61. 61 “Equality” Comparison Computing result table requires “Equality” comparison between 2 symbols

62. 62 Simple Logic System Difficult to decide “best” logic or scoring system As simple as possible Give high accuracy result