1. Heiko Schröder, 2003 Real time image processing on-board satellites

2. © Heiko Schröder, 2003 Parallel image processing 2 Heiko Schröder Srikanthan Thambipillai Timo Bretschneider Tobias Trenschel Ian McLoughlin Doug Maskell Wu Jigang The PPU for X-SAT1 and beyond

3. © Heiko Schröder, 2003 Parallel image processing 3 1000 nm RGB 2500 nm Multispectral

4. © Heiko Schröder, 2003 Parallel image processing 4 CHRIS Multispectral

5. © Heiko Schröder, 2003 Parallel image processing 5 Hyperspectral Precision farming!

6. © Heiko Schröder, 2003 Parallel image processing 6 Increasing Performance of X-SAT1 7 km/s 100 Mbit/sec 4000 s/orbit 400 Gbit/orbit download: 4 Gbit/orbit On-board image analysis and compression 685 km Singapore 100 x output if useful/useless<=1/100 100 x value

7. © Heiko Schröder, 2003 Parallel image processing 7 Performance evaluation P=C/A X-SAT1 without PPU: P=15,000,000/25,000/1200 =.5 $/km 2 $12,500 for 50x500 km 2  air plane Useful? What are we looking for? C -- Cost A --image area: Useful (can be sold)

8. © Heiko Schröder, 2003 Parallel image processing 8 Our aim: High performance via COTS 16 processors (+ spares) off-the-shelf connected via a fault tolerant reconfigurable network Mesh/torus Real-time

9. © Heiko Schröder, 2003 Parallel image processing 9 processors fault tolerant mesh on-board PPU for X-SAT1

10. © Heiko Schröder, 2003 Parallel image processing 10 FPGA ctrl h/v o/e r/w Instructions to PEs link to PE BSP? Mesh with slow recovery Real-time

11. © Heiko Schröder, 2003 Parallel image processing 11 ctrl h/v o/e r/w Instructions to PEs Diagnostic set switches Mesh with fast recovery Not on X-SAT1

12. © Heiko Schröder, 2003 Parallel image processing 12 Available data (320 images) – search task Oil slicks, forest fires, red tide, settlements, … Random selection U=1/5 Output Algorithms: Compression Classification Segmentation

13. © Heiko Schröder, 2003 Parallel image processing 13 Compression ratio (CR=4 loss-less) Segmentation gain (SG=16, 1/16 of a useful image is useful) Classification gain (CG=5, 1 in 5 images contain useful information) U=.8U=.2 U=4 U=1 U=16 The satellite efficiency cube Not likely LOSSY=60 U=32 U=64 (0,0,0)

14. © Heiko Schröder, 2003 Parallel image processing 14 TargetModeClassification gain Segmentation gain Total gain Oil slickSearch>100 ShipsSearch>10>100 Air pollutionSearch/investigate>10 >100 StormsSearch/investigate>100<10>100 FloodsSearch/investigate>100<10>100 LandslidesSearch/investigate>1000>10>100 VolcanicSearch/investigate>100>10>100 Forrest firesSearch/investigate>100>10>100 Assumption: Exhaust download capacity  PPU can achieve price reduction by more than 2 orders of magnitude $100 for 50x500 km 2 image. Enough useful data? – Customers?

15. © Heiko Schröder, 2003 Parallel image processing 15 What is a good algorithm? Fast – real-time Correct – low error rate Classification: Error 1: Does not detect a good image Error 2: Flags a bad image as useful

16. © Heiko Schröder, 2003 Parallel image processing 16 Choice of Image Processing Routines: Evaluation criteria (gain): G=U P /U U – useful area/data received without PPU U P – useful area/data received with PPU Algorithm A: Real-time, flags 50% of good images 5, flags 1% of bad images 10,  5*2/3=3.3 G A =33 Example: 1000 pictures can be taken, 10 pictures are good, 10 pictures can be downloaded .1 picture without PPU Algorithm B: ¼ real-time, flags 90% of good images 2.5x.9, flags.1% of bad images 1,  2.3 G B =23 Algorithm C: Real-time, flags 40% of good images 4, flags.1% of bad images 1,  4 G C =40

17. © Heiko Schröder, 2003 Parallel image processing 17 12345678 910111213141516 1718192021222324 2526272829303132 3334353637383940 4142434445464748 4950515253545556 5758596061626364 LL 1+2+3+4 HL 1+3-2-4 LH 1+2-3-4 HH 1+4-2-3 12 34 LLHL LH HH LLHL LH HH LLHL LH HH LLHL LH HH LLHL LH HH LLHL LH HH LLHL LH HH LLHL LH HH LLHL LH HH LLHL LH HH LLHL LH HH LLHL LH HH LLHL LH HH LLHL LH HH LLHL LH HH LLHL LH HH LL LH HL HH 12345678 910111213141516 1718192021222324 2526272829303132 3334353637383940 4142434445464748 4950515253545556 5758596061626364 L 1+2 H 1-2 L 3+4 H 3-4 Invertible! + /2 - /2

18. © Heiko Schröder, 2003 Parallel image processing 18 HL1 HH1 LH1 LL1 7+8 +15+16 23+24 +31+32 38+40 +47+48 55+56 +63+64 5+6 +13+14 21+22 +29+30 37+38 +24+46 53+54 +61+62 3+4 +11+12 19+20 +27+28 35+36 +43+44 51+52 +59+60 1+2 +9+10 17+18 +25+26 33+34 +41+42 49+50 +57+58 LL2 LH2 HL2 HH2 HL2 HH2 LL2 LH3 HL3 HH3 LL3 33+42 -34-41 35+44 -36-43 49+58 -50-57 51+60 -52-59

19. © Heiko Schröder, 2003 Parallel image processing 19

20. © Heiko Schröder, 2003 Parallel image processing 20 How to find areas of interest Image classification ?

21. © Heiko Schröder, 2003 Parallel image processing 21 Thresholding

22. © Heiko Schröder, 2003 Parallel image processing 22 Mathematical morphology erosion dilation erosion edge detection, thinning, noise removal, enlarging Structural element reference point

23. © Heiko Schröder, 2003 Parallel image processing 23 Thresholding MM-segmentation

24. © Heiko Schröder, 2003 Parallel image processing 24 MM-Hough Transform reference point erosion m d d m a dot leads to one addition if there is a matching point

25. © Heiko Schröder, 2003 Parallel image processing 25 1min 420km Investigative mode 72 sec 500km 36sec 250km

26. © Heiko Schröder, 2003 Parallel image processing 26 30 sec 210km 7 min 2900 km Follow coast Search mode 400 sec: Storm: 3km Ship: 5km Fire: 50m Air plane: 100km

27. © Heiko Schröder, 2003 Parallel image processing 27 2000 km High-performance Computer network Image analysis Classification Segmentation compression Intelligent search Maximize the efficiency/ useful output of the satellite! 200 km

28. © Heiko Schröder, 2003 Parallel image processing 28 skeletons Compression and classification

29. © Heiko Schröder, 2003 Parallel image processing 29 diamond circles 2 3 2 1 5 13

30. © Heiko Schröder, 2003 Parallel image processing 30 odd square circles 8-neighbourhood skeleton 36 13 04

31. © Heiko Schröder, 2003 Parallel image processing 31 square circles red-square-skeleton 1 1 0 3 0 0 0 1 6

32. © Heiko Schröder, 2003 Parallel image processing 32 red square skeletons one-sweep algorithm to produce the red square skeleton wavefront --- ISA new = min{W,NW,N}+1

33. © Heiko Schröder, 2003 Parallel image processing 33 granularity Histograms of skeletons classify images

34. © Heiko Schröder, 2003 Parallel image processing 34 segmentation locate objects partition image thinning

35. © Heiko Schröder, 2003 Parallel image processing 35 Mathematical morphology erosion dilation erosion edge detection, thinning, noise removal, enlarging Structural element reference point

36. © Heiko Schröder, 2003 Parallel image processing 36 Border following X 12 3 56 7 8 4 2333333 4 5 6 5 5 6 6 67 8 8 1 8 1 1 1 Image classification 1 2 3 4 5 6 7 8 Histogram: #

37. © Heiko Schröder, 2003 Parallel image processing 37 Image classification I

38. © Heiko Schröder, 2003 Parallel image processing 38 Image classification II

39. © Heiko Schröder, 2003 Parallel image processing 39 Image classification III

40. © Heiko Schröder, 2003 Parallel image processing 40 Hough transform good line detection method m d d m every dot leads to M (1K) additions d m

41. © Heiko Schröder, 2003 Parallel image processing 41 MM-Hough transform reference point erosion m d d m a dot leads to one addition if there is a matching point

42. © Heiko Schröder, 2003 Parallel image processing 42 HT on the ISA Bertil

43. © Heiko Schröder, 2003 Parallel image processing 43 shearing 0 1 2 3 4 5 column sum 15/2022/10 MM OR AND 15/25/612/28/16 eliminates dirt OR AND 15/1012/16 d d : parameter

44. © Heiko Schröder, 2003 Parallel image processing 44 skewing 0 1 2 3 4 5 6 7 8 23/11 15/25/6 16/21 12/28/16

45. © Heiko Schröder, 2003 Parallel image processing 45 alternatives 0 1 2 3 4 5 6 7 8 MM OR AND } } OR } AND } } OR }

46. © Heiko Schröder, 2003 Parallel image processing 46 advantages of MM-HT higher contrast less additions more flexibility –lines of given thickness –dashed lines –lines of given length –lines of given orientation –circles, … tomography !!

47. © Heiko Schröder, 2003 Parallel image processing 47 circles

48. © Heiko Schröder, 2003 Parallel image processing 48 Hough transform I

49. © Heiko Schröder, 2003 Parallel image processing 49 Hough transform II

50. © Heiko Schröder, 2003 Parallel image processing 50 robot vision projector CCD stereo vision thinning (skeletons or erosion), line detection (MM-HT), trigonometry

52. © Heiko Schröder, 2003 Parallel image processing 52 Design a mathematical morphology algorithm (and demonstrate by means of example), that removes all isolated patterns of size 2 (black black on white and white on black). It does not change any set of 3 neighbouring pixels with identical colour. Write an algorithm that removes all squares of maximal size from a given image. Write a program based on MM, that fills gaps in horizontal and vertical lines up to length 2, but does not prolong the ends of lines.

53. Application specific massive parallelism Low cost alternatives to polygons for visualisation

54. © Heiko Schröder, 2003 Parallel image processing 54 contents scan-line image processing PIPS architecture from landscapes to 3D surface generation for CAD

55. © Heiko Schröder, 2003 Parallel image processing 55 basic architecture 1 1024 high resolution real time

56. © Heiko Schröder, 2003 Parallel image processing 56 PIPS (1990-94) 1 M bit 1 M bit 32x32 torus 16 bit parallel communication 16 bit add prefetch memory control BHP -- CSIRO -- NU -- ADFA 1.4 M

57. © Heiko Schröder, 2003 Parallel image processing 57 elementary operations compress horizontal shear vertical shear

58. © Heiko Schröder, 2003 Parallel image processing 58 scan-line image processing height colour rotate shear 2x transpose P. Robertson 1986 A. Spray 1990-4

59. © Heiko Schröder, 2003 Parallel image processing 59 horizontal projection f h m y a x x = f ((m-y) cos(a) + h sin(a) ) / ( f + m sin(a) - h cos(a))

60. © Heiko Schröder, 2003 Parallel image processing 60 transpose algorithm transpose: 1 diagonal/step 1024 steps 1step: 1 read 1 move (PE-PE) 1 write

61. © Heiko Schröder, 2003 Parallel image processing 61 HC/torus diameter / bandwidth 1024 nodes 12 Diameter 32 56 Diameter 10 56*4= 224 12*16=192 4 bit wide 16 bit wide

62. © Heiko Schröder, 2003 Parallel image processing 62 tailored towards transpose transpose operation ---> torus alternatives: hypercube, linear array, hypercubic networks. off-the-shelf SRAM memory chips determined performance. transpose: read pixel, move pixel, write pixel. average distance of 32x32 torus is 16. read and a write take 8 cycles each. 1 M bit 1 M bit memory control

63. © Heiko Schröder, 2003 Parallel image processing 63 tailored towards interpolation Interpolation is the most frequent operation. linear interpolation (nearest neighbour, spline) 1 multiplication and 2 additions per pixel (18 cycles) overlapping arithmetic and memory access (prefetch 16 cycles) y h1 h2 d y = h1 + (h2-h1)d

64. © Heiko Schröder, 2003 Parallel image processing 64 performance 1 perspective view (464 K) –1 rotation (170 K) 3 shears (3x36 K) 2 transpose (2x 31 K) –1 compress (41 K) –1 projection (219 K) –1 image output (34 K) 464 K x 50ns = 23.2 ms 43 frames / sec

65. © Heiko Schröder, 2003 Parallel image processing 65 performance parameters 20 GOPS (16 bit words) IEEE standard 32bit floating point: –200 instructions / floating point operation –100 MFLOPS / 1024 PEs fast floating point: –80 instructions / floating point operation –250 MFLOPS / 1024 PEs 5 Gbytes/s internal memory bandwidth 40 Gbytes/s inter-processor communication

66. © Heiko Schröder, 2003 Parallel image processing 66 performance Main criterion: high throughput machine A: price P A, time k t B machine B: price k P A, time t B k A-machines cost and produce as much as one B-machine evaluation criterion: Cost x time (cost x period; AT; AP)

67. © Heiko Schröder, 2003 Parallel image processing 67 cost-performance Timecostcost x Time SUN 2 min 3 K 360 MasPar 5 sec 100 K 500 1024 PIPS1/40 sec 20 K 1/2 1024

68. © Heiko Schröder, 2003 Parallel image processing 68 from landscapes to 3D

69. © Heiko Schröder, 2003 Parallel image processing 69 Unsolved problems: partitioning 3D surfaces into landscapes detail on demand target architecture (distributed & parallel)...

70. © Heiko Schröder, 2003 Parallel image processing 70 partitioning the surfaces into pieces of landscapes Bez, May, Schroeder

71. © Heiko Schröder, 2003 Parallel image processing 71 partitioning algorithms fixed set of observer points? how many? observer position data dependent?

72. © Heiko Schröder, 2003 Parallel image processing 72 Detail on demand 1/4 1/16 1/64 1/256 1/1

73. © Heiko Schröder, 2003 Parallel image processing 73 1/16 1/64 1/256 1/4 1/16

74. © Heiko Schröder, 2003 Parallel image processing 74 Levels of resolution Provide image at various levels of resolution

75. © Heiko Schröder, 2003 Parallel image processing 75 detail on demand wavelet transform ? all data should be kept at several levels of resolution R. Lang, P. Lenders, H. Schroeder (1995/6)

76. © Heiko Schröder, 2003 Parallel image processing 76 Wavelet transform (simplified) Easy reconstruction! Low-pass-filter High-pass-filter

77. © Heiko Schröder, 2003 Parallel image processing 77 Spiral (Rein Warmels)

78. © Heiko Schröder, 2003 Parallel image processing 78 Butterfly network for FFT FFT frequency spectrum image classification CM2

79. © Heiko Schröder, 2003 Parallel image processing 79 FFT without butterfly 3579111315 246810121416 112569101314 347811121516 13249111012 576813151416 15372648 913111510141216

80. © Heiko Schröder, 2003 Parallel image processing 80 target architecture ? PCs: How many? ISAs ! -- with every PC? partitioning the screen amongst ISAs distribution of data over PCs ATM switch: PVM/MPI --- BSP ? optical communication? Edinburgh? Jena?

81. © Heiko Schröder, 2003 Parallel image processing 81 visualise what? landscapes gallery of the future physical data simulations medical images CAD

82. © Heiko Schröder, 2003 Parallel image processing 82 + + ++ + + + + + control points: + ++ + + ++ + + ++ + + ++ + CAD 9/16 3/16 1/16 Catmul & Clark (78) Pham & Schröder (89)

83. © Heiko Schröder, 2003 Parallel image processing 83 ++ + + (9A + 3C + 3D + B)/16 C B D A 9 add-shift 9/4 per control point 3 per pixel +

84. © Heiko Schröder, 2003 Parallel image processing 84 move algorithms ? routing algorithm (warping)? –hot potato? (Kaufmann, Schröder 94) warping “cheaper” than general routing?

85. © Heiko Schröder, 2003 Parallel image processing 85 scan-lines ? tiling? -- no transpose! hidden surface removal via “z”-value

86. low cost, real-time, high resolution visualisation can be done !!!