1. ENERGY-PROPORTIONAL IMAGE SENSING FOR Robert LiKamWa Bodhi Priyantha Matthai Philipose Victor Bahl Lin Zhong CONTINUOUS MOBILE VISION http://roblkw.com http://research.microsoft.com

2. CONTINUOUS MOBILE VISION CONTINUOUS MOBILE VISION Conversational Face Detection 2 of 33

3. CONTINUOUS MOBILE VISION CONTINUOUS MOBILE VISION Gestures Object Memory 3 of 33 Face Recognition Victor Fine-grained Localization

4. BATTERY LIFE BATTERY DEATH Google Glass 2-3 hours LooxCie 2-3 hours GoPro Hero 2-3 hours 4 of 33

5. 2000 mWh 10 h 200 mW 5 of 33

6. Image Sensor Goal < 25 mW Sensors ~ 5 mW Processor ~150 mW Network Stack ~20 mW 6 of 33

7. Image Sensor Goal < 25 mW 7 of 33 Image Sensor Reality > 250 mW

8. KEY IDEA: ENERGY α QUALITY Power Frame rate Power Resolution

9. ENERGY PROFILE OF AN IMAGE SENSOR 1 MP, 5 fps 250 mW 1 MP, 5 fps 250 mW 0.3 MP, 15 fps 245 mW 0.3 MP, 15 fps 245 mW 1 MP, 15 fps 295 mW 1 MP, 15 fps 295 mW < 25 mW Goal: Reality: 0.3 MP, 5 fps 232 mW 0.3 MP, 5 fps 232 mW 9 of 38

10. ENERGY-EFFICIENT IMAGE SENSING Image Sensor Characterization Energy Reduction Techniques Energy vs. Vision Performance 10 of 38

11. IMAGE SENSOR MEASUREMENT Camera Module Programmable Clock (I2C) NI DAQ Device Power Rail Resistors Power Rail Resistors Power VDDCLK * Profiled 5 state-of-the-art image sensors from 2 manufacturers * 11 of 38

12. Camera Module Programmable Clock (I2C) NI DAQ Device Power Rail Resistors Power Rail Resistors IMAGE SENSOR MEASUREMENT 12 of 38

13. IMAGE SENSOR WAVEFORMS Active Period Idle Period Analog Digital PLL 13 of 38

14. IMAGE SENSOR WAVEFORMS Active Period Idle Period 14 of 38

15. IMAGE SENSOR WAVEFORMS Active Period Idle Period Pixel Count divided by Clock Frequency Frame Time minus Active Time

16. IMAGE SENSOR PIXEL COUNT (N) Region-of-Interest (Windowing) Active Power Time Power Time 16 of 38 Scaled Resolution (Pixel Skipping)

17. IMAGE SENSOR PIXEL COUNT (N) Active Power Time Power Time Video (30 FPS) Power vs. Resolution 17 of 38

18. Active Frame Readout Active Power Time Active IMAGE SENSOR FRAME RATE (R) Video (0.1 MP) Power vs. FPS Active Frame Readout Active Power Time Active 1.0s 18 of 38

19. CHARACTERIZATION CONCLUSION: NO ENERGY PROPORTIONALITY Video (0.1 MP) Power vs. FPS Video (30 FPS) Power vs. Resolution 19 of 38

20. ENERGY-EFFICIENT IMAGE SENSING Image Sensor Characterization Energy Reduction Techniques Energy vs. Vision Performance 20 of 38

21. TECHNIQUE # 1 : AGGRESSIVE STANDBY Active 21 of 38

22. Active CAVEAT TO AGGRESSIVE STANDBY Active Not enough exposure time This won’t work for long active periods, i.e., high resolution, high frame rate. 22 of 38

23. TECHNIQUE # 2 : CLOCK SCALING (f) One pixel per clock period 23 of 38

24. TECHNIQUE # 2 : CLOCK SCALING (f) Faster clock Lower Active Time Higher Active Power Higher Idle Power Slower clock Higher Active Time Lower Active Power Lower Idle Power 24 of 38

25. TECHNIQUE # 2 : CLOCK SCALING (f) Active Low Pixel Count Low Frame Rate Slowed Clock Optimal clock frequency depends on Pixel Count & Frame Rate 25 of 38

26. AGGRESSIVE STANDBY + CLOCK OPTIMIZATION Sped-up Clock Aggressive Standby Active Readout Active Readout Activ e Read out Activ e Read out Optimal clock frequency depends on Pixel Count & Frame Rate Exposure Time 26 of 38

27. ENERGY α QUALITY Frame rate (FPS) Resolution (MP) 30 25 20 15 10 5 0 21345 Frame rate (FPS) Resolution (MP) 30 25 20 15 10 5 0 21345 Power (mW) 350 300 250 200 150 100 50 0 Aggressive Standby & Clock Optimization Unoptimized Power (mW) 350 300 250 200 150 100 50 0 27 of 38

28. CPU Driver 28 of 37

29. 29 of 38

30. CURRENT IMAGE SENSOR DESIGN Image Processor Pixel Array Column Output Pixel Array Column Output Gain, ADC Analog Signal Chain 70-85% Power Consumption 30 of 38

31. Image Processor Pixel Array Column Output Pixel Array Column Output High-Speed ADC Mid-Speed ADC Low-Speed ADC Gain HETEROGENEOUS SENSOR DESIGN Analog Signal Chain Heterogeneous 31 of 38

32. ENERGY α QUALITY Default Clock select Standby HW Fix Power vs. Framerate (at 0.1 MP) Power vs. Resolution (at 5 FPS) Default Clock select Standby HW Fix 32 of 36

33. ENERGY-EFFICIENT IMAGE SENSING Image Sensor Characterization Energy Reduction Techniques Energy vs. Vision Performance 33 of 38

34. ENERGY vs. VISION: VISION TASK IMAGE REGISTRATION 34 of 36

35. ENERGY vs. VISION: PERFORMANCE Image Registration Success Power Reduction with software assist Estimated Power Reduction with hardware assist Full VGA Resolution 0.1 MP, 30 FPS 99.9%51%84% Frame Rate Reduction 0.1 MP, 3 FPS 95.7%95%98% 30% Window 0.06 MP, 30 FPS 96.5%63%91% Subsampled by 2 0.3 MP, 30 FPS 91.8%71%94% 185 mW Typical Average Power 10 mW With aggr. standby & optimal clock 3 mW With heterogeneous analog signal chain

36. Scalable Computer Vision Algorithms Integrated Systems Design Developer Support Energy Proportional Image Sensing 36 of 36

38. ENERGY-PROPORTIONAL IMAGE SENSING FOR CONTINUOUS MOBILE VISION http://roblkw.com http://research.microsoft.com Image sensors are not energy-proportional… Frame rate (FPS) Resolution (MP) 30 20 10 0 135 …but we can make them energy-proportional… Aggressive Standby Clock Optimization Sensor Modifications … and this is just the beginning.