1. DIGITAL CAMERAS IN EMBEDDED SYSTEMS BY: ANTONIO JIM É NEZGRIFFIN MILLERVARUN PATEL

2. BACKGROUND First cameras built in mid 1800s Pinhole Cameras (not even cameras) Silver-plated sheets of copper Film cameras Digital cameras (CCD then CMOS)  uses a semiconductor (silicon)  light charges up each pixel / sensor and is then processed Who took this? ^ https://i.ytimg.com/vi/43KcHyKN1t8/hqdefault.jpg https://upload.wikimedia.org/wikipedia/commons/f/fd/Camera_Obscura_box18thCentury.jpg

3. Why do WE need digital cameras in embedded systems?

4. READABLE DATA Digital cameras and pictures are a way of transmitting information that’s easy for humans to process Enables embedded systems to “see” http://www.faculty.virginia.edu/rwoclass/astr1230/im/human-eye-crossec.jpg

5. CAPTURING A SPECIFIC IMAGE FOR ITS DATA QR Codes Facial Recognition Security http://cdni.wired.co.uk/1920x1280/g_j/google-glass-facial- recognition.jpg

6. SMART TECHNOLOGY Autonomous Robotics  Collision control  Remote control http://www.seaeye.com/images/falcon_probe.jpg http://i.telegraph.co.uk/multimedia/archive/03130/medium-3-3d53e7185_3130912c.jpg https://www.google.com/maps/streetview/images/understand/fleet/device-carousel1.jpg Smart Cameras Going beyond the image Pre vs post processed Human Detection

7. LIGHT SENSING Indoor Localization  Takes advantage of the time-lapse in digital cameras Light Fidelity  GBs per second  Cheap and easy to implement  different frequencies

8. Image sensors: CCD vs CMOS

9. Same task: photons to electrons http://www.kenrockwell.com/nikon/d60.htm http://dew.globalsystemsscience.org/tools/digital-cameras-overview/effect-of-removing-near-infrared-filter-from-digital-camera

10. CCD: Charge-Coupled Device 2-D array of light-sensing “buckets” on top of a thin silicon wafer Each bucket consists of a positively doped MOS-Capacitor Incident photons hit the oxide and free electrons The greater the light intensity, the greater amount of freed electrons These photo-generated electrons are stored Each bucket in the array is capable of representing a pixel http://www.microscopyu.com/articles/digitalimaging/ccdintro.html

11. Reading the charges (CCD) Charges are shifted in two directions: parallel and serial The serial shift occurs at the edge of the array  Charges are shifted towards an ADC to be digitized  The resulting voltage passes through an amplifier  The value for each pixel is stored in memory After the edge empties, the parallel shift occurs  Every row of capacitors passes its charge to the neighboring row  Each row gets rid old charge and receives new charge The process repeats until every charge is processed http://spiff.rit.edu/classes/phys445/lectures/ccd1/ccd1.html https://commons.wikimedia.org/wiki/File:CCD_charge_transfer_animation.gif

12. CMOS: COMPLEMENTARY METAL OXIDE SEMICONDUCTOR Similar technique for photo-generating electrons Pixel charges are passed to a group of readers Each reader has its own charge-to-voltage conversion, amplifier, and ADC converter No need to shift charges across the pixel array Rolling shutter captures light from top left to bottom right Manufacturing process is the same for regular chips Animation http:// www.diyphotography.net/everything-you-wanted-to-know-about-rolling-shutter / CMOS CCD

13. SO WHICH ONE IS THE BEST? CCD: Pros: ❏ Global shutter provides uniformity ❏ Almost every photon generates an electron ❏ Levels of noise are very reduced Cons: ❏ Special manufacturing process is expensive ❏ Relatively low speed ❏ High power consumption ❏ Blooming CMOS: Pros: ❏ Conventional manufacturing process is cheap ❏ High speed ❏ Low power consumption Cons: ❏ Rolling-shutter effect ❏ Light sensitivity is not the best

14. SO WHICH ONE IS THE BEST? CCD: Pros: ❏ Global shutter provides uniformity ❏ Almost every photon generates an electron ❏ Levels of noise are very reduced Cons: ❏ Special manufacturing process is expensive ❏ Relatively low speed ❏ High power consumption ❏ Blooming CMOS: Pros: ❏ Conventional manufacturing process is cheap ❏ High speed ❏ Low power consumption Cons: ❏ Rolling-shutter effect ❏ Light sensitivity is not the best

15. Blooming http://www.astrosurf.com/luxorion/digital-darkroom.htm CCDnot CCD

16. SO WHICH ONE IS THE BEST? CCD: Pros: ❏ Global shutter provides uniformity ❏ Almost every photon generates an electron ❏ Levels of noise are very reduced Cons: ❏ Special manufacturing process is expensive ❏ Relatively low speed ❏ High power consumption ❏ Blooming CMOS: Pros: ❏ Conventional manufacturing process is cheap ❏ High speed ❏ Low power consumption Cons: ❏ Rolling-shutter effect ❏ Light sensitivity is not the best

17. SO WHICH ONE IS THE BEST? CCD: Pros: ❏ Global shutter provides uniformity ❏ Almost every photon generates an electron ❏ Levels of noise are very reduced Cons: ❏ Special manufacturing process is expensive ❏ Relatively low speed ❏ High power consumption ❏ Blooming CMOS: Pros: ❏ Conventional manufacturing process is cheap ❏ High speed ❏ Low power consumption Cons: ❏ Rolling-shutter effect ❏ Light sensitivity is not the best

18. ROLLING-SHUTTER EFFECT http:// www.diyphotography.net/everything-you-wanted-to-know-about-rolling-shutter /

19. not CMOS CMOS http://www.productionapprentice.com/tutorials/ccd-vs-cmos/ Fast-moving objects

20. Summary of CCD vs CMOS http://blog.ipphone-warehouse.com/ip-camera-image-sensor-guide/ CCDCMOS Light sensitivityHighLow Noise levelLowModerate ShutterGlobalRolling Power consumption HighLow Chip outputAnalogDigital

21. What happens after raw image is captured? Camera’s use SIMD (Single-Instruction Multiple-Data) Microprocessors Redundant tasks are often performed to process images (filters) Latency due to processing the image

22. Functional Block Diagram (CCD) BUS Serial/Parallel shift signals

23. Sampling Optimization Each pixel is a mixture of Red, Green, Blue 12 bits per color = 4096 shades of color Most cameras use 8 bits = 256 steps 256 x 256 x 256 > 16 million colors http://www.astropix.com/HTML/I_ASTROP/HOW.HTM 256 Shades 16 Shades 50 Shades of Grey http://www.rocketroberts.com/astro/calibration.htm http://www.cs.sfu.ca/CourseCentral/166/tjd/_images/colorGrayscales.png

24. CMU CAM4 libraries: https://github.com/EECS-373-Driving-School/CMUCam4 https://github.com/orgicus/RPI-CV-Goodies/tree/master/CMUCam4GUI/source/CMUcam4GUI http://www.societyofrobots.com/member_tutorials/node/321 CMU CAM4 6MHz Clock TTL UART bus protocol Baud Rate : 19,200 to 250,000 Parallax P8X32A Microchip 30 FPS Servo controllers at 50Hz Refresh Rate $39.95 (robotmesh)

25. Few Functions in CMUCAM4 library Getpixel Andpixels cameraBrightness getServoPosition setServoPosition trackColor trackWindow trackShape And many other functions

26. Raspberry Pi Camera 3.3 V 5MP 1080p Video Quality $34.95 (Sparkfun.com) CMOS Camera Module (serial com port) 6 – 20 V PC1089K CMOS Chip NTSC protocol (yellow wire) $49.00 https://www.sparkfun.com/products/11745 http://www.datasheet-pdf.com/PDF/PC1089K-Datasheet-Pixelplus-925068 http://techcrunch.com/2014/04/27/heres-what-happens-when-you-strap-an- oculus-controlled-video-camera-to-a-drone-and-take-it-for-a- spin/?utm_campaign=fb&ncid=fb Raspberry Pi Camera (OV5647) CMOS High Quality Video 5 Mega Pixel 1080p @ 30 FPS (frames/sec) @ 68 MHz 960p @ 45 FPS @ 91.2 MHz Camera Serial Interface $34.95 https://www.sparkfun.com/products/11868 Uart Camera Module with Jpeg compression - C328 3.3 V UART: 115.2Kbps for transferring JPEG still pictures or 160x128 preview @8bpp with 0.75fps. Built-in color conversion circuits $49.00 http://www.seeedstudio.com/depot/Uart-Camera-Module-with-Jpeg-compression-C328-p- 209.html

27. http://lens.blogs.nytimes.com/2015/12/23/the-future-of-computational- photography/?_r=0&module=Slide&region=SlideShowTopBar&version=SlideCard- 3&action=Click&contentCollection=Blogs&slideshowTitle=The%20Future%20of%20Digital%20Photography&currentS lide=3&entrySlide=1&pgtype=imageslideshow Photography of the Future?

28. References http://www.specinst.com/What_Is_A_CCD.html http://www.microscopyu.com/articles/digitalimaging/ccdintro.html https://www.teledynedalsa.com/imaging/knowledge-center/appnotes/ccd-vs-cmos/ http://www.techhive.com/article/246931/cmos_is_winning_the_camera_sensor_battle_an d_heres_why.html http://www.steves-digicams.com/knowledge-center/how-tos/digital-camera- operation/ccd-vs-cmos-whats-the-difference.html#b

29. Q & A