Presentation is loading. Please wait.

Presentation is loading. Please wait.

Dongsoo Kim Heungjun Samsung

Similar presentations


Presentation on theme: "Dongsoo Kim Heungjun Samsung"— Presentation transcript:

1 Dongsoo Kim (dongsoo45.kim@samsung.com)dongsoo45.kim@samsung.com Heungjun Kim(riverful.kim@samsung.com) Samsung Electronicsriverful.kim@samsung.com

2 2 Introducing mobile camera devices SoC camera subsystem Mobile camera module devices Framework for digital camera Video4Linux2 and camera device New APIs for high-end mobile camera devices Dual camera framework for 3G handset Porting issues in SoC camera subsystem

3 3 AP : Application Processor Camera interface : interface device which is syncing image data from external camera device and gives some commands to them Camera module : external camera device packaged with lens, sensor and ISP on it ISP : Image Signal Processor. Also supports lots of additional functions V4L2 : Video for Linux 2

4 4 This document contains some animations. Slideshow mode is recommended

5 5 Exposure control Aperture (Iris) Shutter ISO Lens control Focus Zoom Strobe support Various program mode : slow sync, red-eye reduction and on….

6 6 Introducing mobile camera devices

7 7 History of camera phone First Camera Phone in 2000 with 3.5MP 1.3MP in 2003 2MP in 2004 3MP in2004 5MP in2004 7MP in 2005 10MP in 2006 12MP in 2009

8 8 Old and low-end mobile camera modules FeatureLow end camera Exposure ApertureNot supported ShutterJust change frame rate ISOLow sensitivity & High noise level Lens FocusPan focus ZoomDigital zoom StrobeJust LED light most of time

9 9 Brand new and high-end mobile camera modules FeatureHigh end camera Exposure ApertureSupported with lens packaging ShutterSupported with lens packaging ISOHigh sensitivity & Low noise level Lens FocusMovable focus point (Auto/manual) ZoomOptical zoom & Digital zoom StrobeXENON flash is also supported

10 10 What is SoC camera subsystem? ARM SoC solution ARM Core Peripheral1(Display) Peripheral2 (MMC) Peripheral3 (USB)Camera interface Peripherals External Camera module External Camera module

11 11 Camera interface Buffer handling Buffer handling Resolution handling Data format handling (YUV/RAW/JPEG …) Data format handling (YUV/RAW/JPEG …) Camera module I2C Exposure control Exposure control Lens control Lens control Strobe control Strobe control White balance control Effect control Face detect Face detect Zoom control Zoom control............ Command interface Command interface RAW DATA Image data in expected formatRequest for data

12 12 Various ways of preview (Live view) in SoC camera subsystems Camera Module MemoryMemory Camera Interface ARM core Frame buffer OK Cancel User Application COPY! DATA

13 13 Various ways of preview (Live view) in SoC camera subsystems Camera Module MemoryMemory Camera Interface ARM core Frame buffer OK Cancel User Application COPY! DMADMA Through DMA DATA Save as file

14 14 How to capture still shots in SoC camera subsystem Your camera module has a dedicated JPEG encoder on it MemoryMemory Camera Interface ARM core User Application COPY! DATA MEMORY Camera module RAW DATA PROCES S JPEG PROCES S DATA Save as JPEG file

15 15 How to capture still shots in SoC camera subsystem No JPEG encoder in your camera module Camera Module MemoryMemory Camera Interface ARM core User Application COPY! DATA RAW DATA Codec S/W Save as JPEG file

16 16 How to take motion pictures in SoC camera subsystem Camera Module MemoryMemory Camera Interface ARM core Frame buffer OK Cancel User Application COPY! DATA COPY! Codec S/W Save as…

17 17 In Kernel V4L2 (AKA Video For Linux 2) The second version of the Video For Linux API Kernel interface for analog radio and video capture and output drivers In user space Multimedia middleware Gstreamer Pipeline based multimedia framework written in the C programming language with the type system based on GObject OpenMAX Cross-platform set of C-language programming interfaces that provides abstractions for routines especially useful for audio, video, and still images

18 18 Is V4L2 enough for mobile digital camera modules? No way

19 19 Framework for digital camera

20 20 Camera subsystem in Linux kernel aspect Camera interface Buffer handling Buffer handling Resolution handling Data format handling (YUV/RAW/JPEG…) Data format handling (YUV/RAW/JPEG…) Command interface Command interface V4L2 driver Camera module Exposure control Exposure control Lens control Lens control Strobe control Strobe control White balance control Effect control Face detect Face detect Zoom control Zoom control............ V4L2 driver videobuf videobuf- core videobuf- sg-dma videodev V4l2- dev V4l2- ioctl V4l2- subdev (new) V4l2-device V4l2-int- device (old) V4L2

21 21 How does it work with V4L2? Open device VIDIOC_QUERYCA P Application or Middleware VIDIOC_S_INPUT VIDIOC_REQBU F VIDIOC_QBUF VIDIOC_STREAMO N VIDIOC_QBUF VIDIOC_DQBUF VIDIOC_QUERYBUF Copy memory to userspace

22 22 New V4L2 APIs for digital camera Exposure control

23 23 Need for detailed control in exposure Weve got camera devices with mechanical shutter Iris could be handled also ISO controlled output is somehow useful Why not using enhanced exposure control like a regular digital camera?

24 24 What weve got in mainline V4L2 right now? #define V4L2_CID_EXPOSURE_AUTO (V4L2_CID_CAMERA_CLASS_BASE+1) enum v4l2_exposure_auto_type { V4L2_EXPOSURE_AUTO = 0, V4L2_EXPOSURE_MANUAL = 1, V4L2_EXPOSURE_SHUTTER_PRIORITY = 2, V4L2_EXPOSURE_APERTURE_PRIORITY = 3 }; #define V4L2_CID_EXPOSURE_ABSOLUTE (V4L2_CID_CAMERA_CLASS_BASE+2) #define V4L2_CID_EXPOSURE_AUTO_PRIORITY (V4L2_CID_CAMERA_CLASS_BASE+3) No way to control iris No way to control exposure metering method

25 25 Additional exposure control enum v4l2_exposure_auto_type { V4L2_EXPOSURE_AUTO = 0, V4L2_EXPOSURE_MANUAL = 1, V4L2_EXPOSURE_SHUTTER_PRIORITY = 2, V4L2_EXPOSURE_APERTURE_PRIORITY = 3, +/* Additional features for digital camera */ +V4L2_EXPOSURE_ISO_PRIORITY =4, }; #define V4L2_CID_EXPOSURE_ABSOLUTE (V4L2_CID_CAMERA_CLASS_BASE+2) #define V4L2_CID_EXPOSURE_AUTO_PRIORITY (V4L2_CID_CAMERA_CLASS_BASE+3)... +#define V4L2_CID_EXPOSURE_LOCK (V4L2_CID_CAMERA_CLASS_BASE+18)

26 26 Supported aperture stages, shutter speed, and ISO speed could be different between every different camera module products +#define V4L2_CID_CAM_APERTURE(V4L2_CID_CAMERA_CLASS_BASE+19) +#define V4L2_CID_CAM_SHUTTER(V4L2_CID_CAMERA_CLASS_BASE+20) +#define V4L2_CID_CAM_ISO(V4L2_CID_CAMERA_CLASS_BASE+21) Supported values for aperture, shutter and ISO should be made in V4L2_CTRL_TYPE_MENU {.id = V4L2_CID_CAM_APERTURE,.type = V4L2_CTRL_TYPE_MENU,.name = "Aperture",.minimum = 0,.maximum = 4,.step = 1,.default_value = 0, }, static const char *camera_iris_stages[] = { /* This module supports 5 Iris stages * on it's own but F number depends * how you package Lens module */ "2.8, "4, "5.6, "8, "11, NULL };

27 27 New exposure metering control +/* Exposure Methods */ +#define V4L2_CID_PHOTOMETRY(V4L2_CID_CAMERA_CLASS_BASE+17) +enum v4l2_photometry_mode { + V4L2_PHOTOMETRY_MULTISEG= 0, + V4L2_PHOTOMETRY_CWA = 1, + V4L2_PHOTOMETRY_SPOT = 2, + V4L2_PHOTOMETRY_AFSPOT = 3, +};

28 28 New V4L2 APIs for digital camera Lens control

29 29 Need for enhanced Focus mode API Current V4L2 framework just slightly covers MANUAL & AUTO FOCUS mode for usb webcam High-end camera modules support for various focus mode presets like AF Macro AF-S / AF-C For compatibility Focus control API should be defined

30 30 Focus Mode & Enumeration values #define V4L2_CID_FOCUS_ABSOLUTE (V4L2_CID_CAMERA_CLASS_BASE+10) #define V4L2_CID_FOCUS_RELATIVE (V4L2_CID_CAMERA_CLASS_BASE+11) +#define V4L2_CID_FOCUS_MODE (V4L2_CID_CAMERA_CLASS_BASE+13) +/* Focus Methods */ +enum v4l2_focus_mode { + V4L2_FOCUS_AUTO= 0, + V4L2_FOCUS_MANUAL= 1, + V4L2_FOCUS_MACRO= 2, + V4L2_FOCUS_CONTINUOUS= 3, +};

31 31 New V4L2 APIs for digital camera Object recognition

32 32 Object detection Gee! It recognized my face! Hello everyone

33 33 What do we set? Detect mode Type of object How many objects to detect Triggered action What do we get? How many objects are detected Detected objects type Detected objects coordinates

34 34 New API for object recognition New capability #define V4L2_CAP_ASYNCIO 0x02000000 /* async I/O */ #define V4L2_CAP_STREAMING 0x04000000 /* streaming I/O ioctls */ +#define V4L2_CAP_OBJ_RECOGNITION0x10000000 +#define VIDIOC_S_RECOGNITION_IOWR ('V', 85, struct v4l2_detect) +#define VIDIOC_G_RECOGNITION_IOR ('V', 86, struct v4l2_detect) New IOCTL

35 35 +/* Object detection and triggered actions */ +enum v4l2_recog_mode { + V4L2_RECOGNITION_MODE_OFF= 0, + V4L2_RECOGNITION_MODE_ON= 1, + V4L2_RECOGNITION_MODE_LOCK= 2, +}; + +enum v4l2_recog_action { + V4L2_RECOG_ACTION_NONE= 0,/* only detection */ + V4L2_RECOG_ACTION_BLINK= 1,/* Capture on blinking */ + V4L2_RECOG_ACTION_SMILE= 2,/* Capture on smiling */ +}; + +enum v4l2_recog_pattern { + V4L2_RECOG_PATTERN_FACE= 0; /* Face */ + V4L2_RECOG_PATTERN_HUMAN= 1; /* Human */ + V4L2_RECOG_PATTERN_CHAR= 2; /* Character */ +} New API for object detection

36 36 +struct v4l2_recog_rect { + enumv4l2_recog_pattern p;/* What pattern detected */ + structv4l2_rect o;/* detected area */ + __u32reserved[4]; +} + +struct v4l2_recognition_data { + __u32detect_cnt; /* detected object counter */ + structv4l2_recog_rectobj; /* detected area */ + __u32reserved[4]; +}; + +struct v4l2_recognition { + enum v4l2_recog_mode mode; + enum v4l2_recog_pattern pattern;/* What pattern to detect */ + __u32 obj_num;/* How many object to detect */ + __u32detect_idx;/* select detected object */ + struct v4l2_recog_datadata; /* read only */ + enum v4l2_recognition_actionaction; + __u32reserved[4]; +}; New API for object detection

37 37 What do we set? Detect mode How many objects to detect Triggered action What do we get? How many objects are detected Detected objects coordinates

38 38 New API for object recognition New capability #define V4L2_CAP_ASYNCIO 0x02000000 /* async I/O */ #define V4L2_CAP_STREAMING 0x04000000 /* streaming I/O ioctls */ +#define V4L2_CAP_OBJ_RECOGNITION0x10000000 +#define VIDIOC_S_RECOGNITION_IOWR ('V', 85, struct v4l2_detect) +#define VIDIOC_G_RECOGNITION_IOR ('V', 86, struct v4l2_detect) New IOCTL

39 39 +/* Object detection and triggered actions */ +enum v4l2_recognition_mode { + V4L2_RECOGNITION_MODE_OFF= 0, + V4L2_RECOGNITION_MODE_ON= 1, + V4L2_RECOGNITION_MODE_LOCK= 2, +}; + +enum v4l2_recognition_action { + V4L2_RECOG_ACTION_NONE= 0,/* only detection */ + V4L2_RECOG_ACTION_BLINK= 1,/* Capture on blinking */ + V4L2_RECOG_ACTION_SMILE= 2,/* Capture on smiling */ +}; + +struct v4l2_recognition_data { + __u8detect_cnt; /* detected object counter */ + structv4l2_recto; /* detected area */ +}; + +struct v4l2_recognition { + enum v4l2_recognition_mode mode; + __u8 obj_num;/* How many object to detect */ + __u8detect_idx;/* select detected object */ + struct v4l2_recognition_datadata; /* read only */ + enum v4l2_recognition_actionaction; +}; New API for object detection

40 40 New V4L2 APIs for digital camera Dual camera

41 41 What is dual camera? Dual camera applications Video telephony Camera applications Taking self portrait shots Flip side camera : for still shots Front side camera : for special purpose

42 42 H/W restrictions in Application Processors camera peripheral interface Mega camera VGA camera Data path

43 43 H/W restrictions in Application Processors camera peripheral interface SINGLE camera interface on Application Processor ! SINGLE camera interface on Application Processor ! Only one camera could be using data pins at a time Each camera is different in Working resolution (SYNC) Working MCLK Working PCLK Only one camera could be handled at a time

44 44 We already have API for dual camera Enumerate video inputs Query or select the current video input VIDIOC_ENUMINPUT VIDIOC_G_INPUT VIDIOC_S_INPUT

45 45 What do we need to consider? Context of each camera module Capability : Supported color spaces, resolutions, additional functions… Latest working status : resolution, color space, effect and so on Context of each camera module Capability : Supported color spaces, resolutions, additional functions… Latest working status : resolution, color space, effect and so on Context of camera interface Working information of the camera module getting switched : Clock information and so on Context of camera interface Working information of the camera module getting switched : Clock information and so on Let user space application get noticed about camera input devices Dont forget VIDIOC_ENUMINPUT Check out supported pixel format using VIDIOC_ENUM_FMT and VIDIOC_ENUM_FRAMESIZES Let user space application get noticed about camera input devices Dont forget VIDIOC_ENUMINPUT Check out supported pixel format using VIDIOC_ENUM_FMT and VIDIOC_ENUM_FRAMESIZES

46 46 How to switch active camera Condition 2 Shared things Camera power enable pin Camera data pins MCLK Condition 3 Dedicated things Camera enable pins Camera reset pins Condition 1 How to enable camera module: Give power source Enable expecting camera Give MCLK CAM POWER MCLKDATA PINS Mega Enable Mega Reset VGA Enable VGA Reset Mega cameraVGA camera CPU

47 47 STREAMOFF Camera (A) initialize Camera (A) initialize Start preview Switch to camera (B) Camera (B) initialize Camera interface reset & setting Start preview Reset camera interfaces current setting Make a proper MCLK for camera (B) Setup resolution, color-space, DMA Stop preview Stop DMA Cam power off Camera (B) Enable pin on Camera (A) enable pin off Cam power on Stop preview STREAMON G_INPUT/ S_INPUT G_INPUT/ S_INPUT

48 48 Porting issues in SoC camera subsystem

49 49 I2C protocol has start and stop condition Some camera modules generate unexpected noise until their power up sequence Makes I2C bus in bus busy state

50 50 How to prevent this malfunctioning issue Use a noise free on power up camera module Use a level shifter to block and ignore noise from camera module Change functionality of I2C pins while turning on camera module OR

51 51 All about between camera interface and camera module Expected resolution from camera interface Served resolution from camera module Symptoms depend on camera interface Expecting resolution is bigger than served resolution A select time out error in user space application or Preview image is floating in some direction Serving resolution is bigger than expected resolution A select time out error in user space application or Preview angle looks like to be zoomed in

52 52 Color order mismatched between camera interface and camera module

53 53 Preview freezing while checking lens response Using while() loop while it checks response, preview will freeze until focusing job gets finished If the ISP driver use the kernel thread when it check response from current lens status could make smooth preview on the LCD while lens is on focusing.

54 54 Solving preview freezing while checking lens status struct platform_isp { const struct platform_data *pdata; struct v4l2_int_device *v4l2_int_device; struct i2c_client *i2c_client; int reschk;......... } static int ioctl_s_ctrl(struct v4l2_int_device *s, struct v4l2_control *vc) { struct platform_isp *isp = s->priv; struct i2c_client *client = isp->i2c_client; switch (vc->id) { case V4L2_CID_FOCUS_AUTO: isp->vfm = vc->value; if (vc->value == V4L2_CID_FOCUS_MACRO) { err = i2c_command_for_isp( client, focus_macro ); isp->reschk = 1; } break; } If isp->reschk is Set, The kernel thread is active.

55 55 Solving preview freezing while checking lens status int pseudo_reschk_thread (void *data) { int err = 0; ……… do { switch (isp->reschk) { case 1: err = i2c_command_for_isp(client, focus_check); if (err < 0) { isp->reschk = 1; /* i2c command function failed. */ } else { isp->reschk = 0; /* if succeed, reschk ended. */ } break; …… } } while(1); return 0; } This function return when the ISP is matched the focus(+ return) or timeout(- return). This is able to re-check to the next check time. Platform Data Vairable for isp, or static variable.

56 56 JPEG has no synchronization protocol when it gets transferred All about camera interface and camera modules data synchronization When camera interface supports JPEG sync Follow user manual and make camera module to follow that protocol When camera interface does not support JPEG sync Make synchronization protocol: make JPEG data to be synchronized with VSYNC

57 57 Q&A

58 58 Thank you


Download ppt "Dongsoo Kim Heungjun Samsung"

Similar presentations


Ads by Google