1. Framework for digital camera in Linux
Dongsoo Kim
Heungjun Samsung Electronics

2. Contents Introducing mobile camera devices
SoC camera subsystem
Mobile camera module devices
Framework for digital camera
Video4Linux2 and camera device
New APIs for high-end mobile camera devices
Dual camera framework for 3G handset
Porting issues in SoC camera subsystem

3. Terms AP : Application Processor
Camera interface : interface device which is syncing image data from external camera device and gives some commands to them
Camera module : external camera device packaged with lens, sensor and ISP on it
ISP : Image Signal Processor. Also supports lots of additional functions
V4L2 : Video for Linux 2

4. This document contains some animations. Slideshow mode is recommended

5. What makes camera “a camera”?
Strobe support
Various program mode : slow sync, red-eye reduction and on….
Exposure control
Aperture (Iris)
Shutter
ISO
Lens control
Focus
Zoom

6. Introducing mobile camera devices

7. History of camera phone First Camera Phone in 2000 with 3.5MP
1.3MP in 2003
5MP in2004
3MP in2004
7MP in 2005
2MP in 2004
10MP in 2006
12MP in 2009

8. Mobile camera module devices
Old and low-end mobile camera modules
Feature
Low end camera
Exposure
Aperture
Not supported
Shutter
Just change frame rate
ISO
Low sensitivity & High noise level
Lens
Focus
Pan focus
Zoom
Digital zoom
Strobe
Just LED light most of time

9. Mobile camera module devices
Brand new and high-end mobile camera modules
Feature
High end camera
Exposure
Aperture
Supported with lens packaging
Shutter
ISO
High sensitivity & Low noise level
Lens
Focus
Movable focus point (Auto/manual)
Zoom
Optical zoom & Digital zoom
Strobe
XENON flash is also supported

10. ARM SoC solution ARM Core
SoC camera subsystem
What is SoC camera subsystem?
ARM SoC solution
Peripherals
External
Camera module
Camera interface
Peripheral1(Display)
Peripheral2 (MMC)
Peripheral3 (USB)
ARM
Core

11. Camera interface Camera module
SoC camera subsystem
Request for data
Image data in expected format
Camera interface
Command
interface
Resolution handling
Data format handling
(YUV/RAW/JPEG…)
Buffer
handling
I2C
RAW DATA
Camera module
Exposure
control
Lens
control
White balance control
Effect control
Face
detect
Zoom
control
Strobe
control .

12. SoC camera subsystem COPY! User Application
Various ways of preview (Live view) in SoC camera subsystems
User Application
ARM
core
COPY! M e m o r y
Frame buffer
DATA
Camera
Interface
DATA
Camera
Module OK
Cancel

13. SoC camera subsystem COPY! Through DMA User Application Save as file
Various ways of preview (Live view) in SoC camera subsystems
User Application
ARM
core M e m o r y
COPY!
Save
as file
DATA D M A
Camera
Interface
DATA
Camera
Module
Frame buffer
Through DMA OK
Cancel

14. SoC camera subsystem
How to capture still shots in SoC camera subsystem
Your camera module has a dedicated JPEG encoder on it
User Application
ARM
core
Camera module
DATA
COPY!
RAW
DATA
PROCESS
JPEG
PROCESS M e m o r y
DATA
Save as JPEG file
DATA
Camera
Interface
MEMORY

15. SoC camera subsystem
How to capture still shots in SoC camera subsystem
No JPEG encoder in your camera module
User Application
ARM
core
COPY!
RAW
DATA M e m o r y
DATA
Codec S/W
Camera
Interface
DATA
Camera
Module
Save as
JPEG file

16. SoC camera subsystem COPY! COPY!
How to take motion pictures in SoC camera subsystem
COPY!
Frame buffer
ARM
core
User Application M e m o r y OK
Cancel
DATA
Camera
Interface
Codec S/W
Save as…
DATA
Camera
Module
COPY!

17. What we’ve got in Linux to control camera
In Kernel
V4L2 (AKA Video For Linux 2)
The second version of the Video For Linux API
Kernel interface for analog radio and video capture and output drivers
In user space
Multimedia middleware
Gstreamer
Pipeline based multimedia framework written in the C programming language with the type system based on GObject
OpenMAX
Cross-platform set of C-language programming interfaces that provides abstractions for routines especially useful for audio, video, and still images

18. Mobile camera module devices
Is V4L2 enough for mobile digital camera modules?
No way

19. Framework for digital camera

20. Video4Linux2 and camera device
Camera subsystem in Linux kernel aspect
videobuf
videodev
V4L2
V4l2-dev
V4l2-ioctl
V4l2-int-device
(old)
V4l2-subdev
(new)
videobuf-core
videobuf-sg-dma
V4l2-device
Camera interface
V4L2 driver
Command
interface
Resolution handling
Data format handling
(YUV/RAW/JPEG…)
Buffer
handling
Camera module
V4L2 driver
Exposure
control
Lens
control
White balance control
Effect control
Face
detect
Zoom
control
Strobe
control .

21. Abstract view of V4L2 working with video capture device
How does it work with V4L2?
Application or Middleware
Open device
Copy memory to userspace
VIDIOC_QUERYCAP
VIDIOC_S_INPUT
VIDIOC_REQBUF
VIDIOC_QBUF
VIDIOC_QBUF
VIDIOC_DQBUF
VIDIOC_QUERYBUF
VIDIOC_STREAMON

22. New V4L2 APIs for digital camera Exposure control

23. New V4L2 APIs for exposure control
Need for detailed control in exposure
We’ve got camera devices with mechanical shutter
Iris could be handled also
ISO controlled output is somehow useful
Why not using enhanced exposure control like a regular digital camera?

24. New V4L2 APIs for exposure control
What we’ve got in mainline V4L2 right now?
#define V4L2_CID_EXPOSURE_AUTO (V4L2_CID_CAMERA_CLASS_BASE+1)
enum  v4l2_exposure_auto_type {
        V4L2_EXPOSURE_AUTO = 0,
        V4L2_EXPOSURE_MANUAL = 1,
        V4L2_EXPOSURE_SHUTTER_PRIORITY = 2,
        V4L2_EXPOSURE_APERTURE_PRIORITY = 3 };
#define V4L2_CID_EXPOSURE_ABSOLUTE  (V4L2_CID_CAMERA_CLASS_BASE+2)
#define V4L2_CID_EXPOSURE_AUTO_PRIORITY (V4L2_CID_CAMERA_CLASS_BASE+3)
No way to control iris
No way to control exposure metering method

25. New V4L2 APIs for exposure control
Additional exposure control
enum v4l2_exposure_auto_type {
V4L2_EXPOSURE_AUTO = 0,
V4L2_EXPOSURE_MANUAL = 1,
V4L2_EXPOSURE_SHUTTER_PRIORITY = 2, V4L2_EXPOSURE_APERTURE_PRIORITY = 3,
+ /* Additional features for digital camera */
+ V4L2_EXPOSURE_ISO_PRIORITY =4, };
#define V4L2_CID_EXPOSURE_ABSOLUTE (V4L2_CID_CAMERA_CLASS_BASE+2)
#define V4L2_CID_EXPOSURE_AUTO_PRIORITY (V4L2_CID_CAMERA_CLASS_BASE+3)
. . .
+#define V4L2_CID_EXPOSURE_LOCK (V4L2_CID_CAMERA_CLASS_BASE+18)

26. New V4L2 APIs for exposure control
+#define V4L2_CID_CAM_APERTURE(V4L2_CID_CAMERA_CLASS_BASE+19)
+#define V4L2_CID_CAM_SHUTTER (V4L2_CID_CAMERA_CLASS_BASE+20)
+#define V4L2_CID_CAM_ISO (V4L2_CID_CAMERA_CLASS_BASE+21)
Supported aperture stages, shutter speed, and ISO speed could be different between every different camera module products
Supported values for aperture, shutter and ISO should be made in V4L2_CTRL_TYPE_MENU {
    .id = V4L2_CID_CAM_APERTURE,
    .type = V4L2_CTRL_TYPE_MENU,
    .name = "Aperture",
    .minimum = 0,
    .maximum = 4,
    .step = 1,
    .default_value = 0, },
static const char *camera_iris_stages[] = {
/* This module supports 5 Iris stages 
* on it's own but F number depends 
* how you package Lens module */
"2.8“, "4“, "5.6“, "8“, "11“, NULL };

27. New V4L2 APIs for exposure control
New exposure metering control
+/* Exposure Methods */
+#define V4L2_CID_PHOTOMETRY (V4L2_CID_CAMERA_CLASS_BASE+17)
+enum v4l2_photometry_mode {
+ V4L2_PHOTOMETRY_MULTISEG = 0,
+ V4L2_PHOTOMETRY_CWA = 1,
+ V4L2_PHOTOMETRY_SPOT = 2,
+ V4L2_PHOTOMETRY_AFSPOT = 3,
+};

28. New V4L2 APIs for digital camera Lens control

29. New V4L2 APIs for lens control
Need for enhanced Focus mode API
Current V4L2 framework just slightly covers MANUAL & AUTO FOCUS mode for usb webcam
High-end camera modules support for various focus mode presets like
AF Macro
AF-S / AF-C
For compatibility
Focus control API should be defined

30. New V4L2 APIs for lens control
Focus Mode & Enumeration values
#define V4L2_CID_FOCUS_ABSOLUTE (V4L2_CID_CAMERA_CLASS_BASE+10)
#define V4L2_CID_FOCUS_RELATIVE (V4L2_CID_CAMERA_CLASS_BASE+11)
+#define V4L2_CID_FOCUS_MODE (V4L2_CID_CAMERA_CLASS_BASE+13)
+/* Focus Methods */
+enum v4l2_focus_mode {
+ V4L2_FOCUS_AUTO = 0,
+ V4L2_FOCUS_MANUAL = 1,
+ V4L2_FOCUS_MACRO = 2,
+ V4L2_FOCUS_CONTINUOUS = 3,
+};

31. New V4L2 APIs for digital camera Object recognition

32. What is object detection?
Gee!
It recognized my face!
Hello everyone

33. What should we consider?
What do we set?
Detect mode
Type of object
How many objects to detect
Triggered action
What do we get?
How many objects are detected
Detected object’s type
Detected object’s coordinates

34. New V4L2 APIs for object recognition
New API for object recognition
New capability
#define V4L2_CAP_ASYNCIO 0x /* async I/O */
#define V4L2_CAP_STREAMING 0x /* streaming I/O ioctls */
+#define V4L2_CAP_OBJ_RECOGNITION 0x
New IOCTL
+#define VIDIOC_S_RECOGNITION _IOWR ('V', 85, struct v4l2_detect)
+#define VIDIOC_G_RECOGNITION _IOR ('V', 86, struct v4l2_detect)

35. New V4L2 APIs for object recognition
New API for object detection
+/* Object detection and triggered actions */
+enum v4l2_recog_mode {
+ V4L2_RECOGNITION_MODE_OFF = 0,
+ V4L2_RECOGNITION_MODE_ON = 1,
+ V4L2_RECOGNITION_MODE_LOCK = 2,
+}; +
+enum v4l2_recog_action {
+ V4L2_RECOG_ACTION_NONE = 0,/* only detection */
+ V4L2_RECOG_ACTION_BLINK = 1,/* Capture on blinking */
+ V4L2_RECOG_ACTION_SMILE = 2,/* Capture on smiling */
+enum v4l2_recog_pattern {
+ V4L2_RECOG_PATTERN_FACE = 0; /* Face */
+ V4L2_RECOG_PATTERN_HUMAN = 1; /* Human */
+ V4L2_RECOG_PATTERN_CHAR = 2; /* Character */ +}

36. New V4L2 APIs for object recognition
New API for object detection
+struct v4l2_recog_rect {
+ enum v4l2_recog_pattern p; /* What pattern detected */
+ struct v4l2_rect o; /* detected area */
+ __u32 reserved[4]; +} +
+struct v4l2_recognition_data {
+ __u32 detect_cnt; /* detected object counter */
+ struct v4l2_recog_rect obj; /* detected area */
+};
+struct v4l2_recognition {
+ enum v4l2_recog_mode mode;
+ enum v4l2_recog_pattern pattern; /* What pattern to detect */
+ __u32 obj_num; /* How many object to detect */
+ __u32 detect_idx; /* select detected object */
+ struct v4l2_recog_data data; /* read only */
+ enum v4l2_recognition_action action;

37. What should we consider?
What do we set?
Detect mode
How many objects to detect
Triggered action
What do we get?
How many objects are detected
Detected object’s coordinates

38. New V4L2 APIs for object recognition
New API for object recognition
New capability
#define V4L2_CAP_ASYNCIO 0x /* async I/O */
#define V4L2_CAP_STREAMING 0x /* streaming I/O ioctls */
+#define V4L2_CAP_OBJ_RECOGNITION 0x
New IOCTL
+#define VIDIOC_S_RECOGNITION _IOWR ('V', 85, struct v4l2_detect)
+#define VIDIOC_G_RECOGNITION _IOR ('V', 86, struct v4l2_detect)

39. New V4L2 APIs for object recognition
New API for object detection
+/* Object detection and triggered actions */
+enum v4l2_recognition_mode {
+ V4L2_RECOGNITION_MODE_OFF = 0,
+ V4L2_RECOGNITION_MODE_ON = 1,
+ V4L2_RECOGNITION_MODE_LOCK = 2,
+}; +
+enum v4l2_recognition_action {
+ V4L2_RECOG_ACTION_NONE = 0,/* only detection */
+ V4L2_RECOG_ACTION_BLINK = 1,/* Capture on blinking */
+ V4L2_RECOG_ACTION_SMILE = 2,/* Capture on smiling */
+struct v4l2_recognition_data {
+ __u8 detect_cnt; /* detected object counter */
+ struct v4l2_rect o; /* detected area */
+struct v4l2_recognition {
+ enum v4l2_recognition_mode mode;
+ __u8 obj_num; /* How many object to detect */
+ __u8 detect_idx; /* select detected object */
+ struct v4l2_recognition_data data; /* read only */
+ enum v4l2_recognition_action action;

40. New V4L2 APIs for digital camera Dual camera

41. Dual camera for 3G handset
What is dual camera?
Dual camera applications
Video telephony
Camera applications
Taking self portrait shots
Flip side camera
: for still shots
Front side camera
: for special purpose

42. Dual camera for 3G handset
H/W restrictions in Application Processor’s camera peripheral interface
Mega
camera
VGA
Data
path

43. Why should we make single device node?
H/W restrictions in Application Processor’s camera peripheral interface
“SINGLE” camera interface
on Application Processor !
Only one camera could be handled at a time
Only one camera could be using data pins at a time
Each camera is different in
Working MCLK
Working resolution (SYNC)
Working PCLK

44. What do we have in V4L2? We already have API for dual camera
Enumerate video inputs
 Query or select the current video input
VIDIOC_ENUMINPUT
VIDIOC_G_INPUT
VIDIOC_S_INPUT

45. What do we need to consider for dual camera?
Let user space application get noticed about camera input devices
Don’t forget VIDIOC_ENUMINPUT
Check out supported pixel format using VIDIOC_ENUM_FMT and VIDIOC_ENUM_FRAMESIZES
Context of each camera module
Capability : Supported color spaces, resolutions, additional functions…
Latest working status : resolution, color space, effect and so on
Context of camera interface
Working information of the camera module getting switched : Clock information and so on

46. Example of switching camera
How to switch active camera
Mega camera
VGA camera
Mega
Enable
VGA
Enable
Condition 1
How to enable camera module:
Give power source
Enable expecting camera
Give MCLK
Mega
Reset
VGA
Reset
CAM
POWER
MCLK
DATA PINS
CPU
Condition 2
Shared things
Camera power enable pin
Camera data pins
MCLK
Condition 3
Dedicated things
Camera enable pins
Camera reset pins

47. Abstract work flow of switching camera
power off
Camera (A) enable pin off
Camera (B) Enable pin on
Cam
power on
Switch to camera (B)
STREAMON
STREAMOFF
Camera (A)
initialize
Start preview
Stop preview
G_INPUT/
S_INPUT
Camera (B) initialize
Start preview
Camera interface reset & setting
Stop preview
Stop DMA
Reset camera interface’s current setting
Make a proper MCLK for camera (B)
Setup resolution, color-space, DMA

48. Porting issues in SoC camera subsystem

49. I2C issue I2C protocol has start and stop condition
Some camera modules generate unexpected noise until their power up sequence
Makes I2C bus in bus busy state

50. I2C issue OR OR How to prevent this malfunctioning issue
Use a “noise free on power up” camera module OR
Use a level shifter to block and ignore noise from camera module OR
Change functionality of I2C pins while turning on camera module

51. Resolution issue All about between camera interface and camera module
Expected resolution from camera interface
Served resolution from camera module
Symptoms depend on camera interface
Expecting resolution is bigger than served resolution
A “select time out” error in user space application or
Preview image is floating in some direction
Serving resolution is bigger than expected resolution
Preview angle looks like to be zoomed in

52. Color space issue
Color order mismatched between camera interface and camera module

53. Lens control issue Preview freezing while checking lens response
Using while() loop while it checks response, preview will freeze until focusing job gets finished
If the ISP driver use the kernel thread when it check response from current lens status could make smooth preview on the LCD while lens is on focusing.

54. Lens control issue Solving preview freezing while checking lens status
struct platform_isp {
const struct platform_data *pdata;
struct v4l2_int_device *v4l2_int_device;
struct i2c_client *i2c_client;
int reschk; }
static int ioctl_s_ctrl(struct v4l2_int_device *s,
struct v4l2_control *vc) {
struct platform_isp *isp = s->priv;
struct i2c_client *client = isp->i2c_client;
switch (vc->id) {
case V4L2_CID_FOCUS_AUTO:
isp->vfm = vc->value;
if (vc->value == V4L2_CID_FOCUS_MACRO) {
err = i2c_command_for_isp( client, focus_macro );
isp->reschk = 1;
break;
If isp->reschk is Set,
The kernel thread is active.

55. Lens control issues
Solving preview freezing while checking lens status
int pseudo_reschk_thread (void *data) {
int err = 0;
………
do {
switch (isp->reschk) {
case 1:
err = i2c_command_for_isp(client, focus_check);
if (err < 0) {
isp->reschk = 1; /* i2c command function failed. */
} else {
isp->reschk = 0; /* if succeed, reschk ended. */ }
break; ……
} while(1);
return 0;
This function return when the ISP is
matched the focus(+ return) or timeout(- return).
This is able to re-check to the next check time.
Platform Data Vairable for isp, or
static variable.

56. JPEG data synchronization issues
JPEG has no synchronization protocol when it gets transferred
All about camera interface and camera module’s data synchronization
When camera interface supports JPEG sync
Follow user manual and make camera module to follow that protocol
When camera interface does not support JPEG sync
Make synchronization protocol: make JPEG data to be synchronized with VSYNC

57. Q&A

58. Thank you