1. Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research http://research.microsoft.com/users/lhe

2. Motivation u There are 3 elements in computer graphics –lights, scene objects, and –the camera (focus of this talk) u Camera control is hard (7 DOFs) –position (3), direction (3), field of view (1) u We may learn from cinematography

3. Roles in cinematography u Low-level responsibilities –Cameraman Position and move the cameraPosition and move the camera –Film editor Keep the film to proper lengthKeep the film to proper length Decide shot transitionsDecide shot transitions u High-level responsibilities –Script writer and director Story-tellingStory-telling

4. Camera control level 1 u Input: camera position and direction u Output: camera transformation u Application: low-level graphics library routines (Direct3D)

5. Camera control level 2 u Input: show both A and B, follow C, etc. u Output: camera position and direction u Application: 3D game, computer animation package

6. Camera control level 3 u Input: high-level user directions –show a conversation –show a car chase scene u Output: a sequence of level 2 camera spec u Applications: graphical chat, tele- conferencing, Virtual Reality games

7. Camera control level 3 (cont.) u Encodes cinematographic expertise –an interesting problem in itself u Provides an interface that is –real-time –visually entertaining –more informative

9. Eyepeep BARG07011 Feldegast Microsoft V-Chat Lunar Island Eyepeep says "really" Feldegast says "yup" BARG07011 has joined the conversation Eyepeep says "im still older" Feldegast says "lol" DGeste309 does something silly. BARG07011 DGerste309 Eyepeep Feldegast GrenDayGod

10. Principles of Cinematography

11. Film structure Film...... Scenes Shots

12. Camera distance Close up Close shot Medium shot Full shot Long shot

13. The line of interest AB

14. External camera AB

15. AB

16. Internal camera AB

17. Apex camera AB

18. Moving cameras AB pan(A) track(B)

19. Some rules in film editing u Don't cross the line of interest u u Avoid jump cuts u u Let the actor lead u u Break movement

20. Don’t cross the line of interest 1 2

25. Some rules in film editing u u Don't cross the line of interest u Avoid jump cuts u u Let the actor lead u u Break movement

26. Avoid jump cut

27. Some rules in film editing u u Don't cross the line of interest u Avoid jump cuts u u Let the actor lead u u Break movement

28. Two-person conversation AB 1 1 2 2 3 3 4 4

29. 2 3 1 ABAB A 4 B

30. Three-person conversation 2 2 4 4 3 3 1 1 B A C

31. 4 3 2 AB C AB 1 AB C

34. Bad

35. Good

36. Three spaces The universe space

37. Three spaces The eye space (z-axis is the look at direction)

38. Three spaces The screen space (screen is ctan(fov/2) away from eye point)

39. Standard look-at transformation Given eye position P from, a look at direction Want a rotation matrix R and a translation to transform a vector from universe space to eye space

40. Standard look-at transformation But this is not unique

41. Standard look-at transformation But this is not unique

42. Standard look-at transformation But this is not unique

43. Standard look-at transformation We will need to specify an up vector, usually [0,1,0]

44. Standard look-at transformation In eye space Z-axis is X-axis is Y-axis is Rotation matrix Translation is -P from

45. Internal close-up of actor A

46. Look at [x at,y at ] transformation Given eye position P from, look at point P at, and P at in screen space [x at, y at ] Want In universe space

47. Look at [x at,y at ] transformation In screen space: H = [x at, y at, ctan(fov/2)] Solve from the following equations:

48. External of actor A and actor B

49. Look at [x A,y A ] and [x B,y B ] Given P A,P B in universe space and eye space, and distance of eye position P from and P a. Want P from and

50. Use numerical method: 1.Set P from to 0 2.Solve R using the Look at [x at,y at ] method 3.Transform E by inverse of R, getting a new P from approximation 4.Goto 2 Look at [x A,y A ] and [x B,y B ]

51. Medium panning shot of actor A

52. References u Jim Blinn’s CG&A ’88 article –Where am I? What am I looking at? u Steve Drucker’s Interactive 3D ’92, ’94, ‘95 papers –Cast camera positioning as an optimization problem

53. System Design

54. Overall system diagram Real-time Application Virtual Cinematographer Renderer

55. At each time step 1. Application generates events to VC –( subject, verb, object ) 2. VC determines – –camera specifications – –acting hints 3. Renderer outputs the image Real-time Application Virtual Cinematographer Renderer

56. At each time step 1. Application generates events to VC – –( subject, verb, object ) 2. VC determines –camera specifications –acting hints 3. Renderer outputs the image Real-time Application Virtual Cinematographer Renderer

57. At each time step 1. Application generates events to VC – –( subject, verb, object ) 2. VC determines – –camera specifications – –acting hints 3. Renderer outputs the image Real-time Application Virtual Cinematographer Renderer

58. A networked virtual party game

59. Actors are simulated Actor’s mind Loneliness Thirst Boredom u Walk, converse, look around, drink, etc. u Can be controlled by the users

60. Networked virtual party game Server Client User actions Events

61. Inside the Virtual Cinematographer

62. VC architecture Idioms Camera modules

63. VC architecture Idioms Camera modules

64. u Geometric placement of specific cameras for each shot u u Choose the side of the line of interest u u Influence acting

65. Camera module: ext(B,A) AB

66. Static camera modules AB ext(B,A)ext(A,B) apex(A,B) int(B)

67. Moving camera modules AB pan(A) track(B)

68. Camera modules u u Geometric placement of specific cameras for each shot u Choose the side of the line of interest u u Influence acting

69. Don’t cross the line of interest 1 2

70. Camera modules u u Geometric placement of specific cameras for each shot u u Choose the side of the line of interest u Influence acting

71. Acting hints ABA’B’

72. VC architecture Idioms Camera modules

73. Previous works u Dave Christianson’s AAAI ’96 paper –Declarative camera control for automatic cinematography –An off-line algorithm –Need to know all events ahead –Use plan algorithms in AI

74. Film idioms u Capture a particular type of scene u Register relevant events u Select shot types u Determine the transitions between shots

75. Hierarchical idiom structure Lookaround Master Converse Drinking Moving 3Talk 2Talk

76. Hierarchical idiom structure u Avoids exponential growth of states u Allows expertise to be reused u Provides robustness

77. Moving idiom track(C)pan(A) ext(A,B) ext(B,A)apex(A,B)

78. Moving idiom track(C)pan(A) ext(A,B)ext(B,A) apex(A,B) T > 8

79. Moving idiom track(C) pan(A) ext(A,B)ext(B,A)apex(A,B) T > 8

80. Moving idiom track(C)pan(A)ext(A,B)ext(B,A) apex(A,B) D < 10

81. Moving idiom track(C)pan(A) ext(A,B) ext(B,A)apex(A,B) D < 5

82. Moving idiom apex(A,B) D < 5 apex(A,B) pan(A) ext(B,A) T > 8 ext(A,B) D < 10 T > 8

84. 3Talk idiom int(C) ext1to2 2Talk(A,B)

85. Invoking a sub-idiom int(C) ext1to2 2Talk(A,B) ext(B,A) ext(A,B) 2Talk

86. Contributions u Real-time camera control –lightweight –automatic camera placement –automatic shot transitions u Do a reasonable job –not to replace human in computer animation

87. Work in Progress

88. Camera control for chat

90. Camera control for live meeting

91. Image after warping

92. Camera control for classroom u Multiple cameras per classroom –Lecturer cameras –Audience camera u Use audio and vision techniques to –Track the lecturer –Activate the audience camera

93. Dramatic lighting

94. Virtual Cinematography Theory and Practice for Automatic Real- Time Camera Control and Directing Liwei He Microsoft Research http://research.microsoft.com/users/lhe