Cinematography in Augmented Reality Richard Shemaka

1.Background on Camera and Cinematography 2.Introduction to AR 3.Camera in AR 4.Two Approaches to Cinematography in AR Exploring the notion of camera and presentation in augmented reality

Camera and Cinematography Ever since the advent of film, the notion of camera has become an import tool for media makers to convey their ideas The moving photographic image gave rise to a new language of emotion and meaning-making: cinematography Some larger considerations of cinematography include lighting, camera movement, composition (framing, blocking, etc), and special effects

Camera in Entertainment Media While traditional cinematography is concerned with the photographic image, we can derive important aspects with respect to any moving image The lessons learned from photography and cinematography have been applied to great affect in new media such as websites and video games Uncharted 2: Among Thieves Naughty Dog

Augmented Reality An emerging media which blends reality with computer images First appeared in the early 1990’s In recent years, advances in mobile technologies have allowed for more wide- spread development

History of Camera in AR Wearable AR systems focused on first-person presentation of material Focus was more on the presentation of information – Ease-of-use and transparency have been the goals of many AR camera systems

Camera in AR Most AR cameras rely on user controlled position and orientation This alone renders many of the traditional techniques of cinematography irrelevant – E.g. calculated camera movement

Leveraging Cinematography in AR As with most new forms of media, we can bring our knowledge of older media to bear when designing new experiences (remediation) Recently, video games have started to integrate cinematic camerawork with interactive controls Because most interactive AR applications rely on a user-controller camera, new techniques and approaches will be needed

Lighting Lighting is a powerful cinematic tool that could be easily adapted to AR Both scene lighting and interactive light-based mechanics could contribute to a scene’s presentation

Framing and Composition Framing and composition in film are generally accomplished with careful camera placement I am proposing two separate approaches to accomplishing these effects in a typical handheld augmented reality interface – Guiding the user’s gaze within the space – Adjusting the scene according to the user’s view

Two Separate Approaches Focus of Attention The user’s attention is directed towards particular parts of the scene Borrows directly from film with techniques such as focus, action, framing, etc. Camera-Aware Scenes The scene recognizes and adapts to the way it is being presented Props and characters change themselves to better ‘fit the shot’

Focus of Attention Loaded Scenes – In this approach (originally taken from stage), the scene is constructed in such a way to lead the user towards a particular view – Borrows heavily from composition Action – Actors and events direct the user’s attention towards something particular

Camera-Aware Scenes Actors’ positioning and pose – Having the actors in a shot change to better present themselves Shifting scene – The scene shapes itself based on the way it is viewed

Implementation In order to explore these designs in practice, I plan to build a suite of AR prototypes which focus on cinematic presentation With the nature of prototyping, I will not be going into these with any expectation beyond leveraging the designs I am outlining for the project

Platform Qualcomm’s AR SDK – Natural feature tracking – API for some AR-specific mechanics Unity3D for Android – Game engine + IDE – Integrated with Qualcomm’s SDK