ASIS&T 2010 Building a Collaborative Testbed for Very High-Quality Media Applications Using Very High-Speed Networks April 9, 2010 Laurin Herr President, Pacific Interface Inc.

What is CineGrid? Formed 2004 – non-profit international membership organization Mission – to build an interdisciplinary community Focus – the research, development & demonstration of networked collaborative tools To enable – the production, use, preservation & exchange of very high-quality digital media over high-speed photonic networks Members – media arts schools, research universities, scientific labs, post-production facilities & hardware and software developers around the world Connected via 1 G Ethernet & 10 G Ethernet networks For – research & education

Convergence Motivates CineGrid Media technology historically driven by 3 sectors Entertainment, media, art and culture Science, medicine, education and research Military, intelligence, security and police Adoption of digital media means all 3 face converging needs Fast networking for distributed applications Access to shared devices Specialized computers and massive storage Collaboration tools for distributed teams Robust security for intellectual property Higher quality sound and picture A next generation of trained professionals

CineGrid: A Scalable Approach 4K x 24 2K x 24 HD 2 x 24/25/30 HDTV x 24/25/30/60 HDV x 24/25/30/60 4K 2 x 24/30/60 2K 2 x 24 8K x 60 Consumer HD HDTV Stereo HD Digital Cinema Stereo 4K (future) UHDTV (far future) Mbps 20 Mbps Gbps 200 Mbps - 3 Gbps 250 Mbps Gbps 500 Mbps Gbps Gbps More SHD x 24/30/60 Tiled Displays Camera Arrays SHD (Quad HD) 250 Mbs - 6 Gbps

UHDTV(8K) × Picture Height 1.5 × Picture Height 3.0 × Picture Height UHDTV(4K) HDTV (2K ) º 30º Visual acuity=1.0=20/20 Standard viewing distance 60º Yutaka TANAKA SHARP CORPORATION Advanced Image Research Laboratories Why is more resolution is better? 1. More Resolution Allows Closer Viewing of Larger Image 2. Closer Viewing of Larger Image Increases Viewing Angle 3. Increased Viewing Angle Produces Stronger Emotional Response

Saint Issacs cathedral in St. Petersburg, Russia Original art is 3 meters tall by 1 meter wide Original photograph shot from about 3 meters distance. Camera resolution of 12 Mpixels (4K x 3K)

Original shot from about 1 meters distance

Original shot from about 0.3 meters distance

1981 Francis Ford Coppola with Dr. Takashi Fujio First Look at Electronic Cinema

2001 NTT Network Innovations Laboratory First Look at 4K Digital Cinema

2004 OptIPuter Vision for the Next Decade Terabits/sec Source: Jason Leigh, EVL Augmented Reality No Glasses 4K Streaming Video Gigapixel Wall Paper 1 GigaPixel x 3 bytes/pixel x 8 bits/byte x 30 frames/sec ~ 1 Terabit/sec!

CineGrid projects run over the Global Lambda Integrated Facility (GLIF) backbone 2008 GLIF Visualization by Bob Patterson, NCSA/UIUC In California, CineGrid access primarily via CENIC

Learning by Doing Early CineGrid Projects iGrid 2005 AES 2006 GLIF Holland Festival 2007

CineGrid Exchange TERABYTES PILING UP. To store & distribute its own collection of digital media assets. Members access materials for experiments and demonstrations. THE DIGITAL DILEMMA. Report published by AMPAS – lessons learned by NDIPP and NARA + pioneering research at Stanford (LOCKSS) & UCSD (SRB and iRODS) GLOBAL SCALE TESTBED = distributed storage + fast networks + high quality digital media assets. Explore strategic issues in digital media storage, access, distribution and preservation.

Movie-making is going digital and going global. Local talent is key! Post-production is almost all digital already. Increasingly distributed! Many movies still shot using 35mm film cameras, then scanned as the first step in the digital intermediate process, but digital cameras are getting better and better! Digital cinema projection technology is enabling better picture quality: From 2K digital to 4K digital, plus a new generation of 3D stereo movies… Avatar ! Trends in Cinema

Cinema sound quality is improving. From stereo to compressed 5.1, to uncompressed 5.1, to uncompressed 7.1 surround sound! Of the roughly 100,000 screens worldwide, only 15% are digital today. But recent financing announcements in the USA lead to forecasts that 60-70% of American screens will be digital by 2013! Fast networks are becoming critical infrastructure for media companies involved in production, post-production and distribution! Education of the next generation of media professional is critical! Trends in Cinema

Cinema Archives vs. Libraries Archive Goal: preservation without errors, access without end Access is WORSE case: Write Once Read Seldom if Ever Long-term insurance for expensive intellectual property Mixed motivations Potential for future revenue Corporate asset protection Cultural record Library Goal: customer service, active management, temporary storage Access is BEST case: Best Effort Sells Titles Mixed inventory: film, video, digital in many formats Increasing going online/near-line quicker search of inventory faster time-to-market lower cost of replication & distribution

The Digital Dilemma of Cinema 35mm film is the quality standard for motion pictures Properly stored lasts years Survives benign neglect gracefully Directly readable without reliance on software stack Very cost effective storage media 4K digital cinema needed to match 35mm analog film quality More than 8 TB per master version of 2 hour theatrical movie More than 5 million objects, ~ 2 PB total all elements per production Hollywood produce approx. 400 motion pictures per year Hollywood archives are privately funded for future profit Digital cinema preservation relying on endless migration is less reliable and much more expensive (TCO) than film preservation

CineGrid Exchange 2008 Geographically Distributed Repositories + Fast Networks San UCSD/Calit2 40 TB with 10 Gbps connectivity UvA 30 TB with 10 Gbps connectivity Keio/DMC 6 TB with 10 Gbps connectivity Total storage = 76 TB High-quality digital media assets: 4K, 2K, HD, mono & stereo, still & motion pictures + audio

CineGrid Exchange 2009 Building on Basic Concept Receive seed funding from AMPAS to start building multi-layer open source asset management & user access framework for distributed digital media repository Establish Working Group Bi-annual workshops Weekly conference calls Basecamp for online coordination Write CineGrid Exchange functional requirements document Expand CineGrid Exchange storage capacity with more nodes Diversify to reduce risk!

CineGrid Exchange 2009 Phase 1 Development Goals Define basic workflows for ingest, preservation, distribution, & deletion Implement basic workflows to automate management of collection Expand distributed repository: more nodes, more assets in collection Deploy consistently at all CineGrid Exchange nodes Enable easier large data transfers between nodes Prepare for integration of cataloging application + distributed repository Define and validate CineGrid Exchange metadata schema

CineGrid Exchange Development Multi-layer Open Software Stack User Interface & Access Open Source Application for Collections Management & On-Line Access Open Source Digital Repository Interface Open Source Middleware for Rule-based Management of Distributed Digital Repository Resources & Asset Testbed Infrastructure of Distributed Storage and Network Links iRODS by DICE For CineGrid By Calit2 Collective Access Extended by Whirl-I-Gig for AMPAS CineGrid Exchange Access Portal For CineGrid by CineGrid Resource Description Framework For GLIF and CineGrid by UvA

Programmable micro-services to implement management policies as non-ambiguous rules for automated operations across distributed storage repository Transparent management of CX files in a distributed repository Dedicated CX storage Non-dedicated CX storage Centralized catalog (iCAT) File transfer using parallel TCP & UDP Suitable for experimenting with digital media preservation strategies CineGrid Exchange (CX) MiddleWare iRODS (Integrated Rules Oriented Data Services)

CineGrid Exchange 2010 Build multi-layer open-source framework for distributed digital media repository Refine iRODS middleware rules for robustness Integrate iRODS + Collective Access CX Node SiteStorage TypeCX Allocation UCSD/Calit2, San Diego, USASun Thumper (x4540)66 TB UvA, Amsterdam, NetherlandsSun Thumper (x4540)30 TB UIC/EVL, Chicago, USARAID Array10 TB Keio U./DMC, Tokyo, JapanRAID Array8 TB CESNET, Prague, Czech Republic Sun Thumper (x4540)48 TB Ryerson U, Toronto, CanadaSun Thumper (x4540)57 TB AMPAS, Los Angeles, USASun Thumper (x4540)24 TB Total CineGrid Exchange Capacity243 TB

Acknowledgements Academy of Motion Picture Arts and Sciences, STC Naval Postgraduate School, MOVES Institute University of California San Diego, Calit2 University of Illinois Chicago, EVL University of Washington, Research Channel Ryerson University, Rogers Communications Ctr. CESNET NTT Network Innovation Laboratory Keio University, DMC University of Amsterdam Pacific Interface DICE at UNC & SDSC

CineGrid International Workshop December 13-15, UCSD