Presentation on theme: "Preservation-Worthy Digital Video: or, How to Drive your Library into Chapter 11 Jerome McDonough New York University February 6, 2014."— Presentation transcript:
Preservation-Worthy Digital Video: or, How to Drive your Library into Chapter 11 Jerome McDonough New York University February 6, 2014
Digital Video Basics A video signal consists of luminance and chrominance information Luminance – brightness, varying from white to black (abbreviated as Y) Chrominance – color (hue & saturation), conveyed as a pair of color difference signals: R-Y (hue & saturation for red, without luminance) B-Y (hue & saturation for blue, without luminance)
Digital Video Basics Wheres the green? Spectral response of the human eye peaks in the green frequencies. The perceived brightness of an item can be constructed using weighted values for its red, green and blue components: Y = 0.299R + 0.587G +0.114B, or
Digital Video Basics 4:2:2 sampling4:2:0 sampling 4:1:1 sampling 4:2:2 – High End DV (Digital Betacam, DVCPro50) 4:2:0 – MPEG 1 & 2 4:1:1 – DV and DVCAM
Digital Video Basics Why not 4:4:4 sampling? 720 x 486 resolution = 349,920 pixels per frame 349,920 pixels x 10 bits/sample x 3 samples/pixel = 10,497,600 bits per frame 10,497,600 bits/frame X 29.97 frames/second = 314,613,072 bits per second 314,613,072 bps x 3600 seconds = ~141.58 GB/hour For 1920x1080 HDTV, more like 840 GB/hour 4:2:2 sampling drops that rate by a third with almost no perceptible difference in quality. 4:2:0 and 4:1:1 drop it in half.
Digital Video Basics MPEG 2 Compression Further subsampling down sample to 8 bits/sample down sample to 4:2:0 sampling regime Discrete Cosine Transformation + Requantizing of coefficients from DCT Variable Length Encoding & Run Length Encoding Interframe compression (motion compensation) all of which can take a 209 mbps video rate (for 4:2:2 video) and reduce it to around 8 mbps with no apparent visual loss.
Digital Video Basics Raw digital video is extremely storage and bandwidth intensive. As a result, almost all digital video processing systems employ a mix of lossless and lossy compression mechanisms.
Preservation-Worthy Digital Video Desired characteristics for digital video we feel we can preserve include: Content can be migrated to new formats and new media without introducing artifacts Stored in non-proprietary, standard format which is openly documented Easy to produce derivatives for end-user distribution Minimize costs of production, distribution & migration
Sampling, Migration & Artifacts As in still image digitization, employing lossy compression can lead to artifacting when you migrate. Unlike still images, lossy compression is assumed in almost all video processing technology today.
One River Media Codec Test Image Images courtesy of Marco Solorio, One River Media
Black Magic 8-Bit 4:2:2 Uncompressed Codec Images courtesy of Marco Solorio, One River Media
Black Magic 8-Bit: 10 th Generation Images courtesy of Marco Solorio, One River Media
One River Codec Test Images courtesy of Marco Solorio, One River Media
Digital Voodoo 10-bit Codec Images courtesy of Marco Solorio, One River Media
Digital Voodoo 10-bit: 10 th Gen. Images courtesy of Marco Solorio, One River Media
One River Codec Test Images courtesy of Marco Solorio, One River Media
Apple 4:4:4 None: 10 th Gen. Images courtesy of Marco Solorio, One River Media
Sampling, Migration & Artifacts 4:4:4 sampling fulfills the digital promise of perfect copies across generations, but Most video equipment doesnt actually support it. Most high-end video editing packages on computers do and will store 4:4:4 to disk. Lesson: if you want to store 4:4:4 uncompressed video, prepare to buy a lot of disk (or HSM), and abandon videotape.
Storing 4:4:4 Uncompressed Video QuickTime – Proprietary, but publicly documented and does the task, and software support is available MJPEG 2000 – Open Standard (ISO/IEC 15444), supports 4:4:4 uncompressed. Software support iffy, but growing. Material Exchange Format – Open Standard, but software support is weak, and some vendor issues
Storing 4:4:4 Uncompressed Video Videotape is a non-starter. D1 tape decks for uncompressed video cost $200K, and use 4:2:2. Disk vs. HSM Tape Cost vs. Speed Opportunities to detect bit rot, ability to migrate, time to produce derivatives Waiting on grid storage….
Mind Games I: Conversion NYU has approximately 30,000 hours of moving image material, undigitized, in its special collections. Lets digitize 1/10 of that. 9 Digitization/Editing workstations: $1,125,000 9 conversion staff full time for 1 year: $350,000 425 TB of Disk Storage @ $10k/TB = $4,250,000 Grand Total: $5,725,000 FYI, according to ARL, thats about half our entire 2002 materials budget
Mind Games II: On-going costs Assume migration every 10 years. Assume time to migrate = 2x time of source material. 6,000 hours x staff salary = ~$120K / 10 = $12k/year Assume new derivatives every 5 years, and time to migrate = 2x time of source material 12,000 hours x staff salary = ~$240k / 10 = $24k/year 3% disk loss/year x 425 TB = 12.75 TB replaced/year. Assuming disk prices are halved every two years, for next 10 years wed have $38,750 total replacement costs, or $3,875/year. Grand Total: ~$40K/year maintenance costs
Mind Games III On-going maintenance costs for 3,000 hours of video on disk arent particularly bad. Initial conversion costs, however, are nightmarish. If you dont spend the money, however, your digital video is unlikely to prove any more preservation-worthy than analog.
NYU Production/Storage Chain Capture/editing done on Apple G4 system, producing: Digital Betacam tape (master copy), edited to add initial SMPTE color bars, AES/EBU tones, and NYU titling information DVD for in-house use in library MPEG4 (high bit rate and low bit rate) for streaming delivery (hoping for better client support soon….) Betacam tapes stored off-site, DVDs to library media center, MPEG4 files FTPd to Sun server
NYU Production/Storage Chain Descriptive metadata – For items in special collections, collection-level MARC w/$856 for EAD finding aid with links to items For materials outside special collections, item-level MARC record w/$856 to item Administrative metadata – Looking at ODRL for rights metadata Local SMPTE W25-based technical metadata, but looking to merge local work with LC A/V prototyping work Waiting on OCLC/RLG PREMIS group for preservation Structural Metadata – Damned little until we figure out how to automate production of structural metadata But it will be in METS
NYU Costs: Capture Hardware Complete system cost: ~$125,000.00
NYU Costs: Repository Hardware Sun Enterprise 15K w/L700 Tape Backup: ~$400K/year Sun T3 Disk Arrays (10 TB): ~$100K ($10K/TB)
NYU Costs: Conversion Personnel Currently conversion takes approximately 8 hours for every hour of tape. Minimum conversion cost of ~$120/hour Hope to lower conversion time with practice (and better equipment), but at best, probably around $90/hour of tape.
NYU Costs: Repository Personnel Fractional part of NYU ITS Unix SysAdmin, Network Support Specialist, Tape Backup support, equivalent to about 1 FTE ~$75K/year
NYU Costs: Summary About $475K/year to keep our server alive and happy; $10K to add another terabyte About $125K to add a new video capture/editing workstation About $90-120 per hour of video capture/conversion costs