Michał Kwiatek IT-IS1 Media Archive Status 29/10/2007

Motivation 1.Summarise the past a.DV MOV 2 AVI conversion b.Recent data migration c.FLV external encoder d.Recent configuration changes e.Recent changes of the Media Archiver software f.Test of HD master format 2.Convince you that at present Media Archive can grow without active help from IT-IS 3.Discuss the future a.Space requirements b.WMV playback on CDS c.FLV format on CDS d.HD video master format Michał Kwiatek IT-IS2

Overview NB: No master file is encoded if Media Archiver cannot read its metadata 3Michał Kwiatek IT-IS

1.The Past Michał Kwiatek IT-IS4

DV MOV to AVI conversion How to convert DV MOV files into DV AVI files Impossible on a Mac? Found, bought (~150 CHF) and tested: FireStore DV Conversion Suite Converted 39 files provided It’s been decided to keep both DV MOV and DV AVI files in the archive Handed over to the Video team Michał Kwiatek IT-IS5

After CERNDATA26ACERNDATA26BCERNDATA27ACERNDATA27BCERNDATA28A CERNDATA28 BCERNDATA29ACERNDATA29B capacity free Audio450 Photo/ConversionSettings1 Photo/Masters 550 Photo/Public5 Photo/Public21 Photo/Workspace 605 Software1 Video/Masters/Conferences Video/Masters/Movies Video/Masters/Rush 340 Video/ConversionSettings1 Video/Public237 Video/Public2 622 Video/TestMasters62 Video/TestSlaves5 Video/Workspace 2790 TempVideoWorkspace 1740 On CERNDATA26ACERNDATA26BCERNDATA27ACERNDATA27BCERNDATA28A CERNDATA28 BCERNDATA29ACERNDATA29B capacity free Audio450 Photo/ConversionSettings1 Photo/Masters 550 Photo/Public5 Photo/Public21 Photo/Workspace 605 Software1 Video/Masters/Conferences 2790 Video/Masters/Movies 1862 Video/Masters/Rush 340 Video/ConversionSettings1 Video/Public237 Video/Public2 622 Video/TestMasters62 Video/TestSlaves5 Video/Workspace 2790 TempVideoWorkspace 1740 Video Data Migration Michał Kwiatek IT-IS6 Legend: ok, don't touch can put more data need to free some space moved from to split between video quota owners audio quota owners photo quota owners operations quota owners 2008 growth estimates Video/Masters/Movies2000 Video/Masters/Conferences1000 Video/Masters/Rush1000 Video/Workspace0 TempVideoWorkspace0 Video/Public2200 Total4200

Recent configuration changes New worker node dedicated to photos Retry erroneous slave files after 4 hours instead of 1 day; abandon after 8 days: Michał Kwiatek IT-IS7 Try 4 hours Try 4 hours Try 4 hours Try... 8 days

Recent changes in the Media Archiver software Notify CDS only when all slave files are ready (complex – have to synchronise on master files to be sure of semantic correctness) Implemented a “DFS offline” check Support for external encoders Michał Kwiatek IT-IS8

FLV External Encoder We’ve tested ffmpeg.exe to generate FLV videos: ffmpeg.mplayerhq.hu/download.htmlffmpeg.mplayerhq.hu/download.html Current statistics: 100% OK (the only 5 errors were related to bad master files) IT-UDS has complete control over: –The version of ffmpeg used –How it is invoked –Generation logs Michał Kwiatek IT-IS9

Tests of HD master formats We’ve looked at H.264 MPEG-4 codec (AVC) We’ve decided it is still too early Michał Kwiatek IT-IS10

2.The Present Michał Kwiatek IT-IS11

Formats supported by Media Archiver Master formats VideoAudioPhoto DV AVIWAVEJPEG MPEG-2 DVD-quality Michał Kwiatek IT-IS12 Slave formats VideoAudioPhoto WMVWMAJPEG MP3 ? FLV ?OGG

Formats stored in Media Archive Michał Kwiatek IT-IS13 Master formats VideoAudioPhoto DV AVIWAVEJPEG MPEG-2 DVD-quality TIFF HDV MOVRAW DVCProHD MOV DV MOV Current funtional needs are satisfied

Media Archive interventions More space needed Media Archiver does not work according to specification Michał Kwiatek IT-IS14 Metadata not present or not found Bad master file Specification changes

Current statistics Photo: –ERROR1737 –INSYNC17425 Audio: –INSYNC543 –NEW1 Video: –BUSY3 –ERROR4 –INSYNC563 –MSTRCHD19 Michał Kwiatek IT-IS15

3.The future Michał Kwiatek IT-IS16

Space requirements in 2008 Michał Kwiatek IT-IS17 Video/Masters/Movies2000 Video/Masters/Conferences1000 Video/Masters/Rush1000 Video/Workspace0 TempVideoWorkspace0 Video/Public2200 Total4200

WMV playback on CDS Playback windows should be bigger Higher resolution file should be used Streaming server should be used instead of the HTTP server multimedia/test/webcast/templates/Artificial_Intelligence/tes t/default_CDS_WMV_vs_improved_WMV_4_3.htm multimedia/test/webcast/templates/Artificial_Intelligence/tes t/default_CDS_WMV_vs_improved_WMV_4_3.htm Michał Kwiatek IT-IS18

FLV on CDS Let’s see how it works: multimedia/test/webcast/templates/Artificial_Intell igence/test/improved_WMV_vs_flash_4_3.htm multimedia/test/webcast/templates/Artificial_Intell igence/test/improved_WMV_vs_flash_4_3.htm Implemented as external encoder, no support from IT-IS for the encoder Michał Kwiatek IT-IS19

External Encoder Requirements It supports all current (and future!) master formats It is reliable: –Always works for valid files (and returns 0) –Returns non-zero for invalid files –Does not get stuck Michał Kwiatek IT-IS20

(Minor) problem with ffmpeg Michał Kwiatek IT-IS21

Future master format requirements be supported and easy to read, have good chances to stay supported and easy to read in the future, be recognised by external customers of the archive: television companies (for video files) and newspapers and publishing houses (for photo files), be the format in which the teams work and save their work, be easy to process automatically with the available, well-supported tools Michał Kwiatek IT-IS22

H.264 requirements for WMV directshow filters needed: –mp4 splitter - takes the files and splits the video and audio into two –H.264 codec Michał Kwiatek IT-IS23

4.Discussion Michał Kwiatek IT-IS24