1. Michał Kwiatek IT-IS1 Media Archive Status 29/10/2007

2. Motivation 1.Summarise the past a.DV MOV 2 AVI conversion b.Recent data migration c.FLV external encoder d.Recent configuration changes e.Recent changes of the Media Archiver software f.Test of HD master format 2.Convince you that at present Media Archive can grow without active help from IT-IS 3.Discuss the future a.Space requirements b.WMV playback on CDS c.FLV format on CDS d.HD video master format Michał Kwiatek IT-IS2

3. Overview NB: No master file is encoded if Media Archiver cannot read its metadata 3Michał Kwiatek IT-IS

4. 1.The Past Michał Kwiatek IT-IS4

5. DV MOV to AVI conversion How to convert DV MOV files into DV AVI files Impossible on a Mac? Found, bought (~150 CHF) and tested: FireStore DV Conversion Suite Converted 39 files provided It’s been decided to keep both DV MOV and DV AVI files in the archive Handed over to the Video team Michał Kwiatek IT-IS5

6. After 2007-10-10CERNDATA26ACERNDATA26BCERNDATA27ACERNDATA27BCERNDATA28A CERNDATA28 BCERNDATA29ACERNDATA29B capacity18622793186227931862279318622793 free1099370743186214537771088 Audio450 Photo/ConversionSettings1 Photo/Masters 550 Photo/Public5 Photo/Public21 Photo/Workspace 605 Software1 Video/Masters/Conferences 10851705 Video/Masters/Movies 66013400 Video/Masters/Rush 340 Video/ConversionSettings1 Video/Public237 Video/Public2 622 Video/TestMasters62 Video/TestSlaves5 Video/Workspace 2790 TempVideoWorkspace 1740 On 2007-10-02CERNDATA26ACERNDATA26BCERNDATA27ACERNDATA27BCERNDATA28A CERNDATA28 BCERNDATA29ACERNDATA29B capacity18622793186227931862279318622793 free1099370710531240245303 Audio450 Photo/ConversionSettings1 Photo/Masters 550 Photo/Public5 Photo/Public21 Photo/Workspace 605 Software1 Video/Masters/Conferences 2790 Video/Masters/Movies 1862 Video/Masters/Rush 340 Video/ConversionSettings1 Video/Public237 Video/Public2 622 Video/TestMasters62 Video/TestSlaves5 Video/Workspace 2790 TempVideoWorkspace 1740 Video Data Migration 03.10.2007 Michał Kwiatek IT-IS6 Legend: ok, don't touch can put more data need to free some space moved from to split between video quota owners audio quota owners photo quota owners operations quota owners 2008 growth estimates Video/Masters/Movies2000 Video/Masters/Conferences1000 Video/Masters/Rush1000 Video/Workspace0 TempVideoWorkspace0 Video/Public2200 Total4200

7. Recent configuration changes New worker node dedicated to photos Retry erroneous slave files after 4 hours instead of 1 day; abandon after 8 days: Michał Kwiatek IT-IS7 Try 4 hours Try 4 hours Try 4 hours Try... 8 days

8. Recent changes in the Media Archiver software Notify CDS only when all slave files are ready (complex – have to synchronise on master files to be sure of semantic correctness) Implemented a “DFS offline” check Support for external encoders Michał Kwiatek IT-IS8

9. FLV External Encoder We’ve tested ffmpeg.exe to generate FLV videos: ffmpeg.mplayerhq.hu/download.htmlffmpeg.mplayerhq.hu/download.html Current statistics: 100% OK (the only 5 errors were related to bad master files) IT-UDS has complete control over: –The version of ffmpeg used –How it is invoked –Generation logs Michał Kwiatek IT-IS9

10. Tests of HD master formats We’ve looked at H.264 MPEG-4 codec (AVC) We’ve decided it is still too early Michał Kwiatek IT-IS10

11. 2.The Present Michał Kwiatek IT-IS11

12. Formats supported by Media Archiver Master formats VideoAudioPhoto DV AVIWAVEJPEG MPEG-2 DVD-quality Michał Kwiatek IT-IS12 Slave formats VideoAudioPhoto WMVWMAJPEG MP3 ? FLV ?OGG

13. Formats stored in Media Archive Michał Kwiatek IT-IS13 Master formats VideoAudioPhoto DV AVIWAVEJPEG MPEG-2 DVD-quality TIFF HDV MOVRAW DVCProHD MOV DV MOV Current funtional needs are satisfied

14. Media Archive interventions More space needed Media Archiver does not work according to specification Michał Kwiatek IT-IS14 Metadata not present or not found Bad master file Specification changes

15. Current statistics Photo: –ERROR1737 –INSYNC17425 Audio: –INSYNC543 –NEW1 Video: –BUSY3 –ERROR4 –INSYNC563 –MSTRCHD19 Michał Kwiatek IT-IS15

16. 3.The future Michał Kwiatek IT-IS16

17. Space requirements in 2008 Michał Kwiatek IT-IS17 Video/Masters/Movies2000 Video/Masters/Conferences1000 Video/Masters/Rush1000 Video/Workspace0 TempVideoWorkspace0 Video/Public2200 Total4200

18. WMV playback on CDS Playback windows should be bigger Higher resolution file should be used Streaming server should be used instead of the HTTP server http://it-multimedia.web.cern.ch/it- multimedia/test/webcast/templates/Artificial_Intelligence/tes t/default_CDS_WMV_vs_improved_WMV_4_3.htm http://it-multimedia.web.cern.ch/it- multimedia/test/webcast/templates/Artificial_Intelligence/tes t/default_CDS_WMV_vs_improved_WMV_4_3.htm Michał Kwiatek IT-IS18

19. FLV on CDS Let’s see how it works: http://it-multimedia.web.cern.ch/it- multimedia/test/webcast/templates/Artificial_Intell igence/test/improved_WMV_vs_flash_4_3.htm http://it-multimedia.web.cern.ch/it- multimedia/test/webcast/templates/Artificial_Intell igence/test/improved_WMV_vs_flash_4_3.htm Implemented as external encoder, no support from IT-IS for the encoder Michał Kwiatek IT-IS19

20. External Encoder Requirements It supports all current (and future!) master formats It is reliable: –Always works for valid files (and returns 0) –Returns non-zero for invalid files –Does not get stuck Michał Kwiatek IT-IS20

21. (Minor) problem with ffmpeg Michał Kwiatek IT-IS21

22. Future master format requirements be supported and easy to read, have good chances to stay supported and easy to read in the future, be recognised by external customers of the archive: television companies (for video files) and newspapers and publishing houses (for photo files), be the format in which the teams work and save their work, be easy to process automatically with the available, well-supported tools Michał Kwiatek IT-IS22

23. H.264 requirements for WMV directshow filters needed: –mp4 splitter - takes the files and splits the video and audio into two –H.264 codec Michał Kwiatek IT-IS23

24. 4.Discussion Michał Kwiatek IT-IS24