1. Multimedia File Format Heejune AHN Embedded Communications Laboratory Seoul National Univ. of Technology Fall 2011 Last updated 2011. 03. 02

2. Heejune AHN: Image and Video Compressionp. 2 Agenda File Format Important file Format AVI MPEG-4 (ISO-

3. Heejune AHN: Image and Video Compressionp. 3 1. AVI Format AVI format (Audio-Video-Interleave) Microsoft’s definition for “interleaving synchronized stream” (1992) Based on RIFF (Resource Interchange File Format) AVI 1.0 and AVI2.0 (OpenDML AVI file extension, 1997) Format Structure RIFF File = RIFF header + {List|Chunk}s stream data (interleaved chunks) overall info stream info decoder info indexing info

4. Heejune AHN: Image and Video Compressionp. 4 LIST, CHUNKs RIFF Header FOURCC (4 Character Code) = "RIFF" Length (4B) the rest of the file ( < 4G), multiple RIFF headers for longer file Type of file FOURCC (4B) "AVI " for avi file, "WAV " for wav file Contain LISTs and Chunks Chunk & LIST CHUNK = FOURCC(chunk type) + Length(4B) + data LIST = FOURCC("LIST") + Length(4B) + FOURCC(type) + data List type = "hdrl", "movi", and so on "hdrl" for format of streams "movi" for stream data Format style little-endian: e.g. 4bytes E5 A2 00 00 = 0xA2E5 4Byte aligned

5. Heejune AHN: Image and Video Compressionp. 5 AVI header typedef struct _avimainheader { FOURCC fcc; // "avih" DWORD cb;// chunk length DWORD dwMicroSecPerFrame;// frame duration ( in usec) DWORD dwMaxBytesPerSec;// peak rate (in Bps) DWORD dwPaddingGranularity;// alignment (typically, 4) DWORD dwFlags;//... DWORD dwTotalFrames;// as the name says DWORD dwInitialFrames;// how much audio leads video (??) DWORD dwStreams;// DWORD dwSuggestedBufferSize;// usually, max chunk size DWORD dwWidth;// resolution DWORD dwHeight;// resolution DWORD dwReserved[4]; } AVIMAINHEADER; Flag AVIF_HASINDEX : The AVI file has an index. (We’ll cover the index later.) AVIF_MUSTUSEINDEX : The application should use the index, rather than the physical ordering of the chunks in the file, to determine the presentation order of the data. AVIF_ISINTERLEAVED: The AVI file contains interleaved data. AVIF_WASCAPTUREFILE: The file is a specially allocated file used for capturing real-time video. AVIF_COPYRIGHTED: The file contains copyrighted data and should not be duplicated.

6. Heejune AHN: Image and Video Compressionp. 6 The Stream Header List and Chunk ("strl", "strh") typedef struct _avistreamheader { FOURCC fcc; // "strh" DWORD cb; // length FOURCC fccType;// type of stream: "auds", "vids", "txts", "mids", FOURCC fccHandler;// codec suggested: "dvds" for DV, " DWORD dwFlags;// flags for disabled and so on WORD wPriority;// priority WORD wLanguage; DWORD dwInitialFrames; DWORD dwScale;// dwRate/dwScale fps DWORD dwRate; DWORD dwStart;// start time (usually 0) DWORD dwLength;// DWORD dwSuggestedBufferSize;// buffer smax DWORD dwQuality;// qualiy value DWORD dwSampleSize;// 0 for varying sample size, non-zero for fixed size struct { short int left, top, right, bottom; } rcFrame;// target location (used for multiple stream in one window) } AVISTREAMHEADER;

7. Heejune AHN: Image and Video Compressionp. 7 The Stream Format Chunk ("strf") any of a number of legal formats (e.g BITMAPINFOHEADER, WAVEFORMATEX e.g. format 'strf' 4bytes 문자열로 strf 의 시작임을 나타내고 있다. size 4bytes size 로 strf 의 사이즈를 나타내고 있다. format Tag 오디오 압축 코덱 (0x40 : G.721, 0x64 : G.726, 0x1:pcm(?))(2bytes) Channels 오디오 채널 수 (2bytes) Sample Rate 오디오 샘플 레이트 (4bytes) Average per Ser 1 초동안 처리되어지는 오디오 크기 (byte 단위 )(4bytes) block Alignent 데이터 처리의 기본이 되는 크기 (2bytes) Bit Per Sample 하나를 표현하는데 필요한 비트수 (2bytes)

8. Heejune AHN: Image and Video Compressionp. 8 Sample

9. Heejune AHN: Image and Video Compressionp. 9 The Stream Data List ("movi") interleaved chunks of steams, interleaved for synchronization easy Stream Chunk = FOURCC + length + data FOURCC = stream id + "db"|"dc" |"pc" | "wb" (e.g. 00dc, 01wb) AVI Index Entries ("idx1", "indx") For random and easy location, same order of stream chunks in movi List typedef struct _avioldindex { FOURCC fcc;// "idx1" DWORD cb;// length struct _avioldindex_entry { DWORD dwChunkId;// stream id DWORD dwFlags;// DWORD dwOffset;// offset from movi list DWORD dwSize;// size of data chunk } aIndex[]; } AVIOLDINDEX;

10. Heejune AHN: Image and Video Compressionp. 10

11. Heejune AHN: Image and Video Compressionp. 11 More Container Format comparison http://en.wikipedia.org/wiki/Comparison_of_container_formats Avi (MSDN) http://msdn.microsoft.com/en-us/library/ms779636.aspx AVI 2.0

12. Heejune AHN: Image and Video Compressionp. 12 1. MP4 File Format History ISO/IEC 14496-12 “Info. Tech. Coding of AV objects” Part 12 ISO Base Media File Format, 2008 Many extensions from Quicktime, Sony, etc Purpose Interchange (one file, all media data) Content Creation (separated meta info and media data) Streaming (one file with protocol hints)

13. Heejune AHN: Image and Video Compressionp. 13

14. Heejune AHN: Image and Video Compressionp. 14 Structure of mp4 file Object-oriented, hierarchical ‘Box’ structure previously called ‘Atom’ ‘Box is a parent or holds data’ rule box = length (4B) + type (4B) + {subbox}es | data But it sometimes breaks length + type + data + subboxes length field includes lengths of all sub boxes. (so min length of box >= 8). 0 for ‘end of file’, 1 for large length (64 bits) encoder Input Video Bit-rate Delay Loss decoder Input Video

15. Heejune AHN: Image and Video Compressionp. 15 Types Increasing types MP4 Registration Authority: http://www.mp4ra.org Full Box Extended type(32), version (8), flag (24) Sample

16. Heejune AHN: Image and Video Compressionp. 16 Data (mdat, stco) 1 chunk = N samples ‘mdat’ has chunks (data units) ‘stco’ (sample table chunk ‘offset’): locate the offset from file start ‘stsz’ (sample table sample size) : Bytes for each sample ‘stsc’ (sample table sample to chunk) : first sample, sample_per_chunk i-th sample location = chunk offset + accum. sample sizes. stco_j + sum_{0tof}{stsz} for i = sum_{0toj}{stsc(n)} + f issues the offset values changes when new boxes are inserted/removed. chunk mdat moov.trak.mdia.minf.stbl stco stsz stsc

17. Heejune AHN: Image and Video Compressionp. 17

18. Heejune AHN: Image and Video Compressionp. 18 mp4 box ISO Code Rule ftyp*(file type, version) pdin (progressive download) moov *(container of all meta boxes) mvhd*(movie header) trak* (each stream) + tkhd* (track header) + tref * (track reference container) + edts (edit list container) + elst (edit list) + mdia*(media info for the track) + mdhd*(media header) + hdlr*(media handler type) + minf*(media info container) + vmhd (video media header) + smhd (sound media header) + hmhd (hint media header) + nmhd (null media header) sample

19. Heejune AHN: Image and Video Compressionp. 19 Rule + minf*(media info container) + dinf*(data info box) + dref*(data reference) + stbl*(sample table) + stsd*(sample desc.) + stts* (decoding time to s.) + ctts (composition time to s.) + stsz (sample sizes) + stz2 (compact sample size) + stco*(chunk offset) + co64 (64 bit chunk offset) + stss +… moof (movie fragment) … mfra (movie fragment random access) … mdat (media data container)…. free (free space)

20. Heejune AHN: Image and Video Compressionp. 20 Rule meta (meta data) … +hdlr + dinf + ipmc + iloc + ipro meco