1. ELIS – Multimedia Lab Davy Van Deursen, Wim Van Lancker Requirements for automatic evaluation of MF TCs

2. 2 ELIS – Multimedia Lab Parsing of a media fragment Retrieval of the media fragment scenarios retrieve the full media resource UA performs mapping and sends HTTP byte range request UA sends HTTP time/track range request with or without redirect … with or without setup information … Interpretation/rendering of the HTTP response play/visualize the media fragment Testing a MF-aware UA

3. 3 ELIS – Multimedia Lab Automatic testing of parsing+retrieval catch HTTP requests from the UA information necessary in the TC media fragment URI + corresponding HTTP request(s) conclusion: should be feasible Automatic testing of interpretation/rendering computer vision algorithms -> lot of work, not feasible manual testing (inspired on for example SVG test cases) create an artificial a/v resource enabling the manual validation of the result (e.g., show a clock for temporal fragment evaluation) http://ninsuna.elis.ugent.be/content/TC_spatial_25fps_with_audio.mp4 Testing a MF-aware UA

4. 4 ELIS – Multimedia Lab Parsing of HTTP Range header Extraction of the requested media fragment server-triggered redirect (e.g., for tracks) calculate byte-time/track mapping and return bytes corresponding to the found byte range Automatic testing of parsing and extraction catch response information necessary in the TC Range header and corresponding response headers we assume that the server is able to provide the correct bytes according to its calculated byte ranges Testing a MF-aware server

5. 5 ELIS – Multimedia Lab Automatic evaluation of the TCs demands detailed TC inputs existing media resource media fragment (e.g., t=10,20) possible MF-specific HTTP request headers (i.e., Range and Accept-Range-Redirect) combinations possible MF-specific HTTP response headers (i.e., Content-Range-Mapping) Conclusions Server validation UA validation

6. ELIS – Multimedia Lab End2End examples … … incompliant behavior compliant behavior, should be specified in the TC not sure if compliant or not misuse of MF-specific headers

7. 7 ELIS – Multimedia Lab Valid temporal example t=10 UA get m.ogv m.ogv get m.ogv t=10- get m.ogv t=10- get m.ogv bytes=349- get m.ogv bytes=349- get m.ogv … get m.ogv … get m.ogv t=10- / redirect get m.ogv t=10- / redirect Server 206, t=9- 307, bytes=349- … … … … 200 307, m.ogv?t=10 Server 200 307, m.ogv?t=10 Cases to add: include-setup

8. 8 ELIS – Multimedia Lab UA-detectable ‘invalid’ temporal example t=foo UA get m.ogv m.ogv get m.ogv include-setup get m.ogv include-setup get m.ogv bytes=0-22 get m.ogv bytes=0-22 get m.ogv … get m.ogv … Server 206, include-setup … … 200 Cases to add: redirect

9. 9 ELIS – Multimedia Lab UA-undetectable ‘invalid’ temporal example UA get m.ogv m.ogv get m.ogv t=20-;incl-setup get m.ogv t=20-;incl-setup get m.ogv … get m.ogv … Server ??? Cases to add: redirect t=20 length of m.ogv is 15s get m.ogv t=20- get m.ogv t=20- get m.ogv incl-setup get m.ogv incl-setup Server 206, include-setup … … 200

10. 10 ELIS – Multimedia Lab Spatial example xywh= 2,2,20,20 xywh= 2,2,20,20 UA get m.ogv m.ogv get m.ogv xywh=2,2,20,20 get m.ogv xywh=2,2,20,20 Server … … 200 307, m.ogv?xywh=2,2,20,20 get m.ogv … get m.ogv …

11. 11 ELIS – Multimedia Lab Valid track example track=1 UA get m.ogv m.ogv get m.ogv track=1 get m.ogv track=1 get m.ogv bytes=…-… get m.ogv bytes=…-… get m.ogv … get m.ogv … get m.ogv track=1/redirect get m.ogv track=1/redirect Server 206, track=1 307, bytes=…-… … … … … 200 307, m.ogv?track=1 Server 200 307, m.ogv?track=1 Cases to add: include-setup

12. 12 ELIS – Multimedia Lab UA-undetectable ‘invalid’ track example track=54 UA get m.ogv m.ogv get m.ogv include-setup get m.ogv include-setup get m.ogv track=54 get m.ogv track=54 Server 206, include-setup … … 200 Server … … ??? get m.ogv track=54; incl-setup get m.ogv track=54; incl-setup Server … … 206, include-setup get m.ogv bytes=0-22 get m.ogv bytes=0-22 Cases to add: redirect

13. 13 ELIS – Multimedia Lab TODOs provide detailed, machine-readable TC descriptions and extend Corrib? write evaluation software Questions which and how much media resources to use? we can vary in container formats, codecs, codec profiles, codec settings (intra frame period), … URI fragments vs. URI queries? for the moment: only URI fragments URI queries e.g., media.mp4?xywh=100,100,50,50 -> transcoding on server? Issues what if the server chooses what it returns (not exactly the shortest time range possible …)? TODOs – questions – Issues