Presentation is loading. Please wait.

Presentation is loading. Please wait.

1 The VQEG HDTV Test Plan A presentation by Vivaik Balasubrawmanian (Intel) Stephen Wolf (NTIA/ITS)

Similar presentations


Presentation on theme: "1 The VQEG HDTV Test Plan A presentation by Vivaik Balasubrawmanian (Intel) Stephen Wolf (NTIA/ITS)"— Presentation transcript:

1 1 The VQEG HDTV Test Plan A presentation by Vivaik Balasubrawmanian (Intel) Stephen Wolf (NTIA/ITS)

2 2 Outline Introduction: Setting the stage Introduction: Setting the stage HDTV Source Scenes and HRCs HDTV Source Scenes and HRCs Test methodologies Test methodologies Key challenges Key challenges Proposed VQEG approach Proposed VQEG approach Final thoughts Final thoughts

3 3 HD is no longer the future. It is the present! “NBC will upgrade the rest of the network's top-rated late-night lineup into HDTV in 2005” “NBC will upgrade the rest of the network's top-rated late-night lineup into HDTV in 2005” “The Fox network is better prepared then ever for its hi- def broadcast of Super Bowl XXXIX.” “The Fox network is better prepared then ever for its hi- def broadcast of Super Bowl XXXIX.” “The swanky future of hotels is a big-screen HDTV set.” “The swanky future of hotels is a big-screen HDTV set.” “2005 would be the year of high definition video and TV” “2005 would be the year of high definition video and TV” “Shipments of High Definition televisions and set-top boxes are gaining momentum” “Shipments of High Definition televisions and set-top boxes are gaining momentum” “HDTV content is increasing rapidly” “HDTV content is increasing rapidly”

4 4 HD studio workflows are mainstream “Adobe, HP, Intel, Microsoft, and Dell have joined together to deliver a line of open, scalable, certified desktop HDV and HD solutions to deliver optimal performance at a price point that is unbeatable today.” “Adobe, HP, Intel, Microsoft, and Dell have joined together to deliver a line of open, scalable, certified desktop HDV and HD solutions to deliver optimal performance at a price point that is unbeatable today.” “JVC's ProHD initiative represents an economical solution to getting the stations into the HD production business. The company has products to help with this transition on two fronts.” “JVC's ProHD initiative represents an economical solution to getting the stations into the HD production business. The company has products to help with this transition on two fronts.”

5 5 Content acquisition is affordable Panasonic: The AG-HVX200 DVCPRO HD P2 represents the company's first HD solid-state handheld camcorder. The AG-HVX200 has two card slots and records in 1080/60i, 30p and 24p; 720/60p, 30p and 24p; and DVCPRO50, DVCPRO or DV. It also records 480i and 480p on Mini-DV tape from a built-in drive. AG-HVX200 will come available in the fall for about $5,995. Panasonic: The AG-HVX200 DVCPRO HD P2 represents the company's first HD solid-state handheld camcorder. The AG-HVX200 has two card slots and records in 1080/60i, 30p and 24p; 720/60p, 30p and 24p; and DVCPRO50, DVCPRO or DV. It also records 480i and 480p on Mini-DV tape from a built-in drive. AG-HVX200 will come available in the fall for about $5,995. Alec Shapiro, senior vice president of Sony Electronics' Broadcast and Production Systems Division, touted the acceptance Sony's HDC-1000 1080P camera and the XDCAM tapeless acquisition system have found in the industry. Alec Shapiro, senior vice president of Sony Electronics' Broadcast and Production Systems Division, touted the acceptance Sony's HDC-1000 1080P camera and the XDCAM tapeless acquisition system have found in the industry. In the high-end HD production arena, Shapiro noted the Sony's HDCAM format is used by virtually 100 percent of the network sitcoms produced in the United States. In the high-end HD production arena, Shapiro noted the Sony's HDCAM format is used by virtually 100 percent of the network sitcoms produced in the United States.

6 6 Viewer adoption is accelerating Maxent (tm), announced the availability of its value-priced, high- performance 26-inch WXGA LCD Display through Costco Wholesale stores nationwide at a retail price of $1,199. Maxent (tm), announced the availability of its value-priced, high- performance 26-inch WXGA LCD Display through Costco Wholesale stores nationwide at a retail price of $1,199. HDTVs keep getting more affordable. Olevia's new 32-inch LCD HDTV has a lot of the right ingredients for a good low-cost HDTV. HDTVs keep getting more affordable. Olevia's new 32-inch LCD HDTV has a lot of the right ingredients for a good low-cost HDTV. 10m homes around the world were watching HDTV programming on a high- definition TV set, according to research from market analysts In-Stat. 10m homes around the world were watching HDTV programming on a high- definition TV set, according to research from market analysts In-Stat. By the end of 2005, the worldwide total of these HDTV service households is projected to reach 15.5m. By 2009, HDTV households worldwide are forecast to reach 52m. By the end of 2005, the worldwide total of these HDTV service households is projected to reach 15.5m. By 2009, HDTV households worldwide are forecast to reach 52m. HDTV services are currently widely available in five countries: Australia, Canada, Japan, the United States, and South Korea. HDTV services are currently widely available in five countries: Australia, Canada, Japan, the United States, and South Korea. Nearly half of all consumers plan to make their next television purchase a high-definition (HDTV) television set, according to the CEA. Nearly half of all consumers plan to make their next television purchase a high-definition (HDTV) television set, according to the CEA.

7 7 Technology building blocks are in place British Sky Broadcasting (BSkyB) has selected Thomson as its primary HD decoder set top supplier as it prepares to roll out a new digital HDTV service in 2006. The company will use H.264-based advanced video coding for its HD service. British Sky Broadcasting (BSkyB) has selected Thomson as its primary HD decoder set top supplier as it prepares to roll out a new digital HDTV service in 2006. The company will use H.264-based advanced video coding for its HD service. Royal Philips Electronics today unveiled a High-Definition Multimedia Interface™ (HDMI™ ) receiver integrated circuit (IC), the TDA9975A, to meet consumer demands for superior image quality in next-generation TVs. Royal Philips Electronics today unveiled a High-Definition Multimedia Interface™ (HDMI™ ) receiver integrated circuit (IC), the TDA9975A, to meet consumer demands for superior image quality in next-generation TVs. The single cable digital multimedia solution has arrived. High- Definition video and audio capability quietly rolled into the NZ home entertainment market in the lead-up to Christmas, with the release of a number of DVD players, projectors and TV sets from various manufacturers featuring the new High-Definition Multimedia Interface (HDMI TM ) connectivity standard. The single cable digital multimedia solution has arrived. High- Definition video and audio capability quietly rolled into the NZ home entertainment market in the lead-up to Christmas, with the release of a number of DVD players, projectors and TV sets from various manufacturers featuring the new High-Definition Multimedia Interface (HDMI TM ) connectivity standard.

8 8 Differentiation based on quality is the next step OnHD.TV has generated international attention for our list of the 10 celebrities who look better or worse in crystal-clear High-Definition TV. The list has been featured in such publications as The Chicago Tribune, The National Enquirer, The New York Post and TV shows such as Inside Edition and The Fox Report With Shepard Smith. OnHD.TV has generated international attention for our list of the 10 celebrities who look better or worse in crystal-clear High-Definition TV. The list has been featured in such publications as The Chicago Tribune, The National Enquirer, The New York Post and TV shows such as Inside Edition and The Fox Report With Shepard Smith. Driving much of the company's growth is the worldwide move to HD production, and Thomson has products for the acquisition, editing and distribution of both SD and HD content. The top of the company's line in terms of image quality is its Viper FilmStream digital cinema camera. Driving much of the company's growth is the worldwide move to HD production, and Thomson has products for the acquisition, editing and distribution of both SD and HD content. The top of the company's line in terms of image quality is its Viper FilmStream digital cinema camera.

9 9 Hence the need for standardized metrics To verify and make credible claims To verify and make credible claims To objectively validate the promised media experience To objectively validate the promised media experience

10 10 Outline Introduction: Setting the stage Introduction: Setting the stage HDTV Source Scenes and HRCs HDTV Source Scenes and HRCs Test methodologies Test methodologies Key challenges Key challenges Proposed VQEG approach Proposed VQEG approach Final thoughts Final thoughts

11 11 HD landscape today Broadcast Head-end Set-top Box DTH Satellite Cable Distribution IP TV

12 12 Content acquisition Quality: Consumer ($6K), Professional ($50K), Studio ($110K) cameras in use. Quality: Consumer ($6K), Professional ($50K), Studio ($110K) cameras in use. Storage: HDV, HDCAM, DVCPRO-HD decks are prevalent. Storage: HDV, HDCAM, DVCPRO-HD decks are prevalent. Frame Rate: 24p, 29.97i, 30i, 59.94p, 60p. Frame Rate: 24p, 29.97i, 30i, 59.94p, 60p. Frame Size: 1920 x 1080 (1080p), 1920 x 540 (1080i), 1280 x 720 (720p). Frame Size: 1920 x 1080 (1080p), 1920 x 540 (1080i), 1280 x 720 (720p).

13 13 Source Scenes - Copyright Copyright Issues Copyright Issues Public Domain Public Domain Shot by VQEG Members (put into public domain or approved for unlimited research use) Shot by VQEG Members (put into public domain or approved for unlimited research use) Purchased specifically for testing (e.g., VClips) Purchased specifically for testing (e.g., VClips) Proposal: All scenes must have non- expiring use clause for VQEG testing and be obtainable by any interested party Proposal: All scenes must have non- expiring use clause for VQEG testing and be obtainable by any interested party

14 14 Source Scenes - Quality Formats? Formats? Camera/recording quality? Camera/recording quality? Uncompressed? Uncompressed? Mildly compressed? Mildly compressed? Professional Consumer (e.g., Sony HVR-Z1U)? Professional Consumer (e.g., Sony HVR-Z1U)? Allowing some percentage of lower grade material will greatly increase available sources Allowing some percentage of lower grade material will greatly increase available sources

15 15 Source Scenes - Characteristics Motion? Motion? Detail? Detail? Color? Color? Size of source scene pool? Size of source scene pool? Percent unknown to proponents? Percent unknown to proponents?

16 16 HRC Characteristics Bit-rates? Bit-rates? Coder/decoders? Coder/decoders? Error conditions? Error conditions? Pre/post processing? Pre/post processing? Percent unknown to proponents? Percent unknown to proponents?

17 17 Compression Technologies: MPEG4-AVC, MPEG2, VC-1, DV compression. Technologies: MPEG4-AVC, MPEG2, VC-1, DV compression. Bit rate: 8 Mbps – 25 Mbps Bit rate: 8 Mbps – 25 Mbps Rate Control: Variable and constant. Rate Control: Variable and constant. Rate Conversion: 3:2 pull down. Rate Conversion: 3:2 pull down.

18 18 Transmission Technologies: Satellite, Cable, IP Technologies: Satellite, Cable, IP Packet loss: 0-25% Packet loss: 0-25%

19 19 Decode and display processing 3:2 pull down 3:2 pull down Image enhancement Image enhancement De-interlacing De-interlacing Scaling Scaling

20 20 Display Conditions Should we test many types of video displays or should we use a high quality “transparent” video display? Should we test many types of video displays or should we use a high quality “transparent” video display? What combinations of display types and video formats should be tested? What combinations of display types and video formats should be tested?

21 21 Display technologies Broadcast CRT Broadcast CRT Rear projection Rear projection LCD LCD Plasma Plasma Projectors Projectors

22 22 HRC vectors at a glance Content acquisition: Content acquisition: Capture quality Capture quality Storage format Storage format Frame rate Frame rate Frame size Frame size Compression: Compression: Coding technology Coding technology Bit rate Bit rate Rate control Rate control Rate conversion Rate conversion Transmission: Transmission: Technology Technology Packet loss Packet loss Decode and display Processing: Decode and display Processing: 3:2 pull down 3:2 pull down Image enhancement Image enhancement De-interlacing De-interlacing Scaling Scaling Display: Display: Broadcast CRT Broadcast CRT Rear Projection Rear Projection LCD LCD Plasma Plasma Projectors Projectors

23 23 Outline Introduction: Setting the stage Introduction: Setting the stage HDTV HRCs HDTV HRCs Test methodologies Test methodologies Key challenges Key challenges Proposed VQEG approach Proposed VQEG approach Final thoughts Final thoughts

24 24 Test methods Subjective testing Subjective testing Sequence processing Sequence processing

25 25 Subjective Methodology What subjective testing methodology should we use? What subjective testing methodology should we use? Double/single stimulus? Double/single stimulus? Explicit/hidden reference? Explicit/hidden reference? Continuous/discrete assessment? Continuous/discrete assessment?

26 26 Subjective testing Double Stimulus Impairment Scale Double Stimulus Impairment Scale 4 times picture height for all HD formats 4 times picture height for all HD formats Different display technologies Different display technologies 24 subjects per test with: 24 subjects per test with: 20/20 visual acuity with or without eye glasses 20/20 visual acuity with or without eye glasses Normal color vision Normal color vision Familiarity with the language Familiarity with the language No audio No audio Subjects screened per BT.500 Subjects screened per BT.500 Subjects rejected if failed votes >1 in one session. Subjects rejected if failed votes >1 in one session.

27 27 Sequence processing 1080p, 1080i and 720p formats. 1080p, 1080i and 720p formats. No up-sampling or down-sampling. No up-sampling or down-sampling.

28 28 Outline Introduction: Setting the stage Introduction: Setting the stage HDTV HRCs HDTV HRCs Test methodologies Test methodologies Key challenges Key challenges Proposed VQEG approach Proposed VQEG approach Final thoughts Final thoughts

29 29 Key Challenges HD Content HD Content Reliable HD playback for subjective testing Reliable HD playback for subjective testing Investments required for HD testing Investments required for HD testing Tools for HRC error injection Tools for HRC error injection Reference display processing algorithms Reference display processing algorithms Selection of back channel bandwidth for HD Selection of back channel bandwidth for HD

30 30 Outline Introduction: Setting the stage Introduction: Setting the stage HDTV HRCs HDTV HRCs Test methodologies Test methodologies Key challenges Key challenges Proposed VQEG approach Proposed VQEG approach Final thoughts Final thoughts

31 31 Step 1: Subjective testing Select sequences and run a subjective test to get MOS results. Select sequences and run a subjective test to get MOS results. Include as many display technologies as possible. Include as many display technologies as possible. Make sure a variety of HRCs are used for the test sequences. Make sure a variety of HRCs are used for the test sequences. Compile results from multiple labs to get a consolidated score for the sequences. Compile results from multiple labs to get a consolidated score for the sequences.

32 32 Step 2: Evaluation of existing models Run through existing (already submitted) models through the same sequences. Run through existing (already submitted) models through the same sequences. Follow procedure and rules for calibration in the test plan. Follow procedure and rules for calibration in the test plan. Include full reference, reduced reference and no reference models in the test. Include full reference, reduced reference and no reference models in the test.

33 33 Step 3: Results analysis and next steps Analyze objective results to get a correlation to the subjective results. Analyze objective results to get a correlation to the subjective results. Follow evaluation metrics in the test plan. Follow evaluation metrics in the test plan. Decide on next steps based on the results. Decide on next steps based on the results. If correlations are high, we may have a model for HD TV. If correlations are high, we may have a model for HD TV. If correlations are low, proponents will have a test set to tweak their models. If correlations are low, proponents will have a test set to tweak their models.

34 34 Lab readiness poll Of the labs represented here: Of the labs represented here: How many of them have HD playback capability? How many of them have HD playback capability? Of those who don’t have it now, how many of them are planning to have this capability in the next 3 - 6 months. Of those who don’t have it now, how many of them are planning to have this capability in the next 3 - 6 months. How many labs are willing to run HD subjective tests in the next 3 months? How many labs are willing to run HD subjective tests in the next 3 months?

35 35 Outline Introduction: Setting the stage Introduction: Setting the stage HDTV HRCs HDTV HRCs Test methodologies Test methodologies Key challenges Key challenges Proposed VQEG approach Proposed VQEG approach Final thoughts Final thoughts

36 36 Final thoughts HD TV metrics are in urgent need. HD TV metrics are in urgent need. We should test existing models for their correlations before seeking new ones. We should test existing models for their correlations before seeking new ones. HD testing is happening now in organizations. A timely standardized metric will have the greatest impact. HD testing is happening now in organizations. A timely standardized metric will have the greatest impact. VQEG should complete testing the existing models in the next 6 months. VQEG should complete testing the existing models in the next 6 months.

37 37 Next steps Off line meetings to coordinate HD testing activities. Off line meetings to coordinate HD testing activities. Date for next meeting? Date for next meeting? Action items captured so far: Action items captured so far:

38 38 Calibration of model inputs Peak video level +/- 10% Peak video level +/- 10% Block level +/- 10% Block level +/- 10% Horizontal shift +/- 1 pixel Horizontal shift +/- 1 pixel Vertical shift +/- 1 line Vertical shift +/- 1 line Horizontal cropping < 30 pixels Horizontal cropping < 30 pixels Vertical cropping < 20 lines Vertical cropping < 20 lines No horizontal or vertical re-scaling No horizontal or vertical re-scaling Temporal alignment +/- 1 frame excluding dropped/repeated frames Temporal alignment +/- 1 frame excluding dropped/repeated frames No visible chroma differential timing No visible chroma differential timing No visible picture jitter No visible picture jitter

39 39 Model evaluation metrics Attributes: Attributes: Prediction accuracy Prediction accuracy Prediction monotonicity Prediction monotonicity Prediction consistency Prediction consistency Metrics: Metrics: Root mean square error Root mean square error Pearson linear correlation Pearson linear correlation Outlier ratio Outlier ratio

40 40 Results analysis ILG is responsible ILG is responsible


Download ppt "1 The VQEG HDTV Test Plan A presentation by Vivaik Balasubrawmanian (Intel) Stephen Wolf (NTIA/ITS)"

Similar presentations


Ads by Google