1. AVS Standard – Status and Future Plan
Wen Gao,
Chinese Academy of Sciences

2. Outline History of video coding technologies Status of AVS standard
Summary
2018/11/19

3. History of video coding technology
2018/11/19

4. History of A/V coding standard
ISO/IEC：MPEG-1, MPEG-2, MPEG-4, MPEG-4 AVC
ITU-T ： H.261, H.263, H.263+, H.263++,H.264
1st generation
2nd generation
MPEG-China
AVS
H.264 /
MPEG-4 AVC
China
2018/11/19

5. Second generation source coding standard
ISO/IEC MEPG，MPEG-4（MPEG-4 part 2）
ITU VCEG，H.263, H.263+, H.26L
ISO/IEC MPEG，MPEG-4 AVC/H.264 （MPEG-4 part 10）
ITU VCEG，H.264
VC9
2002-now
AVS
ISO/IEC MPEG and ITU VCEG JVT, SVC, 3DAVC
2007-
ISO/IEC MPEG，next video coding
ITU VCEG，H.265
2018/11/19

6. Progress of source coding technology
Compression
ratio
AVS-?/MPEG-?
250
AVS-1/MPEG-4 AVC
150
MPEG-2 50
MPEG-1
1991
1994
2003
2030
2018/11/19

7. AVC Structure - Decoder Input Video Signal Split into Macroblocks
16x16 pixels
Coder
Control
Control
Data
Transform/ Scal./Quant.
Quant. Transf. coeffs -
Decoder
Scaling & Inv. Transform
Entropy
Coding
Deblocking
Filter
Intra-frame
Prediction
Output
Video
Signal
Motion-
Compensation
Intra/Inter
Motion
Data
Motion
Estimation
2018/11/19

8. Periods of key video coding tools contributed
~1985
1999
1950
DPCM
Interlace
B-frames
P-frames
Scene Adaptive Coder
Block Motion Estimation
DCT
Macroblocks
Hybrid Coding
Huffman Coding
Transform Coding
Motion Vector Prediction
Video Object Planes
Object-Based Scalability
Error Resilience
Generic
B-pictures
Advanced Deblocking Filter
C Reader, 2002
Multiple Reference
Integer
Transform
Variable Block-size Motion
Compensation
Resource: Cliff Reader 2002
2018/11/19

9. Overview of video coding tools contributed
Early period – 1970s-1980s
Most basic coding tools invented
Few patents filed
Some key patents expired
Middle period – Late 1980s-Mid 1990s
Many patents filed on Interlace and B-frames
Some techniques reinvented and patented
Many patents on small improvements/syntax
Recent period – Late 1990s-Now
Many new big improvements and patents
No patents on some of these, especially Telenor
2018/11/19

10. Video coding tools and its contribution
Transform and bit allocation
Color space Conversion
Entropy coding
ME and MC
1st G
* * * ≈ 50
2nd G
* * * ≈
2018/11/19

11. C(h,v)=C(v)C(h)ΣΣ[P(x,y)COS{πv(2y+1)/16}COS{πh(2x+1)/16}]
175
172
169
162
158
163
181
171
174
168
165
167
177
179
178
176
170
173
185
184
186
180
199
196
189
182
194
191
193
187
192
198
190 Y U V
RGB
YUV
DCT 2 -3 -1 4
-10 -4 5 3 6 -2 -8 8
-12 1
-16 7 -6 -7 -9 64
-32
1430
Quant -1 4 -2
179
VLC/
CABAC
C(h,v)=C(v)C(h)ΣΣ[P(x,y)COS{πv(2y+1)/16}COS{πh(2x+1)/16}]
C(0)=1/2√2
C(v)=C(h)=1/2 [v,h≠0]
x=0
y=0 7 83 69 56 46 38 35 29 27 34 26 58 48 40 32 22 37 19 24 16 8 99
103
100
112 98 95 92 72
101
120
121 87 78 64 49
113
104 81 55 77
109 68 18 62 80 51 17 14 57 13 60 12 61 10 11 66 47 21 83 69 56 46 38 35 29 27 34 26 58 48 40 32 22 37 19 24 16 8 99
103
100
112 98 95 92 72
101
120
121 87 78 64 49
113
104 81 55 77
109 68 18 62 80 51 17 14 57 13 60 12 61 10 11 66 47 21
2018/11/19

12. Color space conversion
RGBYUV
YUV(4:4:4, 4:2:2, 4:2:0): Sample reduction
2018/11/19

13. Macroblock
4:2:0
4:2:2
2018/11/19

14. Transform Decorrelation Energy compaction DCT
Some high frequency coefficients can be removed with the following quantization process
175
172
169
162
158
163
181
171
174
168
165
167
177
179
178
176
170
173
185
184
186
180
199
196
189
182
194
191
193
187
192
198
190 2 -3 -1 4
-10 -4 5 3 6 -2 -8 8
-12 1
-16 7 -6 -7 -9 64
-32
1430
DCT
2018/11/19

15. Bit-allocation Histograms of DCT coefficients: Laplacian distribution
Bit allocation for N transform coefficients with rate R
minimize the Lagrangian cost
2018/11/19

16. Motion estimation and motion compensation
Multiple reference picture
2018/11/19

17. Motion estimation and motion compensation
Variable block-size motion compensation
2018/11/19

18. Motion estimation and motion compensation
Fractional pixel accuracy
720p 32 33 34 35 36 37 38 39 40 41
2000
4000
6000
8000
10000
12000
14000
16000
Bitrate(kbps)
PSNRY(dB)
Crew 1/4 pel MC
Crew 1/2 pel MC
Harbour 1/4 pel MC
Harbour 1/2 pel MC
Night 1/4 pel MC
Night 1/2 pel MC
2018/11/19

19. Entropy coding Reduce the statistics redundancy Huffman Code
Golomb Code
Arithmetic Coding ……
2018/11/19

20. What’s difference between two generations?
Efficient motion estimation and motion compensation
More modes (block sizes)
More directions
More references
Detail interpolation
Deblocking
Better entropy coding
2018/11/19

21. IP issues in international standard
2018/11/19

22. How to deal with IP in standard
Request from standard organization to IP contributor
Non-commitment
most exist standards in China
Rand
ITU-T VCEG
ISO/IEC/JTC1/SC29/WG11 (MPEG)
RandRF
ISO/IEC/JTC1/SC29/WG1 (JPEG)
Patent Pool
AVS
MPEG by MPEG LA, not by ISO-MPEG, much later than standard fixed
2018/11/19

23. Advantage and disadvantage
Non-commitment from IP holder
High risk
RAND
Too ideal, definition not clear, hard to handle
RAND-RF
Everyone happy in the start
but somehow with risk, like the happening at JPEG case
Patent Pool
Later patent pool approach, like MPEG LA, may be not accepted by market, if the pool wants to charge too high
Patent pool approach, like AVS, need to find a way to get all key patent holders join-in
But not adopted by international standard organization yet
2018/11/19

24. Reason of AVS requested by China
What we learn from DVD IP trial in 2002
Problem in the case of design, manufacture, and sale in different region
There is no transfer mechanism from sale region to manufacture region
Problem in over charge of IP
In most major DVD manufacturers in China, a DVD device cost about 30$ in China, but IP charge is about $20
IP cost = 20$/(30+20)$ = 40%
Some data show,
In US, 3-5% of total price for IP charge is acceptable
In Japan, under 10% of total price for tax and so on is acceptable
But DVD charge is legal! How we can solve the problem?
2018/11/19

25. MPEG-4 AVC/H.264 IPR issue 1991：MPEG-1 1994：MPEG-2
No IP charge
1994：MPEG-2
1997, MPEG LA Patent Pool start, one stop shop
Before 2002, 4 US$/device
After 2002, 2.5 US$/device
1996：MPEG LA Inc. no relationship with MPEG
1999：MPEG-4 (Part 2)
Complex, Charge to device, content (per title), usage (per year)
AOL-Times Warner against
2003：H.264/MPEG-4 AVC (Part 10)
licensing term announced
EBU against（2003第96号声明）
licensing term fixed
2018/11/19

26. MPEG4 AVC licensing term for content and usage
Categories
MPEGLA
Via
Total
Title-by-title
Less than 12 minutes
12 – 30 minutes
31 – 89 minutes
More than 90 minutes
$0.00
2% or $0.02
$0.005
$0.015
$0.025
2% or $0.025
2% or $0.035
2% or $0.045
Subscriber/year
0 – 10,000
10,000 – 250,000
250,000 – 500,000
500,000 – 1,000,000
More than 1,000,000
$25,000
$50,000
$75,000
$100,000 -
Free to Air Broadcasting
Viewers/year
100,000 – 500,000
$2,500
$5,000
$10,000

27. MPEG4 AVC licensing terms for devices
Unit/Year
MPEGLA
Via
Total
0 – 50,000
$0.00
50, ,000
$0.25
100, ,000,000
$0.20
$0.45
5,000,000 – 20,000,000
$0.10
(Reach the CAP
at 10M or 16M)
$0.35
More than 20,000,000
at 30M, 37.5M or 45M) -
2018/11/19

28. Risk of MPEG-4 AVC/H.264 Licensing
Another DVD IP trial
MPEG-2 patent pool, one pool
MPEG LA pool，
MPEG-4 AVC/H.264，2 pools + x
MPEG LA pool
17 companies attending
Licensing term change within 10%
VIA pool
5 companies attending
Thomson, not attend any pool
Individual licensing
Risk？
17C+5C+1C+？
2018/11/19

29. AVS try to solve the problem
Create a organization, in which all member agree to put their contributions into a pool, to engage members make money by product, not by patent
Simple to license the package of all central patents in low price
Who will join?
Use patent to promote new technology, =>partnership
Big companies who make product
Universities and research institutes who grant by government
Basis of AVS working group
Use patent to protect itself, =>friendship
work with
Use patent to make money, =>other society
Pay attention to no infringe their patents
2018/11/19

30. AVS Objectives
Create an national audio-video standard for China broadcasting
Extend technology worldwide after China deployment
Engage technology through wide participation
From within China Academia and Industry
From existing worldwide expertise
Create favorable IP licensing terms
Affordable for current National Industries
One stop shop for entire suite of standards
2018/11/19

31. AVS IPR Experts Group formation
Late 2003 the President of AVS asked that a group of experts be assembled to create a recommendation for an IPR policy for AVS
Create policies that:
reflect and respect China law and culture
reflect WTO requirements
reflect successful global practices
balance the rights of the inventor and needs of the implementers
innovative and forward looking
evolves with practice and law in China
2018/11/19

32. Assumption Worldwide licensing Patent pool membership optional
Globally acceptable Intellectual Property policy
Separate pricing possible for China and Rest-of-World
Single domestic license (video, audio.. etc), separate licenses for all other geographies
2018/11/19

33. AVS IPR Experts Group Experienced members:经验丰富的成员
Patent Attorneys专利律师
Technical Experts技术专家
Contract Attorneys处理合同的律师
Previous experience in setting up patent pools具有建立专利池相关经验的成员
Members are from:成员来自
Asia 亚洲
Europe 欧洲
North America 北美
Members represent: 成员代表了
Consumer Electronics 消费电子产品界
Information Technology 信息技术界
Microelectronics微电子产业界
2018/11/19

34. AVS IPR Experts group Members BroadCom IBM Intel Matsushita/Panasonic
Microsoft
Nokia
Sony
Sun Microsystems
2018/11/19

35. Experts Achievements
Dual language (Chinese/English) suite of documents to govern standardization and IPR policy.
Recommendation to the AVS Membership
Constitution 章程
Member Agreement including 包括下列文件的会员协议
IPR Policy 知识产权管理办法
Legacy Agreement 更新协议
Bylaws 章程细则
Recommendations accepted by AVS membership Sept 2004
2018/11/19

36. Leading edge elements
Election of licensing options upon joining standardization effort.
Different IPR options for patents inside PRC than patents outside PRC
Disclosure of existence of unpublished patents with contributions
2018/11/19

37. Licensing Obligations
Non-Participant:
RAND RF
POOL
RAND
No License
Participant:
RAND RF
POOL
RAND
Join AVS
Sign Member Agreement
Declare default option
Participate in Subgroup
Commit to license
License options
RAND RF
POOL
RAND
Submit Contribution
Disclose patents
Declare license option
China Patent:
RAND RF
POOL
Worldwide Patent:
RAND RF
POOL
RAND
2018/11/19

38. Disclosure Obligations
When making a contribution
Unpublished, published and granted patents
Ongoing
When a patent is discovered
When a Patent Application is published
During Final Draft Standard Review
90 days to declare licensing option, OR:
Default applies
2018/11/19

39. AVS has a procedure to remove some tools
If some tools was found own by non-AVS member, if that patent holder is not willing to cooperate, then we will remove that tool out from AVS standard
2018/11/19

40. Status of AVS
2018/11/19

41. Milestone of AVS Mar18-21. 2002 June 21, 2002 Aug 23-24, 2002
178th Xiangshan Science Conference, Beijing, “Broad-band Network and Security Stream Media Technology”
June 21, 2002
“Audio Video Coding Standard Working Group” was set up in Beijing.
Aug 23-24, 2002
first meeting of AVS, AVS united with MPEG-China. Website of AVS opened to the members formally.
Dec 9, 2002
MII formally approve AVS working group to process AV coding related standard in China national standard
Dec 19, 2003
In the 7th AVS meeting, FCD of AVS-video (1.0) and AVS-system (1.0) was finalized.
Dec 12, 2004
In the 11st AVS meeting, FCD of AVS-M was finalized.
2018/11/19

42. AVS members Total 133 members (by May 31, 2005)
CE, 11%；
Com, 12%；
Comp and Soft, 44%；
IC, 14%；
Local research entities, 24%
Worldwide research entities, 30%
2018/11/19

43. Local Members
2018/11/19

44. Multinational members
2018/11/19

45. AVS schedule AVS部分 国家标准计划号 小组草案(WD) 工作组草案(CD) 最终草案(FCD) 送审稿（FD）
国家标准(GB)
AVS1-P1（系统-广播）
T-339
2004.4
2004.8
AVS1-P2（视频-基准）
T-339
AVS1-P2（视频-增强）
2005.9
2006.3
2006.9
AVS1-P3（音频-双声道）
T-339
2005.3
2005.7
AVS1-P3（音频-5.1声道）
AVS1-P4（一致性测试）
T-339
2005.6
AVS1-P5（参考软件）
T-339
AVS1-P6（DRM）
T-339
AVS1-P7（视频-移动1）
T-339
AVS1-P7（视频-移动2）
AVS1-P8.1（P2 over IP）
TBD
AVS1-P8.2（P7 over IP）
2006.6
AVS1-P9.1（P2文件格式）
AVS1-P9.2（P7文件格式）
2018/11/19

46. AVS video family AVS 1.0 video(AVS1-P2) AVS-M(AVS1-P7) 2018/11/19

47. Performance- HD (1280 x720)
2018/11/19

48. Performance- HD (1280 x720)
2018/11/19

49. Performance – SD (720x576)
2018/11/19

50. Performance – SD (720x576)
2018/11/19

51. Performance – CIF (352x288)
2018/11/19

52. Performance – CIF (352x288)
2018/11/19

53. Differences between AVS, H.264, and MPEG-2
tools
AVS
H.264
MPEG-2
¼ pixel MC
½ pixels 4-tap
¼ pixels 4-tap
½ pixels 6-tap
¼ pixels 2-tap
½ pixels 2-tap
Transform and quantization
8x8 integer transform，encoding site normalization only
4x4 integer transform, both encoding and decoding sites need to normalize
8x8 float DCT
Entropy coding
Adaptive 2D VLC
CAVLC
CABAC
VLC
Loop filter
8x8 based
Less boundaries
Less BS-levels (0..2), Less pixels filtered (p0, p1,q0, q1)
4x4 based
More boundaries
More BS-levels (0..4), More pixels filtered (p0..p3,q0..q3)
N/A
2018/11/19

54. Cost efficiency AVS-?/MPEG-? AVS-1 MPEG-4 AVC MPEG-2 MPEG-1
Compression
ratio
AVS-?/MPEG-?
250
150
AVS-1
MPEG-4 AVC
MPEG-2 50
1.0
Complexity
9.0
MPEG-1
2018/11/19

55. Cost efficiency analysis
Tools
Estimated cost increase
AVS
H.264
Multiple reference 1 2
Variable block-size MC
Quarter pixel 3
Entropy coding
0.5
Deblock filter
Total 6 9
2018/11/19

56. AVS video family AVS 1.0 video(AVS1-P2) AVS-M(AVS1-P7) 2018/11/19

57. Testing on stream of BUS
Testing by China Mobile
2018/11/19

58. Testing on stream of FOREMAN
Testing by China Mobile
2018/11/19

59. Testing on stream of MOBILE
Testing by China Mobile
2018/11/19

60. Testing on stream of TEMPETE
Testing by China Mobile
2018/11/19

61. AVS1-P7: video for mobile
Stage 1, targeting features
Software implementation oriented
Low complexity, no cut needed
2018/11/19

62. AVS1-P7 history 2004.8，WD（working draft）
2004.9，10th AVS meeting, WD 2.0
，CD（committee draft）
2005.9，14th AVS meeting, FCD (Final CD)
2006.1, FD
2006.3, GB
2018/11/19

63. AVS1-P7 reference software
2004.9，WM 1.0
，WM 2.0
2005.1，WM 2.2
2005.3，WM 2.3
2005.3，WM 2.4，and Gold C
2005.4，WM 2.5
2005.4，WM 2.5a
2005.6，WM 2.7
, WM 3.3
2018/11/19

64. AVS1 -P7 vs. H.264 功能模块 AVS-P7 H.264 complexity （AVS1-P7 vs. H.264）
Inter prediction
Luma
Intra4x4 mode
Intra Direct mode
Intra16x16 mode
lower
Chroma
3 modes
4 modes
Intra prediction
¼ pel
Inter16x16~4x4
same
¼ pel interpolation
水平1/2位置8tap
垂直1/2位置4tap
1/4位置线性插值
1/2位置6tap
Same/lower
Quantization
系数矩阵(3, 2)
没有scale
系数矩阵(2, 1)
Entropy Coding
2D-VLC
Exp-Golomb
CAVLC
Huffman codes
Deblocking Filter
像素最多滤波一次
滤波点数减少
一次或两次
2018/11/19

65. AVS Implementation Chip for AVS AVS Encoder & Transcoder
NSCC, AVS101, AVS decoder in HD resolution, March 2, 2005
Celestial Semi Inc., Virgo, AVS decoder in HD resolution, Oct. 2005
ICT, Phoenix, AVS decoder in HD resolution, Dec. 2005
AVS Encoder & Transcoder
Real-time encoder & transcoder in D1 resolution, Oct. 2005
Celestial Semi Inc., real-time encoder in D1 resolution, Oct. 2005
NSCC, real-time encoder in D1 resolution, Dec. 2005
Intel, real-time software encoder in D1 resolution, Dec. 2005
AVS decoder software
NSCC, AVS player 1.6, freeware,
Intel, AVS player …
AVS system
DMB-T, Tsinghua U
3G video streaming, HKUST
广州富年、上海文广、四川长城软件
2018/11/19

66. Summary AVS is a co-design to balance technology and IPR
AVS try to provide a clean and cost effective solution to industry
Next step?
2018/11/19

67. Questions?

68. Thanks
2018/11/19