I. INTRODUCTION
Given the central role that the internet plays in modern life, internet bandwidth should be considered a scarce resource, demanding careful consideration as to how we consume it to maximize its utility.
Global internet traffic is predicted to increase nearly threefold between 2016 and 2021, and the proportion of video traffic is expected to rise from 73 to 82% over the same period [1], driven in part by the move of content from broadcast to online delivery channels and to the rapid evolution of virtual reality applications. There is considerable concern that the growth rate of intellectual property (IP) traffic will exceed the rate at which network capacity is being expanded.
Adding network capacity means increasing network speed, particularly the last mile where the traffic leaves the high-speed trunk networks as it is delivered to individual consumers. This is both expensive and time consuming for the network operators, and is not always possible. Thus the efficiency achieved by video compression is becoming increasingly important.
In this paper, we present the argument in favor of an open source, a royalty-free video codec that will keep pace with the evolution of video traffic. Additionally, we argue that the availability of a state-of-the-art, royalty-free codec levels the playing field, allowing small content owners and application developers to compete with the larger companies that operate in this space. This will ultimately result in a richer and more diverse internet.
II. BACKGROUND
The first digital video codec standards, H.120 (1984) and H.261 (1988), were developed at the International Telecommunication Union (ITU) and primarily intended for videophone and video-conferencing use-cases. Both standards were royalty-free. It was not until the 1995 development of MPEG-2/H.262 that standards defined video codecs were monetized when the US Department of Justice approved MPEG-LA [2], a company unaffiliated with either MPEG or ISO, to form a patent pool with the aim of collecting royalties from the patents considered “essential” for implementation. Royalties were set at $6 per unit for devices that could both encode and decode, and $4 per unit for a device capable only of decoding.
This pricing was seen as a problem in early 2001 when the research work around AVC/H.264 was begun by the Joint Video Team (JVT), a group of video coding experts from ITU-T Study Group 16 (VCEG) and ISO/IEC JTC 1 SC 29/WG 11 (MPEG). A guiding principle was set for the development of a “baseline profile” that would be royalty-free:
“The JVT codec should have a simple royalty-free “baseline” profile (both on the encoder and decoder) in order to promote the wide implementation and use of the JVT codec. All implementations should have such a common baseline profile core, in order to allow minimal interoperability among all JVT codecs. The above requirement means that all technology applied in the baseline profile shall have no IPR, expired IPR, or valid but royalty-fee-free IPR.” [3]
However, when it was time to declare IP for this effort, multiple participants refused to declare their IP royalty-free despite the fact that a majority of IP owners at the time voiced their support. Even though the baseline profile performs significantly worse than the “high” and “main” profiles, not all of the IP owners approved its usage for a royalty-free baseline.
MPEG made a further attempt to define a royalty-free baseline when in July 2011 they issued a call for proposals (CfP) for internet video coding that anticipated that patent declarations associated with the baseline profile would be contributed on a royalty-free basis. Each of the resulting three submissions took a different approach:
-
• Web Video Coding (WVC) spun out the Constrained Baseline profile from AVC/H.264 as an independent standard,
-
• Video Coding for Browsers (VCB) is Google's VP8 video codec, and,
-
• Internet Video Coding (IVC) is built from a base of expired patent IP with new technology added.
WVC was published in 2013 and while some of the patent holders again refused to license their technology royalty-free, the hope is that it will become royalty-free once those patents expire in the near future. VCB has been adopted by MPEG but the status at ISO is uncertain. IVC is nearing technical completion.
MPEG's laudable efforts in this area have been hampered by the ISO/IEC rules regarding IP declarations. Standards bodies are provided with guidance to avoid consideration of intellectual property rights (IPR) in technical committees and are asked to let outside organizations (e.g. the World Intellectual Property Organization) handle licensing. The only IPR statement participants are asked to make is a patent declaration that asks them to select one of the following options [4]:
-
2.1 The patent holder is willing to negotiate licenses free of charge with other parties on a nondiscriminatory basis on reasonable terms and conditions. Such negotiations are left to the parties concerned and are performed outside ITU-T/ITU-R/ISO/IEC. (Type-1)
-
2.2 The patent holder is willing to negotiate licenses with other parties on a non-discriminatory basis on reasonable terms and conditions. Such negotiations are left to the parties concerned and are performed outside ITU-T/ITU-R/ISO/IEC. (Type-2)
-
2.3 The patent holder is not willing to comply with the provisions of either paragraph 2.1 or paragraph 2.2; in such case, the Recommendation | Deliverable shall not include provisions depending on the patent. (Type-3)
If a codec is to be considered royalty-free, all technology submissions have to be accompanied by a Type-1 declaration. However, an intellectual property rights (IPR) holder gives up certain patent rights when they make a Type-1 declaration, or when they fail to make a declaration. Consequently, large corporations often file Type-2 declarations even if they have not yet decided whether they will ever charge a royalty, in order to protect their future patent rights. Anyone who believes they have IP that reads on a codec can make a Type-3 declaration, potentially stalling the effort or requiring significant work around. Even if all of the submissions are accompanied by Type-1 declarations, nonparticipants may use patents that put royalty free status in question.
Six years after the CfP was issued we still cannot point to a single published royalty-free standardized codec and for those still in the standardization process, it seems very unlikely that they will see wider adoption than they already have today.
Independent of the work taking place at MPEG, and at the request of the World Wide Web Consortium (W3C), in 2011 the Internet Engineering Task Force (IETF) began an effort to define a “mandatory to implement” (MTI) video codec for the WebRTC [Reference Summerfield5] project that was preferably royalty-free. WebRTC applications [6] would fall back to use the MTI codec in cases where they are unable to negotiate an alternative, guaranteeing interoperability. After much debate two MTI codecs were adopted, Google's VP8, and the AVC/H.264 Constrained Baseline profile. Both are in active use and widely deployed in most major browsers.
Outside of the MPEG, there have been a couple of notable efforts to produce a royalty-free codec. Microsoft published their VC-1 codec through SMPTE, but MPEG-LA established a patent pool to collect royalties [7]. Google created the WebM project [8] in 2010 after acquiring On2 Technologies [9]. Their VP8 codec is widely deployed, has support in most browsers, and is utilized by many WebRTC applications, and its successor VP9 continues to achieve adoption through its use by companies such as Google/YouTube and Netflix.
III. STANDARDIZATION
A) Advantages
Multimedia standardization ensures interoperability, and the ubiquitous support for content that plays back independent of the equipment manufacturer or application vendor. The technical complexity of video codecs makes this difficult to achieve. Standards bodies achieve this by: (1) attracting participation from a large cross-section of the manufacturing community; (2) having mature development processes; and (3) writing precise and accurate and highly scrutinized specification documents.
From the perspective of a hardware vendor, standards provide a degree of certainty that the specification will not change that is less obvious when a technology is developed under the control of a single company. Considering the huge investment required to implement these standards in hardware and the complexity and cost of resolving these errors, this preference is understandable.
The process of standardization also surfaces industry requirements and provides some level of order to how these requirements are collected, rationalized, agreed upon, and eventually fulfilled.
B) Disadvantages
The most successful video codec standards have been published by the ITU and MPEG, both individually and in collaboration, and they enabled the digital broadcast and multimedia industries as we know them today. MPEG was founded by a group of consumer electronics equipment manufacturers with the common goal of solving the interoperability problem, to ensure that content can be consumed by products irrespective of manufacturer. However, when MPEG-LA's AVC/H.264 pool was created, non-manufacturing organizations used it to monetize their IPR. The addition of monetization objectives changed the dynamics of the standardization process. During the development of HEVC/H.265, meetings were held roughly four times a year. At each meeting, MPEG received around 1000 submissions as proponents tried to get their technology integrated into the standard. Most offered vanishingly small improvements and very few were adopted, but meetings had to be extended by several days to process them all. As a result, MPEG is a relatively small sub-committee within ISO, but it accounts for a disproportionately large number of patents produced through activities across the whole of ISO.
In recent years there has been a marked shift away from the traditional broadcast model, where content is pushed over fixed channels on a defined timetable, to one where content is made available on-demand through the internet. The 10–15 year hardware refresh cycle that works for the broadcast model no longer meets the needs of the more dynamic and fast-evolving needs of the internet model that relies on the ubiquitous availability of high-speed, general purpose computing devices (e.g., desktop, laptop, and tablet computers) that can be updated as new applications and use cases are developed.
One of the major disadvantages of developing a codec through royalty-bearing standards bodies is that while it produces good codecs, the time required to do so is excessively long. The pace that codecs are evolving is insufficient to meet the expected rate of growth of video content in the coming years. User-generated content is no longer the only form of video on the internet; professionally produced and curated content, security monitoring systems, video communication devices, and in particular virtual and augmented reality applications are predicted to generate a significantly higher volume of traffic in the coming years. Codecs need to evolve at a faster rate to keep pace. This is made possible by usable software implementations and/or upgradability of software on internet-connected devices.
The biggest problem with royalty-bearing standards-based video codecs is that they are difficult and expensive to license. The original intent of the patent pool – to be a one-stop-shop and make it easier to license the technology – has not worked out in practice. For example, there are currently three publicly known separate patent pools (MPEG-LA, HEVC Advance, and Velos Media) licensing HEVC/H.265, and there are also several other companies claiming to have HEVC/H.265 standards-essential patents and who are not participating in any of the three pools. In addition, there is an unknown number of patent holders who have yet to declare their licensing terms; and each pool has different and complicated rate structures. This uncertainty has hindered the adoption of HEVC/H.265 and led MPEG to lobby for a change in the rules within ISO that govern the handling of patents to prevent the problem from recurring when the Joint Video Exploration Team (JVET), a collaboration between ITU-T VCEG (Q6/16) and ISO/IEC MPEG (JTC 1/SC 29/WG 11) release their new codec, expected in 2021.
IV. ROYALTY-FREE
A) The case against
There is an argument [Reference Glidden10] that the potential payback afforded to technology IP licensing is a necessary motivation for research and development work. While this might be true for some, the huge storage and networking costs related to video have proven incentive enough for three companies (i.e. Google, Cisco, and Mozilla) to dedicate over a 100 engineers to royalty-free coding. Keeping the cost of the new technologies as low as possible motivates another set of companies who participate in royalty-bearing standards. Others that contribute do not seek direct licensing income but rather seek to use patents as a defensive hedge against being sued. For those that do seek royalty revenues from patents, courts have recently set the value of these patents at fractions of a cent per device, a number far lower than some had hoped for, e.g., Motorola versus Microsoft [11].
Another argument that some have offered is that a royalty-free video codec is a virtual impossibility because of the overwhelming number of video compression patents that have been filed. While there are over 100 000 filed patents in this space, many if not most of those relating to the fundamental technology have now expired, and a large number of patents not expired are held by companies that actually desire a royalty-free video codec. Of the remaining patents, many have a narrow scope and presumably can be easily circumvented. Others which might have been challenged with prior art remain unchallenged only by lack of incentive. Participants are reticent to challenge others' IP claims in fear of reciprocal action and a state of quid pro quo persists.
An oft-cited hurdle to royalty-free video compression is the great success of the existing royalty-bearing video compression standards. The argument goes “the existing process has produced great technology – H.261, MPEG-1, MPEG-2/H.262, H.263, AVC/H.264 and HEVC/H.265 – with hundreds of companies participating to great effect. Any change from the existing process is both less likely to succeed and unnecessary.” While it is hard to argue against the past success of MPEG, the current situation is untenable. HEVC/H.265 unquestionably provides impressive compression gains, but despite being published 4 years ago its usage remains largely stalled by licensing concerns. There are currently three licensing bodies charging fees for HEVC/H.265 technology (MPEG-LA, HEVC Advance, and Velos Media) and technicolor is negotiating its own licensing terms with licensees on an individual basis. In addition, there are rumored to be a number of hold-outs who have not yet announced their licensing intentions but want to charge even higher fees for the use of their patents, for example [12]. Over a similar period VP9, which was published at around the same time as HEVC, has been in active use and adoption has risen considerably.
Some argue that royalty-free does not mean less expensive. They argue that every new codec costs space on chip and costs content providers time and money for support. The entire industry must thus pay a toll to support multiple codecs. It is true that each additional codec costs money but it is not true that this is any more a problem for a new royalty-free codec than a new royalty-bearing codec. Any format that is used by consumers provides advantages for both the content provider and the end user: when used it is a proof that the chip vendor made the right choice. A much bigger waste of time and cost comes when a video codec sits unused on a chip for lack of content or use cases or because of licensing uncertainty.
B) The case for
The internet has become the key conduit for communicating and consuming information. When you write an article or publish a web page and share it, you do not pay for the means by which that text is compressed or delivered across the internet. Neither does the manufacturer of your phone or laptop computer nor for that matter the creator of the program which you use to distribute that article. That is because the core web technologies such as JPEG, HTML, HTTP, TCP/IP, etc. are open and freely implementable.
Unfortunately, although the video is now a key web experience, there has not until recently been a broadly-adopted, free video technology. Instead, manufacturers of devices and organizations that build tools to enable multimedia experiences on the web have been forced to pay a toll to technology licensors to use the digital video compression standards.
1) A high-quality, free video technology is necessary to allow innovation
A high-quality, royalty-free video technology is necessary to foster innovation. Broadly speaking, the prevailing licensing terms around royalty-bearing standards codecs require developers to pay royalties on each unit of the developer's product that incorporates the codec. In some cases, developers and manufacturers are not required to pay royalties when they produce their products in comparatively low volumes. Ironically, this pricing policy can benefit established companies with low-volume, high-priced products, while not providing any meaningful benefit to startups that are distributing millions of video-enabled products but are not yet deriving meaningful revenue from those products.
Many of the web-based video experiences that we now take for granted, such as web-based video conferencing and access to episodic video content, were first introduced to the world by bold, pre-revenue start-ups. Skype took the world by storm, by enabling one-to-one video chatting through its free video communicator. And Move Networks first gave the world access to television content over the web from providers such as ABC, HBO, Fox, ESPN, Discovery, CBS, WB, and Televisa.
Both Skype and Move Networks enabled these novel experiences by distributing free client applications that included integrated codec components. If those companies had used royalty bearing standards-based technologies, they would have incurred millions of dollars in royalty fees, costs that they would likely have been unable to bear during the early years of their existence. Without a free technology that is technically competitive, innovation and experimentation will be stifled or will be restricted to large, established companies that either can afford the luxury of subsidizing free products or, because they are already paying the annual cap for codec usage (where such caps exist), can effectively distribute new, experimental products without incurring additional costs.
2) A royalty-free codec eliminates licensing complexity
Navigating the licensing landscape for royalty-bearing standards technology can pose an expensive and sometimes insurmountable hurdle for small companies. A royalty-free standard eliminates that complexity.
Royalty-bearing standards-based codecs are generally licensed through patent pools – in some cases multiple pools (e.g., HEVC). In addition, there are companies that purport to own standard-essential patents but do not participate in the patent pools. The patents controlled by those companies must be licensed through individually negotiated arrangements.
Just analyzing this complex licensing terrain poses a challenging and expensive legal and administrative hurdle for even the most sophisticated user. Large companies have the legal and business manpower to absorb this work and may be able to take advantage of their own patent portfolios to negotiate unique cross-licensing arrangements with individual licensors. Small companies, in particular, are left to face proportionately larger costs to enter into license agreements and may very well lack the heft to negotiate individual deals with licensors. Similarly, monitoring and reporting of royalties add administrative costs.
A technically competitive royalty-free codec reduces or eliminates the administrative and legal costs associated with the use of the codec. This advantage may provide the greatest benefit to small and early-stage companies that have historically produced some of the great technical innovations.
3) Royalty-free codecs are necessary to reach the next billion users
For users in developing economies to access the internet, they need affordable multimedia-capable devices and services. Even comparatively inexpensive royalty-bearing technologies add to the total cost of already expensive mobile phones and devices, pushing them out of the reach of many users in the developing markets.
4) The market is increasingly open to a royalty-free codec
One important piece of evidence supporting the case for a royalty-free codec is the market's increasing embrace of such royalty-free technologies. In 2010 Google and partners founded the WebM project [7] and simultaneously launched the VP8 codec, followed by its successor, VP9 in 2013.
Since their respective launches, VP8 and VP9 have enjoyed significant success. VP8 has reached the Final Draft International Standard (FDIS) stage, which designates that the technology has achieved technical acceptance and will become a published standard pending a final ballot of national bodies as a royalty-free codec at MPEG and is, along with AVC/H.264, one of two codecs that are designated as mandatory to implement in the IETF WebRTC standard. Based on recent data, it is far and away the most popular video codec for WebRTC applications [8].
Although newer, VP9 has also seen significant adoption, particularly for streaming media services, and is supported by billions of endpoints, including:
-
• Chrome, Firefox, Edge, and Opera browsers;
-
• Virtually all major 4 K smart TVs;
-
• Mobile devices powered by chips from Qualcomm, Mediatek, Samsung, Huawei, Intel.
This broad platform has allowed content distributors such as YouTube and Netflix to rely increasingly heavily on VP9.
V. CHARACTERISTICS OF A SUCCESSFUL RF CODEC
At least four things are needed to lay the foundation for a successful royalty-free technology. First, potential users need to feel confident that there is little risk of a patent owner successfully claiming that users require a royalty-bearing license to use the technology. Second, the technology has to offer an advantage such as technical superiority, lower cost or even licensing surety. Third, the codec must provide sufficiently high performance to satisfy developers' design needs. And finally, because of the increasing complexity of the video ecosystem, the codec must enjoy broad adoption throughout the ecosystem. This section explores these characteristics in more depth.
A) There must be some advantage to using the technology
No one would go through the pain of supporting a new codec unless it provided some distinct advantage. A new codec could require fewer bits to achieve the same quality. It could encode and decode on cheaper hardware or cover special use cases that the alternatives do not. It might allow a content provider a means for distribution which enables massive scale without fear of crushing licensing costs or it could just be available for use sooner than the alternative.
At an inflection point where existing technologies are suddenly being outperformed by both a new royalty-bearing and a new royalty-free technology, there is no incumbent advantage to use either choice. The cost of supporting the higher performance royalty-bearing codec would, therefore, be additive to current costs, making the royalty-free technology an attractive alternative.
B) Royalty-free
There are numerous open source implementations of standards-based codecs that can be used without paying for a copyright license for the software, but that nevertheless requires a patent license to use the proprietary techniques that power the codec. A successful royalty-free codec, therefore, must use only techniques that are outside the scope of the relevant patents owned by licensors seeking royalties or are available for use because they have been anticipated by prior art or because any patents on such techniques have expired.
For application and web developers to adopt the codec, they must perceive it as safe to use, without the substantial and credible threat of IP-based lawsuits or royalties. There is no single key to achieving that perception, but there are several factors that contribute to it.
The codec developer(s) should begin by having a defined process for reviewing codec tools to provide assurance that they are free from encumbrance from third-party patents. Among the steps that the developer can take to provide that assurance are:
-
Developing new techniques that the developer owns that are outside the scope of patents owned by licensors seeking royalties.
-
Actively exploring techniques covered by expired patents.
-
Encouraging contributions from entities that have useful IP assets.
Because codec development is frequently evolutionary rather than revolutionary, developers often use existing codecs as the launching point for the next generation. Being able to start from a previous, successful royalty-free baseline means that the development team does not have to start building a new codec from scratch.
C) High quality and high performance
Just as higher performance may lend traction to a new, royalty-free codec, poor performance will likely hinder adoption. In general, therefore, a royalty-free codec must have at least comparable performance to the alternatives. When offered as an alternative to an incumbent codec that already has broad adoption, the royalty-free codec will likely have to show technical superiority for at least some important use-cases.
D) Widespread industry adoption
Even a groundbreaking new compression format is of little use if a content distributor cannot easily prepare content in the format or if a consumer cannot view that content on the device of their choice. Enabling broad adoption of a new codec requires more than just the dedication of a motivated group of open source developers. It requires the participation of the myriad players in the video ecosystem.
Ultimately, the starting point should include high quality, performant open source implementations that can serve as the basis for production encoders (or real-time encoders for RTC scenarios) and highly-optimized decoders for operation on x86- and ARM-based platforms and GPU processors. Great open source implementations of x264, VP8, VP9, and the Opus audio codec that are fast to encode and decode have brought these tools to the masses, with hobbyists bringing further great gains.
From this launch point, the codec project needs to ensure that content distributors can easily use the format. Users generally do not want to be prompted to download a new application to play their content. Likewise, developers would prefer to build a user playback experience with minimal effort and by reusing existing tools and, ideally, a widely adopted player. Having those tools integrated into HTML5 browsers and operating systems such as Windows or Android, gives an important advantage.
Similarly, content creators do not want to build a new encoding toolchain to support the new format. They want to continue encoding using their existing tools, so integration of the new encoder into those tools greases the wheels for adoption.
It is also very important that there is widespread hardware support. While there are some use cases for which codecs can achieve acceptable performance in software, the growing demands of high-end consumers for ever-increasing pixel counts, higher definition color, high frame rates and new use cases (like VR) necessitate support in hardware. For that reason, any royalty-free codec needs support across a wide range of chips including chips for cell phones, set-top boxes, televisions, and laptop computers.
VI. DELIVERING A ROYALTY-FREE VIDEO CODEC
In September 2015, a group of global technology leaders launched The Alliance for Open Media (“AOM”) with the express purpose of creating a royalty-free video codec that is available for everyone. That effort is scheduled to produce its first video codec, dubbed AV1, by the end of 2017, with significant quality gains over the present state-of-the art codecs, HEVC/H.265, and VP9.
The broad coalition of technology companies that have assembled to form AOM provides strong evidence that leaders in the web video ecosystem recognize the promise of a royalty-free video codec. At present, 30 companies have joined the Alliance, including Adobe, Allegro, Amazon, AMD, Amlogic, Argon Design, ARM, Ateme, BBC, Bitmovin, Broadcom, Chips & Media, Cisco, Google, Hulu, IBM, Intel, Ittiam, Microsoft, Mozilla, Netflix, NGCodec, nVidia, Polycom, Realtek, Socionext, VeriSilicon, VideoLAN, Vidyo and Xilinx. Many of these companies already support open source codecs such as VP8, VP9, and Opus in their products – for instance, in the Chrome, Edge, and Firefox browsers; in streaming video from YouTube and Netflix; and on chips from Intel and Broadcom – and their early support for AV1 should provide an important boost for the format and should help ensure interoperable implementations and a complete specification.
From the recruitment of members to the organization of working groups, AOM's members have worked to address the challenges of establishing a new royalty-free video codec. At the outset, AOM's founders sought new members who could help broaden the IP foundation underlying AV1. A number of members bring decades of experience in codec development. That experience provides not only technical know-how but also IP coverage in terms of either patents or prior art offered by early implementations of coding techniques. All members agree up front to license all their patents that pertain to the final codec free of royalty.
In addition to having access to techniques that members have developed, AOM's researchers are also exploring expired patents and experimenting with new approaches to video compression in an effort to avoid thickets of video-related patents. Two of the founding AOM members, Mozilla and Cisco, each had a royalty-free codec effort and were cooperating in an effort to standardize such a codec through the IETF, called NetVC [13]. While each pursued all avenues for developing their codecs, both devoted significant effort to exploiting techniques that had not been previously applied to video coding or which used expired patents. An IP vetting process is built into the AOM review process to minimize IP risk.
The AV1 codec is already performing markedly better than x265. In January 2017, Mozilla ran comparisons for AV1 in comparison with x265. With just 11 of the 90 or so experiments available in the toolset, the gains were already in the order of 20% bitrate savings for the same quality [Reference Terriberry14]. Since that time over 40 tools have been adopted by the organization with over 50 still being reviewed. We expect to show gains on video sets used by the industry of well over 30%.
VII. CONCLUSION
In this paper, we present the argument in favor of royalty-free video codec technology that keeps pace with the evolution of video traffic. The historical standards development process and IPR policies add delay and cost to the resulting royalty-bearing codecs that impede their adoption. The Alliance for Open Media will provide a viable solution that should achieve widespread adoption and high quality without royalties.
ACKNOWLEDGEMENT
The authors would like to acknowledge the great work done by the members of the Alliance For Open Media and to all of the royalty free media efforts that have come before. We'd also like to thank Anne Aaron, Iraj Sodagar, and Mo Zanaty for their careful proof reading and edits of this paper.
James Bankoski is an Engineering Director working on Google's WebM project. He is the former CTO of On2 Technologies, and a technical contributor to all of On2s and later Google's codecs from Tm2x through VP9; including video codecs widely used in Flash and Skype and now WebM.
Matthew Frost is a Group Product Manager working on Google's WebM project. Prior to joining Google, Matthew was CEO at On2, which was acquired by Google in 2010.
Adrian Grange is a software engineer working on Google's WebM project and is Chair of the Codec Working Group at the Alliance for Open Media. He was a technical contributor to all of On2/Google's video codecs from VP3 to VP9.
Open access
