Warm up: Why configuration management?

When working with large-scale software systems, configuration management becomes crucial - supporting non-uniform environments gets greatly simplified, if you decouple code from configuration. While building complex software/products such as Flickr, we had to come up with a simple yet powerful way to manage configuration. Popular approaches to solving this problem include using configuration files or having a dedicated configuration service. Our new solution combines both the extremely popular GitHub and cfg4j library, giving you a very flexible approach that will work with applications of any size.

Why should I decouple configuration from the code?

• Faster configuration changes (e.g. flipping feature toggles): Configuration can simply be injected without requiring parts of your code to be reloaded and re-executed. Config-only updates tend to be faster than code deployment
• Different configuration for different environments: Running your app on a laptop or in a test environment requires a different set of settings than production instance
• Keeping credentials private: If you don’t have a dedicated credential store, it may be convenient to keep credentials as part of configuration. They usually aren’t supposed to be “public” but the code still may be. Be a good sport and don’t keep credentials in a public GitHub repo :)

Meet the Gang: Overview of configuration management players

Let’s see what configuration-specific components we’ll be working with today:

• Configuration repository and editor: Where your configuration lives. We’re using Git for storing configuration files and GitHub as an ad hoc editor.
  
• Push cache : Intermediary store that we use to improve fetch speed and to ease load on GitHub servers.
• CD pipeline: Continuous deployment pipeline pushing changes from repository to push cache and validating config correctness.
• Configuration library: Fetches configs from push cache and exposing them to your business logic.
• Bootstrap configuration : Initial configuration specifying where your push cache is located (so that library knows where to get configuration from).

All these players work as a team to provide an end-to-end configuration management solution.

The Coach: Configuration repository and editor

The first thing you might expect from the configuration repository and editor is ease of use. Let’s enumerate what that means:

• Configuration should be easy to read and write
• It should be straightforward to add a new configuration set
  
• You most certainly want to be able to review changes if your team is bigger than one person
• It’s nice to see a history of changes, especially when you’re trying to fix a bug at the middle of night
• Support from popular IDEs - freedom of choice is priceless
• Multi-tenancy support (optional) is often pragmatic

So what options are out there that may satisfy those requirements? The three very popular formats for storing configuration are YAML, Java Property files, and XML files. We use YAML because it is widely supported by multiple programming languages and IDEs and it’s very readable and easy to understand, even for the non-engineer.

We could use a dedicated configuration store; however, the great thing about files is that they can be easily versioned by version control tools like Git, which we decided to use as it’s widely known and proven.

Git provides us with a history of changes and an easy way to branch off configuration. It also has great support in the form of GitHub which we use both as an editor (built-in support for YAML files) and collaboration tool (pull requests, forks, review tool). Both are nicely glued together by following the Git flow branching model. Here’s an example of configuration file that we use:

One of the goals was to make managing multiple configuration sets (execution environments) a breeze. We needed the ability to add and remove environments quickly. If you look at the screenshot below, you’ll notice a “prod-us-east” directory in the path. For every environment, we stored a separate directory with config files in Git. All of them have the exact same structure and only differ in YAML file contents.

This solution makes working with environments simple and comes in very handy during local development or new production fleet rollout (see use cases at the end of this article). Here’s a sample config repo for a project that has only one “feature”:

Some of the products that we work with at Yahoo have a very granular architecture - hundreds of micro-services working together. For scenarios like this, it’s convenient to store configurations for all services in a single repository, which greatly reduces the overhead of maintaining multiple repositories. We support this use case by having multiple top-level directories each holding configurations for one service only.

The Sprinter: Push cache

The main role of push cache is to decrease load put on the GitHub server and improve configuration fetch time. Since speed is the only concern here, we decided to keep the push cache simple - it’s just a key-value store. Consul was our choice: the nice thing is that it’s fully distributed.

You can install Consul clients on the edge nodes and they will keep being synchronized across the fleet. This greatly improves both reliability and performance of the system. If performance is not a concern, any key-value store will do. You can skip using push cache altogether and connect directly to Github, which comes in handy during development (see use cases below to learn more about this).

The Manager: CD Pipeline

When the configuration repository is updated, a CD pipeline kicks in. This fetches configuration, converts it into a more optimized format and pushes it to the cache. Additionally, the CD pipeline validates the configuration (once at the pull-request stage and again after being merged to master) and controls multi-phase deployment by deploying config change to only 20% of production hosts at one time.

The Mascot: Bootstrap configuration

Before we can connect to the push cache to fetch configuration we need to know where it is. That’s where bootstrap configuration comes into play - it’s very simple. The config contains the hostname, port to connect to, and the name of the environment to use. You need to put this config with your code or as part of the CD pipeline. This simple yaml file binding Spring profiles to different Consul hosts suffices for our needs:

The Cool Guy: Configuration library

The configuration library takes care of fetching the configuration from push cache and exposing it to your business logic. We use the library called cfg4j (“configuration for java”). This library re-loads configurations from the push cache every few seconds and injects them into configuration objects that our code uses. It also takes care of local caching, merging properties from different repositories, and falling back to user-provided defaults when necessary (read more at http://www.cfg4j.org/).

Briefly summarizing how we use cfg4j’s features:

• Configuration auto-reloading: Each service reloads configuration every ~30 seconds and auto re-configures itself.
• Multi-environment support: for our multiple environments (beta, performance, canary, production-us-west, production-us-east, etc.).
• Local caching: Remedies service interruption when the push cache or configuration repository is down and also improves the performance for obtaining configs.
• Fallback and merge strategies: Simplifies local development and provides support for multiple configuration repositories.
• Integration with Dependency Injection containers: because we love DI :D.

If you want to play with this library yourself, there’s plenty of examples both in its documentation and cfg4j-sample-apps Github repository.

The Heavy Lifter: Configurable code

The most important piece is business logic. To best make use of a configuration service, the business logic has to be able to re-configure itself in runtime. Here are a few rules of thumb and code samples:

• Use dependency injection for injecting configuration. This is how we do it using Spring Framework (see the bootstrap configuration above for host/port values):

• Use configuration objects to inject configuration instead of providing configuration directly - here’s where the difference is:
  

Direct configuration injection (won’t reload as config changes)

Configuration injection via “interface binding” (will reload as config changes):

The exercise: Common use-cases (applying our simple solution)

Configuration during development (local overrides)

When you develop a feature, a main concern is the ability to evolve your code quickly. A full configuration management pipeline is not conducive to this ability. We use the following approaches when doing local development:

• Add a temporary configuration file to the project and use cfg4j’s MergeConfigurationSource for reading config both from the configuration store and your file. By making your local file a primary configuration source, you provide an override mechanism. If the property is found in your file, it will be used. If not, cfg4j will fall back to using values from configuration store. Here’s an example (reference examples above to get a complete code):
  

• Fork the configuration repository, make changes to the fork and use cfg4j’s GitConfigurationSource to access it directly (no push cache required):
  

• Set up your private push cache, point your service to the cache and edit values in it directly.
  

Configuration defaults

When you work with multiple environments, some of them may share a common configuration. That’s when using configuration defaults may be convenient. You can do this by creating a “default” environment and using cfg4j’s MergeConfigurationSource for reading config first from the original environment and then (as a fallback) from the “default” environment.

Dealing with outages

Configuration repository, push cache and configuration CD pipeline can experience outages. To minimize impact of such events, it’s good practice to cache the configuration locally (in-memory) after each fetch. cfg4j does that automatically.

Responding to incidents - ultra fast configuration updates (skipping configuration CD pipeline)

Tests can’t always detect all problems. Bugs leak to production environment and at times it’s important to make a config change as fast as possible to stop the fire. If you’re using push cache, the fastest way to modify config values is to make changes directly within the cache. Consul offers a rich REST API and web UI for updating configuration in the key-value store.

Keeping code and configuration in sync

Verifying that code and configuration are kept in sync happens at the configuration CD pipeline level. One part of the continuous deployment process deploys the code into a temporary execution environment, and pointing it to the branch that contains the configuration changes. Once the service is up, we execute a batch of functional tests to verify configuration correctness.

The cool down: Summary

The presented solution is the result of work that we put into building huge-scale photo-serving services. We needed a simple yet flexible configuration management system, and combining Git, Github, Consul and cfg4j provided a very satisfactory solution that we encourage you to try.

I want to thank the following people for reviewing this article: Bhautik Joshi, Elanna Belanger, Archie Russell.

Sid Reddy, Director of Engineering, VDEO

On October 25, 2015, Yahoo live streamed the first-ever regular season NFL game over the Internet to a global audience. This was a phenomenal effort, as several teams at Yahoo came together to deliver a flawless experience to viewers across devices and geographies. To give you a sense of the scale involved, the NFL live stream on Yahoo attracted over 15 million unique viewers, over 33 million views, and had a peak of over 3 million concurrent viewers. 33% of these views came from international viewers from 185 countries, who watched over 460 million minutes of the game.

Our innovative API systems were one of the key drivers for the broadcast’s success. During the 4 hours of the NFL game, the API platform fielded over 215 million calls, with a median latency of 11ms, and a 95th percentile latency of 16ms. The APIs showed six 9s of availability during this time period, despite failure of dependencies during spikes in the game.

In this post, we will discuss the importance of APIs to the NFL video experience, examine the Key Performance Indicators (KPIs) to pay attention to, and highlight several essential elements that comprise a robust API platform.

Why are the APIs important?

The APIs are critical to powering the player and the video experiences. As an example, the APIs provide the player with several pieces of information about the video, including the title, description, as well as URLs to fetch “streams” of video (each stream represents a particular combination of resolution, bitrate, and encoding profile); the APIs also filter and provide only those streams to the player that are supported by the given device.

The Lifecycle of an API request

The lifecycle starts with a client doing a DNS lookup for the API, then issuing an HTTPS request that is terminated at the Edge; the Edge then forwards this request to the API origin which calls several dependency services, processes the data received, and builds the final response (see the figure below).

In the remainder of this post, we will discuss our KPIs, the Yahoo ecosystem that we leveraged, and several lessons from having examined our API stack end-to-end. The sections below are organized around components as they are encountered in the lifecycle of an API request.

Know your KPIs

Always start by defining your KPIs and measure your performance against them. This will help ensure that changes you make to the system lead to improved performance. For our system, we had 4 KPIs

• Latency: We needed to provide users with an instant-on video experience.
  
• Availability: We only had one shot with live events, so we had better be highly available!
  
• Error rate: We wanted our system to operate without errors, for reasons of dependency degradation, data store unreachability, etc.
  
• Throughput: We wanted to maximize our hardware capabilities to ensure higher scalability for a given cost.
  

Yahoo ecosystem

Yahoo has a long history of operating services at the highest scale, and as such, several infrastructure components are readily available for use. For example, to avoid a single point of failure, API systems at Yahoo are present by default in several data centers around the world (serving the Americas, Europe, and APAC regions). Paradigms exist to replicate data across these data centers, while ensuring consistency. Also, code is deployed to these data centers with a CI/CD (Continuous Integration/Continuous Delivery) pipeline. Given this baseline Yahoo ecosystem, we will identify some of the most interesting enhancements below.

DNS

This is the first stop for any client talking to APIs. Should a data center become unavailable, clients should get routed appropriately for high availability. Yahoo operates such a DNS-based GSLB (Global Server Load Balancing) system. There are typically two components to such a system: periodic health checks for the different data centers, and setting the TTL (Time To Live) for DNS records. We decreased these times by 5x to improve availability, with the tradeoff being an increase in health check traffic to our origin and DNS servers.

Use the Edge

While our APIs are already present in several data centers, adding an edge layer (aka ADN: Application Delivery Network) helps in lowering latencies significantly, by up to 25% in our case, because the HTTPS termination happens closer to the user. SSL costs 2 additional round trips (1 additional round trip if session is resumed), and optimizing this latency helps significantly.

Abuse protection

We have a system in place to detect abuse, and are able to dynamically accept/reject/degrade the request. This system helps protect the APIs against potential DDoS attacks, preventing a meltdown of either the APIs or the dependencies. Our systems detect abuse from IP addresses as well as “users”, and enables whitelisting/blacklisting specific entities. This system runs on the edge layer, and relays only “good” requests to the API system.

Caching

We have a caching layer on each of the API nodes. This technique not only improves latency (by up to 20%), but enhances availability when dependency services go down. In addition, the load on dependency systems reduces significantly (by up to 70% in our case). For maximum flexibility, we place the cache close to the dependency calls, and use Guava in our implementation.

Dependency protection

Occasionally, we notice patterns where a dependency service would degrade, causing threads in the API system to stall. This dramatically reduces throughput, and can bring the system down, due to thread exhaustion. To avoid this, we implemented aggressive timeouts on dependencies, and an automated mechanism to eliminate calls to dependency services when they are down. This technique improves scalability significantly as threads can now proceed instead of waiting for a timeout.

Speculative retry

In our analysis, some dependencies tend to have a disproportionately large p99 (aka 99th-percentile) latency, compared to say the p95 latency. More generally, we see patterns where there is a steep jump in latency. At the point where the latency jump occurs, we introduced a parallel retry, and consume the first response received. The tradeoff is an increase in traffic to our dependency systems, in exchange for lower latency and error rate. In our case, this approach reduced latency by up to 22%, and error rate by up to 81%.

Throughput benchmarking

We  invested a significant amount of time in optimizing our API performance. We have high scalability targets, and squeezing performance from hardware is critical to keeping costs under control. We improved our performance 2.3X, with a further 3X improvement planned for the future (by optimizing our core Java platform). An important lesson from this exercise is to benchmark a dummy API as well as the APIs of interest, and measure where the throughput drops are happening; then systematically target the causes for the biggest drops. Another lesson is to pre-materialize the data in an off-stage process, and keep the serving APIs simple; this technique, along with an in-memory data store, improves throughput significantly.

Monitoring/Alerting

When you measure, you can act. We developed extensive monitoring capabilities for the entire API system, including dependencies, on a variety of metrics to track the KPIs. We also have proactive alerting, whenever the SLAs are not met. This proved critical during the NFL live stream, as we detected some dependency services that were briefly unavailable, and were able to repair the services involved. Given the robust design and fallbacks, such intermittent issues did not significantly degrade the end-user experience.

Summary

Building a robust and highly scalable API platform was key to delivering a flawless experience for the first NFL game that was live streamed over the Internet. In this post, we examined several techniques and components that enabled such a robust API platform. Here’s a table summarizing the techniques, and the KPIs they promote.

Future work

We have several interesting projects on our roadmap to make our APIs more resilient, including incorporating a destructive testing framework into our API platform to ensure continued robustness, as well as an automated framework to take bad servers/data centers out of rotation.

If you are excited about this sort of work, reach out to sidreddy at yahoo-inc dot com.

By Roberto Osorio-Goenaga, iOS Developer

Unit testing networking code can be problematic, due mostly to its asynchronous nature. Using a staging server introduces lag and external factors that can cause your tests to run slowly, or not at all. Frameworks like OCMock exist to specify how an object responds to a specific query to address this behavior, but a mock object must still be set up for each type of behavior being mocked.

Using Apple’s NSURLProtocol, we can create a test suite that eschews these problems by mocking the response to our network calls centrally, essentially letting your test focus only on business logic. This protocol can be used not only with the built-in NSURLSession class, but can also be used to test classes and structs written with modern third party networking libraries, such as the popular Alamofire. In this article, we will look at mocking network responses in Swift for requests made using Alamofire. The sample project can be found on github.

NSURLProtocol’s main purpose is to extend URL loading to incorporate custom schemes or enhance existing ones. A secondary, yet extremely powerful, use of NSURLProtocol is to mock a server by sending canned responses back to callbacks and delegates. Say we have a very simple struct that uses Alamofire to make an HTTP GET request.

Fig 1 - A simple struct that serves as a REST client

The sample in Figure 1 creates a struct with an NSURL as an init parameter, and a sole method, getAvailableItems(), taking in a completion block as an argument, making a rest call to the NSURL and populating an array of MyItem in the block sent into it. From a testing perspective, we’d like to have a JSON response that matches the expected response, containing an object called items whose value pair is an array of strings. In order to make our tests as thorough and robust as possible, we’d also include at least two other mock responses: a JSON response that does not match this expectation, to test the else clause, and a garbage or erroneous response to check our error handling.

 
Fig 2 - A valid response

 
Fig 3 - A non-valid response

 
Fig 4 - A throw-away garbage response
 

Figures 2, 3 and 4 show a valid response for our purposes, a non-valid yet correct JSON response, and a throw-away string that isn’t even valid JSON, respectively. Without having to make a full-blown staging server, let’s see how we could go about testing these using NSURLProtocol.

To understand where NSURLProtocol fits into this problem, it’s important to look at a bit of the architecture Alamofire employs. Alamofire works as a singleton, as one can see from the above example. There is no instantiation required. Just feed a URL in, and make a request. Under the hood, the entity making the request is called the Manager. Manager is the entity that actually stores the URL and parameters, and is responsible for firing off an NSURLSession request abstracted from the caller class.

The manager for Alamofire can be initialized with a custom configuration of type NSURLSessionConfiguration, which has a property called protocolClasses, an array of NSURLProtocol members. By creating a new protocol that defines what happens when NSURLSession tries to reach a certain type of endpoint, loading it into the protocol array of a new configuration at index 0 (the default configuration), and initializing a new Manager object with this configuration, we can inject Alamofire with a simple, local mock server that will return whatever we want, given any request. Let’s start setting up a test class for our REST client by extending NSURLProtocol to respond to GET requests, and creating an Alamofire.Manager object with a custom NSURLSessionConfiguration that employs our protocol.

Fig 5 - Setting up a testing class for our client

Great, now we have an NSURLProtocol class that takes a GET request, checks the URL, and returns either a valid JSON response, or a simple “GARBAGE” response. This should allow us to test how our client responds. We still haven’t written any cases. We have a MyRESTClient property, as well as a Manager property. We also have a setup initial method that instantiates and loads our custom protocol into the manager instance. We now need a way to inject this manager instance into our Alamofire singleton. Let’s extend our client to the following.

Fig 6 - The REST client with an injectable “manager” parameter

We’ve added an initializer to our struct that allows us to send either a custom manager or nil into Alamofire. When the parameter is nil, the manager will load with its standard configuration. We also edited the request execution to be called via the manager we selected instead of directly through Alamofire. We can now add the following test case to our test class.

Fig 7 - Our first test case

In this test case, we create a new client, and give it our custom manager through the new initializer. We set a testing expectation, since the result comes back on a closure, and, after loading our itemsArray inside, fulfill the expectation. We tell the test case to wait for said expectation to be fulfilled, and, once it is, we make sure the itemsArray contains three items. If so, our test is successful, and our business logic is tested for getAvailableItems. Notice that we have used a bogus URL of “http://notNil”, which we have defined in the protocol to be selected in the conditional for populating the response correctly. To test the “garbage” case, we could write a test like the following.

In the third and final test case, our protocol class will return a “concepts” array instead of an “items” array, so the end result should still be a nil array in the closure.

As you can see, using NSURLProtocol we have created what amounts to a tiny server that responds to our requests and replies as specified, perfect for testing our asynchronous net calls. Now, go forth and test!

P.P.S. Narayan, VP of Engineering

Introduction

About sixty-five years ago, Alan Turing, considered to be the father of Computer Science and modern computing machines, put forth a deep and philosophical question: can machines think? Asked differently, “Can a machine exhibit (intelligent) behavior indistinguishable from a human [1]?” In order to confirm the conclusion, he devised a test, coined the imitation game.  While there have been various interpretations (and extensive debate) of the test, our objective is to look at its application to video streaming by understanding the premise and appreciating the concept.

Turing says:

I believe that in about fifty years’ time it will be possible to program computers, with a storage capacity of about 109, to make them play the imitation game so well that an average interrogator will not have more than 70 percent chance of making the right identification after five minutes of questioning. … I believe that at the end of the century the use of words and general educated opinion will have altered so much that one will be able to speak of machines thinking without expecting to be contradicted.

Over the years, the Turing test has become an important concept in artificial intelligence and the evolution of computing in modern times. In fact, some modifications to the Turing test have been proposed and have been adopted widely. CAPTCHA is a form of reverse Turing test, where we have participant C replaced by a machine, and the task of C, as a gatekeeper, is to distinguish the humans (in A or B or …) from machines [1].

TV on the internet

Television and video broadcasting has been around for decades. Starting with the first television broadcasts in the 1930s to modern broadcasts in 3D and 4K UHD, the evolution has been phenomenal and eye-popping. Over the years, TV broadcast has moved from terrestrial over-the-air radio waves to satellite or fiber delivery to the home. Analog signals have modernized to digital; black-and-white to color. Televisions have evolved from mechanical to electronic to digital, and traditional CRT displays have changed to ultra thin LED displays.

In the mid-1990s, with the explosion of the consumer Internet, a new wave of video consumption started on devices other than the traditional television sets. Video streaming was soon available on desktops and then laptops, and with the WiFi and mobile revolution, we began streaming video on our phones and tablets.

Nearly a century of evolution of standards, technology, infrastructure, and innovation helped the US TV industry grow to more than a $100B [6]. We are in a new generation where internet streaming is a revolution and growing rapidly. “Over-the-Top” video streaming, or OTT as it is commonly referred to, is delivered through the open unmanaged internet, with the “last-mile” companies (e.g., Comcast) acting only as the internet service provider [3]. Netflix, an OTT video streaming service, already accounts for more than 35% of peak US internet traffic. In fact, all of OTT video streaming accounts for more than 50% of both internet and mobile traffic in the US [4].

This new form of video consumption is pushing the infrastructure and technology built for the Internet beyond its original design parameters and capabilities. While content and user interfaces have advanced quite drastically with OTT, the overall quality of experience is still years behind.

The “OTT Turing Test”

Several months ago, I attended the Streaming Media West conference. And, during one of the panels, the question was asked about the “success metrics” for OTT streaming. Rather than invent what we will call as “success”, I believe we need to use existing TV quality as a baseline benchmark.

Say, we modify the original Turing test, to replace A and B with simple LED TV-like monitors. The video inputs for A and B will have the similar videos (e.g. say any NFL game) from either over-the-air HD signals or from an OTT stream. The participant C, or the interrogator, has to interact with the two monitors to determine which of A or B is displaying video that is OTT streamed. We coin this as the “OTT Turing test”.

And, to make it even more challenging, C could be either a human or a machine. Therefore, any variance in human perception will be eliminated via more systematic and objective evaluation of the inputs.  

If we did this test today, would OTT pass the Turing test? My answer is no. The reason is because there are a number of challenges that have yet to be completely solved. Let’s examine some of these challenges, why they exist, and how far away the technology is for us to achieve this goal.

Startup Time

When we want to start watching television today, we switch “on” the TV set and/or our connected device (e.g. cable box). Typically, these startup in at most 2-3 seconds, and you can see the video on your screen in that time. At times, the audio even starts up sooner. In traditional television, the latency for startup can be broken down into two parts: the time taken for the first few frames of the video to reach the device which is determined by the speed of light, and then to the additional time taken for decoding the signal and constructing the frames to be rendered on the display. This holds true for the over-the-air broadcasts as well as for cable or satellite transmissions.

Video Quality

These days, most displays support HD and resolution of 720p or 1080i/p – table stakes in video broadcasting. In fact, most television HDTV broadcasts are shot at 1080i/60 fields-per-sec. Modern television sets have inbuilt sophisticated upscaling algorithms and technologies that can take lower resolution incoming signals and upscale them to 1080p or even 4K. However, these algorithms vary from device to device, manufacturer to manufacturer, and upscaling can have artifacts introduced which can appear on certain types of content (e.g., high motion sports like NFL or video games).

The Spinner

Anyone who has watched video on the internet knows (or should know) what the “Spinner” is. It is the non-technical term given to the manifestation of not being able to deliver content reliably, over a network which is not dedicated.

Buffering and re-buffering (after the video has started) have become a pervasive pain point for viewers of OTT streaming. And there are many different causes of rebuffering. The reasons vary from end viewer device capabilities (and software), to network congestion inside the viewer’s home/premise, to ISP or local area bandwidth issues, to the core getting clogged and so on.  

Rebuffering ratio is a typical industry measure to determine the perceived user quality. It is a ratio, measured for a time window, as the total time spent rebuffering across all users to the total time spent viewing the stream. Rebuffering ratios of 2-4% are considered acceptable across the industry today. For example, if you are watching a video online for 2 minutes, it is normal for you (and all users) to see the spinner for 2-3 seconds. This typically gets worse during live streams, so if you watch an NFL game for more than 2 hours, you are more than likely to experience many interruptions in your viewing totaling up to 2-3 minutes!! Imagine that experience as compared to what you get on TV or cable today.

At Yahoo, our goal for re-buffering is ZERO. Plus, we’ve focused on some new metrics. For example, the percentage of rebuffering impacted views is more important and relevant. Having 100% of your viewers impacted for 1 minute is far worse than 1% of your viewers impacted for 100 minutes. Basically, this would mean that 99% of your viewers had a FLAWLESS experience. That number needs to grow to five 9s like traditional television.

Visual Artifacts

Video encoding is a complex process from signal acquisition to mixing to adding audio tracks and then shipping the packets to the device. The receiving device capabilities can vary quite drastically. And, this is handled when encoding video digitally for transmission, by picking a few different resolutions. This allows for rendering the video well on various form factors (e.g., a 4K display vs a VGA device), and also allows for optimizing the number of bits transmitted depending on the viewing device. The optimization comes from the fact that lower resolutions would require lower number of bits to encode.

Usual OTT streams are encoded in a few different resolutions or “bitrates”. The availability of various bitrates has become a mechanism for handling poor network conditions. Video players on devices “adapt” and deliver a continuous experience (i.e. avoid rebuffering). The tradeoff has become straightforward – reduce resolution rather than stop the video.

Ad Transition

Advertising on television is native and seamless. What do I mean by native? A viewer sees very little or no difference between content and advertising. In other words, when there is an imperceptible break of under 500ms between the content and ads and from the viewer perspective, it is considered “seamless”.  Over time, the television ad ecosystem has evolved and local stations have the ability to insert local video ads into global or national content.

Synchronization

Consider watching a live (American) football, cricket or soccer game on television. Maybe with a bunch of friends or strangers in a bar, with multiple big screen televisions. We quite frequently do this, to enjoy a social viewing experience. With the current television technology, typically all the television sets in the bar are within a few milliseconds of each other – not much lag or lack of synchronization.

However, with OTT, multiple viewers of the same “live” event cannot be guaranteed a synchronized viewing experience. Even if they all started streaming at the same exact moment! This common problem is due to the inherent nature of various streaming protocols and buffer management. In OTT streaming, especially for live events, it is likely that a viewer may watch or experience a goal at a significantly different moment in time as compared to his neighbors in the same apartment complex, which creates a bad user experience.

Summary

In summary, as an industry, we have a lot of challenges to deliver a TV-like experience for OTT streaming. Some of them are easy, while the others are quite difficult to overcome.

It is likely that very soon, startup time, video quality and ad transitions will improve significantly and be indistinguishable from the current television experience. The tougher technical challenges will be to get to ZERO rebuffering and to enable synchronized watching experience for OTT. These are fundamental challenges in the today’s technology, that will require significant innovation and even some revolutionary new concepts such as P2P or new protocols to up the game for OTT.

However, I am very confident that we will overcome these challenges, and win the OTT Turing Test – much before the original Turing Test is solved!!

References

[1] https://en.wikipedia.org/wiki/Turing_test

[2] http://plato.stanford.edu/entries/turing-test/#Tur195ImiGam

[3] https://en.wikipedia.org/wiki/Internet_television

[4] http://www.statista.com/chart/1620/top-10-traffic-hogs/

[5] https://en.wikipedia.org/wiki/Display_resolution#/media/File:Vector_Video_Standards8.svg

[6] http://www.statista.com/topics/977/television/

[7] http://www.bestshareware.net/howto/img1/how-to-remove-pixellation-from-video-1.jpg

[8] http://blogs.tigarus.com/patcoola/wp-content/uploads/sites/patcoola/2014/buffering-2014.png

Yesterday, Yahoo hosted the Streaming Video Alliance’s quarterly member meeting where over 70 executives from across the streaming video landscape convened to advance discussions on a broad range of streaming video topics and reach agreements on best practices, policy and proposed standards. Ron Jacoby, VP of Engineering, Yahoo Video & TV Applications, and P.P.S. Narayan, VP of Engineering, Yahoo Video were the morning’s featured keynotes.

Ron kicked off discussing the challenges and complexities behind building a strong streaming video experience. The root of which stemmed from the rapidly changing consumption patterns of today’s audiences.

Millennials, which now represent over 30% of the US population, consume 283% more media via the internet than non-millennial age groups – a vast change from how their parents watched TV. This reflects a dramatic shift in how TV is being consumed, and is accelerating in key demographics.

Additionally, the 18-24 year old demographic saw a 37% decline in traditional TV viewing. Ron attributed the shift to the pervasiveness of online video content to premium video services and social media across portable media devices, including laptops, tablets, smartphones, etc.

“In order for the industry to succeed in the face of these trends, it needs to look at content and delivery differently,” said Jacoby. “investments in live streaming and innovation in video protocols and delivery is necessary.”

P.P.S. Narayan followed up with a presentation about the technical opportunities and challenges in video streaming. To echo Ron’s statements, PPSN said “folks are moving away from TV..and watching video across different social and OTT platforms. Gone are the days of sitting in the same room with everyone watching the same show.”

He added, “the shift in consumer behavior and consumption patterns is leading to the disaggregation of content – providers are taking content from TV and cable, and making it accessible on multiple platforms, such as phones, tablets, and connected devices. Services, like Hulu, HBOGo, and MLB.TV have invested heavily in this which is a clear indication that they, and the rest of the industry, are serious about embracing this consumer shift.”

This move is indicative of bigger technology shifts, which begs the question “Can the quality of the video be as good as what we see on TV?” Almost. PPSN explained that when Yahoo hosted the first ever NFL live stream, the technological considerations he and his team had to account for, included resolution, bandwidth, encoding, ads, and latency.

He also talked about the next-generation immersive experiences, including time-based immersion, made possible by cloud DVRs and live scrubbing; space-based immersion with VR and 360 degree videos; and people-based immersion, evidenced by the sharing of content on social media. Additionally, he covered how the disaggregation of content without having a “TV Guide” is leading to gaps in content discovery and personalization. The Yahoo Video Guide, is one such example of addressing the growing needs for users to discover and consume relevant and contextual content.

PPSN concluded by expressing the importance of the groups like the SVA, as they are critical to working together as an industry and help move the ball forward in streaming video.

By P.P.S. Narayan, VP of Engineering

Today, we are excited to announce the debut of a Full HD experience on Yahoo’s video programming.  Going forward, we will be delivering, both live and video-on-demand content, at 1080p – a high definition standard, characterised by a resolution of 1920 pixels wide by 1080 pixels in height, with progressive scan – to our users, based on devices, platforms and network capabilities. In addition to this, users will also be able to enjoy the content at up to 60 frames-per-second (fps).

Back in October 2015,  we delivered the first, global live stream of a regular season NFL game at a maximum of 720p/60fps. And this past week, we streamed the AT&T Pebble Beach Pro-Am from PGA TOUR LIVE on Yahoo, at formats up to 1080p/60fps, with TV-like quality. Fans were able to enjoy panoramic views of Pebble Beach, while watching players hit shots off the tight fairways, follow balls as they sailed across the horizon, and see the professional and celebrity golfers nail putts – all thanks to this high resolution and smooth motion video.  

We’re also excited to announce that on April 30, you’ll be able to experience this enhanced video quality as Yahoo Finance hosts the first-ever, global livestream of the Berkshire Hathaway annual shareholders meeting, from Omaha, NE. The “Woodstock of Capitalism” will be broadcast across all devices - with exclusive video-on-demand (VOD) of the event available on Yahoo Finance for 30 days following the meeting.

We have also upgraded our studios to support 1080p/60fps. Starting with Yahoo’s cameras, the whole video pipeline, including video signal acquisition, video mixing, encoding, transcoding, and all the packetization, will be 1080p capable, when we produce the content.

These days, while most of our television displays support Full HD, most television HDTV broadcasts are delivered at 1080i/60 fields-per-second, a lower resolution than 1080p/60 frames-per-second. And, over-the-top (OTT) streaming experiences do not widely support a top quality resolution with purity in signal from start to finish. We are thrilled at Yahoo to be able to deliver this top-notch quality.

We recently delivered our Yahoo Sports’ analysts preview of Super Bowl 50 from our new HD-supported studio. The lineup included: Shaun King, former NFL Quarterback and current Yahoo Sports’ NFL and College Football analyst, Frank Schwab, Editor of Yahoo Sports’ Shutdown Corner blog, and Tank Williams, former NFL safety and Yahoo Sports NFL/Fantasy contributor.

This is an exciting milestone for our team and for Yahoo. We look forward to delivering the best possible viewing experience to users as we continue to iterate on and improve the quality of our stream. Sit back, relax and enjoy the next round.

Tech evangelist, Robert Scoble, got a rare glimpse into Yahoo’s research center in Champaign, IL where he spoke with engineers about about the innovations in big data that the teams were working on. 

Cathy Singer, senior director of engineering, served as Robert’s gracious tour guide.