Qualcomm Ref.

2207506TW

EARLY NOTIFICATION FOR A LOW LATENCY VIDEO DECODER

TECHNICAL FIELD

[0001] This application is generally related to video processing. For example, aspects of the
application relate to improving video coding techniques (e.g., encoding and/or decoding video)
with respect to early notification for low latency video decoders.

BACKGROUND

[0002] Digital video capabilities can be incorporated into a wide range of devices, including
digital televisions, digital direct broadcast systems, wireless broadcast systems, personal digital
assistants (PDAs), laptop or desktop computers, tablet computers, e-book readers, digital cameras,
digital recording devices, digital media players, video gaming devices, video game consoles,
cellular or satellite radio telephones, so-called “smart phones,” video teleconferencing devices,
video streaming devices, and the like. Such devices allow video data to be processed and output
for consumption. Digital video data includes large amounts of data to meet the demands of
consumers and video providers. For example, consumers of video data desire video of the utmost
quality, with high fidelity, resolutions, frame rates, and the like. As a result, the large amount of
video data that is required to meet these demands places a burden on communication networks and
devices that process and store the video data.

[0003] Digital video devices can implement video coding techniques to compress video data.
Video coding is performed according to one or more video coding standards or formats. For
example, video coding standards or formats include versatile video coding (VVC), high-efficiency
video coding (HEVC), advanced video coding (AVC), MPEG-2 Part 2 coding (MPEG stands for
moving picture experts group), Essential Video Coding (EVC), among others, as well as
proprietary video coder-decoder (codecs)/formats such as AOMedia Video 1 (AV1) that was
developed by the Alliance for Open Media or universal bandwidth compression (UBWC). A goal
of video coding techniques is to compress video data into a form that uses a lower bit rate, while
avoiding or minimizing degradations to video quality. With ever-evolving video services

1
Qualcomm Ref. 2207506TW

becoming available techniques to accelerate video decoding to allow video data to be

decompressed for display in as little time as possible are needed.

SUMMARY

[0004] Systems and techniques are described herein for improved video processing, such as
video encoding and/or decoding. For example, a system can provide an early notification for low
latency video decoders to help parallelize decoding the image data and sending the image data .
According to at least one example, an apparatus for processing video data is provided. The
apparatus includes a memory and a processor (e.g., configured in circuitry) coupled to a memory.
The processor is configured to: determine a number of rows of pixels for one or more portions of
an image; obtain first encoded data for a first portion of the image; decode the first encoded data
to generate first pixel data for the number of rows of pixels of the first portion of the image; output
the first pixel data for the first portion of the image to the memory; and output an indication that
the first portion of the image is available.

[0005] As another illustrative example, a method for processing video data is provided. The
method includes: obtaining second encoded data for a second portion of the image, wherein the
second portion of the image is nonoverlapping with the first portion of the image; decoding the
second encoded data to generate second pixel data for the number of rows of pixels for the second
portion of the image; outputting the second pixel data for the second portion of the image to the
memory; and outputting an indication that the second portion of the image is available.

[0006] In another example, a non-transitory computer-readable medium is provided. The non-

transitory computer-readable medium has stored thereon instructions that, when executed by a
processor, cause the processor to: determine a number of rows of pixels for one or more portions
of an image; obtain first encoded data for a first portion of the image; decode the first encoded data
to generate first pixel data for the number of rows of pixels of the first portion of the image; output
the first pixel data for the first portion of the image to a memory; and output an indication that the
first portion of the image is available.

2
Qualcomm Ref. 2207506TW

[0007] As another example, an apparatus for processing video data is provided. The apparatus
includes: means for obtaining second encoded data for a second portion of the image, wherein the
second portion of the image is nonoverlapping with the first portion of the image; means for
decoding the second encoded data to generate second pixel data for the number of rows of pixels
for the second portion of the image; means for outputting the second pixel data for the second
portion of the image to the memory; and means for outputting an indication that the second portion
of the image is available.

[0008] In some aspects, one or more of the apparatuses or devices described herein is, is part of,
and/or includes a mobile device (e.g., a mobile telephone or so-called “smart phone” or other
mobile device), a wearable device, an extended reality device (e.g., a virtual reality (VR) device,
an augmented reality (AR) device, or a mixed reality (MR) device), a camera, a personal computer,
a laptop computer, a server computer, a vehicle or a computing device or component of a vehicle,
a robotics device or system, a television, or other device. In some aspects, the apparatuses or
devices include a camera or multiple cameras for capturing one or more pictures, images, or
frames. In some aspects, the apparatuses or devices include a display for displaying one or more
images, notifications, and/or other displayable data. In some aspects, the apparatuses or devices
can include one or more sensors (e.g., one or more inertial measurement units (IMUs), such as one
or more gyrometers, one or more accelerometers, any combination thereof, and/or other sensor.

[0009] This summary is not intended to identify key or essential features of the claimed subject
matter, nor is it intended to be used in isolation to determine the scope of the claimed subject
matter. The subject matter should be understood by reference to appropriate portions of the entire
specification of this patent, any or all drawings, and each claim.

[0010] The foregoing, together with other features and aspects, will become more apparent upon
referring to the following specification, claims, and accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Illustrative examples of the present application are described in detail below with
reference to the following figures:

3
Qualcomm Ref. 2207506TW

[0012] FIG. 1 is a block diagram illustrating an example of an encoding device and a decoding
device, in accordance with some aspects of the disclosure;

[0013] FIG. 2 is a block diagram illustrating an example architecture of a video coding hardware
engine, in accordance with some aspects of the disclosure;

[0014] FIG. 3 is a block diagram illustrating an example architecture of a video coding system,
in accordance with some aspects of the disclosure;

[0015] FIG. 4 is a block diagram illustrating a mapping of image data in grid to a bitstream, in
accordance with aspects of the present disclosure;

[0016] FIG. 5 is a block diagram illustrating an example architecture of an early notification

technique for a low latency video coding system, in accordance with aspects of the present
disclosure;

[0017] FIG. 6 illustrates an example application of in-loop filtering to an image, in accordance

with aspects of the present disclosure;

[0018] FIG. 7 is an example illustrating how partial images for early notifications interact with
image tiles for an image, in accordance with aspects of the present disclosure;

[0019] FIG. 8 is a flow diagram for a process for processing video data, in accordance with
aspects of the present disclosure

[0020] FIG. 9 is a block diagram illustrating an example video encoding device, in accordance
with some aspects of the disclosure; and

[0021] FIG. 10 is a block diagram illustrating an example video decoding device, in accordance
with some aspects of the disclosure.

DETAILED DESCRIPTION

[0022] Certain aspects of this disclosure are provided below. Some of these aspects may be
applied independently and some of them may be applied in combination as would be apparent to
those of skill in the art. In the following description, for the purposes of explanation, specific

4
Qualcomm Ref. 2207506TW

details are set forth in order to provide a thorough understanding of aspects of the application.
However, it will be apparent that various aspects may be practiced without these specific details.
The figures and description are not intended to be restrictive.

[0023] The ensuing description provides example aspects only, and is not intended to limit the
scope, applicability, or configuration of the disclosure. Rather, the ensuing description of the
example aspects will provide those skilled in the art with an enabling description for implementing
an example aspect. It should be understood that various changes may be made in the function and
arrangement of elements without departing from the spirit and scope of the application as set forth
in the appended claims.

[0024] Video coding devices implement video compression techniques to encode and decode
video data efficiently. Video compression techniques may include applying different prediction
modes, including spatial prediction (e.g., intra-frame prediction or intra-prediction), temporal
prediction (e.g., inter-frame prediction or inter-prediction), inter-layer prediction (across different
layers of video data), and/or other prediction techniques to reduce or remove redundancy inherent
in video sequences. A video encoder can partition each picture of an original video sequence into
rectangular regions referred to as video blocks or coding units (described in greater detail below).
These video blocks may be encoded using a particular prediction mode.

[0025] Video blocks may be divided in one or more ways into one or more groups of smaller
blocks. Blocks can include coding tree blocks, prediction blocks, transform blocks, or other
suitable blocks. References generally to a “block,” unless otherwise specified, may refer to such
video blocks (e.g., coding tree blocks, coding blocks, prediction blocks, transform blocks, or other
appropriate blocks or sub-blocks, as would be understood by one of ordinary skill). Further, each
of these blocks may also interchangeably be referred to herein as “units” (e.g., coding tree unit
(CTU), coding unit, prediction unit (PU), transform unit (TU), or the like). In some cases, a unit
may indicate a coding logical unit that is encoded in a bitstream, while a block may indicate a
portion of video frame buffer a process is target to.

[0026] For inter-prediction modes, a video encoder can search for a block similar to the block
being encoded in a frame (or picture) located in another temporal location, referred to as a

5
Qualcomm Ref. 2207506TW

reference frame or a reference picture. The video encoder may restrict the search to a certain spatial
displacement from the block to be encoded. A best match may be located using a two-dimensional
(2D) motion vector that includes a horizontal displacement component and a vertical displacement
component. For intra-prediction modes, a video encoder may form the predicted block using
spatial prediction techniques based on data from previously encoded neighboring blocks within
the same picture.

[0027] The video encoder may determine a prediction error. For example, the prediction can be
determined as the difference between the pixel values in the block being encoded and the predicted
block. The prediction error can also be referred to as the residual. The video encoder may also
apply a transform to the prediction error (e.g., a discrete cosine transform (DCT) or other suitable
transform) to generate transform coefficients. After transformation, the video encoder may
quantize the transform coefficients. The quantized transform coefficients and motion vectors may
be represented using syntax elements and, along with control information, form a coded
representation of a video sequence. In some instances, the video encoder may entropy code syntax
elements, thereby further reducing the number of bits needed for their representation.

[0028] A video decoder may construct, using the syntax elements and control information
discussed above, predictive data (e.g., a predictive block) for decoding a current frame. For
example, the video decoder may add the predicted block and the compressed prediction error. The
video decoder may determine the compressed prediction error by weighting the transform basis
functions using the quantized coefficients. The difference between the reconstructed frame and the
original frame is called reconstruction error.

[0029] In some cases, after an image is decoded, a notification may be sent indicating that the
decoded image is ready for display. As a part of handling the notification, the image data may be
sent to a downstream device. In some cases, sending all of the image data may take a certain
amount of time in addition to time used to decode the image. In some cases, it may be desirable to
reduce a total amount of time to decode and send the image data.

[0030] Systems, apparatuses, processes (also referred to as methods), and computer-readable

media (collectively referred to herein as “systems and techniques”) are described herein for

6
Qualcomm Ref. 2207506TW

providing early notification for low latency video decoders. In some aspects, early notification
may help in parallelize decoding the image data and sending the image data. For example, an image
may be divided into image portions, where the image portions may include a number of rows of
pixels. After a first image portion is decoded, a notification may be sent indicating that image data
for the first image portion is ready. The image data for the first image portion may then be sent
(e.g., accessed/loaded) to a downstream device while image data for a second image portion is
decoded. In some cases, a number of rows in a particular image portion may be determined based
on a target latency. In some cases, the number of rows in the image portion may additionally or
alternatively be determined based on a size of a portion (e.g., a tile size) of the coding used to
encode the image. In some cases, the early notification may be adjusted for in loop filtering.

[0031] The early notification systems and techniques described herein can reduce latency
associated with decoding video data. Such reduced latency can be useful and in some cases
necessary for certain applications, such as extended reality (XR) applications (e.g., a virtual reality
(VR), augmented reality (AR), mixed reality (MR), etc.), vehicle applications, robotics
applications, mobile applications (e.g., media streaming for mobile devices), among others.

[0032] Various aspects of the application will be described herein with respect to the figures.

[0033] The systems and techniques described herein can be applied to any of the existing video
codecs, such as Versatile Video Coding (VVC), High Efficiency Video Coding (HEVC),
Advanced Video Coding (AVC), Essential Video Coding (EVC), VP9, the AV1 format/codec,
and/or other video coding standard, codec, format, etc. in development or to be developed.

[0034] FIG. 1 is a block diagram illustrating an example of a system 100 including an encoding
device 104 and a decoding device 112. The encoding device 104 may be part of a source device,
and the decoding device 112 may be part of a receiving device. The source device and/or the
receiving device may include an electronic device, such as a mobile or stationary telephone handset
(e.g., smartphone, cellular telephone, or the like), a desktop computer, a laptop or notebook
computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media
player, a video gaming console, a video streaming device, an Internet Protocol (IP) camera, or any
other suitable electronic device. In some examples, the source device and the receiving device may

7
Qualcomm Ref. 2207506TW

include one or more wireless transceivers for wireless communications. The coding techniques
described herein are applicable to video coding in various multimedia applications, including
streaming video transmissions (e.g., over the Internet), television broadcasts or transmissions,
encoding of digital video for storage on a data storage medium, decoding of digital video stored
on a data storage medium, or other applications. As used herein, the term coding can refer to
encoding and/or decoding. In some examples, system 100 can support one-way or two-way video
transmission to support applications such as video conferencing, video streaming, video playback,
video broadcasting, gaming, and/or video telephony.

[0035] The encoding device 104 (or encoder) can be used to encode video data using a video
coding standard, format, codec, or protocol to generate an encoded video bitstream. Examples of
video coding standards and formats/codecs include ITU-T H.261, ISO/IEC MPEG-1 Visual, ITU-
T H.262 or ISO/IEC MPEG-2 Visual, ITU-T H.263, ISO/IEC MPEG-4 Visual, ITU-T H.264 (also
known as ISO/IEC MPEG-4 AVC), including its Scalable Video Coding (SVC) and Multiview
Video Coding (MVC) extensions, High Efficiency Video Coding (HEVC) or ITU-T H.265, and
Versatile Video Coding (VVC) or ITU-T H.266. Various extensions to HEVC deal with multi-
layer video coding exist, including the range and screen content coding extensions, 3D video
coding (3D-HEVC) and multiview extensions (MV-HEVC) and scalable extension (SHVC). The
HEVC and its extensions have been developed by the Joint Collaboration Team on Video Coding
(JCT-VC) as well as Joint Collaboration Team on 3D Video Coding Extension Development (JCT-
3V) of ITU-T Video Coding Experts Group (VCEG) and ISO/IEC Motion Picture Experts Group
(MPEG). VP9, AOMedia Video 1 (AV1) developed by the Alliance for Open Media Alliance of
Open Media (AOMedia), and Essential Video Coding (EVC) are other video coding standards for
which the techniques described herein can be applied.

[0036] The techniques described herein can be applied to any of the existing video codecs (e.g.,
High Efficiency Video Coding (HEVC), Advanced Video Coding (AVC), or other suitable
existing video codec), and/or can be an efficient coding tool for any video coding standards being
developed and/or future video coding standards, such as, for example, VVC and/or other video
coding standard in development or to be developed. For example, examples described herein can
be performed using video codecs such as VVC, HEVC, AVC, and/or extensions thereof. However,

8
Qualcomm Ref. 2207506TW

the techniques and systems described herein may also be applicable to other coding standards,
codecs, or formats, such as MPEG, JPEG (or other coding standard for still images), EVC, VP9,
AV1, extensions thereof, or other suitable coding standards already available or not yet available
or developed. For instance, in some examples, the encoding device 104 and/or the decoding device
112 may operate according to a proprietary video codec/format, such as AV1, extensions of AVI,
and/or successor versions of AV1 (e.g., AV2), or other proprietary formats or industry standards.
Accordingly, while the techniques and systems described herein may be described with reference
to a particular video coding standard, one of ordinary skill in the art will appreciate that the
description should not be interpreted to apply only to that particular standard.

[0037] Referring to FIG. 1, a video source 102 may provide the video data to the encoding device
104. The video source 102 may be part of the source device, or may be part of a device other than
the source device. The video source 102 may include a video capture device (e.g., a video camera,
a camera phone, a video phone, or the like), a video archive containing stored video, a video server
or content provider providing video data, a video feed interface receiving video from a video server
or content provider, a computer graphics system for generating computer graphics video data, a
combination of such sources, or any other suitable video source.

[0038] The video data from the video source 102 may include one or more input pictures or
frames. A picture or frame is a still image that, in some cases, is part of a video. In some examples,
data from the video source 102 can be a still image that is not a part of a video. In HEVC, VVC,
and other video coding specifications, a video sequence can include a series of pictures. A picture
may include three sample arrays, denoted SL, SCb, and SCr. SL is a two-dimensional array of
luma samples, SCb is a two-dimensional array of Cb chrominance samples, and SCr is a two-
dimensional array of Cr chrominance samples. Chrominance samples may also be referred to
herein as “chroma” samples. A pixel can refer to all three components (luma and chroma samples)
for a given location in an array of a picture. In other instances, a picture may be monochrome and
may only include an array of luma samples, in which case the terms pixel and sample can be used
interchangeably. With respect to example techniques described herein that refer to individual
samples for illustrative purposes, the same techniques can be applied to pixels (e.g., all three
sample components for a given location in an array of a picture). With respect to example

9
Qualcomm Ref. 2207506TW

techniques described herein that refer to pixels (e.g., all three sample components for a given
location in an array of a picture) for illustrative purposes, the same techniques can be applied to
individual samples.

[0039] The encoder engine 106 (or encoder) of the encoding device 104 encodes the video data
to generate an encoded video bitstream. In some examples, an encoded video bitstream (or “video
bitstream” or “bitstream”) is a series of one or more coded video sequences. A coded video
sequence (CVS) includes a series of access units (AUs) starting with an AU that has a random
access point picture in the base layer and with certain properties up to and not including a next AU
that has a random access point picture in the base layer and with certain properties. For example,
the certain properties of a random access point picture that starts a CVS may include a RASL flag
(e.g., NoRaslOutputFlag) equal to 1. Otherwise, a random access point picture (with RASL flag
equal to 0) does not start a CVS. An access unit (AU) includes one or more coded pictures and
control information corresponding to the coded pictures that share the same output time. Coded
slices of pictures are encapsulated in the bitstream level into data units called network abstraction
layer (NAL) units. For example, an HEVC video bitstream may include one or more CVSs
including NAL units. Each of the NAL units has a NAL unit header. In one example, the header
is one-byte for H.264/AVC (except for multi-layer extensions) and two-byte for HEVC. The
syntax elements in the NAL unit header take the designated bits and therefore are visible to all
kinds of systems and transport layers, such as Transport Stream, Real-time Transport (RTP)
Protocol, File Format, among others.

[0040] Two classes of NAL units exist in the HEVC standard, including video coding layer
(VCL) NAL units and non-VCL NAL units. VCL NAL units include coded picture data forming
a coded video bitstream. For example, a sequence of bits forming the coded video bitstream is
present in VCL NAL units. A VCL NAL unit can include one slice or slice segment (described
below) of coded picture data, and a non-VCL NAL unit includes control information that relates
to one or more coded pictures. In some cases, a NAL unit can be referred to as a packet. An HEVC
AU includes VCL NAL units containing coded picture data and non-VCL NAL units (if any)
corresponding to the coded picture data. Non-VCL NAL units may contain parameter sets with
high-level information relating to the encoded video bitstream, in addition to other information.

10
Qualcomm Ref. 2207506TW

For example, a parameter set may include a video parameter set (VPS), a sequence parameter set
(SPS), and a picture parameter set (PPS). In some cases, each slice or other portion of a bitstream
can reference a single active PPS, SPS, and/or VPS to allow the decoding device 112 to access
information that may be used for decoding the slice or other portion of the bitstream.

[0041] NAL units may contain a sequence of bits forming a coded representation of the video
data (e.g., an encoded video bitstream, a CVS of a bitstream, or the like), such as coded
representations of pictures in a video. The encoder engine 106 generates coded representations of
pictures by partitioning each picture into multiple slices. A slice is independent of other slices so
that information in the slice is coded without dependency on data from other slices within the same
picture. A slice includes one or more slice segments including an independent slice segment and,
if present, one or more dependent slice segments that depend on previous slice segments.

[0042] In HEVC, the slices are then partitioned into coding tree blocks (CTBs) of luma samples
and chroma samples. A CTB of luma samples and one or more CTBs of chroma samples, along
with syntax for the samples, are referred to as a coding tree unit (CTU). A CTU may also be
referred to as a “tree block” or a “largest coding unit” (LCU). A CTU is the basic processing unit
for HEVC encoding. A CTU can be split into multiple coding units (CUs) of varying sizes. A CU
contains luma and chroma sample arrays that are referred to as coding blocks (CBs).

[0043] The luma and chroma CBs can be further split into prediction blocks (PBs). A PB is a
block of samples of the luma component or a chroma component that uses the same motion
parameters for inter-prediction or intra-block copy (IBC) prediction (when available or enabled
for use). The luma PB and one or more chroma PBs, together with associated syntax, form a
prediction unit (PU). For inter-prediction, a set of motion parameters (e.g., one or more motion
vectors, reference indices, or the like) is signaled in the bitstream for each PU and is used for inter-
prediction of the luma PB and the one or more chroma PBs. The motion parameters can also be
referred to as motion information. A CB can also be partitioned into one or more transform blocks
(TBs). A TB represents a square block of samples of a color component on which a residual
transform (e.g., the same two-dimensional transform in some cases) is applied for coding a
prediction residual signal. A transform unit (TU) represents the TBs of luma and chroma samples,
and corresponding syntax elements. Transform coding is described in more detail below.

11
Qualcomm Ref. 2207506TW

[0044] A size of a CU corresponds to a size of the coding mode and may be square in shape. For
example, a size of a CU may be 8 x 8 samples, 16 x 16 samples, 32 x 32 samples, 64 x 64 samples,
or any other appropriate size up to the size of the corresponding CTU. The phrase "N x N" is used
herein to refer to pixel dimensions of a video block in terms of vertical and horizontal dimensions
(e.g., 8 pixels x 8 pixels). The pixels in a block may be arranged in rows and columns. In some
implementations, blocks may not have the same number of pixels in a horizontal direction as in a
vertical direction. Syntax data associated with a CU may describe, for example, partitioning of the
CU into one or more PUs. Partitioning modes may differ between whether the CU is intra-
prediction mode encoded or inter-prediction mode encoded. PUs may be partitioned to be non-
square in shape. Syntax data associated with a CU may also describe, for example, partitioning of
the CU into one or more TUs according to a CTU. A TU can be square or non-square in shape.

[0045] According to the HEVC standard, transformations may be performed using transform
units (TUs). TUs may vary for different CUs. The TUs may be sized based on the size of PUs
within a given CU. The TUs may be the same size or smaller than the PUs. In some examples,
residual samples corresponding to a CU may be subdivided into smaller units using a quadtree
structure known as residual quad tree (RQT). Leaf nodes of the RQT may correspond to TUs. Pixel
difference values associated with the TUs may be transformed to produce transform coefficients.
The transform coefficients may then be quantized by the encoder engine 106.

[0046] Once the pictures of the video data are partitioned into CUs, the encoder engine 106
predicts each PU using a prediction mode. The prediction unit or prediction block is then subtracted
from the original video data to get residuals (described below). For each CU, a prediction mode
may be signaled inside the bitstream using syntax data. A prediction mode may include intra-
prediction (or intra-picture prediction) or inter-prediction (or inter-picture prediction). Intra-
prediction utilizes the correlation between spatially neighboring samples within a picture. For
example, using intra-prediction, each PU is predicted from neighboring image data in the same
picture using, for example, DC prediction to find an average value for the PU, planar prediction to
fit a planar surface to the PU, direction prediction to extrapolate from neighboring data, or any
other suitable types of prediction. Inter-prediction uses the temporal correlation between pictures
in order to derive a motion-compensated prediction for a block of image samples. For example,

12
Qualcomm Ref. 2207506TW

using inter-prediction, each PU is predicted using motion compensation prediction from image
data in one or more reference pictures (before or after the current picture in output order). The
decision whether to code a picture area using inter-picture or intra-picture prediction may be made,
for example, at the CU level.

[0047] The encoder engine 106 and decoder engine 116 (described in more detail below) may
be configured to operate according to VVC. According to VVC, a video coder (such as encoder
engine 106 and/or decoder engine 116) partitions a picture into a plurality of coding tree units
(CTUs) (where a CTB of luma samples and one or more CTBs of chroma samples, along with
syntax for the samples, are referred to as a CTU). The video coder can partition a CTU according
to a tree structure, such as a quadtree-binary tree (QTBT) structure or Multi-Type Tree (MTT)
structure. The QTBT structure removes the concepts of multiple partition types, such as the
separation between CUs, PUs, and TUs of HEVC. A QTBT structure includes two levels,
including a first level partitioned according to quadtree partitioning, and a second level partitioned
according to binary tree partitioning. A root node of the QTBT structure corresponds to a CTU.
Leaf nodes of the binary trees correspond to coding units (CUs).

[0048] In an MTT partitioning structure, blocks may be partitioned using a quadtree partition, a
binary tree partition, and one or more types of triple tree partitions. A triple tree partition is a
partition where a block is split into three sub-blocks. In some examples, a triple tree partition
divides a block into three sub-blocks without dividing the original block through the center. The
partitioning types in MTT (e.g., quadtree, binary tree, and tripe tree) may be symmetrical or
asymmetrical.

[0049] When operating according to the AV1 codec, encoder engine 106 and decoder engine
116 may be configured to code video data in blocks. In AV1, the largest coding block that can be
processed is called a superblock. In AV1, a superblock can be either 128x128 luma samples or
64x64 luma samples. However, in successor video coding formats (e.g., AV2), a superblock may
be defined by different (e.g., larger) luma sample sizes. In some examples, a superblock is the top
level of a block quadtree. Encoder engine 106 may further partition a superblock into smaller
coding blocks. Encoder engine 106 may partition a superblock and other coding blocks into smaller
blocks using square or non-square partitioning. Non-square blocks may include N/2xN, NxN/2,

13
Qualcomm Ref. 2207506TW

N/4xN, and NxN/4 blocks. Encoder engine 106 and decoder engine 116 may perform separate
prediction and transform processes on each of the coding blocks.

[0050] AV1 also defines a tile of video data. A tile is a rectangular array of superblocks that may
be coded independently of other tiles. That is, encoder engine 106 and decoder engine 116 may
encode and decode, respectively, coding blocks within a tile without using video data from other
tiles. However, encoder engine 106 and decoder engine 116 may perform filtering across tile
boundaries. Tiles may be uniform or non-uniform in size. Tile-based coding may enables parallel
processing and/or multi-threading for encoder and decoder implementations.

[0051] In some examples, the video coder can use a single QTBT or MTT structure to represent
each of the luminance and chrominance components, while in other examples, the video coder can
use two or more QTBT or MTT structures, such as one QTBT or MTT structure for the luminance
component and another QTBT or MTT structure for both chrominance components (or two QTBT
and/or MTT structures for respective chrominance components).

[0052] The video coder can be configured to use quadtree partitioning, QTBT partitioning, MTT
partitioning, superblock partitioning, or other partitioning structure.

[0053] In some examples, the one or more slices of a picture are assigned a slice type. Slice
types include an intra-coded slice (I-slice), an inter-coded P-slice, and an inter-coded B-slice. An
I-slice (intra-coded frames, independently decodable) is a slice of a picture that is only coded by
intra-prediction, and therefore is independently decodable since the I-slice requires only the data
within the frame to predict any prediction unit or prediction block of the slice. A P-slice (uni-
directional predicted frames) is a slice of a picture that may be coded with intra-prediction and
with uni-directional inter-prediction. Each prediction unit or prediction block within a P-slice is
either coded with intra-prediction or inter-prediction. When the inter-prediction applies, the
prediction unit or prediction block is only predicted by one reference picture, and therefore
reference samples are only from one reference region of one frame. A B-slice (bi-directional
predictive frames) is a slice of a picture that may be coded with intra-prediction and with inter-
prediction (e.g., either bi-prediction or uni-prediction). A prediction unit or prediction block of a
B-slice may be bi-directionally predicted from two reference pictures, where each picture

14
Qualcomm Ref. 2207506TW

contributes one reference region and sample sets of the two reference regions are weighted (e.g.,
with equal weights or with different weights) to produce the prediction signal of the bi-directional
predicted block. As explained above, slices of one picture are independently coded. In some cases,
a picture can be coded as just one slice.

[0054] As noted above, intra-picture prediction of a picture utilizes the correlation between
spatially neighboring samples within the picture. There is a plurality of intra-prediction modes
(also referred to as “intra modes”). In some examples, the intra prediction of a luma block includes
35 modes, including the Planar mode, DC mode, and 33 angular modes (e.g., diagonal intra
prediction modes and angular modes adjacent to the diagonal intra prediction modes). The 35
modes of the intra prediction are indexed as shown in Table 1 below. In other examples, more intra
modes may be defined including prediction angles that may not already be represented by the 33
angular modes. In other examples, the prediction angles associated with the angular modes may
be different from those used in HEVC.

Intra-prediction mode Associated name

0 INTRA_PLANAR

1 INTRA_DC

2..34 INTRA_ANGULAR2..INTRA_ANGULAR34

Table 1 – Specification of intra prediction mode and associated names

[0055] Inter-picture prediction uses the temporal correlation between pictures in order to derive
a motion-compensated prediction for a current block of image samples. Using a translational
motion model, the position of a block in a previously decoded picture (a reference picture) is
indicated by a motion vector (∆𝑥𝑥, ∆𝑦𝑦), with ∆𝑥𝑥 specifying the horizontal displacement and ∆𝑦𝑦
specifying the vertical displacement of the reference block relative to the position of the current
block. In some cases, a motion vector (∆𝑥𝑥, ∆𝑦𝑦) can be in integer sample accuracy (also referred to
as integer accuracy), in which case the motion vector points to the integer-pel grid (or integer-pixel
sampling grid) of the reference frame. In some cases, a motion vector (∆𝑥𝑥, ∆𝑦𝑦) can be of fractional
sample accuracy (also referred to as fractional-pel accuracy or non-integer accuracy) to more

15
Qualcomm Ref. 2207506TW

accurately capture the movement of the underlying object, without being restricted to the integer-
pel grid of the reference frame. Accuracy of motion vectors may be expressed by the quantization
level of the motion vectors. For example, the quantization level may be integer accuracy (e.g., 1-
pixel) or fractional-pel accuracy (e.g., ¼-pixel, ½-pixel, or other sub-pixel value). Interpolation is
applied on reference pictures to derive the prediction signal when the corresponding motion vector
has fractional sample accuracy. For example, samples available at integer positions can be filtered
(e.g., using one or more interpolation filters) to estimate values at fractional positions. The
previously decoded reference picture is indicated by a reference index (refIdx) to a reference
picture list. The motion vectors and reference indices can be referred to as motion parameters. Two
kinds of inter-picture prediction can be performed, including uni-prediction and bi-prediction.

[0056] With inter-prediction using bi-prediction (also referred to as bi-directional inter-

prediction), two sets of motion parameters (∆𝑥𝑥0 , 𝑦𝑦0 , 𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟0 and ∆𝑥𝑥1 , 𝑦𝑦1 , 𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟1) are used to
generate two motion compensated predictions (from the same reference picture or possibly from
different reference pictures). For example, with bi-prediction, each prediction block uses two
motion compensated prediction signals, and generates B prediction units. The two motion
compensated predictions are then combined to get the final motion compensated prediction. For
example, the two motion compensated predictions can be combined by averaging. In another
example, weighted prediction can be used, in which case different weights can be applied to each
motion compensated prediction. The reference pictures that can be used in bi-prediction are stored
in two separate lists, denoted as list 0 and list 1. Motion parameters can be derived at the encoder
using a motion estimation process.

[0057] With inter-prediction using uni-prediction (also referred to as uni-directional inter-

prediction), one set of motion parameters (∆𝑥𝑥0 , 𝑦𝑦0 , 𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟0 ) is used to generate a motion
compensated prediction from a reference picture. For example, with uni-prediction, each
prediction block uses at most one motion compensated prediction signal, and generates P
prediction units.

[0058] A PU may include the data (e.g., motion parameters or other suitable data) related to the
prediction process. For example, when the PU is encoded using intra-prediction, the PU may
include data describing an intra-prediction mode for the PU. As another example, when the PU is

16
Qualcomm Ref. 2207506TW

encoded using inter-prediction, the PU may include data defining a motion vector for the PU. The
data defining the motion vector for a PU may describe, for example, a horizontal component of
the motion vector (∆𝑥𝑥), a vertical component of the motion vector (∆𝑦𝑦), a resolution for the motion
vector (e.g., integer precision, one-quarter pixel precision or one-eighth pixel precision), a
reference picture to which the motion vector points, a reference index, a reference picture list (e.g.,
List 0, List 1, or List C) for the motion vector, or any combination thereof.

[0059] AV1 includes two general techniques for encoding and decoding a coding block of video
data. The two general techniques are intra prediction (e.g., intra frame prediction or spatial
prediction) and inter prediction (e.g., inter frame prediction or temporal prediction). In the context
of AV1, when predicting blocks of a current frame of video data using an intra prediction mode,
encoder engine 106 and decoder engine 116 do not use video data from other frames of video data.
For most intra prediction modes, the encoding device 104 encodes blocks of a current frame based
on the difference between sample values in the current block and predicted values generated from
reference samples in the same frame. The encoding device 104 determines predicted values
generated from the reference samples based on the intra prediction mode.

[0060] After performing prediction using intra- and/or inter-prediction, the encoding device 104
can perform transformation and quantization. For example, following prediction, the encoder
engine 106 may calculate residual values corresponding to the PU. Residual values may comprise
pixel difference values between the current block of pixels being coded (the PU) and the prediction
block used to predict the current block (e.g., the predicted version of the current block). For
example, after generating a prediction block (e.g., issuing inter-prediction or intra-prediction), the
encoder engine 106 can generate a residual block by subtracting the prediction block produced by
a prediction unit from the current block. The residual block includes a set of pixel difference values
that quantify differences between pixel values of the current block and pixel values of the
prediction block. In some examples, the residual block may be represented in a two-dimensional
block format (e.g., a two-dimensional matrix or array of pixel values). In such examples, the
residual block is a two-dimensional representation of the pixel values.

[0061] Any residual data that may be remaining after prediction is performed is transformed
using a block transform, which may be based on discrete cosine transform, discrete sine transform,

17
Qualcomm Ref. 2207506TW

an integer transform, a wavelet transform, other suitable transform function, or any combination
thereof. In some cases, one or more block transforms (e.g., sizes 32 x 32, 16 x 16, 8 x 8, 4 x 4, or
other suitable size) may be applied to residual data in each CU. In some aspects, a TU may be used
for the transform and quantization processes implemented by the encoder engine 106. A given CU
having one or more PUs may also include one or more TUs. As described in further detail below,
the residual values may be transformed into transform coefficients using the block transforms, and
then may be quantized and scanned using TUs to produce serialized transform coefficients for
entropy coding.

[0062] In some aspects, following intra-predictive or inter-predictive coding using PUs of a CU,
the encoder engine 106 may calculate residual data for the TUs of the CU. The PUs may comprise
pixel data in the spatial domain (or pixel domain). The TUs may comprise coefficients in the
transform domain following application of a block transform. As previously noted, the residual
data may correspond to pixel difference values between pixels of the unencoded picture and
prediction values corresponding to the PUs. Encoder engine 106 may form the TUs including the
residual data for the CU, and may then transform the TUs to produce transform coefficients for
the CU.

[0063] The encoder engine 106 may perform quantization of the transform coefficients.
Quantization provides further compression by quantizing the transform coefficients to reduce the
amount of data used to represent the coefficients. For example, quantization may reduce the bit
depth associated with some or all of the coefficients. In one example, a coefficient with an n-bit
value may be rounded down to an m-bit value during quantization, with n being greater than m.

[0064] Once quantization is performed, the coded video bitstream includes quantized transform
coefficients, prediction information (e.g., prediction modes, motion vectors, block vectors, or the
like), partitioning information, and any other suitable data, such as other syntax data. The different
elements of the coded video bitstream may then be entropy encoded by the encoder engine 106. In
some examples, the encoder engine 106 may utilize a predefined scan order to scan the quantized
transform coefficients to produce a serialized vector that can be entropy encoded. In some
examples, encoder engine 106 may perform an adaptive scan. After scanning the quantized
transform coefficients to form a vector (e.g., a one-dimensional vector), the encoder engine 106

18
Qualcomm Ref. 2207506TW

may entropy encode the vector. For example, the encoder engine 106 may use context adaptive
variable length coding, context adaptive binary arithmetic coding, syntax-based context-adaptive
binary arithmetic coding, probability interval partitioning entropy coding, or another suitable
entropy encoding technique.

[0065] The output 110 of the encoding device 104 may send the NAL units making up the
encoded video bitstream data over the communications link 120 to the decoding device 112 of the
receiving device. The input 114 of the decoding device 112 may receive the NAL units. The
communications link 120 may include a channel provided by a wireless network, a wired network,
or a combination of a wired and wireless network. A wireless network may include any wireless
interface or combination of wireless interfaces and may include any suitable wireless network
(e.g., the Internet or other wide area network, a packet-based network, WiFiTM, radio frequency
(RF), ultra-wideband (UWB), WiFi-Direct, cellular, Long-Term Evolution (LTE), WiMaxTM, or
the like). A wired network may include any wired interface (e.g., fiber, ethernet, powerline
ethernet, ethernet over coaxial cable, digital signal line (DSL), or the like). The wired and/or
wireless networks may be implemented using various equipment, such as base stations, routers,
access points, bridges, gateways, switches, or the like. The encoded video bitstream data may be
modulated according to a communication standard, such as a wireless communication protocol,
and transmitted to the receiving device.

[0066] In some examples, the encoding device 104 may store encoded video bitstream data in
storage 108. The output 110 may retrieve the encoded video bitstream data from the encoder engine
106 or from the storage 108. Storage 108 may include any of a variety of distributed or locally
accessed data storage media. For example, the storage 108 may include a hard drive, a storage
disc, flash memory, volatile or non-volatile memory, or any other suitable digital storage media
for storing encoded video data. The storage 108 can also include a decoded picture buffer (DPB)
for storing reference pictures for use in inter-prediction. In a further example, the storage 108 can
correspond to a file server or another intermediate storage device that may store the encoded video
generated by the source device. In such cases, the receiving device including the decoding device
112 can access stored video data from the storage device via streaming or download. The file
server may be any type of server capable of storing encoded video data and transmitting that

19
Qualcomm Ref. 2207506TW

encoded video data to the receiving device. Example file servers include a web server (e.g., for a
website), an FTP server, network attached storage (NAS) devices, or a local disk drive. The
receiving device may access the encoded video data through any standard data connection,
including an Internet connection. This may include a wireless channel (e.g., a Wi-Fi connection),
a wired connection (e.g., DSL, cable modem, etc.), or a combination of both that is suitable for
accessing encoded video data stored on a file server. The transmission of encoded video data from
the storage 108 may be a streaming transmission, a download transmission, or a combination
thereof.

[0067] The input 114 of the decoding device 112 receives the encoded video bitstream data and
may provide the video bitstream data to the decoder engine 116, or to storage 118 for later use by
the decoder engine 116. For example, the storage 118 can include a DPB for storing reference
pictures for use in inter-prediction. The receiving device including the decoding device 112 can
receive the encoded video data to be decoded via the storage 108. The encoded video data may be
modulated according to a communication standard, such as a wireless communication protocol,
and transmitted to the receiving device. The communication medium for transmitting the encoded
video data can comprise any wireless or wired communication medium, such as a radio frequency
(RF) spectrum or one or more physical transmission lines. The communication medium may form
part of a packet-based network, such as a local area network, a wide-area network, or a global
network such as the Internet. The communication medium may include routers, switches, base
stations, or any other equipment that may be useful to facilitate communication from the source
device to the receiving device.

[0068] The decoder engine 116 may decode the encoded video bitstream data by entropy
decoding (e.g., using an entropy decoder) and extracting the elements of one or more coded video
sequences making up the encoded video data. The decoder engine 116 may then rescale and
perform an inverse transform on the encoded video bitstream data. Residual data is then passed to
a prediction stage of the decoder engine 116. The decoder engine 116 then predicts a current block
of pixels (e.g., a PU). In some examples, the prediction is added to the output of the inverse
transform (the residual data).

20
Qualcomm Ref. 2207506TW

[0069] The decoding device 112 may output the decoded video to a video destination device
122, which may include a display or other output device for displaying the decoded video data to
a consumer of the content. In some aspects, the video destination device 122 may be part of the
receiving device that includes the decoding device 112. In some aspects, the video destination
device 122 may be part of a separate device other than the receiving device.

[0070] In some aspects, the encoding device 104 and/or the decoding device 112 may be
integrated with an audio encoding device and audio decoding device, respectively. The encoding
device 104 and/or the decoding device 112 may also include other hardware or software that is
necessary to implement the coding techniques described above, such as one or more
microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs),
field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware or any
combinations thereof. The encoding device 104 and the decoding device 112 may be integrated as
part of a combined encoder/decoder (codec) in a respective device.

[0071] The example system shown in FIG. 1 is one illustrative example that can be used herein.
Techniques for processing video data using the techniques described herein can be performed by
any digital video encoding and/or decoding device. Although generally the techniques of this
disclosure are performed by a video encoding device or a video decoding device, the techniques
may also be performed by a combined video encoder-decoder, typically referred to as a “CODEC.”
Moreover, the techniques of this disclosure may also be performed by a video preprocessor. The
source device and the receiving device are merely examples of such coding devices in which the
source device generates coded video data for transmission to the receiving device. In some
examples, the source and receiving devices may operate in a substantially symmetrical manner
such that each of the devices include video encoding and decoding components. Hence, example
systems may support one-way or two-way video transmission between video devices, e.g., for
video streaming, video playback, video broadcasting, or video telephony.

[0072] Extensions to the HEVC standard include the Multiview Video Coding extension,
referred to as MV-HEVC, and the Scalable Video Coding extension, referred to as SHVC. The
MV-HEVC and SHVC extensions share the concept of layered coding, with different layers being
included in the encoded video bitstream. Each layer in a coded video sequence is addressed by a

21
Qualcomm Ref. 2207506TW

unique layer identifier (ID). A layer ID may be present in a header of a NAL unit to identify a
layer with which the NAL unit is associated. In MV-HEVC, different layers usually represent
different views of the same scene in the video bitstream. In SHVC, different scalable layers are
provided that represent the video bitstream in different spatial resolutions (or picture resolution)
or in different reconstruction fidelities. The scalable layers may include a base layer (with layer
ID = 0) and one or more enhancement layers (with layer IDs = 1, 2, … n). The base layer may
conform to a profile of the first version of HEVC, and represents the lowest available layer in a
bitstream. The enhancement layers have increased spatial resolution, temporal resolution or frame
rate, and/or reconstruction fidelity (or quality) as compared to the base layer. The enhancement
layers are hierarchically organized and may (or may not) depend on lower layers. In some
examples, the different layers may be coded using a single standard codec (e.g., all layers are
encoded using HEVC, SHVC, or other coding standard). In some examples, different layers may
be coded using a multi-standard codec. For example, a base layer may be coded using AVC, while
one or more enhancement layers may be coded using SHVC and/or MV-HEVC extensions to the
HEVC standard.

[0073] In general, a layer includes a set of VCL NAL units and a corresponding set of non-VCL
NAL units. The NAL units are assigned a particular layer ID value. Layers can be hierarchical in
the sense that a layer may depend on a lower layer. A layer set refers to a set of layers represented
within a bitstream that are self-contained, meaning that the layers within a layer set can depend on
other layers in the layer set in the decoding process, but do not depend on any other layers for
decoding. Accordingly, the layers in a layer set can form an independent bitstream that can
represent video content. The set of layers in a layer set may be obtained from another bitstream by
operation of a sub-bitstream extraction process. A layer set may correspond to the set of layers that
is to be decoded when a decoder wants to operate according to certain parameters.

[0074] As previously described, an HEVC bitstream includes a group of NAL units, including
VCL NAL units and non-VCL NAL units. VCL NAL units include coded picture data forming a
coded video bitstream. For example, a sequence of bits forming the coded video bitstream is
present in VCL NAL units. Non-VCL NAL units may contain parameter sets with high-level
information relating to the encoded video bitstream, in addition to other information. For example,

22
Qualcomm Ref. 2207506TW

a parameter set may include a video parameter set (VPS), a sequence parameter set (SPS), and a
picture parameter set (PPS). Examples of goals of the parameter sets include bit rate efficiency,
error resiliency, and providing systems layer interfaces. Each slice references a single active PPS,
SPS, and VPS to access information that the decoding device 112 may use for decoding the slice.
An identifier (ID) may be coded for each parameter set, including a VPS ID, an SPS ID, and a PPS
ID. An SPS includes an SPS ID and a VPS ID. A PPS includes a PPS ID and an SPS ID. Each
slice header includes a PPS ID. Using the IDs, active parameter sets can be identified for a given
slice.

[0075] A PPS includes information that applies to all slices in a given picture. Because of this,
all slices in a picture refer to the same PPS. Slices in different pictures may also refer to the same
PPS. An SPS includes information that applies to all pictures in a same coded video sequence
(CVS) or bitstream. As previously described, a coded video sequence is a series of access units
(AUs) that starts with a random access point picture (e.g., an instantaneous decode reference (IDR)
picture or broken link access (BLA) picture, or other appropriate random access point picture) in
the base layer and with certain properties (described above) up to and not including a next AU that
has a random access point picture in the base layer and with certain properties (or the end of the
bitstream). The information in an SPS may not change from picture to picture within a coded video
sequence. Pictures in a coded video sequence may use the same SPS. The VPS includes
information that applies to all layers within a coded video sequence or bitstream. The VPS includes
a syntax structure with syntax elements that apply to entire coded video sequences. In some
aspects, the VPS, SPS, or PPS may be transmitted in-band with the encoded bitstream. In some
aspects, the VPS, SPS, or PPS may be transmitted out-of-band in a separate transmission than the
NAL units containing coded video data.

[0076] This disclosure may generally refer to “signaling” certain information, such as syntax
elements. The term “signaling” may generally refer to the communication of values for syntax
elements and/or other data used to decode encoded video data. For example, the encoding device
104 may signal values for syntax elements in the bitstream. In general, signaling refers to
generating a value in the bitstream. As noted above, video source 102 may transport the bitstream

23
Qualcomm Ref. 2207506TW

to video destination device 122 substantially in real time, or not in real time, such as might occur
when storing syntax elements to storage 108 for later retrieval by the video destination device 122.

[0077] A video bitstream can also include Supplemental Enhancement Information (SEI)
messages. For example, an SEI NAL unit can be part of the video bitstream. In some cases, an SEI
message can contain information that is not needed by the decoding process. For example, the
information in an SEI message may not be essential for the decoder to decode the video pictures
of the bitstream, but the decoder can be use the information to improve the display or processing
of the pictures (e.g., the decoded output). The information in an SEI message can be embedded
metadata. In one illustrative example, the information in an SEI message could be used by decoder-
side entities to improve the viewability of the content. In some instances, certain application
standards may mandate the presence of such SEI messages in the bitstream so that the
improvement in quality can be brought to all devices that conform to the application standard (e.g.,
the carriage of the frame-packing SEI message for frame-compatible plano-stereoscopic 3DTV
video format, where the SEI message is carried for every frame of the video, handling of a recovery
point SEI message, use of pan-scan scan rectangle SEI message in DVB, in addition to many other
examples).

[0078] FIG. 2 is a block diagram illustrating an example architecture 200 of a video coding
hardware engine. In some cases, the architecture 200 can be implemented by the encoding device
104 and/or decoding device 112 shown in FIG. 1. In some examples, the architecture 200 can be
implemented by the encoder engine 106 of the encoding device 104 or by the decoder engine 116
of the decoding device 112, as shown in FIG. 1.

[0079] In this example, the architecture 200 of the video coding hardware engine can include a
control processor 210, an interface 222, a video stream processor (VSP) 212, processing pipelines
214-220 (also referred to as “pipes”), a direct memory access (DMA) subsystem 230, and one or
more buffers 232. In some examples, the architecture 200 can include memory 240 for storing data
such as frames, videos, coding information, outputs, etc. In other examples, the memory 240 can
be external memory on the coding device implementing the video coding hardware engine.

24
Qualcomm Ref. 2207506TW

[0080] The interface 222 can transfer data between components of the video coding hardware
engine and/or the video coding device through a communication system or system bus on the video
coding hardware engine and/or the coding device implementing the video coding hardware engine.
For example, the interface 222 can connect the control processor 210, VSP 212, processing
pipelines 214-220 (e.g., video pixel processor (VPP)), DMA subsystem 230, and/or one or more
buffers 232 with a system bus on the video coding hardware engine and/or the coding device. In
some examples, the interface 222 can include a network-based communications subsystem, such
as a network-on-chip (NoC).

[0081] The DMA subsystem 230 can allow other components of the video coding hardware
engine (e.g., other components in the architecture 200) to access memory on the video coding
hardware engine and/or the video coding device implementing the video coding hardware engine.
For example, the DMA subsystem 230 can provide access to the memory 240 and/or the one or
more buffers 232. In some examples, the DMA subsystem 230 can manage access to common
memory units and associated data traffic (e.g., tile 202, blocks 204A-D, bitstream 236, entropy
decoded data 238, etc.).

[0082] The memory 240 can include one or more internal or external memory devices such as,
for example and without limitation, one or more random access memory (RAM) components, read-
only memory (ROM) components, cache memory components, buffer components, and/or other
memory devices. The memory 240 can store data used by the video coding hardware engine and/or
the video coding device, such as frames, processing parameters, input data, output data, and/or any
other type of data.

[0083] The control processor 210 can include one or more processors. The control processor 210
can control and/or program components of the video coding hardware engine (e.g., other
components in the architecture 200). In some examples, the control processor 210 can interface
with other drivers, applications, and/or components that are not shown in FIG. 2. For example, in
some cases, the control processor 210 can interface with an application processor on the video
coding device.

25
Qualcomm Ref. 2207506TW

[0084] The VSP 212 can perform bitstream parsing (e.g., separating a network abstraction layer,
a picture layer, and a slice layer) and entropy coding operations. In some examples, the VSP 212
can perform coding functions such as variable length encoding or decoding. For example, the VSP
212 can implement a lossless compression/decompression algorithm to compress or decompress a
bitstream 236. In some examples, the VSP 212 can perform arithmetic coding, such as context,
adaptive binary arithmetic coding (CABAC), and/or any other coding algorithm.

[0085] The processing pipelines 214-220 can perform video pixel operations such as motion
estimation, motion compensation, transform and quantization, image deblocking, and/or any other
video pixel operations. In some cases, the processing pipelines 214-220 may perform video pixel
operations based on output of the VSP 212. In some cases, output of one VSP 212 may be
processed by multiple processing pipelines 214-220. The processing pipelines 214-220 (and/or
each individual processing pipeline) can perform specific video pixel operations in parallel. For
example, each processing pipeline can perform multiple operations (and/or process data)
simultaneously and/or significantly in parallel. As another example, multiple processing pipelines
can perform operations (and/or process data) simultaneously and/or significantly in parallel.

[0086] In FIG. 2, the processing pipelines 214-220 can store and retrieve video pixel processing
data (e.g., video pixel processing outputs, inputs, parameters, pixel data, processing
synchronization data, etc.) to and from the one or more buffers 232. In some cases, the one or more
buffers 232 can include a single buffer. In other cases, the one or more buffers 232 can include
multiple buffers. In some examples, the one or more buffers 232 can include a global input/output
line buffer and a pipeline synchronization buffer. In some cases, the pipeline synchronization
buffer can temporarily store data used to synchronize data and/or results from video pixel
processing operations performed by the processing pipelines 214-220.

[0087] In some examples, the VSP 212 can decompress a bitstream 236 associated with a video
or sequence of frames, and store entropy decoded data 238 associated with the bitstream 236 for
processing by the processing pipelines 214-220. In some cases, the entropy decoded data 238 may
be stored in a memory or buffer and this memory or buffer may be a part of, or separate from
buffer 232. In some cases, the VSP 212 can retrieve the bitstream 236 and store the entropy
decoded data 238 to and from memory using the DMA subsystem 230, which can manage access

26
Qualcomm Ref. 2207506TW

to memory components and/or units as previously noted. In some cases, the VSP 212 may store
the decoded data in an order based on the bitstream. For example, where the bitstream organizes
image information based on tiles, the decoded data may be grouped such that decoded data for a
tile is stored together, in an order that the tiles are decoded (e.g., in tile order, also referred to as
bitstream order). The processing pipelines 214-220 can retrieve the entropy decoded data 238 (e.g.,
via the DMA subsystem 230) and perform video pixel processing operations on blocks 204A-D of
a tile 202 associated with the bitstream 236.

[0088] The processing pipelines 214-220 can perform video pixel processing operations in
parallel, as previously described. The processing pipelines 214-220 can retrieve and store video
pixel processing inputs and outputs from/in the one or more buffers 232 (e.g., via DMA subsystem
230). For example, a motion estimation algorithm implemented by the processing pipeline 214 can
perform motion estimation on block 204A and store motion estimation information calculated for
block 204A in the one or more buffers 232. A motion compensation algorithm implemented by
the processing pipeline 214 can retrieve the motion estimation information from the one or more
buffers 232, and use the motion estimation information to perform motion compensation for block
204A. While the motion compensation algorithm is performing the motion compensation, the
motion estimation algorithm can perform motion estimation for a next block.

[0089] The motion compensation algorithm can store motion compensation results in the one or
more buffers 232, which can be accessed and used by transform, quantization, and deblocking
algorithms to perform transform, quantization and deblocking for the block 204A. The motion
compensation algorithm can perform motion compensation for a next block while the transform,
quantization, and/or deblocking algorithms perform the transform, quantization and/or deblocking
for the block 204A. The transform, quantization and deblocking algorithms can similarly perform
respective operations for the block 204A and the next block in parallel. In some examples, the
motion estimation, motion compensation, transform, quantization, and deblocking algorithms can
perform respective operations on different blocks in parallel.

[0090] The processing pipelines 214-220 can be implemented by hardware and/or software
components. For example, the processing pipelines 214-220 can be implemented by one or more
pixel processors. In some examples, each processing pipeline can be implemented by one or more

27
Qualcomm Ref. 2207506TW

hardware components. In some cases, each processing pipeline can use different hardware units
and/or components to implement different stages in a pipeline of the processing pipeline. After the
video pixel operations are performed to generate output pixels for display, the output pixels for
display may be output to a memory, such as the memory 240 or the one or more buffers 232, such
as a display buffer. In some cases, the memory 204 may be a system memory or similar memory
device, such as a double data rate (DDR) synchronous dynamic random-access memory (SDRAM)
or any other memory device. The memory 240 may store the output pixels pending display on a
display device.

[0091] The number of processing pipelines shown in FIG. 2 is merely an example provided for
explanation purposes. One of ordinary skill in the art will appreciate that the architecture 200 can
include more or less processing pipelines than shown in FIG. 2. For example, the number of
processing pipelines implemented by the architecture 200 can be increased or reduced to include
more or less processing pipelines. Moreover, while the architecture 200 is shown to include certain
components, one of ordinary skill will appreciate that the architecture 200 can include more or
fewer components than those shown in FIG. 2. For example, the architecture 200 can also include,
in some instances, other memory devices (e.g., one or more random access memory (RAM)
components, read-only memory (ROM) components, cache memory components, buffer
components, database components, and/or other memory devices), processing devices (e.g., one
or more CPUs, GPUs, and/or other processing devices), interfaces (e.g., internal bus, etc.), and/or
other components that are not shown in FIG. 2.

[0092] FIG. 3 is a block diagram illustrating an example architecture 300 of a video coding
system. In architecture 300, application software 302 may direct video firmware 304 and video
hardware 306 to decode a bitstream 308 to memory 310 for downstream device 312 (e.g., a display
device, network device to transmit the decoded image to a display device, and the like). In some
cases, the application software 302 may be a driver, operating system, higher level user software,
and the like. In some cases, the application software 302 may be executing on a CPU or other
general purpose processor. The application software 302 may indicate to the video firmware 304
to decode bitstream 308. In some cases, the video firmware 304 may be a control processor for
video firmware 304, such as control processor 210 of FIG. 2. The video hardware 306 may include

28
Qualcomm Ref. 2207506TW

video hardware components for processing video data, such as components from FIG. 2 including
VSP 212, processing pipelines 214-220, DMA subsystem 230, interface 222, and the like.

[0093] The video firmware 304 may configure the video hardware 306 to obtain and decode the
bitstream 308. In some cases, as the video hardware 306 decodes the bitstream 308 into portions
of the image, the video hardware 306 may store the portions of the one or more image in the
memory 310. In some examples, memory 310 may be similar to memory 240 of FIG. 2. In some
cases, after an image is decoded and ready for display, the image may be stored in the memory
310 by the video hardware 306. The video hardware 306 may also send an interrupt 320 to the
video firmware 304 indicating that the image is ready for display. The video firmware 304 send
an interrupt 322 to the application software 302 indicating the image is ready for display. The
application software 302 may receive the interrupt 322 and the application software 302 may
indicate 324 to the downstream device 312 to obtain 326 the decoded image for display. In some
cases, the downstream device 312 may obtain (e.g., receive) 326 the decoded image from memory
310.

[0094] FIG. 4 is a block diagram illustrating a mapping of image data in grid 400 to a bitstream,
in accordance with aspects of the present disclosure. In FIG. 4, image data correlating to each pixel
may be represented by individual squares of a grid 400, which are arranged, in grid 400, as the
pixels would be displayed. Thus, square 402 represents image data for an upper left hand corner
of the image. In some cases, image data representing pixels of an image may be sent to a
downstream device in a raster scan order. In raster scan order, image data for pixels may be sent
starting from the left-hand side of a top row and proceed down the row until the last (e.g.,
rightmost) pixel. Then image data for pixels of the next row may be sent, again staring from the
left-hand side. Thus, image data for a pixel represented by square 402 may be sent first, then square
404 and on down to square 406. After image data for square 406 is sent, image data for square 408
is sent and so forth until all of the image data is sent. In some cases, sending all of the image data
may take a certain amount of time. In some cases, if image data is sent in response to the interrupt
to indicate that the image is ready, a latency for displaying the image would include, in addition to
an amount of time to render the image, an amount of time for processing the interrupt and an
amount of time to send all of the image data. In some cases, sending an interrupt before the entire

29
Qualcomm Ref. 2207506TW

image is rendered may provide early notification for a low latency video decoder. The early
interrupt (e.g., notification) may allow the decoding/rendering of the image from the bitstream and
transmitting of the image data to be overlapped, thus reducing latency.

[0095] FIG. 5 is a block diagram illustrating an example architecture 500 of an early notification
technique for a low latency video coding system, in accordance with aspects of the present
disclosure. In architecture 500, application software 502 may direct video firmware 504 and video
hardware 506 to decode a bitstream 508 to memory 510 for downstream device 512. In some cases,
the application software 502 may be a driver, operating system, higher level user software, and the
like. In some cases, the application software 502 may be executing on a CPU or other general
purpose processor. The application software 502 may indicate to the video firmware 504 to decode
bitstream 508. In some cases, the video firmware 504 may be a control processor for video
firmware 504, such as control processor 210 of FIG. 2. The video hardware 506 may include video
hardware components for processing video data, such as components from FIG. 2 including VSP
212, processing pipelines 214-220, DMA subsystem 230, interface 222, and the like.

[0096] The application software 502 may send, to the video firmware 504, timing information
550 for the decoding. For example, the application software 502 may send an indication of a target
latency for decoded images from the bitstream 508. For example, the application software 502
may indicate a target latency of 5 milliseconds (ms). Based on the target latency, the video
firmware 504 may determine a portion of the image that may be updated for the target latency. For
example, if the images are being processed at 60 frames per second (FPS) or one image every 16.6
milliseconds (ms) at a resolution of 1920x1080, number of rows of an image that may be 1080 x
5/16.6 = 325.3. In some cases, the number of rows may be rounded downward, for example, based
on a size of a tile, CTU, or other portion of an image. For example, the number of rows may be
rounded downward to 320 rows. Based on the number of rows, the image may be divided into set
of rows (e.g., partial images). For example, the 180 total rows may be divided into four sets of
nonoverlapping rows (e.g., nonoverlapping partial images), the first set including a first 320 rows,
a second set including rows 321-640, a third set including rows 641-960, and a fourth set including
rows 961-1080. The video firmware 504 may configure the video hardware 506 to obtain and
decode the bitstream 508. The video firmware 504 may also indicate to the video hardware 506 a

30
Qualcomm Ref. 2207506TW

number of rows 552 to process before sending an interrupt. While discussed in a context of an
interrupt, it may be understood that any hardware notification system may be used, such as flags,
memory bits, etc. For example, rather than sending an interrupt to indicate an image is ready to be
displayed (e.g., output to a downstream device), a flag or memory bit may be set to indicate the
image is ready.

[0097] In some cases, the video hardware 506 may and decode the bitstream 508 in raster order
and stored the decoded portions of the image in the memory 510 as the bitstream 508 is decoded.
In some cases, a first interrupt 556 may be sent by the video hardware 506 to the video firmware
504 based on the indicated number of rows, such as a first partial image 554, being decoded and
sent to the memory 510 for storage. The video firmware 504 may send a second interrupt 558 to
the application software 502 to indicate that the first partial image 554 of the decoded image is
ready. In some cases, the video firmware 504 may continue to decode the bitstream 508. The
application software 502 may receive the second interrupt 558 and the application software 502
may indicate 524 to the downstream device 512 to obtain 526 the first partial image 554 of the
decoded image for display.

[0098] In some cases, the downstream device 512 may obtain 526 the first partial image 554 in
a raster scan order. As indicated above, the video hardware 506 may continue to decode a second
partial image 560 from the bitstream 508 while the downstream device 512 obtains 526 the first
partial image 554. In some cases, prior to the downstream device 512 obtaining 526 all of the first
partial image 554, the video hardware 506 may decode the rows of the second partial image 560
(e.g., rows 321-640) and store the decoded second partial image 560 to memory 510. The video
hardware 506 may send a third interrupt 562 to the video firmware 504 to indicate that the second
partial image 560 are ready. The video firmware 504 may send a fourth interrupt 564 to the
application software 502 to indicate that the second partial image 560 of the decoded image is
ready. The application software 502 may indicate 572 to the downstream device 512 that the
second partial image 560 of the decoded image are ready. The downstream device 512 may then
obtain 566 the second partial image 560. The above-described technique may then be repeated
with the third partial image 568 and fourth partial image 570.

31
Qualcomm Ref. 2207506TW

[0099] In some cases, in loop filtering may be performed as an image is decoded. The in loop
filtering may be filtering that is applied in a coding loop. In some cases, in loop filtering may be
used to smooth pixel transitions, or to otherwise improve the video quality. Examples of in loop
filters may include a deblocking filter, an adaptive loop filter (ALF), a sample adaptive offset
(SAO) filter, a cross-component adaptive loop filter, and the like. In some cases, after a first row
is decoded, in-loop filtering may be applied to pixels of the first row based on pixels in a second
row below the first row in raster scan order.

[0100] FIG. 6 illustrates an example application of in-loop filtering to an image 600, in

accordance with aspects of the present disclosure. Image 600 includes at least CTU A 602, CTU
B 604, CTU x 606, and CTU y 608. Each CTU includes a number of rows N. For simplicity, a
single column 610 of pixels with multiple rows is shown. Operations performed for rows of this
single column 610 of pixels is intended to be representative of operations performed across the
row. In some cases, row N-2, row N-1, and row N of CTU A 602 may be decoded in raster scan
order. Row 1, row 2, and row 3 of CTU B 604 may then be decoded and in loop filtering applied
along the edge of CTU B 604, for example, to smooth transitions across different CTUs. This in
loop filtering, such as a deblocking filter, may cause pixels of rows N-2, N-1, N of CTU A 602 to
be changed.

[0101] As discussed above, for early notifications, if an edge of a partial image is aligned with
an edge of a CTU, such as CTU A, an interrupt may be sent after row N of CTU A 602 is decoded
and stored in memory indicating the partial image are ready for display. This interrupt may be sent
before portions (e.g., rows 1, 2, and 3) of CTU B 604 are decoded and in-loop filtering applied.
Thus, the early notification may, in some cases, cause the interrupt to be sent before in-loop
filtering can be applied. This may cause visual artifacts along boundaries of sets of rows. Rather
than turning off in loop filtering, the early notification may be adapted to accommodate for in loop
filtering.

[0102] In some cases, for early notification to be adapted to accommodate in loop filtering, the
early notification (e.g., interrupt) may be delayed, as compared to when a last row of the partial
image is decoded, until after the in loop filtering is complete across the rows of the partial image.
For example, assuming that a first partial image includes rows N-2, N-1, and N of CTU A 602

32
Qualcomm Ref. 2207506TW

while a second partial image includes rows 1, 2, and 3 of CTU B 604, rows N-2, N-1, and N of
CTU A 602 may be decoded. Rows 1, 2, and 3 of CTU B 604 may also be decoded and in loop
filtering applied to the edges of CTU B 604 in accordance with the in loop filter being used. Any
changes to pixels in rows N-2, N-1, and N of CTU A 602 based on the in loop filtering may be
made. After the in loop filtering is completed, the decoded first partial image, including rows N-
2, N-1, and N of CTU A 602, may be output to memory and the interrupt indicating that the first
partial image are ready for display may be sent.

[0103] As discussed above, in some cases, the number of rows for a partial image may be based
on a tile size. In some cases, this tile size may defined based on a coding applied to the image. In
some cases, the tile size may also be defined, for some codecs, to help optimize reading/writing
image data to a memory and/or cache. Thus, a codec may have fixed tile sizes for certain
resolutions and/or bit rates. As an example, some non-codec tiles (e.g., data tile format for
optimizing image storage in memory) may have, for a 1920x1080 resolution with an 8 bit pixel
depth, a resolution of 256x64 and a 1920x1080 image may include 17 such tiles.

[0104] FIG. 7 is an example illustrating how partial images for early notifications interact with
image tiles for an image 700, in accordance with aspects of the present disclosure. Image 700 may
have a resolution of 1920x1080. In some cases, the application software may indicate a target
latency that would result in two partial images. For an image with a height of 1080 pixels, the
number of rows of pixels for two partial images would be 540 rows of pixels. In some cases, the
number of rows in a partial image may be based on the latency as well as a tile height or other
partition height. For a non-codec tile, the tile height may be 64 rows of pixels and thus the number
of rows of pixels for the partial image may be a factor of the number of rows in the tile (e.g., 64).
This helps avoid a splitting a tile across multiple partial images.

[0105] In some cases, the tile size may be selected to aid memory transactions and/or cache
coherency. It may be useful to avoid tiles being in multiple partial images. In some cases, the
number of rows of pixels may be rounded down from 540 to a nearest factor of the tile height.
Thus, in this example, the 540 rows may be rounded down to 512 rows (e.g., 64x8) and image 700
may be split into three partial images, partial image 1 702, partial image 2 704, and partial image

33
Qualcomm Ref. 2207506TW

3 706. In some cases, rounding down the number of rows in the partial image helps ensure the
target latency is met. As shown, partial images may have a different number of rows of pixels.

[0106] FIG. 8 is a flow diagram for a process 800 for processing video data, in accordance with
aspects of the present disclosure. The process 800 may be performed by a computing device (or
apparatus) or a component (e.g., a chipset, codec, etc.) of the computing device. The computing
device may be a mobile device (e.g., a mobile phone), a network-connected wearable such as a
watch, an extended reality (XR) device such as a virtual reality (VR) device or augmented reality
(AR) device, a vehicle or component or system of a vehicle, or other type of computing device.
The operations of the process 800 may be implemented as software components that are executed
and run on one or more processors.

[0107] At block 802, the computing device (or component thereof) may determine a number of
rows of pixels for one or more portions (e.g., partial image) of an image. The computing device
(or component thereof) may obtain an indication of a target latency (e.g., from an application
executing on the computing device). The computing device (or component thereof) may determine
the number of rows of pixels for the first portion of the image based on the target latency. The
computing device (or component thereof) may determine the number of rows of pixels for the first
portion of the image further based on a partition height of the first encoded data. In some cases,
the number of rows of pixels for the first portion of the image is aligned with the partition height
of the first encoded data.

[0108] At block 804, the computing device (or component thereof) may obtain first encoded
data for a first portion of the image.

[0109] At block 806, the computing device (or component thereof) may decode the first encoded
data to generate first pixel data for the number of rows of pixels of the first portion of the image.

[0110] At block 808, the computing device (or component thereof) may output the first pixel
data for the first portion of the image to the memory. The computing device (or component thereof)
may obtain second encoded data for a second portion of the image. In some cases, the second
portion of the image is nonoverlapping with the first portion of the image. The computing device
(or component thereof) may decode the second encoded data to generate second pixel data for the

34
Qualcomm Ref. 2207506TW

number of rows of pixels for the second portion of the image. The computing device (or component
thereof) may output the second pixel data for the second portion of the image to the memory. The
computing device (or component thereof) may output an indication that the second portion of the
image is available.

[0111] At block 810, the computing device (or component thereof) may output an indication that
the first portion of the image is available. In some cases, the indication comprises at least one of
an interrupt, a flag, or a memory bit. The computing device (or component thereof) may output
the indication that the first portion of the image is available after decoding the first encoded data
and before decoding the second encoded data. The computing device (or component thereof) may
delay output of the indication until at least a part of the second encoded data is decoded. The
computing device (or component thereof) may apply an in loop filter to update one or more rows
of pixels of the first portion of the image based on the second pixel data. In some cases, the delayed
output is based on an amount of time to apply the in loop filter.

[0112] The processes (or methods) described herein can be used individually or in any
combination. In some implementations, the processes (or methods) described herein can be
performed by a computing device or an apparatus, such as the system 100 shown in FIG. 1. For
example, the processes can be performed by the decoding device 112 shown in FIG. 1 and FIG.
10, and/or by another client-side device, such as a player device, a display, or any other client-side
device. In some cases, the computing device or apparatus may include one or more input devices,
one or more output devices, one or more processors, one or more microprocessors, one or more
microcomputers, and/or other component(s) that is/are configured to carry out the steps of one or
more processes described herein.

[0113] In some examples, the computing device may include a mobile device, a desktop
computer, a server computer and/or server system, or other type of computing device. The
components of the computing device (e.g., the one or more input devices, one or more output
devices, one or more processors, one or more microprocessors, one or more microcomputers,
and/or other component) can be implemented in circuitry. For example, the components can
include and/or can be implemented using electronic circuits or other electronic hardware, which
can include one or more programmable electronic circuits (e.g., microprocessors, graphics

35
Qualcomm Ref. 2207506TW

processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and/or
other suitable electronic circuits), and/or can include and/or be implemented using computer
software, firmware, or any combination thereof, to perform the various operations described
herein. In some examples, the computing device or apparatus may include a camera configured to
capture video data (e.g., a video sequence) including video frames. In some examples, a camera or
other capture device that captures the video data is separate from the computing device, in which
case the computing device receives or obtains the captured video data. The computing device may
include a network interface configured to communicate the video data. The network interface may
be configured to communicate Internet Protocol (IP) based data or other type of data. In some
examples, the computing device or apparatus may include a display for displaying output video
content, such as samples of pictures of a video bitstream.

[0114] The processes can be described with respect to logical flow diagrams, the operation of
which represent a sequence of operations that can be implemented in hardware, computer
instructions, or a combination thereof. In the context of computer instructions, the operations
represent computer-executable instructions stored on one or more computer-readable storage
media that, when executed by one or more processors, perform the recited operations. Generally,
computer-executable instructions include routines, programs, objects, components, data structures,
and the like that perform particular functions or implement particular data types. The order in
which the operations are described is not intended to be construed as a limitation, and any number
of the described operations can be combined in any order and/or in parallel to implement the
processes.

[0115] Additionally, the processes may be performed under the control of one or more computer
systems configured with executable instructions and may be implemented as code (e.g., executable
instructions, one or more computer programs, or one or more applications) executing collectively
on one or more processors, by hardware, or combinations thereof. As noted above, the code may
be stored on a computer-readable or machine-readable storage medium, for example, in the form
of a computer program comprising a plurality of instructions executable by one or more processors.
The computer-readable or machine-readable storage medium may be non-transitory.

36
Qualcomm Ref. 2207506TW

[0116] The coding techniques discussed herein may be implemented in an example video
encoding and decoding system (e.g., system 100). In some examples, a system includes a source
device that provides encoded video data to be decoded at a later time by a destination device. In
particular, the source device provides the video data to destination device via a computer-readable
medium. The source device and the destination device may comprise any of a wide range of
devices, including desktop computers, notebook (i.e., laptop) computers, tablet computers, set-top
boxes, telephone handsets such as so-called “smart” phones, so-called “smart” pads, televisions,
cameras, display devices, digital media players, video gaming consoles, video streaming device,
or the like. In some cases, the source device and the destination device may be equipped for
wireless communication.

[0117] The destination device may receive the encoded video data to be decoded via the
computer-readable medium. The computer-readable medium may comprise any type of medium
or device capable of moving the encoded video data from source device to destination device. In
one example, computer-readable medium may comprise a communication medium to enable
source device to transmit encoded video data directly to destination device in real-time. The
encoded video data may be modulated according to a communication standard, such as a wireless
communication protocol, and transmitted to destination device. The communication medium may
comprise any wireless or wired communication medium, such as a radio frequency (RF) spectrum
or one or more physical transmission lines. The communication medium may form part of a packet-
based network, such as a local area network, a wide-area network, or a global network such as the
Internet. The communication medium may include routers, switches, base stations, or any other
equipment that may be useful to facilitate communication from source device to destination device.

[0118] In some examples, encoded data may be output from output interface to a storage device.
Similarly, encoded data may be accessed from the storage device by input interface. The storage
device may include any of a variety of distributed or locally accessed data storage media such as
a hard drive, Blu-ray discs, DVDs, CD-ROMs, flash memory, volatile or non-volatile memory, or
any other suitable digital storage media for storing encoded video data. In a further example, the
storage device may correspond to a file server or another intermediate storage device that may
store the encoded video generated by source device. Destination device may access stored video

37
Qualcomm Ref. 2207506TW

data from the storage device via streaming or download. The file server may be any type of server
capable of storing encoded video data and transmitting that encoded video data to the destination
device. Example file servers include a web server (e.g., for a website), an FTP server, network
attached storage (NAS) devices, or a local disk drive. Destination device may access the encoded
video data through any standard data connection, including an Internet connection. This may
include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., DSL, cable modem,
etc.), or a combination of both that is suitable for accessing encoded video data stored on a file
server. The transmission of encoded video data from the storage device may be a streaming
transmission, a download transmission, or a combination thereof.

[0119] The techniques of this disclosure are not necessarily limited to wireless applications or
settings. The techniques may be applied to video coding in support of any of a variety of
multimedia applications, such as over-the-air television broadcasts, cable television transmissions,
satellite television transmissions, Internet streaming video transmissions, such as dynamic
adaptive streaming over HTTP (DASH), digital video that is encoded onto a data storage medium,
decoding of digital video stored on a data storage medium, or other applications. In some examples,
system may be configured to support one-way or two-way video transmission to support
applications such as video streaming, video playback, video broadcasting, and/or video telephony.

[0120] In one example the source device includes a video source, a video encoder, and a output
interface. The destination device may include an input interface, a video decoder, and a display
device. The video encoder of source device may be configured to apply the techniques disclosed
herein. In other examples, a source device and a destination device may include other components
or arrangements. For example, the source device may receive video data from an external video
source, such as an external camera. Likewise, the destination device may interface with an external
display device, rather than including an integrated display device.

[0121] The example system above is merely one example. Techniques for processing video data
in parallel may be performed by any digital video encoding and/or decoding device. Although
generally the techniques of this disclosure are performed by a video encoding device, the
techniques may also be performed by a video encoder/decoder, typically referred to as a
“CODEC.” Moreover, the techniques of this disclosure may also be performed by a video

38
Qualcomm Ref. 2207506TW

preprocessor. Source device and destination device are merely examples of such coding devices
in which source device generates coded video data for transmission to destination device. In some
examples, the source and destination devices may operate in a substantially symmetrical manner
such that each of the devices include video encoding and decoding components. Hence, example
systems may support one-way or two-way video transmission between video devices, e.g., for
video streaming, video playback, video broadcasting, or video telephony.

[0122] The video source may include a video capture device, such as a video camera, a video
archive containing previously captured video, and/or a video feed interface to receive video from
a video content provider. As a further alternative, the video source may generate computer
graphics-based data as the source video, or a combination of live video, archived video, and
computer-generated video. In some cases, if video source is a video camera, source device and
destination device may form so-called camera phones or video phones. As mentioned above,
however, the techniques described in this disclosure may be applicable to video coding in general,
and may be applied to wireless and/or wired applications. In each case, the captured, pre-captured,
or computer-generated video may be encoded by the video encoder. The encoded video
information may then be output by output interface onto the computer-readable medium.

[0123] As noted the computer-readable medium may include transient media, such as a wireless
broadcast or wired network transmission, or storage media (that is, non-transitory storage media),
such as a hard disk, flash drive, compact disc, digital video disc, Blu-ray disc, or other computer-
readable media. In some examples, a network server (not shown) may receive encoded video data
from the source device and provide the encoded video data to the destination device, e.g., via
network transmission. Similarly, a computing device of a medium production facility, such as a
disc stamping facility, may receive encoded video data from the source device and produce a disc
containing the encoded video data. Therefore, the computer-readable medium may be understood
to include one or more computer-readable media of various forms, in various examples.

[0124] The input interface of the destination device receives information from the computer-
readable medium. The information of the computer-readable medium may include syntax
information defined by the video encoder, which is also used by the video decoder, that includes
syntax elements that describe characteristics and/or processing of blocks and other coded units,

39
Qualcomm Ref. 2207506TW

e.g., group of pictures (GOP). A display device displays the decoded video data to a user, and may
comprise any of a variety of display devices such as a cathode ray tube (CRT), a liquid crystal
display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type
of display device. Various aspects of the application have been described.

[0125] Specific details of the encoding device 104 and the decoding device 112 are shown in
FIG. 9 and FIG. 10, respectively. FIG. 9 is a block diagram illustrating an example encoding device
104 that may implement one or more of the techniques described in this disclosure. Encoding
device 104 may, for example, generate the syntax elements and/or structures described herein (e.g.,
the syntax elements and/or structures of a green metadata, such as Complexity Metrics (CM), or
other syntax elements and/or structures). Encoding device 104 may perform intra-prediction and
inter-prediction coding of video blocks within video slices, tiles, sub-pictures, etc. As previously
described, intra-coding relies, at least in part, on spatial prediction to reduce or remove spatial
redundancy within a given video frame or picture. Inter-coding relies, at least in part, on temporal
prediction to reduce or remove temporal redundancy within adjacent or surrounding frames of a
video sequence. Intra-mode (I mode) may refer to any of several spatial based compression modes.
Inter-modes, such as uni-directional prediction (P mode) or bi-prediction (B mode), may refer to
any of several temporal-based compression modes.

[0126] The encoding device 104 includes a partitioning unit 35, prediction processing unit 41,
filter unit 63, picture memory 64, summer 50, transform processing unit 52, quantization unit 54,
and entropy encoding unit 56. Prediction processing unit 41 includes motion estimation unit 42,
motion compensation unit 44, and intra-prediction processing unit 46. For video block
reconstruction, encoding device 104 also includes inverse quantization unit 58, inverse transform
processing unit 60, and summer 62. Filter unit 63 is intended to represent one or more loop filters
such as a deblocking filter, an adaptive loop filter (ALF), and a sample adaptive offset (SAO) filter.
Although filter unit 63 is shown in FIG. 6 as being an in loop filter, in other configurations, filter
unit 63 may be implemented as a post loop filter. A post processing device 57 may perform
additional processing on encoded video data generated by the encoding device 104. The techniques
of this disclosure may in some instances be implemented by the encoding device 104. In other

40
Qualcomm Ref. 2207506TW

instances, however, one or more of the techniques of this disclosure may be implemented by post
processing device 57.

[0127] As shown in FIG. 9, the encoding device 104 receives video data, and partitioning unit
35 partitions the data into video blocks. The partitioning may also include partitioning into slices,
slice segments, tiles, or other larger units, as wells as video block partitioning, e.g., according to a
quadtree structure of LCUs and CUs. The encoding device 104 generally illustrates the
components that encode video blocks within a video slice to be encoded. The slice may be divided
into multiple video blocks (and possibly into sets of video blocks referred to as tiles). Prediction
processing unit 41 may select one of a plurality of possible coding modes, such as one of a plurality
of intra-prediction coding modes or one of a plurality of inter-prediction coding modes, for the
current video block based on error results (e.g., coding rate and the level of distortion, or the like).
Prediction processing unit 41 may provide the resulting intra- or inter-coded block to summer 50
to generate residual block data and to summer 62 to reconstruct the encoded block for use as a
reference picture.

[0128] Intra-prediction processing unit 46 within prediction processing unit 41 may perform
intra-prediction coding of the current video block relative to one or more neighboring blocks in
the same frame or slice as the current block to be coded to provide spatial compression. Motion
estimation unit 42 and motion compensation unit 44 within prediction processing unit 41 perform
inter-predictive coding of the current video block relative to one or more predictive blocks in one
or more reference pictures to provide temporal compression.

[0129] Motion estimation unit 42 may be configured to determine the inter-prediction mode for
a video slice according to a predetermined pattern for a video sequence. The predetermined pattern
may designate video slices in the sequence as P slices, B slices, or GPB slices. Motion estimation
unit 42 and motion compensation unit 44 may be highly integrated, but are illustrated separately
for conceptual purposes. Motion estimation, performed by motion estimation unit 42, is the process
of generating motion vectors, which estimate motion for video blocks. A motion vector, for
example, may indicate the displacement of a prediction unit (PU) of a video block within a current
video frame or picture relative to a predictive block within a reference picture.

41
Qualcomm Ref. 2207506TW

[0130] A predictive block is a block that is found to closely match the PU of the video block to
be coded in terms of pixel difference, which may be determined by sum of absolute difference
(SAD), sum of square difference (SSD), or other difference metrics. In some examples, the
encoding device 104 may calculate values for sub-integer pixel positions of reference pictures
stored in picture memory 64. For example, the encoding device 104 may interpolate values of one-
quarter pixel positions, one-eighth pixel positions, or other fractional pixel positions of the
reference picture. Therefore, motion estimation unit 42 may perform a motion search relative to
the full pixel positions and fractional pixel positions and output a motion vector with fractional
pixel precision.

[0131] Motion estimation unit 42 calculates a motion vector for a PU of a video block in an
inter-coded slice by comparing the position of the PU to the position of a predictive block of a
reference picture. The reference picture may be selected from a first reference picture list (List 0)
or a second reference picture list (List 1), each of which identify one or more reference pictures
stored in picture memory 64. Motion estimation unit 42 sends the calculated motion vector to
entropy encoding unit 56 and motion compensation unit 44.

[0132] Motion compensation, performed by motion compensation unit 44, may involve fetching
or generating the predictive block based on the motion vector determined by motion estimation,
possibly performing interpolations to sub-pixel precision. Upon receiving the motion vector for
the PU of the current video block, motion compensation unit 44 may locate the predictive block to
which the motion vector points in a reference picture list. The encoding device 104 forms a residual
video block by subtracting pixel values of the predictive block from the pixel values of the current
video block being coded, forming pixel difference values. The pixel difference values form
residual data for the block, and may include both luma and chroma difference components.
Summer 50 represents the component or components that perform this subtraction operation.
Motion compensation unit 44 may also generate syntax elements associated with the video blocks
and the video slice for use by the decoding device 112 in decoding the video blocks of the video
slice.

[0133] Intra-prediction processing unit 46 may intra-predict a current block, as an alternative to

the inter-prediction performed by motion estimation unit 42 and motion compensation unit 44, as

42
Qualcomm Ref. 2207506TW

described above. In particular, intra-prediction processing unit 46 may determine an intra-

prediction mode to use to encode a current block. In some examples, intra-prediction processing
unit 46 may encode a current block using various intra-prediction modes, e.g., during separate
encoding passes, and intra-prediction processing unit 46 may select an appropriate intra-prediction
mode to use from the tested modes. For example, intra-prediction processing unit 46 may calculate
rate-distortion values using a rate-distortion analysis for the various tested intra-prediction modes,
and may select the intra-prediction mode having the best rate-distortion characteristics among the
tested modes. Rate-distortion analysis generally determines an amount of distortion (or error)
between an encoded block and an original, unencoded block that was encoded to produce the
encoded block, as well as a bit rate (that is, a number of bits) used to produce the encoded block.
Intra-prediction processing unit 46 may calculate ratios from the distortions and rates for the
various encoded blocks to determine which intra-prediction mode exhibits the best rate-distortion
value for the block.

[0134] In any case, after selecting an intra-prediction mode for a block, intra-prediction
processing unit 46 may provide information indicative of the selected intra-prediction mode for
the block to entropy encoding unit 56. Entropy encoding unit 56 may encode the information
indicating the selected intra-prediction mode. The encoding device 104 may include in the
transmitted bitstream configuration data definitions of encoding contexts for various blocks as well
as indications of a most probable intra-prediction mode, an intra-prediction mode index table, and
a modified intra-prediction mode index table to use for each of the contexts. The bitstream
configuration data may include a plurality of intra-prediction mode index tables and a plurality of
modified intra-prediction mode index tables (also referred to as codeword mapping tables).

[0135] After prediction processing unit 41 generates the predictive block for the current video
block via either inter-prediction or intra-prediction, the encoding device 104 forms a residual video
block by subtracting the predictive block from the current video block. The residual video data in
the residual block may be included in one or more TUs and applied to transform processing unit
52. Transform processing unit 52 transforms the residual video data into residual transform
coefficients using a transform, such as a discrete cosine transform (DCT) or a conceptually similar

43
Qualcomm Ref. 2207506TW

transform. Transform processing unit 52 may convert the residual video data from a pixel domain
to a transform domain, such as a frequency domain.

[0136] Transform processing unit 52 may send the resulting transform coefficients to
quantization unit 54. Quantization unit 54 quantizes the transform coefficients to further reduce
bit rate. The quantization process may reduce the bit depth associated with some or all of the
coefficients. The degree of quantization may be modified by adjusting a quantization parameter.
In some examples, quantization unit 54 may then perform a scan of the matrix including the
quantized transform coefficients. Alternatively, entropy encoding unit 56 may perform the scan.

[0137] Following quantization, entropy encoding unit 56 entropy encodes the quantized
transform coefficients. For example, entropy encoding unit 56 may perform context adaptive
variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), syntax-
based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy
(PIPE) coding or another entropy encoding technique. Following the entropy encoding by entropy
encoding unit 56, the encoded bitstream may be transmitted to the decoding device 112, or
archived for later transmission or retrieval by the decoding device 112. Entropy encoding unit 56
may also entropy encode the motion vectors and the other syntax elements for the current video
slice being coded.

[0138] Inverse quantization unit 58 and inverse transform processing unit 60 apply inverse
quantization and inverse transformation, respectively, to reconstruct the residual block in the pixel
domain for later use as a reference block of a reference picture. Motion compensation unit 44 may
calculate a reference block by adding the residual block to a predictive block of one of the reference
pictures within a reference picture list. Motion compensation unit 44 may also apply one or more
interpolation filters to the reconstructed residual block to calculate sub-integer pixel values for use
in motion estimation. Summer 62 adds the reconstructed residual block to the motion compensated
prediction block produced by motion compensation unit 44 to produce a reference block for
storage in picture memory 64. The reference block may be used by motion estimation unit 42 and
motion compensation unit 44 as a reference block to inter-predict a block in a subsequent video
frame or picture.

44
Qualcomm Ref. 2207506TW

[0139] In this manner, the encoding device 104 of FIG. 9 represents an example of a video
encoder configured to perform any of the techniques described herein. In some cases, some of the
techniques of this disclosure may also be implemented by post processing device 57.

[0140] FIG. 10 is a block diagram illustrating an example decoding device 112. The decoding
device 112 includes an entropy decoding unit 80, prediction processing unit 81, inverse
quantization unit 86, inverse transform processing unit 88, summer 90, filter unit 91, and picture
memory 92. Prediction processing unit 81 includes motion compensation unit 82 and intra
prediction processing unit 84. The decoding device 112 may, in some examples, perform a
decoding pass generally reciprocal to the encoding pass described with respect to the encoding
device 104 from FIG. 6.

[0141] During the decoding process, the decoding device 112 receives an encoded video
bitstream that represents video blocks of an encoded video slice and associated syntax elements
sent by the encoding device 104. In some aspects, the decoding device 112 may receive the
encoded video bitstream from the encoding device 104. In some aspects, the decoding device 112
may receive the encoded video bitstream from a network entity 79, such as a server, a media-aware
network element (MANE), a video editor/splicer, or other such device configured to implement
one or more of the techniques described above. Network entity 79 may or may not include the
encoding device 104. Some of the techniques described in this disclosure may be implemented by
network entity 79 prior to network entity 79 transmitting the encoded video bitstream to the
decoding device 112. In some video decoding systems, network entity 79 and the decoding device
112 may be parts of separate devices, while in other instances, the functionality described with
respect to network entity 79 may be performed by the same device that comprises the decoding
device 112.

[0142] The entropy decoding unit 80 of the decoding device 112 entropy decodes the bitstream
to generate quantized coefficients, motion vectors, and other syntax elements. Entropy decoding
unit 80 forwards the motion vectors and other syntax elements to prediction processing unit 81.
The decoding device 112 may receive the syntax elements at the video slice level and/or the video
block level. Entropy decoding unit 80 may process and parse both fixed-length syntax elements
and variable-length syntax elements in or more parameter sets, such as a VPS, SPS, and PPS.

45
Qualcomm Ref. 2207506TW

[0143] When the video slice is coded as an intra-coded (I) slice, intra prediction processing unit
84 of prediction processing unit 81 may generate prediction data for a video block of the current
video slice based on a signaled intra-prediction mode and data from previously decoded blocks of
the current frame or picture. When the video frame is coded as an inter-coded (i.e., B, P or GPB)
slice, motion compensation unit 82 of prediction processing unit 81 produces predictive blocks for
a video block of the current video slice based on the motion vectors and other syntax elements
received from entropy decoding unit 80. The predictive blocks may be produced from one of the
reference pictures within a reference picture list. The decoding device 112 may construct the
reference frame lists, List 0 and List 1, using default construction techniques based on reference
pictures stored in picture memory 92.

[0144] Motion compensation unit 82 determines prediction information for a video block of the
current video slice by parsing the motion vectors and other syntax elements, and uses the prediction
information to produce the predictive blocks for the current video block being decoded. For
example, motion compensation unit 82 may use one or more syntax elements in a parameter set to
determine a prediction mode (e.g., intra- or inter-prediction) used to code the video blocks of the
video slice, an inter-prediction slice type (e.g., B slice, P slice, or GPB slice), construction
information for one or more reference picture lists for the slice, motion vectors for each inter-
encoded video block of the slice, inter-prediction status for each inter-coded video block of the
slice, and other information to decode the video blocks in the current video slice.

[0145] Motion compensation unit 82 may also perform interpolation based on interpolation
filters. Motion compensation unit 82 may use interpolation filters as used by the encoding device
104 during encoding of the video blocks to calculate interpolated values for sub-integer pixels of
reference blocks. In this case, motion compensation unit 82 may determine the interpolation filters
used by the encoding device 104 from the received syntax elements, and may use the interpolation
filters to produce predictive blocks.

[0146] Inverse quantization unit 86 inverse quantizes, or de-quantizes, the quantized transform
coefficients provided in the bitstream and decoded by entropy decoding unit 80. The inverse
quantization process may include use of a quantization parameter calculated by the encoding
device 104 for each video block in the video slice to determine a degree of quantization and,

46
Qualcomm Ref. 2207506TW

likewise, a degree of inverse quantization that should be applied. Inverse transform processing unit
88 applies an inverse transform (e.g., an inverse DCT or other suitable inverse transform), an
inverse integer transform, or a conceptually similar inverse transform process, to the transform
coefficients in order to produce residual blocks in the pixel domain.

[0147] After motion compensation unit 82 generates the predictive block for the current video
block based on the motion vectors and other syntax elements, the decoding device 112 forms a
decoded video block by summing the residual blocks from inverse transform processing unit 88
with the corresponding predictive blocks generated by motion compensation unit 82. Summer 90
represents the component or components that perform this summation operation. If desired, loop
filters (either in the coding loop or after the coding loop) may also be used to smooth pixel
transitions, or to otherwise improve the video quality. Filter unit 91 is intended to represent one or
more loop filters such as a deblocking filter, an adaptive loop filter (ALF), and a sample adaptive
offset (SAO) filter. Although filter unit 91 is shown in FIG. 10 as being an in loop filter, in other
configurations, filter unit 91 may be implemented as a post loop filter. The decoded video blocks
in a given frame or picture are then stored in picture memory 92, which stores reference pictures
used for subsequent motion compensation. Picture memory 92 also stores decoded video for later
presentation on a display device, such as video destination device 122 shown in FIG. 1.

[0148] In this manner, the decoding device 112 of FIG. 10 represents an example of a video
decoder configured to perform any of the techniques described herein.

[0149] As used herein, the term “computer-readable medium” includes, but is not limited to,
portable or non-portable storage devices, optical storage devices, and various other mediums
capable of storing, containing, or carrying instruction(s) and/or data. A computer-readable medium
may include a non-transitory medium in which data can be stored and that does not include carrier
waves and/or transitory electronic signals propagating wirelessly or over wired connections.
Examples of a non-transitory medium may include, but are not limited to, a magnetic disk or tape,
optical storage media such as compact disk (CD) or digital versatile disk (DVD), flash memory,
memory or memory devices. A computer-readable medium may have stored thereon code and/or
machine-executable instructions that may represent a procedure, a function, a subprogram, a
program, a routine, a subroutine, a module, a software package, a class, or any combination of

47
Qualcomm Ref. 2207506TW

instructions, data structures, or program statements. A code segment may be coupled to another
code segment or a hardware circuit by passing and/or receiving information, data, arguments,
parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed,
forwarded, or transmitted via any suitable means including memory sharing, message passing,
token passing, network transmission, or the like.

[0150] In some aspects the computer-readable storage devices, mediums, and memories can
include a cable or wireless signal containing a bit stream and the like. However, when mentioned,
non-transitory computer-readable storage media expressly exclude media such as energy, carrier
signals, electromagnetic waves, and signals per se.

[0151] Specific details are provided in the description above to provide a thorough
understanding of the aspects and examples provided herein. However, it will be understood by one
of ordinary skill in the art that the aspects may be practiced without these specific details. For
clarity of explanation, in some instances the present technology may be presented as including
individual functional blocks including functional blocks comprising devices, device components,
steps or routines in a method embodied in software, or combinations of hardware and software.
Additional components may be used other than those shown in the figures and/or described herein.
For example, circuits, systems, networks, processes, and other components may be shown as
components in block diagram form in order not to obscure the aspects in unnecessary detail. In
other instances, well-known circuits, processes, algorithms, structures, and techniques may be
shown without unnecessary detail in order to avoid obscuring the aspects.

[0152] Individual aspects may be described above as a process or method which is depicted as
a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram.
Although a flowchart may describe the operations as a sequential process, many of the operations
can be performed in parallel or concurrently. In addition, the order of the operations may be re-
arranged. A process is terminated when its operations are completed, but could have additional
steps not included in a figure. A process may correspond to a method, a function, a procedure, a
subroutine, a subprogram, etc. When a process corresponds to a function, its termination can
correspond to a return of the function to the calling function or the main function.

48
Qualcomm Ref. 2207506TW

[0153] Processes and methods according to the above-described examples can be implemented
using computer-executable instructions that are stored or otherwise available from computer-
readable media. Such instructions can include, for example, instructions and data which cause or
otherwise configure a general purpose computer, special purpose computer, or a processing device
to perform a certain function or group of functions. Portions of computer resources used can be
accessible over a network. The computer executable instructions may be, for example, binaries,
intermediate format instructions such as assembly language, firmware, source code, etc. Examples
of computer-readable media that may be used to store instructions, information used, and/or
information created during methods according to described examples include magnetic or optical
disks, flash memory, USB devices provided with non-volatile memory, networked storage devices,
and so on.

[0154] Devices implementing processes and methods according to these disclosures can include
hardware, software, firmware, middleware, microcode, hardware description languages, or any
combination thereof, and can take any of a variety of form factors. When implemented in software,
firmware, middleware, or microcode, the program code or code segments to perform the necessary
tasks (e.g., a computer-program product) may be stored in a computer-readable or machine-
readable medium. A processor(s) may perform the necessary tasks. Typical examples of form
factors include laptops, smart phones, mobile phones, tablet devices or other small form factor
personal computers, personal digital assistants, rackmount devices, standalone devices, and so on.
Functionality described herein also can be embodied in peripherals or add-in cards. Such
functionality can also be implemented on a circuit board among different chips or different
processes executing in a single device, by way of further example.

[0155] The instructions, media for conveying such instructions, computing resources for
executing them, and other structures for supporting such computing resources are example means
for providing the functions described in the disclosure.

[0156] In the foregoing description, aspects of the application are described with reference to
specific aspects thereof, but those skilled in the art will recognize that the application is not limited
thereto. Thus, while illustrative aspects of the application have been described in detail herein, it
is to be understood that the inventive concepts may be otherwise variously embodied and

49
Qualcomm Ref. 2207506TW

employed, and that the appended claims are intended to be construed to include such variations,
except as limited by the prior art. Various features and aspects of the above-described application
may be used individually or jointly. Further, aspects can be utilized in any number of environments
and applications beyond those described herein without departing from the broader spirit and scope
of the specification. The specification and drawings are, accordingly, to be regarded as illustrative
rather than restrictive. For the purposes of illustration, methods were described in a particular
order. It should be appreciated that in alternate aspects, the methods may be performed in a
different order than that described.

[0157] One of ordinary skill will appreciate that the less than (“<”) and greater than (“>”)
symbols or terminology used herein can be replaced with less than or equal to (“≤”) and greater
than or equal to (“≥”) symbols, respectively, without departing from the scope of this description.

[0158] Where components are described as being “configured to” perform certain operations,
such configuration can be accomplished, for example, by designing electronic circuits or other
hardware to perform the operation, by programming programmable electronic circuits (e.g.,
microprocessors, or other suitable electronic circuits) to perform the operation, or any combination
thereof.

[0159] The phrase “coupled to” refers to any component that is physically connected to another
component either directly or indirectly, and/or any component that is in communication with
another component (e.g., connected to the other component over a wired or wireless connection,
and/or other suitable communication interface) either directly or indirectly.

[0160] Claim language or other language in the disclosure reciting “at least one of” a set and/or
“one or more” of a set indicates that one member of the set or multiple members of the set (in any
combination) satisfy the claim. For example, claim language reciting “at least one of A and B”
means A, B, or A and B. In another example, claim language reciting “at least one of A, B, and C”
means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language “at least one
of” a set and/or “one or more” of a set does not limit the set to the items listed in the set. For
example, claim language reciting “at least one of A and B” can mean A, B, or A and B, and can
additionally include items not listed in the set of A and B.

50
Qualcomm Ref. 2207506TW

[0161] The various illustrative logical blocks, modules, circuits, and algorithm steps described
in connection with the aspects disclosed herein may be implemented as electronic hardware,
computer software, firmware, or combinations thereof. To clearly illustrate this interchangeability
of hardware and software, various illustrative components, blocks, modules, circuits, and steps
have been described above generally in terms of their functionality. Whether such functionality is
implemented as hardware or software depends upon the particular application and design
constraints imposed on the overall system. Skilled artisans may implement the described
functionality in varying ways for each particular application, but such implementation decisions
should not be interpreted as causing a departure from the scope of the present application.

[0162] The techniques described herein may also be implemented in electronic hardware,
computer software, firmware, or any combination thereof. Such techniques may be implemented
in any of a variety of devices such as general purposes computers, wireless communication device
handsets, or integrated circuit devices having multiple uses including application in wireless
communication device handsets and other devices. Any features described as modules or
components may be implemented together in an integrated logic device or separately as discrete
but interoperable logic devices. If implemented in software, the techniques may be realized at least
in part by a computer-readable data storage medium comprising program code including
instructions that, when executed, performs one or more of the methods described above. The
computer-readable data storage medium may form part of a computer program product, which
may include packaging materials. The computer-readable medium may comprise memory or data
storage media, such as random access memory (RAM) such as synchronous dynamic random
access memory (SDRAM), read-only memory (ROM), non-volatile random access memory
(NVRAM), electrically erasable programmable read-only memory (EEPROM), FLASH memory,
magnetic or optical data storage media, and the like. The techniques additionally, or alternatively,
may be realized at least in part by a computer-readable communication medium that carries or
communicates program code in the form of instructions or data structures and that can be accessed,
read, and/or executed by a computer, such as propagated signals or waves.

[0163] The program code may be executed by a processor, which may include one or more
processors, such as one or more digital signal processors (DSPs), general purpose microprocessors,

51
Qualcomm Ref. 2207506TW

an application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or
other equivalent integrated or discrete logic circuitry. Such a processor may be configured to
perform any of the techniques described in this disclosure. A general purpose processor may be a
microprocessor; but in the alternative, the processor may be any conventional processor, controller,
microcontroller, or state machine. A processor may also be implemented as a combination of
computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of
microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such
configuration. Accordingly, the term “processor,” as used herein may refer to any of the foregoing
structure, any combination of the foregoing structure, or any other structure or apparatus suitable
for implementation of the techniques described herein. In addition, in some aspects, the
functionality described herein may be provided within dedicated software modules or hardware
modules configured for encoding and decoding, or incorporated in a combined video encoder-
decoder (CODEC).

[0164] Illustrative aspects of the disclosure include:

[0165] Aspect 1. An apparatus for processing video data, comprising: a memory; and a processor
coupled to a memory, the processor being configured to: determine a number of rows of pixels for
one or more portions of an image; obtain first encoded data for a first portion of the image; decode
the first encoded data to generate first pixel data for the number of rows of pixels of the first portion
of the image; output the first pixel data for the first portion of the image to the memory; and output
an indication that the first portion of the image is available.

[0166] Aspect 2. The apparatus of Aspect 1, wherein the processor is further configured to:
obtain an indication of a target latency; and determine the number of rows of pixels for the first
portion of the image based on the target latency.

[0167] Aspect 3. The apparatus of Aspect 2, wherein the processor is configured to determine
the number of rows of pixels for the first portion of the image further based on a partition height
of the first encoded data.

[0168] Aspect 4. The apparatus of Aspect 3, wherein the number of rows of pixels for the first
portion of the image is aligned with the partition height of the first encoded data.

52
Qualcomm Ref. 2207506TW

[0169] Aspect 5. The apparatus of any of Aspects 1-4, wherein the indication comprises at least
one of an interrupt, a flag, or a memory bit.

[0170] Aspect 6. The apparatus of any of Aspects 1-5, wherein the processor is further
configured to: obtain second encoded data for a second portion of the image, wherein the second
portion of the image is nonoverlapping with the first portion of the image; decode the second
encoded data to generate second pixel data for the number of rows of pixels for the second portion
of the image; output the second pixel data for the second portion of the image to the memory; and
output an indication that the second portion of the image is available.

[0171] Aspect 7. The apparatus of Aspect 6, wherein the processor is configured to output the
indication that the first portion of the image is available after decoding the first encoded data and
before decoding the second encoded data.

[0172] Aspect 8. The apparatus of Aspect 6, wherein the processor is further configured to delay
output of the indication until at least a part of the second encoded data is decoded.

[0173] Aspect 9. The apparatus of Aspect 8, wherein the processor is further configured to apply
an in loop filter to update one or more rows of pixels of the first portion of the image based on the
second pixel data, and wherein the delayed output is based on an amount of time to apply the in
loop filter.

[0174] Aspect 10. The apparatus of any of Aspects 1-9, wherein the apparatus comprises a
decoder device.

[0175] Aspect 11. A method for processing video data, comprising: determining a number of
rows of pixels for one or more portions of an image; obtaining first encoded data for a first portion
of the image; decoding the first encoded data to generate first pixel data for the number of rows of
pixels of the first portion of the image; outputting the first pixel data for the first portion of the
image to a memory; and outputting an indication that the first portion of the image is available.

[0176] Aspect 12. The method of Aspect 11, further comprising: obtain an indication of a target
latency; and determining the number of rows of pixels for the first portion of the image based on
the target latency.

53
Qualcomm Ref. 2207506TW

[0177] Aspect 13. The method of Aspect 12, wherein determining the number of rows of pixels
for the first portion of the image is further based on a partition height of the first encoded data.

[0178] Aspect 14. The method of Aspect 13, wherein the number of rows of pixels for the first
portion of the image is aligned with the partition height of the first encoded data.

[0179] Aspect 15. The method of any of Aspects 11-14, wherein the indication comprises at
least one of an interrupt, a flag, or a memory bit.

[0180] Aspect 16. The method of any of Aspects 11-15, further comprising: obtaining second
encoded data for a second portion of the image, wherein the second portion of the image is
nonoverlapping with the first portion of the image; decoding the second encoded data to generate
second pixel data for the number of rows of pixels for the second portion of the image; outputting
the second pixel data for the second portion of the image to the memory; and outputting an
indication that the second portion of the image is available.

[0181] Aspect 17. The method of Aspect 16, further comprising outputting the indication that
the first portion of the image is available after decoding the first encoded data and before decoding
the second encoded data.

[0182] Aspect 18. The method of Aspect 16, further comprising delaying output of the indication
until at least a part of the second encoded data is decoded.

[0183] Aspect 19. The method of Aspect 18, further comprising applying an in loop filter to
update one or more rows of pixels of the first portion of the image based on the second pixel data,
and wherein the delayed output is based on an amount of time to apply the in loop filter.

[0184] Aspect 20. A non-transitory computer-readable medium having stored thereon

instructions that, when executed by a processor, cause the processor to: determine a number of
rows of pixels for one or more portions of an image; obtain first encoded data for a first portion of
the image; decode the first encoded data to generate first pixel data for the number of rows of
pixels of the first portion of the image; output the first pixel data for the first portion of the image
to a memory; and output an indication that the first portion of the image is available.

54
Qualcomm Ref. 2207506TW

[0185] Aspect 21. The non-transitory computer-readable medium of Aspect 20, wherein the
instructions further cause the processor to: obtain an indication of a target latency; and determine
the number of rows of pixels for the first portion of the image based on the target latency.

[0186] Aspect 22. The non-transitory computer-readable medium of Aspect 21, wherein the
instructions further cause the processor to determine the number of rows of pixels for the first
portion of the image further based on a partition height of the first encoded data.

[0187] Aspect 23. The non-transitory computer-readable medium of Aspect 22, wherein the
number of rows of pixels for the first portion of the image is aligned with the partition height of
the first encoded data.

[0188] Aspect 24. The non-transitory computer-readable medium of any of Aspects 20-23,
wherein the indication comprises at least one of an interrupt, a flag, or a memory bit.

[0189] Aspect 25. The non-transitory computer-readable medium of any of Aspects 20-24,
wherein the instructions further cause the processor to: obtain second encoded data for a second
portion of the image, wherein the second portion of the image is nonoverlapping with the first
portion of the image; decode the second encoded data to generate second pixel data for the number
of rows of pixels for the second portion of the image; output the second pixel data for the second
portion of the image to the memory; and output an indication that the second portion of the image
is available.

[0190] Aspect 26. The non-transitory computer-readable medium of Aspect 25, wherein the
instructions further cause the processor to output the indication that the first portion of the image
is available after decoding the first encoded data and before decoding the second encoded data.

[0191] Aspect 27. The non-transitory computer-readable medium of Aspect 25, wherein the
instructions further cause the processor to delay output of the indication until at least a part of the
second encoded data is decoded.

[0192] Aspect 28. The non-transitory computer-readable medium of Aspect 27, wherein the
instructions further cause the processor to apply an in loop filter to update one or more rows of

55
Qualcomm Ref. 2207506TW

pixels of the first portion of the image based on the second pixel data, and wherein the delayed
output is based on an amount of time to apply the in loop filter.

[0193] Aspect 29. An apparatus for processing video data, comprising: means for determining a
number of rows of pixels for one or more portions of an image; means for obtaining first encoded
data for a first portion of the image; means for decoding the first encoded data to generate first
pixel data for the number of rows of pixels of the first portion of the image; means for outputting
the first pixel data for the first portion of the image to a memory; and means for outputting an
indication that the first portion of the image is available.

[0194] Aspect 30. The apparatus of Aspect 29, further comprising: means for obtain an
indication of a target latency; and means for determining the number of rows of pixels for the first
portion of the image based on the target latency.

[0195] Aspect 32: An apparatus for processing video data, comprising one or more means for
performing operations according to Aspects 12 to 19.

56
Qualcomm Ref. 2207506TW

CLAIMS

WHAT IS CLAIMED IS:

1. An apparatus for processing video data, comprising:

a memory; and
a processor coupled to a memory, the processor being configured to:
determine a number of rows of pixels for one or more portions of an image;
obtain first encoded data for a first portion of the image;
decode the first encoded data to generate first pixel data for the number of rows of
pixels of the first portion of the image;
output the first pixel data for the first portion of the image to the memory; and
output an indication that the first portion of the image is available.

2. The apparatus of claim 1, wherein the processor is further configured to:

obtain an indication of a target latency; and
determine the number of rows of pixels for the first portion of the image based on the target
latency.

3. The apparatus of claim 2, wherein the processor is configured to determine the number of
rows of pixels for the first portion of the image further based on a partition height of the first
encoded data.

4. The apparatus of claim 3, wherein the number of rows of pixels for the first portion of the
image is aligned with the partition height of the first encoded data.

5. The apparatus of claim 1, wherein the indication comprises at least one of an interrupt, a
flag, or a memory bit.

6. The apparatus of claim 1, wherein the processor is further configured to:

57
Qualcomm Ref. 2207506TW

obtain second encoded data for a second portion of the image, wherein the second portion
of the image is nonoverlapping with the first portion of the image;
decode the second encoded data to generate second pixel data for the number of rows of
pixels for the second portion of the image;
output the second pixel data for the second portion of the image to the memory; and
output an indication that the second portion of the image is available.

7. The apparatus of claim 6, wherein the processor is configured to output the indication that
the first portion of the image is available after decoding the first encoded data and before decoding
the second encoded data.

8. The apparatus of claim 6, wherein the processor is further configured to delay output of the
indication until at least a part of the second encoded data is decoded.

9. The apparatus of claim 8, wherein the processor is further configured to apply an in loop
filter to update one or more rows of pixels of the first portion of the image based on the second
pixel data, and wherein the delayed output is based on an amount of time to apply the in loop filter.

10. The apparatus of claim 1, wherein the apparatus comprises a decoder device.

11. A method for processing video data, comprising:

determining a number of rows of pixels for one or more portions of an image;
obtaining first encoded data for a first portion of the image;
decoding the first encoded data to generate first pixel data for the number of rows of
pixels of the first portion of the image;
outputting the first pixel data for the first portion of the image to a memory; and
outputting an indication that the first portion of the image is available.

12. The method of claim 11, further comprising:

obtain an indication of a target latency; and

58
Qualcomm Ref. 2207506TW

determining the number of rows of pixels for the first portion of the image based on the
target latency.

13. The method of claim 12, wherein determining the number of rows of pixels for the first
portion of the image is further based on a partition height of the first encoded data.

14. The method of claim 13, wherein the number of rows of pixels for the first portion of the
image is aligned with the partition height of the first encoded data.

15. The method of claim 11, wherein the indication comprises at least one of an interrupt, a
flag, or a memory bit.

16. The method of claim 11, further comprising:

obtaining second encoded data for a second portion of the image, wherein the second
portion of the image is nonoverlapping with the first portion of the image;
decoding the second encoded data to generate second pixel data for the number of rows
of pixels for the second portion of the image;
outputting the second pixel data for the second portion of the image to the memory; and
outputting an indication that the second portion of the image is available.

17. The method of claim 16, further comprising outputting the indication that the first portion
of the image is available after decoding the first encoded data and before decoding the second
encoded data.

18. The method of claim 16, further comprising delaying output of the indication until at
least a part of the second encoded data is decoded.

19. The method of claim 18, further comprising applying an in loop filter to update one or
more rows of pixels of the first portion of the image based on the second pixel data, and wherein
the delayed output is based on an amount of time to apply the in loop filter.

59
Qualcomm Ref. 2207506TW

20. A non-transitory computer-readable medium having stored thereon instructions that,

when executed by a processor, cause the processor to:
determine a number of rows of pixels for one or more portions of an image;
obtain first encoded data for a first portion of the image;
decode the first encoded data to generate first pixel data for the number of rows of pixels
of the first portion of the image;
output the first pixel data for the first portion of the image to a memory; and output an
indication that the first portion of the image is available.

21. The non-transitory computer-readable medium of claim 20, wherein the instructions
further cause the processor to:
obtain an indication of a target latency; and
determine the number of rows of pixels for the first portion of the image based on the
target latency.

22. The non-transitory computer-readable medium of claim 21, wherein the instructions
further cause the processor to determine the number of rows of pixels for the first portion of the
image further based on a partition height of the first encoded data.

23. The non-transitory computer-readable medium of claim 22, wherein the number of rows
of pixels for the first portion of the image is aligned with the partition height of the first encoded
data.

24. The non-transitory computer-readable medium of claim 20, wherein the indication
comprises at least one of an interrupt, a flag, or a memory bit.

25. The non-transitory computer-readable medium of claim 20, wherein the instructions
further cause the processor to:

60
Qualcomm Ref. 2207506TW

obtain second encoded data for a second portion of the image, wherein the second portion
of the image is nonoverlapping with the first portion of the image;
decode the second encoded data to generate second pixel data for the number of rows of
pixels for the second portion of the image;
output the second pixel data for the second portion of the image to the memory; and
output an indication that the second portion of the image is available.

26. The non-transitory computer-readable medium of claim 25, wherein the instructions
further cause the processor to output the indication that the first portion of the image is available
after decoding the first encoded data and before decoding the second encoded data.

27. The non-transitory computer-readable medium of claim 25, wherein the instructions
further cause the processor to delay output of the indication until at least a part of the second
encoded data is decoded.

28. The non-transitory computer-readable medium of claim 27, wherein the instructions
further cause the processor to apply an in loop filter to update one or more rows of pixels of the
first portion of the image based on the second pixel data, and wherein the delayed output is based
on an amount of time to apply the in loop filter.

29. An apparatus for processing video data, comprising:

means for determining a number of rows of pixels for one or more portions of an image;
means for obtaining first encoded data for a first portion of the image;
means for decoding the first encoded data to generate first pixel data for the number of
rows of pixels of the first portion of the image;
means for outputting the first pixel data for the first portion of the image to a memory;
and
means for outputting an indication that the first portion of the image is available.

30. The apparatus of claim 29, further comprising:

61
Qualcomm Ref. 2207506TW

means for obtain an indication of a target latency; and

means for determining the number of rows of pixels for the first portion of the image
based on the target latency.

62
Qualcomm Ref. 2207506TW

ABSTRACT

Systems and techniques are described herein for processing video data. For instance, a
process can include determining a number of rows of pixels for one or more portions of an image.
The process can further include obtaining first encoded data for a first portion of the image,
decoding the first encoded data to generate first pixel data for the number of rows of pixels of the
first portion of the image, outputting the first pixel data for the first portion of the image to a
memory, and outputting an indication that the first portion of the image is available.

63