Apple Endorses New Image Format, HEIF

Most streaming professionals have heard about Apple's including HEVC encode/decode in iOS11/tvOS11 and the next version of MacOS. If not, you can read my technical description in Streaming Media Magazine here or Dan Rayburn's excellent opinion piece on his blog here.

Less well known is Apple's use of the High-Efficiency Image File (HEIF) format. I wrote a bit about HEIF in my Streaming Media article, and several days later received an email from Dror Gill (shown below), CTO of image optimization and codec company Beamr, asking if I'd like a longer explanation. This led to the Q&A presented below, which details what HEIF is, why you care, what Apple announced, what HEIF costs, and how Beamr makes HEIF and HEVC simple to use for large scale producers. I hope you find it informative and useful.

Ozer: What is HEIF?

Gill: HEIF (High-Efficiency Image File format) is a standard for storing images and image sequences. It was developed by MPEG, the same standards body that was involved in the developed of HEVC (High-Efficiency Video Codec), and all of the standard video compression technologies we are using today (MPEG-1, MPEG-2, MPEG-4, and H.264/AVC). HEIF is based on the ISO Base Media Format, which we all know as MP4, the most common file format for video today.

In principal, HEIF can store various types of images and other media, but the main application of HEIF is to store images that have been compressed using the image tools of the HEVC video compression standard. By itself HEIF has some interesting features: It can store both single images and image sequences (photo bursts or video), it can store images and video that were captured simultaneously, and it can also store audio and text which are synchronized with the image sequences.

Ozer: Why do we care?

Gill: Of particular interest to photographers is the ability of HEIF to support both lossy and lossless compression, so the full quality of your captured photos will be preserved. But the most interesting feature for photographers is the ability to store image editing operations. These operations, such as rotations, cropping, title, and overlays result in “derived” images that are created from a base image. The base image and editing operations are stored separately in the file, and the derived image is created when the file is rendered for display. This gives photographers unprecedented freedom in applying non-destructive editing operations that can be stored in the image file and applied or removed at will.

Ozer: How is Apple implementing it?

Gill: Starting with the release of iOS 11, iPhones and iPads will capture and store images in HEIF format, saving 50% of storage capacity. Today, an iPhone 7 with 128 GB of memory can store around 50,000 photos of 12 Megapixels each. With HEIF, the capacity of that iPhone 7 will grow to around 100,000 photos, which means that you can take 50 photos every day for 3 consecutive years without having to delete a single photo, and still have half of your storage capacity available for videos and apps. Apple will also support HEIF viewing and editing in its MacOS platforms, starting from the next version of MacOS called High Sierra.

Ozer:  What does this mean for sharing images snapped on your iPhone?

Gill: Until every application supports HEIF, you’ll have to convert your images to JPEG if you want to use them on a website or share with non-Apple computers and devices. The Settings app on iOS 11 will let you select whether you want your photos and videos stored in “High Efficiency” formats (HEIF and HEVC), or “Most Compatible” formats (JPEG and H.264).

At WWDC, Apple outlined its strategy for sharing HEIF photos with other devices: When you share photos via mail or apps, they will always be transcoded to JPEG, since the receiver is unknown and hence might not be able to support HEIF. When you use PTP sharing and AirDrop, the conversion will depend on a process of capability exchange with the receiving device: If the receiving device supports HEIF, no conversion will be performed. Otherwise, the HEIF image will be converted to JPEG.

Ozer: Is there a royalty on HEIF? If so, do we know what it is, or will it be another HEVC nightmare?

Gill: HEIF is a subset of HEVC, as it contains a single HEVC “keyframe” or “I frame”. These are the frames in a video codec that are encoded as a whole, and not predicted from other frames. Therefore, only the HEVC patents that cover still image encoding are relevant to HEIF, and not the patents that deal with frame predictions, motion estimation and compensation, temporal modes, etc.

The current patent pools don’t make a distinction between “still image” patents and the rest of the HEVC patents, so it’s not clear how HEIF can be licensed, other than paying the full HEVC royalties. It makes sense that if the format becomes popular, the patent pools would offer a reduced “HEIF-only” license fee.

That said, I can’t think of many companies that will only use HEIF and not HEVC, since today image and video workflows are tied closely together. Camera vendors such as Canon, Nikon and Sony all make cameras that shoot both still images and videos, software vendors such as Adobe create image and video software, and most online photo service such as Flickr, Snapfish, Shutterfly etc. support both photo and video sharing. In all of these examples, vendors who license HEVC for their video flow will get the HEIF license “for free”.

Ozer: What’s Beamr doing to make it easier for producers and photographs to use HEVC and HEIF?

Gill: Until the whole industry supports HEIF and HEVC, conversions of videos and photos captured on iOS 11 devices in HEVC and HEIF to H.264 and JPEG will be required. The simple way to perform these conversions is to take a “safety” measure and use a high bitrate / file size to make sure quality isn’t hurt.

Beamr has a better solution: We developed a perceptual quality measure that has very high correlation with human results, and can tell use how much compression can be applied (without losing quality) when performing these conversions. So under the control of our quality measure, we can ensure “optimal” conversion of HEIF to JPEG and HEVC to H.264, and conversions in the other direction as well. We don’t have any specific product plans in these areas yet, but you can assume that once the market picks up, we will offer products that enable “optimal” conversions which preserve quality but minimize bitrate and file size.

Another interesting direction we are exploring is “minimizing” HEIF files - reducing their file size without affecting their perceptual quality. We already have a line of products called JPEGmini, which do exactly this for JPEG images. And, we have a video optimizer that reduces the bitrate of both H.264 and HEVC video streams. So combining these technologies to optimize HEIF images makes a lot of sense.

Ozer: Thanks, Dror.

Gill: My pleasure.

Jan Ozer is the author of Video Encoding by the Numbers: Eliminate the Guesswork from Your Streaming Video and a contributing editor to Streaming Media Magazine. He blogs at the Streaming Learning Center.