Live Photo and HDR Media Library Compression Scheme
Published:
Existence precedes perfection.
My photo/video library has grown to hundreds of gigabytes and tens of thousands of files, stored separately on a NAS and on my phone. For a long time, to reduce the cost of services like iCloud, I’ve relied on Synology Photos for management, but its pain points have become increasingly apparent:
- Lack of offline access: Outside the local network, the browsing experience is severely constrained.
- Poor support for new features: Although it’s compatible with Live Photo, support for modern imaging features like HDR photos is not ideal.
- Weak smart features: AI face recognition and scene search are far behind mainstream cloud albums.
To address these issues, a natural idea is to build a tiered storage system: keep the untouched original files on the NAS, while syncing a heavily compressed “optimized” copy to the phone—similar to Apple’s “Optimize iPhone Storage” feature.
On the technology side, state-of-the-art codecs can achieve astonishing compression ratios. Video can use AV1 + Opus, and images can use the AV1-based AVIF format. For example, in the comparison below, I downscaled an image and compressed it to the same size with different encoders; you can see that AVIF maintains excellent quality even at very high compression ratios. You can find the reproduction code and images here: Image Codec Quality Compare)
Inevitable Trade-offs: What Do We Lose?
However, when we compress a rich multimedia object (like a Live Photo or HDR photo) into a single AVIF file aimed at being “small,” loss of information is inevitable. The main costs of this approach are:
- The loss of motion: The vitality of a Live Photo lies in its attached short video. After compression, the motion disappears and only a static “still” remains; those vivid moments are gone.
- Flattening of light and shadow: Modern HDR photos can display the full detail from deep shadows to dazzling highlights on compatible devices. During compression, to pursue extreme size reduction, “tone mapping” is often performed to squeeze the wide dynamic range into standard dynamic range (SDR). This makes originally nuanced light and shadow look “flat,” losing the immersive visual impact.
Next, we’ll look at the implementation for each in turn.
Live Photo: Structure, Principles, and Compatibility Solution
Although Apple’s Live Photo feels seamless, its technical implementation is quite straightforward. It’s not a single special-format file, but a pair consisting of a still image (usually HEIC) and a short video (MOV).
These two files are linked by a shared metadata identifier, allowing the system to know they should be presented as a single entity. Whether you export from the macOS Photos app or inspect the backend storage of Synology Photos, you can verify this structure.
This mechanism gives us an excellent entry point for our compression scheme. We can convert the original HEIC and MOV into a higher-compression AVIF image and an AV1+Opus-encoded MP4 video, respectively. As long as the original EXIF and the key association IDs are migrated into the new files, Apple Photos can still recognize and correctly display them as a Live Photo.
You can save the sample files below to your Photos library to experience this yourself. If you encounter issues, try dragging them into the Photos app on macOS to import.
HDR: Standards Diversity and the Current State of Conversion
Compared with the “combo” nature of Live Photos, compressing HDR photos is much more complex. HDR has many standards, and compatibility varies across devices and software (for in-depth discussion, see Jackchou’s series), so our discussion will focus on the implementation within the Apple ecosystem.
At the time of writing, mainstream open-source imaging libraries (such as ImageMagick, libheif, and FFmpeg) still cannot preserve the core HDR information—Apple’s Gain Map—when converting HDR photos shot on iPhone.
Fortunately, color science expert Jackchou’s hdr-conversion tool solves this problem. It can read HEIC files shot on iPhone that contain a Gain Map and convert them into PQ AVIF while preserving the high dynamic range. Note that this tool depends on an older version of the pillow_heif library for export, because newer Pillow has native AVIF support and the third-party plugin no longer supports it. If you find the library versions bundled with the old pillow_heif too dated, refer to my fork pillow_heif_avif, which currently offers Windows wheel packages.
If you saved the sample image from the previous section to your Photos library and view it on a supported device (e.g., an iPhone/iPad Pro/Mac bearing the XDR label), you’ll see true HDR highlights. If you can’t tell in the browser, be sure to add it to the system Photos library to view.
Finally, the good news is HDR handling for video is somewhat simpler. Whether it’s natively recorded video or the MOV attached to a Live Photo, their Dolby Vision HDR information can now be parsed and preserved well by newer FFmpeg builds, and the conversion usually requires no special intervention.
Photo Library Compression
Building on the complete solution for Live Photo and HDR features above, we can further implement an engineered photo library compression workflow. For implementation details and code, see the MediaConverter project. Using this workflow, my library size shrank dramatically from about 300 GB to about 40 GB. After importing on a Mac and enabling iCloud sync, Apple devices including the iPhone can access the entire library offline at acceptable image quality. If you need the originals, you can always fetch them from Synology Photos.
Future Work
Currently the entire conversion flow still requires manual steps—triggering conversion, deleting originals, and importing the compressed versions—and has not been automated. In the future, I plan to explore using iCloud-related APIs to perform automatic triggering, conversion, deletion of originals, and import of new files on the NAS side, making the whole process transparent to the user and further improving the experience.