Request: Mac media-engine headroom for 3840-per-eye stereo encoding to Apple Vision Pro
We develop a Mac app that streams OpenXR VR applications, rendered on the Mac's GPU, to Apple Vision Pro as stereo video. The Mac encodes with VideoToolbox, as MV-HEVC (one layer per eye) or side-by-side HEVC. Vision Pro decodes the stream and presents it in an immersive space.
We'd like to ask for future Mac hardware to support stereo encoding at Vision Pro-class resolution: 3840 pixels per eye at 90 fps.
Why resolution matters without eye tracking
Vision Pro does not give applications eye-tracking data. We understand and respect the privacy reasons. It does mean a streaming app can not use gaze-driven foveated encoding, which sends full detail where the wearer is looking and less elsewhere. Streaming solutions on other headsets use that to stay within their encoder budget. Without gaze data, every part of each eye's image has to be sent at full detail, so the only route to a sharper image is encoding more pixels. Today the encoder is where that stops.
What we measured (tested on an M4 Max)
M4 Max, macOS 27.0.1, hardware HEVC through VideoToolbox, synthetic frames at 200 Mbps, no frame reordering, RealTime off, speed prioritised, frames submitted at 90 Hz with at most 4 in flight, eye heights at 20:19. Results are frames per second, with median encode latency in brackets.
MV-HEVC:
- 2048×1952 per eye: 90 fps (16 ms)
- 2240×2128 per eye: 90 fps (19 ms), the largest size that holds 90
- 2304×2192 per eye: 87 fps (46 ms)
- 3072×2928 per eye: 50 fps (79 ms)
- 3840×3648 per eye: 33 fps (121 ms)
Side-by-side HEVC:
- 2048×1952 per eye: 90 fps (12 ms)
- 3072×2928 per eye: 90 fps (25 ms)
- 3200×3040 per eye: 90 fps (27 ms), the largest size that holds 90
- 3328×3168 per eye: 86 fps (46 ms)
- 3840×3648 per eye: 65 fps (61 ms)
What the numbers show:
- One MV-HEVC session uses one media engine (about 0.92 Gpx/s at every size). The second engine sits idle: two MV-HEVC sessions at 3840 per eye run at the same time at 33 fps each.
- One single-layer HEVC session spreads across both engines (about 1.82 Gpx/s), but only while nothing else is encoding. With a second session open (we used a 1080p30 desktop share), it drops to one engine: 3072 per eye falls from 90 to 52 fps, and 3840 per eye from 65 to 33 fps.
- GPU load does not slow the media engine. We repeated the tests beside a synthetic GPU load using 7 to 9.6 ms of every 11.1 ms frame, as a VR title would, and every result was unchanged.
The encoding window we actually have
At 90 Hz a frame lasts 11.1 ms, and the VR application uses most of that rendering on the GPU. Rendering and encoding run in sequence for each frame, so every millisecond of encoding adds directly to the wearer's motion-to-photon latency, on top of the app's own frame. In practice we need two things from the encoder:
- Latency: finish each frame within about one frame period (around 11 ms), so the stream adds no more than one frame beyond the app. Today 2048 per eye MV-HEVC takes 16 ms, or about 1.5 frames.
- Headroom: run at no more than about 75 % of capacity. Real scenes vary in complexity, and near the limit latency climbs steeply (side-by-side HEVC goes from 27 ms at 3200 per eye to 46 ms at 3328 per eye).
At 75 % load an M4 Max supports about 2000 per eye for MV-HEVC, which is the 2048 we ship today, and about 2800 per eye for side-by-side HEVC when no other session is running.
3840×3648 per eye at 90 fps is 2.52 Gpx/s. With the 25 % headroom, that is about 3.4 Gpx/s of sustained encode capacity: roughly 1.9× both M4 Max engines together, or 3.7× the single engine an MV-HEVC session gets today.
For comparison, our Windows host encodes MV-HEVC at 3840×3648 per eye at 90 fps on a current desktop NVIDIA GPU, in about 11 ms per frame, by splitting the work across its encoder engines. Vision Pro decodes and presents that stream at 90 fps, so the headset handles this resolution. The gap is on the Mac's encoding side.
What we don't know yet
We do not yet know whether the Mac GPU can render VR titles at 3840 per eye at 90 fps; that will depend on the title. We can not usefully find out while the encoder limits us first, so in practice we ship MV-HEVC/HEVC at 2048 per eye by default.
What we're asking for
- About 3.4 Gpx/s of HEVC / MV-HEVC encode capacity in future Mac media engines, enough for 3840×3648 per eye at 90 fps with headroom, plus room for a smaller second session such as a desktop share.
- MV-HEVC/HEVC sessions that can use more than one media engine on chips that have them. On current chips this alone would roughly double the stereo ceiling.
- A single-layer HEVC session that keeps its multi-engine split while another session is running, or a property to request that.
- If the platform can offer it, a privacy-preserving foveation path for streaming apps, where the system supplies a foveation region to the encoder without the app ever seeing raw gaze.
If a VideoToolbox configuration already lets one MV-HEVC/HEVC session use both engines, we'd be glad to hear about it. We're happy to file a Feedback report with our test tool attached.