Orb Player — iOS¶

How the in-house Rust playback engine runs on iOS. One engine shared across all native platforms; siblings: Desktop (Windows/macOS/Linux) · Android.

iOS embeds the engine in-process (no subprocess spawning in the sandbox): AVFoundation/VideoToolbox decoders feed the same renderer, clock, and colour pipeline the other platforms use. Status: the full core feature set is implemented and cargo check/clippy-verified for aarch64-apple-ios, and the Swift host is written — but nothing has been built or run on Apple hardware (no Mac in the dev environment), so this is the least-validated platform. AVPlayer remains the default until the on-Mac bring-up below.

Role in the app¶

Mirroring Android, the app keeps two players, selected per launch:

• OrbVideoPlayerViewController (this engine) for direct streams and local files.
• VideoPlayerViewController (AVPlayer) for HLS segment streams — AVAssetReader cannot read m3u8; AVPlayer plays HLS natively — and as the stock/default backend. OrbVideoPlayerLauncher.present(isHls:) performs the delegation.

Both report through the shared VideoPlayerBridge → video-native-* events. The Tauri-side iOS plugin wiring is staged (no gen/ios project exists yet); the integration checklist lives in web/src-tauri/ios-source/README.md.

Architecture¶

WKWebView (Svelte watch route)
│ invoke launch_video_player(url, title, resume_ms, auth_token, is_hls)
▼
Tauri iOS plugin (staged — see ios-source/README.md)
│            is_hls? ──yes──►  VideoPlayerViewController (AVPlayer, HLS)
│ no
▼
OrbVideoPlayerViewController (Swift)
│  OrbMetalView (layerClass = CAMetalLayer) ──► orb_player_create_ios (C ABI)
│  CADisplayLink (vsync) ────────────────────► orb_player_on_redraw
│  per frame ──► orb_player_subtitle_generation/text ──► cue UILabel
│  per second ─► position/duration/state ─► VideoPlayerBridge ─► video-native-*
│  transport bar: play/pause · scrubber · speed · ♪ audio · CC · mute · ✕
▼
orb-player-ffi (liborb_player_ffi.a staticlib + orb_player.h bridging header)
▼
┌─ Player (web/src-tauri/crates/orb-player) ────────────────────────────────────────────┐
│                                                                         │
│ video    AVURLAsset ─► AVAssetReader ─► AVAssetReaderTrackOutput        │
│          '420v' CVPixelBuffer → NV12   (SDR)                            │
│          'x420' CVPixelBuffer → P010   (PQ/HLG, 16-bit-norm GPU)        │
│                              └──► wgpu (Metal) ─ nv12.wgsl ─ CAMetalLayer│
│ audio    AVAssetReader (LPCM 16-bit output settings) ─► PCM-16          │
│          ─► WSOLA stretch (rate≠1) ─► AudioQueue  [master clock:        │
│                                       AudioQueueGetCurrentTime]         │
│ subs     AVAssetReader ("text"/"sbtl" track) ─► tx3g parse ─► CueStore  │
└─────────────────────────────────────────────────────────────────────────┘

Each elementary stream uses its own reader (video / audio / subtitles); seek recreates the reader with a new timeRange (readers are one-pass) — the same worker-restart model as every other backend.

Video pipeline¶

AVURLAsset (mp4/mov/m4v over HTTP Range or local file; auth headers via the AVURLAssetHTTPHeaderFieldsKey option) → AVAssetReaderTrackOutput requesting bi-planar 420v (NV12) — or x420 (10-bit, P010 layout) when the source transfer is PQ/HLG and the GPU exposes TEXTURE_FORMAT_16BIT_NORM — plane-copied and uploaded through the shared VideoFrame path. Colour maps from the format description's transfer/matrix/range extensions; HDR10 mastering metadata parses from the MasteringDisplayColorVolume + ContentLightLevelInfo extensions (big-endian SEI payloads). MKV/TS containers are out of scope for AVFoundation — such sources arrive as HLS and take the AVPlayer path. Zero-copy (CVPixelBuffer→CVMetalTextureCache→wgpu) replaces the plane copy later.

Audio¶

audio_ios.rs: a second AVAssetReader decodes the selected audio track to interleaved PCM-16 (LPCM output settings), played through an AudioToolbox AudioQueue (callback-refilled buffer pool). The output is the engine's master clock via AudioQueueGetCurrentTime with a frames-written fallback. The full DSP set runs in the worker on the decoded PCM, matching the desktop order: pitch-correct 0.25–4× speed (WSOLA stretch.rs) → 10-band EQ + night-mode DRC + stereo remap (pure-Rust dsp.rs) → volume/mute — the same shared, unit-tested modules Android uses.

Subtitles¶

Embedded tx3g timed-text tracks ("text"/"sbtl") are read by a side reader paced to the play head, parsed (subtitle_text.rs) into the shared CueStore, and rendered by the host: the view controller polls a generation counter each frame and updates a shadowed UILabel; the CC button cycles tracks. Sidecar/HLS subtitles stay with the AVPlayer path.

HDR & colour¶

Two paths, like desktop. EDR HDR passthrough (blind, // VERIFY:): when the screen has EDR headroom (UIScreen.potentialEDRHeadroom > 1, passed through create_ios(hdr_capable)), the Swift host configures the owning CAMetalLayer for HDR — pixelFormat = .rgba16Float, colorspace = itur_2100_PQ, wantsExtendedDynamicRangeContent = true — and the engine selects the matching Rgba16Float surface and emits its PQ-encoded BT.2020 output (the same shader path as every other platform; the PQ colour space on the layer makes the panel show it as HDR). Otherwise: 10-bit P010 decode + mastering metadata feed the shared GPU tone-map. Apple EDR is the cleanest HDR pipeline of all platforms; validating it here also unlocks macOS EDR. (An alternative is extended-linear EDR output — a different shader branch — but the PQ-layer route reuses the whole existing passthrough pipeline.)

Source types¶

Host UI & controls¶

Tap toggles a transport overlay (UIKit, Auto Layout): play/pause, scrubber (real duration), time label, speed cycle, audio-track cycle (♪, shown when >1 track), subtitle cycle (CC), mute, close. Subtitle cue label sits above the bar. A Skip Intro / Skip Credits button (bottom-right) appears inside the server's marker windows (passed to OrbVideoPlayerLauncher.present) and seeks past them. Progress/ended/error flow through VideoPlayerLauncher.bridge exactly like AVPlayer's.

Build & verify (the on-Mac bring-up)¶

Everything below needs a Mac + Xcode — none exists in the current dev environment, which is why this platform is check-only:

rustup target add aarch64-apple-ios aarch64-apple-ios-sim
cd web && bunx tauri ios init          # generates gen/apple
cd ../crates && cargo build -p orb-player-ffi --release --target aarch64-apple-ios

Then follow ios-source/README.md: move the staged Swift files into the Xcode project, link the staticlib + frameworks, add orb_player.h as the bridging header, wire the plugin commands + bridge, and extend platform.ts to fire on iOS.

Suggested validation order (each independently demoable): 1. Bring-up — colour-bar source into the CAMetalLayer via create_ios (proves wgpu-Metal, the FFI attach, and the CADisplayLink loop). 2. Decode — a local 1080p H.264 clip with A/V sync; verify the objc2 selectors / struct returns / output-settings keys (// VERIFY: markers). 3. Pipeline — HTTP + auth headers, seek/resume, subtitles, speed, bridge events against an Orb server. 4. Parity + EDR — 4K HDR clips, then EDR passthrough; flip the AVPlayer default after a device pass. Reuse the EDR work for macOS.

What's verified without a Mac: cargo check + clippy for aarch64-apple-ios, and the platform-free DSP/parsing (WSOLA stretcher, cue store, payload parsers) unit-tested on desktop.

Status & remaining work¶

Feature-complete in code (including EDR HDR passthrough), entirely blind/unverified (the objc2 msg_send! selectors, AudioQueue timing, output-settings constants, and the CAMetalLayer EDR config all carry // VERIFY: markers). After bring-up: zero-copy CVMetalTextureCache import, on-device EDR validation, and the Tauri-side plugin wiring once tauri ios init exists.