Orb Player — Desktop (Windows / macOS / Linux)¶

How the in-house Rust playback engine runs on the desktop. All three desktop OSes link the same orb-player engine in-process and drive it the same way — they differ only in the GPU backend, the audio host API, and the HDR / window paths (the per-OS deltas below). One engine shared across all native platforms; siblings: Android · iOS.

Windows is the reference platform. The engine (decode, sync, HDR10 passthrough on an RTX 5090 + HDR panel) is runtime-verified and presents through a DirectComposition underlay of the main window — no second window. The macOS / Linux hosts render into a second top-level window stacked behind the transparent main window and are pending end-to-end re-verification on-device — see per-OS deltas.

Role in the app¶

The desktop video backend is launch_video_player / stop_video_player + video_* (play/pause/seek/volume/mute/rate/select-audio/select-subtitle/subtitle- style) in web/src-tauri, emitting video-native-progress/ended/error events. The engine is linked in-process (no separate orb-player.exe). How its picture reaches the screen differs per OS:

• Windows — the engine presents through DirectComposition as a compositor underlay of the main window (Player::open_composition), beneath wry's transparent WebView2. No second window. DWM composites a DirectComposition surface unconditionally — it isn't occlusion-culled like a windowed swapchain — so HDR10 passthrough stays reliable even when another app is frontmost, and the picture is part of the one window's alt-tab / taskbar thumbnail.
• macOS / Linux — the engine renders into a second top-level window stacked behind the transparent main window (Player::open_raw), kept glued to it as the window moves / resizes. Native HDR layering (EDR CAMetalLayer / Vulkan + Wayland subsurface) is a follow-up, so HDR is tone-mapped to SDR here for now.

Either way the web VideoPlayer.svelte controls — the exact same UI as the browser player — paint on top through the transparent "stage" and drive the engine via the video_* commands. On the watch route this is the default on desktop (automatic, full passthrough/HDR); the in-page <video> is the fallback if the engine fails. Tauri (tao) owns the single event loop; only GPU / decode / present runs on the render thread (the same model the mobile FFI uses).

Architecture¶

Tauri app (web/src-tauri) — ONE event loop (tao), main thread
│
│  ┌──────────── main WebviewWindow (transparent) ───────────────┐
│  │  VideoPlayer.svelte controls (opaque) over a transparent     │
│  │  "stage" (html.orb-immersive makes the shell see-through)    │
│  └──────────────────────────────────────────────────────────────┘
│              ▲ controls on top; engine picture below (z-order)
│  ┌─ DirectComposition underlay (Windows) / 2nd window (macOS·Linux) ─┐
│  │  wgpu surface, fills the main window's client area                │
│  └────────────────────────────────────────────────────────────────────┘
│
│ launch_video_player(url, resume_ms, auth_token, …) ── creates the window,
│ video_play/pause/seek/… ──► HostCmd channel ──►  spawns the render thread:
◄── video-native-progress/ended/error ───────────┐
                                                  ▼
                ┌──────────────────────────────────────────────┐
                │ Player (web/src-tauri/crates/orb-player)       │
                │  Player::open_raw(handle) + on_redraw() loop   │
       ffmpeg ◄─┤ decode::ffmpeg_cli   NV12/P010 frames (pipe)  │
       ffprobe◄─┤ probe                tracks/chapters/HDR meta │
       ffmpeg ◄─┤ audio                PCM → cpal               │──► speakers
       ffmpeg ◄─┤ suboverlay           libass → RGBA overlay    │   [master clock]
       ffmpeg ◄─┤ trickplay            seek-preview thumbnails  │
                │                  ▼                            │
                │   wgpu — R8/Rg8 (NV12) or R16/Rg16 (P010)     │
                │   render/nv12.wgsl: YUV→RGB, EOTF, gamut,     │
                │   tone-map, colour adjust, zoom/aspect        │
                └──────────────┬───────────────────────────────┘
                               ▼
                swapchain ── SDR sRGB   (every display)
                         └── HDR10 PQ/BT.2020 passthrough (Windows HDR display)

Video pipeline¶

Demux + decode run in an ffmpeg subprocess (-hwaccel auto picks the platform HW decoder — D3D11VA / VideoToolbox / VAAPI / NVDEC; toggleable to forced software) streaming raw NV12 — or P010 for 10-bit PQ/HLG sources when the GPU exposes TEXTURE_FORMAT_16BIT_NORM — over a pipe on a worker thread (bounded channel; the render thread never blocks on decode). Frames upload as wgpu plane textures and are paced to the audio master clock with a scheduled-present lead, holding the content cadence on any refresh rate (pinned by the frame_cadence_matches_fps_at_every_refresh test).

A Windows-only zero-copy ID3D11Texture2D → wgpu import (DXGI shared NT handle, proven in the Spike C prototype) is kept in decode::d3d11_interop as the future fast path for an in-process ffmpeg-next decoder.

Audio¶

An ffmpeg subprocess decodes the selected audio track to PCM; cpal plays it (WASAPI / CoreAudio / ALSA per OS) and acts as the master clock. Full DSP feature set via the ffmpeg filter graph: pitch-correct 0.25–4× speed (atempo), 10-band EQ, night-mode DRC (dynaudnorm), stereo routing (pan), surround downmix honoring max_audio_channels, output-device selection, A/V delay trim, volume to amplification, mute.

Subtitles¶

• Text subs (SRT/ASS/VTT, embedded or external): a dedicated ffmpeg+libass process renders cues onto a transparent canvas (two-pass alpha recovery); the host composites the RGBA overlay — the video decoder is never restarted, so styling/delay/track changes are instant and A/V never desyncs.
• Bitmap subs (PGS/VOBSUB): burned in by the video decoder.
• Styling (size/colour/outline/position/font), charset override, delay trim.

HDR & colour¶

Two presentation paths, chosen per display at open (choose_hdr):

• HDR10 passthrough (Windows only today) — when DXGI reports an HDR output (IDXGIOutput6::GetDesc1), the surface is configured Rgb10a2Unorm and the underlying DXGI swapchain is tagged via the as_hal escape hatch: SetColorSpace1(G2084_P2020) + SetHDRMetaData (real mastering metadata from ffprobe). PQ output is bit-exact (the panel applies the EOTF); HLG is re-encoded to PQ. Requires the D3D12 backend (forced on Windows). Reapplied after every resize (wgpu recreates the swapchain). Verified live on an RTX 5090 + HDR panel, plus offscreen bit-exactness + tone-map readback tests that run on every cargo test.
• GPU tone-map to SDR — PQ/HLG EOTF → linear, BT.2020→709 gamut, selectable operator (Hable/Reinhard/Clip) with peak override → sRGB. This is the path on macOS and Linux today (hdr::detect() returns "no HDR display" off Windows), so HDR content is watchable everywhere; their native passthrough paths (EDR / Vulkan) are the per-OS deltas below.

Source types¶

Host UI & controls¶

The controls are the web VideoPlayer.svelte — the same component the browser player uses — running in its backend="native" mode. Instead of an HTML <video> it draws a transparent stage (the engine shows through from behind) and routes transport (play/pause, seek, volume, mute, speed, audio/subtitle select) to the video_* Tauri commands; progress/duration/ended come from the video-native-* events. The engine always plays the original bytes — true passthrough, no transcode ladder — so the quality picker reflects the server-offered options (Direct passthrough is the default). "Picture-in-picture" becomes a desktop mini-player: the window shrinks to a pinned, always-on-top tile in the corner (with a size floor so it can't be dragged degenerately small). Skip Intro/Credits and the next-up card work from the server markers as on the web; fullscreen toggles the OS window and restores a snapped / split-screen layout on exit. The rest matches the browser player: scroll-wheel / arrow-key volume with a boost up to 200 % (an amber warning shows above the read-out past 100 %), per-track subtitle styling + sync, playback speed, always-on-top pin, and a keyboard-shortcuts overlay — open it from the circular ? button beside the top-right badges or with the ? key. The standalone orb-player-app egui overlay still exists as a separate dev binary but the Tauri app no longer uses it.

Per-OS deltas¶

On Windows the DirectComposition underlay is composited by DWM beneath the transparent WebView2 (reliable even when the window is covered). The macOS / Linux second-window compositing + their native-HDR work (EDR, Vulkan colour-space) are the open items needing on-device verification.

Build & verify¶

# The engine is linked in-process — no separate binary to build or ORB_PLAYER_BIN.
cd web && cargo tauri dev

# App bundles:  make windows-build | macos-build[-arm|-universal] | linux-build[-arm]

The standalone orb-player-app (egui) binary is still buildable for engine dev (cd web/src-tauri/crates && cargo build --release -p orb-player-app) but the Tauri app no longer launches it.

Runtime requirement: ffmpeg/ffprobe on PATH (winget install Gyan.FFmpeg / brew install ffmpeg / apt install ffmpeg) or ORB_FFMPEG/ORB_FFPROBE. Tests: cargo test -p orb-player (headless — shader validation, sync regression, GPU readback when an adapter is present); end-to-end photon-level sync: python scripts/sync-check.py (Windows + a display).

Status & remaining work¶

Engine fully implemented and runtime-verified on Windows (DirectComposition underlay + web-controls integration); portable by construction (wgpu + cpal are mature; the engine is backend-portable across Vulkan + D3D12 + Metal). Remaining: end-to-end re-verification of the macOS / Linux second-window transparent compositing + geometry/z-order sync (the one part that can't be checked at build time); macOS / Linux native HDR passthrough (EDR / Vulkan) + a recorded macOS run; in-process zero-copy decoder (ffmpeg-next over the proven D3D11 interop) for max 4K perf; Atmos/TrueHD bitstream passthrough (WASAPI exclusive); HDR10+ dynamic metadata (treated as static HDR10 today); Dolby Vision (P8 plays via its HDR10 base layer).