Orb Player — Android¶

How the in-house Rust playback engine runs on Android. One engine shared across all native platforms; siblings: Desktop (Windows/macOS/Linux) · iOS.

Android embeds the engine in-process (no ffmpeg binary exists in an app sandbox, and 4K HDR demands hardware decode): platform decoders feed the same renderer, clock, and colour pipeline the desktop uses. Status: the full core feature set is implemented and compile/link-verified (Rust .so for both ABIs, Kotlin host builds); on-device runtime validation is pending — ExoPlayer remains the shipping default until a device pass.

Role in the app¶

The Tauri Android app keeps two players, selected per launch by android_bridge::launch_video_player:

• OrbVideoPlayerActivity (this engine) for direct streams and local files — only with the native_rust_player Cargo feature.
• VideoPlayerActivity (ExoPlayer) for HLS segment streams (AMediaExtractor cannot demux m3u8) and as the stock/default backend.

Both report through the same VideoPlayerBridge → video-native-* events, so the Svelte watch route is identical either way.

Architecture¶

WebView (Svelte watch route)
│ invoke launch_video_player(url, title, resume_ms, auth_token, is_hls)
▼
android_bridge.rs (Rust, JNI)
│            is_hls? ──yes──►  VideoPlayerActivity (ExoPlayer, HLS)
│ no
▼
OrbVideoPlayerActivity (Kotlin)
│  SurfaceView ── Surface ── ANativeWindow ──► orb_player_create_android (FFI)
│  Choreographer (vsync) ─────────────────► nativeOnRedraw
│  per frame ──► nativeSubtitleGeneration/Text ──► cue TextView overlay
│  500 ms ticker ─► nativePositionMs/State ─► VideoPlayerBridge ─► video-native-*
│  transport bar: play/pause · scrubber · speed · ♪ audio · CC · mute
▼
orb-player-ffi (liborb_player_ffi.so, JNI exports + C ABI)
▼
┌─ Player (web/src-tauri/crates/orb-player) ────────────────────────────────────────────┐
│                                                                         │
│ video    AMediaExtractor ─► AMediaCodec ─► AImageReader                 │
│          YUV_420_888 → NV12   (SDR)                                     │
│          YCbCr_P010  → P010   (PQ/HLG, API 29+, 16-bit-norm GPU)        │
│                              └──► wgpu (Vulkan) ─ nv12.wgsl ─ surface   │
│ audio    AMediaExtractor ─► AMediaCodec ─► PCM-16                       │
│          ─► WSOLA stretch (rate≠1) ─► AAudio stream  [master clock:     │
│                                       AAudioStream_getTimestamp]        │
│ subs     AMediaExtractor (text track) ─► subtitle_text parse ─► CueStore│
│          (SRT/ASS/VTT/tx3g; paced to the play head)                     │
└─────────────────────────────────────────────────────────────────────────┘

Each elementary stream uses its own extractor (video / audio / subtitles), restarted on seek/track-change — the same worker-restart model as the desktop subprocess decoders.

Video pipeline¶

AMediaExtractor (demux: mp4/mkv/webm/ts over HTTP or local path) → AMediaCodec (HW H.264/HEVC/AV1/VP9) → AImageReader, repacked to tightly packed NV12 — or P010 when the source is PQ/HLG, the device is API 29+, and the GPU exposes TEXTURE_FORMAT_16BIT_NORM — and uploaded through the shared VideoFrame path. Colour (BT.601/709/2020, limited/full, PQ/HLG) maps from the MediaFormat keys; HDR10 mastering metadata parses from HDR_STATIC_INFO (ST 2086 + CLL). The CPU plane repack is the functional baseline; the zero-copy AHardwareBuffer→Vulkan import is seamed in decode::hwbuffer_android behind the zerocopy feature (a device spike).

Auth: request headers ride an AMediaDataSource_newUri data source (API 28+; the build targets --platform 28), so authenticated Orb streams direct-play. Local files (file:// stripped) open without headers.

Audio¶

audio_android.rs: a second extractor + AMediaCodec decodes the selected audio track to PCM-16, played through AAudio (low-latency mode). The output is the engine's master clock, preferring the device's presented frames (AAudioStream_getTimestamp) with a frames-written fallback. The full DSP set runs in the worker on the decoded PCM, matching the desktop order: pitch-correct 0.25–4× speed (the streaming WSOLA stretcher, stretch.rs) → 10-band EQ + night-mode DRC + stereo remap (the pure-Rust dsp.rs, the in-process analog of the desktop ffmpeg equalizer/dynaudnorm/pan filters) → volume/mute. Both DSP modules are unit-tested on desktop.

Subtitles¶

Two kinds, both read by a side extractor paced to the play head and polled by the host through a per-kind generation counter (one cheap JNI call/frame):

• Text (SRT application/x-subrip, ASS, WebVTT, tx3g): parsed to plain text (subtitle_text.rs — markup stripped, unit-tested) into the shared CueStore (timing + "until next cue" fallback); the activity renders a shadowed TextView.
• Bitmap (Blu-ray PGS S_HDMV/PGS, and broadcast DVBSub ETSI EN 300 743): decoded in pure Rust (subtitle_pgs.rs and subtitle_dvbsub.rs — segment parsers + RLE expanders + palette→RGBA, unit-tested) behind a unified BitmapDecoder in subtitle_android.rs, into the shared BitmapCueStore; the JNI hands the host an ARGB int[] and the activity composites a canvas-placed ImageView. Both codecs share that ImageView path, so DVBSub is just another track the CC button cycles — no extra host code.

Both bitmap decoders are unit-verified; the on-device unknown (// VERIFY:) is whether AMediaExtractor surfaces a PGS/DVB track's raw segments at all.

HDR & colour¶

Two paths, like desktop. HDR10 passthrough (blind, // VERIFY:): when the host reports the display is HDR-capable (Display.HdrCapabilities → HDR_TYPE_HDR10, passed through nativeCreate(hdrCapable)), the activity puts the Window in HDR colour mode, and the engine switches the surface to a 10-bit/float format and tags the ANativeWindow's buffers as BT.2020 / ST 2084 (PQ) via ANativeWindow_setBuffersDataSpace (crate::hdr:: apply_hdr10_android, linked through libandroid). The renderer emits the same PQ-encoded BT.2020 R'G'B' it feeds the Windows DXGI swapchain, so SurfaceFlinger composites a real HDR10 signal. Otherwise (SDR display, or the dataspace call fails): 10-bit P010 decode + mastering metadata feed the shared GPU tone-map (PQ/HLG EOTF → BT.2020→709 → operator → sRGB), correct on any screen. The dataspace-after-swapchain-creation behaviour and the buffer format negotiation are the on-device verification points.

Source types¶

Host UI & controls¶

Tap toggles a transport overlay: play/pause, scrubber (real duration), time label, speed cycle (1× → 1.25 → 1.5 → 2 → 0.5 → 0.75), audio-track cycle (♪, shown when >1 track), subtitle cycle (CC, dimmed when off), mute. Subtitle cue label sits above the bar. A Skip Intro / Skip Credits button (bottom-right, independent of the auto-hiding bar) appears while the playhead is inside the server's marker windows (passed as intent extras through the launch) and seeks past them. Progress/ended/error flow through VideoPlayerBridge exactly like ExoPlayer's.

Build & verify¶

# FFI .so for both ABIs into the app's jniLibs (from web/src-tauri/crates/, needs cargo-ndk):
cd crates && cargo ndk -t arm64-v8a -t x86_64 \
  -o ../web/src-tauri/gen/android/app/src/main/jniLibs \
  --platform 28 build -p orb-player-ffi --release
# Full signed app with the engine:
cd ../web && cargo tauri android build --apk --features native_rust_player
#   (or: make android-build)
# Kotlin-only compile check (no Developer Mode needed):
gen/android/gradlew :app:compileArm64DebugKotlin -x :app:rustBuildArm64Debug

Verified today: engine + FFI cargo check/clippy clean for aarch64-linux-android, both ABIs link with all JNI symbols exported (llvm-nm), the Tauri app crate checks for the Android target with the feature, Kotlin host compiles, and the platform-free DSP/parsing (WSOLA, cue store, payload parsers) is unit-tested on desktop.

Status & remaining work¶

The mobile chain is blind code until it runs: the decode/JNI/AAudio paths carry // VERIFY: markers for on-device bring-up (suggested order: colour-bars via create_android first, then a local 1080p H.264 clip, then HTTP + auth + seek/resume + bridge events, then 4K HDR/subs parity vs ExoPlayer (incl. PGS — confirming AMediaExtractor exposes the track), then HDR10 passthrough on an HDR panel — confirming the setBuffersDataSpace PQ tag actually lights the display). After that the only structural gap is zero-copy AHardwareBuffer→Vulkan import.