android(call): Increment 2b — native camera video + hole-punch tiles

Mirrors the iOS native-video path on Android, gated by ANDROID_NATIVE_VIDEO.

- CallVideo (new): hole-punch (transparent WebView over a black parent with a
  container behind) + per-tile LiveKit TextureViewRenderer, positioned from the
  web's CSS-px tile rects scaled by display density. TextureView (not Surface)
  so it composites behind the transparent WebView. Local camera mirrored;
  camera = aspect-fill, screen = aspect-fit; stale renderers released.
- CallMedia: setCameraEnabled / switchCamera (LocalVideoTrack.switchCamera) +
  cameraTrack/screenTrack lookup (by LiveKit identity = mesh peerId) +
  initRenderer. All LiveKit 2.29.2 APIs verified against source.
- NativeCallPlugin: setCamera / switchCamera / setHolePunch / syncVideoTiles get
  real bodies (+ CAMERA runtime-permission prompt, video teardown on endCall).
  Screen-share (MediaProjection), tile zoom and #5 transcript stay no-ops.

Web needs no change: the shared NativeCall bridge already drives these for any
meetNative call; bzNativeVideoOn() enables hole-punch + the tile poll on Android
once the flag is on. Flip ANDROID_NATIVE_VIDEO=1 after a 2b APK is installed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
SravanandClaude Opus 4.8 committed 2026-11-03 20:26:31 +05:30
1 parent ae607f44b5
commit 4baace83bc
4 files changed
+269 -8

No files matched your search

+8 -2
View File
@@ -1,6 +1,6 @@
# Tech Stack — BizGaze Connect
> Living document. **Last updated: 2026-10-30** (rev 3). Update this file whenever a dependency, platform, service, or
> Living document. **Last updated: 2026-11-03** (rev 4). Update this file whenever a dependency, platform, service, or
> tool is added/removed/version-bumped. See [Maintaining this file](#maintaining-this-file) and the
> [Changelog](#changelog) at the bottom.
@@ -109,7 +109,9 @@ hash, per-user salt, timing-safe compare), opaque random session tokens, SHA-256
(`SFSpeechRecognizer`). **Android:** Java, full-screen incoming-call UI over the lock screen + a
`FirebaseMessagingService` woken by high-priority FCM data messages. **Increment 2a** adds native LiveKit
**audio** (Kotlin `CallMedia` + a `microphone` foreground service for background audio + `AudioSwitchHandler`
routing), gated by the server flag `ANDROID_NATIVE_MEDIA`; native video = Increment 2b.
routing), gated by the server flag `ANDROID_NATIVE_MEDIA`. **Increment 2b** adds native **video** (Kotlin
`CallVideo`: camera publish/flip + remote/local `TextureViewRenderer` tiles hole-punched behind a transparent
WebView), gated by `ANDROID_NATIVE_VIDEO`; screen-share (MediaProjection) is still pending.
- `audio-route` (iOS audio output routing), `media-library` (save to Photos/gallery), `share-inbox` (inbound
share sheet), `file-opener` (native Quick Look / open-with).
- **iOS build:** Capacitor 8 + **SPM** (CocoaPods retired); Xcode 26; a share extension + a broadcast extension.
@@ -201,6 +203,10 @@ Bump the **Last updated** date at the top and add a line to the Changelog below.
## Changelog
- **2026-11-03 (rev 4)** — Android native call **video** (Increment 2b): `CallVideo` renders camera/screen tiles
with LiveKit `TextureViewRenderer` hole-punched behind a transparent WebView; `CallMedia` gains camera
publish/flip + track lookup; new `ANDROID_NATIVE_VIDEO` server flag. Also fixed the white meeting background
(hole-punch no longer turns on without native video).
- **2026-10-30 (rev 3)** — Android native call media (Increment 2a): added `io.livekit:livekit-android` 2.29.2 +
`com.twilio:audioswitch` + Kotlin to the native-call plugin, a `microphone` foreground service, the
`ANDROID_NATIVE_MEDIA` server flag, and raised the Android app to minSdk 24. Build fix: `android-patch.js`
@@ -7,7 +7,11 @@ import io.livekit.android.LiveKit
import io.livekit.android.LiveKitOverrides
import io.livekit.android.AudioOptions
import io.livekit.android.audio.AudioSwitchHandler
import io.livekit.android.renderer.TextureViewRenderer
import io.livekit.android.room.Room
import io.livekit.android.room.track.LocalVideoTrack
import io.livekit.android.room.track.Track
import io.livekit.android.room.track.VideoTrack
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.SupervisorJob
@@ -114,4 +118,55 @@ object CallMedia {
audio = null
if (r != null) scope.launch { try { r.disconnect() } catch (_: Throwable) {} }
}
// ---- Native video (Increment 2b) -------------------------------------------------------------------
// The WebView owns the meeting grid but has no LiveKit connection on a native call, so it can't render any
// video. CallVideo draws native TextureViewRenderers behind the (transparent) WebView, positioned to match
// each web tile; these helpers give it the camera control + the live tracks to bind. Mirrors the iOS plugin.
// Publish/unpublish the local camera (front by default). Fire-and-forget: the web re-syncs tiles right after,
// and a tile only shows video once a live track exists — so a failure just leaves the web avatar showing.
@JvmStatic
fun setCameraEnabled(enabled: Boolean) {
val r = room ?: return
scope.launch {
try { r.localParticipant.setCameraEnabled(enabled) }
catch (e: Throwable) { Log.w(TAG, "setCamera($enabled) failed: ${e.message}") }
}
}
// Flip front/back. No-arg switchCamera() moves to the next available camera.
@JvmStatic
fun switchCamera() {
val r = room ?: return
val track = r.localParticipant.getTrackPublication(Track.Source.CAMERA)?.track as? LocalVideoTrack ?: return
try { track.switchCamera() } catch (e: Throwable) { Log.w(TAG, "switchCamera failed: ${e.message}") }
}
// The live camera track for a participant (local, or remote by LiveKit identity = mesh peerId), or null when
// the camera is off/muted so the web avatar shows through instead.
@JvmStatic
fun cameraTrack(uid: String, isLocal: Boolean): VideoTrack? {
val r = room ?: return null
val p = if (isLocal) r.localParticipant else (r.getParticipantByIdentity(uid) ?: return null)
val pub = p.getTrackPublication(Track.Source.CAMERA) ?: return null
if (pub.muted) return null
return pub.track as? VideoTrack
}
// The remote screen-share track for a participant, or null if they aren't sharing (or not subscribed yet).
@JvmStatic
fun screenTrack(uid: String): VideoTrack? {
val r = room ?: return null
val p = r.getParticipantByIdentity(uid) ?: return null
val pub = p.getTrackPublication(Track.Source.SCREEN_SHARE) ?: return null
if (pub.muted) return null
return pub.track as? VideoTrack
}
// Init a renderer against the room's EGL context before a track is attached. No-op if no active room.
@JvmStatic
fun initRenderer(renderer: TextureViewRenderer) {
try { room?.initVideoRenderer(renderer) } catch (e: Throwable) { Log.w(TAG, "initRenderer failed: ${e.message}") }
}
}
@@ -0,0 +1,137 @@
package com.bizgaze.connect.nativecall
import android.app.Activity
import android.graphics.Color
import android.graphics.drawable.ColorDrawable
import android.util.Log
import android.view.ViewGroup
import android.webkit.WebView
import android.widget.FrameLayout
import io.livekit.android.renderer.TextureViewRenderer
import io.livekit.android.room.track.VideoTrack
import livekit.org.webrtc.RendererCommon
// Native video tiles for Android native calls (Increment 2b) — the Android counterpart to the iOS plugin's
// TileVideoView path. The WebView owns the meeting UI (grid, roster, controls) but, on a native call, has NO
// LiveKit connection — the media lives only in the plugin's native LiveKit room (CallMedia). So we render the
// video NATIVELY:
//
// • Hole-punch: the WebView is made transparent and its parent painted black; a container FrameLayout is
// inserted BEHIND the WebView to host the video. The web meeting UI (opaque controls, avatars for
// camera-off tiles) then paints on top, while tiles flagged `bz-hasvid` go transparent so the native
// video behind shows through. We use TextureViewRenderer (NOT SurfaceViewRenderer) because a TextureView
// composites normally in the view tree and shows through a transparent sibling on top — a SurfaceView
// punches its own window hole and can't.
// • The web polls syncVideoTiles([{uid, local, screen, x, y, w, h}]) with each tile's on-screen rect in CSS
// px; we place/size a renderer per participant that has a live camera (or screen) track, scaling CSS px →
// device px by display density, and bind the matching LiveKit track. Camera-off participants get no
// renderer, so their web avatar shows.
//
// All view work runs on the UI thread. Track lookup + renderer init go through CallMedia (which owns the room).
object CallVideo {
private const val TAG = "bzcall"
private var holePunchOn = false
private var container: FrameLayout? = null
private var savedParentBg: Int? = null
// key ("__local" | peerId) -> renderer, and the track currently bound to it (to avoid needless re-binds).
private val renderers = HashMap<String, TextureViewRenderer>()
private val bound = HashMap<String, VideoTrack>()
// One meeting tile from the web (coords are CSS px, viewport-relative — same space as getBoundingClientRect).
data class Tile(
val key: String, val uid: String, val isLocal: Boolean, val wantScreen: Boolean,
val x: Double, val y: Double, val w: Double, val h: Double
)
@JvmStatic
fun setHolePunch(activity: Activity?, webView: WebView?, on: Boolean) {
if (activity == null || webView == null) return
activity.runOnUiThread {
try {
if (on) {
webView.setBackgroundColor(Color.TRANSPARENT)
val parent = webView.parent as? ViewGroup
if (parent != null) {
if (!holePunchOn) savedParentBg = (parent.background as? ColorDrawable)?.color
parent.setBackgroundColor(Color.BLACK)
if (container == null) {
val c = FrameLayout(webView.context)
val idx = parent.indexOfChild(webView)
// Insert at the WebView's index → container sits BEHIND it (drawn first).
parent.addView(c, if (idx < 0) 0 else idx,
ViewGroup.LayoutParams(ViewGroup.LayoutParams.MATCH_PARENT, ViewGroup.LayoutParams.MATCH_PARENT))
container = c
}
}
holePunchOn = true
Log.i(TAG, "hole-punch ON (native video)")
} else {
teardown(webView)
}
} catch (e: Throwable) { Log.w(TAG, "setHolePunch($on) failed: ${e.message}") }
}
}
@JvmStatic
fun syncTiles(activity: Activity?, webView: WebView?, tiles: List<Tile>) {
if (activity == null || webView == null) return
activity.runOnUiThread {
val c = container ?: return@runOnUiThread
val density = webView.resources.displayMetrics.density
val wanted = HashSet<String>()
for (t in tiles) {
if (t.w < 2 || t.h < 2) continue
val track = (if (t.wantScreen) CallMedia.screenTrack(t.uid) else CallMedia.cameraTrack(t.uid, t.isLocal)) ?: continue
wanted.add(t.key)
var r = renderers[t.key]
if (r == null) {
r = TextureViewRenderer(webView.context)
CallMedia.initRenderer(r)
c.addView(r)
renderers[t.key] = r
}
r.setMirror(t.isLocal && !t.wantScreen) // mirror the local camera preview only
r.setScalingType(if (t.wantScreen) RendererCommon.ScalingType.SCALE_ASPECT_FIT else RendererCommon.ScalingType.SCALE_ASPECT_FILL)
val lp = FrameLayout.LayoutParams((t.w * density).toInt(), (t.h * density).toInt())
lp.leftMargin = (t.x * density).toInt()
lp.topMargin = (t.y * density).toInt()
r.layoutParams = lp
val cur = bound[t.key]
if (cur !== track) {
if (cur != null) try { cur.removeRenderer(r) } catch (_: Throwable) {}
try { track.addRenderer(r) } catch (_: Throwable) {}
bound[t.key] = track
}
}
// Drop renderers for tiles that are gone or whose camera turned off (so the web avatar shows).
val it = renderers.entries.iterator()
while (it.hasNext()) {
val e = it.next()
if (!wanted.contains(e.key)) {
val r = e.value
val tr = bound.remove(e.key)
if (tr != null) try { tr.removeRenderer(r) } catch (_: Throwable) {}
try { c.removeView(r) } catch (_: Throwable) {}
try { r.release() } catch (_: Throwable) {}
it.remove()
}
}
}
}
private fun teardown(webView: WebView) {
for ((key, r) in renderers) {
val tr = bound[key]
if (tr != null) try { tr.removeRenderer(r) } catch (_: Throwable) {}
try { r.release() } catch (_: Throwable) {}
}
renderers.clear()
bound.clear()
val c = container
if (c != null) { try { (c.parent as? ViewGroup)?.removeView(c) } catch (_: Throwable) {}; container = null }
try { (webView.parent as? ViewGroup)?.setBackgroundColor(savedParentBg ?: Color.TRANSPARENT) } catch (_: Throwable) {}
try { webView.setBackgroundColor(Color.WHITE) } catch (_: Throwable) {} // restore the normal (opaque) app background
holePunchOn = false
Log.i(TAG, "hole-punch OFF")
}
}
@@ -14,6 +14,8 @@ import com.getcapacitor.PluginMethod;
import com.getcapacitor.annotation.CapacitorPlugin;
import com.google.firebase.messaging.FirebaseMessaging;
import org.json.JSONObject;
// Native calling for Biz Connect (Android). Registered by `cap sync` as window.Capacitor.Plugins.NativeCall —
// the SAME jsName the iOS plugin uses, so the web bridge (setupNativeCall in home.html) drives both platforms
// through one code path.
@@ -86,6 +88,18 @@ public class NativeCallPlugin extends Plugin {
} catch (Exception ignored) {}
}
// Native camera capture (Increment 2b) needs the CAMERA runtime grant. Best-effort like the mic: prompt
// without blocking; the grant persists so the camera works from this toggle on (or the next).
private void ensureCameraPermission() {
try {
if (getContext() != null
&& ContextCompat.checkSelfPermission(getContext(), Manifest.permission.CAMERA) != PackageManager.PERMISSION_GRANTED
&& getActivity() != null) {
ActivityCompat.requestPermissions(getActivity(), new String[]{ Manifest.permission.CAMERA }, 7732);
}
} catch (Exception ignored) {}
}
@Override
protected void handleOnDestroy() {
super.handleOnDestroy();
@@ -156,6 +170,7 @@ public class NativeCallPlugin extends Plugin {
if (!uuid.isEmpty()) CallStore.markEnded(uuid);
CallStore.cancel(getContext());
IncomingCallActivity.dismissIfShowing();
try { CallVideo.setHolePunch(getActivity(), getBridge() != null ? getBridge().getWebView() : null, false); } catch (Exception ignored) {}
CallMedia.disconnect();
CallForegroundService.stop(getContext());
call.resolve();
@@ -208,17 +223,65 @@ public class NativeCallPlugin extends Plugin {
CallMedia.connect(getContext(), url, token, true); // start muted; web re-applies on callConnected
}
// --- Increment 2b methods (video tiles / screen-share / hole-punch / #5 transcript). Safe no-ops for now so
// the shared web bridge never rejects on Android; real bodies land with the native-video build. ---
@PluginMethod public void setCamera(PluginCall call) { call.resolve(); }
@PluginMethod public void switchCamera(PluginCall call) { call.resolve(); }
// --- Increment 2b: native video tiles (camera + hole-punch rendering). ---
// Publish/unpublish the local camera. The web re-syncs tiles right after this resolves so the local tile
// picks up (or drops) the native video.
@PluginMethod
public void setCamera(PluginCall call) {
boolean on = Boolean.TRUE.equals(call.getBoolean("on", false));
if (on) ensureCameraPermission();
CallMedia.setCameraEnabled(on);
JSObject ret = new JSObject();
ret.put("on", on);
call.resolve(ret);
}
// Flip the local camera front/back.
@PluginMethod
public void switchCamera(PluginCall call) {
CallMedia.switchCamera();
call.resolve();
}
// Hole-punch: transparent WebView over native video behind it. Toggled by the web when a native call's
// meeting view is shown/left (gated on the ANDROID_NATIVE_VIDEO server flag).
@PluginMethod
public void setHolePunch(PluginCall call) {
boolean on = Boolean.TRUE.equals(call.getBoolean("on", false));
CallVideo.setHolePunch(getActivity(), getBridge() != null ? getBridge().getWebView() : null, on);
call.resolve();
}
// Position native video renderers to match the web meeting tiles. tiles = [{uid, local, screen, x, y, w, h}]
// in CSS px. Polled by the web while a native call is on screen.
@PluginMethod
public void syncVideoTiles(PluginCall call) {
java.util.List<CallVideo.Tile> list = new java.util.ArrayList<>();
JSArray arr = call.getArray("tiles");
if (arr != null) {
for (int i = 0; i < arr.length(); i++) {
JSONObject o = arr.optJSONObject(i);
if (o == null) continue;
String uid = o.optString("uid", "");
if (uid.isEmpty()) continue;
boolean local = o.optBoolean("local", false);
boolean screen = o.optBoolean("screen", false);
String key = local ? "__local" : uid;
list.add(new CallVideo.Tile(key, uid, local, screen,
o.optDouble("x", 0), o.optDouble("y", 0), o.optDouble("w", 0), o.optDouble("h", 0)));
}
}
CallVideo.syncTiles(getActivity(), getBridge() != null ? getBridge().getWebView() : null, list);
call.resolve();
}
// --- Still no-ops: screen-share (MediaProjection) + tile zoom + #5 transcript land in a later increment. ---
@PluginMethod public void startScreenShare(PluginCall call) { call.resolve(); }
@PluginMethod public void stopScreenShare(PluginCall call) { call.resolve(); }
@PluginMethod public void startMeetingScreenShare(PluginCall call) { call.resolve(); }
@PluginMethod public void stopMeetingScreenShare(PluginCall call) { call.resolve(); }
@PluginMethod public void setHolePunch(PluginCall call) { call.resolve(); }
@PluginMethod public void setTileZoom(PluginCall call) { call.resolve(); }
@PluginMethod public void syncVideoTiles(PluginCall call) { call.resolve(); }
@PluginMethod public void startTranscription(PluginCall call) { call.resolve(); }
@PluginMethod public void stopTranscription(PluginCall call) { call.resolve(); }
@PluginMethod public void feedAudio(PluginCall call) { call.resolve(); }