Sravan/BizGaze_Remote
0
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Hole-punch: render native video BEHIND a transparent WebView

Browse Source 
Permanent fix for the z-order whack-a-mole (controls/menus/panels hiding
behind the native video). Instead of drawing native video ON TOP of the
WebView, draw it BEHIND a transparent WebView so ALL web UI floats on top
naturally — no suppressing, no clamping, no docked-only bar.

Plugin:
- setHolePunch(on): webView.isOpaque=false + black layer bg + clear scroll
  bg (restored on off / call end). Tiles inserted belowSubview:scrollView.
- Dropped native name/mute overlays (the web tile's own .nm/.meet-mute/avatar
  render on top now) and the PaddingLabel.
- Zoom re-plumbed: touches hit the WebView, so native gesture zoom can't work;
  new setTileZoom({uid,scale,tx,ty}) applies a web-forwarded transform to the
  tile's inner video. TileVideoView simplified to a container + applyZoom.

Web:
- bzNativeStartTiles/StopTiles toggle NC.setHolePunch + a body.bz-hp class
  (only when the plugin supports it — old builds keep the suppress fallback).
- Tiles with live native video get .bz-hasvid → CSS makes them transparent +
  hides the web avatar so the video shows through; name/mute/border stay.
- New web-forwarded pinch/pan/double-tap on the shared screen → setTileZoom.
- Suppress-on-overlay + the height clamp now only apply when NOT hole-punched.

Needs a Codemagic build (plugin). Web deployed; no-ops to the prior behavior
on builds without setHolePunch.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Sravan
2026-08-07 11:34:09 +05:30
parent f971908d80
commit bdd0058c5c
2 changed files with 139 additions and 135 deletions
Download Patch File Download Diff File Expand all files Collapse all files
mobile/plugins/native-call/ios/Sources/NativeCallPlugin/NativeCallPlugin.swift
+71 -118
View File
@@ -1,5 +1,6 @@
import Foundation
import UIKit
import WebKit
import Capacitor
import PushKit
import CallKit
@@ -31,6 +32,8 @@ public class NativeCallPlugin: CAPPlugin, CAPBridgedPlugin, PKPushRegistryDelega
CAPPluginMethod(name: "switchCamera", returnType: CAPPluginReturnPromise),
CAPPluginMethod(name: "startScreenShare", returnType: CAPPluginReturnPromise),
CAPPluginMethod(name: "stopScreenShare", returnType: CAPPluginReturnPromise),
CAPPluginMethod(name: "setHolePunch", returnType: CAPPluginReturnPromise),
CAPPluginMethod(name: "setTileZoom", returnType: CAPPluginReturnPromise),
CAPPluginMethod(name: "syncVideoTiles", returnType: CAPPluginReturnPromise),
CAPPluginMethod(name: "endCall", returnType: CAPPluginReturnPromise)
]
@@ -49,6 +52,12 @@ public class NativeCallPlugin: CAPPlugin, CAPBridgedPlugin, PKPushRegistryDelega
// drawn over the WebView and positioned to match the web meeting tiles (see syncVideoTiles). UIKit views —
// only ever touched on the main thread. TileVideoView adds pinch-zoom/pan for shared screens.
private var tileViews: [String: TileVideoView] = [:]
// Hole-punch: draw the native video BEHIND a transparent WebView so all web UI (bar, menus, panels) floats
// on top. Saved so we can restore the WebView when the call ends.
private var holePunchOn = false
private var savedWebOpaque = true
private var savedWebBg: UIColor?
private var savedScrollBg: UIColor?
override public func load() {
let config = CXProviderConfiguration(localizedName: "Biz Connect")
@@ -122,6 +131,7 @@ public class NativeCallPlugin: CAPPlugin, CAPBridgedPlugin, PKPushRegistryDelega
room = nil
Task { await r?.disconnect() }
removeAllTileViews()
DispatchQueue.main.async { [weak self] in guard let self = self, let web = self.bridge?.webView else { return }; self.applyHolePunchRestore(web) }
}
// MARK: - Native video (tile rendering, Increment 2b)
@@ -167,65 +177,60 @@ public class NativeCallPlugin: CAPPlugin, CAPBridgedPlugin, PKPushRegistryDelega
}
}
// Tags for the overlay subviews we add to each tile's VideoView (name label + mute badge).
private static let nameTag = 9001
private static let muteTag = 9002
// Build a tile view (a container holding a LiveKit VideoView) with its native name label (bottom-left) +
// mute badge (top-left) — the video covers the web tile, so these redraw the essentials.
private func makeTileView(host: UIView, key: String) -> TileVideoView {
let v = TileVideoView(frame: .zero) // sets up its inner VideoView + zoom gestures + cosmetics in init
host.addSubview(v)
// Name chip (bottom-left): white text on a dark translucent pill so it's legible over ANY video
// (plain white text vanished on bright footage).
let label = PaddingLabel()
label.tag = NativeCallPlugin.nameTag
label.font = .systemFont(ofSize: 12, weight: .semibold)
label.textColor = .white
label.backgroundColor = UIColor.black.withAlphaComponent(0.5)
label.layer.cornerRadius = 7
label.clipsToBounds = true
label.translatesAutoresizingMaskIntoConstraints = false
v.addSubview(label)
// Mute badge (top-left): a RED circle with a white mic-slash — matches the web tile's .meet-mute (#dc2626).
let badge = UIImageView(image: UIImage(systemName: "mic.slash.fill")?.withConfiguration(
UIImage.SymbolConfiguration(pointSize: 11, weight: .semibold)))
badge.tag = NativeCallPlugin.muteTag
badge.tintColor = .white
badge.contentMode = .center
badge.backgroundColor = UIColor(red: 0.863, green: 0.149, blue: 0.149, alpha: 1) // #dc2626
badge.layer.cornerRadius = 10
badge.clipsToBounds = true
badge.translatesAutoresizingMaskIntoConstraints = false
v.addSubview(badge)
NSLayoutConstraint.activate([
label.leadingAnchor.constraint(equalTo: v.leadingAnchor, constant: 6),
label.bottomAnchor.constraint(equalTo: v.bottomAnchor, constant: -6),
label.trailingAnchor.constraint(lessThanOrEqualTo: v.trailingAnchor, constant: -6),
badge.leadingAnchor.constraint(equalTo: v.leadingAnchor, constant: 6),
badge.topAnchor.constraint(equalTo: v.topAnchor, constant: 6),
badge.widthAnchor.constraint(equalToConstant: 20),
badge.heightAnchor.constraint(equalToConstant: 20),
])
// Create a tile view and place it BEHIND the web content (hole-punch), so the web tile's own name/mute/
// border/avatar render on top. No native overlays needed — the WebView draws them.
private func makeTileView(host: WKWebView, key: String) -> TileVideoView {
let v = TileVideoView(frame: .zero)
host.insertSubview(v, belowSubview: host.scrollView)
tileViews[key] = v
return v
}
private func updateTileOverlay(_ vv: TileVideoView, name: String, muted: Bool) {
// Keep the overlays above the video renderer (VideoView adds its renderer when the track is set).
// Hide them while zoomed into a screen — a scaled name chip over content just gets in the way.
let zoomed = vv.isZoomed
if let label = vv.viewWithTag(NativeCallPlugin.nameTag) as? UILabel {
label.text = name; label.isHidden = zoomed || name.isEmpty; vv.bringSubviewToFront(label)
}
if let badge = vv.viewWithTag(NativeCallPlugin.muteTag) {
badge.isHidden = zoomed || !muted; vv.bringSubviewToFront(badge)
// Turn hole-punch on/off. On: make the WebView transparent (its black layer bg shows where the web page is
// transparent, i.e., over the meeting tiles) so the native video BEHIND it shows through. Off: restore.
@objc func setHolePunch(_ call: CAPPluginCall) {
let on = call.getBool("on") ?? false
DispatchQueue.main.async { [weak self] in
guard let self = self, let web = self.bridge?.webView else { call.resolve(); return }
if on {
if !self.holePunchOn {
self.savedWebOpaque = web.isOpaque
self.savedWebBg = web.backgroundColor
self.savedScrollBg = web.scrollView.backgroundColor
}
web.isOpaque = false
web.backgroundColor = .black // shows through the transparent meeting; also the letterbox bg
web.scrollView.backgroundColor = .clear
self.holePunchOn = true
} else {
self.applyHolePunchRestore(web)
for (_, vv) in self.tileViews { vv.removeFromSuperview() }
self.tileViews.removeAll()
}
call.resolve()
}
}
// Restore the WebView to opaque (must run on main).
private func applyHolePunchRestore(_ web: WKWebView) {
guard holePunchOn else { return }
web.isOpaque = savedWebOpaque
web.backgroundColor = savedWebBg
web.scrollView.backgroundColor = savedScrollBg
holePunchOn = false
}
// Web-forwarded zoom: touches land on the WebView (on top), so the web captures pinch/pan on the shared
// screen and forwards the transform here; we apply it to that tile's inner video.
@objc func setTileZoom(_ call: CAPPluginCall) {
let key = tileKey(uid: call.getString("uid") ?? "", isLocal: call.getBool("local") ?? false)
let scale = CGFloat(call.getDouble("scale") ?? 1)
let tx = CGFloat(call.getDouble("tx") ?? 0)
let ty = CGFloat(call.getDouble("ty") ?? 0)
DispatchQueue.main.async { [weak self] in self?.tileViews[key]?.applyZoom(scale: scale, tx: tx, ty: ty) }
call.resolve()
}
// Route call audio to the LOUDSPEAKER when we'd otherwise be on the quiet earpiece. Guarded so a connected
// wired/Bluetooth headset (any non-receiver route) still wins. Re-asserted on mute toggles because enabling
// the mic can flip the route back to the earpiece.
@@ -379,15 +384,12 @@ public class NativeCallPlugin: CAPPlugin, CAPBridgedPlugin, PKPushRegistryDelega
: self.cameraTrack(forUid: uid, isLocal: isLocal) else { continue }
wanted.insert(key)
let vv = self.tileViews[key] ?? self.makeTileView(host: host, key: key)
if vv.superview !== host { host.addSubview(vv) }
vv.isScreen = wantScreen // enables pinch-zoom on shared screens (resets zoom if turned off)
if vv.superview !== host { host.insertSubview(vv, belowSubview: host.scrollView) } // keep BEHIND web UI
vv.layoutMode = wantScreen ? .fit : .fill
if vv.track !== track { vv.track = track }
// The container always tracks the tile rect; the zoom transform lives on the INNER video, so
// this never fights an active zoom.
// The container tracks the tile rect; the zoom transform (web-forwarded via setTileZoom) lives on
// the INNER video, so this never fights an active zoom.
vv.frame = CGRect(x: x, y: y, width: w, height: h)
// Native video covers the web tile, so redraw the essentials (name + muted) natively.
self.updateTileOverlay(vv, name: (t["name"] as? String) ?? "", muted: (t["muted"] as? Bool) ?? false)
}
// Drop views for participants no longer present / camera turned off.
for (key, vv) in self.tileViews where !wanted.contains(key) {
@@ -547,16 +549,12 @@ public class NativeCallPlugin: CAPPlugin, CAPBridgedPlugin, PKPushRegistryDelega
}
// A tile view = a container holding a LiveKit VideoView. We CANNOT subclass VideoView (it's `public`, not
// `open`, so subclassing outside its module is illegal), so we compose instead. Pinch-to-zoom + pan apply to
// the INNER video while it's showing a shared SCREEN (so fine print is legible); double-tap resets. The
// container itself stays frame-synced to the web tile rect, so zooming never fights the position poll, and the
// name/mute overlays (added to the container by makeTileView) don't scale with the zoom.
final class TileVideoView: UIView, UIGestureRecognizerDelegate {
// `open`, so subclassing outside its module is illegal), so we compose. Under hole-punch the video sits BEHIND
// the (transparent) WebView, so touches never reach it — pinch-zoom is captured by the web and forwarded via
// applyZoom(). The container stays frame-synced to the web tile rect; the zoom transform lives on the inner
// video, so the two never fight.
final class TileVideoView: UIView {
let video = VideoView()
var isScreen = false { didSet { if !isScreen { resetZoom() } } }
var isZoomed: Bool { !video.transform.isIdentity }
private var zoomScale: CGFloat = 1
private var zoomOffset: CGPoint = .zero
// Forward the two properties the plugin sets so call sites read like a VideoView.
var track: VideoTrack? { get { video.track } set { video.track = newValue } }
@@ -564,12 +562,11 @@ final class TileVideoView: UIView, UIGestureRecognizerDelegate {
override init(frame: CGRect) {
super.init(frame: frame)
backgroundColor = .black
backgroundColor = .clear // the web tile draws its own frame; gaps show the WebView's black bg
clipsToBounds = true
layer.cornerRadius = 8
layer.cornerRadius = 12 // match .meet-tile's border-radius so corners don't poke past the web border
video.layoutMode = .fill
addSubview(video)
installZoomGestures()
}
required init?(coder: NSCoder) { fatalError("init(coder:) has not been implemented") }
@@ -580,53 +577,9 @@ final class TileVideoView: UIView, UIGestureRecognizerDelegate {
video.center = CGPoint(x: bounds.midX, y: bounds.midY)
}
private func installZoomGestures() {
isUserInteractionEnabled = true
let pinch = UIPinchGestureRecognizer(target: self, action: #selector(onPinch(_:))); pinch.delegate = self
let pan = UIPanGestureRecognizer(target: self, action: #selector(onPan(_:))); pan.delegate = self
let dtap = UITapGestureRecognizer(target: self, action: #selector(onDoubleTap)); dtap.numberOfTapsRequired = 2
addGestureRecognizer(pinch); addGestureRecognizer(pan); addGestureRecognizer(dtap)
}
func resetZoom() {
zoomScale = 1; zoomOffset = .zero
if !video.transform.isIdentity { video.transform = .identity }
}
private func apply() {
video.transform = CGAffineTransform(translationX: zoomOffset.x, y: zoomOffset.y).scaledBy(x: zoomScale, y: zoomScale)
}
@objc private func onPinch(_ g: UIPinchGestureRecognizer) {
guard isScreen else { return }
if g.state == .changed {
zoomScale = min(5, max(1, zoomScale * g.scale)); g.scale = 1
if zoomScale <= 1 { zoomOffset = .zero }
apply()
} else if (g.state == .ended || g.state == .cancelled), zoomScale <= 1.01 {
resetZoom()
}
}
@objc private func onPan(_ g: UIPanGestureRecognizer) {
guard isScreen, zoomScale > 1 else { return }
let t = g.translation(in: self); g.setTranslation(.zero, in: self)
zoomOffset.x += t.x; zoomOffset.y += t.y
apply()
}
@objc private func onDoubleTap() { guard isScreen else { return }; resetZoom() }
// Let pinch + pan run together for a natural zoom/pan.
func gestureRecognizer(_ g: UIGestureRecognizer, shouldRecognizeSimultaneouslyWith other: UIGestureRecognizer) -> Bool { true }
}
// A UILabel with padding, so the name chip has breathing room around its text.
final class PaddingLabel: UILabel {
var insets = UIEdgeInsets(top: 2, left: 6, bottom: 2, right: 6)
override func drawText(in rect: CGRect) { super.drawText(in: rect.inset(by: insets)) }
override var intrinsicContentSize: CGSize {
let s = super.intrinsicContentSize
return CGSize(width: s.width + insets.left + insets.right, height: s.height + insets.top + insets.bottom)
// Web-forwarded zoom (scale + translation in points). scale<=1 clears the transform.
func applyZoom(scale: CGFloat, tx: CGFloat, ty: CGFloat) {
if scale <= 1.001 { if !video.transform.isIdentity { video.transform = .identity } }
else { video.transform = CGAffineTransform(translationX: tx, y: ty).scaledBy(x: scale, y: scale) }
}
}