Hi All, I'm working on a camera system extension where the main app is supposed to transfer a video stream using IOSurface memory sharing to the cam extension.
I have built a sample app that does contains all the logic, but without a camera extension. So I'm essentially using IOSurface to render a video in one SwiftUI view and show the result in another SwiftUI view. Just for testing purposes. And everything works fine so far.
Now, when moving the receiver code to the camera extensions, I'm having problems in accessing the IOSurface via ID. I am sharing the IOSurface ID via UserDefaults. I know from the logs the ID is correctly transferred.
Here is the code that uses IOSurfaceLookup to get the IOSurface. But this fails with the given message. The error message prints the surface ID which is the correct one. I know this from the main app where I get the ID and print it as well.
private var surfaceId: Int = -1 {
didSet {
logger.info("surfaceId has changed")
if surfaceId == -1 {
stopReceivingFrames()
ioSurface = nil
} else {
guard let surface = IOSurfaceLookup(IOSurfaceID(surfaceId)) else {
logger.error("failed to lookup IOSurface with ID: \(self.surfaceId)")
return
}
self.ioSurface = surface
logger.info("surface set, now starting receiving frames")
startReceivingFrames()
}
}
}
My gut feeling says that this issue might be related to some missing entitlement, sandboxing. In general, I have a working camera extension. I'm just not able to render a video in the main app, and send it over to the camera extension to overlay another web cam.
Both, the main app and camera extension are in the same XCode workspace and share the same AppGroup.
In short, my actual questions are:
- Is there any entitlement required for using IOSurface between app and camera system extension?
- Is using IOSurface actually possible in system extensions?
- Is there any specific setting/requirement that I need to handle to make this work?
Your single-app test shows why the lookup fails in the extension. In a quick test with macOS 27.0.1, IOSurfaceLookup(_:) found a surface by its ID in the process that created it. A second, unsandboxed process got nil for the same ID, and found the surface only when I created it as global. Both of your SwiftUI views run in one process, so the lookup works there. As Create camera extensions with Core Media IO explains, your extension runs in "its own daemon process that's sandboxed and run as a role user."
The header for kIOSurfaceIsGlobal says "If true, the IOSurface may be looked up by any task in the system by its ID." That key is deprecated with the message "Global surfaces are insecure," so don't rely on it. Without it, sharing the ID through your app group can't make the lookup work, and no documented entitlement changes that. IOSurface itself works in a system extension. The Camera Extension template in Xcode creates its pixel buffers with kCVPixelBufferIOSurfacePropertiesKey, which backs them with IOSurfaces. What doesn't work is finding another process's surface by its ID.
The documented way for an app to send video frames to a camera extension is a sink stream. The same session describes it: "Sink streams consume data from an app. Clients feed your sink stream by inserting sample buffers into a simple queue."
- In the extension, create another stream with the
.sinkdirection. After a client starts it, callconsumeSampleBuffer(from:completionHandler:)to receive each buffer, thennotifyScheduledOutputChanged(_:)once you've consumed it. - In the app, find your device and its sink stream by reading
kCMIOHardwarePropertyDevicesandkCMIODevicePropertyStreamswithCMIOObjectGetPropertyData(_:_:_:_:_:_:_:). Get the stream's queue withCMIOStreamCopyBufferQueue(_:_:_:_:), start the stream withCMIODeviceStartStream(_:_:), and add each frame as aCMSampleBufferwithCMSimpleQueueEnqueue(_:element:). AVFoundation has no equivalent for sending frames to a device, so the app side uses the C API directly.
Your extension can then combine those frames with the frames from the other webcam and send the result through its source stream. I haven't built a sink stream for this reply, so these steps come from the session and the headers, not from a test.