AVFoundation

RSS for tag

Work with audiovisual assets, control device cameras, process audio, and configure system audio interactions using AVFoundation.

Posts under AVFoundation tag

200 Posts

Post

Replies

Boosts

Views

Activity

iPhone 18 Pro: AVAudioEngine input tap returns all zeros after the Always-On Display turns off overnight
My app records audio overnight in the background: UIBackgroundModes audio, AVAudioSession category playAndRecord with mode measurement, and an AVAudioEngine with a tap on inputNode. On a customer's iPhone 18 Pro running iOS 27.2.1 (24B156), the tap starts delivering buffers of exact zeros about two hours into the night. It keeps delivering zeros until she picks the phone up in the morning. Her sysdiagnose shows the same sequence on three consecutive nights: powerd logs SmartPowerNap: enterSmartPowerNap. The backlight state changes from inactiveOnLive to suppressed, with the explanation "alwaysOn suppressed - SmartPowerNap". The panel reaches 0 nits about two seconds later. Under a second after that, every sample in the recording is 0. PowerLog's per-app mic RMS reads -240 from that point on. Hours later powerd logs Exiting SmartPowerNap. Session end reason MotionDetected. The display wakes and mic input returns within a second. Timing on the third night: 23:55:57.234 Smart Power Nap entry 23:55:59.14 panel at 0 nits 23:55:59.8 first zero sample in the recording 05:46:58.032 Smart Power Nap exit about 05:46:59 mic input returns During the silent hours: The app receives no interruption, route change, media services reset or AVAudioEngineConfigurationChange notification. The engine keeps running and the tap keeps firing on schedule. audiomxd logs no deactivation for the session and holds its power assertions throughout. The system never sleeps. I can't reproduce it. My iPhone 16 Pro on the same build logs "SmartPowerNap: feature not enabled or predictor not initialized" on every lock, so it never takes this path. Questions: Does iOS gate mic input by design while Smart Power Nap suppresses the Always-On Display? Can an app keep recording through it, or get a notification that input has stopped? Does restarting AVAudioEngine while the session stays active restore input in this state? I can detect a run of exact zeros and restart the engine, but I have no device to test it on. Has anyone else with iPhone 18 Pro users seen this? I filed FB25663635 with the sysdiagnose attached.
0
0
73
1d
Camera preview rotated 90 degrees on initial presentation on iPadOS 27.2.1, corrected after returning from Control Center
Dear Apple Support Team, We are investigating a camera preview orientation issue on an iPad and would appreciate guidance on the relevant UIKit and AVFoundation behavior. Our app uses Flutter and the third-party Regula Document Reader SDK for passport MRZ scanning. We understand that third-party SDK internals are outside Apple’s support scope. Our question concerns the relationship between the active interface orientation, presentation lifecycle, and camera preview connection orientation. Environment Device: iPad (A16), hardware identifier iPad15,7 iPadOS: 27.2.1, as reported by the device and diagnostic export App distribution: TestFlight App version in the diagnostic log: 2.0.14, build 7 Flutter plugin: flutter_document_reader_api 9.8.1718 Runtime Regula SDK API version: 9.8.6944 Runtime Regula SDK Core version: 9.8.20492 Steps to reproduce Hold the affected iPad in landscape orientation and open the app. Open the passport scanner. The scanner interface appears in landscape, but the camera image appears rotated by 90 degrees. Open Control Center, then dismiss it and return to the app. The preview can then appear in the correct orientation. Close and reopen the scanner; the issue may recur. Physically rotating the device does not consistently resolve the issue. Expected: The preview should match the current interface orientation from initial presentation. Actual: The preview initially has an orientation mismatch and may recover only after leaving and returning to the app. Diagnostic observations We collected initial-entry and app-resume observations within the same scanner session: Before scanner presentation, the active UIWindowScene.interfaceOrientation is landscapeRight. The SDK’s presenting view controller belongs to the active application window. The scanner controller has landscape dimensions of approximately 1180 × 820. The third-party SDK’s supported interface orientation is configured for landscape right. However, the initial preview connection reports videoOrientation = portrait and videoRotationAngle = 90. Updating supported interface orientations and requesting layout updates did not resolve the initial mismatch in this test. A subsequent app-resume observation reports videoOrientation = landscapeRight and videoRotationAngle = 0. This change is recorded before our app’s delayed layout refresh. We recognize that interface orientation and capture rotation are different concepts, and that connection metadata alone does not establish the displayed pixel orientation. We are providing these observations alongside the visible symptom rather than treating a particular angle as universally correct. Scope and investigation status Our separate QR scanner and receipt camera work normally on the same device. The passport scanner also works normally on many other iPads. We have not established that this affects all devices running this iPadOS version. An app-side orientation enum comparison issue was corrected before collecting the build 7 log, but the preview mismatch remained. We have not isolated the cause to iPadOS, our presentation code, or the SDK’s capture initialization. We do not yet have a focused standalone Xcode reproduction project. We are also preparing an inquiry to the SDK vendor. Questions Are there known circumstances in which a camera preview initially uses portrait orientation despite being presented in a landscape UIWindowScene? What is the recommended lifecycle point for determining the initial interface orientation and configuring the preview connection, particularly while presentation or rotation transitions are in progress? Which supported APIs should be used to keep preview rotation synchronized with interface changes on this iPadOS version? Could the recovery after dismissing Control Center indicate an initialization or lifecycle ordering issue? What additional observations would help distinguish these possibilities? What should we include in a minimal native reproduction or system diagnostic collection to make this issue actionable? We can provide a symptom video and an application diagnostic export. These were collected in separate tests on the same affected device and are not time-synchronized. The application export is not an Apple sysdiagnose. Thank you for any guidance on how to investigate this further. With Best Regards, GTF_Japan_Service
0
0
38
3d
AVAudioSession interrupted with no .ended when a window changes under an open ImmersiveSpace
Title: AVAudioSession interrupted with no .ended when the window changes while an ImmersiveSpace is open Hi, I'm seeing an issue where an active voice call (mic capture + playback) goes silent as soon as the app enters its immersive experience. The system posts AVAudioSession.interruptionNotification with type .began, but .ended never arrives while the app stays in the foreground, so the audio never resumes on its own. Audio only comes back after the app goes inactive and active again (for example after leaving the immersive space). This looks like the same issue reported in https://developer.apple.com/forums/thread/761721 (2024), and it still reproduces on visionOS 26 and 27. Setup: App with a WindowGroup and an ImmersiveSpace. Voice runs on a WebRTC stack (LiveKit, AVAudioEngine-based) on top of AVAudioSession, configured as: category .playAndRecord, mode .videoChat, options [.mixWithOthers, .allowBluetoothHFP, .allowBluetoothA2DP, .allowAirPlay], then setActive(true). When the immersive space opens, an ARKitSession runs with WorldTrackingProvider, HandTrackingProvider and SharedCoordinateSpaceProvider. Steps to Reproduce: Start a voice call from the main window. Mic capture and playback both work. Open the ImmersiveSpace and replace the main window with another window (open the new window, dismiss the main one). Shortly after the space opens, interruptionNotification arrives with type .began and reason .default (0). The audio engine stops. Mic capture and playback both go silent. Expected: Audio keeps running when the window changes. If the system has to interrupt the session, .ended is posted so the app can resume. Actual: .began arrives with no matching .ended while the app remains in the foreground. Audio stays silent until the app goes inactive and active again. At that point .ended arrives with .shouldResume and the engine starts again. The same interruption sometimes hits the very first microphone start right after the session is activated, so the start times out. Nothing else in the app activates or deactivates the audio session at that point. Observation code: NotificationCenter.default.addObserver( forName: AVAudioSession.interruptionNotification, object: nil, queue: .main ) { note in let type = note.userInfo?[AVAudioSessionInterruptionTypeKey] as? UInt let reason = note.userInfo?[AVAudioSessionInterruptionReasonKey] as? UInt print("interruption type=\(String(describing: type)) reason=\(String(describing: reason))") // Logged: type=1 (.began), reason=0 (.default). No type=0 (.ended) while the app stays active. }Current workaround: On every interruptionNotification, and again on UIApplication.didBecomeActiveNotification, call setActive(true) and restart the audio engine if it isn't running. This restores audio while the immersive space stays open. Device: Apple Vision Pro visionOS version: visionOS 27.2 (24N120), also seen on visionOS 26.5 (23O471) Xcode version: Version 27.0 (27A266a) Questions: Is the app's audio session expected to be interrupted when a window closes or is replaced while an ImmersiveSpace is open? If so, why is no .ended posted while the app remains active, and is reactivating the session and restarting the engine the recommended way to resume? Thank you!
0
0
167
4d
AVSpeechSynthesizer pulls words out of thin air.
Hi, I'm working on a project that uses the AVSpeechSynthesizer and AVSpeechUtterance. I discovered by chance that the AVSpeechSynthesizer automatically completes some words instead of just outputting what it's supposed to. These are abbreviations for days of the week or months, but not all of them. I don't want either of them automatically completed, but only the specified text. The completion transcends languages. I have written a short example program for demonstration purposes. import SwiftUI import AVFoundation import Foundation let synthesizer: AVSpeechSynthesizer = AVSpeechSynthesizer() struct ContentView: View { var body: some View { VStack { Button { utter("mon") } label: { Text("mon") } .buttonStyle(.borderedProminent) Button { utter("tue") } label: { Text("tue") } .buttonStyle(.borderedProminent) Button { utter("thu") } label: { Text("thu") } .buttonStyle(.borderedProminent) Button { utter("feb") } label: { Text("feb") } .buttonStyle(.borderedProminent) Button { utter("feb", lang: "de-DE") } label: { Text("feb DE") } .buttonStyle(.borderedProminent) Button { utter("wed") } label: { Text("wed") } .buttonStyle(.borderedProminent) } .padding() } private func utter(_ text: String, lang: String = "en-US") { let utterance = AVSpeechUtterance(string: text) let voice = AVSpeechSynthesisVoice(language: lang) utterance.voice = voice synthesizer.speak(utterance) } } #Preview { ContentView() } Thank you Christian
5
0
1.9k
1w
iOS 27.2: iPhone 13 Pro / 14 Pro heat up sooner during 4K camera capture than on iOS 27.0
We make a camera app (AVCaptureSession at 4K, live preview, recording with AVAssetWriter). Since updating to iOS 27.2, a user with an iPhone 13 Pro reports that the phone gets hot while using our app. Our own iPhone 14 Pro also runs warmer on 27.2 than it did on 27.0. To check, we reran on the 14 Pro the same automated test we ran there on iOS 27.0 in September. Each run lasts 300 seconds with the phone on power and fully charged, Low Power Mode off, and starts at ProcessInfo.thermalState == .nominal. We log thermalState every 5 seconds. Capture is 3840x2160 at 60 fps in all three runs: Idle preview, no processing: iOS 27.0: stayed Nominal for all 300 s iOS 27.2 (24B115): Fair at 278 s Idle preview with a Metal overlay: iOS 27.0: stayed Nominal for all 300 s iOS 27.2: Fair at 123 s Recording, HEVC 10-bit: iOS 27.0: Fair at 38 s, Serious at 108 s iOS 27.2: Fair at 33 s, Serious at 83 s Frame delivery held up. The 5-minute recording lost no frames, even at Serious. Questions: Are there known changes in iOS 27.2 to power use or thermal management during camera capture (AVCaptureSession at 4K) on A15 / A16 devices? Have the thresholds behind ProcessInfo.thermalState or AVCaptureDevice.SystemPressureState changed in 27.2? What data would be most useful in a Feedback report for this: a sysdiagnose taken right after a run, or a specific Instruments template?
1
0
556
1w
AVAudioSession ioBufferDuration always default to 512 samples on MacOS27 no matter what is requested
Hello, I develop an audio processing demo app that has been working fine on iOS/iPad and on Mac ("developed for iPad") until I upgraded to MacOS27. The app is demonstrating low latency / high temporal resolution audio spectral analysis. I am using AVAudioSession, I request the audio IO buffer to be 32 samples. This still works fine on iOS/iPadOS, but it seems that when running on MacOS27 the buffer size always defaults to 512 samples no matter what I request (smaller or larger). This applies to the "Designed for iPad" version running on MacOS and any version running in the iPhone and iPad simulators on MacOS. The app seems to be working as intended on physical iPhones and iPads. I made a catalyst version and used AudioObjectSetPropertyData to force the IO bugger sizes to 32 samples on the system's active hardware device. This seems to succeed but still results in incorrect behaviour in my app; I am still diagnosing but I believe it's because this leaves AVAudioSession with incorrect data, and I am using AudioKit which relies on AVAudioSession (not confirmed). Regardless, is this change in behaviour intentional in MacOS27? Is there a way around it? The whole point of my app is to demonstrate efficient audio processing with very low latency - I need small input buffer size for that. Thank you for any help with this
2
0
442
1w
Can a third-party app capture microphone audio while a cellular call is active?
We're building a companion app that people use during a regular cellular call (VoIP isn't an option for this call). The user wants to briefly mute the call, speak a short question into our app, then unmute. Is there any supported way for a third-party app to record from the microphone during an active cellular call, even if the call is muted or on hold? Does Siri with App Intents accept voice input during an active call? If so, is Siri's spoken response heard by the other person on the call? Can a watchOS app record audio while the paired iPhone is on a call? If none of these work, what approach do you recommend?
1
0
165
2w
Swift 6: Supported AVCaptureSession actor isolation and MainActor preview attachment
I’m developing a SwiftUI camera preview for iPhone and iPad using Xcode 27.2 and Swift 6.4.2. I’m looking for clarification about the supported concurrency boundary between an actor-owned AVCaptureSession and a MainActor-owned AVCaptureVideoPreviewLayer. Environment Xcode 27.2 (27B52) Apple Swift 6.4.2 iPhoneOS SDK 27.2 Deployment target: iOS 26.0 Swift 6 language mode Complete concurrency checking Default actor isolation: MainActor Approachable Concurrency enabled The isolated compiler reproductions also explicitly enable NonisolatedNonsendingByDefault, InferIsolatedConformances, and MemberImportVisibility. Architecture The intended architecture follows the general pattern demonstrated in Apple’s AVCam sample: A capture actor uses a dedicated DispatchSerialQueue executor. The capture actor creates, configures, starts, and stops AVCaptureSession. A MainActor adapter owns AVCaptureVideoPreviewLayer. The preview layer retains the same session after attachment. The capture actor retains its session reference because it must perform subsequent capture operations. Compiler behavior In a minimal reproduction, transferring the retained session directly to the MainActor adapter: await preview.attach(session)produces: Sources/CaptureActor.swift:19:23: error: sending 'session' risks causing data races [#RegionIsolation::SendingRisksDataRace]The compiler identifies a potential race between actor-isolated and MainActor-isolated uses. This reproduction produces one error and zero warnings. A second reproduction uses a small PreviewSource: Sendable value containing a private AVCaptureSession reference, with @preconcurrency import AVFoundation confined to that source file. Its MainActor method attaches the session to the preview layer. Under otherwise identical compiler settings, this second reproduction compiles with zero diagnostics. We understand that compiler acceptance does not establish runtime safety and that the compatibility mechanism does not itself synchronize access to the retained session. Apple AVCam reference Apple’s official AVCam sample demonstrates a similar interoperability pattern: CaptureService.swift: dedicated capture executor and retained preview source. PreviewSource.swift: compatibility import and MainActor attachment. CameraPreview.swift: preview-layer association. Reference: Apple’s AVCam: Building a Camera App sample. Pinned sample SHA-256: 5a1981f90bb30827121324d4b33198d58de9b9d68e3508b64f0aabc18bbc0e53The sample revision date is not established; we are not assuming this archive is identical to the latest downloadable revision. Questions 1. Supported Swift 6 interoperability pattern Is the narrow PreviewSource compatibility approach demonstrated by AVCam the recommended way to associate an actor-retained AVCaptureSession with a MainActor-owned AVCaptureVideoPreviewLayer under complete Swift 6 concurrency checking? Is there a newer recommended API, annotation, or ownership pattern? 2. Capture and preview synchronization Is the following operation sequence supported? Configure and commit the capture session on the dedicated capture executor. Attach the session to the preview layer on MainActor. Call startRunning() on the capture executor. Perform ordinary preview layout on MainActor while capture runs. Call stopRunning() on the capture executor. Detach the session from the preview layer on MainActor. Which preview-layer or preview-connection mutations, if any, must be excluded while startRunning() or stopRunning() is executing? 3. Supported detachment and release sequence After stopRunning() returns and MainActor executes: previewLayer.session = nilis it supported to release the remaining application-held session references once application-owned observers and asynchronous consumers have terminated? Does AVFoundation require or recommend any additional step before those references are released? We are asking about the supported framework lifecycle contract, not assuming that application-level cancellation drains internal AVFoundation work. Sources AVCam: Building a camera app Overview Apple’s official AVCam sample
0
1
293
2w
iOS – Custom voice keyword detection when iPhone is locked
Hello, I am currently working on an iOS app and studying the feasibility of a custom voice detection feature. The user might set up a secret voice keyword (a custom word or phrase). When this keyword is spoken, even when the iPhone is locked and stored in a pocket, the application would automatically trigger an action. My questions are: Can an iOS app continuously detect a custom voice keyword when running in the background and iPhone is locked? Is there an officially supported and App Store compatible method to perform this detection locally, without recording or transmitting audio? Can an application automatically trigger an action following this detection? Would using AVAudioSession and the audio mode in the background be an acceptable solution for this use case? Are there any Apple frameworks suitable for this type of feature? I would be particularly interested in feedback from Apple engineers or developers who have experimented with similar features. Thank you in advance for your help.
0
0
100
2w
AVCaptureSession multitasking camera access on iPhone without VoIP background mode
I'm developing an iPhone screen-recording application that simultaneously records the front-facing camera and displays the live camera feed using Picture in Picture while the user navigates other apps. The implementation uses AVFoundation (AVCaptureSession) for camera capture and a system Picture in Picture controller for the floating window. I've encountered the following behavior on a physical iPhone: With voip declared in UIBackgroundModes, isMultitaskingCameraAccessSupported returns true, and the front camera continues capturing while another app is foregrounded. Without voip, isMultitaskingCameraAccessSupported returns false. When the application enters the background, the capture session is interrupted and camera frames stop. Screen recording and the Picture in Picture window continue, but the camera feed freezes or becomes blank. Apple App Review has confirmed that declaring voip without actual VoIP functionality is not permitted under Guideline 2.5.4. I understand and respect that restriction and do not want to misuse background modes. My question: Is there a supported public API, entitlement, or configuration that allows a non-VoIP iPhone application to continue capturing front-camera video while another application is in the foreground? Does Picture in Picture provide any supported mechanism for this use case, or is multitasking camera capture unavailable to non-VoIP applications on iPhone? I'm looking for a supported implementation rather than a workaround that violates App Review guidelines. Any guidance from Apple's AVFoundation engineers would be greatly appreciated.
2
0
485
2w
CallKit VoIP call on the receiver: the side (lock) button only locks the screen, no CXEndCallAction is delivered
Device and versions iPhone 15, iOS 27.0.1 App: Hush, a private messenger. Capacitor 8.5.2 iOS app; the UI is a web page in a WKWebView. Call signalling runs in the page; call audio runs in a native Swift Capacitor plugin on stasel/WebRTC 154.0.0, integrated with CallKit. Built on Codemagic with the latest Xcode, distributed through TestFlight. UIBackgroundModes: audio, voip. No PushKit yet: calls are placed and answered in the app's own screen while it is in the foreground. Setup RTCAudioSession manual audio (useManualAudio = true). isAudioEnabled is set only after ICE connects AND provider(_:didActivate:) is called (a 3-second fallback exists but never fired in these tests). We call RTCAudioSession.audioSessionDidActivate / audioSessionDidDeactivate from the provider delegate. Session: playAndRecord, mode voiceChat, options allowBluetoothHFP, no defaultToSpeaker. overrideOutputAudioPort is only called when the user picks Speaker (never in the failing tests). One CXProvider, created lazily, delegate on the main queue. Configuration: supportsVideo false (also tested true), maximumCallGroups 1, maximumCallsPerCallGroup 1, supportedHandleTypes [.generic], includesCallsInRecents false (also tested true). Outgoing: CXStartCallAction is requested via CXCallController when ICE first connects (not at dial time; the page plays its own ringback until then), handle generic, value "Hush". In perform(CXStartCallAction) we configure the audio session (no activation), reportCall(with:updated:), fulfil, then reportOutgoingCall(startedConnectingAt:) and (connectedAt:). Answering side: reportNewIncomingCall, then an immediate CXAnswerCallAction (the user already accepted in the app's screen). Mute: CXSetMutedCallAction. In-app End reported the call with reportCall(with:endedAt:reason: .remoteEnded) in all tests below (we are changing this to a CXEndCallAction request; not yet tested). Proximity: isProximityMonitoringEnabled = true from dialling while on the receiver. The screen does go dark at the ear. Also tested with our proximity monitoring off. The problem With a Hush call live, route Receiver, CallKit holding the call (the system call UI shows it), didActivate delivered exactly once, our audio running inside that activation, proximity on and the screen dark at the ear: pressing the side button only locks the screen. No CXEndCallAction arrives and the call continues. In the Phone app on the same iPhone the side button ends the call at the ear. Snapchat calls on this iPhone end on the side button while Snapchat's call screen is open. What works: End on the system lock-screen call UI delivers a CXEndCallAction and we end the call. Mute from the system UI arrives as CXSetMutedCallAction. So provider delegate delivery works. What we tried, in order (each tested on device, with a readout of the plugin's state after the call) First CallKit build: side button did nothing; lock-screen End worked. Readout: "CallKit off, its audio off (0x), fallback no, route speaker, proximity on". Cause: CXStartCallAction was refused (missing voip background mode) and the error swallowed. Fixed: voip added; on CXErrorCodeRequestTransactionError.unknownCallProvider we invalidate and recreate the provider and retry once. Next build: CallKit held the call, didActivate once, route Receiver. Side button still did nothing, no CXEndCallAction. Our route log showed two route changes caused by our own overrideOutputAudioPort(.none) on every activation and route change. Removed: no override unless Speaker is selected. Result: no change. Diagnostic switches, tested in combinations: our proximity monitoring on/off; supportsVideo false/true; includesCallsInRecents false/true; callee now reported with reportNewIncomingCall + CXAnswerCallAction (previously both sides used CXStartCallAction). Result: the side button ended the call EXACTLY ONCE: the first call after the provider was recreated (changing supportsVideo makes us invalidate() and create a new CXProvider). Every later call with the same configuration failed. Readout for the failures: "CallKit on, its audio on (1x), fallback no, route Receiver, proximity on", screen dark at the ear. Following thread 837211 (creating a provider implicitly pushes the audio session ID to callservicesd, which can keep a stale one; set the configuration before reporting each call): we now assign provider.configuration = a fresh CXProviderConfiguration at call start and again immediately before each CXStartCallAction request / reportNewIncomingCall. We also added a switch that recreates the CXProvider for every call. With the re-push alone: no end action. With a fresh provider every call: no end action either. Readout each time: "provider calls 1, config re-pushed yes, CallKit on, its audio on (1x), route Receiver, proximity on". The single success in step 3 has not been reproduced. Not tried on purpose: detecting the lock ourselves (protectedDataWillBecomeUnavailable arrives about 10 seconds late and needs a passcode; a 2017 thread says Home and Lock cannot be told apart); reporting the CallKit call at dial time instead of ICE connect; PushKit. Threads we relied on 819363 (April 2026): the lock button hangs up a CallKit call handled directly by the device (not through a headset), delivered as the provider's end action. 837211 (August 2026): stale audio session ID in callservicesd after provider creation; set the configuration before reporting a call. How we differ from apps where it works WKWebView: WebKit's GPU process holds a second AVAudioSession in our app's name. During the call the page captures no mic (the plugin owns it); before connect the page plays a Web Audio ringback (Ambient category). The CallKit call starts at ICE connect, not at dial time. In-app End reported as .remoteEnded rather than requested as a CXEndCallAction. The callee's immediate CXAnswerCallAction after reportNewIncomingCall. Questions Which conditions must hold for iOS to treat the side button as "end call" for a CallKit call, and which is our app failing? Is there a log subsystem or category (callservicesd, TelephonyUtilities) we can read in a sysdiagnose to see why the press is handled as lock? Could WebKit's GPU-process audio session in the same app confuse CallKit about which session carries the call? Would reporting the call at dial time, before any audio, change the outcome? Can ending the in-app call with reportCall(.remoteEnded) instead of a CXEndCallAction leave state in callservicesd that affects the NEXT call? That would fit "works once after a fresh provider, never again". We have no Mac. Is a sysdiagnose captured on the iPhone right after a failing press enough to diagnose this? If you need a project, we can provide the plugin's Swift source (one file) and the Capacitor app project that Codemagic builds.
1
0
156
2w
Public API for torch beam width (Adaptive True Tone flash)?
On iPhone 14 Pro and later, the system flashlight in Control Center can adjust beam width (narrow ↔ wide). As far as I can tell, AVCaptureDevice only exposes torchMode, torchLevel and setTorchModeOn(level:) I found nothing related to beam width Is there any supported way for a third-party app to control the torch beam width? If not, is this on the roadmap?
0
0
338
2w
Improving metric scale consistency of ARWorldTrackingConfiguration on devices without LiDAR
We build a medical scanning app. A practitioner walks a slow orbit (~1 m radius, two laps, 25–35 s) around a standing subject while an ARSession runs, and we capture ~50 high-resolution stills with captureHighResolutionFrame. The ARKit camera poses for those frames are later used to set the metric scale of a photogrammetry reconstruction. We need the resulting scale to be within about 2 % of true. What we observe, measured against a printed A4 sheet on the floor in every scan: Devices with LiDAR (iPhone 16 Pro Max, iPad Pro), using the sceneDepth frame semantic as a check: scale within ±0.8 % across 11 sessions. Devices without LiDAR (iPhone 13, iPad 9th and 11th gen, plus non-LiDAR runs on the Pro devices): the world-frame metre is on average 2–3 % short of true, and varies by a further ±2 % from one session to the next, across 24 sessions in four rooms. Within a session the scale is stable: lap-to-lap variation is under 0.4 %. The error appears to be set during initialisation and held. Nothing we record about capture correlates with the error: time spent before capture, distance walked, vertical camera movement, orbit radius, rotation speed, tracking state, or the residual of our pose fit. Questions: Is there any session setup, initialisation practice, or motion during the first seconds of tracking that is known to improve the metric scale estimate on devices without LiDAR? Does the choice of video format, resolution or frame rate affect scale estimation? Is there any signal, beyond ARCamera.trackingState, that indicates how well scale has been determined for a session, so we could flag sessions rather than accept them? Does anything in the configuration above, in particular locking exposure on the capture device or using the high-resolution video format, work against tracking quality or scale?
1
0
912
2w
Is activeLockedVideoFrameDuration supported on iPhone 16 Pro? Hardware or iOS limitation?
Hello Apple AVFoundation team, Could you clarify the supported iPhone models and OS versions for standalone locked video frame duration using AVCaptureDeviceInput.activeLockedVideoFrameDuration? The API is documented as available from iOS 26.0, but I understand that OS availability alone does not mean every camera device/configuration supports it. I have read the API documentation and the DTS reply in iPhone 17 Pro's Genlock feature, but could not find a model compatibility list for the standalone locked-duration API. Scope: I am investigating actual capture timing for multi-iPhone motion capture. This question is about locking each camera's frame period using its internal timing, without an external genlock accessory. I am not asking about encoding a constant-frame-rate file, ordinary equal min/max frame durations, timecode metadata, or synchronizing the first exposure across phones. Physical-device test iPhone 16 Pro, iOS 26.2 (23C55). Built with Xcode 26.3 and the iOS 26.2 SDK. One AVCaptureSession, one physical rear .builtInWideAngleCamera input, and one AVCaptureVideoDataOutput. 8-bit SDR 420v formats, tested at 1920 x 1080 and 3840 x 2160, each at 30 and 60 fps. Auto video frame rate disabled; no spatial video capture or multicam session. Manual exposure approximately 2 ms, shorter than half a frame at both tested rates. The session is running and delivering frames. Two separate probe runs produced the same results, including capability readbacks after a settling period. For all four tested configurations: input.isLockedVideoFrameDurationSupported = false device.minSupportedLockedVideoFrameDuration = kCMTimeInvalid input.activeLockedVideoFrameDuration = kCMTimeInvalid device.isVideoFrameDurationLocked = falseOrdinary activeVideoMinFrameDuration and activeVideoMaxFrameDuration can both be set to the requested CMTime(value: 1, timescale: 30) or CMTime(value: 1, timescale: 60), and frames are delivered. This is only a configuration readback, not a claim of exact capture timing in the ordinary mode. The app directly queries the public support properties; it has no model whitelist. It does not attempt to set a valid locked duration when support is false, as the documented setter contract would reject that. These tests cover the selected formats, not every camera/format on this phone. Questions Which iPhone models support this standalone API? In particular, is false expected on iPhone 16 Pro? Is there an official compatibility list covering iPhone 13/14/15/16 and iPhone 17 models? Is the iPhone 16 Pro result a hardware capability restriction, an iOS 26.2 limitation/known issue, or a configuration requirement? Is there a released iOS version that enables this on iPhone 16 Pro, or is newer hardware required? What is the minimum supported capture configuration and correct time to query support? Are there restrictions involving the physical versus virtual camera input, pixel format, SDR/HDR, or exposure mode that could explain this result? If so, which configuration should be tested before concluding that the device does not support it? On supported hardware, can this standalone lock be enabled without an external accessory or a special entitlement, independently of follow(_:videoFrameDuration:delegate:)? Relevant documentation: activeLockedVideoFrameDuration isLockedVideoFrameDurationSupported minSupportedLockedVideoFrameDuration I can provide a focused reproducer if needed. Thank you for clarifying the device requirements and whether an OS upgrade can change the result.
1
0
655
3w
iPhone 17 Pro: Is Native Apple Log 2 ISP Processing Available for 4224×3024/60fps RAW?
Hi everyone, I'm developing an iOS camera app that aims to record 4224×3024 Open Gate video at 60fps in 10-bit HEVC with Apple Log 2. Native Apple ISP-processed Apple Log 2 capture works at 3840×2160/60fps. However, at 4224×3024/60fps, the tested capture configuration only provides Bayer RAW (btp2). I've successfully decoded ProRes RAW using Apple's decoder into linear Rec.2020 and applied the published Apple Gamut and Apple Log 2 transforms. However, the resulting chroma is only approximately 72–74% of the native ISP output in controlled color-chart comparisons. I've also confirmed that VTRAWProcessingSession and related RAW processing controls are not publicly available to iOS apps in the iPhoneOS 27.0 SDK. My questions are: Is there a public iOS API that applies the same color and tone rendering as Apple's native camera ISP to captured Bayer RAW or ProRes RAW? Is there a supported configuration for obtaining full-resolution 4224×3024/60fps ISP-processed Apple Log 2 output (x420 or x422), without preview proxies or upscaling? If neither is supported, what is Apple's recommended approach? Is this a hardware limitation or a public API limitation? The goal is to use Apple's official color processing pipeline without empirically fitted matrices, LUTs, or arbitrary color corrections. Any guidance would be greatly appreciated. Thank you!
0
0
357
3w
Radiometric interpretation of Apple ProRAW and Bayer RAW access via AVFoundation
I am working on a computational photography research project involving multi-exposure HDR reconstruction using Bayer RAW and Apple ProRAW captures. I would like to clarify the radiometric interpretation of Apple ProRAW and the availability of Bayer RAW capture through AVFoundation. My questions are: On current iPhone Pro devices, is it possible for third-party apps to capture and export true Bayer-pattern RAW DNG files through AVFoundation, rather than Apple ProRAW linear DNG files? If so, which availableRawPhotoPixelFormatTypes correspond to Bayer RAW, and what device or format restrictions apply? Apple ProRAW appears to be demosaiced and computationally processed, and may include multi-frame fusion. Is the decoded ProRAW image intended to be radiometrically linear and scene-referred? For a bracketed ProRAW sequence captured with fixed ISO, white balance, lens, and focus, but different exposure times, can one assume that the decoded linear pixel values Y_i(p) satisfy an exposure-proportional model in non-saturated regions, such as Y_i(p) ≈ t_i R(p), across brackets? This question is about radiometric consistency for algorithmic use, not about visual editing or tone mapping. Thank you for your help.
2
0
2.1k
3w
Radiometric interpretation of Apple ProRAW and Bayer RAW access via AVFoundation
I am working on a computational photography research project involving multi-exposure HDR reconstruction using Bayer RAW and Apple ProRAW captures. I would like to clarify the radiometric interpretation of Apple ProRAW and the availability of Bayer RAW capture through AVFoundation. My questions are: 1.On current iPhone Pro devices, is it possible for third-party apps to capture and export true Bayer-pattern RAW DNG files through AVFoundation, rather than Apple ProRAW linear DNG files? If so, which availableRawPhotoPixelFormatTypes correspond to Bayer RAW, and what device or format restrictions apply? 2.Apple ProRAW appears to be demosaiced and computationally processed, and may include multi-frame fusion. Is the decoded ProRAW image intended to be radiometrically linear and scene-referred? 3.For a bracketed ProRAW sequence captured with fixed ISO, white balance, lens, and focus, but different exposure times, can one assume that the decoded linear pixel values Y_i(p) satisfy an exposure-proportional model in non-saturated regions, such as Y_i(p) ≈ t_i R(p), across brackets? This question is about radiometric consistency for algorithmic use, not about visual editing or tone mapping. Thank you for your help.
2
0
1.7k
3w
TTS Stuttering & Noise
TTS Stuttering & Noise When using Apple’s native TTS API on an iPhone or iPad—specifically while connected to Bluetooth earphones or headsets—users are experiencing audio noise or intermittent gaps in speech, causing parts of sentences to drop out during playback. There were no reports or inquiries regarding such issues up through iOS/iPadOS 26. Since the release of iOS/iPadOS 27, however, there has been a significant increase in reports and inquiries about this problem. If you happen to know the cause of this issue or have any suggestions for a fix, please leave a reply. The issue persists even after removing the AirPods or headset from the device and reconnecting them. The issue persists even after rebooting the device. The issue occurs even on the latest versions, from iOS/iPadOS 27 through 27.2.
0
0
129
3w
AVFoundation: full-resolution Quad Bayer RAW capture path for third-party computational photography? (FB25352517)
At the WWDC26 Camera and Photo Technologies Group Lab (session 8018, ~29:55 to 32:46), the panel explained why ProRAW can produce 48 MP output from Quad Bayer sensors while native Bayer RAW is limited to the binned resolution. ProRAW is already demosaiced and linearized, and may incorporate Photonic Engine multi-frame processing. The native RAW path exposes the binned Bayer result. A true full-resolution RAW output would be Quad Bayer, not conventional Bayer, and supporting it properly needs a decoding path around it. The panel suggested filing Feedback Assistant requests if developers need this, so I filed FB25352517. My use case is computational photography that wants minimally processed sensor-domain samples on purpose — its own multi-frame alignment, denoising, HDR fusion, motion rejection, and super-resolution. Adobe Project Indigo is a public example of that kind of pipeline. It captures RAW bursts and uses measured sub-pixel variation between frames to reconstruct detail. Processed 48 MP photos and ProRAW don't cover that. They've already been through Apple's reconstruction or demosaic. Is filing enhancement requests still the intended path for a supported full-resolution Quad Bayer RAW or mosaic capture path? If so, what would make these reports more useful — mosaic metadata, DNG representation, CIRAWFilter support, AVCaptureDevice.Format capability discovery, something else? WWDC26 session: https://developer.apple.com/videos/play/wwdc2026/8018/ Project Indigo technical overview: https://research.adobe.com/articles/indigo/indigo.html
2
0
530
3w
Official API for Apple Log / Apple Log 2 to SDR Rec.709 and HDR HLG Conversion?
Hello, I'm developing a video application for Apple platforms and would like to implement accurate Apple Log and Apple Log 2 color conversion using official Apple APIs. Final Cut Pro and Final Cut Camera support SDR Rec.709 and HDR Rec.2020 HLG monitoring. I have found Apple's official Apple Log white papers and Rec.709 LUTs, but I could not find an official HDR HLG LUT or a public API that provides equivalent color conversion. I have three questions: Does Apple provide a public API through AVFoundation, Core Image, ColorSync, or another framework to convert Apple Log / Apple Log 2 to SDR Rec.709 and HDR Rec.2020 HLG? Are the color transformations used by Final Cut Pro and Final Cut Camera available to third-party developers, either through public APIs or official LUT resources? If no such API exists, is implementing the transfer functions and gamut conversions from Apple's white papers, followed by standard SDR/HDR rendering, the recommended approach? My goal is to preserve the original Apple Log footage while allowing independent SDR/HDR monitoring without modifying the recorded data. Any guidance, documentation, or sample code from Apple engineers would be greatly appreciated. Thank you!
1
0
505
3w
iPhone 18 Pro: AVAudioEngine input tap returns all zeros after the Always-On Display turns off overnight
My app records audio overnight in the background: UIBackgroundModes audio, AVAudioSession category playAndRecord with mode measurement, and an AVAudioEngine with a tap on inputNode. On a customer's iPhone 18 Pro running iOS 27.2.1 (24B156), the tap starts delivering buffers of exact zeros about two hours into the night. It keeps delivering zeros until she picks the phone up in the morning. Her sysdiagnose shows the same sequence on three consecutive nights: powerd logs SmartPowerNap: enterSmartPowerNap. The backlight state changes from inactiveOnLive to suppressed, with the explanation "alwaysOn suppressed - SmartPowerNap". The panel reaches 0 nits about two seconds later. Under a second after that, every sample in the recording is 0. PowerLog's per-app mic RMS reads -240 from that point on. Hours later powerd logs Exiting SmartPowerNap. Session end reason MotionDetected. The display wakes and mic input returns within a second. Timing on the third night: 23:55:57.234 Smart Power Nap entry 23:55:59.14 panel at 0 nits 23:55:59.8 first zero sample in the recording 05:46:58.032 Smart Power Nap exit about 05:46:59 mic input returns During the silent hours: The app receives no interruption, route change, media services reset or AVAudioEngineConfigurationChange notification. The engine keeps running and the tap keeps firing on schedule. audiomxd logs no deactivation for the session and holds its power assertions throughout. The system never sleeps. I can't reproduce it. My iPhone 16 Pro on the same build logs "SmartPowerNap: feature not enabled or predictor not initialized" on every lock, so it never takes this path. Questions: Does iOS gate mic input by design while Smart Power Nap suppresses the Always-On Display? Can an app keep recording through it, or get a notification that input has stopped? Does restarting AVAudioEngine while the session stays active restore input in this state? I can detect a run of exact zeros and restart the engine, but I have no device to test it on. Has anyone else with iPhone 18 Pro users seen this? I filed FB25663635 with the sysdiagnose attached.
Replies
0
Boosts
0
Views
73
Activity
1d
Camera preview rotated 90 degrees on initial presentation on iPadOS 27.2.1, corrected after returning from Control Center
Dear Apple Support Team, We are investigating a camera preview orientation issue on an iPad and would appreciate guidance on the relevant UIKit and AVFoundation behavior. Our app uses Flutter and the third-party Regula Document Reader SDK for passport MRZ scanning. We understand that third-party SDK internals are outside Apple’s support scope. Our question concerns the relationship between the active interface orientation, presentation lifecycle, and camera preview connection orientation. Environment Device: iPad (A16), hardware identifier iPad15,7 iPadOS: 27.2.1, as reported by the device and diagnostic export App distribution: TestFlight App version in the diagnostic log: 2.0.14, build 7 Flutter plugin: flutter_document_reader_api 9.8.1718 Runtime Regula SDK API version: 9.8.6944 Runtime Regula SDK Core version: 9.8.20492 Steps to reproduce Hold the affected iPad in landscape orientation and open the app. Open the passport scanner. The scanner interface appears in landscape, but the camera image appears rotated by 90 degrees. Open Control Center, then dismiss it and return to the app. The preview can then appear in the correct orientation. Close and reopen the scanner; the issue may recur. Physically rotating the device does not consistently resolve the issue. Expected: The preview should match the current interface orientation from initial presentation. Actual: The preview initially has an orientation mismatch and may recover only after leaving and returning to the app. Diagnostic observations We collected initial-entry and app-resume observations within the same scanner session: Before scanner presentation, the active UIWindowScene.interfaceOrientation is landscapeRight. The SDK’s presenting view controller belongs to the active application window. The scanner controller has landscape dimensions of approximately 1180 × 820. The third-party SDK’s supported interface orientation is configured for landscape right. However, the initial preview connection reports videoOrientation = portrait and videoRotationAngle = 90. Updating supported interface orientations and requesting layout updates did not resolve the initial mismatch in this test. A subsequent app-resume observation reports videoOrientation = landscapeRight and videoRotationAngle = 0. This change is recorded before our app’s delayed layout refresh. We recognize that interface orientation and capture rotation are different concepts, and that connection metadata alone does not establish the displayed pixel orientation. We are providing these observations alongside the visible symptom rather than treating a particular angle as universally correct. Scope and investigation status Our separate QR scanner and receipt camera work normally on the same device. The passport scanner also works normally on many other iPads. We have not established that this affects all devices running this iPadOS version. An app-side orientation enum comparison issue was corrected before collecting the build 7 log, but the preview mismatch remained. We have not isolated the cause to iPadOS, our presentation code, or the SDK’s capture initialization. We do not yet have a focused standalone Xcode reproduction project. We are also preparing an inquiry to the SDK vendor. Questions Are there known circumstances in which a camera preview initially uses portrait orientation despite being presented in a landscape UIWindowScene? What is the recommended lifecycle point for determining the initial interface orientation and configuring the preview connection, particularly while presentation or rotation transitions are in progress? Which supported APIs should be used to keep preview rotation synchronized with interface changes on this iPadOS version? Could the recovery after dismissing Control Center indicate an initialization or lifecycle ordering issue? What additional observations would help distinguish these possibilities? What should we include in a minimal native reproduction or system diagnostic collection to make this issue actionable? We can provide a symptom video and an application diagnostic export. These were collected in separate tests on the same affected device and are not time-synchronized. The application export is not an Apple sysdiagnose. Thank you for any guidance on how to investigate this further. With Best Regards, GTF_Japan_Service
Replies
0
Boosts
0
Views
38
Activity
3d
AVAudioSession interrupted with no .ended when a window changes under an open ImmersiveSpace
Title: AVAudioSession interrupted with no .ended when the window changes while an ImmersiveSpace is open Hi, I'm seeing an issue where an active voice call (mic capture + playback) goes silent as soon as the app enters its immersive experience. The system posts AVAudioSession.interruptionNotification with type .began, but .ended never arrives while the app stays in the foreground, so the audio never resumes on its own. Audio only comes back after the app goes inactive and active again (for example after leaving the immersive space). This looks like the same issue reported in https://developer.apple.com/forums/thread/761721 (2024), and it still reproduces on visionOS 26 and 27. Setup: App with a WindowGroup and an ImmersiveSpace. Voice runs on a WebRTC stack (LiveKit, AVAudioEngine-based) on top of AVAudioSession, configured as: category .playAndRecord, mode .videoChat, options [.mixWithOthers, .allowBluetoothHFP, .allowBluetoothA2DP, .allowAirPlay], then setActive(true). When the immersive space opens, an ARKitSession runs with WorldTrackingProvider, HandTrackingProvider and SharedCoordinateSpaceProvider. Steps to Reproduce: Start a voice call from the main window. Mic capture and playback both work. Open the ImmersiveSpace and replace the main window with another window (open the new window, dismiss the main one). Shortly after the space opens, interruptionNotification arrives with type .began and reason .default (0). The audio engine stops. Mic capture and playback both go silent. Expected: Audio keeps running when the window changes. If the system has to interrupt the session, .ended is posted so the app can resume. Actual: .began arrives with no matching .ended while the app remains in the foreground. Audio stays silent until the app goes inactive and active again. At that point .ended arrives with .shouldResume and the engine starts again. The same interruption sometimes hits the very first microphone start right after the session is activated, so the start times out. Nothing else in the app activates or deactivates the audio session at that point. Observation code: NotificationCenter.default.addObserver( forName: AVAudioSession.interruptionNotification, object: nil, queue: .main ) { note in let type = note.userInfo?[AVAudioSessionInterruptionTypeKey] as? UInt let reason = note.userInfo?[AVAudioSessionInterruptionReasonKey] as? UInt print("interruption type=\(String(describing: type)) reason=\(String(describing: reason))") // Logged: type=1 (.began), reason=0 (.default). No type=0 (.ended) while the app stays active. }Current workaround: On every interruptionNotification, and again on UIApplication.didBecomeActiveNotification, call setActive(true) and restart the audio engine if it isn't running. This restores audio while the immersive space stays open. Device: Apple Vision Pro visionOS version: visionOS 27.2 (24N120), also seen on visionOS 26.5 (23O471) Xcode version: Version 27.0 (27A266a) Questions: Is the app's audio session expected to be interrupted when a window closes or is replaced while an ImmersiveSpace is open? If so, why is no .ended posted while the app remains active, and is reactivating the session and restarting the engine the recommended way to resume? Thank you!
Replies
0
Boosts
0
Views
167
Activity
4d
AVSpeechSynthesizer pulls words out of thin air.
Hi, I'm working on a project that uses the AVSpeechSynthesizer and AVSpeechUtterance. I discovered by chance that the AVSpeechSynthesizer automatically completes some words instead of just outputting what it's supposed to. These are abbreviations for days of the week or months, but not all of them. I don't want either of them automatically completed, but only the specified text. The completion transcends languages. I have written a short example program for demonstration purposes. import SwiftUI import AVFoundation import Foundation let synthesizer: AVSpeechSynthesizer = AVSpeechSynthesizer() struct ContentView: View { var body: some View { VStack { Button { utter("mon") } label: { Text("mon") } .buttonStyle(.borderedProminent) Button { utter("tue") } label: { Text("tue") } .buttonStyle(.borderedProminent) Button { utter("thu") } label: { Text("thu") } .buttonStyle(.borderedProminent) Button { utter("feb") } label: { Text("feb") } .buttonStyle(.borderedProminent) Button { utter("feb", lang: "de-DE") } label: { Text("feb DE") } .buttonStyle(.borderedProminent) Button { utter("wed") } label: { Text("wed") } .buttonStyle(.borderedProminent) } .padding() } private func utter(_ text: String, lang: String = "en-US") { let utterance = AVSpeechUtterance(string: text) let voice = AVSpeechSynthesisVoice(language: lang) utterance.voice = voice synthesizer.speak(utterance) } } #Preview { ContentView() } Thank you Christian
Replies
5
Boosts
0
Views
1.9k
Activity
1w
iOS 27.2: iPhone 13 Pro / 14 Pro heat up sooner during 4K camera capture than on iOS 27.0
We make a camera app (AVCaptureSession at 4K, live preview, recording with AVAssetWriter). Since updating to iOS 27.2, a user with an iPhone 13 Pro reports that the phone gets hot while using our app. Our own iPhone 14 Pro also runs warmer on 27.2 than it did on 27.0. To check, we reran on the 14 Pro the same automated test we ran there on iOS 27.0 in September. Each run lasts 300 seconds with the phone on power and fully charged, Low Power Mode off, and starts at ProcessInfo.thermalState == .nominal. We log thermalState every 5 seconds. Capture is 3840x2160 at 60 fps in all three runs: Idle preview, no processing: iOS 27.0: stayed Nominal for all 300 s iOS 27.2 (24B115): Fair at 278 s Idle preview with a Metal overlay: iOS 27.0: stayed Nominal for all 300 s iOS 27.2: Fair at 123 s Recording, HEVC 10-bit: iOS 27.0: Fair at 38 s, Serious at 108 s iOS 27.2: Fair at 33 s, Serious at 83 s Frame delivery held up. The 5-minute recording lost no frames, even at Serious. Questions: Are there known changes in iOS 27.2 to power use or thermal management during camera capture (AVCaptureSession at 4K) on A15 / A16 devices? Have the thresholds behind ProcessInfo.thermalState or AVCaptureDevice.SystemPressureState changed in 27.2? What data would be most useful in a Feedback report for this: a sysdiagnose taken right after a run, or a specific Instruments template?
Replies
1
Boosts
0
Views
556
Activity
1w
AVAudioSession ioBufferDuration always default to 512 samples on MacOS27 no matter what is requested
Hello, I develop an audio processing demo app that has been working fine on iOS/iPad and on Mac ("developed for iPad") until I upgraded to MacOS27. The app is demonstrating low latency / high temporal resolution audio spectral analysis. I am using AVAudioSession, I request the audio IO buffer to be 32 samples. This still works fine on iOS/iPadOS, but it seems that when running on MacOS27 the buffer size always defaults to 512 samples no matter what I request (smaller or larger). This applies to the "Designed for iPad" version running on MacOS and any version running in the iPhone and iPad simulators on MacOS. The app seems to be working as intended on physical iPhones and iPads. I made a catalyst version and used AudioObjectSetPropertyData to force the IO bugger sizes to 32 samples on the system's active hardware device. This seems to succeed but still results in incorrect behaviour in my app; I am still diagnosing but I believe it's because this leaves AVAudioSession with incorrect data, and I am using AudioKit which relies on AVAudioSession (not confirmed). Regardless, is this change in behaviour intentional in MacOS27? Is there a way around it? The whole point of my app is to demonstrate efficient audio processing with very low latency - I need small input buffer size for that. Thank you for any help with this
Replies
2
Boosts
0
Views
442
Activity
1w
Can a third-party app capture microphone audio while a cellular call is active?
We're building a companion app that people use during a regular cellular call (VoIP isn't an option for this call). The user wants to briefly mute the call, speak a short question into our app, then unmute. Is there any supported way for a third-party app to record from the microphone during an active cellular call, even if the call is muted or on hold? Does Siri with App Intents accept voice input during an active call? If so, is Siri's spoken response heard by the other person on the call? Can a watchOS app record audio while the paired iPhone is on a call? If none of these work, what approach do you recommend?
Replies
1
Boosts
0
Views
165
Activity
2w
Swift 6: Supported AVCaptureSession actor isolation and MainActor preview attachment
I’m developing a SwiftUI camera preview for iPhone and iPad using Xcode 27.2 and Swift 6.4.2. I’m looking for clarification about the supported concurrency boundary between an actor-owned AVCaptureSession and a MainActor-owned AVCaptureVideoPreviewLayer. Environment Xcode 27.2 (27B52) Apple Swift 6.4.2 iPhoneOS SDK 27.2 Deployment target: iOS 26.0 Swift 6 language mode Complete concurrency checking Default actor isolation: MainActor Approachable Concurrency enabled The isolated compiler reproductions also explicitly enable NonisolatedNonsendingByDefault, InferIsolatedConformances, and MemberImportVisibility. Architecture The intended architecture follows the general pattern demonstrated in Apple’s AVCam sample: A capture actor uses a dedicated DispatchSerialQueue executor. The capture actor creates, configures, starts, and stops AVCaptureSession. A MainActor adapter owns AVCaptureVideoPreviewLayer. The preview layer retains the same session after attachment. The capture actor retains its session reference because it must perform subsequent capture operations. Compiler behavior In a minimal reproduction, transferring the retained session directly to the MainActor adapter: await preview.attach(session)produces: Sources/CaptureActor.swift:19:23: error: sending 'session' risks causing data races [#RegionIsolation::SendingRisksDataRace]The compiler identifies a potential race between actor-isolated and MainActor-isolated uses. This reproduction produces one error and zero warnings. A second reproduction uses a small PreviewSource: Sendable value containing a private AVCaptureSession reference, with @preconcurrency import AVFoundation confined to that source file. Its MainActor method attaches the session to the preview layer. Under otherwise identical compiler settings, this second reproduction compiles with zero diagnostics. We understand that compiler acceptance does not establish runtime safety and that the compatibility mechanism does not itself synchronize access to the retained session. Apple AVCam reference Apple’s official AVCam sample demonstrates a similar interoperability pattern: CaptureService.swift: dedicated capture executor and retained preview source. PreviewSource.swift: compatibility import and MainActor attachment. CameraPreview.swift: preview-layer association. Reference: Apple’s AVCam: Building a Camera App sample. Pinned sample SHA-256: 5a1981f90bb30827121324d4b33198d58de9b9d68e3508b64f0aabc18bbc0e53The sample revision date is not established; we are not assuming this archive is identical to the latest downloadable revision. Questions 1. Supported Swift 6 interoperability pattern Is the narrow PreviewSource compatibility approach demonstrated by AVCam the recommended way to associate an actor-retained AVCaptureSession with a MainActor-owned AVCaptureVideoPreviewLayer under complete Swift 6 concurrency checking? Is there a newer recommended API, annotation, or ownership pattern? 2. Capture and preview synchronization Is the following operation sequence supported? Configure and commit the capture session on the dedicated capture executor. Attach the session to the preview layer on MainActor. Call startRunning() on the capture executor. Perform ordinary preview layout on MainActor while capture runs. Call stopRunning() on the capture executor. Detach the session from the preview layer on MainActor. Which preview-layer or preview-connection mutations, if any, must be excluded while startRunning() or stopRunning() is executing? 3. Supported detachment and release sequence After stopRunning() returns and MainActor executes: previewLayer.session = nilis it supported to release the remaining application-held session references once application-owned observers and asynchronous consumers have terminated? Does AVFoundation require or recommend any additional step before those references are released? We are asking about the supported framework lifecycle contract, not assuming that application-level cancellation drains internal AVFoundation work. Sources AVCam: Building a camera app Overview Apple’s official AVCam sample
Replies
0
Boosts
1
Views
293
Activity
2w
iOS – Custom voice keyword detection when iPhone is locked
Hello, I am currently working on an iOS app and studying the feasibility of a custom voice detection feature. The user might set up a secret voice keyword (a custom word or phrase). When this keyword is spoken, even when the iPhone is locked and stored in a pocket, the application would automatically trigger an action. My questions are: Can an iOS app continuously detect a custom voice keyword when running in the background and iPhone is locked? Is there an officially supported and App Store compatible method to perform this detection locally, without recording or transmitting audio? Can an application automatically trigger an action following this detection? Would using AVAudioSession and the audio mode in the background be an acceptable solution for this use case? Are there any Apple frameworks suitable for this type of feature? I would be particularly interested in feedback from Apple engineers or developers who have experimented with similar features. Thank you in advance for your help.
Replies
0
Boosts
0
Views
100
Activity
2w
AVCaptureSession multitasking camera access on iPhone without VoIP background mode
I'm developing an iPhone screen-recording application that simultaneously records the front-facing camera and displays the live camera feed using Picture in Picture while the user navigates other apps. The implementation uses AVFoundation (AVCaptureSession) for camera capture and a system Picture in Picture controller for the floating window. I've encountered the following behavior on a physical iPhone: With voip declared in UIBackgroundModes, isMultitaskingCameraAccessSupported returns true, and the front camera continues capturing while another app is foregrounded. Without voip, isMultitaskingCameraAccessSupported returns false. When the application enters the background, the capture session is interrupted and camera frames stop. Screen recording and the Picture in Picture window continue, but the camera feed freezes or becomes blank. Apple App Review has confirmed that declaring voip without actual VoIP functionality is not permitted under Guideline 2.5.4. I understand and respect that restriction and do not want to misuse background modes. My question: Is there a supported public API, entitlement, or configuration that allows a non-VoIP iPhone application to continue capturing front-camera video while another application is in the foreground? Does Picture in Picture provide any supported mechanism for this use case, or is multitasking camera capture unavailable to non-VoIP applications on iPhone? I'm looking for a supported implementation rather than a workaround that violates App Review guidelines. Any guidance from Apple's AVFoundation engineers would be greatly appreciated.
Replies
2
Boosts
0
Views
485
Activity
2w
CallKit VoIP call on the receiver: the side (lock) button only locks the screen, no CXEndCallAction is delivered
Device and versions iPhone 15, iOS 27.0.1 App: Hush, a private messenger. Capacitor 8.5.2 iOS app; the UI is a web page in a WKWebView. Call signalling runs in the page; call audio runs in a native Swift Capacitor plugin on stasel/WebRTC 154.0.0, integrated with CallKit. Built on Codemagic with the latest Xcode, distributed through TestFlight. UIBackgroundModes: audio, voip. No PushKit yet: calls are placed and answered in the app's own screen while it is in the foreground. Setup RTCAudioSession manual audio (useManualAudio = true). isAudioEnabled is set only after ICE connects AND provider(_:didActivate:) is called (a 3-second fallback exists but never fired in these tests). We call RTCAudioSession.audioSessionDidActivate / audioSessionDidDeactivate from the provider delegate. Session: playAndRecord, mode voiceChat, options allowBluetoothHFP, no defaultToSpeaker. overrideOutputAudioPort is only called when the user picks Speaker (never in the failing tests). One CXProvider, created lazily, delegate on the main queue. Configuration: supportsVideo false (also tested true), maximumCallGroups 1, maximumCallsPerCallGroup 1, supportedHandleTypes [.generic], includesCallsInRecents false (also tested true). Outgoing: CXStartCallAction is requested via CXCallController when ICE first connects (not at dial time; the page plays its own ringback until then), handle generic, value "Hush". In perform(CXStartCallAction) we configure the audio session (no activation), reportCall(with:updated:), fulfil, then reportOutgoingCall(startedConnectingAt:) and (connectedAt:). Answering side: reportNewIncomingCall, then an immediate CXAnswerCallAction (the user already accepted in the app's screen). Mute: CXSetMutedCallAction. In-app End reported the call with reportCall(with:endedAt:reason: .remoteEnded) in all tests below (we are changing this to a CXEndCallAction request; not yet tested). Proximity: isProximityMonitoringEnabled = true from dialling while on the receiver. The screen does go dark at the ear. Also tested with our proximity monitoring off. The problem With a Hush call live, route Receiver, CallKit holding the call (the system call UI shows it), didActivate delivered exactly once, our audio running inside that activation, proximity on and the screen dark at the ear: pressing the side button only locks the screen. No CXEndCallAction arrives and the call continues. In the Phone app on the same iPhone the side button ends the call at the ear. Snapchat calls on this iPhone end on the side button while Snapchat's call screen is open. What works: End on the system lock-screen call UI delivers a CXEndCallAction and we end the call. Mute from the system UI arrives as CXSetMutedCallAction. So provider delegate delivery works. What we tried, in order (each tested on device, with a readout of the plugin's state after the call) First CallKit build: side button did nothing; lock-screen End worked. Readout: "CallKit off, its audio off (0x), fallback no, route speaker, proximity on". Cause: CXStartCallAction was refused (missing voip background mode) and the error swallowed. Fixed: voip added; on CXErrorCodeRequestTransactionError.unknownCallProvider we invalidate and recreate the provider and retry once. Next build: CallKit held the call, didActivate once, route Receiver. Side button still did nothing, no CXEndCallAction. Our route log showed two route changes caused by our own overrideOutputAudioPort(.none) on every activation and route change. Removed: no override unless Speaker is selected. Result: no change. Diagnostic switches, tested in combinations: our proximity monitoring on/off; supportsVideo false/true; includesCallsInRecents false/true; callee now reported with reportNewIncomingCall + CXAnswerCallAction (previously both sides used CXStartCallAction). Result: the side button ended the call EXACTLY ONCE: the first call after the provider was recreated (changing supportsVideo makes us invalidate() and create a new CXProvider). Every later call with the same configuration failed. Readout for the failures: "CallKit on, its audio on (1x), fallback no, route Receiver, proximity on", screen dark at the ear. Following thread 837211 (creating a provider implicitly pushes the audio session ID to callservicesd, which can keep a stale one; set the configuration before reporting each call): we now assign provider.configuration = a fresh CXProviderConfiguration at call start and again immediately before each CXStartCallAction request / reportNewIncomingCall. We also added a switch that recreates the CXProvider for every call. With the re-push alone: no end action. With a fresh provider every call: no end action either. Readout each time: "provider calls 1, config re-pushed yes, CallKit on, its audio on (1x), route Receiver, proximity on". The single success in step 3 has not been reproduced. Not tried on purpose: detecting the lock ourselves (protectedDataWillBecomeUnavailable arrives about 10 seconds late and needs a passcode; a 2017 thread says Home and Lock cannot be told apart); reporting the CallKit call at dial time instead of ICE connect; PushKit. Threads we relied on 819363 (April 2026): the lock button hangs up a CallKit call handled directly by the device (not through a headset), delivered as the provider's end action. 837211 (August 2026): stale audio session ID in callservicesd after provider creation; set the configuration before reporting a call. How we differ from apps where it works WKWebView: WebKit's GPU process holds a second AVAudioSession in our app's name. During the call the page captures no mic (the plugin owns it); before connect the page plays a Web Audio ringback (Ambient category). The CallKit call starts at ICE connect, not at dial time. In-app End reported as .remoteEnded rather than requested as a CXEndCallAction. The callee's immediate CXAnswerCallAction after reportNewIncomingCall. Questions Which conditions must hold for iOS to treat the side button as "end call" for a CallKit call, and which is our app failing? Is there a log subsystem or category (callservicesd, TelephonyUtilities) we can read in a sysdiagnose to see why the press is handled as lock? Could WebKit's GPU-process audio session in the same app confuse CallKit about which session carries the call? Would reporting the call at dial time, before any audio, change the outcome? Can ending the in-app call with reportCall(.remoteEnded) instead of a CXEndCallAction leave state in callservicesd that affects the NEXT call? That would fit "works once after a fresh provider, never again". We have no Mac. Is a sysdiagnose captured on the iPhone right after a failing press enough to diagnose this? If you need a project, we can provide the plugin's Swift source (one file) and the Capacitor app project that Codemagic builds.
Replies
1
Boosts
0
Views
156
Activity
2w
Public API for torch beam width (Adaptive True Tone flash)?
On iPhone 14 Pro and later, the system flashlight in Control Center can adjust beam width (narrow ↔ wide). As far as I can tell, AVCaptureDevice only exposes torchMode, torchLevel and setTorchModeOn(level:) I found nothing related to beam width Is there any supported way for a third-party app to control the torch beam width? If not, is this on the roadmap?
Replies
0
Boosts
0
Views
338
Activity
2w
Improving metric scale consistency of ARWorldTrackingConfiguration on devices without LiDAR
We build a medical scanning app. A practitioner walks a slow orbit (~1 m radius, two laps, 25–35 s) around a standing subject while an ARSession runs, and we capture ~50 high-resolution stills with captureHighResolutionFrame. The ARKit camera poses for those frames are later used to set the metric scale of a photogrammetry reconstruction. We need the resulting scale to be within about 2 % of true. What we observe, measured against a printed A4 sheet on the floor in every scan: Devices with LiDAR (iPhone 16 Pro Max, iPad Pro), using the sceneDepth frame semantic as a check: scale within ±0.8 % across 11 sessions. Devices without LiDAR (iPhone 13, iPad 9th and 11th gen, plus non-LiDAR runs on the Pro devices): the world-frame metre is on average 2–3 % short of true, and varies by a further ±2 % from one session to the next, across 24 sessions in four rooms. Within a session the scale is stable: lap-to-lap variation is under 0.4 %. The error appears to be set during initialisation and held. Nothing we record about capture correlates with the error: time spent before capture, distance walked, vertical camera movement, orbit radius, rotation speed, tracking state, or the residual of our pose fit. Questions: Is there any session setup, initialisation practice, or motion during the first seconds of tracking that is known to improve the metric scale estimate on devices without LiDAR? Does the choice of video format, resolution or frame rate affect scale estimation? Is there any signal, beyond ARCamera.trackingState, that indicates how well scale has been determined for a session, so we could flag sessions rather than accept them? Does anything in the configuration above, in particular locking exposure on the capture device or using the high-resolution video format, work against tracking quality or scale?
Replies
1
Boosts
0
Views
912
Activity
2w
Is activeLockedVideoFrameDuration supported on iPhone 16 Pro? Hardware or iOS limitation?
Hello Apple AVFoundation team, Could you clarify the supported iPhone models and OS versions for standalone locked video frame duration using AVCaptureDeviceInput.activeLockedVideoFrameDuration? The API is documented as available from iOS 26.0, but I understand that OS availability alone does not mean every camera device/configuration supports it. I have read the API documentation and the DTS reply in iPhone 17 Pro's Genlock feature, but could not find a model compatibility list for the standalone locked-duration API. Scope: I am investigating actual capture timing for multi-iPhone motion capture. This question is about locking each camera's frame period using its internal timing, without an external genlock accessory. I am not asking about encoding a constant-frame-rate file, ordinary equal min/max frame durations, timecode metadata, or synchronizing the first exposure across phones. Physical-device test iPhone 16 Pro, iOS 26.2 (23C55). Built with Xcode 26.3 and the iOS 26.2 SDK. One AVCaptureSession, one physical rear .builtInWideAngleCamera input, and one AVCaptureVideoDataOutput. 8-bit SDR 420v formats, tested at 1920 x 1080 and 3840 x 2160, each at 30 and 60 fps. Auto video frame rate disabled; no spatial video capture or multicam session. Manual exposure approximately 2 ms, shorter than half a frame at both tested rates. The session is running and delivering frames. Two separate probe runs produced the same results, including capability readbacks after a settling period. For all four tested configurations: input.isLockedVideoFrameDurationSupported = false device.minSupportedLockedVideoFrameDuration = kCMTimeInvalid input.activeLockedVideoFrameDuration = kCMTimeInvalid device.isVideoFrameDurationLocked = falseOrdinary activeVideoMinFrameDuration and activeVideoMaxFrameDuration can both be set to the requested CMTime(value: 1, timescale: 30) or CMTime(value: 1, timescale: 60), and frames are delivered. This is only a configuration readback, not a claim of exact capture timing in the ordinary mode. The app directly queries the public support properties; it has no model whitelist. It does not attempt to set a valid locked duration when support is false, as the documented setter contract would reject that. These tests cover the selected formats, not every camera/format on this phone. Questions Which iPhone models support this standalone API? In particular, is false expected on iPhone 16 Pro? Is there an official compatibility list covering iPhone 13/14/15/16 and iPhone 17 models? Is the iPhone 16 Pro result a hardware capability restriction, an iOS 26.2 limitation/known issue, or a configuration requirement? Is there a released iOS version that enables this on iPhone 16 Pro, or is newer hardware required? What is the minimum supported capture configuration and correct time to query support? Are there restrictions involving the physical versus virtual camera input, pixel format, SDR/HDR, or exposure mode that could explain this result? If so, which configuration should be tested before concluding that the device does not support it? On supported hardware, can this standalone lock be enabled without an external accessory or a special entitlement, independently of follow(_:videoFrameDuration:delegate:)? Relevant documentation: activeLockedVideoFrameDuration isLockedVideoFrameDurationSupported minSupportedLockedVideoFrameDuration I can provide a focused reproducer if needed. Thank you for clarifying the device requirements and whether an OS upgrade can change the result.
Replies
1
Boosts
0
Views
655
Activity
3w
iPhone 17 Pro: Is Native Apple Log 2 ISP Processing Available for 4224×3024/60fps RAW?
Hi everyone, I'm developing an iOS camera app that aims to record 4224×3024 Open Gate video at 60fps in 10-bit HEVC with Apple Log 2. Native Apple ISP-processed Apple Log 2 capture works at 3840×2160/60fps. However, at 4224×3024/60fps, the tested capture configuration only provides Bayer RAW (btp2). I've successfully decoded ProRes RAW using Apple's decoder into linear Rec.2020 and applied the published Apple Gamut and Apple Log 2 transforms. However, the resulting chroma is only approximately 72–74% of the native ISP output in controlled color-chart comparisons. I've also confirmed that VTRAWProcessingSession and related RAW processing controls are not publicly available to iOS apps in the iPhoneOS 27.0 SDK. My questions are: Is there a public iOS API that applies the same color and tone rendering as Apple's native camera ISP to captured Bayer RAW or ProRes RAW? Is there a supported configuration for obtaining full-resolution 4224×3024/60fps ISP-processed Apple Log 2 output (x420 or x422), without preview proxies or upscaling? If neither is supported, what is Apple's recommended approach? Is this a hardware limitation or a public API limitation? The goal is to use Apple's official color processing pipeline without empirically fitted matrices, LUTs, or arbitrary color corrections. Any guidance would be greatly appreciated. Thank you!
Replies
0
Boosts
0
Views
357
Activity
3w
Radiometric interpretation of Apple ProRAW and Bayer RAW access via AVFoundation
I am working on a computational photography research project involving multi-exposure HDR reconstruction using Bayer RAW and Apple ProRAW captures. I would like to clarify the radiometric interpretation of Apple ProRAW and the availability of Bayer RAW capture through AVFoundation. My questions are: On current iPhone Pro devices, is it possible for third-party apps to capture and export true Bayer-pattern RAW DNG files through AVFoundation, rather than Apple ProRAW linear DNG files? If so, which availableRawPhotoPixelFormatTypes correspond to Bayer RAW, and what device or format restrictions apply? Apple ProRAW appears to be demosaiced and computationally processed, and may include multi-frame fusion. Is the decoded ProRAW image intended to be radiometrically linear and scene-referred? For a bracketed ProRAW sequence captured with fixed ISO, white balance, lens, and focus, but different exposure times, can one assume that the decoded linear pixel values Y_i(p) satisfy an exposure-proportional model in non-saturated regions, such as Y_i(p) ≈ t_i R(p), across brackets? This question is about radiometric consistency for algorithmic use, not about visual editing or tone mapping. Thank you for your help.
Replies
2
Boosts
0
Views
2.1k
Activity
3w
Radiometric interpretation of Apple ProRAW and Bayer RAW access via AVFoundation
I am working on a computational photography research project involving multi-exposure HDR reconstruction using Bayer RAW and Apple ProRAW captures. I would like to clarify the radiometric interpretation of Apple ProRAW and the availability of Bayer RAW capture through AVFoundation. My questions are: 1.On current iPhone Pro devices, is it possible for third-party apps to capture and export true Bayer-pattern RAW DNG files through AVFoundation, rather than Apple ProRAW linear DNG files? If so, which availableRawPhotoPixelFormatTypes correspond to Bayer RAW, and what device or format restrictions apply? 2.Apple ProRAW appears to be demosaiced and computationally processed, and may include multi-frame fusion. Is the decoded ProRAW image intended to be radiometrically linear and scene-referred? 3.For a bracketed ProRAW sequence captured with fixed ISO, white balance, lens, and focus, but different exposure times, can one assume that the decoded linear pixel values Y_i(p) satisfy an exposure-proportional model in non-saturated regions, such as Y_i(p) ≈ t_i R(p), across brackets? This question is about radiometric consistency for algorithmic use, not about visual editing or tone mapping. Thank you for your help.
Replies
2
Boosts
0
Views
1.7k
Activity
3w
TTS Stuttering & Noise
TTS Stuttering & Noise When using Apple’s native TTS API on an iPhone or iPad—specifically while connected to Bluetooth earphones or headsets—users are experiencing audio noise or intermittent gaps in speech, causing parts of sentences to drop out during playback. There were no reports or inquiries regarding such issues up through iOS/iPadOS 26. Since the release of iOS/iPadOS 27, however, there has been a significant increase in reports and inquiries about this problem. If you happen to know the cause of this issue or have any suggestions for a fix, please leave a reply. The issue persists even after removing the AirPods or headset from the device and reconnecting them. The issue persists even after rebooting the device. The issue occurs even on the latest versions, from iOS/iPadOS 27 through 27.2.
Replies
0
Boosts
0
Views
129
Activity
3w
AVFoundation: full-resolution Quad Bayer RAW capture path for third-party computational photography? (FB25352517)
At the WWDC26 Camera and Photo Technologies Group Lab (session 8018, ~29:55 to 32:46), the panel explained why ProRAW can produce 48 MP output from Quad Bayer sensors while native Bayer RAW is limited to the binned resolution. ProRAW is already demosaiced and linearized, and may incorporate Photonic Engine multi-frame processing. The native RAW path exposes the binned Bayer result. A true full-resolution RAW output would be Quad Bayer, not conventional Bayer, and supporting it properly needs a decoding path around it. The panel suggested filing Feedback Assistant requests if developers need this, so I filed FB25352517. My use case is computational photography that wants minimally processed sensor-domain samples on purpose — its own multi-frame alignment, denoising, HDR fusion, motion rejection, and super-resolution. Adobe Project Indigo is a public example of that kind of pipeline. It captures RAW bursts and uses measured sub-pixel variation between frames to reconstruct detail. Processed 48 MP photos and ProRAW don't cover that. They've already been through Apple's reconstruction or demosaic. Is filing enhancement requests still the intended path for a supported full-resolution Quad Bayer RAW or mosaic capture path? If so, what would make these reports more useful — mosaic metadata, DNG representation, CIRAWFilter support, AVCaptureDevice.Format capability discovery, something else? WWDC26 session: https://developer.apple.com/videos/play/wwdc2026/8018/ Project Indigo technical overview: https://research.adobe.com/articles/indigo/indigo.html
Replies
2
Boosts
0
Views
530
Activity
3w
Official API for Apple Log / Apple Log 2 to SDR Rec.709 and HDR HLG Conversion?
Hello, I'm developing a video application for Apple platforms and would like to implement accurate Apple Log and Apple Log 2 color conversion using official Apple APIs. Final Cut Pro and Final Cut Camera support SDR Rec.709 and HDR Rec.2020 HLG monitoring. I have found Apple's official Apple Log white papers and Rec.709 LUTs, but I could not find an official HDR HLG LUT or a public API that provides equivalent color conversion. I have three questions: Does Apple provide a public API through AVFoundation, Core Image, ColorSync, or another framework to convert Apple Log / Apple Log 2 to SDR Rec.709 and HDR Rec.2020 HLG? Are the color transformations used by Final Cut Pro and Final Cut Camera available to third-party developers, either through public APIs or official LUT resources? If no such API exists, is implementing the transfer functions and gamut conversions from Apple's white papers, followed by standard SDR/HDR rendering, the recommended approach? My goal is to preserve the original Apple Log footage while allowing independent SDR/HDR monitoring without modifying the recorded data. Any guidance, documentation, or sample code from Apple engineers would be greatly appreciated. Thank you!
Replies
1
Boosts
0
Views
505
Activity
3w