Foundation Models

RSS for tag

Discuss the Foundation Models framework which provides access to Apple’s on-device large language model that powers Apple Intelligence to help you perform intelligent tasks specific to your app.

Foundation Models Documentation

Posts under Foundation Models subtopic

Post

Replies

Boosts

Views

Activity

Questions About Apple Foundation Models, Context Window Limits, and the New Core AI Framework
After reviewing the WWDC sessions on Foundation Models and Core AI, I had a few questions around the practical limits and architectural direction of the platform. From my understanding, on-device Foundation Models remain optimized for privacy, latency, and efficiency, which naturally introduces constraints around context length and agent complexity. Has anything changed regarding the effective context window available to developers, or should we still design around similar context-management constraints as before? Core AI appears to introduce a more structured approach to building AI-powered applications. For developers building sophisticated assistants, how should we think about the boundary between application-level orchestration and framework-level orchestration? For example, are advanced patterns such as sub-agents, hierarchical planning, dynamic tool availability, and workflow decomposition expected to remain developer-managed, or are these areas Core AI aims to support more directly over time? I am also curious about Apple's vision for model interoperability. While Foundation Models provide an excellent on-device experience, many production-grade agent systems combine multiple specialized models for planning, reasoning, retrieval, and execution. Does Apple envision future pathways for integrating external models into Core AI driven workflows while maintaining the privacy and performance principles of the platform? Finally, for teams pushing the limits of on-device AI assistants, what architectural patterns do you recommend for handling long-horizon tasks, large context requirements, evolving toolsets, and multi-step reasoning within the current Foundation Models ecosystem?
0
0
339
Jun ’26
Questions About Apple Foundation Models, Context Window Limits and the New Core AI Framework
After reviewing the WWDC sessions on Foundation Models and Core AI, I had a few questions around the practical limits and architectural direction of the platform. From my understanding, on-device Foundation Models remain optimized for privacy, latency, and efficiency, which naturally introduces constraints around context length and agent complexity. Has anything changed regarding the effective context window available to developers, or should we still design around similar context-management constraints as before? Core AI appears to introduce a more structured approach to building AI-powered applications. For developers building sophisticated assistants, how should we think about the boundary between application-level orchestration and framework-level orchestration? For example, are advanced patterns such as sub-agents, hierarchical planning, dynamic tool availability, and workflow decomposition expected to remain developer-managed, or are these areas Core AI aims to support more directly over time? I am also curious about Apple's vision for model interoperability. While Foundation Models provide an excellent on-device experience, many production-grade agent systems combine multiple specialized models for planning, reasoning, retrieval, and execution. Does Apple envision future pathways for integrating external models into Core AI driven workflows while maintaining the privacy and performance principles of the platform? Finally, for teams pushing the limits of on-device AI assistants, what architectural patterns do you recommend for handling long-horizon tasks, large context requirements, evolving toolsets, and multi-step reasoning within the current Foundation Models ecosystem?
0
0
267
Jun ’26
Disambiguation when multiple entities match
When a spoken phrase could match several entities in our catalog — same region, similar names, or partial matches — who is responsible for disambiguation: Siri via App Schemas and entity resolution, or the app via EntityStringQuery returning multiple candidates? What’s the recommended UX pattern for ‘Did you mean A or B?’
5
0
371
Jun ’26
On Protocol Extensibility & Multi-Modal Data
The Foundation Models framework is adding built-in OCR and barcode reader tools this year . If we implement a custom backend using the Language Model Protocol, can we return complex multi-modal objects (like bounding boxes or segmentation masks) back to the agentic flow, or is the protocol currently limited to text-based responses? For the 'Phone a Friend' pattern, is there a standard way to pass 'privacy-preserving embeddings' instead of raw text when calling a third-party model to maintain a higher level of user data protection?
1
0
162
Jun ’26
On Agentic Testing & Accessibility
Since agents in Xcode 27 can now interact with the accessibility tree and screenshots, can we provide 'developer hints' in our code to help the agent distinguish between decorative UI and critical interactive elements during automated testing? Can the Evaluations framework be used to 'score' the efficiency of an agent’s navigation path through the app, helping us identify where our App Intents might be creating confusing or redundant loops for Apple Intelligence?
0
0
233
Jun ’26
In-app text input vs system speech paths
If users dictate into a standard TextField via the keyboard mic instead of a dedicated in-app record button, does that text still benefit from App Intents entity resolution and indexed entities — or is keyboard dictation a separate pipeline where we lose domain vocabulary unless the user invokes Siri directly?
0
1
138
Jun ’26
On Performance & Backgrounding
While we now know about the continued-processing.gpu entitlement for background tasks, is there a similar NPU-specific entitlement or priority flag to ensure that an on-device foundation model isn't preempted by system-level Apple Intelligence features while the app is in the background?
1
0
259
Jun ’26
On-device model capabilities, limits, and versioning
What is the context window of the on-device model (AFM 3 Core Advanced and the 3B Core), and how should developers handle prompts that exceed it — automatic truncation, error, or developer-managed chunking? For guided/structured generation into typed Swift values, what are the limits on schema complexity (nesting depth, enums, arrays, optionals), and what is the failure mode when the model cannot satisfy the schema? How deterministic and reliable is on-device tool calling under the Tool protocol — are there guarantees on argument validity, and a recommended pattern for validating/repairing tool arguments before execution? For the new image input: what are the constraints on resolution, image count per prompt, and formats, and does passing images change which device tiers or which model (on-device vs PCC) services the request? Since the on-device model ships and updates with the OS, how should developers detect the active model version at runtime and guard against behavioral drift between OS releases? Is there a pinning or capability-query API? What are the realistic latency and concurrency expectations on supported hardware, and is there a supported way to run multiple sessions or background inference without thermal/throttling penalties?
2
0
301
Jun ’26
Creating an in-universe AI computer in my app
Last year after Apple foundation models framework was introduced, I begin working on a separate test Playground project to see how to use the foundation model framework to create an AI computer in my app that only has knowledge of in universe content that comes from within my app. Now with the OS 27 updates released I’m going back to work on that. I believe I can use the on-device system foundation model framework comfortably because I don’t think there’s a lot of content in my app that the AI has to know about Do you have any advice for using instructions to tell the model to focus on only the knowledge boundaries from within my app universe or might there be new tools this year in using foundation models framework that might help me achieve the limited knowledge scope that I want the AI to recognize and respond to for my app users.
1
0
174
Jun ’26
Guidance Around PCC
If a developer is eligible for Private Cloud Compute and then crosses the threshold, what happens to PCC calls? Is there a paid program for PCC that you fall back on or does a developer need to already have built into their app another model ready in the wings to take over once that threshold is reached?
4
1
261
Jun ’26
Questions About Apple Foundation Models, Context Window Limits, and the New Core AI Framework
After reviewing the WWDC sessions on Foundation Models and Core AI, I had a few questions around the practical limits and architectural direction of the platform. From my understanding, on-device Foundation Models remain optimized for privacy, latency, and efficiency, which naturally introduces constraints around context length and agent complexity. Has anything changed regarding the effective context window available to developers, or should we still design around similar context-management constraints as before? Core AI appears to introduce a more structured approach to building AI-powered applications. For developers building sophisticated assistants, how should we think about the boundary between application-level orchestration and framework-level orchestration? For example, are advanced patterns such as sub-agents, hierarchical planning, dynamic tool availability, and workflow decomposition expected to remain developer-managed, or are these areas Core AI aims to support more directly over time? I am also curious about Apple's vision for model interoperability. While Foundation Models provide an excellent on-device experience, many production-grade agent systems combine multiple specialized models for planning, reasoning, retrieval, and execution. Does Apple envision future pathways for integrating external models into Core AI driven workflows while maintaining the privacy and performance principles of the platform? Finally, for teams pushing the limits of on-device AI assistants, what architectural patterns do you recommend for handling long-horizon tasks, large context requirements, evolving toolsets, and multi-step reasoning within the current Foundation Models ecosystem?
Replies
0
Boosts
0
Views
339
Activity
Jun ’26
Questions About Apple Foundation Models, Context Window Limits and the New Core AI Framework
After reviewing the WWDC sessions on Foundation Models and Core AI, I had a few questions around the practical limits and architectural direction of the platform. From my understanding, on-device Foundation Models remain optimized for privacy, latency, and efficiency, which naturally introduces constraints around context length and agent complexity. Has anything changed regarding the effective context window available to developers, or should we still design around similar context-management constraints as before? Core AI appears to introduce a more structured approach to building AI-powered applications. For developers building sophisticated assistants, how should we think about the boundary between application-level orchestration and framework-level orchestration? For example, are advanced patterns such as sub-agents, hierarchical planning, dynamic tool availability, and workflow decomposition expected to remain developer-managed, or are these areas Core AI aims to support more directly over time? I am also curious about Apple's vision for model interoperability. While Foundation Models provide an excellent on-device experience, many production-grade agent systems combine multiple specialized models for planning, reasoning, retrieval, and execution. Does Apple envision future pathways for integrating external models into Core AI driven workflows while maintaining the privacy and performance principles of the platform? Finally, for teams pushing the limits of on-device AI assistants, what architectural patterns do you recommend for handling long-horizon tasks, large context requirements, evolving toolsets, and multi-step reasoning within the current Foundation Models ecosystem?
Replies
0
Boosts
0
Views
267
Activity
Jun ’26
Disambiguation when multiple entities match
When a spoken phrase could match several entities in our catalog — same region, similar names, or partial matches — who is responsible for disambiguation: Siri via App Schemas and entity resolution, or the app via EntityStringQuery returning multiple candidates? What’s the recommended UX pattern for ‘Did you mean A or B?’
Replies
5
Boosts
0
Views
371
Activity
Jun ’26
On Protocol Extensibility & Multi-Modal Data
The Foundation Models framework is adding built-in OCR and barcode reader tools this year . If we implement a custom backend using the Language Model Protocol, can we return complex multi-modal objects (like bounding boxes or segmentation masks) back to the agentic flow, or is the protocol currently limited to text-based responses? For the 'Phone a Friend' pattern, is there a standard way to pass 'privacy-preserving embeddings' instead of raw text when calling a third-party model to maintain a higher level of user data protection?
Replies
1
Boosts
0
Views
162
Activity
Jun ’26
On Agentic Testing & Accessibility
Since agents in Xcode 27 can now interact with the accessibility tree and screenshots, can we provide 'developer hints' in our code to help the agent distinguish between decorative UI and critical interactive elements during automated testing? Can the Evaluations framework be used to 'score' the efficiency of an agent’s navigation path through the app, helping us identify where our App Intents might be creating confusing or redundant loops for Apple Intelligence?
Replies
0
Boosts
0
Views
233
Activity
Jun ’26
In-app text input vs system speech paths
If users dictate into a standard TextField via the keyboard mic instead of a dedicated in-app record button, does that text still benefit from App Intents entity resolution and indexed entities — or is keyboard dictation a separate pipeline where we lose domain vocabulary unless the user invokes Siri directly?
Replies
0
Boosts
1
Views
138
Activity
Jun ’26
On Performance & Backgrounding
While we now know about the continued-processing.gpu entitlement for background tasks, is there a similar NPU-specific entitlement or priority flag to ensure that an on-device foundation model isn't preempted by system-level Apple Intelligence features while the app is in the background?
Replies
1
Boosts
0
Views
259
Activity
Jun ’26
On-device model capabilities, limits, and versioning
What is the context window of the on-device model (AFM 3 Core Advanced and the 3B Core), and how should developers handle prompts that exceed it — automatic truncation, error, or developer-managed chunking? For guided/structured generation into typed Swift values, what are the limits on schema complexity (nesting depth, enums, arrays, optionals), and what is the failure mode when the model cannot satisfy the schema? How deterministic and reliable is on-device tool calling under the Tool protocol — are there guarantees on argument validity, and a recommended pattern for validating/repairing tool arguments before execution? For the new image input: what are the constraints on resolution, image count per prompt, and formats, and does passing images change which device tiers or which model (on-device vs PCC) services the request? Since the on-device model ships and updates with the OS, how should developers detect the active model version at runtime and guard against behavioral drift between OS releases? Is there a pinning or capability-query API? What are the realistic latency and concurrency expectations on supported hardware, and is there a supported way to run multiple sessions or background inference without thermal/throttling penalties?
Replies
2
Boosts
0
Views
301
Activity
Jun ’26
Mixed languages and foreign proper nouns
If the user’s device language is French but they speak English, or they use one language for the sentence and another for proper nouns, how does Siri handle transcription and entity resolution? Do we need per-locale entity indexing, aliases, or can semantic indexing work across languages?
Replies
0
Boosts
0
Views
135
Activity
Jun ’26
Creating an in-universe AI computer in my app
Last year after Apple foundation models framework was introduced, I begin working on a separate test Playground project to see how to use the foundation model framework to create an AI computer in my app that only has knowledge of in universe content that comes from within my app. Now with the OS 27 updates released I’m going back to work on that. I believe I can use the on-device system foundation model framework comfortably because I don’t think there’s a lot of content in my app that the AI has to know about Do you have any advice for using instructions to tell the model to focus on only the knowledge boundaries from within my app universe or might there be new tools this year in using foundation models framework that might help me achieve the limited knowledge scope that I want the AI to recognize and respond to for my app users.
Replies
1
Boosts
0
Views
174
Activity
Jun ’26
Guidance Around PCC
If a developer is eligible for Private Cloud Compute and then crosses the threshold, what happens to PCC calls? Is there a paid program for PCC that you fall back on or does a developer need to already have built into their app another model ready in the wings to take over once that threshold is reached?
Replies
4
Boosts
1
Views
261
Activity
Jun ’26