Study Guide

The Foundry Models catalog holds thousands of models, filterable by capability (reasoning, tool calling, multimodal), provider, inference task, and industry. Large language models such as GPT-5 and Mistral Large handle deep reasoning, complex generation, and long context, at a higher compute cost. Small language models such as Phi-4 handle common language tasks efficiently and can run on lower-end hardware or edge devices, so they win when cost and speed matter more than hard reasoning.

Reasoning is its own axis. GPT-5 is a reasoning model with adjustable effort (minimal, low, medium, high) that suits planning, multistep analysis, and agentic tool calling, but its time to first token is higher. GPT-4.1 is a non-reasoning model tuned for low latency and high throughput, so it fits live chat and short factual answers. When you can't decide per request, model router is deployed like a single model and picks an underlying model for each prompt in Balanced, Quality, or Cost mode.

Specialized models cover other modalities: embedding models (Ada, Cohere) for semantic search and RAG, image generation (GPT-image-1), video generation (Sora 2), text to speech (GPT-4o-tts), and speech to text (GPT-4o-transcribe). Image analysis models such as GPT-4.1 accept images alongside text.

Not every task needs a generative model. Foundry Tools (formerly Azure AI Services, and before that Cognitive Services) are prebuilt APIs for language, speech, translation, document field extraction, and content understanding. They are often cheaper and more predictable than prompting an LLM for the same result.

Foundry Agent Service has two agent types. A prompt agent is defined entirely by instructions, a model, and tools, and Foundry runs it with no code or containers for you to maintain. A hosted agent is your own code or framework, such as Microsoft Agent Framework or LangGraph, shipped as a container or source .zip that Foundry runs with a managed endpoint, autoscaling, and a dedicated Microsoft Entra identity. Hosted agents add container compute to the bill. If you already run agent code elsewhere, you can call the Responses API directly without creating an agent resource.

When a task spans several agents or stages, a workflow orchestrates them, with branching, variables, and human-in-the-loop steps. The capability map is explicit that if a single agent with tools can finish the job, you should stay with the simpler design. Foundry workflows are in preview and retire on December 1, 2026; Microsoft recommends Microsoft Agent Framework for new orchestration.

For grounding, Azure AI Search is the recommended index store for RAG and supports keyword, semantic, vector, and hybrid retrieval. Foundry IQ builds on it to provide agentic retrieval over a knowledge base that several agents can share. The file search tool is the lighter option when an agent just needs to search a few uploaded files.

For multimodal processing, Azure Content Understanding in Foundry Tools extracts data from documents, images, video, and audio. Azure Document Intelligence focuses on fields from invoices, receipts, and forms.

RAG retrieves relevant content from your data, adds it to the prompt as grounding, and generates a response that can cite its sources. Choose RAG when answers depend on private or frequently changing information. Choose fine-tuning when the goal is to change the model's behavior, style, or performance on a task rather than to supply new facts.

Classic RAG sends a single query. Agentic retrieval uses a model to plan: it reads the conversation history, splits a complex question into focused subqueries, runs them in parallel, applies semantic ranking, and returns structured grounding data with citations. Follow-up questions keep their context because query planning uses the chat history.

Foundry IQ knowledge sources are chosen by where data lives. SharePoint (Remote) queries SharePoint live, needs no index, and respects SharePoint permissions. SharePoint (Indexed) preprocesses content into Azure AI Search for faster responses and custom pipelines. Blob Storage suits files in Azure, OneLake suits Fabric lakehouse content, an existing Azure AI Search index reuses prior investment, and Web grounds on current public information through Bing.

The file search tool manages its own vector store. Defaults are 800-token chunks with 400-token overlap, text-embedding-3-large embeddings at 256 dimensions, and up to 20 chunks added to context. An agent and a conversation can each have at most one vector store attached.

Memory (preview) in Foundry Agent Service is managed long-term memory. It extracts key facts from conversations, consolidates duplicates and conflicts, and retrieves the relevant items later. It stores three types: user profile memory (durable preferences, retrieved near the start of a conversation), chat summary memory (summaries of prior topics, retrieved per turn), and procedural memory (reusable routines inferred from earlier tasks). Microsoft's guidance separates the three knowledge options clearly: memory for learned user context, a Foundry IQ knowledge base for curated organizational content, and file search for documents a user provides during an interaction.

A toolbox curates tools once, including MCP servers, web search, Azure AI Search, code interpreter, file search, OpenAPI, and A2A, and exposes them through a single MCP-compatible endpoint. Authentication, guardrails, observability, and versioning are managed at the toolbox level, so updating or promoting a toolbox version reaches every consuming agent without code changes.

Large toolboxes use tool search (preview), which hides tools by default and exposes only tool_search and call_tool. That keeps tool definitions from flooding the context window, which would raise token cost and hurt tool selection.

Some tools can't live in a toolbox. Function calling runs on the client, so it attaches directly to the agent, as do Grounding with Bing, SharePoint, computer use, and image generation.

A Foundry resource (Microsoft.CognitiveServices/accounts, kind AIServices) is where you manage networking, security, and model deployments. Projects are child resources that give teams an isolated place to build agents, run evaluations, and store files while reusing the resource's deployments and connections. Connected services such as Storage, Key Vault, and Azure AI Search are separate Azure resources with their own governance.

Agent Service offers basic setup, which stores conversations, files, and vector stores in Microsoft-managed storage, and standard setup, which uses your own resources: Azure Storage for files, Azure AI Search for vector stores, and Azure Cosmos DB for messages, conversation history, and agent metadata. Standard setup is required for customer-managed keys, and standard setup with private networking adds bring-your-own virtual network isolation.

For topology, Microsoft recommends isolation by default in production: one Foundry resource per production workload boundary, which gives independent access control, quota, and cost. Colocating many projects in one resource suits experimentation, at the price of a shared blast radius.

Not every API honors project boundaries. Agent capabilities support project-scoped roles, but some Foundry Tools APIs need access at the parent resource scope, and Translator isn't organized by project at all. If such a workload needs isolation, give it a separate Foundry resource.

Deployment types combine where inferencing is processed with how you pay. Global types can process in any Azure region, Data Zone types stay within the US, EU, or APAC data zone, and Standard or Regional Provisioned types stay within an Azure geography. Standard types bill per token, Provisioned types bill per provisioned throughput unit (PTU) per hour, and Batch types bill discounted tokens. Data at rest stays in the Azure geography regardless of type.

Start with Global Standard (GlobalStandard): new models arrive there first, and it has the lowest price, the broadest region coverage, and the highest default quota. Global Batch and Data Zone Batch cost 50% less than Global Standard, target a 24-hour turnaround, and draw on separate quota so they never slow online traffic. The Developer type exists only for evaluating fine-tuned models, with a 24-hour lifetime and no SLA.

Provisioned throughput reserves model processing capacity for your deployment, which gives predictable throughput and consistent latency for high-volume, latency-sensitive production workloads. PTU quota and actual capacity are separate: quota lets you deploy, but capacity must be available in the region. Azure reservations discount the PTU meter for one month or one year but also don't guarantee capacity, so create deployments before buying.

Serverless API is the preferred deployment option for Foundry Models. Managed compute (preview) runs open-source, partner, and custom-weight models on dedicated GPUs, bills per accelerator hour, can scale to zero, and has no content filtering during the preview.

When you create a deployment, you give it a name, a model, a version, a SKU (the deployment type), and a capacity. Requests pass the deployment name in the model parameter, not the model's catalog name, so one model can be deployed several times under different names, for example to test different content filters. With the Azure CLI, az cognitiveservices account deployment create sets the deployment type through --sku-name.

Model versions change over time. The versionUpgradeOption property has three values: OnceNewDefaultVersionAvailable upgrades within two weeks of a new default version, OnceCurrentVersionExpired upgrades only when the current version retires (a null value behaves the same), and NoAutoUpgrade never upgrades, so the deployment stops working at retirement. Automatic updates apply only to Standard deployment types; provisioned deployments migrate in place or by moving traffic to a new deployment.

Prompt agents are versioned. Each saved version is immutable, and code refers to a specific one as <agent_name>:<version>, which lets you roll out and roll back deliberately. Unsaved changes can be tried in the playground, but history, monitoring, and full evaluations need a saved version. An agent's name can't be changed after creation.

Infrastructure as code makes environments repeatable. Microsoft publishes Bicep and Terraform templates for Foundry, and a hand-configured resource can be exported from the Azure portal as Bicep, after which you replace hardcoded subscription and resource IDs with parameters.

For hosted agents, azd pipeline config detects GitHub or Azure DevOps, creates a service principal, stores environment values as secrets and variables, and generates a workflow that runs azd provision and azd deploy on pushes to main. Runner images don't include the Foundry extensions, so the pipeline must run azd ext install microsoft.foundry first.

Unattended jobs need pipeline-friendly flags. --no-prompt makes commands fail fast instead of hanging on input, and --output json produces parseable results. After deploying, azd ai agent eval run can fail the job when scores drop.

The AI agent evaluation GitHub Action (and the equivalent Azure DevOps extension, both in preview) runs a dataset of queries against one or more agent versions with the evaluators you choose, then reports scores, confidence intervals, and whether differences between versions are statistically significant. Running it on every commit is discouraged because of cost.

Standard quota is measured in tokens per minute (TPM) per subscription, per region, per model, and per deployment type. You divide it among deployments however you like, and assigning TPM also sets a proportional requests-per-minute limit. When quota runs out, free TPM from another deployment of the same model in that region or request an increase.

Rate limiting happens when a request arrives, based on an estimate that includes the prompt and max_tokens, so 429 responses can appear while token usage metrics look low. Requests are also checked over short 1-second or 10-second windows, so bursts get throttled even if the per-minute total is fine. Set max_tokens close to the expected response size, spread traffic evenly, and retry with exponential backoff and jitter while honoring retry-after-ms. The openai SDK retries twice by default; raise max_retries on the client, or set it to 0 if you use your own retry library.

Provisioned deployments throttle at 100% utilization. Spillover sends overflow requests to a standard deployment in the same resource, either for all requests (the spilloverDeploymentName property) or per request (the x-ms-spillover-deployment header). Spilled requests are billed at standard token rates.

To stop one project from consuming shared capacity, Foundry Control Plane can apply AI Gateway token limits per project: a TPM limit returns 429, and a total token quota for an hour, day, week, month, or year returns 403. Track spend with Cost Management, which remains the source of truth for billing.

Model deployments emit Azure Monitor metrics with no configuration: availability, request counts, latency (Time To Response is the recommended measure for streaming), input and output tokens, and for provisioned deployments Provisioned Utilization, which triggers 429s at 100%. Resource logs such as RequestResponse and Audit need a diagnostic setting before they're collected.

The Agent Monitoring Dashboard, on an agent's Monitor tab, reads from the Application Insights resource connected to the project. It shows token usage, latency, run success rate, evaluation scores, and red teaming results, and supports alerts. A run success rate below 95% is worth investigating.

Quality and safety monitoring runs through evaluations. Continuous evaluation samples live traffic and scores it, capped by max_hourly_runs (100 per hour by default). Scheduled evaluation reruns a test dataset on a fixed schedule, which is how you detect drift: a decline in quality or safety over time. Grounding quality is monitored the same way by including evaluators such as groundedness.

Indexer runs show In Progress, Success, Failed, or Reset in the portal. A run can report Success while individual documents fail, as long as errors stay under the Max failed items setting; exceed it and the run fails. Each run lists its errors and warnings, and warnings aren't always problems (for example, an image with no text). Programmatically, Get Indexer Status returns a top-level status, where running only means the indexer is available, plus lastResult and up to 50 runs of executionHistory, each with its own status.

Service health is watched through Azure Monitor. The most common alerts are average search latency and throttled search queries percentage, with storage usage and service deletion as other candidates. Query strings, durations, and index names are only captured after you enable resource logging with a diagnostic setting.

Relevance comes from ranking. Keyword queries are scored with BM25, vector queries with HNSW or exhaustive KNN, hybrid queries are fused with Reciprocal Rank Fusion (RRF), and the semantic ranker rescores the top results into @search.rerankerScore. Scoring profiles boost nonvector fields by weight, freshness, or other functions at no extra cost. Microsoft's two recommended strategies are hybrid search with the semantic ranker, and agentic retrieval.

Foundry accepts Microsoft Entra ID tokens and API keys. Keys are static, all-or-nothing, and bypass RBAC, so Microsoft recommends Entra ID for production. In Python, get_bearer_token_provider(DefaultAzureCredential(), "https://ai.azure.com/.default") supplies tokens in place of a key, and once every caller uses tokens you can disable key-based authentication with disableLocalAuth. Agent Service and evaluations support Entra ID only.

Foundry separates control plane actions (creating resources, projects, and deployments) from data plane actions (building agents, running evaluations). Azure Owner and Contributor can manage resources but can't build agents. The Foundry roles, recently renamed from the Azure AI role names, are Foundry User for developers, Foundry Project Manager for team leads, Foundry Account Owner for resource administrators, Foundry Owner for both, and Foundry Agent Consumer for principals that only call agent endpoints, assignable down to a single agent.

Each agent authenticates to tools with a Microsoft Entra agent identity. Unpublished agents in a project share one identity; publishing creates a distinct identity, so downstream roles such as Storage Blob Data Contributor must be assigned again to the published agent's identity.

Networking has two decisions. Inbound: public (optionally selected IPs) or a private endpoint. Outbound for agents: public, a bring-your-own virtual network with a subnet delegated to Microsoft.App/environments (set at account creation and not changeable later), or a Microsoft-managed virtual network when you don't want to manage IP ranges.

Guardrails apply to models and, in preview, to agents. Each control names a risk (hate, sexual, violence, self-harm, user prompt attacks, indirect attacks, protected material, PII, task adherence, and others), the intervention points to scan, and an action. Models support Annotate or Annotate and block; agents support only Annotate and block. Models get the Microsoft.DefaultV2 guardrail unless you assign another, and you can override the guardrail for a single request with the x-policy-id header.

The four intervention points are user input, tool call (preview), tool response (preview), and output. Tool call and tool response apply only to agents, and only for tools that support moderation, such as Azure AI Search, OpenAPI, and Bing Grounding. An agent's own guardrail fully replaces its model's guardrail; an agent without one inherits the model's. Groundedness and Spotlighting controls don't take effect on agents yet.

Prompt Shields detect two attack types. User prompt attacks (formerly jailbreak risk detection) try to override system rules through the user's own input. Document attacks hide instructions in third-party content such as emails, web pages, or tool responses. Spotlighting (preview) adds defense by base-64 encoding documents so the model trusts them less, at the cost of more tokens.

By default, Azure OpenAI text models filter hate, sexual, violence, and self-harm content at medium severity on prompts and completions, detect jailbreaks on prompts, and detect protected material (text and code) on completions. Content at the safe level is annotated but never filtered. Only customers approved for modified guardrails can turn filters off.

Foundry ships evaluator families. General purpose (coherence, fluency) and textual similarity (F1, BLEU, ROUGE, METEOR, GLEU) measure writing. RAG evaluators split into process evaluation of retrieval (Retrieval, Document Retrieval) and system evaluation of the answer (Groundedness, Groundedness Pro, Relevance, Response Completeness). Groundedness checks precision, meaning nothing beyond the context, and Response Completeness checks recall against ground truth. Groundedness Pro uses Azure AI Content Safety and returns pass or fail without a model deployment.

Agent evaluators also split in two. System evaluators judge outcomes (Task Completion, Task Adherence, Intent Resolution, Task Navigation Efficiency, Customer Satisfaction). Process evaluators judge each step (Tool Call Accuracy, Tool Selection, Tool Input Accuracy, Tool Output Utilization, Tool Call Success). Risk and safety evaluators run on Microsoft-hosted models and include content harms, protected material, indirect attack, code vulnerability, ungrounded attributes, and, for agents only, prohibited actions and sensitive data leakage. Content harm evaluators score severity from 0 to 7.

The AI Red Teaming Agent automates adversarial probing with PyRIT attack strategies (encodings, jailbreaks, multi-turn escalation, and more) and scores results with the safety evaluators. Its headline metric is Attack Success Rate (ASR). Agentic risk categories run only in cloud red teaming.

When many samples fail, cluster analysis (preview) groups them by semantic similarity and explains each cluster with a description and recommended next steps. Results aren't stored, so download them before leaving the page.

Tracing is off by default and starts when a project is connected to an Application Insights resource. Built on OpenTelemetry, traces capture user inputs, agent outputs, tool calls and results, token usage, and latency, organized as spans. Because traces can hold personal data, redact secrets and personal information, and control who holds Log Analytics Reader (and Privileged Monitoring Data Reader for protected tables).

Content provenance marks AI-generated content from supported Foundry models in two ways: Content Credentials (C2PA), which are cryptographically signed metadata about origin, and invisible watermarks embedded in the content itself, which can survive when metadata is stripped by cropping or re-saving. Provenance describes where content came from; it doesn't establish whether content is accurate or who authored it.

Approval workflows put a human in the loop for consequential actions. For MCP tools, require_approval="always" makes the agent return an mcp_approval_request; your application reviews the tool name and arguments and replies with an mcp_approval_response. Microsoft recommends requiring approval for tools that write data and logging every approval.

Tool access starts with configuration. allowed_tools restricts an MCP connection to an allow list, and selecting a subset in the portal catalog does the same. tool_choice controls whether the model must call a tool (required), must not (none), or decides for itself (auto). Tool descriptions and results from remote MCP servers are untrusted input that can carry prompt injection, so validate critical values and require approval before consequential actions.

Task Adherence (preview) is a signal for oversight. Given the tools and the conversation, it flags tool calls that don't match what the user asked, such as calling apply_leave() when the user only wanted a balance, and returns taskRiskDetected with reasoning so you can block the call or escalate to human review. It's also available as a guardrail risk and an agent evaluator.

At organization scale, an AI gateway backed by Azure API Management can front MCP tools created in the Foundry portal, adding authentication, rate limits, IP filtering, and logging without changing agents or servers. Foundry Control Plane adds fleet-wide guardrail policies, compliance posture, and red teaming schedules across projects.

Two chat APIs are available through an OpenAI-compatible client. The Responses API combines the ChatCompletions and Assistants patterns, keeps conversation state across turns, and is recommended for new development. ChatCompletions is stateless and well established across platforms, so it still matters for existing code and portability.

Two SDKs reach those APIs. The Foundry SDK (AIProjectClient) is the choice when an app needs agents, tool approvals, cloud evaluations, tracing, Foundry direct models, or project connections. The OpenAI SDK is the choice for straightforward inference with maximum OpenAI compatibility. One app can use both.

Model choice is mostly a deployment-name change. Through the project endpoint, responses.create() works with Azure OpenAI models and Foundry direct models such as Phi and DeepSeek. For multi-turn chat, pass previous_response_id, but remember that instructions, history, tool schemas, tool outputs, and retrieved documents are all sent on every call, so token usage grows with conversation length.

For responsiveness, set stream=True and handle response.output_text.delta events, reading the response ID from response.completed if you chain turns. For many concurrent calls, use AsyncOpenAI and await.

RAG grounds a model in data it was never trained on. Retrieve relevant content from a data source, augment the prompt with that content, and generate the response. Without grounding, a model can produce fluent answers that contain fabricated details.

Retrieval works on meaning through embeddings, which are vectors produced by an embedding model. Texts with similar meaning have nearby vectors, and cosine similarity (near 1 means very similar) measures that closeness. Azure AI Search stores the index and supports keyword, semantic, vector, and hybrid search; hybrid search is recommended for generative AI apps.

In code, get a client with project.get_openai_client(), query the index, and put the returned documents into the prompt, for example in the system message, before calling the Responses API. For agents, the Azure AI Search tool does retrieval for you: project_connection_id and index_name are required, query_type defaults to vector_semantic_hybrid, top_k defaults to 5, and filter applies to every query.

A tool-augmented flow starts with the tools list in responses.create(). Common tools are code_interpreter (a sandboxed Python runtime with no external network access), web_search, file_search, and function. By default the model decides whether to call a tool, and instructions can steer that choice.

Foundry workflows (preview) orchestrate agents as connected nodes. Templates cover sequential (a fixed order), group chat (control passes between agents based on context or rules), and human in the loop (pause for user input). Node types include Invoke for agents; Flow nodes such as If/Else, Go To, and For Each; Data transformation nodes such as Set Variable and Parse value; Basic chat; and End. Workflows retire on December 1, 2026; Microsoft recommends Microsoft Agent Framework for new orchestration.

Multistep reasoning can also live in the prompt. Breaking a task into steps, such as extracting claims first and then writing queries for them, often makes models more reliable than one large request.

RAG evaluators separate retrieval from generation. Groundedness checks that a response does not fabricate beyond its context. Retrieval judges whether retrieved chunks are relevant, and Relevance judges whether the answer addresses the query; neither needs ground truth. Document Retrieval gives precise search metrics but requires relevance labels.

General-purpose evaluators judge writing quality independent of facts. builtin.coherence measures logical flow and builtin.fluency measures grammar and readability, each on a 1–5 scale with a default pass threshold of 3. These LLM-as-judge evaluators need a deployment_name.

Risk and safety evaluators run on Microsoft's hosted models, so they don't need a deployment. They cover hate and unfairness, sexual, violence, self-harm, protected material, code vulnerability, and ungrounded attributes. Indirect Attack applies to models only, while Prohibited actions and Sensitive data leakage (both preview) apply to agents only and need tool_calls.

In the portal, choose a target: Agent, Model, Dataset (score outputs you already have), or Traces (score interactions in Application Insights). When built-ins fall short, write a custom evaluator: code-based grade() functions for deterministic checks, prompt-based judges for subjective criteria, or endpoint-based evaluators hosted on your infrastructure.

A workflow or agent built in Foundry is called by reference. Create a conversation with openai_client.conversations.create() and call responses.create() with extra_body={"agent": {"name": workflow_name, "type": "agent_reference"}}. When streaming, response.output_item.done marks each completed action and response.completed marks the end.

Code that runs outside Foundry can still use it. Microsoft Agent Framework's FoundryChatClient calls the Responses API at the project endpoint, which brings catalog models, platform tools such as file search and memory, and the project's tracing, content filters, and identity configuration. The same code can later be deployed as a hosted agent.

An agent's endpoint is live as soon as the agent exists and keeps its URL across versions. Callers authenticate with Microsoft Entra ID and need the Foundry Agent Consumer role or higher; API keys aren't supported. Publishing to Microsoft 365 Copilot and Teams enables the activity protocol and either BotServiceRbac (Just you, no admin approval) or BotServiceTenant (the organization, with admin approval).

AIProjectClient provides the Foundry-native operations around all of this: retrieving connections, reading project configuration, enabling tracing, and managing datasets and indexes.

Each project has two endpoints. The project endpoint, https://{resource-name}.services.ai.azure.com/api/projects/<project-name>, is used to create an AIProjectClient, and its get_openai_client() method returns a client for inference. The Azure OpenAI endpoint, https://{resource-name}.openai.azure.com/openai/v1, is used with the OpenAI SDK.

Production apps should authenticate with Microsoft Entra ID. With the OpenAI SDK, create a token provider with get_bearer_token_provider, DefaultAzureCredential(), and the scope https://ai.azure.com/.default, and pass it as api_key. For local testing, sign in with az login.

If you must use keys, keep them out of code; store them in Azure Key Vault. OpenAI() with no arguments reads OPENAI_BASE_URL and OPENAI_API_KEY from the environment. Use the AzureOpenAI client, with azure_endpoint and api_version, only when you need a specific Azure OpenAI API version.

Foundry Agent Service has three runtime components. An agent supplies a reusable model, instructions, and tools. A conversation persists input and output items across turns on the service. A response is one unit of execution. Without a conversation, an app can chain stored responses or resend earlier items; with ChatCompletions, it must resend the full history itself.

Agent versions are immutable, and any change creates a new one. The agent's version_selector routes traffic either to the latest version (the default, which also updates Teams and Microsoft 365) or to a pinned active version, which is safer in production.

Roles and goals live in the instructions. A good system message states the role and expected outcome, boundaries, output format, and a "when unsure" policy. Structured inputs let one definition serve many contexts: handlebar placeholders such as {{variableName}} in instructions or tool settings are filled at runtime. Don't pass secrets through them.

Tool schemas decide how reliably the model calls functions. Give the function and parameters clear descriptions, list required parameters, and set strict=True with "additionalProperties": False.

Function calling is a loop. The model returns a function call with a name and arguments, your application runs the function, and you send the result back as a function_call_output item so the model can finish. With Foundry agents, runs expire 10 minutes after creation, and function definitions can be changed only through the SDK or REST API, not the portal.

For document retrieval, upload files into a vector store and add a file_search tool with vector_store_ids. During development, include=["file_search_call.results"] shows what was retrieved.

For memory (preview), attach the memory search tool with scope set to {{$userId}}. The user ID comes from the x-memory-user-id header when a backend calls on a user's behalf, or otherwise from the caller's Entra tenant and object IDs. Direct memory store API calls must specify scope explicitly.

In Microsoft Agent Framework, the Foundry Agent Service provider keeps chat history on the service, which is why it is recommended for production. Chat Completion providers leave history management to you.

OpenAPI tools need an operationId on every operation and support anonymous, API key, and managed identity authentication. Azure Functions tools are queue-based: the agent writes to an input queue, a queue trigger runs the function, and the result goes to an output queue with the original CorrelationId so the agent can match it.

Knowledge and search tools have their own constraints. The Azure AI Search tool on a private virtual network must use the project managed identity, because key-based authentication isn't supported there. Web search uses Grounding with Bing, whose data transfers occur outside compliance and geographic boundaries; Bing Custom Search limits grounding to domains you choose.

Toolboxes separate two identities. The agent's own identity authenticates to the toolbox, and Foundry gives the downstream service credentials for the signed-in user, so each user sees only their own data. Authentication is configured once on the connection, not in agent code.

Microsoft Agent Framework provides five patterns with one programming model. Concurrent orchestration sends the same task to several agents at once and collects independent results. Sequential orchestration passes each output to the next agent in a fixed order. Handoff transfers full control to whichever specialist the context calls for, one agent at a time.

Group chat puts agents, and optionally a person, in one conversation controlled by a chat manager. Each round the manager calls should_request_user_input, then should_terminate, and then either filter_results or select_next_agent. Keep group chats small, around three agents.

Magentic orchestration uses a manager that plans, delegates, and adapts for open-ended problems, maintaining a task ledger that people can review. It favors planning over speed.

Across service boundaries, use Agent2Agent (A2A). The classic Connected Agents tool isn't available in the new Foundry Agent Service; use the a2a tool to call an A2A endpoint, including another Foundry agent with incoming A2A enabled, or orchestrate the agents with a workflow.

A human-in-the-loop workflow pauses to ask for approval or input and resumes with the answer. For long-running hosted agents (preview), a @multi_turn_task chain suspends under a task_id and resumes when your app starts a new turn on the same task_id, even days later and across container restarts.

Rules decide when to involve a person. In a workflow, store an agent's confidence in a variable and branch with a Power Fx condition such as Local.Confidence > 0.8, escalating the rest.

Tool access needs central control. An AI gateway (preview) connected to the Foundry resource enforces authentication, rate limits, IP restrictions, and audit logging on new MCP tools created in the portal that don't use managed OAuth, without changing agents or servers.

Security by design ties it together: least-privilege RBAC, prompt filtering against injection, human approval before sensitive operations, and comprehensive logging of agent actions.

Once Application Insights is connected, Foundry logs server-side traces for prompt agents, hosted agents, and workflows with no code changes, keeping 90 days available. Add client-side tracing through the Foundry SDK and OpenTelemetry for your own code, or trace locally with the Foundry Toolkit for Visual Studio Code.

For agent evaluation, a rubric evaluator is the recommended primary measure: weighted dimensions an LLM judge applies to every response, which can be generated from the agent's instructions and tools. Choose Full conversations (preview) for end-to-end quality and Individual turns to debug specific behaviors, such as tool selection.

Production traffic closes the loop. A dataset from traces (preview) uses intelligent sampling to pick a diverse, representative set, and it needs the project's managed identity to have Log Analytics Reader on Application Insights. End-user feedback (preview) is logged as gen_ai.evaluation.result events, binary for thumbs up or down and Likert 5-point for ratings, linked to the trace.

Temperature controls randomness: around 0.2 for focused, factual output and around 0.7 for creative output. top_p limits generation to the most probable tokens. Adjust one or the other.

Prompt structure matters as much as parameters. End a prompt with a cue in the format you want to prime the output, and because models can show recency bias, repeat key instructions at the end of long prompts and evaluate the effect.

Reasoning models add reasoning_effort. none suits latency-critical tasks such as voice and classification, medium is a balanced starting point, and high, xhigh, and max spend more time and tokens on hard problems. Supported values vary by model.

When output must match a schema, use structured outputs (text.format in the Responses API or response_format in Chat Completions). JSON mode only guarantees valid JSON.

Chain-of-thought prompting asks a non-reasoning model to work step by step and show each step, which reduces errors and makes answers easier to check. Reasoning models reason internally; the supported way to inspect that reasoning is the reasoning.summary parameter in the Responses API. Other attempts to extract raw reasoning can lead to throttling or suspension.

Self-critique between agents is a maker-checker loop in group chat orchestration: one agent proposes, another reviews and sends feedback, and the manager repeats until the result is acceptable or a termination rule stops it.

The agent optimizer (preview) automates reflection at the configuration level. It scores a baseline against a dataset, generates candidates such as rewritten instructions, better tool descriptions, skills, or other models, scores each candidate, and ranks them by a 0.0–1.0 composite score. Because it invokes the agent for every task, mock tools that have side effects.

Latency follows TTLT = TTFT + (TBT × tokens generated). Because total time scales with output tokens, check Generated Completion Tokens before calling a slowdown a regression. For streaming apps, Time to Response (AzureOpenAITimeToResponse) shows first-token latency; for non-streaming requests, use Time to Last Byte (AzureOpenAITTLTInMS).

Reasoning tokens are hidden from message content but count against the context window and are billed as output tokens. Read them from output_tokens_details.reasoning_tokens (Responses) or completion_tokens_details.reasoning_tokens (Chat Completions), and cap total generation with max_output_tokens or max_completion_tokens.

Agent traces use OpenTelemetry GenAI conventions. An invoke_agent span can contain child invoke_agent, plan, execute_tool, and memory spans, with attributes such as gen_ai.tool.call.arguments. Use consistent attribute names, link traces to evaluation run IDs, and redact personal data and secrets before they reach telemetry.

Model router picks an underlying model per request, including for agents that use it as their base model. With the stateless Chat Completions API, session affinity (preview) uses an application-owned session ID so later turns try the same model first.

Multi-agent orchestration in the Responses API (preview, GPT-5.6) lets a root agent create subagents when work splits into independent, bounded workstreams. Keep one agent when steps depend on each other, share mutable state, or need a fixed, deterministic graph.

Rules and models can share a graph. In Microsoft Agent Framework workflows, executors are agents or custom logic, and edges route between them: direct, conditional, switch-case for predefined routing, and fan-out and fan-in to split and recombine work.

Foundry visual workflows retire on December 1, 2026. Microsoft Agent Framework is the recommended code-first replacement, Azure Logic Apps keeps a visual designer that mixes deterministic steps with agent reasoning, and A2A covers a lightweight handoff between two agents.

Foundry offers several image generation models, including the GPT-image series and Black Forest Labs FLUX models. In code, client.images.generate takes the deployment as model, plus prompt, n, and size. GPT-image models return base64 data in b64_json rather than URLs, so the app decodes and saves the bytes.

To build on existing media, call the image edit API. It accepts multipart/form-data rather than JSON, and up to 16 input images, each a PNG or JPG under 50 MB. The DALL-E 3 model was retired on March 4, 2026; use a GPT-image model instead.

Deploy Sora 2 from the model catalog. In the OpenAI Python SDK, client.videos.create starts a job with a prompt, size, and seconds; poll client.videos.retrieve until the status is completed, failed, or cancelled; and then download with client.videos.download_content. You can run two creation jobs at once, and completed videos stay available for 24 hours.

input_reference supplies an image as the first frame, and the prompt describes what happens next. The image must match the video size exactly and must not contain human faces. Sora 2 also rejects real people, including public figures, and copyrighted characters and music.

The image edit API changes an image according to a prompt. A mask limits the change: its fully transparent pixels (alpha of zero) mark the region to edit, and it must be a PNG with the same dimensions as the input image.

input_fidelity controls how closely the output preserves the input's style and features, especially faces, which makes subtle edits safer. It isn't supported by gpt-image-1-mini.

Foundry agents have an image generation tool (preview) for creating images from text during a conversation. For edits, masks, or partial-image streaming, call the Azure OpenAI Image API directly.

Call client.videos.remix with the completed video's ID and a prompt. Remix preserves scene transitions, visual layout, and overall structure, so it's the way to adjust a result without regenerating it from scratch.

Remix prompts work best when they describe only what changes, limited to one clearly stated adjustment, such as a new lens or a new color palette. Narrow, precise edits keep more of the source video.

Image requests accept quality (low, medium, high; default high, and lower is faster), n from 1 to 10, output_format of PNG (default) or JPEG, and output_compression from 0 to 100. Setting stream with partial_images from 1 to 3 shows progress, and a transparent background needs PNG output.

Sora 2 requests set size to 1280x720 (landscape) or 720x1280 (portrait) and seconds to 4, 8, or 12, with 4 as the default. Shorter clips follow instructions more reliably, so two 4-second clips can beat one 8-second clip. Strong video prompts read like a storyboard shot: framing, subject, action in beats, lighting, and style.

Vision-capable models such as gpt-4.1, gpt-4.1-mini, and Phi-4-multimodal-instruct accept images alongside text, and you can try them in the chat playground by uploading an image.

In the Responses API, a user message contains an input_text item and an input_image item. The image can be a web URL or a Base64 data URL such as data:image/jpeg;base64,.... For models that don't support the Responses API, ChatCompletions uses text and image_url content items, and the answer is in choices[0].message.content.

The detail setting on an image input controls how closely the model looks. low processes a 512x512 version for speed and fewer tokens, high adds detailed 512x512 segments at double the token budget each, and auto (the default) decides based on image size. A chat request can include up to 10 images of up to 20 MB each.

Caption style comes from the prompt: define the output format and length, add examples of good captions, and give the model context, such as writing for an outdoor product catalog.

A generic question gets a generic description. Stating the task and context, such as analyzing car damage for an insurance report, focuses the answer on the details that matter. Breaking a complex request into steps also helps.

When a model refuses, refine the prompt: ask what information it needs, place the image before the text in single-image prompts, ask it to describe the image in detail before the task, and ask it to explain its answer.

Content Understanding's prebuilt-imageSearch analyzer generates a one-paragraph description of an image for search and retrieval. prebuilt-documentSearch describes figures, charts, and diagrams inside documents.

For controlled output, build a custom analyzer on prebuilt-image with fields that use the generate method, such as a short alt text field and a longer description field, and state the length and style in each field description. Learn doesn't publish accessibility-specific guidance for these fields, so define the rules your accessibility standard requires in the descriptions and review the results.

A Content Understanding analyzer combines a base analyzer, model deployments, and a field schema. Each field uses a method: classify picks from predefined categories, generate infers a value such as a description, and extract copies values verbatim but works only in document analyzers.

Image analyzers suit visual tasks such as chart analysis, defect detection, and shelf analysis. They aren't optimized for analysis that depends mainly on extracted text; for scanned forms or labels, use a document field extraction schema. Write detailed, affirmative field descriptions in the language of the content.

Video analysis runs in two stages: content extraction (transcripts, key frames, shot detection) and then generative field extraction and segmentation. prebuilt-videoSearch returns RAG-ready Markdown and JSON with per-segment descriptions.

enableSegment set to false analyzes the whole video as one segment, which suits compliance checks and full summaries. Set to true, a natural-language contentCategories description, such as one segment per news story, drives custom segmentation. Segmentation consumes model tokens even without fields.

Frames are sampled at about one per second and resized to 512 × 512 pixels, and only speech is transcribed, so brief events, small text, and music or sound effects can be missed.

The outline's pro mode existed only in the 2025-05-01-preview API, which is retired. Its successor is agentic mode in 2026-06-01-preview, enabled by setting config.workflow to "agentic" when you create a document analyzer. It reasons across a document, performs calculations, validates values, and interprets tables and figures to build answers from evidence.

For straightforward extraction, a standard analyzer (the default workflow) is simpler, faster, and cheaper. The agentic preview supports one input file per request, document analyzers only, no extract fields, and no labeled samples, and it needs about 400,000 TPM on the model deployment. Use GA version 2025-11-01 for production.

Custom image analyzers can detect defects and anomalies, such as scratches, cracks, or misalignment on production lines, and detect and count products on retail shelves, returning the fields you define.

Video output places content in time. keyFrameTimesMs lists key frames, cameraShotTimesMs marks shot boundaries, and startTimeMs and endTimeMs bound each content object or segment. Camera shots and transcript phrases appear only when returnDetails is true.

Images and text share four harm categories: hate, sexual, violence, and self-harm. The image model returns a trimmed severity scale of 0, 2, 4, and 6. The image-with-text model supports the full 0–7 scale.

The multimodal API (preview) analyzes an image, its associated text, and, with enableOcr set to true, up to 1,000 characters of text read from the image. Images must be at most 4 MB and between 50 × 50 and 7,200 × 7,200 pixels.

Prompt Shields distinguishes user prompt attacks, typed by the user, from document attacks, where a third party hides instructions in content the system processes. Document attack subtypes include manipulated content, information gathering, fraud, malware, and attempts to change system rules.

When an app reads text from uploaded images and passes it to a model, that text is third-party content. Learn doesn't describe an image-specific shield, so treat extracted image text like any other external document: scan it with Prompt Shields for documents, keep it separate from the user prompt, and don't let it trigger actions without checks.

The default guardrail policy for image generation models filters hate, violence, sexual, and self-harm content at Medium on prompts and completions, adds content credentials to completions, and checks prompts for public figures, deceptive political content, protected art and studio characters, profanity, and jailbreaks.

For emerging prohibited content, such as a new symbol, custom categories (rapid) (preview) lets you create an incident, upload image or text samples, and include the incident in image analysis requests with no training step. Blocklists match exact text only.

To verify origin, the Content Provenance Detection API checks media for Microsoft C2PA manifests and invisible watermarks. A negative result doesn't prove content wasn't AI-generated, and provenance says nothing about truth or safety.

Named entity recognition (recognize_entities) returns entities by category, such as Person, Location, DateTime, and Organization, without training. It is one of Azure Language's core capabilities, along with PII detection, language detection, custom NER, and Text Analytics for health.

Key phrase extraction, sentiment analysis, summarization, and other legacy features retire from Azure Language on March 31, 2029, and Microsoft directs new projects to Foundry models. Summarization still illustrates the two styles you may see: extractive (ranked original sentences with positions) and abstractive (newly written sentences).

When a language model does the extraction, ask for JSON that matches a schema with structured outputs (see domain 2). For agents, the Azure Language MCP server exposes language detection, NER, PII redaction, and Text Analytics for health as tools, so the agent chooses the right one without routing code.

recognize_pii_entities returns PII entities with categories and confidence scores, plus redacted_text. Redaction policies choose how values are hidden: CharacterMask (the default), EntityMask (type labels such as [PERSON_1]), NoMask, or SyntheticReplacement (preview, realistic fake values). A default policy can be overridden per entity type.

Sentiment analysis labels sentences and documents positive, neutral, or negative with 0–1 confidence scores, and opinion mining ties opinions to specific aspects. Both retire from Azure Language on March 31, 2029, so tone and sentiment in new work are typically prompted from a Foundry model.

For model output, the PII filter scans completions. Annotate mode flags personal data, and Annotate and Block mode blocks the whole response, set per category such as financial information or government IDs. Harmful-content safety filtering is covered with guardrails in domain 1.

TextTranslationClient.translate takes InputTextItem objects and a to_language list, returning a translation per target. Omit from_language to auto-detect the source. transliterate converts script instead of language, for example Japanese from Jpan to Latn.

Translator's text translation API version 2026-06-06 (GA) has breaking changes from v3 and lets you pick standard neural machine translation (NMT) or an LLM deployment such as GPT-5.1, which needs a Foundry resource. LLM translation adds tone (formal, informal, neutral) and gender controls but has smaller limits (50 array elements of up to 5,000 characters) and token-based billing, so keep high-volume traffic on NMT where it fits.

To keep text untranslated, mark it with the notranslate class or translate="no", which work only when textType is HTML; or use a dynamic dictionary, a Custom Translator dictionary, or simply leave the string out.

Adaptive custom translation guides LLM translation with 5 to 10,000 aligned sentence pairs, built in minutes and updated without retraining, which suits fast-changing terminology. Custom Translator trains a dedicated NMT model on large parallel corpora (typically 10,000+ pairs, up to about 48 hours of training) for high-volume, strict terminology.

Custom NER builds a model for your own entity types: define the schema, label data precisely and consistently, train, evaluate, deploy, and call the Analyze API. For clinical text, Text Analytics for health needs no training and performs NER, relation extraction, UMLS entity linking, and assertion detection in one call.

PII detection can be adapted without training: entitySynonyms maps your terms (such as BAN) to supported entity types, and valueExclusionPolicy lists values that should never be detected or redacted. Microsoft recommends testing defaults first. For compliance-focused summaries from language models, combine clear instructions and output formats with grounding (see domain 2).

Speech to text uses a SpeechConfig, an optional AudioConfig (microphone by default, or a file), and a SpeechRecognizer. recognize_once_async returns a result whose Reason is RecognizedSpeech on success, NoMatch when no speech was recognized, or Canceled on error.

Text to speech uses a SpeechSynthesizer and speak_text_async; SynthesizingAudioCompleted means the audio is ready. SSML, sent with speak_ssml_async, adds speaking styles, pauses, phonemes, prosody, say-as rules, inserted audio, and multiple voices.

Generative models also transcribe and speak. The Audio API transcribes prerecorded files, and TTS models accept voice instructions. Azure Speech remains the recommended choice for real-time transcription, live captions, agent assist, dictation, pronunciation assessment, and live translation.

The Azure Speech MCP server exposes speech-to-text and text-to-speech tools to agents. It needs a blob container SAS URL: synthesized audio is saved there, and audio to transcribe is read from a public or SAS URL. Treat keys and SAS URLs as secrets.

The Voice Live API delivers low-latency speech-to-speech over WebSocket. Clients usually start with session.update to set voice, modalities, turn detection such as azure_semantic_vad, audio formats, noise suppression, and echo cancellation. Connecting Voice Live to a Foundry agent keeps instructions and logic in the agent, so client code doesn't change when behavior does.

Custom speech improves recognition of domain vocabulary or audio conditions. Compare base model output with human transcripts; a high word error rate suggests training. Custom models need a deployed endpoint except for batch transcription, and Voice Live can use custom speech for input and custom voice for output.

Audio-enabled models such as gpt-4o-audio-preview accept input_audio content in Chat Completions and return text, audio, or both, so a model can reason about a recording directly.

The GPT Realtime API supports speech-in, speech-out interaction. Connect through WebRTC for client apps (about 100 ms, generally recommended), WebSocket for server-to-server (about 200 ms), or SIP for telephony. Sessions can be voice-agent, translation, or transcription sessions.

LLM Speech enhances fast transcription with an LLM, adding translation and custom prompting. In Voice Live, GPT realtime models handle audio natively, while models such as gpt-4.1 and gpt-5 hear through Azure speech to text and speak through Azure text to speech.

Configure a SpeechTranslationConfig with speech_recognition_language and one add_target_language call per target, then pass it with an AudioConfig to a TranslationRecognizer. The result includes one translation per target language.

To speak translations, manual synthesis runs a SpeechSynthesizer for each translated text. Event-based synthesis sets a voice and reads audio from the Synthesizing event, but only for one target language.

Live Interpreter identifies the spoken language continuously, handles switching languages in one session, and preserves the speaker's style and tone. For GPT realtime models, deploy gpt-realtime-translate and use a translation session on /openai/v1/realtime/translations.

An indexer is a pull-model crawler: it reads a supported data source, such as Blob Storage, Azure SQL, Cosmos DB, or OneLake, and fills one index on demand or on a schedule as often as every five minutes. Faster updates need the push model. Each indexer has one data source and one target index, though many indexers can write to one index.

During indexing, images in documents can be extracted into normalized_images so skills like OCR run per image. Field mappings copy raw source values to the index; output field mappings copy nodes that skills add to the enriched document tree.

For audio, video, and images alongside documents, Content Understanding analyzers extract content and optional generated fields for each modality, which you then embed and load into one Azure AI Search index for retrieval.

A hybrid query sends search and vectorQueries in one request. Both run in parallel, and Reciprocal Rank Fusion scores each result by its rank in each list and merges them. Avoid orderby, which overrides relevance, and set vector k to 50 when semantic ranking follows.

Semantic ranker reranks the top 50 BM25 or RRF results with language understanding models, reporting @search.rerankerScore from 0 to 4. It also adds captions, highlights, optional answers, and optional query rewrite. Order semantic configuration fields by priority, because inputs are trimmed.

A vectorizer, defined in the index and assigned through a vector profile, converts text queries into vectors at query time. It must use the same embedding model as the indexing-time embedding skill.

Built-in skills bring Foundry Tools capabilities into indexing: language detection, entities, key phrases, translation, PII, OCR, and image captions and tags. Beyond a small free allowance, attach a Foundry Tools resource in the same region as the search service.

The Image Analysis skill returns visualFeatures such as description, tags, objects, and brands. The Document Layout skill uses the Document Intelligence layout model to output Markdown or text that keeps headings and image positions; it times out on documents needing more than five minutes of processing.

A Custom Web API skill calls your HTTPS endpoint, such as an Azure function, with batches of records identified by recordId. The default timeout is 30 seconds and the maximum is 230 seconds. Use it for domain logic or newer service versions that built-in skills don't cover. A skillset can also project enriched output to a knowledge store in Azure Storage (tables, JSON objects, or image files) for analytics outside search.

Chunking keeps text within embedding and chat model limits and represents sub-topics better. A recommended fixed-size start is 512 tokens with 25% overlap; the Content Understanding skill can instead produce semantic chunks that span pages.

Integrated vectorization chunks and embeds during indexing through a skillset with a chunking step and an embedding skill, and embeds queries through a matching vectorizer. The OCR skill reads /document/normalized_images and returns text and layoutText; set defaultLanguageCode to unk for mixed languages.

The multimodal pipeline extracts text and inline images, chunks text, verbalizes images with the GenAI Prompt skill, embeds text and image descriptions, and stores extracted images so answers can show them. Multimodal embedding models can replace the verbalization step.

An Azure AI Search knowledge base can be queried with the retrieve action through REST or an SDK, or used as a tool through the MCP endpoint every knowledge base exposes. Knowledge bases combine indexed sources, which are ingested and can return citation URLs, with remote sources, such as Web or remote SharePoint, that are queried live.

To connect a knowledge base to a Foundry agent, give the project's managed identity Search Index Data Reader on the search service. For indexed content with access control lists, pass the user's token in x-ms-query-source-authorization so results respect permissions.

Instructions decide whether an agent actually uses retrieval: require searching before answering, specify the citation format, and define a fallback when the answer is not in the knowledge base. Test factual, synthesis, out-of-scope, and ambiguous questions, and monitor citation and fallback rates.

The read model extracts printed and handwritten text and detects languages. The layout model adds tables with cell positions, selection marks, structure, and optional key-value pairs. Every model builds on read's OCR.

Prebuilt models extract fields from common documents, including invoices, receipts, bank statements, tax and mortgage forms, and ID documents, with no training. Check for a prebuilt model before building a custom one.

Custom template models suit fixed layouts: fast, cheap training, 100+ languages. Custom neural models suit varied or semi-structured layouts with higher accuracy but longer training. Composed models bundle custom models and pick the best one per document, and custom classifiers route documents to extraction models.

prebuilt-documentSearch is recommended for document RAG ingestion. It extracts paragraphs, tables, and figures, describes and analyzes charts and diagrams, captures handwritten annotations, summarizes the document, and returns chunked output ready for embedding.

In Azure AI Search, the Azure Content Understanding skill outputs tables and figures as Markdown, keeps cross-page tables whole, allows chunks to span pages, and costs less than the Document Layout skill. Figures in Markdown can include a generated description (enableFigureDescription) and an appended chart, table, or Mermaid diagram (enableFigureAnalysis).

For fields that automation or agents act on, set estimateFieldSourceAndConfidence to true to get confidence scores and source locations, which are off by default in the GA API. Then route low-confidence values to human review.

A custom analyzer definition names a baseAnalyzerId (such as prebuilt-document), a fieldSchema of typed fields with methods and descriptions, and the completion and embedding models. Create it with begin_create_analyzer or a REST PUT.

Analysis is asynchronous. In the Python SDK, begin_analyze returns a poller whose .result() waits for completion; over REST you poll analyzerResults with the operation ID. Results contain contents with fields and Markdown. analyze takes a URL, and analyzeBinary takes uploaded file bytes.

Choose output by need: extractionMode selects layout-aware extraction when tables and reading order matter, or text-only extraction for fast, clean Markdown. The content extraction analyzers prebuilt-read, prebuilt-layout, and prebuilt-digitalParse need no language or embedding model; use read or layout for scanned documents.