你当前正在访问 Microsoft Azure Global Edition 技术文档网站。 如果需要访问由世纪互联运营的 Microsoft Azure 中国技术文档网站,请访问 https://docs.azure.cn。
Develop Agents using the Azure AI Foundry platform, leveraging an extensive ecosystem of models, tools, and capabilities from OpenAI, Microsoft, and other LLM providers.
The client library uses a single service version v1 of the AI Foundry data plane REST APIs.
Important
Preview and beta features
- Build
Beta*ClientandBeta*AsyncClientinstances throughAgentsClientBuilder.beta(). These clients automatically opt in to their preview service area; you do not needallowPreview(true)for them. - Use
AgentsClientBuilder.allowPreview(true)when calling preview APIs on non-Beta clients, such as preview agent definitions, draft agent versions, hosted-agent sessions, session files, and code package operations onAgentsClient/AgentsAsyncClient. - Classes and methods annotated with
@Betaare preview API surface and may change in future releases. See Preview operation groups and beta clients for details.
Documentation
Various documentation is available to help you get started
Getting started
Prerequisites
- Java Development Kit (JDK) with version 8 or above
- Azure Subscription
Adding the package to your product
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-ai-agents</artifactId>
<version>2.10.0</version>
</dependency>
Authentication
Azure Identity package provides the default implementation for authenticating the client.
Key concepts
Create an AgentsClient
To interact with the Azure Agents service, you'll need to create an instance of the AgentsClient class.
AgentsClient agentsClient = new AgentsClientBuilder()
.credential(new DefaultAzureCredentialBuilder().build())
.endpoint(endpoint)
.buildAgentsClient();
Alternatively, you can create an asynchronous client using the AgentsAsyncClient class.
AgentsAsyncClient agentsAsyncClient = new AgentsClientBuilder()
.credential(new DefaultAzureCredentialBuilder().build())
.endpoint(endpoint)
.buildAgentsAsyncClient();
The Agents client library has the following sub-clients which group the different operations that can be performed:
AgentsClient/AgentsAsyncClient: Perform operations related to agents, including agent generation and optimization. WhenallowPreview(true)is configured, these clients can also use preview definitions, hosted-agent sessions, session files, and code package operations.ResponsesClient/ResponsesAsyncClient: Create responses that require Azure-specific request fields, such as an explicitAgentReferenceor structured inputs. For standard OpenAI Responses API calls through an agent endpoint, use an agent-scoped OpenAI client. See the OpenAI Responses API documentation for more information.BetaMemoryStoresClient/BetaMemoryStoresAsyncClient(preview): Manage memory stores and individual memory items for agents.ToolboxesClient/ToolboxesAsyncClient: Manage toolboxes and toolbox versions.VoiceAgentsWebSocketClient/VoiceAgentsWebSocketAsyncClient: Open typed realtime WebSocket sessions with voice agents.BetaVoiceAgentsLiveWebSocketClient/BetaVoiceAgentsLiveWebSocketAsyncClient(preview): Open native GPT Live WebSocket sessions with persisted Live voice agents.VoiceAgentsTelephonyClient/VoiceAgentsTelephonyAsyncClient: Manage telephony bindings, calls, transfer targets, and outbound call jobs.BetaVoiceAgentsTelephonyClient/BetaVoiceAgentsTelephonyAsyncClient(preview): Manage outbound telephony campaigns.VoiceAgentsConversationsClient/VoiceAgentsConversationsAsyncClient: Read and delete persisted voice-agent conversations and retrieve their responses, items, and audio.
OpenAI conversation operations are accessed through the OpenAI Official Java SDK's
ConversationService. Persisted voice-agent conversation records use the voice conversation clients listed above.
See the OpenAI Conversation API documentation for more information.
To access each sub-client you need to use your AgentsClientBuilder(). The Agents client library takes the Official OpenAI SDK as a dependency, which is used for all operations, except the ones corresponding to direct Agent management.
AgentsClientBuilder builder = new AgentsClientBuilder()
.credential(new DefaultAzureCredentialBuilder().build())
.endpoint(endpoint)
.allowPreview(true); // Only needed for preview APIs on non-Beta clients that support them.
// Agents sub-clients
AgentsClient agentsClient = builder.buildAgentsClient();
AgentsAsyncClient agentsAsyncClient = builder.buildAgentsAsyncClient();
// Responses sub-clients.
ResponsesClient responsesClient = builder.buildResponsesClient();
ResponsesAsyncClient responsesAsyncClient = builder.buildResponsesAsyncClient();
// Memory Stores sub-clients (preview).
BetaMemoryStoresClient memoryStoresClient = builder.beta().buildBetaMemoryStoresClient();
BetaMemoryStoresAsyncClient memoryStoresAsyncClient = builder.beta().buildBetaMemoryStoresAsyncClient();
// Toolboxes sub-clients.
ToolboxesClient toolboxesClient = builder.buildToolboxesClient();
ToolboxesAsyncClient toolboxesAsyncClient = builder.buildToolboxesAsyncClient();
The OpenAI Official Java SDK is imported transitively and can be built directly from the AgentsClientBuilder. Use it to access conversation operations and other OpenAI services:
OpenAIClient openAIClient = builder.buildOpenAIClient();
OpenAIClientAsync openAIAsyncClient = builder.buildOpenAIAsyncClient();
// Agent-scoped OpenAI clients for invoking an agent endpoint.
OpenAIClient agentScopedOpenAIClient = builder.buildAgentScopedOpenAIClient(agentName);
OpenAIClientAsync agentScopedOpenAIAsyncClient = builder.buildAgentScopedOpenAIAsyncClient(agentName);
// ResponsesClient wraps the OpenAI SDK's ResponseService with Azure-specific options.
ResponsesClient responsesClient = builder.buildResponsesClient();
ResponseService responseService = responsesClient.getResponseService();
// OpenAI SDK ConversationService accessed from OpenAIClient
ConversationService conversationService = agentScopedOpenAIClient.conversations();
Agent version drafts
Draft agent versions are preview candidates that are not promoted to the agent's latest released version. Create one with
CreateAgentVersionInput.setDraft(true), and pass true as the includeDrafts argument to
listAgentVersions when you need to list draft versions. Build the non-Beta client with
allowPreview(true) to opt in to the DraftAgents=V1Preview service feature.
See the full samples in AgentDraftSample.java and AgentDraftAsyncSample.java.
Agent tools
The SDK supports a variety of tools that can be attached to agent definitions. Some tools are generally available, while others are in preview and may change in future releases.
Generally available tools:
| Tool class | Description |
|---|---|
A2ATool |
Agent-to-agent (A2A) protocol |
AzureAISearchTool |
Azure AI Search |
AzureFunctionTool |
Azure Functions |
BingGroundingTool |
Bing grounding |
BrowserAutomationTool |
Browser automation |
CaptureStructuredOutputsTool |
Structured output capture |
CodeInterpreterTool |
Code interpreter |
FileSearchTool |
File search |
FunctionTool |
Custom function calling |
ImageGenTool |
Image generation |
McpTool |
Model Context Protocol (MCP) |
NamespaceTool |
Namespaces for grouping function and custom tools |
OpenApiTool |
OpenAPI spec-based tools |
ToolSearchTool |
Deferred tool search |
WebSearchTool |
Web search |
Preview tools:
| Tool class | Description |
|---|---|
BingCustomSearchPreviewTool |
Bing custom search |
BrowserAutomationPreviewTool |
Deprecated; use BrowserAutomationTool |
ComputerUsePreviewTool |
Computer use |
FabricIqPreviewTool |
Fabric IQ |
MemorySearchPreviewTool |
Memory search |
MicrosoftCopilotToolsetPreview |
Microsoft Copilot built-in tools |
MicrosoftFabricPreviewTool |
Microsoft Fabric |
ReminderPreviewTool |
Reminder scheduling |
SharepointPreviewTool |
SharePoint grounding |
WebIqPreviewTool |
WebIQ MCP servers |
WebSearchPreviewTool |
Web search |
WorkIqPreviewTool |
Work IQ |
Supported tool classes may also expose optional name, description, and toolConfigs properties for user-defined labels and per-tool configuration.
Preview operation groups and beta clients
Several operation groups in the Agents client library expose preview service features. These features require the Foundry-Features HTTP header. The SDK populates that header for you; you do not need to set the header value manually.
APIs annotated with @Beta are part of the SDK's preview surface, even when they appear on a non-Beta client. These APIs are subject to breaking changes in future releases and should be used with the same compatibility expectations as other preview features.
Use AgentsClientBuilder.allowPreview(true) when building non-Beta clients that support preview service behavior. For example, AgentsClient and AgentsAsyncClient use this builder setting to allow the service to return preview response types:
AgentsClientBuilder builder = new AgentsClientBuilder()
.credential(new DefaultAzureCredentialBuilder().build())
.endpoint(endpoint)
.allowPreview(true);
AgentsClient agentsClient = builder.buildAgentsClient();
Build clients whose names start with Beta from AgentsClientBuilder.beta(). These clients always opt in to their corresponding preview service area. Requests sent by these clients automatically include the appropriate Foundry-Features header, and their APIs can send or return preview/beta request and response types. You do not need to call allowPreview(true) to use a Beta*Client.
| Beta sub-client | Automatically populated Foundry-Features value |
|---|---|
BetaMemoryStoresClient |
MemoryStores=V1Preview |
BetaVoiceAgentsLiveWebSocketClient / BetaVoiceAgentsLiveWebSocketAsyncClient |
VoiceAgents=V1Preview |
BetaVoiceAgentsTelephonyClient |
VoiceAgents=V1Preview |
BetaVoiceAgentsLiveConversationsClient |
VoiceAgents=V1Preview |
The async Beta*AsyncClient counterparts follow the same behavior.
Realtime voice-agent sessions
Create and manage preview voice agents with VoiceAgentDefinition and an AgentsClient or AgentsAsyncClient built
using allowPreview(true). Use VoiceAgentsWebSocketClient or VoiceAgentsWebSocketAsyncClient to open a typed,
bidirectional session with an existing voice agent. The client acquires a token for https://ai.azure.com/.default,
negotiates the realtime WebSocket subprotocol, and sends the Foundry-Features: VoiceAgents=V1Preview header
only when built with allowPreview(true). GPT Live WebSocket clients always send that preview header,
independently of allowPreview.
The session API supports text and audio in the format configured on the voice agent, typed streaming server events, response cancellation, client-executed function tools, and optional persisted conversations. See Realtime voice-agent WebSocket examples for complete samples.
Use VoiceAgentWebSocketConnectionOptions with openWebSocketSession to configure session IDs, agent version
selection, transport, structured inputs, persistence, buffering, and timeouts. Options are copied when the session is opened, so
later changes do not affect the active session.
VoiceAgentWebSocketConnectionOptions options = new VoiceAgentWebSocketConnectionOptions()
.setAgentSessionId("session-id")
.setAgentVersionOverride("2")
.setStructuredInputs("{\"language\":\"en\"}")
.setStoreEnabled(true);
The SDK owns the WebSocket route, API version, and authentication scope. Endpoint,
credential, service version, configuration-based proxy settings, and ClientOptions are reused from
AgentsClientBuilder. Custom HTTP clients, pipelines, policies, and retry settings are rejected when building a
WebSocket client because the native WebSocket transports cannot apply them.
Agent optimization
AgentsClient and AgentsAsyncClient create and monitor agent optimization jobs.
Construct AgentOptimizationJob with AgentOptimizationModelConfiguration and AgentOptimizationConfiguration
(or PromptOptimizationConfiguration for prompt optimization). These jobs return scored candidates for instructions,
skills, tools, or model improvements. Use estimateOptimizationJob, listOptimizationCandidates,
getOptimizationCandidate, and promoteOptimizationCandidate to estimate costs and manage candidates.
See Agent optimizer in Foundry Agent Service for the service workflow and the complete
examples in AgentOptimizationSample.java
and AgentOptimizationAsyncSample.java.
Memory item management
BetaMemoryStoresClient and BetaMemoryStoresAsyncClient manage memory stores and individual memory items. In addition to store-level operations, use createMemory, updateMemory, listMemories, getMemory, and deleteMemory to manage individual memories. ListMemoriesOptions supports filtering by scope and MemoryItemKind, including MemoryItemKind.PROCEDURAL. See MemoryStoreItemsSample and MemoryStoreItemsAsyncSample for complete examples.
For conversational memory workflows, use beginUpdateMemories to extract memories from conversation items, searchMemories to retrieve relevant memories, and deleteScope to remove all memories for a scope. See MemoryStoreAdvancedSample and MemoryStoreAdvancedAsyncSample for complete synchronous and asynchronous examples.
Using OpenAI's official library
If you prefer using the OpenAI official Java client library instead, you can do so by including that dependency in your project instead and following the instructions in the linked repository. Additionally, you will have to set up your OpenAIClient as shown below:
OpenAIClient client = OpenAIOkHttpClient.builder()
.baseUrl(endpoint.endsWith("/") ? endpoint + "openai/v1" : endpoint + "/openai/v1")
.credential(BearerTokenCredential.create(AuthenticationUtil.getBearerTokenSupplier(
new DefaultAzureCredentialBuilder().build(), "https://ai.azure.com/.default")))
.build();
ResponseCreateParams responseRequest = new ResponseCreateParams.Builder()
.input("Hello, how can you help me?")
.model(model)
.build();
Response result = client.responses().create(responseRequest);
For this direct setup, ensure that the AI Foundry project endpoint path ends with openai/v1, as shown above.
Examples
Prompt Agent
This example creates a prompt agent and uses a conversation to retain context across requests.
Create an Agent
Creating an Agent can be done like in the following code snippet:
PromptAgentDefinition promptAgentDefinition = new PromptAgentDefinition("gpt-4o");
AgentVersionDetails agent = agentsClient.createAgentVersion("my-agent", promptAgentDefinition);
This returns an AgentVersionDetails containing the agent's name and version. By default, the agent endpoint serves the latest version through the Responses protocol using Microsoft Entra authentication.
To pin the endpoint to a specific version, see ConfigureAgentEndpoint.java and ConfigureAgentEndpointAsync.java.
Create conversation
Use the same agent-scoped OpenAI client for conversations and responses:
OpenAIClient agentScopedClient = builder.buildAgentScopedOpenAIClient(agent.getName());
ConversationService conversationsClient = agentScopedClient.conversations();
Conversation conversation = conversationsClient.create();
Use conversation.id() to add messages and create subsequent responses for the same agent:
conversationsClient.items().create(
ItemCreateParams.builder()
.conversationId(conversation.id())
.addItem(EasyInputMessage.builder()
.role(EasyInputMessage.Role.SYSTEM)
.content("You are a helpful assistant that speaks like a pirate.")
.build()
).addItem(EasyInputMessage.builder()
.role(EasyInputMessage.Role.USER)
.content("Hello, agent!")
.build()
).build()
);
To scope conversation operations to a delegated end user, set FOUNDRY_USER_IDENTITY to an opaque application-generated value and apply it as the x-ms-user-identity header. The caller must have the agents/endpoints/UserIdentityImpersonation/action RBAC permission. See the sync UserIdentityConversation.java and async UserIdentityConversationAsync.java samples.
Text generation with Responses
Invoke the OpenAI Responses API through the same client:
Response response = agentScopedClient.responses().create(ResponseCreateParams.builder()
.conversation(conversation.id())
.build());
// To extract Azure-specific response details:
AzureCreateResponseDetails azureResults = ResponsesClient.getAzureFields(response);
For asynchronous calls, use buildAgentScopedOpenAIAsyncClient.
Using Agent tools
Agents can be enhanced with specialized tools for various capabilities. For complete working examples, see the tools/ folder under samples.
In the description below, tools are organized by their Foundry connection requirements: "Built-in Tools" (which do not require a Foundry connection) and "Connection-based Tools" (which require a Foundry connection).
Built-in Tools
These tools work immediately without requiring external connections.
Code Interpreter (documentation)
Write and run Python code in a sandboxed environment, process files and work with diverse data formats.
// Create a CodeInterpreterTool with default auto container configuration
CodeInterpreterTool tool = new CodeInterpreterTool();
See the full sample in CodeInterpreterSync.java.
File Search (documentation)
Search through files in a vector store for knowledge retrieval:
// Create a FileSearchTool with the vector store ID
FileSearchTool tool = new FileSearchTool(Collections.singletonList(vectorStore.id()));
See the full sample in FileSearchSync.java.
Image Generation (documentation)
Generate images from text descriptions:
// Create image generation tool with model, quality, and size
ImageGenTool imageGenTool = new ImageGenTool()
.setModel(ImageGenToolModel.fromString(imageModel))
.setQuality(ImageGenToolQuality.LOW)
.setSize(ImageGenToolSize.fromString("1024x1024"));
See the full sample in ImageGenerationSync.java.
Web Search (Preview) (documentation)
Search the web for current information:
// Create a WebSearchPreviewTool
WebSearchPreviewTool tool = new WebSearchPreviewTool();
See the full sample in WebSearchSync.java.
Computer Use (documentation)
Interact with computer interfaces through simulated actions and screenshots:
ComputerTool tool = new ComputerTool();
See the full sample in ComputerUseSync.java.
Model Context Protocol (MCP) (documentation)
Connect agents to external MCP servers:
// Uses gitmcp.io to expose a GitHub repository as an MCP-compatible server
McpTool tool = new McpTool("api-specs")
.setServerUrl("https://gitmcp.io/Azure/azure-rest-api-specs")
.setRequireApproval("always");
See the full sample in McpSync.java.
OpenAPI (documentation)
Call external APIs defined by OpenAPI specifications without additional client-side code:
// Load the OpenAPI spec from a JSON file
Map<String, BinaryData> spec = OpenApiFunctionDefinition.readSpecFromFile(
SampleUtils.getResourcePath("assets/httpbin_openapi.json"));
OpenApiTool tool = new OpenApiTool(
new OpenApiFunctionDefinition(
"httpbin_get",
spec,
new OpenApiAnonymousAuthDetails())
.setDescription("Get request metadata from an OpenAPI endpoint."));
See the full sample in OpenApiSync.java.
Function Tool (documentation)
Define custom functions that allow agents to interact with external APIs, databases, or application logic:
Map<String, Object> locationProp = new LinkedHashMap<String, Object>();
locationProp.put("type", "string");
locationProp.put("description", "The city and state, e.g. Seattle, WA");
Map<String, Object> unitProp = new LinkedHashMap<String, Object>();
unitProp.put("type", "string");
unitProp.put("enum", Arrays.asList("celsius", "fahrenheit"));
Map<String, Object> properties = new LinkedHashMap<String, Object>();
properties.put("location", locationProp);
properties.put("unit", unitProp);
Map<String, BinaryData> parameters = new HashMap<String, BinaryData>();
parameters.put("type", BinaryData.fromObject("object"));
parameters.put("properties", BinaryData.fromObject(properties));
parameters.put("required", BinaryData.fromObject(Arrays.asList("location", "unit")));
parameters.put("additionalProperties", BinaryData.fromObject(false));
FunctionTool tool = new FunctionTool("get_weather", parameters, true)
.setDescription("Get the current weather in a given location");
See the full sample in FunctionCallSync.java.
Azure Functions
Integrate Azure Functions with agents to extend capabilities via serverless compute. Functions are invoked through Azure Storage Queue triggers, allowing asynchronous execution of custom logic:
// Create Azure Function tool with Storage Queue bindings
AzureFunctionTool azureFunctionTool = new AzureFunctionTool(
new AzureFunctionDefinition(
new AzureFunctionDefinitionDetails("queue_trigger", parameters)
.setDescription("Get weather for a given location"),
new AzureFunctionBinding(
new AzureFunctionStorageQueue(queueServiceEndpoint, inputQueueName)),
new AzureFunctionBinding(
new AzureFunctionStorageQueue(queueServiceEndpoint, outputQueueName))
)
);
When the agent handles a response, it enqueues function arguments to the input queue. Your Azure Function processes the request and returns results through the output queue.
See the full sample in AzureFunctionSync.java.
Memory Search (Preview) (documentation)
The Memory Search tool adds memory to an agent, allowing the agent's AI model to search for past information related to the current user prompt:
// Create memory search tool
MemorySearchPreviewTool tool = new MemorySearchPreviewTool(memoryStore.getName(), scope)
.setUpdateDelaySeconds(1);
See the full sample in MemorySearchSync.java showing how to create an agent with a memory store and use it across multiple conversations.
Connection-Based Tools
These tools require configuring connections in your Microsoft Foundry project and use a projectConnectionId.
Azure AI Search (documentation)
Integrate with Azure AI Search indexes for powerful knowledge retrieval and semantic search capabilities:
// Create Azure AI Search tool with index configuration
AzureAISearchTool aiSearchTool = new AzureAISearchTool(
new AzureAISearchToolResource(Arrays.asList(
new AISearchIndexResource()
.setProjectConnectionId(connectionId)
.setIndexName(indexName)
.setQueryType(AzureAISearchQueryType.SIMPLE)
))
);
See the full sample in AzureAISearchSync.java.
Bing Grounding (documentation)
Ground agent responses with real-time web search results from Bing to provide up-to-date information:
// Create Bing grounding tool with connection configuration
BingGroundingTool bingTool = new BingGroundingTool(
new BingGroundingSearchToolParameters(Arrays.asList(
new BingGroundingSearchConfiguration(bingConnectionId)
))
);
See the full sample in BingGroundingSync.java.
Bing Custom Search (Preview) (documentation)
Warning: Grounding with Bing Custom Search uses Grounding with Bing, which has additional costs and terms: terms of use and privacy statement. Customer data will flow outside the Azure compliance boundary.
Use custom-configured Bing search instances for domain-specific or filtered web search results:
// Create Bing Custom Search tool with connection and instance configuration
BingCustomSearchPreviewTool bingCustomSearchTool = new BingCustomSearchPreviewTool(
new BingCustomSearchToolParameters(Arrays.asList(
new BingCustomSearchConfiguration(connectionId, instanceName)
))
);
See the full sample in BingCustomSearchSync.java.
Microsoft Fabric (Preview) (documentation)
Query data from Microsoft Fabric data sources:
// Create Microsoft Fabric tool with connection configuration
MicrosoftFabricPreviewTool fabricTool = new MicrosoftFabricPreviewTool(
new FabricDataAgentToolParameters()
.setProjectConnections(Arrays.asList(
new ToolProjectConnection(fabricConnectionId)
))
);
See the full sample in FabricSync.java.
Fabric IQ (Preview) (documentation)
Connect agents to Fabric IQ project connections for enterprise data grounding:
FabricIqPreviewTool fabricIqTool = new FabricIqPreviewTool(fabricIqConnectionId)
.setServerLabel("fabric-iq-tool")
.setRequireApproval("never");
The samples use FOUNDRY_PROJECT_ENDPOINT, FOUNDRY_MODEL_NAME, and the fully qualified
FABRIC_IQ_PROJECT_CONNECTION_ID. FOUNDRY_AGENT_NAME and FABRIC_IQ_USER_INPUT are optional.
The response text and any returned annotations are printed before the temporary agent version is deleted.
See the full samples in FabricIQSync.java and FabricIQAsync.java.
Work IQ (Preview) (documentation)
Ground agent responses in the signed-in user's Microsoft 365 work context through a Work IQ project connection:
// Create a Work IQ tool with a fully qualified project connection resource ID
WorkIqPreviewTool workIqTool = new WorkIqPreviewTool(workIqConnectionId);
Set FOUNDRY_PROJECT_ENDPOINT, FOUNDRY_MODEL_NAME, and WORK_IQ_PROJECT_CONNECTION_ID before running the
sample. FOUNDRY_AGENT_NAME and WORK_IQ_USER_INPUT are optional. Work IQ uses delegated authentication and
honors the signed-in user's Microsoft 365 permissions.
See the full samples in WorkIQSync.java and WorkIQAsync.java.
Microsoft SharePoint (Preview) (documentation)
Search through SharePoint documents for grounding:
// Create SharePoint grounding tool with connection configuration
SharepointPreviewTool sharepointTool = new SharepointPreviewTool(
new SharepointGroundingToolParameters()
.setProjectConnections(Arrays.asList(
new ToolProjectConnection(sharepointConnectionId)
))
);
See the full sample in SharePointGroundingSync.java.
Browser Automation (documentation)
Interact with web pages using BrowserAutomationTool. For toolbox configuration, use BrowserAutomationToolboxTool.
The legacy BrowserAutomationPreviewTool and BrowserAutomationPreviewToolboxTool are deprecated and no longer available.
Use BrowserAutomationTool and BrowserAutomationToolboxTool instead. See Browser Automation documentation.
// Create browser automation tool with connection configuration
BrowserAutomationTool browserTool = new BrowserAutomationTool(
new BrowserAutomationToolParameters(
new BrowserAutomationToolConnectionParameters(connectionId)
)
);
See the full sample in BrowserAutomationSync.java.
Agent-to-Agent (A2A) (documentation)
Enable agent-to-agent communication with remote A2A endpoints:
// Create agent-to-agent tool with A2A protocol version and connection ID
A2ATool a2aTool = new A2ATool(A2AProtocolVersion.V1_0)
.setProjectConnectionId(a2aConnectionId);
See the full sample in AgentToAgentSync.java.
MCP with Project Connection (documentation)
MCP integration using project-specific connections for accessing connected MCP servers:
// Create MCP tool with project connection authentication
McpTool mcpTool = new McpTool("api-specs")
.setServerUrl("https://api.githubcopilot.com/mcp")
.setProjectConnectionId(mcpConnectionId)
.setRequireApproval("always");
See the full sample in McpWithConnectionSync.java.
OpenAPI with Project Connection (documentation)
This example calls TripAdvisor using tripadvisor_openapi.json. Set OPENAPI_PROJECT_CONNECTION_ID to a Custom Keys connection containing your TripAdvisor API key under key.
// Create OpenAPI tool with project connection authentication
OpenApiTool openApiTool = new OpenApiTool(
new OpenApiFunctionDefinition(
"tripadvisor",
spec,
new OpenApiProjectConnectionAuthDetails(
new OpenApiProjectConnectionSecurityScheme(connectionId)))
.setDescription("TripAdvisor API to get travel information."));
See the full sample in OpenApiWithConnectionSync.java.
Toolbox Tools
Toolbox tools are defined in toolbox versions and managed through ToolboxesClient / ToolboxesAsyncClient. Toolbox versions use ToolboxTool subclasses rather than agent Tool subclasses. Use invokeLatestToolboxMcp to invoke the latest toolbox version through its MCP endpoint.
Toolbox Search
Toolbox Search lets an agent search the available toolbox tools at runtime. The GA implementation is ToolSearchToolboxTool (toolbox_search), and the preview implementation ToolboxSearchPreviewToolboxTool (toolbox_search_preview) is maintained alongside it for backward compatibility.
ToolSearchToolboxTool toolboxSearchTool = new ToolSearchToolboxTool()
.setName("search_tools")
.setDescription("Search over available toolbox tools at runtime.");
ToolboxVersionDetails version = toolboxesClient.createToolboxVersion(
toolboxName,
Collections.singletonList(toolboxSearchTool),
"Toolbox version with a Toolbox Search tool.",
null,
null,
null);
System.out.printf("Created toolbox: %s%n", version.getName());
System.out.printf("Toolbox version: %s%n", version.getVersion());
for (ToolboxTool tool : version.getTools()) {
System.out.printf("Tool type: %s%n", tool.getType());
}
See the full sample in ToolboxSearchToolboxSample.java.
Reminder (preview)
The Reminder tool lets a hosted agent schedule itself to run again at a future time. It is connectionless and is available only to hosted agents, not prompt agents.
ReminderPreviewToolboxTool reminderTool = new ReminderPreviewToolboxTool()
.setName("schedule_reminder")
.setDescription("Schedule a reminder that re-invokes this agent at a future time.");
ToolboxVersionDetails version = toolboxesClient.createToolboxVersion(
toolboxName,
Collections.<ToolboxTool>singletonList(reminderTool),
"Built-in reminder tool for a self-scheduling agent.",
null,
null,
null);
System.out.printf("Created toolbox: %s%n", version.getName());
System.out.printf("Toolbox version: %s%n", version.getVersion());
System.out.printf("Tool type: %s%n", version.getTools().get(0).getType());
See the full samples in ReminderPreviewToolboxSample.java and ReminderPreviewToolboxAsyncSample.java.
Shell
The Shell toolbox tool runs commands in an isolated container. This example uses an automatically provisioned container, which has outbound network access disabled by default. A prompt agent consumes the toolbox through its versioned MCP endpoint.
ShellToolboxTool shellTool = new ShellToolboxTool(new ToolboxShellContainerAutoEnvironment())
.setDescription("Runs shell commands in a sandboxed container.");
ToolboxVersionDetails toolboxVersion = toolboxesClient.createToolboxVersion(
toolboxName,
Collections.<ToolboxTool>singletonList(shellTool),
"Toolbox with a shell tool running in an auto-provisioned container.",
null,
null,
null);
See the full end-to-end sample in ShellToolboxSample.java.
Streaming responses
An agent-scoped OpenAI client can stream response events as they arrive instead of waiting for the complete response. This is useful for displaying text in real time and observing tool execution progress.
Synchronous streaming
The OpenAI SDK's synchronous createStreaming method returns a StreamResponse<ResponseStreamEvent>. Close it with try-with-resources, and use ResponseAccumulator to collect the events into a final Response:
// Use ResponseAccumulator to collect streamed events into a final Response
ResponseAccumulator responseAccumulator = ResponseAccumulator.create();
// Stream response - text is printed as it arrives
try (StreamResponse<ResponseStreamEvent> events = openAIClient.responses().createStreaming(
ResponseCreateParams.builder()
.input("Tell me a short story about a brave explorer.")
.build())) {
events.stream().forEach(event -> {
responseAccumulator.accumulate(event);
event.outputTextDelta()
.ifPresent(textEvent -> System.out.print(textEvent.delta()));
});
}
System.out.println(); // newline after streamed text
// Access the complete accumulated response
Response response = responseAccumulator.response();
System.out.println("\nResponse ID: " + response.id());
See the full samples in SimpleStreamingSync.java, FunctionCallStreamingSync.java, and CodeInterpreterStreamingSync.java.
Asynchronous streaming
The OpenAI SDK's asynchronous createStreaming method returns an AsyncStreamResponse<ResponseStreamEvent>.
Use StreamingResponseUtils.toFlux to adapt it to a Reactor Flux and manage the underlying stream lifecycle:
// Adapt OpenAI streaming events to a Reactor Flux.
Mono<Void> streamingCompletion = Mono.defer(() -> {
ResponseAccumulator responseAccumulator = ResponseAccumulator.create();
AsyncStreamResponse<ResponseStreamEvent> stream = openAIAsyncClient.responses().createStreaming(
ResponseCreateParams.builder()
.input("Tell me a short story about a brave explorer.")
.build());
return StreamingResponseUtils.toFlux(stream)
.doOnNext(event -> responseAccumulator.accumulate(event)
.outputTextDelta()
.ifPresent(textEvent -> System.out.print(textEvent.delta())))
.then()
.doOnSuccess(unused -> {
System.out.println(); // newline after streamed text
// Access the complete accumulated response
Response response = responseAccumulator.response();
System.out.println("\nResponse ID: " + response.id());
});
});
See the full samples in SimpleStreamingAsync.java, FunctionCallStreamingAsync.java, and CodeInterpreterStreamingAsync.java.
Stream hosted agent session logs
Hosted agent session logs can be streamed as SessionLogEvent values after a session has been created. The data property contains the log payload as an opaque string.
Session log streams are long-lived and may remain open until the client cancels or the session ends. Bound the stream with take, timeout, by disposing the subscription, or by breaking iteration when appropriate.
Synchronous session log streaming
The synchronous session log method returns IterableStream<SessionLogEvent>, which can be consumed with a standard for-each loop:
IterableStream<SessionLogEvent> sessionLogs =
agentsClient.getSessionLogStream(agentName, agentVersion, sessionId);
int logsRead = 0;
for (SessionLogEvent event : sessionLogs) {
System.out.printf("[%s] %s%n", event.getEvent(), event.getData());
// Session log streams are long-lived; connection is closed on client disconnection
if (++logsRead == 100) {
break;
}
}
Asynchronous session log streaming
The asynchronous session log method returns Flux<SessionLogEvent>, integrating naturally with Reactor pipelines:
agentsAsyncClient.getSessionLogStream(agentName, agentVersion, sessionId)
.take(100)
.doOnNext(event -> System.out.printf("[%s] %s%n", event.getEvent(), event.getData()))
.blockLast();
Structured inputs
Structured inputs allow you to define named parameters on an agent that get substituted into its prompt template at runtime. This is useful when you want the same agent definition to handle different users or contexts by simply changing the input values.
Define structured inputs on an agent
When creating the agent, declare each structured input with a description and whether it is required. Use {{inputName}} placeholders in the instructions to reference them:
// Create an agent with structured input definitions
Map<String, StructuredInputDefinition> structuredInputDefinitions = new LinkedHashMap<>();
structuredInputDefinitions.put("userName",
new StructuredInputDefinition().setDescription("User's name").setRequired(true));
structuredInputDefinitions.put("userRole",
new StructuredInputDefinition().setDescription("User's role").setRequired(true));
AgentVersionDetails agent = agentsClient.createAgentVersion("structured-input-agent",
new PromptAgentDefinition(model)
.setInstructions("You are a helpful assistant. "
+ "The user's name is {{userName}} and their role is {{userRole}}. "
+ "Greet them and confirm their details.")
.setStructuredInputs(structuredInputDefinitions));
Create a response with structured input values
When creating a response, pass a Map<String, BinaryData> whose keys match the structured input names declared on the agent. The values are substituted into the prompt template before the model processes the request:
// Build the structured input values that match the agent's definitions
Map<String, BinaryData> structuredInputValues = new LinkedHashMap<>();
structuredInputValues.put("userName", BinaryData.fromObject("Alice Smith"));
structuredInputValues.put("userRole", BinaryData.fromObject("Senior Developer"));
// Create a response using AzureCreateResponse, which flattens agent_reference
// and structured_inputs as top-level properties in the request body
Response response = responsesClient.createAzureResponse(
new AzureCreateResponseOptions()
.setAgentReference(new AgentReference(agent.getName()).setVersion(agent.getVersion()))
.setStructuredInputs(structuredInputValues),
ResponseCreateParams.builder().input("Hello! Can you confirm my details?")
);
Streaming is also supported via createStreamingAzureResponse, which returns an IterableStream<ResponseStreamEvent> (sync) or Flux<ResponseStreamEvent> (async).
See the full sample in CreateResponseWithStructuredInput.java.
Voice agent samples
The following voice-agent samples cover agent management and persisted conversations.
| Scenario | Samples |
|---|---|
| Lifecycle | VoiceAgentBasicSample.java and VoiceAgentBasicAsyncSample.java create, retrieve, update, list, enable, disable, and delete voice agents. |
| Versions and drafts | VoiceAgentVersionsSample.java creates and lists released and draft versions. |
| Guided generation | VoiceAgentGenerateSample.java generates and creates a voice agent from high-level input. |
| Audio and tools | VoiceAgentWithToolsSample.java configures PCM audio, transcription, voice activity detection, function tools, and system tools. |
| Persisted conversations | VoiceAgentReadConversationSample.java reads responses and transcripts, while VoiceAgentReadConversationAudioSample.java downloads call and item audio. |
Authenticate with DefaultAzureCredential. Every voice sample requires FOUNDRY_PROJECT_ENDPOINT. Samples that create explicit definitions optionally use FOUNDRY_VOICE_MODEL, FOUNDRY_VOICE_MODEL_TYPE, and FOUNDRY_VOICE_AGENT_NAME. The persisted-conversation samples require FOUNDRY_VOICE_AGENT_NAME and FOUNDRY_VOICE_CONVERSATION_ID.
Realtime voice-agent WebSocket examples
Realtime WebSocket sessions provide bidirectional text and audio communication with a voice agent. Create the voice agent before opening a session; the lifecycle samples above demonstrate how to create one.
Create a realtime WebSocket client
Build a synchronous or asynchronous realtime client from AgentsClientBuilder.
Realtime handshakes omit the preview feature header by default; set allowPreview(true) before building
the client to opt into voice-agent preview features.
AgentsClientBuilder builder = new AgentsClientBuilder()
.credential(new DefaultAzureCredentialBuilder().build())
.endpoint(endpoint);
VoiceAgentsWebSocketClient realtimeClient
= builder.buildVoiceAgentsWebSocketClient();
VoiceAgentsWebSocketAsyncClient realtimeAsyncClient
= builder.buildVoiceAgentsWebSocketAsyncClient();
Send a synchronous text turn
Connections require an https:// or wss:// project endpoint. Insecure endpoints are rejected before acquiring a
token. This also applies to localhost; use certificate-verified TLS for local servers.
Unknown server event types are returned as RawRealtimeServerEvent; getRawEvent() preserves the complete JSON object.
Use sendEvent(BinaryData) to send raw JSON objects, including event types or fields not modeled by this SDK. Sessions
receive UTF-8 JSON in text or binary WebSocket messages.
VoiceAgentWebSocketConnectionOptions is copied when a connection is opened. Configure it before connecting; later
changes do not affect the active session:
setReceiveBufferCapacitysets a bounded event queue (default 256, range 1-65536).setOverflowStrategydefaults toERROR, which closes an overflowing connection.DROP_OLDESTandDROP_LATESTexplicitly permit data loss and should only be used when the application can tolerate missing events.setMaxMessageSizelimits accepted message bytes (default 32 MiB). Oversized messages terminate the connection. The sync transport checks size after receiving a complete message; this does not bound the transport's allocation.- Malformed JSON or invalid UTF-8 terminates reception by default. Set
setMalformedEventHandlerto report and skip malformed events while continuing reception. This callback must not block; throwing from it terminates the session.
VoiceAgentWebSocketConnectionOptions options
= new VoiceAgentWebSocketConnectionOptions()
.setReceiveBufferCapacity(512)
.setMaxMessageSize(8 * 1024 * 1024)
.setOverflowStrategy(VoiceAgentWebSocketOverflowStrategy.ERROR);
try (VoiceAgentsWebSocketSessionClient session = realtimeClient.openWebSocketSession(agentName, options)) {
session.sendEvent(BinaryData.fromString(
"{\"type\":\"response.create\",\"event_id\":\"response-1\"}"));
for (RealtimeServerEvent event : session.receiveEvents()) {
if (event instanceof RawRealtimeServerEvent) {
BinaryData payload
= ((RawRealtimeServerEvent) event).getRawEvent();
System.out.println("Received an unrecognized event with " + payload.getLength() + " bytes.");
}
}
}
Connect to the voice agent, add the user's text to the conversation, and request a response. Consume the typed server events until the response finishes. A session supports only one consumer of receiveEvents().
For bounded synchronous waits, use receiveEvents(Duration) with a positive per-event timeout. A timeout raises
IllegalStateException with a TimeoutException cause, leaves the session open, and allows the same iterator to retry.
Use close(code, reason) or asynchronous closeAsync(code, reason) to send a custom close frame. Close reasons must
fit in 123 UTF-8 bytes and close codes must be valid WebSocket codes. The first asynchronous close request wins.
try (VoiceAgentsWebSocketSessionClient session = realtimeClient.openWebSocketSession(agentName)) {
session.sendText("Hello! Tell me about the services you provide.");
session.createResponse();
for (RealtimeServerEvent event : session.receiveEvents()) {
if (event instanceof RealtimeResponseTextDeltaEvent) {
System.out.print(((RealtimeResponseTextDeltaEvent) event).getDelta());
} else if (event instanceof RealtimeErrorEvent) {
RealtimeErrorEvent error = (RealtimeErrorEvent) event;
System.out.println("Session error: " + error.getError().message());
} else if (event instanceof RealtimeResponseDoneEvent) {
break;
}
}
}
Use sendText and createResponse again for subsequent turns while the session remains open. Call cancelResponse to interrupt an active response.
Send an asynchronous text turn
The asynchronous client returns a Mono when connecting and a Flux<RealtimeServerEvent> when receiving events. Mono.usingWhen closes the session on completion, error, or cancellation.
Mono.usingWhen(
realtimeAsyncClient.openWebSocketSession(agentName),
session -> session.sendText("Hello! Tell me about the services you provide.")
.then(session.createResponse())
.thenMany(session.receiveEvents())
.doOnNext(event -> {
if (event instanceof RealtimeResponseTextDeltaEvent) {
System.out.print(((RealtimeResponseTextDeltaEvent) event).getDelta());
}
})
.takeUntil(event -> event instanceof RealtimeResponseDoneEvent)
.then(),
VoiceAgentsWebSocketSessionAsyncClient::closeAsync,
(session, error) -> session.closeAsync(),
VoiceAgentsWebSocketSessionAsyncClient::closeAsync)
.block();
Stream audio and handle function tools
Use appendInputAudio to send bytes in the input format configured on the voice agent (the included live audio sample
uses PCM16). Use commitInputAudio to commit buffered audio when server-side voice activity detection is not configured,
and clearInputAudio to discard pending input. Audio output arrives through RealtimeResponseAudioDeltaEvent events.
When a RealtimeResponseFunctionCallArgumentsDoneEvent requests a client-side tool, execute the function and call
sendFunctionCallOutput with its call ID and serialized result; the helper also requests the next response.
| Scenario | Complete sample |
|---|---|
| Synchronous live text | VoiceAgentLiveTextConversationSample.java |
| Asynchronous live text | VoiceAgentLiveTextConversationAsyncSample.java |
| Asynchronous live audio | VoiceAgentLiveAudioConversationAsyncSample.java |
| Live function tool | VoiceAgentLiveFunctionToolSample.java |
All realtime examples require FOUNDRY_PROJECT_ENDPOINT and optionally use FOUNDRY_VOICE_AGENT_NAME. The function-tool example also optionally uses FOUNDRY_VOICE_MODEL and FOUNDRY_VOICE_MODEL_TYPE. The asynchronous text and audio examples delete their generated agents by default; set FOUNDRY_KEEP_VOICE_AGENT=true to retain them.
The synchronous live text sample deletes and replaces the selected agent if creation returns a conflict,
then deletes the replacement on exit, including on failure. Use a dedicated sample-agent name: this also
applies when FOUNDRY_VOICE_AGENT_NAME is set. This synchronous sample does not support retaining its agent.
The live audio example requires a Java Sound-compatible microphone and speaker. It streams signed, little-endian, mono PCM16 audio at 24 kHz. These examples use WebSocket transport. Although the generated protocol models include WebRTC signaling events, the Java client does not provide a WebRTC peer connection or media implementation.
GPT Live WebSocket examples (preview)
GPT Live is a different wire protocol from Realtime. Create a persisted agent with VoiceAgentLiveDefinition,
then use BetaVoiceAgentsLiveWebSocketClient or BetaVoiceAgentsLiveWebSocketAsyncClient to connect.
These clients use
wss://{project-endpoint}/agents/{agent_name}/endpoint/protocols/voice/live/sessions?api-version=v1,
send the Foundry-Features: VoiceAgents=V1Preview header, and do not negotiate the realtime WebSocket
subprotocol. allowPreview(true) is not required.
Foundry sends the initial native session.start. Wait for session.started before sending application events;
do not send session.start yourself. Use sendEvent(ClientEvent) and receiveEvents() with the
com.openai.models.live event models from the SDK's existing openai-java dependency. These are distinct
from Realtime event models. The synchronous client also offers receiveEvents(Duration) for bounded waits.
The sendEvent(BinaryData) and receiveRawEvents() APIs expose complete JSON objects when the original
wire representation is needed, including unknown properties, nested response.event backend events, and the
Foundry conversation_id on session.started. OpenAI typed models retain extra properties and unknown event
variants, but typed serialization need not preserve the original JSON formatting.
The existing samples named VoiceAgentLive*ConversationSample above demonstrate live connections using the
Realtime protocol; use the separate GPT Live clients for VoiceAgentLiveDefinition agents.
For example, configure and open a synchronous Live session for an agent using Responses delegation:
BetaVoiceAgentsLiveWebSocketClient liveClient = builder.beta().buildBetaVoiceAgentsLiveWebSocketClient();
VoiceAgentLiveWebSocketConnectionOptions options = new VoiceAgentLiveWebSocketConnectionOptions()
.setReceiveBufferCapacity(512)
.setMaxMessageSize(8 * 1024 * 1024)
.setOverflowStrategy(VoiceAgentWebSocketOverflowStrategy.ERROR)
.setHandshakeTimeout(Duration.ofSeconds(30))
.setCloseTimeout(Duration.ofSeconds(10))
.setStoreEnabled(false);
try (BetaVoiceAgentsLiveWebSocketSessionClient session = liveClient.openWebSocketSession(agentName, options)) {
for (ServerEvent event : session.receiveEvents()) {
if (event.isSessionStarted()) {
session.sendEvent(ClientEvent.ofResponseCreate(ResponseCreateEvent.builder().build()));
} else if (event.isSessionOutputTranscriptDelta()) {
System.out.print(event.asSessionOutputTranscriptDelta().delta());
} else if (event.isSessionClosed() || event.isErrorEvent()) {
break;
}
}
}
Send only native Live commands valid for the configured delegation mode, whether typed or raw.
In particular, response.item.create and response.create are Responses-delegation-only commands.
Service error events arrive as events for the application to inspect. Typed receivers decode JSON and
fail on malformed events unless a malformed-event callback was explicitly configured. Raw receivers do not
parse JSON or construct typed models; transport size limits and UTF-8 validation still apply.
Configure timeouts, bounded receive capacity, overflow handling, message size, persistence, agent version,
and structured inputs with VoiceAgentLiveWebSocketConnectionOptions. Structured inputs are passed as
unencoded JSON to setStructuredInputs and URL-encoded by the SDK into the structured_inputs query parameter.
Do not log endpoint URLs containing sensitive template inputs. Realtime-only transport and agent-session-ID
options are not exposed. Options are copied when opening a session.
Both transports retain the realtime clients' credential, proxy, custom-header and builder validation behavior. Custom HTTP clients, pipelines, policies, and retry settings are unsupported and fail at build time rather than being ignored. Each session permits one receiver, shared by typed and raw receive APIs. The default receive buffer holds 256 events; overflow terminates the session unless a drop strategy explicitly permits event loss.
Runnable GPT Live scenarios
These examples create a uniquely named VoiceAgentLiveDefinition agent and connect using native GPT Live WebSockets.
Most use Responses delegation and delete their generated agent on exit, including on failure. The connection
lifecycle sample uses client delegation and deletes only the agent version it created. None modifies or deletes an
existing agent. Set these environment variables before running a sample's main method:
| Variable | Value |
|---|---|
FOUNDRY_PROJECT_ENDPOINT |
Your Foundry project endpoint. Authentication uses DefaultAzureCredential. |
FOUNDRY_GPT_LIVE_MODEL |
Your GPT Live deployment name, not a model catalog ID. Required by the connection lifecycle sample; optional for other samples, which default to gpt-live-1. |
FOUNDRY_GPT_LIVE_BACKEND_MODEL |
Optional. Your Responses backend deployment name in the Live resource. Defaults to gpt-5.6-terra when unset. |
FOUNDRY_GPT_LIVE_STORE |
Optional, text and microphone/speaker examples. Set to true to persist conversation content and audio. Defaults to false. |
FOUNDRY_KEEP_GPT_LIVE_AGENT |
Optional. Set to true to retain the sample-created agent; defaults to false. |
FOUNDRY_VOICE_AGENT_NAME |
Optional prefix used by the connection lifecycle sample. Defaults to sample-live-connection. |
Each sample reads, defaults, and validates these settings and configures its client builder directly in its own main method.
Shared helpers receive explicit values rather than reading environment variables.
Override the model variables if your deployment names differ from these defaults. Explicitly blank values are invalid.
Generated agent names use gpt-live-<scenario>-<UUID> and fit the service's 63-character limit.
| Scenario | Runnable sample |
|---|---|
| Synchronous multi-turn text | GptLiveTextConversationSample.java |
| Asynchronous multi-turn text | GptLiveTextConversationAsyncSample.java |
| Asynchronous connection lifecycle (no audio devices) | GptLiveConnectionAsyncSample.java |
| Asynchronous microphone and speaker | GptLiveAudioConversationAsyncSample.java |
| Responses backend function tool | GptLiveFunctionToolSample.java |
| Advanced synchronous console input and speaker | GptLiveInteractiveTextConversationSample.java |
| Advanced asynchronous console input and speaker | GptLiveInteractiveTextConversationAsyncSample.java |
Read persisted GPT Live conversations and audio
These REST examples read existing persisted data; they do not start a WebSocket session, create an agent, or delete resources.
| Scenario | Runnable sample |
|---|---|
| Synchronous history and transcripts | GptLiveReadConversationSample.java |
| Asynchronous history and transcripts | GptLiveReadConversationAsyncSample.java |
| Synchronous stored audio download | GptLiveReadConversationAudioSample.java |
| Asynchronous stored audio download | GptLiveReadConversationAudioAsyncSample.java |
Use an agent/session configured with store=true and retain the agent. With either text WebSocket example or the
microphone/speaker example, set
both FOUNDRY_GPT_LIVE_STORE=true and FOUNDRY_KEEP_GPT_LIVE_AGENT=true. The first sets the session persistence
override; the second prevents sample cleanup from deleting the agent. Keeping the agent alone does not enable
persistence. The text, advanced interactive, and microphone examples print the Foundry conversation_id from session.started
when provided. If absent, startup output explicitly reports that no persisted conversation ID was provided;
it never substitutes the native session ID.
Use session.started.conversation_id, not the native session ID. Set:
| Variable | Value |
|---|---|
FOUNDRY_PROJECT_ENDPOINT |
Required. Your Foundry project endpoint; authentication uses DefaultAzureCredential. No default. |
FOUNDRY_GPT_LIVE_AGENT_NAME |
Required. The existing Live agent's name. No default. |
FOUNDRY_GPT_LIVE_CONVERSATION_ID |
Required. The persisted Foundry conversation ID. No default. |
FOUNDRY_GPT_LIVE_ITEM_ID |
Optional, audio examples only: a speech item's ID, not a delegation item. Defaults to unset (merged audio only); explicitly blank values are invalid. |
Model deployment settings such as FOUNDRY_GPT_LIVE_MODEL and FOUNDRY_GPT_LIVE_BACKEND_MODEL are not needed:
these samples read existing conversations rather than creating agents.
Run a history sample's main method with no arguments. For audio, pass one existing local output directory;
there is no default output directory.
Audio examples create uniquely named WAV files without overwriting existing files. History examples explicitly
advance the after cursor until an empty page because the generated list clients do not expose automatic
continuation; missing or repeated cursors fail explicitly. They show conversation status and usage seconds,
input/output transcripts, and delegation targets/response IDs. Stored speech segments are not native turns or
proof that audio was played. Treat the printed transcripts and downloaded audio as sensitive data.
Wait for session completion and recording finalization before downloading. HTTP errors propagate: 404 can mean
missing audio; 409 can mean recording_not_ready or recording_unavailable. Inspect Retry-After before retrying.
BYOS metadata is reported without calling Foundry content endpoints; use the metadata's blob URI and your own
Azure Storage credentials to download BYOS audio. The current generated Live REST builder methods require explicit
preview opt-in, so these samples add Foundry-Features: VoiceAgents=V1Preview using an HTTP pipeline header policy.
Using the text WebSocket samples
The primary synchronous and asynchronous text examples show agent and Responses-delegation configuration,
connection, typed sends, event handling, and graceful closure directly. Each command-line argument is a prompt;
with no arguments, they send What can you help me with? and exit after the reply.
They wait for session.started, send response.item.create followed by response.create, and wait for the
nested response.completed before advancing to the next turn. They print backend text without console input,
speaker playback, microphone capture, or silent audio uploads.
Using the advanced interactive WebSocket samples
The separate synchronous and asynchronous interactive examples read terminal input. After each reply,
they prompt for another message; submit a blank line or enter exit or quit to close the session.
Each example owns its conversation flow; shared helpers provide audio-device plumbing and transcript rendering.
Both request a spoken opening greeting with
session.commentary.append, play the voice frontend's 24-kHz PCM16 audio, and print its spoken transcript
rather than backend working notes. The synchronous example uses synchronous session send/receive calls,
with workers for receiving events and reading console input.
Before displaying the first You: prompt, each waits for both greeting audio and transcript, one second without
speech activity, and completion of queued speaker playback. Continuous silent audio does not restart the timer.
For this heuristic, PCM16 frames with peak sample magnitude at most 256 (about -42 dBFS) are treated as silence.
The quiet period is a heuristic because native GPT Live
has no speech-completed event: a long pause within a greeting can be mistaken for its end. Missing greeting output
fails with a timeout instead of silently starting input before the introduction.
It sends 20 milliseconds of silent PCM (960 bytes) every 20 milliseconds after startup, providing continuous
audio input without opening a microphone. This consumes voice-session usage while you type or wait.
The silent stream is stopped and pending audio sends complete before the sample requests session.close.
If sending falls behind, silent-frame production waits for demand instead of accumulating timer ticks or
failing with an interval overflow; it resumes without a burst of stale frames.
Speaker playback runs off the WebSocket receive thread. If a speaker is unavailable, the sample reports that
and still prints transcripts, like the Realtime text example.
Agent creation and deletion occur on the main thread.
Console input is line-oriented: press Enter to submit a message. The sample flushes the You: prompt immediately;
character echo while typing is controlled by Windows CMD/PowerShell and its terminal host, not by the service.
Interactive input uses a normal blocking line read on a worker rather than polling Reader.ready(),
matching the Realtime sample's console interaction. The WebSocket receiver continues independently.
When a native Java console is attached, input comes from System.console().reader() rather than System.in.
Without a console, the sample reads System.in, supporting redirected input. End-of-input (EOF), a blank line,
or an explicit exit command closes the session.
The terminal echoes characters on the You: line as you type. Assistant transcript fragments are assembled into
utterances independently of typed turns and backend completion. A one-second speech-inactivity period marks an
utterance boundary using the same heuristic as the greeting; a long pause within speech can still divide the text.
Completed utterances are printed with one Assistant: label each. While input is active, completed text waits
until Enter so it cannot overwrite typed characters; incomplete text stays buffered across input submissions.
Audio playback and input remain available. This dependency-free approach can delay captions until the next Enter.
Session closure flushes any remaining transcript. Completed and incomplete text share a 65,536-character buffer
limit; exceeding it fails the sample explicitly instead of dropping transcript text or growing memory indefinitely.
The native console's echoed Enter already starts a new line; the sample avoids adding another blank line.
After a remote disconnect, a console read that the operating system cannot interrupt may require Enter to release
the worker. The sample does not close shared standard input to force a pending read to return.
Backend response.completed permits the next typed prompt but does not mean that the frontend finished speaking:
native GPT Live has no output-audio-done event. The sample keeps receiving audio and transcripts while you type.
See GPT-Live audio and transcript events.
Server remote_hangup closes the conversation normally. Other unexpected closures remain failures and include
the complete session.closed event, including its reason, rather than hiding the service diagnostic.
Using the microphone and tool samples
The audio example requires a Java Sound-compatible microphone and speaker supporting raw, mono PCM16LE at
24 kHz. After session.started, it sends a greeting request followed by response.create so the agent can
open the conversation. It labels spoken transcript fragments as You: or Assistant: and omits backend
text deltas, which are not necessarily the words spoken by the voice frontend.
Press Enter to end the call; asking the model to end it does not itself close the session.
Service moderation (content_filter) ends the call with an explicit error and the original service event;
it is not treated as a successful completion. Audio devices and the WebSocket are released on failure.
The standalone audio and text samples' main methods also shut down shared Reactor schedulers on exit
to avoid lingering timer threads under exec:java. Do not invoke those main methods inside an application that
shares Reactor schedulers.
The audio sample sends session.input_audio.append, plays base64-decoded session.output_audio.delta chunks, and
prints native input/output transcript deltas. It does not send Realtime audio-buffer commit or response-cancel
commands. Capture and playback use dedicated threads; the playback backlog is bounded to 60 seconds.
Device errors and overflow end the call explicitly. Press Enter to stop capture and finalize the session.
This example must be the only reader of standard input.
The function-tool example configures get_weather on the Responses backend, consumes completed
function_call items from nested response.output_item.done events, then sends function_call_output items
with the original call_id before requesting continuation. Its weather results are simulated demonstration
data, not live weather. It does not assume lifecycle snapshots contain output items, and does not add
Foundry-managed-agent tool-result routing.
All WebSocket scenarios request session.close and wait for session.closed before normal transport cleanup.
The asynchronous text examples apply a 60-second timeout to startup, each response, and finalization;
the advanced interactive example waits for terminal input without a timeout.
The synchronous text and tool examples use a 60-second event-wait timeout;
the microphone example uses a 60-second startup/finalization timeout.
An unexpected disconnect or native service error fails the sample rather than being reported as success.
Service API versions
The client library targets the latest service API version by default. The service client builder accepts an optional service API version parameter to specify which API version to communicate.
Select a service API version
You have the flexibility to explicitly select a supported service API version when initializing a service client via the service client builder. This ensures that the client can communicate with services using the specified API version.
When selecting an API version, it is important to verify that there are no breaking changes compared to the latest API version. If there are significant differences, API calls may fail due to incompatibility.
Always ensure that the chosen API version is fully supported and operational for your specific use case and that it aligns with the service's versioning policy.
Tracing
azure-ai-agents emits experimental OpenTelemetry GenAI spans
and metrics for agent creation and response (chat / invoke_agent) operations. To use this
experimental feature, set AZURE_EXPERIMENTAL_ENABLE_GENAI_TRACING=true and configure an OpenTelemetry implementation
for the client through ClientOptions or globally.
Customer-controlled prompts, responses, messages, instructions, function arguments, tool results, and
equivalent agent content are omitted unless you also set
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true. Review your privacy requirements before enabling content
recording. The former AZURE_TRACING_GEN_AI_CONTENT_RECORDING_ENABLED setting remains available as a compatibility
fallback when the OpenTelemetry setting is absent.
Typed and raw-response methods are covered, including synchronous and asynchronous streaming. Streaming spans remain open until the stream completes, fails, is cancelled, or is explicitly closed.
// Set AZURE_EXPERIMENTAL_ENABLE_GENAI_TRACING=true before running this sample.
// Configure OpenTelemetry exporters and sampling, then pass the SDK through ClientOptions.
OpenTelemetrySdk openTelemetry = OpenTelemetrySdk.builder().build();
AgentsClient agentsClient = new AgentsClientBuilder().endpoint(endpoint)
.credential(new DefaultAzureCredentialBuilder().build())
.clientOptions(new ClientOptions()
.setTracingOptions(new OpenTelemetryTracingOptions().setOpenTelemetry(openTelemetry)))
.buildAgentsClient();
AgentDefinition definition
= new PromptAgentDefinition("gpt-4o").setInstructions("You are a helpful assistant.");
AgentVersionDetails agent = agentsClient.createAgentVersion("my-agent", definition);
System.out.printf("Created agent %s (traced).%n", agent.getName());
See TracingSample.java for a runnable example.
To inspect exported traces in the Foundry tracing UI, connect the Foundry project to an Application Insights resource
and configure an Azure Monitor OpenTelemetry exporter. See
TracingAzureMonitorSample.java for the exporter
setup.
Troubleshooting
Enable client logging
You can set the AZURE_LOG_LEVEL environment variable to view logging statements made in the client library. For
example, setting AZURE_LOG_LEVEL=2 would show all informational, warning, and error log messages. The log levels can
be found here: log levels.
To log full HTTP request and response bodies (including headers), set:
export AZURE_LOG_LEVEL=verbose
export AZURE_HTTP_LOG_DETAIL_LEVEL=body_and_headers
Default HTTP Client
All client libraries by default use the Netty HTTP client. Configuring or changing the HTTP client is detailed in the HTTP clients wiki.
Default SSL library
All client libraries, by default, use the Tomcat-native Boring SSL library to enable native-level performance for SSL operations. The Boring SSL library is an uber jar containing native libraries for Linux / macOS / Windows, and provides better performance compared to the default SSL implementation within the JDK. For more information, including how to reduce the dependency size, refer to the performance tuning section of the wiki.
Next steps
Contributing
For details on contributing to this repository, see the contributing guide.
- Fork it
- Create your feature branch (
git checkout -b my-new-feature) - Commit your changes (
git commit -am 'Add some feature') - Push to the branch (
git push origin my-new-feature) - Create new Pull Request