Tutorial: Build a multi-turn chat assistant with Foundry Local

Samples repository

You can find the complete sample code for this article in the Foundry samples GitHub repository. To clone the repository and navigate to the sample, use:

git clone https://github.com/microsoft-foundry/foundry-samples.git
cd foundry-samples/samples/csharp/foundry-local/tutorial-chat-assistant

Install packages

dotnet add package Microsoft.AI.Foundry.Local
dotnet add package OpenAI

The C# samples in the GitHub repository are preconfigured projects. If you're building from scratch, you should read the Foundry Local SDK reference for more details on how to set up your C# project with Foundry Local.

Browse the catalog and select a model

The Foundry Local SDK provides a model catalog that lists all available models. In this step, you initialize the SDK and select a model for your chat assistant.

• Open Program.cs and replace its contents with the following code to initialize the SDK and select a model:

  CancellationToken ct = CancellationToken.None;
  
  var config = new Configuration
  {
      AppName = "foundry_local_samples",
      LogLevel = Microsoft.AI.Foundry.Local.LogLevel.Information
  };
  
  using var loggerFactory = LoggerFactory.Create(builder =>
  {
      builder.SetMinimumLevel(Microsoft.Extensions.Logging.LogLevel.Information);
  });
  var logger = loggerFactory.CreateLogger<Program>();
  
  // Initialize the singleton instance
  await FoundryLocalManager.CreateAsync(config, logger);
  var mgr = FoundryLocalManager.Instance;
  
  // Download and register all execution providers.
  var currentEp = "";
  await mgr.DownloadAndRegisterEpsAsync((epName, percent) =>
  {
      if (epName != currentEp)
      {
          if (currentEp != "") Console.WriteLine();
          currentEp = epName;
      }
      Console.Write($"\r  {epName.PadRight(30)}  {percent,6:F1}%");
  });
  if (currentEp != "") Console.WriteLine();
  
  // Select and load a model from the catalog
  var catalog = await mgr.GetCatalogAsync();
  var model = await catalog.GetModelAsync("qwen2.5-0.5b")
      ?? throw new Exception("Model not found");
  
  await model.DownloadAsync(progress =>
  {
      Console.Write($"\rDownloading model: {progress:F2}%");
      if (progress >= 100f) Console.WriteLine();
  });
  
  await model.LoadAsync();
  Console.WriteLine("Model loaded and ready.");
  
  // Create one stateful native chat session for the conversation.
  using var chatSession = new ChatSession(model);
  chatSession.SetStreaming(true);
  

  The GetModelAsync method accepts a model alias, which is a short friendly name that maps to a specific model in the catalog. The DownloadAsync method fetches the model weights to your local cache, and LoadAsync makes the model ready for inference.

Define a system prompt

A system prompt sets the assistant's personality and behavior. It's the first message in the conversation history and the model references it throughout the conversation.

Add a system prompt to shape how the assistant responds:

// Start the conversation with a system prompt
var systemPrompt = "You are a helpful, friendly assistant. Keep your responses " +
                   "concise and conversational. If you don't know something, say so.";

Implement multi-turn conversation

A chat assistant needs to maintain context across multiple exchanges. You achieve this by keeping a list of all messages (system, user, and assistant) and sending the full list with each request. The model uses this history to generate contextually relevant responses.

Add a conversation loop that:

• Reads user input from the console.
• Appends the user message to the history.
• Sends the complete history to the model.
• Appends the assistant's response to the history for the next turn.

while (true)
{
    Console.Write("You: ");
    var userInput = Console.ReadLine();
    if (string.IsNullOrWhiteSpace(userInput) ||
        userInput.Equals("quit", StringComparison.OrdinalIgnoreCase) ||
        userInput.Equals("exit", StringComparison.OrdinalIgnoreCase))
    {
        break;
    }

    using var request = new Request();
    if (chatSession.TurnCount == 0)
    {
        request.AddItem(MessageItem.System(systemPrompt));
    }
    request.AddItem(MessageItem.User(userInput));

    // Stream the response token by token
    Console.Write("Assistant: ");
    await using var streamingResponse = chatSession.ProcessStreamingRequestAsync(request, ct);
    await foreach (var item in streamingResponse)
    {
        using (item)
        {
            if (item is TextItem { Type: TextItemType.Default } text)
            {
                Console.Write(text.Text);
                Console.Out.Flush();
            }
        }
    }
    using var finalResponse = await streamingResponse.FinalResponse;
    Console.WriteLine("\n");
}

Each call to CompleteChatAsync receives the full message history. This is how the model "remembers" previous turns — it doesn't store state between calls.

Add streaming responses

Streaming prints each token as it's generated, which makes the assistant feel more responsive. Replace the CompleteChatAsync call with CompleteChatStreamingAsync to stream the response token by token.

Update the conversation loop to use streaming:

// Stream the response token by token
Console.Write("Assistant: ");
await using var streamingResponse = chatSession.ProcessStreamingRequestAsync(request, ct);
await foreach (var item in streamingResponse)
{
    using (item)
    {
        if (item is TextItem { Type: TextItemType.Default } text)
        {
            Console.Write(text.Text);
            Console.Out.Flush();
        }
    }
}
using var finalResponse = await streamingResponse.FinalResponse;
Console.WriteLine("\n");

The streaming version accumulates the full response so it can be added to the conversation history after the stream completes.

Complete code

Replace the contents of Program.cs with the following complete code:

using Microsoft.AI.Foundry.Local;
using Microsoft.Extensions.Logging;

CancellationToken ct = CancellationToken.None;

var config = new Configuration
{
    AppName = "foundry_local_samples",
    LogLevel = Microsoft.AI.Foundry.Local.LogLevel.Information
};

using var loggerFactory = LoggerFactory.Create(builder =>
{
    builder.SetMinimumLevel(Microsoft.Extensions.Logging.LogLevel.Information);
});
var logger = loggerFactory.CreateLogger<Program>();

// Initialize the singleton instance
await FoundryLocalManager.CreateAsync(config, logger);
var mgr = FoundryLocalManager.Instance;

// Download and register all execution providers.
var currentEp = "";
await mgr.DownloadAndRegisterEpsAsync((epName, percent) =>
{
    if (epName != currentEp)
    {
        if (currentEp != "") Console.WriteLine();
        currentEp = epName;
    }
    Console.Write($"\r  {epName.PadRight(30)}  {percent,6:F1}%");
});
if (currentEp != "") Console.WriteLine();

// Select and load a model from the catalog
var catalog = await mgr.GetCatalogAsync();
var model = await catalog.GetModelAsync("qwen2.5-0.5b")
    ?? throw new Exception("Model not found");

await model.DownloadAsync(progress =>
{
    Console.Write($"\rDownloading model: {progress:F2}%");
    if (progress >= 100f) Console.WriteLine();
});

await model.LoadAsync();
Console.WriteLine("Model loaded and ready.");

// Create one stateful native chat session for the conversation.
using var chatSession = new ChatSession(model);
chatSession.SetStreaming(true);

// Start the conversation with a system prompt
var systemPrompt = "You are a helpful, friendly assistant. Keep your responses " +
                   "concise and conversational. If you don't know something, say so.";

Console.WriteLine("\nChat assistant ready! Type 'quit' to exit.\n");

while (true)
{
    Console.Write("You: ");
    var userInput = Console.ReadLine();
    if (string.IsNullOrWhiteSpace(userInput) ||
        userInput.Equals("quit", StringComparison.OrdinalIgnoreCase) ||
        userInput.Equals("exit", StringComparison.OrdinalIgnoreCase))
    {
        break;
    }

    using var request = new Request();
    if (chatSession.TurnCount == 0)
    {
        request.AddItem(MessageItem.System(systemPrompt));
    }
    request.AddItem(MessageItem.User(userInput));

    // Stream the response token by token
    Console.Write("Assistant: ");
    await using var streamingResponse = chatSession.ProcessStreamingRequestAsync(request, ct);
    await foreach (var item in streamingResponse)
    {
        using (item)
        {
            if (item is TextItem { Type: TextItemType.Default } text)
            {
                Console.Write(text.Text);
                Console.Out.Flush();
            }
        }
    }
    using var finalResponse = await streamingResponse.FinalResponse;
    Console.WriteLine("\n");
}

// Clean up - unload the model
chatSession.Dispose();
await model.UnloadAsync();
Console.WriteLine("Model unloaded. Goodbye!");

Run the chat assistant:

dotnet run

You see output similar to:

Downloading model: 100.00%
Model loaded and ready.

Chat assistant ready! Type 'quit' to exit.

You: What is photosynthesis?
Assistant: Photosynthesis is the process plants use to convert sunlight, water, and carbon
dioxide into glucose and oxygen. It mainly happens in the leaves, inside structures
called chloroplasts.

You: Why is it important for other living things?
Assistant: It's essential because photosynthesis produces the oxygen that most living things
breathe. It also forms the base of the food chain — animals eat plants or eat other
animals that depend on plants for energy.

You: quit
Model unloaded. Goodbye!

Notice how the assistant remembers context from previous turns — when you ask "Why is it important for other living things?", it knows you're still talking about photosynthesis.

Samples repository

The complete sample code for this article is available in the foundry-samples GitHub repository. To clone the repository and navigate to the sample use:

git clone https://github.com/microsoft-foundry/foundry-samples.git
cd foundry-samples/samples/javascript/foundry-local/tutorial-chat-assistant

Install packages

npm install foundry-local-sdk openai

Browse the catalog and select a model

The Foundry Local SDK provides a model catalog that lists all available models. In this step, you initialize the SDK and select a model for your chat assistant.

1. Create a file called index.js.

2. Add the following code to initialize the SDK and select a model:

   // Initialize the Foundry Local SDK
   const manager = FoundryLocalManager.create({
       appName: 'foundry_local_samples',
       logLevel: 'info'
   });
   
   // Download and register all execution providers.
   let currentEp = '';
   await manager.downloadAndRegisterEps((epName, percent) => {
       if (epName !== currentEp) {
           if (currentEp !== '') process.stdout.write('\n');
           currentEp = epName;
       }
       process.stdout.write(`\r  ${epName.padEnd(30)}  ${percent.toFixed(1).padStart(5)}%`);
   });
   if (currentEp !== '') process.stdout.write('\n');
   
   // Select and load a model from the catalog
   const model = await manager.catalog.getModel('qwen2.5-0.5b');
   
   await model.download((progress) => {
       process.stdout.write(`\rDownloading model: ${progress.toFixed(2)}%`);
   });
   console.log('\nModel downloaded.');
   
   await model.load();
   console.log('Model loaded and ready.');
   
   // Create a stateful native chat session
   const chatSession = new ChatSession(model);
   chatSession.setOptions({
       search: { temperature: 0.7, maxOutputTokens: 512 }
   });
   

   The getModel method accepts a model alias, which is a short friendly name that maps to a specific model in the catalog. The download method fetches the model weights to your local cache, and load makes the model ready for inference.

Define a system prompt

A system prompt sets the assistant's personality and behavior. It's the first message in the conversation history and the model references it throughout the conversation.

Add a system prompt to shape how the assistant responds:

// Start the conversation with a system prompt
const systemPrompt =
    'You are a helpful, friendly assistant. Keep your responses ' +
    'concise and conversational. If you don\'t know something, say so.';
let firstTurn = true;

Implement multi-turn conversation

A chat assistant needs to maintain context across multiple exchanges. You achieve this by keeping a list of all messages (system, user, and assistant) and sending the full list with each request. The model uses this history to generate contextually relevant responses.

Add a conversation loop that:

• Reads user input from the console.
• Appends the user message to the history.
• Sends the complete history to the model.
• Appends the assistant's response to the history for the next turn.

while (true) {
    const userInput = await askQuestion('You: ');
    if (userInput.trim().toLowerCase() === 'quit' ||
        userInput.trim().toLowerCase() === 'exit') {
        break;
    }

    const request = new Request();
    if (firstTurn) {
        request.addItem(Item.systemMessage(systemPrompt));
        firstTurn = false;
    }
    request.addItem(Item.userMessage(userInput));

    // Stream the response token by token
    process.stdout.write('Assistant: ');
    const stream = chatSession.processStreamingRequest(request);
    for await (const item of stream) {
        if (item.type === 'text' && item.textType === 'default') {
            process.stdout.write(item.text);
        }
    }
    await stream.response;
    console.log('\n');
}

Each call to completeChat receives the full message history. This is how the model "remembers" previous turns — it doesn't store state between calls.

Add streaming responses

Streaming prints each token as it's generated, which makes the assistant feel more responsive. Replace the completeChat call with completeStreamingChat to stream the response token by token.

Update the conversation loop to use streaming:

// Stream the response token by token
process.stdout.write('Assistant: ');
const stream = chatSession.processStreamingRequest(request);
for await (const item of stream) {
    if (item.type === 'text' && item.textType === 'default') {
        process.stdout.write(item.text);
    }
}
await stream.response;
console.log('\n');

The streaming version accumulates the full response so it can be added to the conversation history after the stream completes.

Complete code

Create a file named index.js and add the following complete code:

import {
    ChatSession,
    FoundryLocalManager,
    Item,
    Request
} from 'foundry-local-sdk';
import * as readline from 'readline';

// Initialize the Foundry Local SDK
const manager = FoundryLocalManager.create({
    appName: 'foundry_local_samples',
    logLevel: 'info'
});

// Download and register all execution providers.
let currentEp = '';
await manager.downloadAndRegisterEps((epName, percent) => {
    if (epName !== currentEp) {
        if (currentEp !== '') process.stdout.write('\n');
        currentEp = epName;
    }
    process.stdout.write(`\r  ${epName.padEnd(30)}  ${percent.toFixed(1).padStart(5)}%`);
});
if (currentEp !== '') process.stdout.write('\n');

// Select and load a model from the catalog
const model = await manager.catalog.getModel('qwen2.5-0.5b');

await model.download((progress) => {
    process.stdout.write(`\rDownloading model: ${progress.toFixed(2)}%`);
});
console.log('\nModel downloaded.');

await model.load();
console.log('Model loaded and ready.');

// Create a stateful native chat session
const chatSession = new ChatSession(model);
chatSession.setOptions({
    search: { temperature: 0.7, maxOutputTokens: 512 }
});

// Start the conversation with a system prompt
const systemPrompt =
    'You are a helpful, friendly assistant. Keep your responses ' +
    'concise and conversational. If you don\'t know something, say so.';
let firstTurn = true;

// Set up readline for console input
const rl = readline.createInterface({
    input: process.stdin,
    output: process.stdout
});

const askQuestion = (prompt) => new Promise((resolve) => {
    if (rl.closed) return resolve('quit');
    const onClose = () => resolve('quit');
    rl.once('close', onClose);
    try {
        rl.question(prompt, (answer) => {
            rl.off('close', onClose);
            resolve(answer);
        });
    } catch {
        rl.off('close', onClose);
        resolve('quit');
    }
});

console.log('\nChat assistant ready! Type \'quit\' to exit.\n');

while (true) {
    const userInput = await askQuestion('You: ');
    if (userInput.trim().toLowerCase() === 'quit' ||
        userInput.trim().toLowerCase() === 'exit') {
        break;
    }

    const request = new Request();
    if (firstTurn) {
        request.addItem(Item.systemMessage(systemPrompt));
        firstTurn = false;
    }
    request.addItem(Item.userMessage(userInput));

    // Stream the response token by token
    process.stdout.write('Assistant: ');
    const stream = chatSession.processStreamingRequest(request);
    for await (const item of stream) {
        if (item.type === 'text' && item.textType === 'default') {
            process.stdout.write(item.text);
        }
    }
    await stream.response;
    console.log('\n');
}

// Clean up - unload the model
chatSession.dispose();
await model.unload();
console.log('Model unloaded. Goodbye!');
rl.close();

Run the chat assistant:

node index.js

You see output similar to:

Downloading model: 100.00%
Model downloaded.
Model loaded and ready.

Chat assistant ready! Type 'quit' to exit.

You: What is photosynthesis?
Assistant: Photosynthesis is the process plants use to convert sunlight, water, and carbon
dioxide into glucose and oxygen. It mainly happens in the leaves, inside structures
called chloroplasts.

You: Why is it important for other living things?
Assistant: It's essential because photosynthesis produces the oxygen that most living things
breathe. It also forms the base of the food chain — animals eat plants or eat other
animals that depend on plants for energy.

You: quit
Model unloaded. Goodbye!

Notice how the assistant remembers context from previous turns — when you ask "Why is it important for other living things?", it knows you're still talking about photosynthesis.

Samples repository

The complete sample code for this article is available in the foundry-samples GitHub repository. To clone the repository and navigate to the sample use:

git clone https://github.com/microsoft-foundry/foundry-samples.git
cd foundry-samples/samples/python/foundry-local/tutorial-chat-assistant

Install packages

pip install foundry-local-sdk openai

Browse the catalog and select a model

The Foundry Local SDK provides a model catalog that lists all available models. In this step, you initialize the SDK and select a model for your chat assistant.

1. Create a file called main.py.

2. Add the following code to initialize the SDK and select a model:

   # Initialize the Foundry Local SDK
   config = Configuration(app_name="foundry_local_samples")
   FoundryLocalManager.initialize(config)
   manager = FoundryLocalManager.instance
   
   # Discover and register execution providers that are not already available.
   missing_eps = [ep.name for ep in manager.discover_eps() if not ep.is_registered]
   if missing_eps:
       manager.download_and_register_eps(missing_eps)
   
   # Select and load a model from the catalog.
   model = manager.catalog.get_model("qwen2.5-0.5b")
   if model is None:
       raise RuntimeError("Model 'qwen2.5-0.5b' was not found in the catalog.")
   if not model.is_cached:
       model.download(
           lambda progress: print(
               f"\rDownloading model: {progress:.2f}%", end="", flush=True
           )
       )
       print()
   model.load()
   print("Model loaded and ready.")
   

   The get_model method accepts a model alias, which is a short friendly name that maps to a specific model in the catalog. The download method fetches the model weights to your local cache, and load makes the model ready for inference.

Define a system prompt

A system prompt sets the assistant's personality and behavior. It's the first message in the conversation history and the model references it throughout the conversation.

Add a system prompt to shape how the assistant responds:

# Start the conversation with a system prompt.
system_prompt = (
    "You are a helpful, friendly assistant. Keep your responses "
    "concise and conversational. If you don't know something, say so."
)

Implement multi-turn conversation

A chat assistant needs to maintain context across multiple exchanges. You achieve this by keeping a list of all messages (system, user, and assistant) and sending the full list with each request. The model uses this history to generate contextually relevant responses.

Add a conversation loop that:

• Reads user input from the console.
• Appends the user message to the history.
• Sends the complete history to the model.
• Appends the assistant's response to the history for the next turn.

while True:
    user_input = input("You: ")
    if user_input.strip().lower() in ("quit", "exit"):
        break

    # The stateful session retains earlier turns automatically.
    with Request() as request:
        if first_turn:
            request.add_item(MessageItem.system(system_prompt))
            first_turn = False
        request.add_item(MessageItem.user(user_input))
        print("Assistant: ", end="", flush=True)
        with session.process_streaming_request(request) as stream:
            for item in stream:
                if (
                    isinstance(item, TextItem)
                    and item.type == TextItemType.DEFAULT
                ):
                    print(item.text, end="", flush=True)
        print("\n")

Each call to complete_chat receives the full message history. This is how the model "remembers" previous turns — it doesn't store state between calls.

Add streaming responses

Streaming prints each token as it's generated, which makes the assistant feel more responsive. Replace the complete_chat call with complete_streaming_chat to stream the response token by token.

Update the conversation loop to use streaming:

# The stateful session retains earlier turns automatically.
with Request() as request:
    if first_turn:
        request.add_item(MessageItem.system(system_prompt))
        first_turn = False
    request.add_item(MessageItem.user(user_input))
    print("Assistant: ", end="", flush=True)
    with session.process_streaming_request(request) as stream:
        for item in stream:
            if (
                isinstance(item, TextItem)
                and item.type == TextItemType.DEFAULT
            ):
                print(item.text, end="", flush=True)
    print("\n")

The streaming version accumulates the full response so it can be added to the conversation history after the stream completes.

Complete code

Create a file named main.py and add the following complete code:

from foundry_local_sdk import (
    ChatSession,
    Configuration,
    FoundryLocalManager,
    MessageItem,
    Request,
    RequestOptions,
    SearchOptions,
    TextItem,
    TextItemType,
)



def main():
    # Initialize the Foundry Local SDK
    config = Configuration(app_name="foundry_local_samples")
    FoundryLocalManager.initialize(config)
    manager = FoundryLocalManager.instance

    # Discover and register execution providers that are not already available.
    missing_eps = [ep.name for ep in manager.discover_eps() if not ep.is_registered]
    if missing_eps:
        manager.download_and_register_eps(missing_eps)

    # Select and load a model from the catalog.
    model = manager.catalog.get_model("qwen2.5-0.5b")
    if model is None:
        raise RuntimeError("Model 'qwen2.5-0.5b' was not found in the catalog.")
    if not model.is_cached:
        model.download(
            lambda progress: print(
                f"\rDownloading model: {progress:.2f}%", end="", flush=True
            )
        )
        print()
    model.load()
    print("Model loaded and ready.")

    try:
        # Start the conversation with a system prompt.
        system_prompt = (
            "You are a helpful, friendly assistant. Keep your responses "
            "concise and conversational. If you don't know something, say so."
        )

        print("\nChat assistant ready! Type 'quit' to exit.\n")

        with ChatSession(model) as session:
            session.set_options(
                RequestOptions(
                    search=SearchOptions(temperature=0.7, max_output_tokens=512)
                )
            )
            session.set_streaming(True)
            first_turn = True

            while True:
                user_input = input("You: ")
                if user_input.strip().lower() in ("quit", "exit"):
                    break

                # The stateful session retains earlier turns automatically.
                with Request() as request:
                    if first_turn:
                        request.add_item(MessageItem.system(system_prompt))
                        first_turn = False
                    request.add_item(MessageItem.user(user_input))
                    print("Assistant: ", end="", flush=True)
                    with session.process_streaming_request(request) as stream:
                        for item in stream:
                            if (
                                isinstance(item, TextItem)
                                and item.type == TextItemType.DEFAULT
                            ):
                                print(item.text, end="", flush=True)
                    print("\n")
    finally:
        # Clean up - unload the model
        if model.is_loaded:
            model.unload()
        manager.close()
        print("Model unloaded. Goodbye!")


if __name__ == "__main__":
    main()

Run the chat assistant:

python main.py

You see output similar to:

Downloading model: 100.00%
Model loaded and ready.

Chat assistant ready! Type 'quit' to exit.

You: What is photosynthesis?
Assistant: Photosynthesis is the process plants use to convert sunlight, water, and carbon
dioxide into glucose and oxygen. It mainly happens in the leaves, inside structures
called chloroplasts.

You: Why is it important for other living things?
Assistant: It's essential because photosynthesis produces the oxygen that most living things
breathe. It also forms the base of the food chain — animals eat plants or eat other
animals that depend on plants for energy.

You: quit
Model unloaded. Goodbye!

Notice how the assistant remembers context from previous turns — when you ask "Why is it important for other living things?", it knows you're still talking about photosynthesis.

Samples repository

The complete sample code for this article is available in the foundry-samples GitHub repository. To clone the repository and navigate to the sample use:

git clone https://github.com/microsoft-foundry/foundry-samples.git
cd foundry-samples/samples/rust/foundry-local/tutorial-chat-assistant

Install packages

cargo add foundry-local-sdk
cargo add tokio --features full
cargo add tokio-stream anyhow

Browse the catalog and select a model

The Foundry Local SDK provides a model catalog that lists all available models. In this step, you initialize the SDK and select a model for your chat assistant.

• Open src/main.rs and replace its contents with the following code to initialize the SDK and select a model:

  // Initialize the Foundry Local SDK
  let manager = FoundryLocalManager::create(FoundryLocalConfig::new("foundry_local_samples"))?;
  
  manager
      .download_and_register_eps_with_progress(None, {
          let mut current_ep = String::new();
          move |ep_name: &str, percent: f64| {
              if ep_name != current_ep {
                  if !current_ep.is_empty() {
                      println!();
                  }
                  current_ep = ep_name.to_string();
              }
              print!("\r  {ep_name:<30}  {percent:5.1}%");
              io::stdout().flush().ok();
          }
      })
      .await?;
  println!();
  
  let model = manager.catalog().get_model("qwen2.5-0.5b").await?;
  if !model.is_cached().await? {
      println!("Downloading model...");
      model
          .download(Some(|progress: f64| {
              print!("\r  {progress:.1}%");
              io::stdout().flush().ok();
          }))
          .await?;
      println!();
  }
  
  model.load().await?;
  let session = ChatSession::new(&model).await?;
  session
      .set_options(RequestOptions {
          search: SearchOptions {
              temperature: Some(0.7),
              max_output_tokens: Some(512),
              ..Default::default()
          },
          ..Default::default()
      })
      .await?;
  println!("Model loaded and ready.");
  

  The get_model method accepts a model alias, which is a short friendly name that maps to a specific model in the catalog. The download method fetches the model weights to your local cache, and load makes the model ready for inference.

Define a system prompt

A system prompt sets the assistant's personality and behavior. It's the first message in the conversation history and the model references it throughout the conversation.

Add a system prompt to shape how the assistant responds:

let system_prompt = "You are a helpful, friendly assistant. Keep your responses concise and \
                     conversational. If you don't know something, say so.";

Implement multi-turn conversation

A chat assistant needs to maintain context across multiple exchanges. You achieve this by keeping a vector of all messages (system, user, and assistant) and sending the full list with each request. The model uses this history to generate contextually relevant responses.

Add a conversation loop that:

• Reads user input from the console.
• Appends the user message to the history.
• Sends the complete history to the model.
• Appends the assistant's response to the history for the next turn.

loop {
    print!("You: ");
    io::stdout().flush()?;

    let mut input = String::new();
    stdin.lock().read_line(&mut input)?;
    let input = input.trim();

    if input.eq_ignore_ascii_case("quit") || input.eq_ignore_ascii_case("exit") {
        break;
    }
    if input.is_empty() {
        continue;
    }

    let mut items = Vec::new();
    if session.turn_count() == 0 {
        items.push(Item::system_message(vec![Item::text(system_prompt)]));
    }
    items.push(Item::user_message(vec![Item::text(input)]));

    print!("Assistant: ");
    io::stdout().flush()?;
    let mut stream = session.process_streaming_request(Request::from_items(items));
    while let Some(item) = stream.next().await {
        if let Item::Text {
            text,
            kind: TextKind::Default,
        } = item?
        {
            print!("{text}");
            io::stdout().flush()?;
        }
    }
    println!("\n");
}

Each call to complete_chat receives the full message history. This is how the model "remembers" previous turns — it doesn't store state between calls.

Add streaming responses

Streaming prints each token as it's generated, which makes the assistant feel more responsive. Replace the complete_chat call with complete_streaming_chat to stream the response token by token.

Update the conversation loop to use streaming:

print!("Assistant: ");
io::stdout().flush()?;
let mut stream = session.process_streaming_request(Request::from_items(items));
while let Some(item) = stream.next().await {
    if let Item::Text {
        text,
        kind: TextKind::Default,
    } = item?
    {
        print!("{text}");
        io::stdout().flush()?;
    }
}
println!("\n");

The streaming version accumulates the full response so it can be added to the conversation history after the stream completes.

Complete code

Replace the contents of src/main.rs with the following complete code:

use foundry_local_sdk::{
    ChatSession, FoundryLocalConfig, FoundryLocalManager, Item, Request, RequestOptions,
    SearchOptions, TextKind,
};
use std::io::{self, BufRead, Write};
use tokio_stream::StreamExt;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Initialize the Foundry Local SDK
    let manager = FoundryLocalManager::create(FoundryLocalConfig::new("foundry_local_samples"))?;

    manager
        .download_and_register_eps_with_progress(None, {
            let mut current_ep = String::new();
            move |ep_name: &str, percent: f64| {
                if ep_name != current_ep {
                    if !current_ep.is_empty() {
                        println!();
                    }
                    current_ep = ep_name.to_string();
                }
                print!("\r  {ep_name:<30}  {percent:5.1}%");
                io::stdout().flush().ok();
            }
        })
        .await?;
    println!();

    let model = manager.catalog().get_model("qwen2.5-0.5b").await?;
    if !model.is_cached().await? {
        println!("Downloading model...");
        model
            .download(Some(|progress: f64| {
                print!("\r  {progress:.1}%");
                io::stdout().flush().ok();
            }))
            .await?;
        println!();
    }

    model.load().await?;
    let session = ChatSession::new(&model).await?;
    session
        .set_options(RequestOptions {
            search: SearchOptions {
                temperature: Some(0.7),
                max_output_tokens: Some(512),
                ..Default::default()
            },
            ..Default::default()
        })
        .await?;
    println!("Model loaded and ready.");

    let system_prompt = "You are a helpful, friendly assistant. Keep your responses concise and \
                         conversational. If you don't know something, say so.";

    println!("\nChat assistant ready! Type 'quit' to exit.\n");

    let stdin = io::stdin();
    loop {
        print!("You: ");
        io::stdout().flush()?;

        let mut input = String::new();
        stdin.lock().read_line(&mut input)?;
        let input = input.trim();

        if input.eq_ignore_ascii_case("quit") || input.eq_ignore_ascii_case("exit") {
            break;
        }
        if input.is_empty() {
            continue;
        }

        let mut items = Vec::new();
        if session.turn_count() == 0 {
            items.push(Item::system_message(vec![Item::text(system_prompt)]));
        }
        items.push(Item::user_message(vec![Item::text(input)]));

        print!("Assistant: ");
        io::stdout().flush()?;
        let mut stream = session.process_streaming_request(Request::from_items(items));
        while let Some(item) = stream.next().await {
            if let Item::Text {
                text,
                kind: TextKind::Default,
            } = item?
            {
                print!("{text}");
                io::stdout().flush()?;
            }
        }
        println!("\n");
    }

    drop(session);
    model.unload().await?;
    println!("Model unloaded. Goodbye!");

    Ok(())
}

Run the chat assistant:

cargo run

You see output similar to:

Downloading model: 100.00%
Model loaded and ready.

Chat assistant ready! Type 'quit' to exit.

You: What is photosynthesis?
Assistant: Photosynthesis is the process plants use to convert sunlight, water, and carbon
dioxide into glucose and oxygen. It mainly happens in the leaves, inside structures
called chloroplasts.

You: Why is it important for other living things?
Assistant: It's essential because photosynthesis produces the oxygen that most living things
breathe. It also forms the base of the food chain — animals eat plants or eat other
animals that depend on plants for energy.

You: quit
Model unloaded. Goodbye!

Notice how the assistant remembers context from previous turns — when you ask "Why is it important for other living things?", it knows you're still talking about photosynthesis.

Samples repository

Clone the Foundry samples GitHub repository and go to the C++ sample:

git clone https://github.com/microsoft-foundry/foundry-samples.git
cd foundry-samples/samples/cpp/foundry-local/tutorial-chat-assistant

Install packages

The C++ samples require:

• CMake 3.20 or later.
• A C++20 compiler.
• The NuGet CLI on your PATH.

Configure and build the sample:

cmake -S . -B build
cmake --build build --config Release

During configuration, the sample restores version 2.2.1 of the Microsoft.AI.Foundry.Local.Runtime NuGet package to build/_packages. The package contains the C++ headers and native libraries for Windows, Linux, and macOS. You don't need a separate WinML package.

The CMake configuration selects the runtime assets for your operating system and architecture and copies the required native libraries beside the executable. On macOS and Linux, run the executable from build. On Windows, run the .exe file from build.

Browse the catalog and select a model

Initialize the SDK, select a model from the catalog, and download and load it:

Configuration config("foundry_local_cpp_tutorial_chat");
Manager manager(std::move(config));
auto model = SelectChatModel(manager.GetCatalog());
if (!model) {
  throw std::runtime_error("No supported CPU chat model was found.");
}
if (!model->IsCached()) {
  model->Download([](float progress) {
    std::cout << "\rDownloading model: " << static_cast<int>(progress) << "%" << std::flush;
    return 0;
  });
  std::cout << "\n";
}
model->Load();

Define a system prompt

Add a system prompt to shape how the assistant responds:

Request system_prompt{
    SystemMessage(
        "You are a concise, friendly local assistant. State when you are unsure."),
};
session.ProcessRequest(system_prompt);

Implement multi-turn conversation

Maintain the conversation history and send it with each request:

std::cout << "Enter a message, or type 'exit'.\n";
for (std::string input; std::cout << "You: " && std::getline(std::cin, input);) {
  if (input == "exit" || input == "quit") {
    break;
  }
  if (input.empty()) {
    continue;
  }

  EnableStreaming(session);
  Request request{UserMessage(input)};
  std::cout << "Assistant: ";
  session.ProcessRequest(request);
  std::cout << "\n";
}

Add streaming responses

Stream response text as the model generates it:

void EnableStreaming(ChatSession& session) {
  session.SetStreamingCallback([](Item item) {
    if (item.GetType() == FOUNDRY_LOCAL_ITEM_TEXT &&
        item.GetText().type == FOUNDRY_LOCAL_TEXT_ITEM_TYPE_DEFAULT) {
      std::cout << item.GetText().text << std::flush;
    }
    return 0;
  });
}

Complete code

#include <foundry_local/foundry_local_cpp.h>

#include <iostream>
#include <memory>
#include <string>

using namespace foundry_local;

std::unique_ptr<IModel> SelectChatModel(ICatalog& catalog) {
  for (const std::string alias : {"qwen2.5-0.5b", "qwen3.5-0.8b"}) {
    auto model = catalog.GetModel(alias);
    if (!model) {
      continue;
    }
    for (const auto& variant : model->GetVariants()) {
      const auto info = variant->GetInfo();
      if (info.DeviceType() == FOUNDRY_LOCAL_DEVICE_CPU &&
          (info.Task() == "chat-completion" || info.Task() == "coding")) {
        model->SelectVariant(*variant);
        return model;
      }
    }
  }
  return nullptr;
}

void EnableStreaming(ChatSession& session);

int main() {
  try {
    Configuration config("foundry_local_cpp_tutorial_chat");
    Manager manager(std::move(config));
    auto model = SelectChatModel(manager.GetCatalog());
    if (!model) {
      throw std::runtime_error("No supported CPU chat model was found.");
    }
    if (!model->IsCached()) {
      model->Download([](float progress) {
        std::cout << "\rDownloading model: " << static_cast<int>(progress) << "%" << std::flush;
        return 0;
      });
      std::cout << "\n";
    }
    model->Load();

    {
      ChatSession session(*model);
      session.SetOptions({
          .search = {.temperature = 0.2F, .max_output_tokens = 512, .seed = 42},
      });

      Request system_prompt{
          SystemMessage(
              "You are a concise, friendly local assistant. State when you are unsure."),
      };
      session.ProcessRequest(system_prompt);

      std::cout << "Enter a message, or type 'exit'.\n";
      for (std::string input; std::cout << "You: " && std::getline(std::cin, input);) {
        if (input == "exit" || input == "quit") {
          break;
        }
        if (input.empty()) {
          continue;
        }

        EnableStreaming(session);
        Request request{UserMessage(input)};
        std::cout << "Assistant: ";
        session.ProcessRequest(request);
        std::cout << "\n";
      }
    }

    model->Unload();
    manager.Shutdown();
  } catch (const Error& error) {
    std::cerr << "Foundry Local error [" << error.Code() << "]: " << error.what() << "\n";
    return 1;
  } catch (const std::exception& error) {
    std::cerr << "Error: " << error.what() << "\n";
    return 1;
  }

  return 0;
}

void EnableStreaming(ChatSession& session) {
  session.SetStreamingCallback([](Item item) {
    if (item.GetType() == FOUNDRY_LOCAL_ITEM_TEXT &&
        item.GetText().type == FOUNDRY_LOCAL_TEXT_ITEM_TYPE_DEFAULT) {
      std::cout << item.GetText().text << std::flush;
    }
    return 0;
  });
}

Run the chat assistant:

./build/tutorial-chat-assistant