Get started with Foundry Local

In this quickstart, you create a console application that downloads a local AI model, generates a streaming chat response, and unloads the model. Everything runs on your device with no cloud dependency or Azure subscription.

Prerequisites

Samples repository

You can find the complete sample code for this article in the Foundry samples GitHub repository. To clone the repository and navigate to the sample, use:

git clone https://github.com/microsoft-foundry/foundry-samples.git
cd foundry-samples/samples/csharp/foundry-local/native-chat-completions

Install packages

If you're developing or shipping on Windows, select the Windows tab. The Windows package integrates with the Windows ML runtime — it provides the same API surface area with a wider breadth of hardware acceleration.

dotnet add package Microsoft.AI.Foundry.Local.WinML
dotnet add package OpenAI

The C# samples in the GitHub repository are preconfigured projects. If you're building from scratch, you should read the Foundry Local SDK reference for more details on how to set up your C# project with Foundry Local.

Use native chat completions API

Copy and paste the following code into a C# file named Program.cs:

using Microsoft.AI.Foundry.Local;

CancellationToken ct = new CancellationToken();

var config = new Configuration
{
    AppName = "foundry_local_samples",
    LogLevel = Microsoft.AI.Foundry.Local.LogLevel.Information
};


// Initialize the singleton instance.
await FoundryLocalManager.CreateAsync(config, Utils.GetAppLogger());
var mgr = FoundryLocalManager.Instance;


// Discover available execution providers and their registration status.
var eps = mgr.DiscoverEps();
int maxNameLen = 30;
Console.WriteLine("Available execution providers:");
Console.WriteLine($"  {"Name".PadRight(maxNameLen)}  Registered");
Console.WriteLine($"  {new string('─', maxNameLen)}  {"──────────"}");
foreach (var ep in eps)
{
    Console.WriteLine($"  {ep.Name.PadRight(maxNameLen)}  {ep.IsRegistered}");
}

// Download and register all execution providers with per-EP progress.
// EP packages include dependencies and may be large.
// Download is only required again if a new version of the EP is released.
// For cross platform builds there is no dynamic EP download and this will return immediately.
Console.WriteLine("\nDownloading execution providers:");
if (eps.Length > 0)
{
    string currentEp = "";
    await mgr.DownloadAndRegisterEpsAsync((epName, percent) =>
    {
        if (epName != currentEp)
        {
            if (currentEp != "")
            {
                Console.WriteLine();
            }
            currentEp = epName;
        }
        Console.Write($"\r  {epName.PadRight(maxNameLen)}  {percent,6:F1}%");
    });
    Console.WriteLine();
}
else
{
    Console.WriteLine("No execution providers to download.");
}


// Get the model catalog
var catalog = await mgr.GetCatalogAsync();


// Get a model using an alias.
var model = await catalog.GetModelAsync("qwen2.5-0.5b") ?? throw new Exception("Model not found");

// Download the model (the method skips download if already cached)
await model.DownloadAsync(progress =>
{
    Console.Write($"\rDownloading model: {progress:F2}%");
    if (progress >= 100f)
    {
        Console.WriteLine();
    }
});

// Load the model
Console.Write($"Loading model {model.Id}...");
await model.LoadAsync();
Console.WriteLine("done.");

// Create a stateful native chat session and enable streaming.
using var chatSession = new ChatSession(model);
chatSession.SetStreaming(true);
using var request = new Request();
request.AddItem(MessageItem.User("Why is the sky blue?"));

// Get a streaming chat completion response
Console.WriteLine("Chat completion response:");
await using var streamingResponse = chatSession.ProcessStreamingRequestAsync(request, ct);
await foreach (var item in streamingResponse)
{
    using (item)
    {
        if (item is TextItem { Type: TextItemType.Default } text)
        {
            Console.Write(text.Text);
            Console.Out.Flush();
        }
    }
}
using var finalResponse = await streamingResponse.FinalResponse;
Console.WriteLine();

// Tidy up - unload the model
finalResponse.Dispose();
await streamingResponse.DisposeAsync();
request.Dispose();
chatSession.Dispose();
await model.UnloadAsync();

Run the code by using the following command:

dotnet run

Note

If you're targeting Windows, use the Windows-specific instructions under the Windows tab for the best performance and experience.

Troubleshooting

  • Build errors referencing net8.0: Install the .NET 8.0 SDK, then rebuild the app.
  • Model not found: Run the optional model listing snippet to find an alias available on your device, then update the alias passed to GetModelAsync.
  • Slow first run: Model downloads can take time the first time you run the app.

Prerequisites

Samples repository

The complete sample code for this article is available in the foundry-samples GitHub repository. To clone the repository and navigate to the sample use:

git clone https://github.com/microsoft-foundry/foundry-samples.git
cd foundry-samples/samples/javascript/foundry-local/native-chat-completions

Install packages

If you're developing or shipping on Windows, select the Windows tab. The Windows package integrates with the Windows ML runtime — it provides the same API surface area with a wider breadth of hardware acceleration.

npm install foundry-local-sdk-winml openai

Use native chat completions API

Copy and paste the following code into a JavaScript file named app.js:

import {
    ChatSession,
    FoundryLocalManager,
    Item,
    Request
} from 'foundry-local-sdk';

// Initialize the Foundry Local SDK
console.log('Initializing Foundry Local SDK...');

const manager = FoundryLocalManager.create({
    appName: 'foundry_local_samples',
    logLevel: 'info'
});
console.log('✓ SDK initialized successfully');

// Discover available execution providers and their registration status.
const eps = manager.discoverEps();
const maxNameLen = 30;
console.log('\nAvailable execution providers:');
console.log(`  ${'Name'.padEnd(maxNameLen)}  Registered`);
console.log(`  ${'─'.repeat(maxNameLen)}  ──────────`);
for (const ep of eps) {
    console.log(`  ${ep.name.padEnd(maxNameLen)}  ${ep.isRegistered}`);
}

// Download and register all execution providers with per-EP progress.
// EP packages include dependencies and may be large.
// Download is only required again if a new version of the EP is released.
console.log('\nDownloading execution providers:');
if (eps.length > 0) {
    let currentEp = '';
    await manager.downloadAndRegisterEps((epName, percent) => {
        if (epName !== currentEp) {
            if (currentEp !== '') {
                process.stdout.write('\n');
            }
            currentEp = epName;
        }
        process.stdout.write(`\r  ${epName.padEnd(maxNameLen)}  ${percent.toFixed(1).padStart(5)}%`);
    });
    process.stdout.write('\n');
} else {
    console.log('No execution providers to download.');
}

// Get the model object
const modelAlias = 'qwen2.5-0.5b'; // Using an available model from the list above
const model = await manager.catalog.getModel(modelAlias);

// Download the model
console.log(`\nDownloading model ${modelAlias}...`);
await model.download((progress) => {
    process.stdout.write(`\rDownloading... ${progress.toFixed(2)}%`);
});
console.log('\n✓ Model downloaded');

// Load the model
console.log(`\nLoading model ${modelAlias}...`);
await model.load();
console.log('✓ Model loaded');

// Create a stateful native chat session
console.log('\nCreating chat session...');
const chatSession = new ChatSession(model);
chatSession.setOptions({
    search: { temperature: 0.7, maxOutputTokens: 256 }
});
console.log('✓ Chat session created');

// Example chat completion
console.log('\nTesting chat completion...');
const completion = await chatSession.processRequest(
    new Request().addItem(Item.userMessage('Why is the sky blue?'))
);

console.log('\nChat completion result:');
for (const item of completion.output) {
    if (item.type === 'message') {
        process.stdout.write(item.content ?? item.parts?.map((part) => part.text ?? '').join('') ?? '');
    } else if (item.type === 'text' && item.textType === 'default') {
        process.stdout.write(item.text);
    }
}
console.log();

// Example streaming completion
console.log('\nTesting streaming completion...');
const stream = chatSession.processStreamingRequest(
    new Request().addItem(Item.userMessage(
        'Write a short poem about programming.'
    ))
);
for await (const item of stream) {
    if (item.type === 'text' && item.textType === 'default') {
        process.stdout.write(item.text);
    }
}
await stream.response;
console.log('\n');

// Unload the model
console.log('Unloading model...');
chatSession.dispose();
await model.unload();
console.log(`✓ Model unloaded`);

Run the code by using the following command:

node app.js

Prerequisites

Samples repository

The complete sample code for this article is available in the foundry-samples GitHub repository. To clone the repository and navigate to the sample use:

git clone https://github.com/microsoft-foundry/foundry-samples.git
cd foundry-samples/samples/python/foundry-local/native-chat-completions

Install packages

If you're developing or shipping on Windows, select the Windows tab. The Windows package integrates with the Windows ML runtime — it provides the same API surface area with a wider breadth of hardware acceleration.

pip install foundry-local-sdk-winml openai

Use native chat completions API

Copy and paste the following code into a Python file named app.py:

from foundry_local_sdk import (
    ChatSession,
    Configuration,
    FoundryLocalManager,
    MessageItem,
    Request,
    RequestOptions,
    SearchOptions,
    TextItem,
    TextItemType,
)



def main():
    # Initialize the Foundry Local SDK
    config = Configuration(app_name="foundry_local_samples")
    FoundryLocalManager.initialize(config)
    manager = FoundryLocalManager.instance

    # Discover and register execution providers that are not already available.
    eps = manager.discover_eps()
    missing_eps = [ep.name for ep in eps if not ep.is_registered]
    if missing_eps:
        manager.download_and_register_eps(missing_eps)

    # Select and load a model from the catalog.
    model = manager.catalog.get_model("qwen2.5-0.5b")
    if model is None:
        raise RuntimeError("Model 'qwen2.5-0.5b' was not found in the catalog.")
    if not model.is_cached:
        model.download(
            lambda progress: print(
                f"\rDownloading model: {progress:.2f}%",
                end="",
                flush=True,
            )
        )
        print()
    model.load()
    print("Model loaded and ready.")

    try:
        # Create a native chat session and stream the response token by token.
        with ChatSession(model) as session:
            session.set_options(
                RequestOptions(
                    search=SearchOptions(temperature=0.0, max_output_tokens=256)
                )
            )
            session.set_streaming(True)
            with Request().add_item(
                MessageItem.user("What is the golden ratio?")
            ) as request:
                print("Assistant: ", end="", flush=True)
                with session.process_streaming_request(request) as stream:
                    for item in stream:
                        if (
                            isinstance(item, TextItem)
                            and item.type == TextItemType.DEFAULT
                        ):
                            print(item.text, end="", flush=True)
                print()
    finally:
        # Clean up
        if model.is_loaded:
            model.unload()
        manager.close()
        print("Model unloaded.")


if __name__ == "__main__":
    main()

Run the code by using the following command:

python app.py

Troubleshooting

  • ModuleNotFoundError: No module named 'foundry_local_sdk': Install the SDK by running pip install foundry-local-sdk.
  • Model not found: Run the optional model listing snippet to find an alias available on your device, then update the alias passed to get_model.
  • Slow first run: Model downloads can take time the first time you run the app.

Prerequisites

Samples repository

The complete sample code for this article is available in the foundry-samples GitHub repository. To clone the repository and navigate to the sample use:

git clone https://github.com/microsoft-foundry/foundry-samples.git
cd foundry-samples/samples/rust/foundry-local/native-chat-completions

Install packages

If you're developing or shipping on Windows, select the Windows tab. The Windows package integrates with the Windows ML runtime — it provides the same API surface area with a wider breadth of hardware acceleration.

cargo add foundry-local-sdk --features winml
cargo add tokio --features full
cargo add tokio-stream anyhow

Use native chat completions API

Replace the contents of main.rs with the following code:

// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.

use std::io::{self, Write};

use foundry_local_sdk::{
    ChatSession, FoundryLocalConfig, FoundryLocalManager, Item, Request, RequestOptions,
    SearchOptions, TextKind,
};
use tokio_stream::StreamExt;

const ALIAS: &str = "qwen2.5-0.5b";

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    println!("Native Chat Completions");
    println!("=======================\n");

    // ── 1. Initialise the manager ────────────────────────────────────────
    let manager = FoundryLocalManager::create(FoundryLocalConfig::new("foundry_local_samples"))?;

    // Download and register all execution providers.
    manager
        .download_and_register_eps_with_progress(None, {
            let mut current_ep = String::new();
            move |ep_name: &str, percent: f64| {
                if ep_name != current_ep {
                    if !current_ep.is_empty() {
                        println!();
                    }
                    current_ep = ep_name.to_string();
                }
                print!("\r  {ep_name:<30}  {percent:5.1}%");
                io::stdout().flush().ok();
            }
        })
        .await?;
    println!();

    // ── 2. Pick a model and ensure it is downloaded ─────────────────────
    let model = manager.catalog().get_model(ALIAS).await?;
    println!("Model: {} (id: {})", model.alias(), model.id());

    if !model.is_cached().await? {
        println!("Downloading model...");
        model
            .download(Some(|progress: f64| {
                print!("\r  {progress:.1}%");
                io::stdout().flush().ok();
            }))
            .await?;
        println!();
    }

    println!("Loading model...");
    model.load().await?;
    println!("✓ Model loaded\n");

    // ── 3. Create a stateful chat session ────────────────────────────────
    let session = ChatSession::new(&model).await?;
    session
        .set_options(RequestOptions {
            search: SearchOptions {
                temperature: Some(0.7),
                max_output_tokens: Some(256),
                ..Default::default()
            },
            ..Default::default()
        })
        .await?;

    // ── 4. Non-streaming chat completion ────────────────────────────────
    let request = Request::from_items(vec![
        Item::system_message(vec![Item::text("You are a helpful assistant.")]),
        Item::user_message(vec![Item::text("What is Rust's ownership model?")]),
    ]);

    println!("--- Non-streaming completion ---");
    let response = session.process_request(request).await?;
    print!("Assistant: ");
    for item in &response.items {
        if let Some(message) = item.as_message() {
            print!("{}", message.text());
        }
    }
    println!();

    // ── 5. Streaming chat completion ────────────────────────────────────
    let request = Request::from_items(vec![Item::user_message(vec![Item::text(
        "Explain the borrow checker in two sentences.",
    )])]);

    println!("\n--- Streaming completion ---");
    print!("Assistant: ");
    let mut stream = session.process_streaming_request(request);
    while let Some(item) = stream.next().await {
        if let Item::Text {
            text,
            kind: TextKind::Default,
        } = item?
        {
            print!("{text}");
            io::stdout().flush().ok();
        }
    }
    println!("\n");

    // ── 6. Unload the model ─────────────────────────────────────────────
    drop(session);
    println!("Unloading model...");
    model.unload().await?;
    println!("Done.");

    Ok(())
}

Run the code by using the following command:

cargo run

Troubleshooting

  • Build errors: Ensure you have Rust 1.70.0 or later installed. Run rustup update to get the latest version.
  • Model not found: Verify the model alias is correct. Use manager.catalog().get_models().await? to list available models.
  • Slow first run: Model downloads can take time the first time you run the app.

Related content

  • Last updated on