ElBruno.LocalLLMs

ElBruno.LocalLLMs

Run local LLMs in .NET through IChatClient — the same interface you'd use for Azure OpenAI, Ollama, or any other provider. Powered by ONNX Runtime GenAI.

Features

• 🔌 IChatClient implementation — seamless integration with Microsoft.Extensions.AI
• 📦 Automatic model download — models are fetched from HuggingFace on first use
• 🚀 Zero friction — works out of the box with sensible defaults (Phi-3.5 mini)
• 🖥️ Multi-hardware — CPU, CUDA, and DirectML execution providers
• 💉 DI-friendly — register with AddLocalLLMs() in ASP.NET Core
• 🔄 Streaming — token-by-token streaming via GetStreamingResponseAsync
• 📊 Multi-model — switch between Phi-3.5, Phi-4, Qwen2.5, Llama 3.2, and more

Installation

dotnet add package ElBruno.LocalLLMs

Quick Start

using ElBruno.LocalLLMs;
using Microsoft.Extensions.AI;

// Create a local chat client (downloads Phi-3.5 mini on first run)
using var client = await LocalChatClient.CreateAsync();

var response = await client.GetResponseAsync([
    new(ChatRole.User, "What is the capital of France?")
]);

Console.WriteLine(response.Text);

Streaming

using ElBruno.LocalLLMs;
using Microsoft.Extensions.AI;

using var client = await LocalChatClient.CreateAsync(new LocalLLMsOptions
{
    Model = KnownModels.Phi35MiniInstruct
});

await foreach (var update in client.GetStreamingResponseAsync([
    new(ChatRole.System, "You are a helpful assistant."),
    new(ChatRole.User, "Explain quantum computing in simple terms.")
]))
{
    Console.Write(update.Text);
}

Dependency Injection

builder.Services.AddLocalLLMs(options =>
{
    options.Model = KnownModels.Phi35MiniInstruct;
    options.ExecutionProvider = ExecutionProvider.DirectML;
});

// Inject IChatClient anywhere
public class MyService(IChatClient chatClient) { ... }

Supported Models

See the Supported Models Guide for detailed model cards, performance benchmarks, and selection guidance.

Samples

Requirements

• .NET 8.0 or .NET 10.0
• CPU (default), NVIDIA GPU (CUDA), or Windows GPU (DirectML)
• ~2-8 GB disk space per model (depending on size and quantization)

Documentation

• Getting Started — installation, first steps, configuration
• Supported Models — full model reference with tiers, specs, decision tree
• Architecture — design decisions and internal structure
• Samples Guide — walkthrough of each sample application
• Benchmarks — how to run and interpret performance benchmarks
• ONNX Conversion — converting HuggingFace models to ONNX format
• Publishing — NuGet package publishing with OIDC
• Contributing — how to contribute
• Changelog — version history

License

MIT © Bruno Capuano