Shiny.Speech.OpenAI

Shiny.Speech

Cross-platform speech services for .NET MAUI and Blazor WebAssembly — speech-to-text, text-to-speech, audio capture, and audio playback with pluggable cloud providers.

Libraries

Getting Started

Native Platform Speech

Use the built-in OS speech engines — no cloud account needed. Works on MAUI (iOS, Android, Windows) and Blazor WebAssembly (via Web Speech API).

builder.Services.AddSpeechServices();
// Registers: ISpeechToTextService, ITextToSpeechService, IAudioSource, IAudioPlayer
// On Browser/WASM: auto-detected via OperatingSystem.IsBrowser()

Azure AI Speech (Cloud)

builder.Services.AddAzureSpeech("your-subscription-key", "your-region");

ElevenLabs (Cloud)

// Register both STT (Scribe) and TTS:
builder.Services.AddElevenLabsSpeech("your-api-key");

// Or pick one:
builder.Services.AddElevenLabsSpeechToText("your-api-key");
builder.Services.AddElevenLabsTextToSpeech("your-api-key");

Typecast (Cloud, TTS only)

builder.Services.AddTypecastSpeech("your-typecast-api-key");
// Set a voice via TypecastConfig.DefaultVoiceId or TextToSpeechOptions.Voice;
// GetVoicesAsync() lists the ids available to your account.

Usage

Text-to-Speech

public class MyService(ITextToSpeechService tts)
{
    public async Task SpeakAsync()
    {
        await tts.SpeakAsync("Hello world!", new TextToSpeechOptions
        {
            SpeechRate = 1.2f,
            Pitch = 1.0f,
            Volume = 0.8f
        });
    }
}

VU Meters (Audio Levels)

Both directions are metered, on the same normalized 0.0–1.0 scale (dBFS mapped from a -50 dB noise floor, so voice actually moves the bar).

Outgoing — playback / TTS. ITextToSpeechService and IAudioPlayer raise AudioLevelChanged while audio is playing. Gate UI on IsPlayerAnalysisSupported.

if (tts.IsPlayerAnalysisSupported)
    tts.AudioLevelChanged += (s, level) =>
        MainThread.BeginInvokeOnMainThread(() => SpeakingBar.Progress = level);

Incoming — microphone. ISpeechToTextService raises InputLevelChanged while listening — gate UI on IsInputAnalysisSupported. The lower-level IAudioSource.InputLevelChanged (raw capture) and IAudioMonitor.InputLevelChanged (live mic-to-output) emit the same signal.

if (stt.IsInputAnalysisSupported)
    stt.InputLevelChanged += (s, level) =>
        MainThread.BeginInvokeOnMainThread(() => ListeningBar.Progress = level);

Cloud recognition meters the IAudioSource feeding the provider, so it works everywhere. Native recognition depends on the platform: Apple taps the recognizer's own input node, Android reports the SpeechRecognizer RMS callback, while Windows' and the browser's recognizers own the mic and expose no level at all.

On Apple platforms, native TTS routes AVSpeechSynthesizer through AVAudioEngine + AVAudioPlayerNode so audio levels can be tapped. The engine is created lazily on first speak and kept warm across utterances — only the first utterance pays ~50–150 ms additional startup.

Levels are raised off the UI thread and throttled to ~20/sec on the capture side; marshal before binding. Reset your bound value to 0 when playback/listening ends so the meter drains.

Speech-to-Text

The ISpeechToTextService uses a Start/Stop model with events, allowing multiple consumers to observe recognition results simultaneously.

public class MyService(ISpeechToTextService stt) : IDisposable
{
    public async Task StartListeningAsync()
    {
        var access = await stt.RequestAccess();
        if (access != AccessState.Available)
            return;

        // Subscribe to events
        stt.ResultReceived += OnResult;
        stt.KeywordHeard += OnKeyword;
        stt.Error += OnError;

        // Start listening (throws if already listening)
        await stt.Start(new SpeechRecognitionOptions
        {
            Culture = CultureInfo.GetCultureInfo("en-US"),
            SilenceTimeout = TimeSpan.FromSeconds(3),
            Keywords = ["Yes", "No", "Maybe"]
        });
    }

    public async Task StopListeningAsync()
    {
        await stt.Stop(); // no-op if not listening
        stt.ResultReceived -= OnResult;
        stt.KeywordHeard -= OnKeyword;
        stt.Error -= OnError;
    }

    void OnResult(object? sender, SpeechRecognitionResult result)
        => Console.WriteLine($"[{(result.IsFinal ? "FINAL" : "partial")}] {result.Text}");

    void OnKeyword(object? sender, string keyword)
        => Console.WriteLine($"Keyword detected: {keyword}");

    void OnError(object? sender, SpeechRecognitionError error)
        => Console.WriteLine($"Error: {error.Message}");

    public void Dispose() => StopListeningAsync().GetAwaiter().GetResult();
}

Extension Methods (Convenience)

// Simple: wait for silence (starts and stops automatically)
var text = await stt.ListenUntilSilence(cancellationToken: ct);

// Wake word: "Hey Computer, do something" → returns "do something"
var command = await stt.StatementAfterKeyword(["Hey Computer"], cancellationToken: ct);

// Wait for a specific keyword (with optional timeout)
var answer = await stt.WaitListenForKeywords(["Yes", "No"], timeout: TimeSpan.FromSeconds(30), cancellationToken: ct);

// Continuous keyword stream
await foreach (var keyword in stt.ListenForKeywords(["Up", "Down", "Left", "Right"], cancellationToken: ct))
{
    Console.WriteLine($"Direction: {keyword}");
}

Custom Cloud Provider

Implement ISpeechToTextProvider and/or ITextToSpeechProvider from Shiny.Speech.Cloud:

public class MyCloudSttProvider : ISpeechToTextProvider
{
    public event EventHandler<SpeechRecognitionError>? Error;

    public async IAsyncEnumerable<SpeechRecognitionResult> RecognizeAsync(
        Stream audioStream,
        SpeechRecognitionOptions? options = null,
        CancellationToken cancellationToken = default)
    {
        // Read PCM audio from audioStream (16kHz, 16-bit, mono)
        // Yield recognition results...

        // For continuous recognition, surface non-fatal errors (e.g. a transient
        // network blip between chunked requests) without aborting the session:
        // Error?.Invoke(this, new SpeechRecognitionError("network blip", ex));
    }
}

// Register:
builder.Services.AddCloudSpeechToText<MyCloudSttProvider>();

CloudSpeechToText subscribes to the provider's Error event and forwards it to the service-level ISpeechToTextService.Error, so app code only needs to wire one handler.

Platform Requirements

Browser (Blazor WebAssembly)

No manifest changes and no <script> tag needed — the browser prompts the user for microphone access automatically, and the JS interop module ships inside the Shiny.Audio package as a static web asset (_content/Shiny.Audio/shiny-audio.js). It is loaded on demand via JSHost.ImportAsync, so referencing Shiny.Speech (or Shiny.Audio directly) is all that's required.

Note: IAudioSource captures raw PCM audio in the browser using the Web Audio API (getUserMedia + ScriptProcessorNode), downsampled to 16kHz 16-bit mono. Audio playback (IAudioPlayer) accepts any browser-supported format via a base64 data URL.

iOS/macOS

Add to Info.plist:

<key>NSSpeechRecognitionUsageDescription</key>
<string>Speech recognition description</string>
<key>NSMicrophoneUsageDescription</key>
<string>Microphone description</string>

Android

Add to AndroidManifest.xml:

<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS" />

MODIFY_AUDIO_SETTINGS is required for the TTS audio-level Visualizer and for the native STT beep suppression.