Microsoft.Windows.SDK.BuildTools.WinApp.UIAutomation

Microsoft.Windows.SDK.BuildTools.WinApp.UIAutomation

Inspect and drive any running Windows desktop app from code — the UI Automation engine behind the winapp ui commands, packaged as a library.

This package does not coordinate with other automation on the desktop. The winapp CLI layers cooperative desktop turns on top of this engine so concurrent winapp ui workflows cannot steal each other's focus or dismiss each other's menus. That arbitration lives in the CLI, not here. Code calling these APIs directly does not participate in it: it drives the desktop immediately, so if you run it alongside winapp ui, or alongside another copy of itself, you are responsible for serializing the two — or for running on a dedicated interactive desktop, as described below.

dotnet add package Microsoft.Windows.SDK.BuildTools.WinApp.UIAutomation
dotnet add package Microsoft.Extensions.DependencyInjection
dotnet add package Microsoft.Extensions.Logging

using Microsoft.Windows.SDK.BuildTools.WinApp.UIAutomation;
using Microsoft.Extensions.DependencyInjection;

var services = new ServiceCollection()
    .AddLogging()
    .AddWinAppUiAutomation()
    .BuildServiceProvider();

var ui = services.GetRequiredService<IUiAutomation>();
var resolver = services.GetRequiredService<IUiTargetResolver>();

var target = await resolver.ResolveAsync("notepad", hwnd: null, CancellationToken.None);
var button = await ui.FindSingleElementAsync(target, new UiSelector { Query = "Save" }, CancellationToken.None);
await ui.InvokeAsync(target, button!, CancellationToken.None);

What you get

• Element inspection and search — walk the UIA tree, or find one element by a stable semantic slug (btn-save-a1b2) or a plain-text query.
• Interaction through UIA patterns — invoke, set value, focus, scroll, scroll-into-view, read text.
• Input injection — keyboard, mouse, touch and pen, including gestures.
• Window capture — screenshots as raw BGRA pixels.

Video recording is a separate package, Microsoft.Windows.SDK.BuildTools.WinApp.UIAutomation.Recording, so that projects which only inspect and drive UI do not take a dependency on SkiaSharp.

Use SlugGenerator.Normalize(text) when constructing searchable, shell-safe labels with the same normalization as UI Automation selectors. It returns the normalized text, not a complete element selector; use the selector returned by inspection to address a specific element.

Choosing a target framework

The package ships two targets with the same public API:

Prefer net10.0-windows unless you need Graphics Capture. Graphics Capture matters when you have to screenshot a window that is occluded or in the background, or one whose content is GPU-composited (WinUI 3, DirectX, video), where PrintWindow can return a blank frame.

IWindowCapture.IsFrameCaptureSupported reports which implementation you got at runtime.

Native desktop pixels

IWindowCapture.GetDesktopBounds() returns the current input desktop's virtual-screen rectangle. Pass its origin and size to CaptureScreenPixels, using that same size for the encoder and display dimensions when you need unscaled BGRA pixels. Measure again afterward and discard the frame if the bounds changed. These operations do not activate a window. They require an unlocked interactive input desktop and a per-monitor-DPI-aware calling process.

CaptureCoordinates describes a source screen rectangle and its content rectangle within an image. ToScreenPoint maps an image pixel center to an integer screen pixel and rejects padding. The screen origin may be negative; right and bottom bounds are exclusive.

Using it with MSTest

The package pairs with MSTest.Windows.UIAutomation, which launches the app and hands you the main window as a UIA2 AutomationElement. Bridge into this library through the window handle:

[STATestClass]
public class CalculatorTests : WindowTest
{
    protected override ProcessStartInfo CreateProcessStartInfo() => new("calc.exe");

    [TestMethod]
    public async Task Clicking_Seven_ShowsSeven()
    {
        var target = UiTarget.FromWindowHandle(MainWindow.Current.NativeWindowHandle);
        var seven = await _ui.FindSingleElementAsync(target, new UiSelector { Query = "Seven" }, default);
        await _ui.InvokeAsync(target, seven!, default);
    }
}

Your test project must target a framework this package supports — for example net10.0-windows.

Input injection drives the real mouse and keyboard

IKeyboardInput, IMouseInput, and IPointerInput send system-wide input, exactly as a person at the machine would: clicks land wherever the cursor is moved, and keystrokes go to whichever window holds focus at that instant. If a popup, a UAC prompt, or a screen lock steals focus mid-test, the input goes there instead of your app.

Run this on a dedicated interactive desktop rather than the one you are working on, and note that injection does nothing useful over a disconnected RDP session, where there is no live desktop to receive it.

ForegroundGuard exposes the two checks these paths need, and they are deliberately different:

• ForegroundBelongsTo(hwnd) — the strict one, for input. It accepts only the target window or the top-level root that owns it, because a dialog in front would swallow your keystrokes.
• ForegroundIsCapturableFor(hwnd) — the capture one, for screen capture and screen recording. It also accepts a foreground window whose owner chain reaches the target, because a modal dialog the target owns is part of that app's UI and is sitting on the pixels you asked for. An unrelated window is still refused, so you never get a picture of somebody else's app labelled as yours.

Requirements

Windows 10 version 1809 or later for synthetic pen and touch injection; other features work on earlier Windows 10 releases.