Askaiser.Marionette

Askaiser.Marionette

Askaiser.Marionette is a test automation framework based on image and text recognition. It includes a C# source generator that allows you to quickly interact with properties generated by images from your project or elsewhere. The framework is built on top of OpenCV and Tesseract OCR and only works on Windows for now.

Why use Askaiser.Marionette

• Unlike other test automation frameworks, Askaiser.Marionette does not rely on hardcoded identifiers, CSS or XPath selectors. It uses image and text recognition to ensure that you interact with elements that are actually visible on the screen.
• Maintaining identifiers, CSS and XPath selectors over time can be hard. Capturing small screenshots and finding text with an OCR is not.
• With the built-in C# source generator, you can start writing the test code right away.
• You can interact with the whole operating system, instead of a single application.
• This means you can also test desktop applications!
• It works well with BDD and SpecFlow.
• You can use it to create a bot.

Askaiser.Marionette in action

• 00:00 : Capture screenshots of the app you're testing,
• 00:08 : Rename and organize your screenshots in a meaningful way,
• 00:22 : Drop your screenshots in your project,
• 00:30 : Use ImageLibraryAttribute to automatically generate properties from your screenshots,
• 01:06 : Use MarionetteDriver to interact with the generated properties (or even text recognized by the OCR)!

https://user-images.githubusercontent.com/14242083/126416123-aebd0fce-825f-4ece-90e9-762503dc4cab.mp4

Getting started

dotnet add package Askaiser.Marionette

It supports .NET Standard 2.0, .NET Standard 2.1 an .NET 6, but only on Windows for now.

[ImageLibrary("<relativeOrAbsolutePathToMyImagesDirectory>")]
public partial class MyLibrary
{
}

var library = new MyLibrary();

using (var driver = MarionetteDriver.Create(/* optional DriverOptions */))
{
    // in this exemple, we enter a username and password in a login page
    await driver.WaitForAsync(library.Pages.Login.Title, waitFor: TimeSpan.FromSeconds(5));

    await driver.SingleClickAsync(library.Pages.Login.Email);
    await driver.TypeTextAsync("[email protected]", sleepAfter: TimeSpan.FromSeconds(0.5));
    await driver.SingleClickAsync(library.Pages.Login.Password);
    await driver.TypeTextAsync("V3ry5ecre7!", sleepAfter: TimeSpan.FromSeconds(0.5));

    await driver.SingleClickAsync(library.Pages.Login.Submit);
    
    // insert more magic here
}

The sample project shows the basics of using this library.

Change the source generator behavior

Given the following partial image library class:

[ImageLibrary("source/screenshots")]
public partial class MyLibrary
{
}

var library = new MyLibrary();

The source generator will behave differently depending on the name of the screenshots/images added to the ImageLibraryAttribute's directory path.

Use lowercase characters. Dashes (-) and underscores (_) are considered as special characters that will change the generated C# code as shown below.

• Adding an image okbutton.png will create an image property library.Okbutton,
• Adding an image ok-button.png will create an image property library.OkButton (here the dash is a word separator),
• Adding an image in a subdirectory my-feature\ok-button.png will create an image property library.MyFeature.OkButton,
• Adding an image with a double dash my-feature--ok-button.png simulates a subdirectory and will create an image property library.MyFeature.OkButton.

Adding multiple images suffixed with a zero-based incremental number such as ok-button_0.png, ok-button_1.png, ok-button_2.png, etc. will create a single array property library.OkButton that can be interacted with methods that accept an array of elements (MoveToAnyAsync, WaitForAnyAsync, SingleClickAnyAsync, etc.). It is very convenient when you have screenshots of an element in multiple states, for instance a button in its normal, pressed, focus state and you want to click on the button no matter what its current state is.

Underscores _ can be also used to specify image recognition options:

• Adding an image foo_gs.png will create an image property library.Foo which will be grayscaled during image recognition,
• Adding an image foo_0.8 will create an image property library.Foo that will use a search threshold of 0.8 instead of the default 0.95,

You can mix these modifiers. Here we will create an single array property library.Header with these images:

• header_gs_0.85_0.png (first item of the array, grayscaled with a 0.85 threshold),
• header_gs_0.9_1.png (second item of the array, grayscaled with a 0.9 threshold),
• header_2.png (second and last item of the array, keep original colors with and use default threshold).

Creating image and text elements manually

Image search

// Instead of relying on the source generator that works with image files, you can create an ImageElement manually
var bytes = await File.ReadAllBytesAsync("path/to/your/image.png");
var image = new ImageElement(name: "sidebar-close-button", content: bytes, threshold: 0.95m, grayscale: false);

• ImageElement.Threshold is a floating number between 0 and 1. It defines the accuracy of the image search process. 0.95 is the default value.
• ImageElement.Grayscale defines whether or not the engine will apply grayscaling preprocessing. Image search is faster with grayscaling.

Image recognition works best with PNG images.

Text search

Although many methods accept a simple string as an element, you can manually create a TextElement
var text = new TextElement("Save changes", options: TextOptions.BlackAndWhite | TextOptions.Negative);

Text options are flags that define the preprocessing behavior of your monitor's screenshots before executing the OCR.

• TextOptions.None : do not use preprocessing,
• TextOptions.Grayscale : Use grayscaling,
• TextOptions.BlackAndWhite : Use grayscaling and binarization (this is the default value),
• TextOptions.Negative : Use negative preprocessing, very helpful with white text on dark background.

MarionetteDriver methods

Many parameters are optional. Most methods that look for an element (image or text) expect to find only one occurrence of this element. ElementNotFoundException and MultipleElementFoundException can be thrown.

You can use DriverOptions.FailureScreenshotPath to automatically save screenshots when these exceptions occur.

Configuration and utilities

static Create()
static Create(DriverOptions options)

GetScreenshotAsync()
GetCurrentMonitorAsync()
GetMonitorsAsync()
SetCurrentMonitor(int monitorIndex)
SetCurrentMonitor(MonitorDescription monitor)
SetMouseSpeed(MouseSpeed speed)
SleepAsync(int millisecondsDelay)
SleepAsync(TimeSpan delay)

Basic methods

WaitForAsync(IElement element, TimeSpan? waitFor, Rectangle? searchRect)
WaitForAllAsync(IEnumerable<IElement> elements, TimeSpan? waitFor, Rectangle? searchRect)
WaitForAnyAsync(IEnumerable<IElement> elements, TimeSpan? waitFor, Rectangle? searchRect)
SingleClickAsync(int x, int y)
DoubleClickAsync(int x, int y)
TripleClickAsync(int x, int y)
RightClickAsync(int x, int y)
MoveToAsync(int x, int y)
DragFromAsync(int x, int y)
DropToAsync(int x, int y)
TypeTextAsync(string text, TimeSpan? sleepAfter)
KeyPressAsync(VirtualKeyCode[] keyCodes)
KeyDownAsync(VirtualKeyCode[] keyCodes)
KeyUpAsync(VirtualKeyCode[] keyCodes)
ScrollDownAsync(int scrollTicks)
ScrollUpAsync(int scrollTicks)
ScrollDownUntilVisibleAsync(IElement element, TimeSpan totalDuration, int scrollTicks, Rectangle? searchRect)
ScrollUpUntilVisibleAsync(IElement element, TimeSpan totalDuration, int scrollTicks, Rectangle? searchRect)

Mouse interaction with an element

MoveToAsync(IElement element, TimeSpan? waitFor, Rectangle? searchRect)
SingleClickAsync(IElement element, TimeSpan? waitFor, Rectangle? searchRect)
DoubleClickAsync(IElement element, TimeSpan? waitFor, Rectangle? searchRect)
TripleClickAsync(IElement element, TimeSpan? waitFor, Rectangle? searchRect)
RightClickAsync(IElement element, TimeSpan? waitFor, Rectangle? searchRect)
DragFromAsync(IElement element, TimeSpan? waitFor, Rectangle? searchRect)
DropToAsync(IElement element, TimeSpan? waitFor, Rectangle? searchRect)

Check for element visibility

IsVisibleAsync(IElement element, TimeSpan? waitFor, Rectangle? searchRect)
IsAnyVisibleAsync(IEnumerable<IElement> elements, TimeSpan? waitFor, Rectangle? searchRect)
AreAllVisibleAsync(IEnumerable<IElement> elements, TimeSpan? waitFor, Rectangle? searchRect)

Mouse interaction with the first available element of a collection

MoveToAnyAsync(IEnumerable<IElement> elements, TimeSpan? waitFor, Rectangle? searchRect)
SingleClickAnyAsync(IEnumerable<IElement> elements, TimeSpan? waitFor, Rectangle? searchRect)
DoubleClickAnyAsync(IEnumerable<IElement> elements, TimeSpan? waitFor, Rectangle? searchRect)
TripleClickAnyAsync(IEnumerable<IElement> elements, TimeSpan? waitFor, Rectangle? searchRect)
RightClickAnyAsync(IEnumerable<IElement> elements, TimeSpan? waitFor, Rectangle? searchRect)
DragFromAnyAsync(IEnumerable<IElement> elements, TimeSpan? waitFor, Rectangle? searchRect)
DropToAnyAsync(IEnumerable<IElement> elements, TimeSpan? waitFor, Rectangle? searchRect)

Text-based actions

WaitForAsync(string text, TimeSpan? waitFor, Rectangle? searchRect, TextOptions? textOptions)
MoveToAsync(string text, TimeSpan? waitFor, Rectangle? searchRect, TextOptions? textOptions)
SingleClickAsync(string text, TimeSpan? waitFor, Rectangle? searchRect, TextOptions? textOptions)
DoubleClickAsync(string text, TimeSpan? waitFor, Rectangle? searchRect, TextOptions? textOptions)
TripleClickAsync(string text, TimeSpan? waitFor, Rectangle? searchRect, TextOptions? textOptions)
RightClickAsync(string text, TimeSpan? waitFor, Rectangle? searchRect, TextOptions? textOptions)
DragFromAsync(string text, TimeSpan? waitFor, Rectangle? searchRect, TextOptions? textOptions)
DropToAsync(string text, TimeSpan? waitFor, Rectangle? searchRect, TextOptions? textOptions)
IsVisibleAsync(string text, TimeSpan? waitFor, Rectangle? searchRect, TextOptions? textOptions)

Mouse interaction with points

MoveToAsync(Point coordinates)
SingleClickAsync(Point coordinates)
DoubleClickAsync(Point coordinates)
TripleClickAsync(Point coordinates)
RightClickAsync(Point coordinates)
DragFromAsync(Point coordinates)
DropToAsync(Point coordinates)

Mouse interaction with WaitFor search result

MoveToAsync(SearchResult searchResult)
SingleClickAsync(SearchResult searchResult)
DoubleClickAsync(SearchResult searchResult)
TripleClickAsync(SearchResult searchResult)
RightClickAsync(SearchResult searchResult)
DragFromAsync(SearchResult searchResult)
DropToAsync(SearchResult searchResult)

Key press with single key code

KeyPressAsync(VirtualKeyCode keyCode, TimeSpan? sleepAfter)
KeyDownAsync(VirtualKeyCode keyCode, TimeSpan? sleepAfter)
KeyUpAsync(VirtualKeyCode keyCode, TimeSpan? sleepAfter)

System.Drawing.Image-based actions

WaitForAsync(Image image, TimeSpan? waitFor, Rectangle? searchRect, decimal? threshold, bool? grayscale)
MoveToAsync(Image image, TimeSpan? waitFor, Rectangle? searchRect, decimal? threshold, bool? grayscale)
SingleClickAsync(Image image, TimeSpan? waitFor, Rectangle? searchRect, decimal? threshold, bool? grayscale)
DoubleClickAsync(Image image, TimeSpan? waitFor, Rectangle? searchRect, decimal? threshold, bool? grayscale)
TripleClickAsync(Image image, TimeSpan? waitFor, Rectangle? searchRect, decimal? threshold, bool? grayscale)
RightClickAsync(Image image, TimeSpan? waitFor, Rectangle? searchRect, decimal? threshold, bool? grayscale)
DragFromAsync(Image image, TimeSpan? waitFor, Rectangle? searchRect, decimal? threshold, bool? grayscale)
DropToAsync(Image image, TimeSpan? waitFor, Rectangle? searchRect, decimal? threshold, bool? grayscale)
IsVisibleAsync(Image image, TimeSpan? waitFor, Rectangle? searchRect, decimal? threshold, bool? grayscale)

Finding elements locations without throwing not found exceptions or multiple element found exceptions

FindLocationsAsync(IElement element, TimeSpan? waitFor, Rectangle? searchRect)
FindLocationsAsync(string text, TimeSpan? waitFor, Rectangle? searchRect, TextOptions? textOptions)
FindLocationsAsync(Image image, TimeSpan? waitFor, Rectangle? searchRect, decimal? threshold, bool? grayscale)