SIL.Harmony.Linq2db

harmony

A CRDT application library for C#, use it to build offline first applications.

Install

dotnet add package SIL.Harmony

It's expected that you use Harmony with the .Net IoC container (IoC intro) and with EF Core. If you're not familier with that you can take a look at the Host docs. If you're using ASP.NET Core you already have this setup for you.

Prerequisites:

• Setup EF Core in your application (Getting started docs)
• Setup a Host, the default host setup for ASP.NET Core will work, or a generic host for desktop apps, depending on your app. Alternatively you could create a ServiceCollection.

Configure DbContext

EF Core needs to be told about the entities used by Harmony, for now these are just Commit, Snapshot, and ChangeEntitiy

public class AppDbContext: DbContext {
  protected override void OnModelCreating(ModelBuilder modelBuilder)
  {
      modelBuilder.UseCrdt(crdtConfig.Value);
  }
}

SampleDbContext has a full example of how to setup the DbContext.

Register CRDT services

Harmony provides the DataModel class as the main way the application will interact with the CRDT model. You first need to register it with the IoC container.

var builder = Host.CreateApplicationBuilder(args);
builder.Service.AddCrdtData<AppDbContext>(config => {});

the config callback passed into AddCrdtData is currently empty, we'll come back to that later.

Pay attention to the generic type when calling AddCrdtData, this will be the type of your application's DbContext.

Define CRDT objects

Now that you have the services setup, you need to define a CRDT object. Take a look the following examples

• Word contains a reference to an Antonym Word.
• Definition references the Word it belongs to. Notice that if the Word Reference is removed, the Definition deletes itself.
• Example this one is special because it uses a YDoc to store the example text in a Yjs compatible format. This allows the example sentence to be edited by multiple users and have those changes merged using the yjs CRDT algorithm.

Once you have created your CRDT objects, you need to tell Harmony about them. Update the config callback passed into AddCrdtData

services.AddCrdtData<SampleDbContext>(config =>
{
// add the following lines
    config.ObjectTypeListBuilder
        .Add<Word>()
        .Add<Definition>()
        .Add<Example>();
});

Define CRDT Changes

Now that you've defined your objects, you need to define your changes. These record user intent when making changes to objects. How detailed and specific you make your changes will directly impact how changes get merged between clients and how often users 'lose' changes that they made.

Example SetWordTextChange

public class SetWordTextChange(Guid entityId, string text) : Change<Word>(entityId), ISelfNamedType<SetWordTextChange>
{
    public string Text { get; } = text;

    public override ValueTask<IObjectBase> NewEntity(Commit commit, IChangeContext context)
    {
        return new(new Word()
        {
            Id = EntityId,
            Text = Text
        });
    }


    public override ValueTask ApplyChange(Word entity, IChangeContext context)
    {
        entity.Text = Text;
        return ValueTask.CompletedTask;
    }
}

This is a fairly simple change, it can either create a new Word entry, or if the entityId passed in matches an object that has previously been created, then it will just set the Text field on the Word entry matching the Id.

Changes will be serialized and stored forever. Try to keep the amount of data stored as small as possible.

This change can either create, or update an object. Most changes will probably be either an update, or a create. In those cases you should inherit from EditChange<T> or CreateChange<T>.

The Sample project contain a number of reference changes which are good examples for a couple different change types. There are also a built in DeleteChange<T>

Once you have created your change types, you need to tell Harmony about them. Again update the config callback passed into AddCrdtData

services.AddCrdtData<SampleDbContext>(config =>
{
// add the following lines
    config.ChangeTypeListBuilder.Add<SetWordTextChange>();
    //every change type must be listed by a model version, see Model versions below
    config.ModelVersionBuilder.Add(0, "Before model versions", v => v.Change<SetWordTextChange>());
    config.ObjectTypeListBuilder
        .Add<Word>()
        .Add<Definition>()
        .Add<Example>();
});

Use change objects to author changes to CRDT objects

Either via DI, or directly from the IoC container get an instance of DataModel and call AddChange

Guid clientId = ... get a stable Guid representing the application instance
Guid objectId = Guid.NewGuid();
await dataModel.AddChange(
  clientId,
  new SetWordTextChange(objectId, "Hello World")
);
var word = await dataModel.GetLatest<Word>(objectId);
Console.WriteLine(word.Text);

The ClientId should be consistent for a project per computer/device. It is used to determine what changes should be synced between clients with the assumption that each client produces changes sequentially. So if a project is on 2 different computers, each copy should have a unique client Id. If they had the same Id, then they would not sync changes properly.

How the ClientId is stored is left up to the application. In FW Lite we created a table to store the ClientId. It's generated automatically when the project is downloaded or created the first time and it should never change after that.

In case of an online web app there could be one ClientId to represent the server. However, if users can author changes offline and sync them later, then each browser would need it's own ClientId.

If you were to regenerate the ClientId for each change or on application start, that would eventually result in poor sync performance, as the sync process checks for new changes to sync per ClientId.

Usage

Queries

DataModel is the primary class for both making changes and getting data. Above you saw an example of making changes, now we'll start querying data.

Query Word objects starting with the letter "A"

DataModel dataModel; //get from IoC, probably via DI
var wordsStartingWithA = await dataModel.GetLatestObjects<Word>()
    .Where(w => w.Text.StartsWith("a"))
    .ToArrayAsync();

Harmony uses EF Core queries under the covers, you can read more about them here.

Submitting Changes

Changes are the only way to modify CRDT data. Here's another example of a change

DataModel dataModel;
Guid clientId; //get a stable Guid representing the application instance
var definitionId = Guid.NewGuid();
Guid wordId; //get the word Id this definition is related to.
await dataModel.AddChange(clientId, new NewDefinitionChange(definitionId)
        {
            WordId = wordId,
            Text = "Hello",
            PartOfSpeech = partOfSpeech,
            Order = order
        });

You can modify data returned by EF Core, and issue updates and inserts yourself, but that data will be lost, and will not sync properly. Do not directly modify the tables produced by Harmony otherwise you risk losing data.

Syncing data

Syncing is primarily done using the DataModel class, however the implementation of the server side is left up to you. You can find the Lexbox implementation here. The sync works by having 2 instances of the ISyncable interface. The local one is implemented by DataModel and the remote implementation depends on your server side. The FW Lite implementation can be found here. You will need to scope the instance to the project as well as deal with authentication.

Once you have a remote representation of the ISyncable interface you just call it like this

DataModel dataModel;
ISyncable remoteModel;
await dataModel.SyncWith(remoteModel);

It's that easy. All the heavy lifting is done by the interface which is fairly simple to implement.

Model versions

Clients on different app versions sync with each other, so an older app receives changes a newer app authored. It skips the ones whose type it doesn't know (stored as OpaqueChange with UnknownChangeHandling.Fallback) and applies the rest with its old code. When the older app upgrades, those changes must be replayed.

Model versions make that possible. Declare them in order, numbered from 0, each listing the change types it added or changed. Version 0 lists every change type that existed before model versions. Every change is stored with the number of the authoring app's latest version, whether or not that version lists the change's type, so a change authored by a newer app carries a number the older app doesn't have yet. Startup fails if no versions are declared, a registered change type isn't listed, or remote resources are enabled without RemoteResourceChanges. Only add versions at the end, never modify a released one:

services.AddCrdtRemoteResources<MediaMetadata>();
services.AddCrdtData<AppDbContext>(config =>
{
    config.ChangeTypeListBuilder
        .Add<NewWordChange>()
        .Add<SetWordTextChange>()
        .Add<SetWordNoteChange>();
    config.ModelVersionBuilder
        .Add(0, "Before model versions", v => v
            .Change<NewWordChange>()
            .Change<SetWordTextChange>()
            //the change types AddCrdtRemoteResources registers
            .RemoteResourceChanges<MediaMetadata>())
        .Add(1, "Word notes", v => v.Change<SetWordNoteChange>())
        //a major version regenerates all snapshots on upgrade
        .AddMajor(2, "Reference rewrite", v => v.Change<SetWordTextChange>());
});

Call ReconcileModelVersions when you open a database, before you use it. A database with nothing stored is at version 0. For each type listed by a version newer than the stored one, it invalidates the changes authored with that version or later, and rebuilds the snapshots from the checkpoint before the oldest of them. A major version, or a downgrade, regenerates all snapshots. It throws if an applied version was modified. Querying changes by type is supported on SQLite only.

await dataModel.ReconcileModelVersions();

Listing a type invalidates only the changes authored by apps on the new version or later. When the new behavior must also apply to older changes, pass invalidateFrom with the model version to invalidate from. That covers a released version that changed how a type applies but forgot to list it: its apps still stamped their changes with its own number, so invalidating from that version replays only the changes those apps and later ones authored:

config.ModelVersionBuilder
    .Add(0, "Before model versions", v => v.Change<SetWordNoteChange>().Change<SetTagChange>())
    .Add(1, "Word notes", v => v.Change<SetWordNoteChange>())
    //changed how SetTagChange applies but forgot to list it, its apps authored tags as version 2
    .Add(2, "Notes again", v => v.Change<SetWordNoteChange>())
    .Add(3, "Fix tags", v => v.Change<SetTagChange>(invalidateFrom: 2));

To make sure a released version is not modified by accident, add a snapshot test of HarmonyConfig.DescribeModelVersions() to your app. A new version only adds a line at the end of the snapshot.

Model versions add an AuthoredModelVersion column to ChangeEntities and a LocalState table, so an EF migration is required.

Development

SemVer commit messages

NuGet package versions are calculated from a combination of tags and commit messages. First, the most recent Git tag matching the pattern v\d+.\d+.\d+ is located. If that is the commit being built, then that version number is used. If there have been any commits since then, the version number will be bumped by looking for one of the following patterns in the commit messages:

• +semver: major or +semver: breaking - update major version number, reset others to 0 (so 2.3.1 would become 3.0.0)
• +semver: minor or +semver: feature - update minor version number, reset patch to 0 (so 2.3.1 would become 2.4.0)
• Anything else, including no +semver lines at all - update patch version number (so 2.3.1 would become 2.3.2)
  • If you want to include +semver lines, then +semver: patch or +semver: fix are the standard ways to increment a patch version bump, but the patch version will be bumped regardless as long as there is at least one commit since the most recent tag.