Microsoft.ML.Tokenizers

v2.0.0-preview.25373.1

Microsoft.ML.Tokenizers contains the implmentation of the tokenization used in the NLP transforms.

dotnetframework Microsoft MLNET

ML.NET ML Machine Learning

NuGet Stats Changelog

Total Downloads

9.2M

9,229,101

Published

Jul 28, 2025

Latest stable version

Install

.NET CLI

dotnet add package Microsoft.ML.Tokenizers --version 2.0.0-preview.25373.1

Package Manager Console

Install-Package Microsoft.ML.Tokenizers -Version 2.0.0-preview.25373.1

PackageReference

<PackageReference Include="Microsoft.ML.Tokenizers" Version="2.0.0-preview.25373.1" />

Frameworks

net8.0 .NETStandard2.0

Dependencies

Google.Protobuf

net8.0

[3.30.2, )

Google.Protobuf

.NETStandard2.0

[3.30.2, )

Microsoft.Bcl.HashCode

.NETStandard2.0

[6.0.0, )

Microsoft.Bcl.Memory

.NETStandard2.0

[9.0.4, )

System.Text.Json

.NETStandard2.0

[9.0.4, )

Microsoft.Bcl.AsyncInterfaces

.NETStandard2.0

[9.0.4, )

System.Buffers

.NETStandard2.0

[4.6.1, )

System.IO.Pipelines

.NETStandard2.0

[9.0.4, )

System.Memory

.NETStandard2.0

[4.6.3, )

System.Runtime.CompilerServices.Unsafe

.NETStandard2.0

[6.1.2, )

System.Text.Encodings.Web

.NETStandard2.0

[9.0.4, )

Adoption guide

A verified publisher signal is available. Assess fit from the package's supported frameworks, license and dependency graph rather than popularity alone.

This version is marked prerelease. Validate its API and upgrade path before production use.

Compatibility metadata lists 2 target frameworks; confirm the exact target used by your application.

How to read this guidance

Sources: published NuGet v3 registry metadata and explicit NuBrowse editorial guides. Refresh: package metadata is refreshed from NuGet when the page loads. Limit: this is not a security, legal or compatibility guarantee; validate the selected version in your project.

Useful next steps

Move from package metadata to a concrete selection or review workflow.

Audit NuGet dependencies in a .NET project Review NuGet licenses before shipping