FastBertTokenizer

v1.1.30-alpha

Fast and memory-efficient WordPiece tokenizer as it is used by BERT and others. Tokenizes text for further processing using NLP/language models.

georgjung

bert tokenizer wordpiece llm semantic-kernel performance ai artificial-intelligence ml machine-learning

NuGet Stats Changelog

Total Downloads

938.3K

938,349

Published

Mar 3, 2025

Latest stable version

Install

.NET CLI

dotnet add package FastBertTokenizer --version 1.1.30-alpha

Package Manager Console

Install-Package FastBertTokenizer -Version 1.1.30-alpha

PackageReference

<PackageReference Include="FastBertTokenizer" Version="1.1.30-alpha" />

Frameworks

net6.0 net8.0 .NETStandard2.0

Dependencies

System.Text.Json

net6.0

[8.0.5, )

System.Memory

.NETStandard2.0

[4.6.0, )

System.Text.Json

.NETStandard2.0

[8.0.5, )

Adoption guide

Assess this package against its supported frameworks, license, maintenance signals and dependency graph before making it a project standard.

This version is marked prerelease. Validate its API and upgrade path before production use.

Compatibility metadata lists 3 target frameworks; confirm the exact target used by your application.

How to read this guidance

Sources: published NuGet v3 registry metadata and explicit NuBrowse editorial guides. Refresh: package metadata is refreshed from NuGet when the page loads. Limit: this is not a security, legal or compatibility guarantee; validate the selected version in your project.

Useful next steps

Move from package metadata to a concrete selection or review workflow.

Audit NuGet dependencies in a .NET project Review NuGet licenses before shipping