Tiktoken.Encodings.Tokenizer

v3.1.1

Load tokenizer.json (HuggingFace format) as a Tiktoken encoding.

tryAGI

chatgpt openai tiktoken tokens huggingface tokenizer bpe llama qwen deepseek gpt2

NuGet Stats Changelog

Total Downloads

78.9K

78,910

Published

Mar 24, 2026

Latest stable version

Install

.NET CLI

dotnet add package Tiktoken.Encodings.Tokenizer --version 3.1.1

Package Manager Console

Install-Package Tiktoken.Encodings.Tokenizer -Version 3.1.1

PackageReference

<PackageReference Include="Tiktoken.Encodings.Tokenizer" Version="3.1.1" />

Frameworks

.NETFramework4.6.2 net8.0 net9.0 net10.0 .NETStandard2.0 .NETStandard2.1

Dependencies

Tiktoken.Encodings.Abstractions

.NETFramework4.6.2

[3.1.1, )

System.Text.Json

.NETFramework4.6.2

[9.0.5, )

System.ValueTuple

.NETFramework4.6.2

[4.6.2, )

Tiktoken.Encodings.Abstractions

net8.0

[3.1.1, )

System.Text.Json

net8.0

[9.0.5, )

Tiktoken.Encodings.Abstractions

net9.0

[3.1.1, )

Tiktoken.Encodings.Abstractions

net10.0

[3.1.1, )

Tiktoken.Encodings.Abstractions

.NETStandard2.0

[3.1.1, )

System.Text.Json

.NETStandard2.0

[9.0.5, )

Tiktoken.Encodings.Abstractions

.NETStandard2.1

[3.1.1, )

System.Text.Json

.NETStandard2.1

[9.0.5, )

Adoption guide

Assess this package against its supported frameworks, license, maintenance signals and dependency graph before making it a project standard.

Compatibility metadata lists 6 target frameworks; confirm the exact target used by your application.

How to read this guidance

Sources: published NuGet v3 registry metadata and explicit NuBrowse editorial guides. Refresh: package metadata is refreshed from NuGet when the page loads. Limit: this is not a security, legal or compatibility guarantee; validate the selected version in your project.

Useful next steps

Move from package metadata to a concrete selection or review workflow.

Audit NuGet dependencies in a .NET project Review NuGet licenses before shipping