Skip to main content
net8.0.NETStandard2.0

The Microsoft.ML.Tokenizers.Data.Cl100kBase class includes the Tiktoken tokenizer data file cl100k_base.tiktoken, which is utilized by models such as GPT-4.

Published
Oct 15, 2026
1mo ago
Total Downloads
4.3M
4,256,207

Install

dotnet add package Microsoft.ML.Tokenizers.Data.Cl100kBase --version 2.0.1
ProductVersions
.NET
net8.0
.NET Standard
netstandard2.0

README

About

The Microsoft.ML.Tokenizers.Data.Cl100kBase includes the Tiktoken tokenizer data file cl100k_base.tiktoken, which is utilized by models such as GPT-4.

Key Features

  • This package mainly contains the cl100k_base.tiktoken file, which is used by the Tiktoken tokenizer. This data file is used by the following models: 1. gpt-4 2. gpt-3.5-turbo 3. gpt-3.5-turbo-16k 4. gpt-35 5. gpt-35-turbo 6. gpt-35-turbo-16k 7. text-embedding-ada-002 8. text-embedding-3-small 9. text-embedding-3-large

How to Use

Reference this package in your project to use the Tiktoken tokenizer with the specified models.


// Create a tokenizer for the specified model or any other listed model name
Tokenizer tokenizer = TiktokenTokenizer.CreateForModel("gpt-4");

// Create a tokenizer for the specified encoding
Tokenizer tokenizer = TiktokenTokenizer.CreateForEncoding("cl100k_base");

Main Types

Users shouldn't use any types exposed by this package directly. This package is intended to provide tokenizer data files.

Additional Documentation

Related Packages

Microsoft.ML.Tokenizers

Feedback & Contributing

Microsoft.ML.Tokenizers.Data.Cl100kBase is released as open source under the MIT license. Bug reports and contributions are welcome at the GitHub repository.

Changes from Previous Version

v2.0.0v2.0.1
Added Frameworks
net8.0
Added Dependencies

Adoption Assessment

A verified publisher signal is available. Assess fit from the package's supported frameworks, license and dependency graph rather than popularity alone.

Compatibility metadata lists 2 target frameworks; confirm the exact target used by your application.The published metadata declares 13 direct dependency entries across framework groups. Inspect the resolved dependency graph for your target framework.

Related Guides & Workflows

Learn how to evaluate, integrate, or audit this package in your .NET project.