Skip to main content
net8.0.NETStandard2.0

The Microsoft.ML.Tokenizers.Data.Cl100kBase class includes the Tiktoken tokenizer data file cl100k_base.tiktoken, which is utilized by models such as GPT-4.

Published
Nov 11, 2026
5d ago
Total Downloads
4.3M
4,256,207

Install

dotnet add package Microsoft.ML.Tokenizers.Data.Cl100kBase --version 3.0.0
ProductVersions
.NET
net8.0
.NET Standard
netstandard2.0

README

About

The Microsoft.ML.Tokenizers.Data.Cl100kBase includes the Tiktoken tokenizer data file cl100k_base.tiktoken, which is utilized by models such as GPT-4.

Key Features

  • This package mainly contains the cl100k_base.tiktoken file, which is used by the Tiktoken tokenizer. This data file is used by the following models: 1. gpt-4 2. gpt-3.5-turbo 3. gpt-3.5-turbo-16k 4. gpt-35 5. gpt-35-turbo 6. gpt-35-turbo-16k 7. text-embedding-ada-002 8. text-embedding-3-small 9. text-embedding-3-large

How to Use

Reference this package in your project to use the Tiktoken tokenizer with the specified models.


// Create a tokenizer for the specified model or any other listed model name
Tokenizer tokenizer = TiktokenTokenizer.CreateForModel("gpt-4");

// Create a tokenizer for the specified encoding
Tokenizer tokenizer = TiktokenTokenizer.CreateForEncoding("cl100k_base");

Main Types

Users shouldn't use any types exposed by this package directly. This package is intended to provide tokenizer data files.

Additional Documentation

Related Packages

Microsoft.ML.Tokenizers

Feedback & Contributing

Microsoft.ML.Tokenizers.Data.Cl100kBase is released as open source under the MIT license. Bug reports and contributions are welcome at the GitHub repository.

Changes from Previous Version

No structural changes between v3.0.0-preview.26513.4 and v3.0.0.

Adoption Assessment

A verified publisher signal is available. Assess fit from the package's supported frameworks, license and dependency graph rather than popularity alone.

Compatibility metadata lists 2 target frameworks; confirm the exact target used by your application.

Related Guides & Workflows

Learn how to evaluate, integrate, or audit this package in your .NET project.