Skip to main content

The Microsoft.ML.Tokenizers.Data.R50kBase includes the Tiktoken tokenizer data file r50k_base.tiktoken, which is utilized by models such as text-davinci-001

Total Downloads
67.5K
67,499
Published
Nov 11, 2026
5d ago

Get Started

dotnet add package Microsoft.ML.Tokenizers.Data.R50kBase --version 3.0.0
ProductVersions
.NET
net8.0
.NET Standard
netstandard2.0

README

About

The Microsoft.ML.Tokenizers.Data.R50kBase includes the Tiktoken tokenizer data file r50k_base.tiktoken, which is utilized by models such as text-davinci-001.

Key Features

  • This package mainly contains the r50k_base.tiktoken file, which is used by the Tiktoken tokenizer. This data file is used by the following models: 1. text-davinci-001 2. text-curie-001 3. text-babbage-001 4. text-ada-001 5. davinci 6. curie 7. babbage 8. ada 9. text-similarity-davinci-001
    1. text-similarity-curie-001
    2. text-similarity-babbage-001
    3. text-similarity-ada-001
    4. text-search-davinci-doc-001
    5. text-search-curie-doc-001
    6. text-search-babbage-doc-001
    7. text-search-ada-doc-001
    8. code-search-babbage-code-001
    9. code-search-ada-code-001

How to Use

Reference this package in your project to use the Tiktoken tokenizer with the specified models.


// Create a tokenizer for the specified model or any other listed model name
Tokenizer tokenizer = TiktokenTokenizer.CreateForModel("text-davinci-001");

// Create a tokenizer for the specified encoding
Tokenizer tokenizer = TiktokenTokenizer.CreateForEncoding("r50k_base");

Main Types

Users shouldn't use any types exposed by this package directly. This package is intended to provide tokenizer data files.

Additional Documentation

Related Packages

Microsoft.ML.Tokenizers

Feedback & Contributing

Microsoft.ML.Tokenizers.Data.R50kBase is released as open source under the MIT license. Bug reports and contributions are welcome at the GitHub repository.

Downloads by Version

Similar & Alternative Packages

Hand-picked matching NuGet libraries based on tags & ecosystem

Depended On By

Find packages that declare this package as a dependency. NuGet does not provide a reverse-dependency endpoint, so candidates are checked on demand.

Adoption Assessment

A verified publisher signal is available. Assess fit from the package's supported frameworks, license and dependency graph rather than popularity alone.

Compatibility metadata lists 2 target frameworks; confirm the exact target used by your application.

Related Guides & Workflows

Learn how to evaluate, integrate, or audit this package in your .NET project.