The Microsoft.ML.Tokenizers.Data.P50kBase includes the Tiktoken tokenizer data file p50k_base.tiktoken, which is utilized by models such as text-davinci-002
Install
| Product | Versions |
|---|---|
| .NET Standard |
README
About
The Microsoft.ML.Tokenizers.Data.P50kBase includes the Tiktoken tokenizer data file p50k_base.tiktoken, which is utilized by models such as text-davinci-002.
Key Features
- This package mainly contains the
p50k_base.tiktokenfile, which is used by the Tiktoken tokenizer. This data file is used by the following models: 1. text-davinci-002 2. text-davinci-003 3. code-davinci-001 4. code-davinci-002 5. code-cushman-001 6. code-cushman-002 7. davinci-codex 8. cushman-codex
How to Use
Reference this package in your project to use the Tiktoken tokenizer with the specified models.
// Create a tokenizer for the specified model or any other listed model name
Tokenizer tokenizer = TiktokenTokenizer.CreateForModel("text-davinci-002");
// Create a tokenizer for the specified encoding
Tokenizer tokenizer = TiktokenTokenizer.CreateForEncoding("p50k_base");
Main Types
Users shouldn't use any types exposed by this package directly. This package is intended to provide tokenizer data files.
Additional Documentation
Related Packages
Microsoft.ML.Tokenizers
Feedback & Contributing
Microsoft.ML.Tokenizers.Data.P50kBase is released as open source under the MIT license. Bug reports and contributions are welcome at the GitHub repository.
Changes from Previous Version
Adoption Assessment
A verified publisher signal is available. Assess fit from the package's supported frameworks, license and dependency graph rather than popularity alone.
Related Guides & Workflows
Learn how to evaluate, integrate, or audit this package in your .NET project.