Microsoft.ML.Tokenizers.Data.P50kBase
The Microsoft.ML.Tokenizers.Data.P50kBase includes the Tiktoken tokenizer data file p50k_base.tiktoken, which is utilized by models such as text-davinci-002
Install
dotnet add package Microsoft.ML.Tokenizers.Data.P50kBase --version 2.0.0
Install-Package Microsoft.ML.Tokenizers.Data.P50kBase -Version 2.0.0
<PackageReference Include="Microsoft.ML.Tokenizers.Data.P50kBase" Version="2.0.0" />
Frameworks
Dependencies
Adoption guide
A verified publisher signal is available. Assess fit from the package's supported frameworks, license and dependency graph rather than popularity alone.
Compatibility metadata lists 1 target framework; confirm the exact target used by your application.
How to read this guidance
Sources: published NuGet v3 registry metadata and explicit NuBrowse editorial guides. Refresh: package metadata is refreshed from NuGet when the page loads. Limit: this is not a security, legal or compatibility guarantee; validate the selected version in your project.
Useful next steps
Move from package metadata to a concrete selection or review workflow.