Tiktoken.Cli.win-arm64

ttok — Token Counter CLI

The fastest .NET CLI tool for counting, encoding, decoding, and exploring BPE tokens. Powered by Tiktoken (618 MiB/s).

Install

# Homebrew (macOS — always installs latest release)
brew install tryAGI/tap/ttok

# Shell script (macOS/Linux — auto-detects OS/arch)
curl -fsSL https://raw.githubusercontent.com/tryAGI/Tiktoken/main/install.sh | sh
# Recommended for Linux — Homebrew cask is macOS-only

# PowerShell (Windows — auto-detects arch)
irm https://raw.githubusercontent.com/tryAGI/Tiktoken/main/install.ps1 | iex

# .NET global tool (NativeAOT on .NET 10+)
dotnet tool install -g Tiktoken.Cli

# Or run without installing (.NET 10+)
dnx Tiktoken.Cli "Hello world"

Standalone binary (no .NET required)

Download a pre-built native binary from GitHub Releases:

Docker

# Use the Alpine/musl binary in a container
docker run --rm -v "$(pwd):/src" -w /src alpine sh -c '
  wget -qO- https://github.com/tryAGI/Tiktoken/releases/latest/download/ttok-linux-musl-x64.tar.gz | tar xz
  echo "Hello world" | ./ttok
'

Usage

Count tokens (default: gpt-4o / o200k_base)

echo "Hello world" | ttok
# 3

cat prompt.md | ttok --model gpt-4
# 1847

Encode / Decode / Explore

# Show token IDs
echo "Hello world" | ttok --encode
# 13225 2375

# Decode token IDs back to text
echo "13225 2375" | ttok --decode
# Hello world

# Show token boundaries
echo "Hello world" | ttok --explore
# |Hello| world|

# Truncate to N tokens
cat long_doc.md | ttok --truncate 4000

Count files

ttok prompt.md system.md
# prompt.md    1,847
# system.md      423
# ─────────────────────
# 2 files      2,270 tokens

Scan directories (respects .gitignore)

ttok src/ --include "*.cs"

# JSON output for CI
ttok . --format json

# Sort by token count
ttok . --sort tokens --top 10

# Group by extension
ttok . --group-by ext

# Compare against model context window
ttok . --context-check

# Fail if over limit (CI gate, exit code 2)
ttok . --max-tokens 50000

Options

Supported models

All OpenAI models are supported via prefix matching: gpt-4o, gpt-4.1, gpt-4, gpt-3.5-turbo, o1, o3, o4-mini, and more.

Direct encoding names: o200k_base, o200k_harmony, cl100k_base, p50k_base, p50k_edit, r50k_base.

Harmony and special-token counting

Use --encoding o200k_harmony or --model gpt-oss-20b / gpt-oss-120b. GPT-5 names, including dotted variants, select o200k_base.

By default, counting treats control marker spellings as literal text. Use --allow-special to recognize all special tokens in trusted serialized input:

printf '%s' '<|start|>user<|message|>Hello<|end|>' | ttok -e o200k_harmony --allow-special
# 5
printf '%s' '<|start|>user<|message|>Hello<|end|>' | ttok -e o200k_harmony --allow-special --encode
# 200006 1428 200008 13225 200007

The flag also applies to file/directory counts and --truncate. Without it, --encode and --truncate reject special tokens; --explore already recognizes all special tokens for visualization. --decode decodes IDs, including control markers (aliases can canonicalize). These counts cover the supplied serialization; they do not add inferred chat-message overhead or promise server-side totals.