tokenize.rs

Where the vocabulary goes

ctok Claude's tokenizer

Measure

Allocation across six tokenizers

Natural-language allocation

Code allocation