Preprint · arXiv 2608.08847

Explicit Boundary Markers for Subword Vocabularies

Sander Land, Clara Meister

A tokenizer keeps a word twice, once with a leading space and once without. Capitalization splits it again, into as many as six forms of one word, which need not be built from the same pieces at all.

No space	Leading space	Boundary markers
`phil` `osoph` `y`	`␣philosophy`	`¦philosophy¦`
`Phil` `osoph` `y`	`␣Philosophy`	`↑` `¦philosophy¦`
`PH` `I` `LOS` `OP` `HY`	`␣PHI` `LOS` `OP` `HY`	`⇑` `¦philosophy¦`

One word in cl100k, the GPT-4 tokenizer. ␣ is a space, ¦ the boundary marker, ↑ title case and ⇑ upper case. Six forms on the left, one on the right, with the case codes as tokens of their own.

The problem

Which form a word gets depends on the character before it, and the forms need not agree on how to split it: 5,178 words are one token in one form and two or more pieces in the other. Each form has its own embedding row, trained on its own occurrences, so the rarer form of a common word gets as few updates as a rare string. Whether a prompt ends in a space decides which of them the model can produce at all.

What we do

The idea comes from Claude's tokenizer, which appears to mark word boundaries instead of attaching spaces and to code capitalization separately; the scheme here differs in the details. One atomic marker delimits a word on both sides. Where two words meet, their markers meet too, and that pair is the space: it is dropped on encoding, and decoding turns every adjacent pair back into one.

the␣cat → ¦the¦¦cat¦ → ¦the¦ ¦cat¦

A word with no space in front of it is marked all the same, so "the gives " ¦the¦: no pair, no space, and the same entry for the word either way. Lone markers are dropped on decoding.

Two shift codes do the same for title case and upper case, lowercasing the span they precede. It happens in pretokenization, it inverts, and it leaves the vocabulary learner untouched, so BPE and Unigram take it alike. The schemes differ only in which spans get marked: words, then punctuation, then digits.

Scheme	Pre-tokens	#
`plain`	`Ash` `␣caught` `␣` `3` `␣SolidGoldMagikarp` `.` `␣WOW` `!`	8
`[w]`	`¦Ash¦` `¦caught¦` `␣` `3` `␣` `¦SolidGoldMagikarp¦` `.` `␣` `¦WOW¦` `!`	10
`[w,p,d]`	`¦Ash¦` `¦caught¦` `¦3¦` `¦SolidGoldMagikarp¦` `.¦` `¦WOW¦` `!`	7
`[w,p,d,↑]`	`↑¦ash¦` `¦caught¦` `¦3¦` `¦SolidGoldMagikarp¦` `.¦` `⇑¦wow¦` `!`	7

Pre-tokens for "Ash caught 3 SolidGoldMagikarp. WOW!", before any vocabulary is learned. Marking words alone leaves the spaces around 3 and after the period, which is where the extra pre-tokens come from. Mixed case is not restorable from one code, so SolidGoldMagikarp stays as it is.

Why it matters

Scatter plot of characters per token against the baseline versus bits per byte, for BPE and MinGram. Every marker scheme sits below its baseline in bits per byte. Marking words alone sits about 9 percent left of the baseline in compression; the schemes that also mark punctuation and digits sit within one percent of it. — Every marker scheme lands below its baseline in bits per byte, and the axes disagree: the scheme that compresses 9% worse is the best BPE model.

Compression is a wash: across six languages, marking words, punctuation and digits lands within one percent of the baseline in characters per token. The models come out better anyway, every scheme and both trainers, at p < 0.01 over three seeds. If you want one form per word, the markers cost close to nothing.

Read the paper on arXiv ↗ Browse the code ↗