pub fn tokenize(bytes: &[u8], mode: WidthMode, eaw_wide: bool) -> Tokenizer<'_> ⓘExpand description
Tokenize an input byte slice into ANSI-aware tokens.
The tokens concatenate back to bytes exactly; the tokenizer reclassifies
bytes but never invents or drops one. bytes need not be valid UTF-8.
See the module docs for which byte ends a control string, and for
why a byte in 0x80..=0x9F is a C1 control in one position and part of a
character in another.