Module text 

ANSI-aware byte-stream tokenizer for text utilities.

Category

The tokenizer classifies an input byte slice as visible grapheme clusters, complete ANSI escape/string sequences, or standalone control bytes. Width, stripping, truncation, and wrapping utilities all build on this stream.

7-bit and 8-bit controls

Both 7-bit forms (ESC [, ESC ], ESC P, ESC _) and 8-bit C1 forms (0x9b, 0x9d, 0x90, 0x9f) are recognized. String controls terminate on BEL, ST (ESC \\), or 8-bit ST where applicable.

Mode interaction

This module does not interpret terminal modes or sequence semantics. Escape bytes are passed through as zero-width tokens so callers can preserve or drop them according to their own policy.

Re-exports

pub use crate::text::WidthMode;

Structs

Tokenizer

Iterator returned by tokenize.

Enums

Token

A single token produced by tokenize.

Functions

string_width

Return the display width of bytes ignoring ANSI escapes.

tokenize

Tokenize an input byte slice into ANSI-aware tokens.