ThinBASIC Modules > Tokenizer > Tokenizer equates

Equates for Tokenizer module.

Symbol	Meaning
%TOKENIZER_FINISHED	The returned token is end of buffer
%TOKENIZER_ERROR	There was an error in parsing next token
%TOKENIZER_UNDEFTOK	The next token is undefined
%TOKENIZER_EOL	The next token is end of line
%TOKENIZER_DELIMITER	The next token is a delimiter
%TOKENIZER_NUMBER	The next token is a number
%TOKENIZER_STRING	The next token is a string
%TOKENIZER_QUOTE	The next token is a quoted string

The following equates are mainly used by the following functions:

Symbol	Meaning
%TOKENIZER_DEFAULT_NewLine	Identify chars belonging to NewLine group
%TOKENIZER_DEFAULT_Space	Identify chars belonging to Space group
%TOKENIZER_DEFAULT_DQuote	Identify chars belonging to Double Quote group
%TOKENIZER_DEFAULT_Delim	Identify chars belonging to Delimiters group
%TOKENIZER_DEFAULT_Numeric	Identify chars belonging to Numeric group
%TOKENIZER_DEFAULT_Alpha	Identify chars belonging to Alphabetic group