Tokenizer: Configurizable?

[Michael Hartlef]

What ever you think is fine. Like I said, even now I can perfectly work with it. Thanks for all the improvements.

[ErosOlmi]

Mike, I've done it. Hope you will like.
You will be able to set any of the 255 ascii table char to any of the following groups: NewLine, Space, DQuote, Delim, Numeric, Alpha
This at any time and anywhere during script execution. Yu will also able to set in group, so pass a list of chars to be set into a group.

An example:
[code=thinbasic]
'---Create a new keywords group. Assign a value >= 100
dim MyDictionary as long value 100

'---Add keywords to the group
tokenizer_KeyAdd("DIM" , MyDictionary, 1)
tokenizer_KeyAdd("AS" , MyDictionary, 2)
tokenizer_KeyAdd("CONSOLE_writeline" , MyDictionary, 3)
'...
'---Add as many keywords you need
'...

'---Now change some default behave in char parsing
Tokenizer_Default_Char("$", %TOKENIZER_DEFAULT_Alpha)
Tokenizer_Default_Char("%", %TOKENIZER_DEFAULT_Alpha)
Tokenizer_Default_Char(":", %TOKENIZER_DEFAULT_newline)
Tokenizer_Default_Char(";", %TOKENIZER_DEFAULT_newline)

'---... or even faster
Tokenizer_Default_Set("$%", %TOKENIZER_DEFAULT_Alpha)
Tokenizer_Default_Set(":;", %TOKENIZER_DEFAULT_newline)
[/code]

I'm going home now (I'm at work right now).
I need to fix some few points and than post a new update. Maybe tomorrow, sorry.

Ciao
Eros

[Michael Hartlef]

YES, I like it! You give to much!