Ciao Petr,
attached a new version of thinBasic_Tokenizer.dll module that implements
<tokenizer>.Char.Set.SubType (sDelimiters, UserSubType)
I'm sorry but due to how I implemented recognition of delimiters, a standard delimiter like []{} cannot be a keys.
But using something like
tokenizer.Char.Set.SubType ("[", 10)
you can associate a subtype number (from 1 to 255) to a single char delimiter.
In this way you can recognize special char delimiters you want to keep track among all other delimiters.
Regarding IF not working ... it is something I knew that sooner or later would have occurred but so far I have no a solution. It is a little tricky.
When thinBasic Core engine encounter an IF ... or a SELECT ... it has to decide if the next expression evaluates in a string or a numeric expression.
To achieve this it put in practice what is the so called "look ahead" technique that is the parser save the actual pointer and then look ahead for one or two tokens in order to try to understand if the next statement evaluates into a string or numeric expression. Than it goes back into the saved pointer and decide how to go on.
But now that we have dotted notation with multiple levels, understanding if the next expressions evaluates into a string or a numeric is not that simple.
Parser should go on for many tokes and also it would need a way to interrogate a module class asking if such sequence of tokens evaluate into a string or a number.
At the moment I have no way to do that. I know it is a BIG PROBLEM and I will go on trying to find a way.
A simple work around is to add en empty string "". In this way thinBasic will find a quote and immediately it will understand that the next will be a string expression
If "" & tokenizer.Token(i).Data = "[" Then
or
If "[" = tokenizer.Token(i).Data Then
otherwise it will be a numeric expression that will evaluate into
If Val(tokenizer.Token(i).Data) = Val("[") Then
and at the end the above will give
Here a complete example
#MINVERSION 1.9.16.6
Uses "Console", "Tokenizer"
String buffer = "[{""a"":1, ""b"":2},{""c"":3, ""d"":4}]"
Dim tokenizer As CTOKENIZER
tokenizer = New CTOKENIZER()
Long brackets = tokenizer.NewMainType("brackets")
tokenizer.Char.Set.Delim (",:")
'tokenizer.Keys.Add("[", brackets, 10)
'tokenizer.Keys.Add("]", brackets, 11)
'tokenizer.Keys.Add("{", brackets, 20)
'tokenizer.Keys.Add("}", brackets, 21)
tokenizer.Char.Set.SubType ("[", 10)
tokenizer.Char.Set.SubType ("]", 11)
tokenizer.Char.Set.SubType ("{", 20)
tokenizer.Char.Set.SubType ("}", 21)
tokenizer.Scan(buffer)
Long i
For i = 1 To tokenizer.Tokens.Count
If tokenizer.Token(i).MainType = %TOKENIZER_DELIMITER Then
If tokenizer.Token(i).SubType Then
PrintL "Data :" & tokenizer.Token(i).Data
PrintL "MainType:" & tokenizer.Token(i).MainType & " (" & tokenizer.Token(i).MainType.ToString & ")"
PrintL "SubType :" & tokenizer.Token(i).SubType
PrintL
End If
End If
Next
WaitKey
Reply With Quote
I think there are missing some Forum-sections as beta-testing and support 
Bookmarks