Is there any existing code that tokenizes a single line?

[TBQuerier]

Hi,

I've been all over the web, and tried some of the code from several Basic dialects. But I can't seem to find any simple code that will just take a single line/string (not a file), which might contain quoted strings, and extract out either symbolic tokens or just the elements. Bint32 was mentioned several times, but I haven't been able to locate the source for that.


The type of code I'm looking for, doesn't require while...wend or if...endif support. It would just need to be able to extract e.g.


from
sVar := "whats happening" (or sVar = "whats happening?" or sVar="whats happening?" )

into
{
"sVar",
":=",
"whats happening?"
}


Is there anything like that around? One of the problems with trying out each mention of 'tokenizer' or 'lexer' on the web, is that you never know ahead of time, whether it will handle quoted strings.


Sorry if this is off-topic or already addressed, but I really have searched a lot, for the past month.

[Petr Schreiber]

Hi TBQuerier,

I think the ThinBASIC Tokenizer module can do what you ask for. Have a look at this basic example (derived from code by Eros):

Uses "Console", "Tokenizer"

Function TBMain()
                       
  String MyBuffer        ' -- Will contain string buffer to be parsed
  Long   CurrentPosition ' -- Current buffer pointer position
  Long   TokenMainType   ' -- Will contain current token main type
  String Token           ' -- Will contain current string token
  Long   TokenSubType    ' -- Will contain current token sub type       
  
  ' -- Parser tuning   
  %CustomKeywords = 100
  %CustomKeyword_Var = 1           
  %CustomKeyword_String = 2           
  Tokenizer_KeyAdd("VAR"                  , %CustomKeywords, %CustomKeyword_Var)
  Tokenizer_KeyAdd("STRING"               , %CustomKeywords, %CustomKeyword_String)
  
  
  Tokenizer_Default_Set(";", %TOKENIZER_DEFAULT_NEWLINE)
  
  ' -- Prepare text for parsing
  MyBuffer = "var sVar : string;" +
             "sVar := ""whats happening""" 
             
  
  
  ' -- Init current buffer position. THIS IS IMPORTANT
  CurrentPosition = 1
  
  ' -- Loops until token is end of buffer
  While TokenMainType <> %TOKENIZER_FINISHED
  
    ' -- Here we are. Most important point here is that all passed parameters
    '   must be a single variable and not an expression. This is necessary because
    '   parameters are passed by reference in order to return information about token
    ' -- 
    '   MyBuffer        must contain the string you want to parse
    '   CurrentPosition must be initialized to 1. After execution this parameter will contains
    '                   current position just after current token
    '   TokenMainType   on exit, it will contain the main type of the token found
    '   Token           on exit, it will contain the string representation of the token found
    '   TokenSubType    on exit, it will contain the sub type of the token found (if relevant)
    ' -- 
    Tokenizer_GetNextToken(MyBuffer, CurrentPosition, TokenMainType, Token, TokenSubType)                                           
    
    ' -- Write some info
    PrintL LSet$(Token, 32) + DecodeType_ToString(TokenMainType, TokenSubType)    
    
  Wend
    
  PrintL "Press any key to quit..."
  WaitKey

End Function

Function DecodeType_ToString( nType As Long, nSubType As Long ) As String
  String sResult
  
  Select Case nType 
    Case %TOKENIZER_FINISHED
      Return "Tokenizer finished..."
    
    Case %TOKENIZER_ERROR
      sResult = "Error"
      
    Case %TOKENIZER_UNDEFTOK
      sResult = "Undefined token"
      
    Case %TOKENIZER_EOL
      sResult = "End of line"
    
    Case %TOKENIZER_DELIMITER
      sResult = "Delimiter"
      
    Case %TOKENIZER_NUMBER
      sResult = "Number"

    Case %TOKENIZER_STRING
      sResult = "String"

    Case %TOKENIZER_QUOTE
      sResult = "Quoted"
      
    Case %CustomKeywords
      sResult = "Custom keyword / " + Choose$(nSubType, "%CustomKeyword_Var", "%CustomKeyword_String")      
  End Select   
  
  Return sResult

End Function

You can check ThinBasic/SampleScripts/Tokenizer for 2 more examples.


Petr

[ErosOlmi]

Hi TBQuerier and welcome to thinBasic community forum.

Petr, thanks a lot for the example.
It should give TBQuerier an idea of Tokenizer module.

Tokenizer module has been designed to be enough general to adapt to any string data in order to identify tokens and transform into something else.
I developed it because I too was quite frustrated to search for simple tokenizer able to do the job without much effort in writing grammars or things like that.

[TBQuerier]

Hi TBQuerier and welcome to thinBasic community forum.

Petr, thanks a lot for the example.
It should give TBQuerier an idea of Tokenizer module.

Tokenizer module has been designed to be enough general to adapt to any string data in order to identify tokens and transform into something else.
I developed it because I too was quite frustrated to search for simple tokenizer able to do the job without much effort in writing grammars or things like that.

Perfect, thanks Ptr and Eros, I'll play around with it this week.

[TBQuerier]

Hi TBQuerier and welcome to thinBasic community forum.

Petr, thanks a lot for the example.
It should give TBQuerier an idea of Tokenizer module.

Tokenizer module has been designed to be enough general to adapt to any string data in order to identify tokens and transform into something else.
I developed it because I too was quite frustrated to search for simple tokenizer able to do the job without much effort in writing grammars or things like that.

Thanks for the foresight, Eros. We're allowed to use any of the ThinBasic DLLs with apps from other Basic dialects (Power, Pure, Oxygen, etc.). Is that correct?

[ErosOlmi]

In theory yes, I have no problem.

In practice none of the thinBasic modules (almost all the DLLs present into \thinBasic\Lib\ directory, for example thinBasic_Tokenizer.dll) can be used without the main thinBasic Core Dll called thinCore.dll because all modules calls special parsing functions present into main thinCore.dll.

So, in few words, you cannot just use functions inside the modules you need like a standard dll. You need to run a thinBasic script or embed thinCore.dll as internal scripting language into another application.

[TBQuerier]

In theory yes, I have no problem.

In practice none of the thinBasic modules (almost all the DLLs present into \thinBasic\Lib\ directory, for example thinBasic_Tokenizer.dll) can be used without the main thinBasic Core Dll called thinCore.dll because all modules calls special parsing functions present into main thinCore.dll.

So, in few words, you cannot just use functions inside the modules you need like a standard dll. You need to run a thinBasic script or embed thinCore.dll as internal scripting language into another application.

Ok, thanks Eros.