Tokenizer-Bug?

[ReneMiner]

I have some problems with classic tokenizer. It does not stop at the end. Does the End of the String not count as EOL automatic?

Make sure the last line of this example is a comment.

' #Filename "test_TokenizerFinished.tBasicU"


Uses "console", "tokenizer"

' --------------------------------------------------------------------
Function TBMain()
' --------------------------------------------------------------------
   ' init tokenizer:
  Tokenizer_Default_Char("#", %TOKENIZER_DEFAULT_ALPHA)
  Tokenizer_Default_Char("$", %TOKENIZER_DEFAULT_ALPHA)
  Tokenizer_Default_Char("%", %TOKENIZER_DEFAULT_ALPHA)
  Tokenizer_Default_Char(".", %TOKENIZER_DEFAULT_DELIM)
  
  PrintL test(Load_File(APP_SourceFullName))
  
  PrintL "done. Key to end"
  WaitKey
    
  
End Function    

' --------------------------------------------------------------------
Function test(ByVal sCode As String) As String
' --------------------------------------------------------------------
   ' usually this function scans for a ' #Filename "name.tBasic[U]"
   ' the situation was similar,
   ' within the script to test there was no 
   ' #filename found in my project.
   
   ' i simply replaced the token to search for by a dummy 
  
  Local sToken             As String
  Local lPos, lMain, lSafe As Long
  
  lPos = 1
  Do  ' Until lPos >= StrPtrLen(StrPtr(sCode))
      ' would solve the situation  
    lSafe = lPos    ' trap Error
    
    Tokenizer_GetNextToken(sCode, lPos, lMain, sToken)
    PrintL "Token " & sToken
   
    ' this would solve the situation but discards last token:
    
    ' If lPos > StrPtrLen(StrPtr(sCode)) Then 
    '  lMain = %TOKENIZER_FINISHED 
    ' EndIf
   
    Select Case lMain
      Case %TOKENIZER_FINISHED 
        PrintL "no dummy found"
        Exit Do
      Case %TOKENIZER_ERROR
        PrintL "token-error"
        lPos = lSafe
        Tokenizer_MoveToEol(sCode, lPos)
      
      Case %TOKENIZER_DELIMITER
        Print "delimiter "
        
        If sToken = "'" Then
          Tokenizer_GetNextToken(sCode, lPos, lMain, sToken)
          PrintL "' " & sToken
          
          If Ucase$(sToken) = "#DUMMY" Then
          '   no its not so I omit this... 
          EndIf
        EndIf
        Tokenizer_MoveToEol(sCode, lPos)   
' at the end of sCode lPos should be higher than Len(sCode) 
' and if requesting a lPos higher than Len(sCode)
' lMain should hold %Tokenizer_Finished
      
      Case %TOKENIZER_EOL 
        PrintL "EOL"
        Nop
      Case Else
        PrintL "move to EOL"
        Tokenizer_MoveToEol(sCode, lPos)
    End Select
  Loop

  Function = "none found"
 
  PrintL "End test()"

End Function 

' comment in the end !                   
' comment in the end !                   
' comment in the end of the code to test!

Weird guessing:

I know that i request more tokens after finding the '-delimter and send to EOL.
There's no EOL in the end i guess??? Does it always return to EOL of the previous line?

If not- shouldn't lMain hold %Tokenizer_Finished in the next loop
if lPos > Strptrlen(strptr(sCode)) then?