FreeBasic strings

[ErosOlmi]

Thanks to José Roca (here (http://www.jose.it-berater.org/smfforum/index.php?topic=976.msg2009#msg2009)), I'm now able to understand how to handle FB strings and how they are allocated/deallocated.

This is the structure used by FreeBasic as a string descriptor:


/** Structure containing information about a specific string.
*
* This structure hols all informations about a specific string. This is
* required to allow BASIC-style strings that may contain NUL characters.
*/
typedef struct _FBSTRING {
char *data; /**< pointer to the real string data */
int len; /**< String length. */
int size; /**< Size of allocated memory block. */
} FBSTRING;


The runtime library is written in C and they use malloc to allocate the memory for the strings.
When string is passed BYREF, a pointer to the above structure is returned, when passed BYVAL, a copy of *data must be allocated locally to functions.

So now I can think a way to handle FB string passed BYREF or BYVAL in/out from thinBasic SDK interfaces.

Ciao
Eros

[kryton9]

That is great. Both he and Sapero seem to be real brains and can do wonders with anything it seems. I am always awed by their posts and submissions on other forums.

FreeBasic looks really good. I think it will just get better and better and will be a great SDK platform for thinBasic. They must have lots of users as they already have so many libraries converted.

[ISAWHIM]

I created a few functions to simplify the slightly complex lines in the sample files...


Function TB_StrOut(StringVal As String) As BSTR
Return SysAllocStringByteLen(StringVal, Len(StringVal))
End Function

Function TB_StrIn(StringPtr As BSTR) As String
Return *Cast(ZString Ptr,StringPtr)
End Function

You pass a "StringValueText" in FreeBasic, to this... (Also in FreeBasic as a function, not a separate API.)

To use them...

' This will output "MyStringText" as the desired OLE STRING which TB needs in the return call.
Return TB_StrOut("MyStringText")

' This will take the Incoming BSTR OLE STRING which TB sent in the call, to the DLL.
' Sent inside thinBasic_ParseString(BSTRval)
SomeValue = TB_StrIn(BSTRval)


Which makes my "Test" code...

Function Exec_MyTest_StringEcho() As BSTR
Dim ParensPresent As Long
Dim BSTRval As BSTR
Dim MyString_1 As String
Dim MyString_2 As String

MyString_1 = "#1 EMPTY"
MyString_2 = "#2 EMPTY"

ParensPresent = thinBasic_CheckOpenParens_Optional
thinBasic_ParseString(BSTRval)
MyString_1 = TB_StrIn(BSTRval)
If thinBasic_CheckComma_Optional = TB_TRUE Then
thinBasic_ParseString(BSTRval)
MyString_2 = TB_StrIn(BSTRval)
End If
If ParensPresent = TB_TRUE Then thinBasic_CheckCloseParens_Mandatory

Return TB_StrOut("TestReturnString: [" & MyString_1 & " : " & MyString_2 & "]")
End Function

NOTE: Only one BSTRval is needed for all String returns... If they are set into a string, directly after each check.

Strings passed in FreeBasic are "Text Data Only"... (For those who may not be use to that.)
Translation, they stop when they hit Chr(0). They are not intended for binary data transfer.

For Binary manipulation, Pass a file-pointer, or other memory-pointer, and manipulate binary data that way. Do not attempt to pass it to the DLL, because it may loose data in the transfer. The help-file says... "Strings will skip-over chr(0) when reading data into strings." But I don't know if that applies to "Fixed-Length" or "Custom Types". I will play with that quote when I wake-up. (I can't imagine that all strings do that, because they have TCP control, which transmits binary data in the packets.)

Looks like I need to make a function to transmit Binary strings. (For small-medium strings.)

[ErosOlmi]

Jason,

I'm sorry but I had very little time in last few days and also in next 2/3 days due to high work load.
I will cehck at the BSTR as soon as I will have few time.

In the while, have a look at http://community.thinbasic.com/index.php?topic=1845.0
Download thinBasic_Oxygen.zip from Charles and have a look at SRC directory inside the ZIP. Charles have posted full Oxygen sources of his fantastic module. I'm sure you will get many tricks on how to use BSTR strings in FreeBasic

Ciao
Eros

[ISAWHIM]

No rush... I am still just playing at the moment... Trying to find all the quirks there too.

I am not 100% sure where there is an issue... but...

BSTR, should be able to transfer full binary. (I don't know why FreeBasic choose to use a format they can't even adhere to?)

The only prerequisite, listed by Microsoft, is that the length be present as 4 bytes and not include the terminating nulls in the length, and the string should be terminated with "Two" NULL characters at the end of the data length.

In any event... BSTR should not terminate at the first NULL, which is what it seems to do, from the limitation in FreeBasic. EG, they are doing it wrong, due to design issues.

I only mentioned this, because according to the BSTR format, it looks possible, but FreeBasic makes it impossible. This may also come in handy on the reverse, for those not familiar with this issue, when sending data to something expecting a BSTR value, but the data is getting corrupted in the process.

http://msdn.microsoft.com/en-us/library/ms221069.aspx

Format is...
4 bytes (String length, not including this 4 bytes, or the two nulls)
x bytes (String data, set by length, includes NULL's in data.)
2 bytes (NULL NULL)

EG, Zstring is not a valid format for BSTR, because it terminates at NULL.

Technically, String is not valid, because it contains a NULL at the end. Thus, length is actually -1 from LEN, and a NULL needs to be appended to the actual string. (I think it is the NULL at the end, and the LEN not being the real LEN, that is messing them up, or us.) EG, LEN() 40 = data size is actually 41 in memory... Add TWO nulls, = 43, but pointer reports 42... Or...LEN(memory size) = 41 plus add TWO nulls... Reads 0-43, but return data is 41, not the expected 40.

Since there is a NULL there... BSTR should only add one NULL, and subtract 1 from the actual data-LEN, so on the other side, when 4 + dataLEN + 2 = 42, which is 40 for the DATA, as expected on the return buffer.

Why would they pad a NULL onto a string in memory, on a string that has NULLs in it? I bet they have fun injecting that data into TYPE strings. They have to strip all the nulls, or inject them into the type... I bet that is fun to debug!

Anywho...

Back to trying to figure out how to get binary data (With nulls) into the DLL... I am not having luck with sending it raw pointers, because it won't let you give the "pointer value", it actually has to be a "pointer type".

Going for API next... Yummy, DLL to DLL

I looked at the oxygen stuff... but he only uses Zstring, from what I saw, which makes sense because he is processing text not binary data. I don't have time to attempt to learn assembler on top of everything I have on my plate. So the rest of the code might as well be Chinese... I just can't contemplate what it is doing. The comments may make sense to someone who programs in assembly, but if you don't, then they are as foreign as the code construct. (Not that I was looking at it with the hopes of learning. I realize it was written for its intended audience.)

I am still trying to figure out where the BSTR type is constructed... I don't see it in any of the project files, or in the FreeBasic help files.

I get a couple of pages in google when searching for, "bstr trouble", without the quotes. But it points to C and C++ resolutions.

[ErosOlmi]

Jason,

see http://msdn.microsoft.com/en-us/library/aa140182(office.10).aspx under "VB Strings" chapter.
Also see http://oreilly.com/catalog/win32api/chapter/ch06.html

thinBasic strings are like VB strings except that thinBasic do not use UNICODE format but just single byte strings plus just one null byte at the end. Null byte is automatically handled by thinBasic and its main purpose is to be able to pass compatible pointer to WIN32 API so when you pass something like:

STRPTR(MyThinBasicString)
to an API function it will not get wrong data or GPF (of course it will get till he first null byte present in the BSTR).

thinBasic has 2 functions to convert in/out from ASCII to unicode formats: ACODE$ (http://www.thinbasic.com/public/products/thinBasic/help/html/acode$.htm) and UCODE$ (http://www.thinbasic.com/public/products/thinBasic/help/html/ucode$.htm)

Back to Frebasic modules and how to deal with strings passe by thnBasic API, check in "thinBasic_FBGFX.bas" FreeBasic example that comes with thinBasic SDK for FreeBasic. Inside check function Exec_FBGFX_InKey that returnsa BSTR string back to thinBasic Core or check Exec_FBGFX_WindowTitle and Exec_FBGFX_Print functions that cast a BSTR string to a pointer to ASCIIZ sting.

If still problems, please post a LITTLE example where you face some problems and I will try to help. I'm not an expert on FreeBasic but I've done something.

Ciao
Eros

[Charles Pegge]

Hi Jason,

you can translate BSTR to FreeBasic String using this procedure. I make no assumptions about the internal descriptors of FB strings. The string may contain any characters including binary. I use assembler to copy the bytes fast and also avoid syntax complications in FB.

You will need to include the Windows API to handle Bstrings

FreeBasic Source


Function BSTR_to_FBstr( byval srcBSTR as BSTR) as string
dim as long i
dim as any ptr j
dim as string s
asm
mov eax,[srcBSTR]
mov ecx,[eax-4]
mov [i],ecx
end asm
if i>0 then
s=space$(i)
j=strptr(s)
asm
mov esi,[srcBSTR]
mov edi,[j]
mov ecx,[i] 'length of data
nexch:
mov al,[esi] ' src
mov [edi],al ' dest
inc esi
inc edi
dec ecx
jnz nexch
end asm
end if
function=s
end function



To return a BSTR from an FB string:

FreeBasic Source


Function FBString_to_BSTR(byref FBstring) as BSTR
dim as any ptr p
p=strptr(FBstring)
Function = SysAllocStringByteLen( p, len(FBstring) )
End Function

[ISAWHIM]

I was looking at that conversion...

My issue is not the BSTR translation itself... It is the fact that a "text string" is being transmitted, not a binary string. EG, Binary 1GB of data, is converted to 2GB of data, because it is treating it like universal "Text Binary" not "Binary", which is 2bytes per byte, or 32bit. (Binary is binary, and is only 8 bit, or 1 byte.)

Like the sample says... "CAT" turns into "0C0A0T" as unicode, or whatever you choose to use for the transfer. That is because "Letters" are double-wide, but binary-data is just binary-data. Unicode text is used, so that they "Translate" in the conversion from one language/format to another.

I guess I am trying to say... that I am looking for an alternative to the bloated, slow BSTR, since I am using it for binary data processing. (RAM-Binary -> Converted-to-BSTR{txt} -> PTR PASS -> Converted-from-BSTR{txt} -> RAM-Binary) is not what I am looking for. I want... (RAM-Binary -> PTR PASS -> RAM-Binary). By RAM-Binary I am saying "StringContents". TB's string contents are not a BSTR, and freeBasic's string contents are not a BSTR. Both documents make that clear. We are not talking to a MS API DLL, so I still don't get the whole BSTR thing.

I am only using "String" because it can hold the raw data. I am not using string because of the text-transfer property. Arg, I hate the moron who decided to call anything that is not a number, a string. I am sure it was a mathematician. (Why else would we have 14 numeric types, and only three string types, none completely binary safe. LOL.)

Ok, I don't want you to waste your time on this any more.

I can transfer strings (Any without Chr(0)), just fine, both ways.

I will try the other code, but I understand the comment, "You will need to include the Windows API to handle Bstrings."

I am using the project with the samples... If I need API to handle Bstrings... I would just use API to CopyMemory... I am not looking for text compatibility transfers.

I am thinking that a module might not be the solution I am looking for. Might be better for me to just make a DLL, and give them the API calls for it. This way it is not limited to only TB.

[Charles Pegge]

These functions will translate any string of data, regardless of its content since the copy length is determined by the length of the source string, not by a null marker.

While most of the BSTR / OLEstring functions available in the Windows api are oriented towards wide (2 byte) chars, they are not restricted to this format and BSTRs are used by PowerBasic and thinBasic for all dynamic string operations. Freebasic does not use them, because it has to maintain Linux compatibility.

I use the BSTR to return compiled machine code strings to thinBasic - no problem with nulls.

There is a theory in quantum physics that the entire universe is made of strings :)

[ISAWHIM]

Not strings... Spaghetti... ;D

Don't take anything I have said above, as being a frustration with ThinBasic...

My frustration completely sits with FreeBasic...

@ Charles...

I believe that is the solution I am looking for, when passing "ThisTinyDataString", to the module... But the "String" that I don't want to pass that way... is this...

MyString = FILE_Load("This2GBFile.CWAD")

The MyString is edited in ThinBasic... and that edit result is sent to MyModule... (This is my boggle)
The MyModule edits some portion, or simply reads the data, to return a complete separate formulated result, that is contrived from the edited data. (Which is not an edit to the original file, just the data in the string-memory.)

This is why I am having a hard time. I don't want the actual string duplicated, sent as a stream of binary, processed, only to be dumped a millisecond later. Or worse... have to make another streamed return trip, to replace the contents of the original string in ThinBasic. (When it could have just altered the string directly, if it knew the pointer.)

Situation... I pass the pointer... The data is edited, and the size change, which returns a new pointer...
I pass the new pointer back... but that was LOCAL, and the data dumps... Now, TB has the pointer of the DUMPED data, and the other data which was the original string, can not be DUMPED, because it replaced the old pointer (String), with the new pointer. (EG, the pointer in the module and TB are both the same, and the old TB pointer has nothing that actually points to it anymore, so it stays there forever, until the computer reboots.)

But I don't want the users of this module to have to create a whole set of code, to make the module function. When all I am looking for is the RAW-Binary data in the TB string, to be edited... without duplicating it, or fear of a massive memory leak, or crash from trying to deallocate a block of memory that was already dumped by the DLL, now attempting to be accessed by TB.

Where there is a will... there is a way...

Again, I am also looking at direct file manipulation. But that will completely insure that the process is slow, but will not have any fear of memory issues while playing string hop-scotch.

I told Petr, that I was looking to switch to PowerBasic... just to avoid all this stuff. This Unicode junk annoyed me in VB and javascript also. Text is the only time that a string is not a string, it is a suggestion of format, or a sub-string. Country and language and program and OS sensitive, and data-in does not always = data-out. Binary, data-in always = data-out... Here, on mars, on a plain, on a train, even with green eggs and ham.

It is ONLY because of my specific needs that this is even an issue. I am sure that few others will have this issue.

[Charles Pegge]

To avoid duplicating huge Bstrings, for processing in FreeBasic, you can treat them like memory files. You can copy small chunks into FreeBasic Strings or patch FBstring content directly into the Bstring (as long as it fits into the same space).

Here is a sort of mid$() for extracting FB substrings directly from the Bstring. It's basically the same code as above slightly tweaked.

FreeBasic Source


Function midBSTR_to_FBstr( byval srcBSTR as BSTR,byval mi as long, byval le as long ) as string
dim as long i,v
dim as any ptr j
dim as string s
asm
mov eax,[srcBSTR]
mov ecx,[eax-4]
mov [i],ecx
end asm
v=i-mi+1 : if le>v then le=v
if le>0 then
s=space$(le) : j=strptr(s)
asm
mov esi,[srcBSTR]
add esi,[mi]
dec esi
mov edi,[j]
mov ecx,[le] 'length of data
nexch:
mov al,[esi] ' src
mov [edi],al ' dest
inc esi
inc edi
dec ecx
jnz nexch
end asm
end if
function=s
end function

[Charles Pegge]

And this sub is for patching FBstrings into Bstrings like: mid$(bst,mi)=fbs....

Again it is based on the previous code.

FreeBasic Source


Sub midBSTR( byval sBSTR as BSTR,byval mi as long, s as string )
dim as long i,v,le
dim as any ptr j
asm
mov eax,[sBSTR]
mov ecx,[eax-4]
mov [i],ecx
end asm
le=len(s)
v=i-mi+1 : if le>v then le=v
if le>0 then
j=strptr(s)
asm
mov edi,[sBSTR]
add edi,[mi]
dec edi
mov esi,[j]
mov ecx,[le] 'length of data
nexch:
mov al,[esi] ' src
mov [edi],al ' dest
inc esi
inc edi
dec ecx
jnz nexch
end asm
end if
end sub

[ISAWHIM]

Edited... I think I got it...

THAT is what I believe I need... because I do know the MID chunks that are edited, and yes, the CWAD data that I am processing like this, is fixed-length... rather... same length. (The length changes from string to string, but the edited portions inside the strings are all the same length. Treated as ASCIIZ, for the text-names, and the DOUBLE values are all the same size also. But one may have 40 and another may have 400... but not from string-in to string-out, just from string-a to string-b, which is not done in the module.)

I will let you know if I get it to work, when I get back here again...

Focusing on the FATE and mini-server at the moment.

TY, Eros an Charles...