LoadLocalSymbols - explanation of dwReturnType

[Randall]

Hi,

I'm playing around learning module development, and I have a question about the dwReturnType parameter to thinBasic_LoadSymbol function.

Is there any documentation on what these predefined constants are used for?

thinBasic_ReturnNone
thinBasic_ReturnNumber
thinBasic_ReturnString
thinBasic_ReturnCodeByte
thinBasic_ReturnCodeInteger
thinBasic_ReturnCodeWord
thinBasic_ReturnCodeDWord
thinBasic_ReturnCodeLong
thinBasic_ReturnCodeQuad
thinBasic_ReturnCodeSingle
thinBasic_ReturnCodeDouble
thinBasic_ReturnCodeCurrency
thinBasic_ReturnCodeExt

For example, when do I use thinBasic_ReturnNumber, and when do I use thinBasic_ReturnCodeLong?

So far, I have a script that creates a long typed variable and stores a value in it, then calls a function in my module passing the memory address of the variable as a parameter. The module calls thinBasic_ParseDWord to get the address, casts it into a pointer, then dereferences the pointer and returns the contents of the memory address. I found I had to specify a dwReturnType of thinBasic_ReturnCodeLong in the LoadSymbol function for it to work.

I'm generally confused (as usual), but especially about how to use these dwReturnType constants.

What I'm working at is for the script to pass the address of an array, the data type, and number of elements, and have the module be able work with the array.

Any help or insight you may offer would be very appreciated!

Thanks,

Randall

[Petr Schreiber]

Hi Randall,

I am not 100% sure now too :), but I think if you will look at thinCore.inc ( in case of PB SDK ) there are some function which should do the job for you.

Have a look at thinBasic_ArrayGetElements and other thinBasic_Array* functions.


Bye,
Petr

[ErosOlmi]

Petr,

thanks for your help on this!
Randall is using C/C++ SDK and not all thinBasic interfaces are present in that SDK due to missing interest at the beginning, but more can be added.

Back to Randal request, that constants are used to tell thinBasic Core engine what type of numeric value the C function will return.

LoadLocalSymbols is a function that every thinBasic module MUST have as exported function. When a module is loaded, thinCore automatically invoke LoadLocalSymbols.
Programmer can use that function mainly for 3 reasons:

1. initialize any needed process in the module
2. define new keywords
3. define new equates


Now, what you are talking about is point number 2: define new keywords.
To define new keywords you have the thinBasic_LoadSymbol function available. With this function you tell thinBasic Core engine you want a new keyword and that the new keyword is connected to a module function passing the function pointer. But most important you must tell also what kind of data the module function will return. Missing to declare the correct return value type can produce GPF when executed because the process stack can get corrupted.

So, for example:



thinBasic_LoadSymbol("myC_Key",thinBasic_ReturnLong &myC_KeyFunction,thinBasic_ForceOverWrite);



tells thinCore you want a new keyword named "myC_Key" that will return a LONG numeric value and will be connected to internal module function called myC_KeyFunction. It is important that myC_KeyFunction will return a LONG value as declared here otherwise memory corruption will take place. More or less is like declaring an external function expecting a DIUBLE numeric variable passed BYREF but you will pass a LONG. Called function expect 8 bytes in the stack while in reality just four will be pushed so when a double will be read or written into the stack a corruption will take place.
The last parameter tells thinCore that even if "myC_Key" is already present as keyword, its behave will be substituted with this one.

Now on return value equates. They are related as follow:
thinBasic_ReturnNone ---> function will return nothing (void)
thinBasic_ReturnNumber ---> in C/C++ SDK this means that function will return a double
thinBasic_ReturnString ---> function will return a string (special consideration must take place here
thinBasic_ReturnCodeByte ---> function will return a byte
thinBasic_ReturnCodeInteger ---> function will return an integer
thinBasic_ReturnCodeWord ---> function will return a word
thinBasic_ReturnCodeDWord ---> function will return a dword
thinBasic_ReturnCodeLong ---> function will return a long
thinBasic_ReturnCodeQuad ---> function will return a quad
thinBasic_ReturnCodeSingle ---> function will return a single
thinBasic_ReturnCodeDouble ---> function will return a double
thinBasic_ReturnCodeCurrency ---> function will return a currency
thinBasic_ReturnCodeExt ---> function will return a ext

For the moment it is better to remain on none, long, double. Some native thinBasic data types have no direct correspondence with C or C++.

Something about arrays.
Arrays in thinBasic are managed like all other compiled languages (most of them): multiple sequences of memory blocks with the size of the base type.
So an array of 10 LONGs is a memory block of 40 bytes: 1 LONG (4 bytes) multiply 10 LONGs
If you have the array type (that will gives you the single element size), the number of elements and a pointer to the first element, you have all the info to manage the array. To get the pointer to the first byte, you need to write in your script something like:


DIM MyArray(10) AS LONG
DIM pArray AS DWORD
pArray = VARPTR(MyArray)


For the moment this is what I can tell you. In any case, consider C/C++ SDK has not been developed at the same level the Power Basic SDK has been. So if you need to have more functionalities, let us know and we will implement them. For example, as Petr suggested, Power Basic SDK has already additional functionalities to give more info to module about variables, their type, pointers, and so on.

Ciao
Eros

[ErosOlmi]

A last note about a C/C++ SDK function I forget to mention: thinBasic_ParsePtr
thinBasic_ParsePtr is a SDK function able to automatically return a pointer to a variable, even if variable is an array or array element.

So in your script you can write something like:


DIM MyArray(10) AS LONG

myC_Key(MyArray)

and in your C code use thinBasic_ParsePtr to get a pointer to MyArray data.

Hope this will help.
Eros

[RobertoBianchi]

Hi Randall,

the thinBasic_ReturnNumber constat is generic constant that should be used to return number from a module function (that is from a DLL) to the interpreter.
For having the maximum size of number the interpreter always expects to receive an PB EXT data type Extended-precision floating-point (10 bytes lenght) so if you declare your C/C++ function (let me say named FuncA()) as thinBasic_ReturnNumber in the thinBasic_LoadSymbol() call, you must return a double from FuncA().
If you use int or unsigned int it's better for clarity to use thinBasic_ReturnCodeLong or thinBasic_ReturnCodeDWord.


I found I had to specify a dwReturnType of thinBasic_ReturnCodeLong in the LoadSymbol function for it to work.

If you specify a dwReturnType of thinBasic_ReturnCodeNumber (or thinBasic_ReturnCodeDouble) in the LoadSymbol and cast the return to a double when return the contents of the memory address, il should work (of course in this case your fuction must be declared for return double).

I've just updated the C/C++ SDK with the following example:


/*
Sample
This sample shows how implement a function that receive one pointer to PB LONG
parameter, dereferences the pointer and returns the contents of the memory address
as number to the interpreter.
*/
double __cdecl Exec_ReturnNumberC(void)
{
int *p;

p = (int *) thinBasic_ParseDWord();

return (double) *p;
}

.
.
.
thinBasic_LoadSymbol("myC_ReturnNumberC", thinBasic_ReturnNumber, &Exec_ReturnNumberC, thinBasic_ForceOverWrite);
.
.
.



that you can test with the following TB script:


dim l as long

l = 7
msgbox 0, myC_ReturnNumberC varptr(l)




What I'm working at is for the script to pass the address of an array, the data type, and number of elements, and have the module be able work with the array.


As Eros said, actually if you need to handle an array you should pass a pointer to it and the number of elements:



DIM MyArray(10) AS LONG
DIM pArray AS DWORD
HandleMyArray(VARPTR(MyArray), UBOUND(MyArray)


Ciao,
Roberto

[Randall]

Thanks guys! I think I have a good understanding of the dwReturnType now, thanks to your detailed explanations. :)

I am able to get my module to work with numeric arrays defined in the script, but have a few more questions.

1. How are strings and arrays of strings stored? Do the first four bytes define the size of the string? And for an array of strings, is the array stored in contiguous memory locations?

2. In the case of UDT's, is the size of a variable just the sum of the size of each element, or is there any additional overhead?

3. Are multi-dimensional arrays stored in memory in a flat manner? For example, if an array is dimensioned as (3,3), it is stored in memory the same as an array dimensioned as (9), and the memory offset is calculated from the array indexes?

Sorry if I am not asking these questions clearly, I barely know what I am talking about, but I hope you understand.

A brief reply is sufficient, please don't waste your valuable time trying to explain it in detail. I'm sorry to be asking these newbie questions, but perhaps others will benefit from your answers.

Thanks again,

Randall

[ErosOlmi]

1. How are strings and arrays of strings stored? Do the first four bytes define the size of the string? And for an array of strings, is the array stored in contiguous memory locations?
_________________________________________________
This is the more complex one because we develop an in house way to pass strings from thinBasic Core engine to C developed module. For the moment I will just explain how strings are managed internally and will left for future explanation how we manage to pass strings from Core to C modules.

thinBasic internally store strings using OLE32 strings. You can get more info in Microsoft web site at: http://msdn2.microsoft.com/en-us/library/ms221069(VS.80).aspx and http://blogs.msdn.com/ericlippert/archive/2003/09/12/52976.aspx
So thinBasic dynamic strings are a handleto a memory area where the first 4 bytes (LONG) contains the len of the string, followed by the real strings data, followed by a null terminator.
An empty string is just a handle with zero value inside. While a not empty string is a handle pointing to a memory area allocated as follows (imagine string contains "CIAO"):


_______________
| | | | |
| | | | | ---------| Handle pointing to the following structure.
|___|___|___|___| V

___________________________________
| | | | | | | | | |
| 4 | 0 | 0 | 0 | C | I | A | O | 0 |
|___|___|___|___|___|___|___|___|___|

| | |
|_____length____|__character____|NUL|
(in bytes) data





thinBasic VARPTR(AnyString) function returns a pointer to the string handle (a LONG pointing the string) while STRPTR(AnyString) will return a pointer to the "Characted data" part of the allocated buffer.
It is important to understand that when you allocate a string, the string handle will remain always allocated in the same position while the string data will be re-allocated every time the string si changed. So VARPTR(AnyString) will always report the same value for the full life of the string while STRPTR(AnyString) will return different values if string is in the meantime changed by any string handling function. A little example to get the len of the string without using LEN function:

DIM MyString as string
dim MyStringLen as long

'---Fill the string with something
MyString = "CIAO"

'---To get the string len you can use the normal LEN(MyString)
'---But here another way. Because thinBasic uses OLE32 strings, the len is stored
'---at -4 bytes offset from the string data
MyStringLen = peek(long, strptr(MyString) - 4)

msgbox 0, "Len of string is: " & MyStringLen

'---Again with more data
MyString = repeat$(10000, "CIAO")
MyStringLen = peek(long, strptr(MyString) - 4)

msgbox 0, "Len of a bigger string is: " & MyStringLen


Also worth to mention that:

thinBasic dynamic strings (regardless of what BSTR are) are not unicode strings. Every characted is stored in single byte.
fixed strings (strings where the max len is specified in script like: DIM MyString AS STRING * 10) are not OLE32 strings but just fixed sequence of bytes managed like UDT. The same arrays of fixed strings.




_________________________________________________

2. In the case of UDT's, is the size of a variable just the sum of the size of each element, or is there any additional overhead?
_________________________________________________
Yes, an UDT is just the size of the sum of the elements. VARPTR(AnUDTVariable) returns a pointer to the first byte of the structure.
In reality every thinBasic variable has an overhead but it is stored in a different place. For every variable thinBasic have special info data stored into internal thinBasic dictionaries used to retrieve and change as fast as possible variable data. But you can forget about that for the moment.

_________________________________________________

3. Are multi-dimensional arrays stored in memory in a flat manner? For example, if an array is dimensioned as (3,3), it is stored in memory the same as an array dimensioned as (9), and the memory offset is calculated from the array indexes?
_________________________________________________
Yes, multi-dimensional arrays are stored in flat manner but in column order sequence. An array of (3,3) in memory can be over-imposed by an array of (9) but you have to understand the way elements are stored. Sequence is column order is: (1,1), (2,1), (3,1), (1,2), (2,2), (3,2), (1,3), (2,3), (3,3)
An example to clarify "over-impose" idea:


'---This create a matrix of 3x3 LONG.
'---In memory it will be a consecutive area of 9 elements multiply 4 bytes each so 36 bytes.
'---To get the memory address where data is stored, just use VARPTR(MyMatrix) or VARPTR(MyMatrix(1,1))
DIM MyMatrix(3, 3) AS LONG

'---Now we will over impose an array of 9 LONGs using AT notation.
'---This will not create a new memory area for a new array but will just create a new variable
'---in thinBasic that will share the same memory area of the above matrix.
DIM MyArray(9) AS LONG AT VARPTR(MyMatrix(1,1))

'---We will fill the array following a sequential order
dim Counter as long
for Counter = 1 to ubound(MyArray)
MyArray(Counter) = Counter
next

'---Now show result and see how sequential order results into the matrix
dim sOut as string
for Counter = 1 to ubound(MyMatrix(1))
sOut += MyMatrix(Counter,1) & $tab & MyMatrix(Counter,2) & $tab & MyMatrix(Counter,3) & $crlf
next
msgbox 0, sOut


Hope this is enough for you to start. Please feel free to ask more if you need.

Ciao
Eros

[Randall]

Wow, thanks again Eros for a very clear and detailed answer!

This is very interesting stuff!

Regarding string concatenation, if I needed to build a large string by concatenating a number of smaller strings, would it be more efficient to first allocate a static string sufficiently large to hold the completed string, and use the MID$ statement to replace sub-strings?

Of course, you would have to keep track of the length of your string, so you can increase the buffer if necessary and know where to trim it when you are done building.

Seems this would avoid having to reallocate memory for every concatenation.

Thanks again for all your help!

Randall

[ErosOlmi]

Regarding string concatenation, if I needed to build a large string by concatenating a number of smaller strings, would it be more efficient to first allocate a static string sufficiently large to hold the completed string, and use the MID$ statement to replace sub-strings?

Of course, you would have to keep track of the length of your string, so you can increase the buffer if necessary and know where to trim it when you are done building.

Seems this would avoid having to reallocate memory for every concatenation.


Yes, of course it is much more efficient to previously allocate the needed space and tha use MID$ or any other way to write into the buffer.
In any case you will discover that even concatenation is done very efficiently by the OLE32 engine. Dynamic strings are a good way top allocate memory. You can store everything you like into dynamic string as far as you keep track of it :D

Ciao
Eros