OK, I need help from ASM men

[ErosOlmi]

Hi all,

it is some days I'm struggled trying to find a fix on how to dynamically correctly call external CDECL functions.
Asper posted a bug in support area: http://www.thinbasic.com/community/project.php?issueid=353&filter=all#note2246
He is trying to use Blitz3d SDK library that is defined as CDECL.
I've created a PowerBasic application that use that library and all is fine but in thinBasic calling that library produces a GPF on function exit.
I'm quite sure I'm making something wrong or missing something in releasing stack used for function parameters when functions are CDECL.

As you may know I'm not an expert in ASM, so I need some help.
Can some ASM men out there help me to understand what to do in ASM for calling a CDECL function?

Attached full PB and TB examples with library.

Thanks a lot
Eros

[ErosOlmi]

I realize that without some code from my side it is quite difficult to get help, isn't it?

The following is part of the method I use to call external functions (API functions or functions contained in 3rd part libraries).
The code mainly push parameters (previously collected) into the stack and than call the external function pointer.
Code is written into PowerBasic with some inline ASM. Somewhere in there I'm missing something.

Added: code changed after Charles suggestion



'----------------------------------------------------------------------------
'Generic function for API function calling
'Parameters:
' lError : return code error
' 2 = Error calling function
' 3 = Error loading library
'----------------------------------------------------------------------------
Function DllApi_Call( ByVal lptr_tFunction As tFunction Ptr, ByRef lResultExt As Ext) As Quad
Local ApiCall_pCount As Long
Local hLib As Long
Local hProc As Long
Local RetVal As Long
Local ApiCall_PushParam As Long
Local ApiCall_PushParam_Single As Single
Local glApiLib As tAPILIB
Local ptr_lApiLib As tAPILIB Ptr
Local ApiCall_WhereFound As Long
'Local HowManyBytesPushed As Long
Local savedSP As Long


'---Save current stack pointer
! mov savedSP,esp
'---

'---Get the lib and proc address from function just in case they were already defined
hProc = @lptr_tFunction.hProc


If @lptr_tFunction.IsAPI Then
hLib = @lptr_tFunction.hLib


If hLib = 0& Then
'---Search if library is already loaded
ptr_lApiLib = HashTable_KeyFind(gScript.APILIB_Dict, @lptr_tFunction.LibName & "")

'---If not, try to load the library
If ptr_lApiLib Then
'---If already loaded, just get its handle
hLib = @ptr_lApiLib.LibHandle
Else
hLib = thinBasic_LoadLibrary(@lptr_tFunction.LibName, ApiCall_WhereFound)
If hLib Then
glApiLib.LibName = @lptr_tFunction.LibName
glApiLib.LibHandle = hLib
APILIB_Add(VarPtr(glApiLib))
End If
End If

If hLib Then
@lptr_tFunction.hLib = hLib
Else
'---
'Error loading library
'---
RunTimeError(%ERR__API_LIB_NOT_FOUND, "LibName: " & Trim$(@lptr_tFunction.LibName))
gdwStatus = %OS_ERROR_LOADLIBRARY


Exit Function

End If

End If


'---Now check hProc
If hProc = 0& Then
hProc = GetProcAddress(hLib, @lptr_tFunction.AliasName )
If hProc Then
@lptr_tFunction.hProc = hProc
Else
'---
'Error calling function
'---
RunTimeError(%ERR__API_FUNCTION_NOT_FOUND_IN_LIB, "LibName: " & Trim$(@lptr_tFunction.LibName) & " - Alias: " & Trim$(@lptr_tFunction.AliasName))
gdwStatus = %OS_ERROR_CALLFUNCTION

Exit Function

End If
End If

End If


'---PUSH parameters reverse order
For ApiCall_pCount = @lptr_tFunction.NumberOfParams To 1& Step -1&



Select Case Long @lptr_tFunction.params(ApiCall_pCount).ParamMainType

Case %ArrayType_IsNumber


If @lptr_tFunction.params(ApiCall_pCount).ParamPassBy = %PARAM_ByVal Then


Select Case Long @lptr_tFunction.params(ApiCall_pCount).ParamSubType


Case %ArrayType_Long
ApiCall_PushParam = @lptr_tFunction.params(ApiCall_pCount).DataValue.lLong
! push ApiCall_PushParam
'HowManyBytesPushed += 4


Case %ArrayType_DWord
ApiCall_PushParam = @lptr_tFunction.params(ApiCall_pCount).DataValue.lDWord
! push ApiCall_PushParam
'HowManyBytesPushed += 4


Case %ArrayType_Ext
ApiCall_PushParam = VarPtr(@lptr_tFunction.params(ApiCall_pCount).DataValue.lExt)
!mov eax,ApiCall_PushParam ;'point EAX To the 8 Byte variable
!mov edx,[eax+8] ;'Get latest 4 bytes
!push edx ;'push them
!mov edx,[eax+4] ;'Get mid 4 bytes
!push edx ;'push them
!mov edx,[eax] ;'Get the 1st 4 bytes
!push edx ;'push them
'HowManyBytesPushed += 12


Case %ArrayType_Double
ApiCall_PushParam = VarPtr(@lptr_tFunction.params(ApiCall_pCount).DataValue.lDouble)
!mov eax,ApiCall_PushParam ;'point EAX To the 8 Byte variable
!mov edx,[eax+4] ;'Get latest 4 bytes
!push edx ;'push them
!mov edx,[eax] ;'Get the 1st 4 bytes
!push edx ;'push them
'HowManyBytesPushed += 8


Case %ArrayType_Single
ApiCall_PushParam_Single = @lptr_tFunction.params(ApiCall_pCount).DataValue.lSingle
! push ApiCall_PushParam_Single
'HowManyBytesPushed += 4


Case %ArrayType_Byte
ApiCall_PushParam = @lptr_tFunction.params(ApiCall_pCount).DataValue.lByte
! push ApiCall_PushParam
'HowManyBytesPushed += 4

Case %ArrayType_Word
ApiCall_PushParam = @lptr_tFunction.params(ApiCall_pCount).DataValue.lWord
! push ApiCall_PushParam
'HowManyBytesPushed += 4

Case %ArrayType_Integer
ApiCall_PushParam = @lptr_tFunction.params(ApiCall_pCount).DataValue.lInteger
! push ApiCall_PushParam
'HowManyBytesPushed += 4


Case %ArrayType_Quad
ApiCall_PushParam = VarPtr(@lptr_tFunction.params(ApiCall_pCount).DataValue.lQuad)
!mov eax,ApiCall_PushParam ;'point EAX To the 8 Byte variable
!mov edx,[eax+4] ;'Get latest 4 bytes
!push edx ;'push them
!mov edx,[eax] ;'Get the 1st 4 bytes
!push edx ;'push them
'HowManyBytesPushed += 8


End Select

Else '---BYREF


ApiCall_PushParam = @lptr_tFunction.params(ApiCall_pCount).DataValue.GenericPtr
! push ApiCall_PushParam
'HowManyBytesPushed += 4

End If

Case %ArrayType_IsString, %ArrayType_IsGUID


'IF @lptr_tFunction.params(ApiCall_pCount).ParamPassBy = %PARAM_ByRef THEN
ApiCall_PushParam = @lptr_tFunction.params(ApiCall_pCount).DataValue.GenericPtr
! push ApiCall_PushParam
'ELSE
' SELECT CASE LONG @lptr_tFunction.params(ApiCall_pCount).ParamSubType
' CASE %ArrayType_String, %ArrayType_Asciiz
' ApiCall_PushParam = @lptr_tFunction.params(ApiCall_pCount).DataValue.lString
' ! push ApiCall_PushParam
' END SELECT
'END IF
'HowManyBytesPushed += 4


Case %ArrayType_UDT
ApiCall_PushParam = @lptr_tFunction.params(ApiCall_pCount).DataValue.GenericPtr
! push ApiCall_PushParam
'HowManyBytesPushed += 4


Case %ArrayType_Variant
If @lptr_tFunction.params(ApiCall_pCount).ParamPassBy = %PARAM_ByRef Then
ApiCall_PushParam = @lptr_tFunction.params(ApiCall_pCount).DataValue.GenericPtr
! push ApiCall_PushParam
'HowManyBytesPushed += 4
Else
'---ORIGINAL
' '---Push 16 bytes into the stack in reverse order
' ApiCall_PushParam = @lptr_tFunction.params(ApiCall_pCount).DataValue.GenericPtr
' !mov eax,ApiCall_PushParam ;point EAX To the 8 Byte variable
' !mov edx,[eax+16] ;Get latest 4 bytes
' !push edx ;push them
' !mov edx,[eax+12] ;Get latest 4 bytes
' !push edx ;push them
' !mov edx,[eax+8] ;Get latest 4 bytes
' !push edx ;push them
' !mov edx,[eax+4] ;Get mid 4 bytes
' !push edx ;push them
' !mov edx,[eax] ;Get the 1st 4 bytes
' !push edx ;push them
'---NEW
'---Push 16 bytes into the stack in reverse order
ApiCall_PushParam = @lptr_tFunction.params(ApiCall_pCount).DataValue.GenericPtr
!mov eax,ApiCall_PushParam ;'point EAX To the 16 Byte variable
!mov edx,[eax+12] ;'Get latest 4 bytes
!push edx ;'push them
!mov edx,[eax+8] ;'Get latest 4 bytes
!push edx ;'push them
!mov edx,[eax+4] ;'Get mid 4 bytes
!push edx ;'push them
!mov edx,[eax] ;'Get the 1st 4 bytes
!push edx ;'push them
'---
'HowManyBytesPushed += 16
End If
End Select
Next

'---
'Call the function and get return code
If @lptr_tFunction.ReturnMainType = %ArrayType_IsNumber Then


Select Case Long @lptr_tFunction.ReturnSubType
Case %ArrayType_Ext
Static ve As Extended
'---
'!Xor eax, eax
!Call hProc
'! fstp Qword Ptr ve
! fstp ve
lResultExt = ve


Case _
%ArrayType_Long , _
%ArrayType_DWord , _
%ArrayType_Word , _
%ArrayType_Integer, _
%ArrayType_Byte
'---
!Xor eax, eax
!Call hProc
!mov RetVal, eax
lResultExt = RetVal

Case %ArrayType_Double
Static vd As Double
'---
'!Xor eax, eax
!Call hProc
! fstp Qword Ptr vd
lResultExt = vd


Case %ArrayType_Single
Static vs As Single
'---
'!Xor eax, eax
!Call hProc
! fstp Dword Ptr vs
lResultExt = vs


Case %ArrayType_Quad
Static vq As Quad
' '---
' '!Xor eax, eax
' !Call hProc
' '! fstp Qword Ptr ve
' ! fistp vq
' lResultExt = vq


'---
'!Xor eax, eax
!Call hProc
! PUSH EBX
! LEA EBX, vq
! MOV [EBX], EAX
! MOV [EBX + 4], EDX
! POP EBX
lResultExt = vq

End Select

ElseIf @lptr_tFunction.ReturnMainType = %ArrayType_IsString Then
'---
!Xor eax, eax
!Call hProc
!mov RetVal, eax
lResultExt = RetVal

End If


'---CDECL
If @lptr_tFunction.CallingMode = %PA__CDECL Then
'---Restore previous stack position
! mov esp,savedSP
'If HowManyBytesPushed Then
' !Add esp, HowManyBytesPushed
'End If
End If


Function = RetVal
End Function

[Charles Pegge]

Hi Eros,

Very briefly.

The easy way to handle CDECL calls is to save the stack pointer before passing parameters and making the call, then restore it immediately after.

In all other respects it is the same as making a STDCALL.

dim savedSP as long
...
! mov savedSP,esp
'push params here (right to left order)
'call function here
! mov esp,savedSP

I hope this helps

Charles

[ErosOlmi]

Thanks a lot Charles.
I applied your changes but still GPF.
The strange thing is that if I start thinBasic script in debug mode and execute it using F5, all is working fine while if I run the script without debugger, it GPF just after "Graphics3D(800, 600, 32, 0)"
There must be something else I missed.

I'm going to reverse engineer PowerBasic example I did and see what is doing before/during/after external function calling compared to my code.

[ErosOlmi]

Maybe different behave between declared procedure is a SUB or a FUNCTION ?

[Charles Pegge]

It may be necessary to push all the params then make the call immediately without intervening BASIC code, since you cannot guarantee that BASIC will not alter the stack pointer in all circumstances. This could explain why it only works with the debugger, which keeps a tight grip on the stack pointer itself :)

It is not necessary to clear eax. (!xor eax,eax) before calling.

Charles

[ErosOlmi]

It may be necessary to push all the params then make the call immediately without intervening BASIC code, since you cannot guarantee that BASIC will not alter the stack pointer in all circumstances.
Yes I thought about that but this method is working with all standard API calling and with calling any DLL function I've tested so far.


This could explain why it only works with the debugger, which keeps a tight grip on the stack pointer itself :)
Yes but the debugger not a real debugger but IS MY thinBasic DEBUGGER that just do a stop/go/stop script execution semaphore :D
It is not checking anything, it just install a hook into main thinBasic Core engine telling Core "hey I'm with you". If core has such a hook, at every line change pass the control to the debugger.
Nothing else.


It is not necessary to clear eax. (!xor eax,eax) before calling.
Thanks I will change it.

[ErosOlmi]

Here it is the debugger session of the GPF
Maybe to someone this tells something

7859

[Charles Pegge]

You could try an experiment, - calling this API in pure Assembler,embedded, and see whether you can get it to work. It will help to isolate the problem.

Another broader test: create your own dll with a set of CDECL functions displaying all received params. Then you have full control of both sides.

I know how time-consuming these bugs can be. You just have to grind them down very fine.

Charles

[ErosOlmi]

I will got with option "Another broader test"
Great idea.
Thanks