The Asmosphere Assembler

[Charles Pegge]

Updated 22 June 2008
Performance improvements

Oxygen / Asmosphere is now available here:

http://community.thinbasic.com/index.php?topic=1845

[Michael Hartlef]

:o :o :o W.....o......W

THANK YOU!!!!

[ErosOlmi]

"wow" is the right word.
It is far from my knowledge. It seems so complex for my poor ASM understanding.
It is a great module done with big generosity: sources are included so all can take advantages studing it.

Thanks a lot
Eros

[Charles Pegge]

There's the rest of the instruction set to come but this establishes the basic principle.

I am also working on procs and string literals so it should be possible to do this directly with system calls:

Proc MessageBox$ 0,"foo","bar",0

or

Proc GetProcAddress$ foo,"bar"

This will save a lot of assembly code lines

[kryton9]

Thanks Charles, looking forward to tinkering with this in due time!

[Petr Schreiber]

Where am I when things move here :D,

so please accept a bit delayed WOW from my side too!
I am leaving now till the evening, but looking forward to spent the night with mov, add, inc and other friends from the assembly quarter :)


Thanks a lot,
incredible job,
Petr

[Charles Pegge]

Hi Petr, you may like to try today's update which supports string literals and system calls. Doesn't look much like assembler but it saves many lines of code.

One word of caution: There is no protection against recursive macros yet. If a macro invokes itself, directly or indirectly, it will expand until your PC runs out of system memory. ;D

This is a MessageBox Hello World:
The latest update is posted at the beginning of this thread




'ASMOSPHERE DEMO
'---------------

' showing use of def (macros) and proc (procedures) to make system calls

Uses "Oxygen"


'---------------------------------------------------------------------------
'---Reference:
'---http://developer.intel.com/design/pentiumii/manuals/243191.htm
'---------------------------------------------------------------------------


DECLARE FUNCTION LoadLibrary LIB "KERNEL32.DLL" _
ALIAS "LoadLibraryA" _
(lpLibFileName AS ASCIIZ) AS LONG

DECLARE FUNCTION FreeLibrary LIB "KERNEL32.DLL" _
ALIAS "FreeLibrary" _
(BYVAL hLibModule AS DWORD) AS LONG

DECLARE FUNCTION GetProcAddress LIB "KERNEL32.DLL" _
ALIAS "GetProcAddress" _
(BYVAL hModule AS DWORD, lpProcName AS ASCIIZ) AS LONG



'SYSTEM INTERFACE
'----------------

dim iKernelLib as long = LoadLibrary("Kernel32.dll")
dim iLoadLibrary as long = GetProcAddress(iKernelLib, "LoadLibraryA")
dim iFreeLibrary as long = GetProcAddress(iKernelLib, "FreeLibrary")
dim iGetProcAddress as long = GetProcAddress(iKernelLib, "GetProcAddress")

'THINBASIC VARIABLES INTERFACE
'------------------------------

dim vv(16) as long
dim ff(16) as single


if (iLoadLibrary)and(iFreeLibrary)and(iGetProcAddress) then


'SOURCE CODE
'-----------

dim src as string="

'DEFINE MACROS

def greet `Hello World!`
def Meaning_of_Life 42
def UserLib [ebx]
def iMessageBox [ebx+04]

def LoadLibrary proc [#iLoadLibrary]
def FreeLibrary proc [#iFreeLibrary]
def GetProcAddress proc [#iGetProcAddress]
def MessageBox proc iMessagebox$


'PRESERVE VITAL REGISTERS

push ebx

'SETUP POINTER TO THINBASIC VARIABLES

mov ebx,#vv

'LOAD LIBRARY

proc [#iLoadLibrary] `LoadLibraryA`

LoadLibrary$ `User32.dll`
mov UserLib$,eax

' GET PROC ADDRESS IN VV(2)

GetProcAddress$ UserLib$ ,`MessageBoxA`
mov iMessageBox$, eax

'INVOKE

MessageBox$ 0,greet$,`Asm Proc Test`,0

'FREE LIBRARY

FreeLibrary$ UserLib$


'RESTORE VITAL REGISTERS

pop ebx

'RETURN OK VALUE

mov eax, Meaning_of_Life$
ret
"

'ASSEMBLE
'--------

dim cod as string=O2_asm(src)

'EXECUTE OR REPORT ERROR
'-----------------------

if asc(cod)=&hc3 then
MsgBox 0,"ERROR:"+$crlf+O2_Error+$crlf+$crlf+_
"CODE:"+$crlf+O2_View(src)
else
dim retval as long
retval = MC_Exec (cod)
'MsgBox 0,"RESULT"+$crlf+retval+$crlf+$crlf+_
'"CODE:"+$crlf+O2_View(src)
end if


end if

'FINISH
'------

if iKernelLib then FreeLibrary(iKernelLib)

[ErosOlmi]

That is magic Charles!
:o

[Petr Schreiber]

Thanks Charles,

this makes coding speed criticall parts a lot easier ;)


Great,
Petr

[Charles Pegge]

New update at the beginning of this thread

This example works with e the latest update which supports hi-level syntax for using system functions. eg:

MessageBox 0,greet,`Asm Proc Test`,0

or

MessageBox (0,greet,`Asm Proc Test`,0)


Return values are also supported (values returned in EAX only)

Userlib=LoadLibrary (`User32.dll`)

The brackets are optional





'ASMOSPHERE DEMO
'---------------

' showing use of def (macros) and proc (procedures) to make system calls
' Supports standard syntax used by C,Basic etc

Uses "Oxygen"


'---------------------------------------------------------------------------
'---Reference:
'---http://developer.intel.com/design/pentiumii/manuals/243191.htm
'---------------------------------------------------------------------------


DECLARE FUNCTION LoadLibrary LIB "KERNEL32.DLL" _
ALIAS "LoadLibraryA" _
(lpLibFileName AS ASCIIZ) AS LONG

DECLARE FUNCTION FreeLibrary LIB "KERNEL32.DLL" _
ALIAS "FreeLibrary" _
(BYVAL hLibModule AS DWORD) AS LONG

DECLARE FUNCTION GetProcAddress LIB "KERNEL32.DLL" _
ALIAS "GetProcAddress" _
(BYVAL hModule AS DWORD, lpProcName AS ASCIIZ) AS LONG



'SYSTEM INTERFACE
'----------------

dim iKernelLib as long = LoadLibrary("Kernel32.dll")
dim iLoadLibrary as long = GetProcAddress(iKernelLib, "LoadLibraryA")
dim iFreeLibrary as long = GetProcAddress(iKernelLib, "FreeLibrary")
dim iGetProcAddress as long = GetProcAddress(iKernelLib, "GetProcAddress")

'THINBASIC VARIABLES INTERFACE
'------------------------------

dim vv(16) as long
dim ff(16) as single


if (iLoadLibrary)and(iFreeLibrary)and(iGetProcAddress) then


'SOURCE CODE
'-----------

dim src as string="

def greet `Hello World!`
def Meaning_of_Life 420.4200e-1
def UserLib [ebp-04]
def iMessageBox [ebp-08]
def workspace 16
def LoadLibrary proc [#iLoadLibrary]
def FreeLibrary proc [#iFreeLibrary]
def GetProcAddress proc [#iGetProcAddress]
def MessageBox proc iMessagebox$
'------------------------------------
push ebp
mov ebp,esp
sub esp,workspace
Userlib=LoadLibrary (`User32.dll`)
iMessageBox=GetProcAddress (UserLib ,`MessageBoxA`)
MessageBox 0,greet,`Asm Proc Test`,0
FreeLibrary (UserLib)
mov eax, Meaning_of_Life
add esp,workspace
pop ebp
ret
"

'ASSEMBLE
'--------

dim cod as string=O2_asm(src)

'EXECUTE OR REPORT ERROR
'-----------------------

if asc(cod)=&hc3 then
MsgBox 0,"ERROR:"+$crlf+O2_Error+$crlf+$crlf+_
"CODE:"+$crlf+O2_View(src)
else
dim retval as long
retval = MC_Exec (cod)
'MsgBox 0,"RESULT"+$crlf+retval+$crlf+$crlf+_
'"CODE:"+$crlf+O2_View(src)
end if


end if

'FINISH
'------

if iKernelLib then FreeLibrary(iKernelLib)

[Petr Schreiber]

Thanks Charles,

very good improvement, we are getting higl-level here :D

One idea - what about CHM help file for Oxygen? If you download "Html Help Workshop" (http://www.microsoft.com/downloads/details.aspx?FamilyID=00535334-c8a6-452f-9aa0-d597d16580cc&displaylang=en), I can send you help file skeleton you could build your documentation on.

There is lot of new stuff added to the module, and after time it could be hard to track them, especially for newcomers.

What do you think?


Petr

[Charles Pegge]

Yes please Petr. I was thinking of doing help documentation in simple HTML since that is a universal standard but I am willing to give CHM a try. Your CHM skeleton will be essential - I did not see anything ready made in the CHM workshop for a quick start. Microsoft are expert at hiding the important things in a mound of verbage.

I hope that, apart from the rest of the instruction set, the assembler will not require many more features. Higher level languages built on top of the assembler can do that. :)

[Petr Schreiber]

Hi Charles,

here is the skeleton. Unmature, but usable :) I am thinking of creating "thinBASIC Module Documentation Wizard for HHW" during the summer. It would be set of scripts making docs editing easier.

How to use? Click thinbasic_o2.hhp to open Html Help Workshop, then click File/Compile and you will have your first CHM :)

Few notes:

functionlist.htm can be generated using included AutoGenerate_FunctionList.tbasic script
Index can be generated using included AutoGenerate_Index.tbasic script
Name command files like <nameofcommand>.htm, it will make everything easier
I recommend using PSPAD (http://www.pspad.com) for editing the HTML files, but that is up to you :)


Hope it will work for you.


Petr

[Charles Pegge]

Many thanks Petr. I can understand that. Writing a manual gives a strong incentive to keep features as simple as possible. There are only 4 O2 functions in the module so far, so the bulk of the manual will be about the features of Oxygen itself and the assembler.

Examples are probably the most important part of a manual since they are often easier to comprehend than an explanation, and can be used directly, or morphed into the required functionality.

[Michael Hartlef]

With http://www.helpndoc.com you don't need a HTML editor. It is build in. ;D

[Petr Schreiber]

I did not tried it yet,

works good! Very nice user interface, quite intuitive :)


Petr

[Charles Pegge]

Progress Report:

I'm going for the full instruction set which is dauntingly large and complex. It includes the MMX and SIMD extensions and all the legacy instructions which were necessary in 16 bit mode. I would envisage that only a fraction of these would be useful in an embedded assembler but they are going in for the sake of completeness.

The finished module should still be modest in size - for 64K, all the opcodes are in, then there remains a small amount of coding logic and error messaging to do.

The major task after this is to verify the encoding. This is not so easy since there are so many quirky groups of instructions with exceptions to the main rules.

[Petr Schreiber]

Thanks Charles,

I would be interested in SSE and SSE2 instructions, I think they are not present in any programming language inline assembler I own, so it would be very much appreciated!

I know it is a mammoth job you do on it now, so please take your time.


Petr

[kryton9]

AS Petr said Charles take your time. This is going to be some module that stands out in the field and really will add to thinBasic's wonderfulness!

[Charles Pegge]

Further Progress

After putting the instruction set, based of appendix B, I worked my way through the entire Intel manual from A to Z, refining the model and writing a test sequence for all the instructions.
There are so many permutations that it would not be practical to do all of them. - I have about 1000 test lines so far, which serves as a reasonable dry check. but this does not exclude the possibility of misinterpreting the manual. Some instructions are only usable with operating system priviledges, so will be difficult to check directly. Also I dont know whether the legacy 16bit segmented addressing mode will cope with the modern PC environment.

Once we move to 64 bit operating systems - the architecture is a lot cleaner. (Unfortunately instructions and registers of the 64 bit modes are inaccessible from 32 bit mode - the operating system locks you in!

There are a few additions to the Oxygen module and the language. You can execute an assembler script directly from file and also include other files in the script. Oxygen now does full label matching instead of on the first 2 characters.

Internally, the preprocessor is now separate from the assembler and constitutes another layer of translatable code,

Anyway I am currently working on a list of nonsense codes which look plausible but are impossible - these will be useful to check the error trapping.

PS
The code Petr wrote to work with FBGFX is now working with all 3 colors - so the problem was assembler side.

[Petr Schreiber]

Hi Charles,

thanks a lot for your work, I am looking forward the public version :)
I am also very happy that problem with colors is gone, I was investigating on that topic and could not find any logical reason.

So great!


Petr

[Michael Hartlef]

Thanks for the update Charles. :)

[kryton9]

Thanks Charles, hope to tinker with this module soon!

[Charles Pegge]

Came across some useful information about the different varieties of SIMD extension:

http://softpixel.com/~cwright/programming/simd/

It might be a good idea to add these instructions to the set at some stage even though they are vendor-specific.

Note that AMD did not support SSE2 until 2003 with the Athlon64 and Opteron processors.

[Petr Schreiber]

Hi Charles,

yes, SSE2 are "quite" new on AMDs. But if you check, many new games support GeForce 5 ( FX ) and up, which is also 2003 stuff. So for wanton developers SSE2 availability is no problem as they look no more than 5 years back :D.

I could not play lot of games on my Duron PC, because they required SSE or SSE2, do you think they gave me error message SSE not found ? No ... I realised it just because I checked my new CPU features and it seems it differed mainly in implementation of this tech.

But as in thinBASIC with Asmosphere we can decide which assembly to use on the fly, script can contain both SSE and non-SSE version and use the most appropriate one.


Thanks,
Petr

P.S. Great website!

[Charles Pegge]

Asmosphere has passed all the dry tests so I have posted the latest version to the beginning of this thread. This version has the full Intel instruction set including SSE2. I will follow on shortly with the more CPU-specific instructions including SSE3 SSE4 and 3DNow. As before the full source code is included. There is also a test list with all the instructions in various modes, but don't try to run it! That would be like going into a chemistry lab and mixing all the chemicals together.

This assembler is quite good at trapping and reporting errors - about a quarter of its code is dedicated to this function, but with the complexity of the instruction set, and the irregularities, this is essential.

My next task is to do a substantial piece of code, making use of the preprocessor, and seeing how well it stands up in This assembler is quite good at trapping and reporting errors - about a quarter of its code is dedicated to this function, but with the complexity of the instruction set, and the irregularities, this is essential.

My next task is to do a substantial piece of code, making use of the preprocessor, to see how well it stands up in practice. Something in Opengl will be a good candidate.. The preprocessor code looks very similar to Basic or C so it should be possible to borrow large hunks of source code without too many alterations.

[Petr Schreiber]

Thanks a lot Charles,

looks pretty good.
I can't wait to put my fingers on SSE!


Thanks,
Petr

[ErosOlmi]

The level you have reached is astonishing Charles.

I will release Oxygen in next thinBasic release.

Hope to see real life usage of this fantastic module, like, for example, speeding big loops calculations and other critical areas where speed is a must or the impossioble can become possible.

Thanks again.
Eros

[kryton9]

Thanks Charles. Hope we mortals can make use of this power!

[Petr Schreiber]

Hi Charles,

I started to play with SSE - found tutorial here:
http://www.neilkemp.us/v3/tutorials/SSE_Tutorial_1.html

I tried to convert following code:


// A 16byte = 128bit vector struct
struct Vector4
{
float x, y, z, w;
};

// Add two constant vectors and return the resulting vector
Vector4 SSE_Add ( const Vector4 &Op_A, const Vector4 &Op_B )
{
Vector4 Ret_Vector;

__asm
{
MOV EAX Op_A // Load pointers into CPU regs
MOV EBX, Op_B

MOVUPS XMM0, [EAX] // Move unaligned vectors to SSE regs
MOVUPS XMM1, [EBX]

ADDPS XMM0, XMM1 // Add vector elements
MOVUPS [Ret_Vector], XMM0 // Save the return vector
}

return Ret_Vector;
}


To thinBASIC:


uses "Oxygen"

type Txyzw
x as single
y as single
z as single
w as single
end type

dim v1, v2, resultV3 as Txyzw

v1.x = 0
v1.y = 0
v1.z = 0
v1.w = 0

v2.x = 1
v2.y = 1
v2.z = 1
v2.w = 1

dim SSE_Demo1 as string = "
MOV EAX, [#v1] ' Load pointers into CPU regs
MOV EBX, [#v2]

movups XMM0, [EAX] ' Move unaligned vectors to SSE regs
movups XMM1, [EBX]

addps XMM0, XMM1 ' Add vector elements
movups [#resultV3], XMM0 ' Save the return vector
ret
"

dim mc_SSE_Demo1 as string = O2_asm(SSE_Demo1)
if mc_SSE_Demo1 = chr$(&hc3) then
msgbox 0, "Assembly error"+$CRLF+O2_Error
else
MC_Exec(mc_SSE_Demo1)
msgbox 0, STR$(resultV3.x)+STR$(resultV3.y)+STR$(resultV3.z)
endif


Maybe it is UDT handling problem / EBX register use?, not sure.


Thanks,
Petr

[Charles Pegge]

Hi Petr,
Just a simple case of indirection - but guaranteed to confuse :)

EAX and EBX need the literal addresses, which is done by removing the square brackets.
(Otherwise EAX and EBX are loaded with the contents of the variables, not their addresses)

PS SSE3, SSE4 and 3DNOW are nearly ready.



uses "Oxygen"

type Txyzw
x as single
y as single
z as single
w as single
end type

dim v1, v2, resultV3 as Txyzw

v1.x = 0
v1.y = 0
v1.z = 0
v1.w = 0

v2.x = 1
v2.y = 1
v2.z = 1
v2.w = 1

dim SSE_Demo1 as string = "
MOV EAX, #v1 ' Load pointers into CPU regs
MOV EBX, #v2

movups XMM0, [EAX] ' Move unaligned vectors to SSE regs
movups XMM1, [EBX]

addps XMM0, XMM1 ' Add vector elements
movups [#resultV3], XMM0 ' Save the return vector
ret
"

dim mc_SSE_Demo1 as string = O2_asm(SSE_Demo1)
if mc_SSE_Demo1 = chr$(&hc3) then
msgbox 0, "Assembly error"+$CRLF+O2_Error
else
MC_Exec(mc_SSE_Demo1)
msgbox 0, STR$(resultV3.x)+STR$(resultV3.y)+STR$(resultV3.z)
msgbox 0,O2_View(SSE_Demo1)
endif

[ErosOlmi]

'...
if mc_SSE_Demo1 = chr$(&hc3) then
'...


To be even more clear, what abount to define a string equate in Oxygen module in place of chr$(&hc3)? So something like:


'...
if mc_SSE_Demo1 = O2_Asm_Err then
'...


Ciao
Eros

[Petr Schreiber]

Fantastic Charles,

thanks a lot!
Works as it should, the brackets in original code confused me :)

I think there are some SSE2 instructions missing, like movupd, at least according to this (http://softpixel.com/~cwright/programming/simd/sse2.php).
But no hurry, I think I will spend loong time with SSE "one".

Eros, good idea.
&hc3 looks cool, but keyword could help :)


Thanks,
Petr

[Charles Pegge]

Hi Eros,
There is a function O2_Error() which returns the last error message from preprocessor, assembler or linker. A null string is returned if the compile was successful.

This works:

If len(O2_Error) then ...

or the converse:

if O2_Error="" then ..


Thanks for your kind comments. I would be delighted to have it included in the next release of thinBasic. After the SSE3 SSE4 additions, it should be stable enough to release - there won't be any radical changes to the functionality but it is likely to be reorganised internally to make it more efficient, catch more errors etc. With a comprehensive set of tests, I can easily rework the source code without accidently breaking anything.

[Charles Pegge]

movupd does not appear in the main Intel manual, but I see it listed in the AMD64 manual - This may be a later addition with the introduction of 64bit cores. The prefix byte 66h which distinguishes it from movups is not used in the standard SSE2.

Please let me know if you find any other missing instructions. Thanks Petr.

[Charles Pegge]

I've just posted an update with the SSE3 SSE4 and 3DNOW instructions plus the SSE2 extensions I know about so far:



movupd xmm0,[ecx] ;!? 66 0f 10 01
movlpd xmm0,[ecx] ;!? 66 0f 12 01
movhpd xmm0,[ecx] ;!? 66 0f 16 01
unpcklpd xmm0,xmm1 ;!? 66 0f 14 c1
unpckhpd xmm0,[ecx] ;!? 66 0f 15 01


Understanding what they are all used for is quite another matter :)

All the instructions have been hard coded into the program. It is tempting to provide a soft coding capability to accommodate new instructions as they appear.

the zip file is posted at the start of this thread

[Petr Schreiber]

Thanks Charles,

that was fast!


Petr

[ErosOlmi]

Charles,

your current Oxygen module is now included into thinBasic 1.6.0.5 version.
I will keep it updated in future releases whenever you will update it here.

Ciao and thanks a lot for your work.
Eros

[Charles Pegge]

Thank you Eros,

I expect there will be frequent minor updates over the next few weeks, but the overall functionality is stable and further adjustments are not going to ruin anyone's assembly scripts. There is also a manual to write and many more examples to follow.

[Charles Pegge]

Another update:

Added O2_VIEW_FILE(src) function - Allows O2 code to be checked from a file source.

Support def #variable overrides (overriding parent binding) - This aids portability of asm code between ThinBasic and other embedded or standalone systems.

Fixed Hexadecimal and octal literals - these are in C notation eg: mov eax,0x2a and mov eax,0o52.

I've also included several versions of Petr's FBGFX plasma code to demonstrate various methods of handling variables - with both inline and file source code

Zip file at beginning of this thread.

[Petr Schreiber]

Thanks Charles,

sounds good!


Petr

[Charles Pegge]

Hi Petr,

I have just posted a low level counterpert of thinBsic_Oxygen, and a compiled demo of Plasma

http://www.jose.it-berater.org/smfforum/index.php?topic=1649

[Charles Pegge]

I've implemented soft coding in a new release. It's the first time I have used a hash algorithm to eliminate scanning for instruction names. In most cases, the instruction is located in a single step and all are resolved within three steps in a linked list.

The instruction names are hashed into a 32 bit code. The lower 9 bits of which are used to address a data array of 512 cells, which can be chained to others. There is a one in 4 million chance :) that the instruction set will contain 2 instructions with the same hash code so this has to be checked when the instruction set is loaded.

Anyway, apart from adding instructions being developed for new processors, it will support those instructions which have alternative names. A kernel of instructions remain hard-coded because their operand types are quite complex to resolve, but I will probably be able to move a few more of them out to the o2asm.data file.

zip file at beginning of this topic

[Petr Schreiber]

Hi Charles,

as usually, great job!
The data file should be placed in same directory as script or DLL?


Thanks,
Petr

[Charles Pegge]

Thanks Petr,

I have not specified a path for o2asm.data so it goes in the current working directory, when the o2_asm() or o2_asm_file() is invoked for the first time.
After this, the module is self-sufficient and o2asm.data is not used again until the module is reloaded.

[ErosOlmi]

Thanks Charles.
Will be present in next thinBasic release.